Create an account to use AI

Create a free account to unlock all features:

Ask questions about videos with AI
Save your favorite videos
Bookmark your favorite categories
Organize your favorite subcategories
Access your personalized favorites page
Access your conversation history
Personalize your homepage with your favorites

Already have an account? Log in

Scrapy Course – Python Web Scraping for Beginners

Name: Scrapy Course – Python Web Scraping for Beginners
Uploaded: 2023-04-27T14:38:08+00:00
Duration: 4 h 37 min 9 s

freeCodeCamp.org

The Scrapy Beginners Course will teach you everything you need to learn to start scraping websites at scale using Python Scrapy.

The course covers:

- Creating your first Scrapy spider

- Crawling through websites & scraping data from each page

- Cleaning data with Items & Item Pipelines

- Saving data to CSV files, MySQL & Postgres databases

- Using fake user-agents & headers to avoid getting blocked

- Using proxies to scale up your web scraping without getting banned

- Deploying your scraper to the cloud & scheduling it to run periodically

✏️ Course created by Joe Kearney.

⭐️ Resources ⭐️

Course Resources

- Scrapy Docs: https://docs.scrapy.org/en/latest/

- Course Guide: https://thepythonscrapyplaybook.com/freecodecamp-beginner-course/

- Course Github: https://github.com/orgs/python-scrapy-playbook/repositories

- The Python Scrapy Playbook: https://thepythonscrapyplaybook.com/

Cloud Environments

- Scrapyd: https://github.com/scrapy/scrapyd

- ScrapydWeb: https://github.com/my8100/scrapydweb

- ScrapeOps Monitor & Scheduler: https://scrapeops.io/monitoring-scheduling/

- Scrapy Cloud: https://www.zyte.com/scrapy-cloud/

Proxies

- Proxy Plan Comparison Tool: https://scrapeops.io/proxy-providers/comparison/free-proxy-providers

- ScrapeOps Proxy Aggregator: https://scrapeops.io/proxy-api-aggregator/

- Smartproxy: https://smartproxy.com/deals/proxyservers/ips

❤️ Try interactive Python courses we love, right in your browser: https://scrimba.com/freeCodeCamp-Python (Made possible by a grant from our friends at Scrimba)

⭐️ Contents ⭐️

⌨️ (0:00:00) Part 1 - Scrapy & Course Introduction

⌨️ (0:08:22) Part 2 - Setup Virtual Env & Scrapy

⌨️ (0:16:28) Part 3 - Creating a Scrapy Project

⌨️ (0:28:17) Part 4 - Build your First Scrapy Spider

⌨️ (0:55:09) Part 5 - Build Discovery & Extraction Spider

⌨️ (1:20:11) Part 6 - Cleaning Data with Item Pipelines

⌨️ (1:44:19) Part 7 - Saving Data to Files & Databases

⌨️ (2:04:33) Part 8 - Fake User-Agents & Browser Headers

⌨️ (2:40:12) Part 9 - Rotating Proxies & Proxy APIs

⌨️ (3:18:12) Part 10 - Run Spiders in Cloud with Scrapyd

⌨️ (4:03:46) Part 11 - Run Spiders in Cloud with ScrapeOps

⌨️ (4:20:04) Part 12 - Run Spiders in Cloud with Scrapy Cloud

⌨️ (4:30:36) Part 13 - Conclusion & Next Steps

🎉 Thanks to our Champion and Sponsor supporters:

👾 davthecoder

👾 jedi-or-sith

👾 南宮千影

👾 Agustín Kussrow

👾 Nattira Maneerat

👾 Heather Wcislo

👾 Serhiy Kalinets

👾 Justin Hual

👾 Otis Morgan

Learn to code for free and get a developer job: https://www.freecodecamp.org

Read hundreds of articles on programming: https://freecodecamp.org/news