In this video, we go through how to scrape data from javascript rendered websites using Scrapy Playwright.
We cover:
How To Install Scrapy Playwright
How To Use Scrapy Playwright In Your Spiders
How To Wait For The Page To Load
How To Scrape Multiple Pages
How To Scroll The Page Elements With Scrapy Playwright
How To Take screenshots With Scrapy Playwright
The codebase we use in this video can be cloned from here:
****** RUNNING ON WINDOWS! *******
As of writing this guide, Scrapy Playwright doesn't work with Windows. However, it is possible to run it with WSL (Windows Subsystem for Linux).
This is a good video tutorial to check out if you need to install WSL on your windows machine:
00:00 - Intro
01:10 - Cloning & Installing the Scrapy Project
02:18 - Adding the settings needed for Scrapy Playwright
02:52 - Adding the scraping code to our Spider
08:50 - Wait for the page to finish loading specific page element CSS selectors
11:54 - Scraping multiple pages by clicking through to the next page
15:47 - Scraping a page with infinite scroll using Playwright
18:11 - Taking a screenshot while scraping with Playwright
20:13 - Outro