Part 3 of our latest Scrapy project / Web Scraping Tutorial. Getting the links.
Web Scraping a large site with "infinite scroll" aka "AJAX load more".
(This is based on a real world project from Upwork).
*** PART 3 of 7 ***
This covers getting the links from behind each thumbnail so we can visit the page and get the details.
As also seen on one of my recent json videos I use :
response.xpath("//script[@type='application/ld+json']/text()").get();
to get the json and load it into a dictionary to parse.
By the end of this video we have 8 links from the thumbnails page in our output CSV file.
Please comment, subscribe, like, as this is taking a lot of effort and the more comments/feedback the better we can tailor future videos to you.
GitHub Code
Cheers,
Dr Pi.
#scrapy #infinitescroll #loadmore