Part 6 of our latest Scrapy project / Web Scraping Tutorial. Getting the "next page"
Web Scraping a large site with "infinite scroll" aka "AJAX load more".
(This is based on a real world project from Upwork).
*** PART 6 of 7 ***
This video covers the the code to handle "next page" - we need to know when to stop - when is there no next page? We check if @rel="next" is true.
Please comment, subscribe, like, as this is taking a lot of effort and the more comments/feedback the better we can tailor future videos to you.
GitHub Code
Cheers,
Dr Pi.
#scrapy #infinitescroll #loadmore