Part 4 of our latest Scrapy project / Web Scraping Tutorial. Parsing the details.
Web Scraping a large site with "infinite scroll" aka "AJAX load more".
(This is based on a real world project from Upwork).
*** PART 4 of 7 ***
This video covers the "parse_detail" function, used to extract the recipes.
Yet again we can use :
res = response.xpath("//script[@type='application/ld+json']/text()").get();
This extracts json and allows us to parse the dictionary that we get from it.
We also get the number of dictionaries in a list and iterate through each dictionary to extract the value from the 'text' KEY that we need to append to our list - the very same list which stores the recipe!
Please comment, subscribe, like, as this is taking a lot of effort and the more comments/feedback the better we can tailor future videos to you.
GitHub Code
Cheers,
Dr Pi.
#scrapy #infinitescroll #loadmore