Part one of a series, web scraping a recipe site using Python and Scrapy.
This video shows how I start a new Scrapy spider in Python and solve the infinite scroll 'problem'. This was not like other infinite scroll pages I had dealt with on a site like food.com.
To solve this I had to think small...and if you want to skip ahead you will find the answer around 30 minutes in, but if you do you'll miss some of the fun. This was recorded 'live' so mistakes have been left in - they may or may not help you - let me know in the comments?
If you are looking for a proxy, then you could do a lot worse than use this one, which I am impressed by and have now subscribed to. Use the promo code and you will achieve a state of eternal bliss.
10% off with this Promo Code: DRPI10
-- CHAPTER TIMES --
0:00 Start
1:20 Looking for some JSON
8:16 Writing Scrapy Code
6:49 Create Virtual Environment
8:45 'cokkpa'
10:32 'sudo chown -R rag:sudo'
17:17 Settings.py
20:16 Scrapy shell
27:57 Monkey Business
30:30 Infinite Scroll solution
Thumbs up yeah? (Cos Algos)
Dr P.
#scrapy #infinitescroll #python