Web scraping has changed and a plain HTTP request with an HTML parser won't get you far anymore. Let Ondra, Head of Delivery Engineering @ Apify, take you on a journey of the latest tech that his team uses every day for web scraping. Headless browsers, fingerprints, generated headers, residential proxies, mobile app traffic monitoring, and other tools can mean the difference between scraping 10 pages and 10 million pages. And don’t worry, it’s perfectly legal - you can check for yourself in Ondra's recent article about the legality of web scraping:
https://blog.apify.com/is-web-scraping-legal/A little bit more about Ondra:
Ondra may be a hacker of the browser age, but that doesn‘t mean he crashes systems, steals personal data, or trades secrets. Instead, he specializes in extracting terabytes of publicly available data and translating them into a language that machines can understand. At Apify, Ondra leads a team of fellow hackers that helps grow their open source projects, breaks anti-scraping walls, and dabbles in AI. If you'd like to join his team, we are hiring!
https://apify.com/jobsThis video is a recording of Ondra's speech he delivered at WebExpo 2021 and it is published with permission from WebExpo Team.
#webscraping #webexpo #Apify