In this video, you will learn how to:
- Make your crawler be seen as a ‘real-user’ and not get blocked
- Select the right tools for browser automation
- Selenium vs. Puppeteer, pro and cons
- Setup Selenium and Puppeteer to work with proxy
- Proxy manipulation for an automated crawler
- Setup your Proxy Manager with Selenium and Puppeteer
How do sites know when you are using a "bot" or "crawler"?
Mostly it is due to cookies, the browser user agent and your IP.
When getting or posting information on a website, the visited website saves cookies on your browser.
The website recognizes a real browser by checking the IP address and reading the request headers which include information about the user agent.
A lack of cookies or the correct user agent may trigger the website to block you from retrieving information.
Both can be programmed to ensure this doesn’t happen.
Your IP however, is the single thing that can’t be coded since it is part of the network infrastructure.
We''ll go to ‘whoer.net’ and check what information my computer in sending over the web.
Our browser supports JavaScript and from this, the target website is able to collect the browser, Operating System, screen colors, screen Resolution, flash version, and Java support.
WebRTC is used for real-time communication from browser to browser and retrieves my IP, port, and protocol to compare with the network data.
Also, your computer time can be collected and compared to IP timezone.
We can also find the *request header* sent to the target website in the browser console by selecting F12 and going to Network tab in Chrome, or selecting ctrl+R.
Clicking on the document we will find the Header information and the request we sent to whoer.net
The request header contains the cookies and user agent which is the browser type and version.
Target websites that compare my pc information and check sessions with cookies require that when we send requests over an API that we provide a new cookie every few requests.
This as well as the exact same user agent and sometimes even the accept-language.
This is because a broken user agent or lack of user agent may in itself trigger an error response by the target website.
We can manage complex scraping operations by using automation tools such as Selenium or Puppeteer, that actually open a real browser, or even use an open source headless browser like chromium.
Browsers are a tool we can use if we need to run JS or don't want to write complex request chains myself.
The downside is that running a browser is slower and takes more RAM than using a custom script.
Automation on top of a real browser will make scraping more simple as this means we don’t need to bother collecting a database of cookies to rotate or correct user agents.
This will increase my success rate with target websites that are checking for these.
Now You may be thinking, why a headless browser?
Mainly for run the scrapping automatically.
A headless browser lacks Flash Player and digital rights management that can send more information about my pc and we can easily increase my success rate by not having them altogether.
Which automation tool should we work with?
Well, this depends on your technical skills and the website you are targeting.
For example, We will compare Puppeteer an easy to use an automation tool to Selenium with requires more technical expertise.
Puppeteer developed by Google is easy to install with one command line: npm install puppeteer It only supports Chromium headless browser and it is based on node.
Very quickly we are able to start working with Puppeteer and test a quick demo.
On the other hand, Puppeteer does not support cross-browser automation and Selenium is the most flexible tool in terms of automation and functionality out there.
Selenium supports many browsers, operating systems and programming languages such as Java, C#, Ruby, Python, and JavaScript.
Selenium will work on most target websites and can be automated for any reasonable scenario.
On the other hand, installing and using Selenium requires some technical skills in terms of understanding web technologies and APIs.
With Selenium WebDriver you can upload or download files, work with pop-ups and overcome dialog barriers.
To summarize, Both Puppeteer and Selenium support desktop web apps, responsive pages, and JavaScript.
However, Puppeteer works with only Chrome and Chromium, whereas Selenium supports all common browsers such as Chrome, Firefox, Explorer and other headless browsers.
We would like to mention that Puppeteer and Selenium DO NOT support native mobile applications out of the box.
However, Appium developed by JS foundation or Selendroid developed by eBay software foundation will allow you to automate native mobile applications.
Both Selendroid and Appium require Selenium and the knowledge of how to work with Selenium API and WebDriver.