Create an account to use AI

Create a free account to unlock all features:

Ask questions about videos with AI
Use smart search (AI)
Save your favorite videos
Bookmark your favorite categories
Organize your favorite subcategories
Access your personalized favorites page
Access your conversation history
Personalize your homepage with your favorites

Create an account

Already have an account? Log in

Using proxies for web scraping: how to crawl without getting blocked

Apify

Getting blocked by a website for web scraping or crawling can happen quite easily if you don't think about its anti-scraping protections in advance 💡 In the first video in our series on blocking, see how you can utilize proxies to make your crawling appear as human-like as possible 🤖

📖 Full article: How to crawl without getting blocked https://blog.apify.com/crawl-without-getting-blocked/

*Useful links 🧑‍💻*

⚙️ Apify Proxy HTTP settings: https://console.apify.com/proxy/access

🌐 Learn more about Apify Proxy: https://apify.com/proxy

❓ What are the different types of proxies? https://blog.apify.com/types-of-proxies/

📝 Sign up for an Apify account: https://console.apify.com/sign-up

💰 Apify pricing: https://apify.com/pricing

🙅🏻‍♂️ Learn more about anti blocking: https://apify.com/anti-blocking

*More resources 📚*

🤖 Learn more about Apify: https://apify.com/

🛍️ Browse 1,500+ ready-made scrapers and automation tools: https://apify.com/store

👾 Join our Discord community: https://discord.com/invite/crawlee-apify-801163717915574323

In this video, you can find ⌛️

00:00 Introduction

00:33 Choosing a proxy

01:16 Adding an authenticated proxy using an HTTP library

03:28 Conclusion

#webscraping #proxy #Apify

Using proxies for web scraping: how to crawl without getting blocked | Knoovi