Even users who have experience with web scraping may find themselves perplexed by the multitude of challenges scaling can often create. At some point, all growing scraping projects have to somehow deal with the scaling issue; therefore, having an in-depth discussion on this topic is of unparalleled importance. This is especially the case when we consider that our host, Augustinas Kalvis, has brought a very special guest with him, Eivydas VilÄŤinskas, who is a Tech Team Lead here at Oxylabs. Having worked with scaling since the very beginning of his five-year Oxylabs career, it would be difficult to find someone more suited to give us crucial insights into such a critical topic as scaling.
In this episode you’ll learn:
0:00 Intro
1:39 Eivydas’ journey with scaling
2:50 How do you go from a small service to thousands of requests per second?
4:31 Understanding the underlying concepts of scaling
5:59 Scalability example
7:35 How do I make my application work for more people?
9:40 Scraping should have light loads on the CPU
11:54 Should I monitor my application before scaling?
13:12 Tools for a small number of requests
16:07 If CPU is my bottleneck, how do I scale it?
17:01 Horizontal vs. Vertical scaling
21:06 Scalability is about the price to time ratio
22:09 What kind of request increase could a better CPU get?
25:29 Load balancers
27:56 What happens if we scale from 1 to 10 applications?
29:01 Is running more applications the best approach to scaling?
30:59 Other resources/bottlenecks and issue of standard internet connections
36:06 How do you horizontally scale a network connection?
40:29 How to get into multi-cloud solutions?
41:40 Making our scraping example/application more complex
43:28 Splitting work is key
45:38 The importance of utilizing your resources
47:30 Monolith vs. multi-service approaches
49:57 Is resource consumption the primary concern?
51:45 What other approaches are there to managing microservices?
54:44 Introducing microservices requires re-thinking but offers value
55:53 Outro
*Information provided in this episode is based on our clients' use of Oxylabs' innovative solutions and tools for scraping publicly available data.
Follow us on social media:
Oxylabs is a premium proxy service provider that offers tools and resources for public data collection. The company believes that every business, big or small, needs to have access to valuable public data.
© 2022 Oxylabs. All rights reserved.
#Oxylabs #OxyCast #Scalability