This session, titled "Geospatial Visualisation with Python, Web Scraping, and Folium" organized by PyData Milton Keynes, features a presentation by Hugh Evans, a Senior Developer Advocate at Aiven. He discusses a personal project where he built custom, interactive maps to track and visualize tech meetup groups around the world.
Here is a summary of the key topics and techniques covered in the presentation:
The Inspiration and Problem: Hugh's goal was to speak at every PyData group in the UK within a year. However, he found that the official map provided by Meetup had several issues, such as placing cities in incorrect locations and failing to show whether a group was currently active or inactive [10:08].
Web Scraping with Playwright: To build a better map, he started by scraping data directly from Meetup. He used Playwright, a headless browser automation tool, to navigate the site, load the elements, and extract information like group names, locations, and upcoming events [13:41].
Data Cleaning and Geoencoding: Because the original coordinates from Meetup were flawed, Hugh used the geopy library and the Nominatim API. By feeding the city and country names into the API, he could retrieve accurate longitude and latitude coordinates to properly place the groups on his map [16:39].
Building Maps with Folium: With the cleaned data in a Pandas DataFrame, he used the folium library to generate interactive maps. He added custom HTML components to create tooltips that display group information on hover, and he implemented color-coding to visually distinguish between highly active groups, inactive groups, and brand-new communities [23:29].
Scaling the Pipeline with Apache Kafka: As the project grew from mapping just UK PyData groups to mapping tens of thousands of tech meetups globally, his single Python scraping script became far too slow. To solve this, he redesigned his architecture using Apache Kafka as a distributed message broker. This allowed him to scale the workload horizontally across multiple worker processes, cutting the scraping time down from days to just a few hours [35:57].
Hosting and Deployment: Finally, he discusses how he hosts the resulting interactive maps and search tools entirely for free using GitHub Pages and a lightweight HTML/CSS frontend backed by a cached JSON data file [51:24].
Here's some links to resources as promised(by Hugh Evans):