Last time we designed the Twitter timeline; this time we go upstream — the front half of a search engine: a web crawler. A program that downloads the entire internet. It sounds insane, but the whole design comes down to three parts: a never-ending URL frontier, a "have I seen this?" filter (the Bloom filter, the star of this episode), and politeness so you don't take sites down. Season two, design problem two.
Do the math first: a billion links to crawl, four billion pages a month, five hundred KB a page, two petabytes a month, seventy-two petabytes over three years, sixteen hundred writes a second. Note — a crawler is bottlenecked on bandwidth, not CPU.
· The loop: take a URL from the queue → fetch → parse → extract new links → enqueue them back. One URL is linked from countless pages; skip dedup and you loop forever, burning bandwidth.
· The Bloom filter (the star): "have I seen this URL?" An exact set stores every URL string — 106 MB for a million URLs, tens of GB at a billion, won't fit in RAM. A Bloom filter uses a fixed 1.2 MB bit array and seven hashes; the answer is only ever "definitely new" or "probably seen." It's about eighty-nine times less memory, at the cost of a ~0.96% false-positive rate — but the false-negative rate is always zero, so it never re-crawls a seen URL. How: k hashes map to k bits; any bit 0 means definitely new, all 1 means probably seen. No false negatives, only occasional false positives — the asymmetry a crawler wants.
· URL frontier + politeness: real links are skewed, one hub host is sixty percent of them. A naive FIFO hits one host 136 of the first 200 fetches (ten in a row) — indistinguishable from a DoS, IP banned. A polite frontier round-robins per host: only ten hits, at most one in a row. The frontier is a politeness scheduler.
· Content dedup: the same article appears under many URLs (session ids, ?print=1, mirror domains). A content signature catches exact duplicates, SimHash catches near-duplicates — and kills crawler traps (infinite calendars).
How AI moves it: (1) Extraction: a regex/CSS scraper is pinned to the HTML structure and breaks on a redesign (demo: regex 4/4 → 0/4), while an LLM extractor works from meaning and stays 4/4 on both layouts. (2) The bigger point: the crawl itself is the training corpus — Common Crawl fed nearly every LLM. AI both consumes the crawl's output and upgrades its parser. The honest note: AI does not change the frontier, the Bloom dedup, or politeness — dedup over a billion URLs is still the Bloom filter's job, not the model's. You would never ask an LLM "have I seen this URL?"
A crawler = frontier + Bloom + politeness, once again assembled from earlier episodes' parts (queue, dedup, cache, shard). All four demos are stdlib, real-run. Code + written walkthrough: github.com/vicenteliu/system-design-ai-era (exercises/web-crawler).
00:00 Design a web crawler
00:38 Estimate · bandwidth is the bottleneck
01:05 The crawl loop · dedup or loop forever
01:38 Bloom filter · 89x less memory
02:30 How Bloom works · no false negatives
03:11 URL frontier · politeness scheduling
04:00 Content dedup · SimHash
04:45 AI · LLM extraction + training data
06:21 The honest note · AI keeps the skeleton
06:43 The traps
07:29 Architecture map + close
—
Derived from donnemartin/system-design-primer (CC BY 4.0) — form, not text.
Borrow My Brain