Copies keep a database alive — but every copy still holds ALL the data. So what happens when the data itself won't fit on one machine? You split it. This is how.
In Episode 9 of The Tech Intern · System Design, we pick up right where Episode 8 left off. Same Dabba order database — but now it's outgrown a single box, and no number of copies fixes that. We spend the episode making it GROW (sharding) and stay FAST (indexing), one deliberate trade-off at a time.
You'll walk away knowing (every term explained on first use, no jargon left dangling):
✦ Partitioning — horizontal (by rows) vs vertical (by columns), with a phone-book-at-two-counters picture
✦ Sharding vs partitioning — the difference people trip over: sharding is horizontal partitioning across SEPARATE machines ("partitioning splits the data; sharding splits it across machines")
✦ Sharding strategies — range vs hash (the fork that decides everything), plus directory, geo, composite & functional
✦ The hot-shard / celebrity problem — why one viral row melts a single shard, and how to cool it (coalesce, dedicated shard, salt, cache)
✦ Real sharding in the wild — Instagram's 64-bit ID (8,192 shards), Notion (480 shards, re-sharded with zero downtime), Discord (177→72 nodes)
✦ Rebalancing — why naive `hash(key) mod N` re-shuffles ~67% of your keys to add one machine
✦ Consistent hashing — the ring that moves only ~1/N of keys, and virtual nodes (Cassandra: 256/node)
✦ Indexing — how a B-tree turns a 23-SECOND scan into 2 MILLISECONDS on 500M rows (~10,000× faster), the read-vs-write trade-off, and local vs global indexes on a sharded DB
One running example (the Dabba food app), real numbers (all sourced), zero hand-waving.
⏭ Next episode: caching — because the fastest query is the one you never send at all.
━━━━━━━━━━━━━━━━━━━━
⏱ CHAPTERS
0:00 Why copies aren't enough (recap)
0:59 Partitioning — rows vs columns
1:53 Sharding vs partitioning: the real difference
3:08 Sharding strategy: range vs hash
4:50 Directory, geo, composite & functional
6:26 The hot-shard problem
7:41 Sharding in the wild (Instagram, Notion, Discord)
8:56 Rebalancing — the mod-N trap
10:00 Consistent hashing — the ring
11:19 Indexing — one row in a billion
12:43 Indexes aren't free (types & trade-offs)
14:31 Recap + what's next
━━━━━━━━━━━━━━━━━━━━
🔔 Subscribe for the rest of the System Design series — we're building one app from "what even is a server" all the way to designing WhatsApp, UPI and Hotstar from scratch.
#systemdesign #databases #sharding #indexing #backend #softwareengineering