System designAdvanced
Design a Social Feed
Users follow other users and see a timeline of their posts. The naive query is a join across 200 followees at 70k reads/s; the fix is to precompute timelines — until one account has 20M followers, which breaks the precomputation. The design is the hybrid that survives both.
Functional requirements
- Publish a post (text, up to 4 images); it appears in every follower’s feed within seconds.
- Home feed: posts from accounts the user follows, newest first (with an optional ranked mode), infinitely scrollable.
- Follow / unfollow; a profile page lists one author’s posts.
- Like and comment counts on each post.
- Deleting a post removes it from all feeds promptly.
- A user who returns after a week sees a sensible feed (not 10,000 posts, not empty).
Non-functional requirements
Scale, latency, availability and durability targets — these decide the architecture.
- Feed load p99 < 300 ms for the first page (20 items).
- A new post is visible to followers within 5 s (p99) — except for accounts with millions of followers, where 30 s is acceptable.
- Scale: 200M DAU, 40M posts/day, 2B feed reads/day; follow graph averages 200 followees, max 50M followers.
- Availability 99.9% for feed reads; posting can be briefly degraded (queued) without users noticing.
- Feed is eventually consistent; a user’s *own* post must appear in their own feed immediately (read-your-writes).
Back-of-the-envelope
Numbers first. Every component below has to be justified by one of these.
| Quantity | Value | Arithmetic |
|---|---|---|
| Post rate | ≈ 460 /s avg, 1,500 /s peak | 40M posts/day ÷ 86,400 s ≈ 463/s; peak 3× ≈ 1,400/s. Writing the post itself is trivial; distributing it is not. |
| Feed reads | ≈ 23k /s avg, 70k /s peak | 2B feed reads/day ÷ 86,400 ≈ 23,150/s; peak 3× ≈ 70k/s. At 200 followees each, a pull-on-read design would issue 200 index lookups per read → 14M index probes/s. That number kills the naive design. |
| Fan-out on write | ≈ 92k feed writes/s avg, 300k peak | 463 posts/s × 200 followers avg = 92,600 timeline appends/s; peak ~300k/s. Redis ZADD at ~100k+/s per node → 3–6 nodes for the average, sharded by user_id. |
| The celebrity post | 20M writes ≈ 67 s at 300k/s | One post from an account with 20M followers = 20M appends. At the cluster’s full 300k/s that is 67 s during which everyone else’s posts queue — and storage grows by 20M × 40 B = 800 MB for one post. Hence the hybrid. |
| Timeline cache | ≈ 1.6 TB for active users | Cache only users active in the last 7 days (~200M) × 500 entries × ~16 B (post id + score in a ZSET, before overhead) ≈ 1.6 TB; with ZSET overhead ~3 TB → ~30 Redis nodes at 100 GB. Older users rebuild on demand. |
| Post storage | ≈ 40 GB/day text, media separate | 40M × ~1 KB (text, ids, counters) = 40 GB/day → 15 TB/yr in the post store. Images (4 × 500 KB) go to object storage + CDN: up to 80 TB/day, never in the database. |
Interface
Endpoints, messages or events.
POST /posts { text, media_ids[] } → 201 { post_id }post_id is a time-sortable 64-bit id (Snowflake-style: ms timestamp + shard + sequence) so ids sort by creation time without a second column. Idempotency-Key header prevents double posts on retry.GET /feed?cursor=&limit=20 → { items[], next_cursor }The hot path. cursor is an opaque base64 of (score, post_id); never an offset. Returns hydrated posts (author, text, media URLs, counts).GET /users/{id}/posts?cursor=&limit=20Profile timeline, read straight from the post store’s (author_id, post_id DESC) index — no fan-out involved.POST /users/{id}/follow → 204 · DELETE /users/{id}/follow → 204Both idempotent. Follow triggers a small backfill (author’s last 50 posts merged into the follower’s timeline); unfollow removes them lazily on next read.POST /posts/{id}/like → 204Idempotent per (user, post); counter incremented in Redis and flushed to the DB in batches — never a synchronous UPDATE posts SET likes = likes + 1 on a viral post.DELETE /posts/{id} → 204Marks the post deleted in the store; feeds filter deleted ids at hydration time, and a background job removes the id from cached timelines.Build it one problem at a time
Each step names the problem first. Decide what you would add before revealing the reference answer.
1
The naive query and why it dies
Problem ·
SELECT * FROM posts WHERE author_id IN (…200 ids…) ORDER BY created_at DESC LIMIT 20. Each read touches 200 index ranges and merges them; at 70k reads/s that is 14M index probes/s. Measured on a mid-size Postgres: p99 of 1.8 s at 2k reads/s, CPU saturated. The requirement is 300 ms at 70k/s — two orders of magnitude off.Work through every step to unlock the data model, the request walkthrough, scaling, failure modes and the open decisions.