You type 280 characters. You hit Tweet. In less than a second, that thought appears on the timelines of every single person who follows you.
If you happen to be a celebrity with 50 million followers — that one tweet needs to reach 50 million people almost instantly. How do you do that without your servers catching fire?
Twitter (now rebranded as X) is one of the most technically fascinating and uniquely challenging systems in existence. The core problem — delivering a tweet at massive scale in real time — led to some of the most creative engineering solutions the industry has ever seen. Let's unpack every single one of them, together.
🐦 500 million+ tweets posted every single day
👥 350 million+ monthly active users globally
⚡ Average tweet delivery latency: under 5 seconds to all followers
🔥 Peak tweet rate record: 143,199 tweets per second (New Year 2013 Japan)
🌐 Twitter processes 18+ billion API requests per day
❤️ 6,000 likes happen on Twitter every second
📺 Twitter's search index handles 600 million queries per day
The unique challenge of Twitter: a single event (World Cup goal, celebrity tweet) can cause a traffic spike of 10–20x in under one second. Nobody else deals with this.
🏛️ Section 1: Think of Twitter Like a Global Town Square + Postal System
Before servers and databases, let's build the right mental model.
Imagine a giant town square where billions of people stand. When someone shouts something, their followers — people who specifically chose to listen to them — immediately hear it. The louder (more followers) the person, the harder it is to relay their message to everyone at once.
A regular person shouting in the square (100 followers) is easy to handle. But when Elon Musk shouts (150 million+ followers)... you need a completely different strategy to relay that message to 150 million ears simultaneously.
| 🏛️ Town Square Analogy | 🐦 Twitter Reality | 🔧 Engineering Concept |
|---|---|---|
| Person shouts an announcement | Posting a tweet | Write Path |
| Relay runners carry the message to each follower | Tweet pushed to all followers | Fan-out on Write |
| Each person goes to check the noticeboard | You open your timeline | Fan-out on Read |
| Celebrity speaks → relay problem for 50M ears | Elon tweets → 150M timeline updates needed | Celebrity / Hot-Key Problem |
| Town crier announces "trending topics" | The Trending section (#WorldCup) | Real-Time Aggregation |
| Each person's personal mailbox of messages | Your Home Timeline | Timeline Cache (Redis) |
🗺️ Section 2: The Big Picture — Twitter's Full Architecture
Twitter is built on a set of interconnected services. Let's see the bird's-eye view first, then zoom into each component one by one.
🏗️ Twitter High-Level Architecture — Animated
Mobile App
Web Browser
Third-Party Apps (API)
Tweet
Service
Timeline
Service
User
Service
Search
Service
Notif.
Service
Trending
Service
Media
Service
Pushes every tweet to all follower timelines
Redis
Timeline Cache
MySQL
Tweets/Users
Manhattan
KV Store
Earlybird
Search Index
Kafka
Event Bus
Blob Store
Media Files
We will deep-dive into every single component above. Keep reading! 👇
❄️ Section 3: Snowflake IDs — The Genius of Twitter's Tweet IDs
Before we talk about tweets being posted or delivered, let's answer a fundamental question: how does Twitter give every tweet a unique ID?
Twitter posts 500 million tweets a day. Every tweet needs a unique identifier. A simple auto-incrementing number (1, 2, 3, 4...) seems obvious — but it creates a massive bottleneck when thousands of tweets arrive per second.
Twitter's solution is called the Snowflake algorithm. It's one of the most elegant pieces of engineering in modern software.
A phone number has different parts: country code + area code + local number. Each part means something. Combine them → globally unique number. No central authority needed!
Snowflake IDs work the same way. A tweet ID is a 64-bit number split into meaningful parts:
❄️ Inside a 64-bit Snowflake Tweet ID
41 bits
Milliseconds since
Twitter's epoch (2006)
10 bits
Which server
generated this
12 bits
Counter within
same millisecond
🔹 No coordination needed — each server generates its own IDs independently
🔹 Naturally time-sortable — sort IDs and you sort by time automatically
🔹 Globally unique — no two servers can ever generate the same ID
🔹 Compact — 64 bits fits perfectly into a database integer column
🔹 Insanely fast — no database roundtrip to generate an ID
Snowflake is now open-source and used by Instagram, Discord, Mastodon, and dozens of other platforms!
🐦 Section 4: What Happens When You Post a Tweet?
You type "Just finished the best pizza of my life 🍕" and hit Tweet. Let's trace exactly what happens on Twitter's side — step by step.
📤 The Tweet Write Path — Every Step
Your tweet content travels over HTTPS to Twitter's load balancer, which routes it to the least-busy Tweet Service server.
Are you logged in? Are you posting too fast? (Twitter limits: 300 tweets per 3 hours.) Basic content checks happen here too (length ≤ 280 chars, no banned content).
A Snowflake ID is generated instantly. The tweet — ID, text, author, timestamp, media references — is written to the MySQL tweets database (sharded by tweet ID). This is the permanent record. Your screen shows the tweet immediately.
The Tweet Service publishes an event to Kafka:
{tweet_id, author_id, text, timestamp}.
This event is picked up by multiple downstream systems simultaneously:
the Fan-out Service, the Search Indexer, the Trending Service, the Notification Service.
The Fan-out Service looks up your follower list. For each follower, it prepends your tweet ID to their timeline cache in Redis. This is the most complex, expensive, and fascinating step. Let's deep dive into it next!
Simultaneously: the tweet is indexed in Earlybird (real-time search), hashtags are counted for trending, and push notifications are sent to followers who have notifications enabled for your account. All in parallel via Kafka.
🌊 Section 5: The Fan-Out Problem — Twitter's #1 Engineering Challenge
This is the most famous and most difficult problem in Twitter's architecture. Nothing else comes close. Understanding fan-out is understanding Twitter's soul.
"Fan-out" is what happens when ONE event needs to trigger updates for MANY recipients. When you post a tweet, your followers' timelines need to be updated. If you have 1,000 followers → 1,000 timeline updates. If you have 50 million followers → 50 million timeline updates. From one tweet.
Now imagine Lady Gaga and 10 other celebrities all tweeting in the same second. That's potentially hundreds of millions of timeline operations per second. This is the fan-out problem.
📝 Approach 1: Fan-Out on WRITE
When you tweet, immediately push the tweet ID to every follower's Redis timeline cache.
📖 Approach 2: Fan-Out on READ
When you open Twitter, dynamically fetch recent tweets from all your followees.
🏆 Twitter's Hybrid Solution — The Best of Both Worlds
Twitter realised: neither approach alone works. So they combined both intelligently. This hybrid approach is one of the greatest examples of pragmatic engineering in the industry.
🏆 Twitter's Hybrid Fan-Out Strategy
Use Fan-out on Write. When they tweet, the Fan-out Service immediately pushes the tweet ID to all their followers' Redis timeline caches. This is fast and the write cost is manageable.
Use Fan-out on Read (lazy loading). Their tweet is NOT pushed to 50 million timelines. Instead, when YOU open your timeline, Twitter sees you follow a celebrity and dynamically fetches their latest tweets and merges them in. The tweet reaches you but WITHOUT a catastrophic write storm.
When you load your home timeline, Twitter's Timeline Service: (1) Reads your pre-computed Redis cache (tweets from regular users you follow), (2) Dynamically fetches recent tweets from any celebrities you follow, (3) Merges and sorts everything by time, (4) Returns a perfectly blended timeline. You see it as one seamless feed. The complexity is completely hidden from you!
🔢 The Numbers Behind Fan-Out at Twitter Scale
User has 700 followers
Fan-out: 700 Redis writes
Easy ✅
50,000 followers
Fan-out: 50K Redis writes
Manageable ⚠️
50M followers
Fan-out: 50M Redis writes
CATASTROPHIC 💥
50M followers
Fan-out: 0 writes!
Elegant ✅
🏠 Section 6: How Your Home Timeline is Assembled
When you open Twitter, how does it build that perfectly sorted stream of tweets in milliseconds? Let's trace the read path.
🏠 Timeline Assembly — What Happens When You Open Twitter
Your Redis key (e.g.,
timeline:user:987654)
holds a sorted list of up to 800 recent tweet IDs from regular followees.
This lookup takes microseconds.
For every celebrity you follow, query their personal "celebrity tweet cache" in Redis for tweets from the last 24 hours. Merge these into the timeline.
The timeline so far is just a list of tweet IDs (very compact in Redis). Now the Timeline Service fetches the full tweet details (text, author info, likes count, retweet count, media) from MySQL. This is done in one batch query — not 800 individual queries!
For Twitter's "For You" algorithm tab, a recommendation model suggests tweets from people you DON'T follow but might be interested in. These are blended into the feed based on engagement predictions.
The full merged set is sorted (by time for "Following" tab, by engagement for "For You"), filtered (block/mute rules applied), and returned to your app. All of this in well under 100ms!
Twitter only keeps the most recent 800 tweet IDs per user in their Redis timeline cache. If you scroll down far enough to go beyond those 800 tweets, Twitter dynamically fetches older tweets from MySQL (slower, but this happens very rarely — who reads 800 old tweets?).
This means Redis only holds tweet IDs (not full tweets!) per user. A tweet ID is 8 bytes. 800 IDs × 350M users = about 2.2TB of Redis memory. That's very manageable compared to storing full tweet text for everyone!
🔍 Section 7: Real-Time Search — Twitter's Unique Superpower
Google indexes the web and results are available within hours or days. Twitter's search shows you tweets from the last 10 seconds. This is an entirely different engineering challenge.
When a plane makes an emergency landing, people tweet about it. Within seconds, searching "plane landing" on Twitter returns live eyewitness tweets. No other platform in the world can do this — show you content posted 5 seconds ago in a search result.
This is what Twitter calls a "real-time index". It's powered by a custom search engine they built called Earlybird.
🔍 How Earlybird Real-Time Search Works
Milliseconds after you tweet, the tweet text lands in Kafka.
Earlybird (built on Apache Lucene) tokenises the tweet — splits it into individual words. Each word is added to an inverted index in memory (RAM). An inverted index maps: word → list of tweet IDs containing that word. "plane" → [tweet#12345, tweet#67890, ...]
Earlybird looks up both words in the in-memory index simultaneously. It finds ALL tweets matching both words from the last 7 days. Results are ranked by: relevance, freshness, engagement (likes, retweets), and author influence.
Because the index lives in RAM (not on disk), queries complete in under 50ms. The entire recent Twitter corpus is spread across hundreds of Earlybird instances. Each instance handles a shard of recent tweets. Queries are broadcast to all shards in parallel.
📈 Section 8: Trending Topics — How Twitter Spots What's Hot
Every few minutes, the trending topics list shifts. How does Twitter figure out, in real time, what the whole world is suddenly talking about?
📈 How Trending Topics Are Calculated
The Trending Service consumes from Kafka. Every tweet that arrives has its hashtags and key phrases extracted and counted in a sliding time window (e.g., last 60 minutes, grouped in 5-minute buckets).
"Cricket" might always be popular in India. Trending is about sudden spikes — terms whose usage jumped dramatically compared to the previous time window. A 1,000% spike beats steady popularity.
Trending is local. What's trending in Mumbai is different from London. The Trending Service maintains separate trend counts per country and major city. Your location determines which trending list you see.
Bots can artificially inflate a hashtag. Twitter's systems analyse: are these tweets from real, diverse accounts? Coordinated inauthentic behaviour is filtered before a topic can trend.
📊 Velocity vs. Volume — Why "Soccer" Doesn't Always Trend
Hour 2: 51,000
Hour 3: 49,000
Steady → NOT trending ❌
Hour 2: 8,000
Hour 3: 890,000 ← GOAL! 🔥
17,700% spike → TRENDING! ✅
🗄️ Section 9: Databases — Twitter's Data Layer
Twitter uses different databases for different jobs. Let's explore each one and understand exactly why it was chosen.
Core relational data. Tweets, user profiles, follower relationships. MySQL is sharded using a tool Twitter built called Gizzard — which manages partitioning across hundreds of MySQL instances. Sharding is done by user ID for follows, and by tweet ID for tweets.
The beating heart of Twitter's timeline system. Each active user has a Redis list containing their 800 most recent timeline tweet IDs. Also used for: rate limiting counters, session storage, trending caches, and social graph caches (Flock). Twitter runs a massive Redis cluster — one of the largest in the world.
Twitter built their own distributed key-value database called Manhattan (inspired by Cassandra and DynamoDB). It stores: tweet counts (likes, retweets, views), direct message metadata, ad impressions, and user preferences. Manhattan is optimised for Twitter's specific access patterns and is highly tuned.
All Twitter events flow to HDFS (Hadoop Distributed File System) for long-term storage. Data scientists use Apache Hive and Spark to run big-batch analysis: training recommendation models, detecting spam patterns, analysing engagement trends. This is the "cold storage" analytical layer — not used for real-time serving.
A custom distributed in-memory search engine. Keeps the last 7 days of tweets fully indexed in RAM across hundreds of servers. Supports full-text search, hashtag search, and user search with sub-second latency. Older content (7+ days) is served from a slower but complete archive index on disk.
Every tweet, like, retweet, follow, and click generates an event. Kafka is the backbone connecting all services. Fan-out workers, search indexers, trending counters, notification systems, analytics — all consume from Kafka topics. Twitter's Kafka cluster handles billions of events every day.
🖼️ Section 10: Handling Media — Photos, Videos, and GIFs
Text tweets are tiny. But Twitter serves billions of images and videos daily. Media requires a completely different pipeline.
🖼️ Image Upload to Delivery Pipeline
a photo
Blob Storage (S3)
multiple sizes
(thumbnail, medium, original)
(global edge servers)
image fast from
nearby CDN node
Twitter uses WebP format for images (30–40% smaller than JPEG). Videos use H.264/H.265 and are adaptive-bitrate streamed (just like Netflix, but shorter clips). GIFs are silently converted to MP4 videos — much smaller file size, same visual result!
A GIF file of a 3-second animation might be 5MB. The same animation as an MP4 video is 200KB — 25x smaller!
When you "attach a GIF" on Twitter, the platform secretly converts it to MP4. Your browser plays it as a silent looping video. You see a GIF experience. Twitter serves 25x less data. Everybody wins! 🎉
🔔 Section 11: Notifications — The Right Alert at the Right Time
When someone likes your tweet, you get a notification almost instantly. How does the notification system work at Twitter's scale?
🔔 Notification Pipeline — Like → Your Phone in Under 3 Seconds
🚦 Section 12: Rate Limiting — Protecting Twitter from Abuse
Without rate limiting, a single bot or bad actor could post millions of tweets per second and crash Twitter's servers. Rate limiting is the bouncer at the door.
🚦 Twitter's Rate Limits (Examples)
| Action | Limit | Window | Why? |
|---|---|---|---|
| Post a tweet | 300 | 3 hours | Prevent spam |
| Follow accounts | 400 | Per day | Prevent mass following bots |
| Like tweets | 1,000 | Per day | Prevent automated engagement |
| API Reads (free) | 500K | Per month | Control 3rd party API cost |
| Timeline reads (X Premium) | Unlimited | — | Revenue model |
Imagine each user has a bucket of tokens. Every action (tweet, like, follow) uses up one token. Tokens refill at a fixed rate (e.g., 300 per 3 hours for tweets).
The token count for each user is stored in Redis (fast in-memory lookup, sub-millisecond).
When you hit the rate limit, your bucket is empty. Twitter returns HTTP 429 Too Many Requests and you must wait for your tokens to refill.
📈 Section 13: Scalability — Handling 500M Tweets and Sudden Spikes
Twitter's hardest scalability challenge isn't the average load. It's the sudden spikes — a World Cup goal can trigger 10x normal traffic in under one second. No warning. No ramp-up time.
Every tweet gets a unique ID without any central counter or database lookup. Each server generates its own IDs independently. This means ID generation scales infinitely with the number of servers — no single bottleneck.
Timeline reads are the most frequent operation (millions per second). By pre-computing timelines into Redis, each read is a simple in-memory list operation — taking microseconds. Twitter's Redis cluster serves millions of timeline reads per second across a large cluster of Redis nodes.
Fan-out (writing to follower timelines) happens asynchronously via Kafka workers. When you tweet, the response to your app is instant — the fan-out happens in the background. If there's a spike, the Kafka queue absorbs it. Fan-out workers process at their own pace. The system stays stable even under huge traffic bursts.
Twitter's MySQL stores billions of tweets. By sharding (splitting) across hundreds of MySQL instances, each individual database stays small and fast. Gizzard (Twitter's sharding framework) manages routing queries to the correct shard automatically.
All of Twitter's microservices are stateless — they don't remember anything between requests. All state lives in Redis, MySQL, or Kafka. During a traffic spike, Twitter can spin up 100 more Tweet Service instances in minutes. The load balancer automatically distributes traffic to the new instances.
💻 Section 14: A Peek at the Code — Simplified Examples
Let's look at simplified pseudocode that brings the core concepts to life. These are teaching examples to illustrate the architecture — not real Twitter code!
This shows what happens inside the Tweet Service when you post a tweet. It does three things in order: (1) generates a unique Snowflake ID so the tweet has a unique identity, (2) saves the tweet to the MySQL database (the permanent record), (3) publishes an event to Kafka so other systems can react to the new tweet asynchronously. Notice the tweet is saved to the database AND returned to the user BEFORE fan-out happens. Fan-out is somebody else's job — done asynchronously in the background.
# Tweet Service — Handle Post Tweet Request (Pseudocode) def post_tweet(user_id, tweet_text, media_ids=[]): # Step 1: Validate content if len(tweet_text) > 280: return error("Tweet too long!") # Step 2: Rate limit check (token bucket in Redis) # Each user has 300 tweet tokens per 3 hours if not rate_limiter.consume(user_id, action="tweet"): return error(status=429, msg="Rate limit exceeded. Try later.") # Step 3: Generate unique Snowflake ID (no DB roundtrip needed!) tweet_id = snowflake_generator.next_id() # e.g. 1683726382837190656 # Step 4: Save tweet to MySQL (sharded by tweet_id) shard = tweet_id % NUM_SHARDS # find which DB shard to write to mysql[shard].insert("tweets", { "id": tweet_id, "user_id": user_id, "text": tweet_text, "media_ids": media_ids, "created_at": now() }) # Step 5: Publish to Kafka — fan-out, search, trending all read from here! kafka.publish(topic="new-tweets", message={ "tweet_id": tweet_id, "user_id": user_id, "text": tweet_text, "timestamp": now() }) # Return to user immediately — DON'T wait for fan-out (async!) return success({"tweet_id": tweet_id, "url": f"twitter.com/tweet/{tweet_id}"})
This is the Fan-out Worker — Twitter's most important background process. It runs continuously, consuming tweet events from Kafka. For each new tweet, it does the following: it looks up all the followers of the tweet's author, checks whether each follower is active (has used Twitter recently), and for active users, it prepends the tweet ID to their personal Redis timeline list. For celebrity accounts (too many followers), it uses the "fan-out on read" approach instead. This is the code that makes your timeline feel real-time!
# Fan-Out Worker — Distributes Tweets to Follower Timelines (Pseudocode) # This runs on thousands of worker machines, consuming from Kafka CELEBRITY_THRESHOLD = 10_000 # accounts with more followers use read-time fan-out TIMELINE_MAX_SIZE = 800 # max tweet IDs stored per user in Redis ACTIVE_USER_DAYS = 30 # only fan-out to users active in last 30 days def fanout_worker(): # Continuously listen to Kafka for new tweets while True: event = kafka.consume("new-tweets") tweet_id = event["tweet_id"] author_id = event["user_id"] # Check if author is a celebrity (too many followers) follower_count = user_service.get_follower_count(author_id) if follower_count >= CELEBRITY_THRESHOLD: # Celebrity path: just update their own tweet list cache. # Followers will fetch it themselves when they open Twitter. redis.lpush(f"user_tweets:{author_id}", tweet_id) redis.ltrim(f"user_tweets:{author_id}", 0, TIMELINE_MAX_SIZE) continue # done — no expensive fan-out needed! # Regular user path: push to all followers' timelines followers = social_graph.get_followers(author_id) # from Flock cache for follower_id in followers: # Only fan-out to ACTIVE users (saves memory!) # Inactive users rebuild their timeline from DB when they return last_active = user_service.last_seen(follower_id) if days_since(last_active) > ACTIVE_USER_DAYS: continue # skip inactive users # Add tweet ID to front of follower's Redis timeline list redis.lpush(f"timeline:{follower_id}", tweet_id) # Keep timeline capped at 800 tweet IDs (trim older ones) redis.ltrim(f"timeline:{follower_id}", 0, TIMELINE_MAX_SIZE - 1) log(f"✅ Tweet {tweet_id} fanned-out to {len(followers)} followers")
This is the Timeline Service — what runs when YOU open Twitter. It assembles your home feed using the hybrid approach: it reads your pre-built Redis timeline (regular users), finds all celebrities you follow and fetches their latest tweets separately, merges everything together sorted by time, and then "hydrates" (fills in full details like text, author name, like count) by fetching from MySQL. This entire process runs in under 100ms!
# Timeline Service — Build Home Timeline for a User (Pseudocode) def get_home_timeline(user_id, page=1, count=20): # Step 1: Read pre-built timeline from Redis (tweet IDs only, very fast!) cached_tweet_ids = redis.lrange(f"timeline:{user_id}", 0, 799) # Step 2: Find celebrities this user follows (uses Flock social graph cache) all_following = social_graph.get_following(user_id) celebrities = [u for u in all_following if user_service.get_follower_count(u) >= CELEBRITY_THRESHOLD] # Step 3: Fetch recent celebrity tweets (fan-out-on-read for celebs) celebrity_tweet_ids = [] for celeb_id in celebrities: recent = redis.lrange(f"user_tweets:{celeb_id}", 0, 19) celebrity_tweet_ids.extend(recent) # Step 4: Merge both lists and sort by tweet_id (Snowflake = time-sorted) all_tweet_ids = list(set(cached_tweet_ids + celebrity_tweet_ids)) all_tweet_ids.sort(reverse=True) # newest first (Snowflake IDs are time-ordered!) # Step 5: Paginate to get just the page we need start = (page - 1) * count page_ids = all_tweet_ids[start : start + count] # Step 6: Hydrate — fetch full tweet data from MySQL in ONE batch query # (NOT 20 separate queries — batching is critical for performance!) full_tweets = mysql.query( "SELECT * FROM tweets WHERE id IN (?)", params=[page_ids] ) # Step 7: Apply user filters (blocks, mutes) and return! filtered = apply_filters(full_tweets, viewer=user_id) return filtered
Notice Step 6 — we fetch all tweet details in ONE batch MySQL query using "WHERE id IN (...)" — not 20 separate queries. This is called N+1 query avoidance and it's one of the most important performance optimisations in any database-backed application. One query with 20 IDs is 20x faster than 20 queries with 1 ID each. 🚀
🛡️ Section 15: Reliability — Why Twitter Keeps Running
Every MySQL shard has one primary (write) and multiple replicas (read-only). Timeline reads go to replicas — spreading read load across many servers. If the primary crashes, a replica is automatically promoted to primary. Zero manual intervention needed.
Kafka stores all events on disk, replicated across multiple brokers. If the fan-out worker crashes mid-processing, it restarts and picks up exactly where it left off — no tweets are ever lost or double-sent.
If the Trending Service goes down, Twitter just hides the trending section. The core experience (posting and reading tweets) continues unaffected. If the Recommendation Service goes down, Twitter shows chronological tweets only. Degraded but functional is always the goal.
What if a fan-out worker processes the same tweet twice due to a retry? Prepending to a Redis list using the tweet ID as deduplication key ensures the same ID isn't added twice. Operations designed to be safe to repeat (idempotent) are fundamental to building reliable distributed systems.
🗺️ Section 16: Everything Together — The Complete Twitter Map
🐦 Twitter / X — Complete Architecture Map
Service
Service
Service
Service
Service
Service
Service
Workers
Topics: new-tweets · likes · follows · retweets · media
Timeline + Rate Limits
Tweets + Users (sharded)
Likes + Counts
Real-Time Search
Media (CDN)
Analytics + ML
📐 Section 17: Twitter's Core Design Principles
Reads vastly outnumber writes on Twitter (millions scroll timelines vs thousands tweet per second). By doing expensive computation at write time (fan-out, timeline cache prep), the very common read operation becomes instant. Optimise for what happens most frequently.
Fan-out, search indexing, trending updates, notification delivery — none of these block the "tweet posted!" response to the user. Everything that can be async SHOULD be async. This keeps core operations fast and prevents cascading failures.
Snowflake shows a critical system design principle: avoid shared state whenever possible. If ID generation required a central counter, it would be a bottleneck at scale. By letting each machine generate its own IDs independently, the system scales infinitely. Always ask: "does this require coordination? Can I eliminate that need?"
The hybrid fan-out approach is a lesson in nuance. A single strategy rarely works for all users at extreme scale. Recognise the different user archetypes (regular vs celebrity) and apply the right strategy to each. One-size-fits-all breaks at scale.
🎓 Section 18: Cheat Sheet
If you're asked to "Design Twitter" in a system design interview, here is your exact 5-step framework:
- Post tweets? (yes — text ≤ 280 chars, media, links)
- Follow users? (yes — asymmetric: I follow you without you following me)
- Home timeline? (yes — tweets from people I follow, reverse chronological)
- Search? (yes — real-time search including tweets from seconds ago)
- Trending topics? (yes — per country/city)
- Notifications? (yes — likes, replies, follows, mentions)
- 350M users, ~100M DAU
- 500M tweets/day = ~6,000 tweets/second average
- Peak: 143,199 tweets/second (New Year's record)
- Average follower count: 700 → average fan-out: 700 writes/tweet
- Timeline reads are 100x more frequent than writes
- Each tweet: 280 chars + metadata ≈ 1KB → 500GB raw tweet data/day
- Write path: Client → LB → API GW → Tweet Service → MySQL + Kafka
- Fan-out: Kafka → Fan-out Workers → Redis timeline caches
- Read path: Client → Timeline Service → Redis cache + celebrity tweets → MySQL hydrate
- Search: Kafka → Earlybird Indexer → in-memory inverted index
- Trending: Kafka → Trending Counter (sliding window) → Cache
- Explain Fan-out on Write vs Fan-out on Read and trade-offs of each
- Explain the Hybrid approach: regular users (write), celebrities (read)
- Explain Snowflake ID generation and why it's time-sortable
- Explain Redis timeline cache (list of IDs, not full tweets)
- Explain Earlybird real-time search with in-memory inverted index
- Celebrity tweets to 150M followers → Hybrid fan-out avoids write storm
- World Cup goal spike (10x traffic in 1 second) → Kafka queue absorbs it; workers catch up
- User comes back after 30 days of inactivity → rebuild timeline from MySQL (cache miss is OK for rare event)
- Duplicate tweet prevention → idempotency key on tweet submission
- Bots inflating trends → velocity detection + account quality signals filter them
- User deletes a tweet → remove from MySQL, invalidate Redis caches, notify Earlybird to un-index
🎉 Final Summary
Twitter's entire system is a masterclass in one core idea: the hardest problem is not what you build — it's understanding which problems to solve differently for different users.
Regular users and celebrities are both "users" — but require completely opposite architectural strategies for timeline delivery. Recognising these distinctions and building systems that handle each case optimally is what separates great engineers from good ones. 🎯
Happy Learning! Keep Building! 🔥
Comments
Post a Comment