Understanding the Challenge: A Feed for 100 Million Users
Designing a social media feed that serves 100 million users is one of the most fascinating and difficult problems in distributed systems engineering. At this scale, every millisecond of latency, every database query, and every network call compounds into massive infrastructure costs and user experience degradation. The feed—often called the timeline, news feed, or home feed—is the central nervous system of any social platform. It must feel instant, remain consistent, and surface relevant content from a user's network, all while handling millions of concurrent requests.
This tutorial walks through the complete architecture, data models, and code patterns needed to build a feed system that scales to 100 million users. We'll cover fan-out strategies, caching layers, ranking pipelines, and the hard-won lessons from platforms like Twitter, Facebook, and Instagram.
Core Terminology and Scale Math
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →Before diving into code, let's establish the numbers we're designing for:
- 100 million registered users — roughly 10-20 million are active daily
- Average follows per user: 200-500 connections
- Celebrity accounts: 1-50 million followers (these break naive architectures)
- New posts created: 50,000 - 500,000 per second during peak
- Feed read requests: 1-5 million per second
- Target latency: Under 200ms for feed generation, under 50ms for cached reads
A naive approach—querying all followed users' posts, sorting by time, and paginating—collapses at around 10,000 users. At 100 million, you need a completely different mindset.
Data Models: Posts, Users, and the Social Graph
Everything starts with clean, shardable data models. Here's the foundational schema using a relational approach with shard keys:
-- users table (sharded on user_id)
CREATE TABLE users (
user_id BIGINT PRIMARY KEY,
username VARCHAR(50) NOT NULL,
follower_count INT DEFAULT 0,
following_count INT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW()
);
-- follows table (sharded on follower_id)
CREATE TABLE follows (
follower_id BIGINT NOT NULL,
followed_id BIGINT NOT NULL,
created_at TIMESTAMP DEFAULT NOW(),
PRIMARY KEY (follower_id, followed_id)
);
-- posts table (sharded on author_id)
CREATE TABLE posts (
post_id BIGINT PRIMARY KEY,
author_id BIGINT NOT NULL,
content TEXT,
created_at TIMESTAMP DEFAULT NOW(),
like_count INT DEFAULT 0,
reply_count INT DEFAULT 0
);
-- Index for time-range queries within a shard
CREATE INDEX idx_posts_author_time ON posts(author_id, created_at DESC);
The key insight: posts is sharded on author_id. When you need all posts from users someone follows, you'd have to query potentially thousands of shards—a scatter-gather nightmare. This is precisely why we need fan-out.
Fan-Out on Write: Pre-Building the Feed
Fan-out on write pushes the computational cost to post creation time. When a user publishes a new post, the system immediately writes that post into the feed cache of every follower. For a user with 500 followers, that's 500 cache writes. For a celebrity with 50 million followers, that's catastrophic without special handling.
Here's the core fan-out service in Python-style pseudocode:
def fanout_on_write(post, author_id):
"""
Called synchronously after a post is created.
For regular users, fan out to all followers.
For celebrities, skip fan-out and use fan-out-on-read instead.
"""
follower_count = get_follower_count(author_id)
if follower_count > CELEBRITY_THRESHOLD: # e.g., 1,000,000
# Celebrities: store post in author's outbox only
# Followers will merge at read time
store_in_author_outbox(author_id, post)
return
# Fetch followers in batches to avoid memory explosion
batch_size = 1000
cursor = 0
while True:
followers = get_followers_paginated(
author_id, cursor, batch_size
)
if not followers:
break
# Pipeline the writes to each follower's feed cache
pipeline = redis_client.pipeline()
for follower_id in followers:
feed_key = f"feed:{follower_id}"
pipeline.zadd(feed_key, {
f"{post['post_id']}:{post['created_at']}": post['created_at']
})
# Trim to keep only last 800 items per feed
pipeline.zremrangebyrank(feed_key, 0, -801)
pipeline.execute()
cursor += batch_size
This approach works beautifully for the 99% of users with reasonable follower counts. The feed is pre-computed, so reads are a simple Redis ZREVRANGE—blazing fast.
Fan-Out on Read: Handling Celebrities
For accounts with millions of followers, fan-out on write would require millions of cache writes per post. Instead, we use fan-out on read for these "celebrity" accounts. Their posts stay in their own outbox, and followers merge them at read time.
def get_celebrity_posts_for_user(user_id, last_timestamp, limit=20):
"""
Fetch recent posts from all celebrities the user follows.
These are NOT pre-populated in the user's feed cache.
"""
followed_celebrities = get_followed_celebrities(user_id)
# Parallel fetch from each celebrity's outbox
results = []
for celeb_id in followed_celebrities:
posts = get_posts_from_outbox(
celeb_id,
since=last_timestamp,
limit=10
)
results.extend(posts)
# Sort merged results by time
results.sort(key=lambda p: p['created_at'], reverse=True)
return results[:limit]
The Hybrid Feed Generation Pipeline
Real production systems combine both approaches. The feed endpoint merges pre-computed cached feeds with real-time celebrity inserts. Here's the complete feed serving logic:
def serve_feed(user_id, cursor=None, limit=20):
"""
Hybrid feed generation:
1. Pull from pre-computed feed cache (regular friends)
2. Inject celebrity posts at their chronological positions
3. Apply ranking model
"""
feed_key = f"feed:{user_id}"
if cursor is None:
# First page: get latest from cache
cached_feed = redis_client.zrevrange(
feed_key, 0, limit * 2 - 1, withscores=True
)
else:
# Paginated: get items older than cursor
cached_feed = redis_client.zrevrangebyscore(
feed_key, max='(cursor', min='-inf',
start=0, num=limit * 2, withscores=True
)
# Extract post IDs from cache entries
cached_posts = [
decode_feed_entry(entry) for entry in cached_feed
]
# Fetch celebrity posts for insertion
celebrity_posts = get_celebrity_posts_for_user(
user_id,
last_timestamp=get_oldest_timestamp(cached_posts)
)
# Merge and deduplicate
merged = merge_sorted_feeds(cached_posts, celebrity_posts)
# Apply lightweight ranking before returning
ranked = apply_ranking_model(merged, user_id)
next_cursor = ranked[-1]['timestamp'] if ranked else None
return {
'items': ranked[:limit],
'next_cursor': next_cursor,
'has_more': len(ranked) > limit
}
def merge_sorted_feeds(cached, celebrity):
"""Both lists are sorted by timestamp descending. Merge in O(n)."""
merged = []
i, j = 0, 0
seen_ids = set()
while i < len(cached) and j < len(celebrity):
if cached[i]['timestamp'] >= celebrity[j]['timestamp']:
post = cached[i]
i += 1
else:
post = celebrity[j]
j += 1
if post['post_id'] not in seen_ids:
seen_ids.add(post['post_id'])
merged.append(post)
# Drain remaining
while i < len(cached):
if cached[i]['post_id'] not in seen_ids:
merged.append(cached[i])
i += 1
while j < len(celebrity):
if celebrity[j]['post_id'] not in seen_ids:
merged.append(celebrity[j])
j += 1
return merged
Caching Architecture with Redis
The feed cache is the heart of the system. We use Redis sorted sets with timestamps as scores. This allows range queries, pagination, and efficient trimming:
# Feed cache structure (per user)
# Key: feed:{user_id}
# Type: Sorted Set
# Members: "post_id:timestamp" (ensures uniqueness)
# Score: Unix timestamp microseconds
# Writing to a user's feed
ZADD feed:42 1700000000123456 "post_789:1700000000123456"
ZADD feed:42 1700000000987654 "post_790:1700000000987654"
# Trimming to last 800 entries (critical for memory control)
ZREMRANGEBYRANK feed:42 0 -801
# Reading latest 20 posts
ZREVRANGE feed:42 0 19 WITHSCORES
# Pagination with cursor
ZREVRANGEBYSCORE feed:42 (cursor_timestamp) -inf LIMIT 0 20 WITHSCORES
# Expiry: feeds inactive for 30 days get TTL'd
EXPIRE feed:42 2592000
Handling the Cold Start Problem
New users or users returning after 30 days have empty or expired feed caches. We need a warm-up path:
def warm_feed_cache(user_id):
"""
Rebuild feed cache for a user who hasn't visited in a while.
Called lazily on first feed request, or proactively via batch job.
"""
followed_users = get_followed_non_celebrity_users(user_id)
# Fetch recent posts from each followed user
recent_posts = []
for followed_id in followed_users:
posts = get_recent_posts_from_shard(
followed_id,
limit=100,
days=7
)
recent_posts.extend(posts)
# Sort and populate Redis sorted set
recent_posts.sort(key=lambda p: p['created_at'], reverse=True)
pipeline = redis_client.pipeline()
feed_key = f"feed:{user_id}"
for post in recent_posts[:800]:
pipeline.zadd(feed_key, {
f"{post['post_id']}:{post['created_at']}": post['created_at']
})
pipeline.zremrangebyrank(feed_key, 0, -801)
pipeline.expire(feed_key, 2592000) # 30 days TTL
pipeline.execute()
return recent_posts[:20]
Sharding Strategy for 100 Million Users
At 100 million users, no single database or Redis instance can hold everything. Sharding is mandatory. Here's how to shard the feed cache across a Redis cluster:
# Consistent hashing ring for Redis cluster
# Each Redis instance handles ~5 million feed keys
def get_feed_shard(user_id):
"""
Map user_id to Redis shard using consistent hashing.
20 Redis instances, each with 2 replicas = 40 nodes.
"""
shard_count = 20 # logical shards
shard_number = user_id % shard_count
return f"redis-feed-shard-{shard_number}.internal:6379"
# In practice, use Redis Cluster with 16384 hash slots
# Key hashing: CRC16(feed:{user_id}) mod 16384
# Redis Cluster handles failover, resharding automatically
# Example Redis Cluster configuration (redis-cluster.conf)
# 20 master nodes, each with 1 replica
# Total: 40 Redis instances
# Each master handles ~819 hash slots
For the post database, shard by author_id using range-based or hash-based partitioning. At 100 million users with 20 shards, each shard holds ~5 million users' posts. Use PostgreSQL or MySQL with partitioning by date within each shard:
-- Partitioning posts by month within a shard
CREATE TABLE posts_shard_07 (
post_id BIGINT,
author_id BIGINT NOT NULL,
content TEXT,
created_at TIMESTAMP NOT NULL
) PARTITION BY RANGE (EXTRACT(YEAR FROM created_at) * 100 + EXTRACT(MONTH FROM created_at));
-- Monthly partitions
CREATE TABLE posts_shard_07_202501 PARTITION OF posts_shard_07
FOR VALUES FROM (202501) TO (202502);
CREATE TABLE posts_shard_07_202502 PARTITION OF posts_shard_07
FOR VALUES FROM (202502) TO (202503);
-- ... and so on
Ranking: Beyond Chronological Order
A raw chronological feed is suboptimal. Ranking boosts engagement. At scale, you can't run a heavy ML model on every feed request. Use a two-phase approach: lightweight scoring at serve time, heavy personalization offline:
def apply_ranking_model(posts, user_id):
"""
Phase 1: Lightweight real-time scoring.
Combines recency, affinity, and post quality signals.
"""
user_affinities = get_cached_affinities(user_id) # pre-computed
scored = []
for post in posts:
age_hours = (now() - post['created_at']) / 3600
# Recency decay (exponential)
recency_score = max(0, 1.0 - age_hours * 0.02)
# Affinity: how much user interacts with this author
affinity = user_affinities.get(post['author_id'], 0.1)
# Post quality: engagement rate normalized
quality = min(1.0, (post['like_count'] + post['reply_count']) / 100)
# Combine scores with configurable weights
final_score = (
0.4 * recency_score +
0.35 * affinity +
0.25 * quality
)
scored.append((post, final_score))
scored.sort(key=lambda x: x[1], reverse=True)
return [p for p, _ in scored]
# Offline: affinity computation via batch processing
def compute_user_affinities():
"""
Runs daily in Spark/MapReduce.
Calculates interaction strength between user pairs.
"""
# Pseudocode for a Spark job
# Input: interaction_logs (likes, comments, profile visits)
# Output: affinity_matrix stored in Redis
# affinity_score = weighted_sum(last_30_days_interactions)
# Stored as Redis Hash: affinity:{user_id} -> {author_id: score}
pass
The Complete Feed API Endpoint
Tying everything together, here's the production-ready feed endpoint with error handling, timeouts, and fallbacks:
# REST API endpoint (Node.js/Express style for clarity)
async function feedHandler(req, res) {
const userId = req.user.id;
const cursor = req.query.cursor || null;
const limit = Math.min(req.query.limit || 20, 50);
const startTime = Date.now();
try {
// Check if feed cache exists and is warm
const feedKey = `feed:${userId}`;
const feedExists = await redis.exists(feedKey);
if (!feedExists) {
// Cold start: warm the cache asynchronously
// Return stale or partial feed immediately
const warmPromise = warmFeedCache(userId);
// Return what we can in < 200ms
const fallbackFeed = await getFallbackFeed(userId, limit);
// Don't await - fire and forget for next request
warmPromise.catch(err => log.error('Warm failed', err));
return res.json({
items: fallbackFeed,
cursor: null,
source: 'fallback'
});
}
// Normal path: serve from hybrid feed
const result = await serveFeed(userId, cursor, limit);
const latency = Date.now() - startTime;
recordLatencyMetric('feed_serve', latency);
res.json({
items: result.items,
next_cursor: result.next_cursor,
has_more: result.has_more,
latency_ms: latency
});
} catch (error) {
log.error('Feed serve failed', error);
// Graceful degradation: return cached snapshot
const snapshot = await getFeedSnapshot(userId, limit);
res.json({
items: snapshot,
cursor: null,
source: 'degraded'
});
}
}
async function getFallbackFeed(userId, limit) {
// Query most active followed users directly
const topFollowed = await getTopFollowedUsers(userId, 10);
const posts = [];
for (const followedId of topFollowed) {
const recent = await getRecentPostsFromShard(followedId, 5);
posts.push(...recent);
}
posts.sort((a, b) => b.created_at - a.created_at);
return posts.slice(0, limit);
}
Event-Driven Fan-Out with Message Queues
For true production scale, the fan-out on write shouldn't block the post creation response. Use an asynchronous event pipeline:
# Post creation flow with Kafka
# 1. Post saved to database (synchronous, < 10ms)
# 2. Event published to Kafka (synchronous, < 5ms)
# 3. Fan-out workers consume events asynchronously
# Kafka topic: post_events
# Partition key: author_id (ensures ordering per author)
def handle_post_creation(post):
"""
Called by the API server after saving post to DB.
"""
event = {
'event_type': 'post_created',
'post_id': post.post_id,
'author_id': post.author_id,
'content': post.content,
'created_at': post.created_at.isoformat(),
'follower_count': post.author_follower_count
}
# Publish to Kafka, not blocking on fan-out completion
kafka_producer.send('post_events',
key=str(post.author_id),
value=json.dumps(event)
)
return {'status': 'accepted', 'post_id': post.post_id}
# Fan-out worker (runs in separate process/pod)
def fanout_worker():
consumer = kafka_consumer.subscribe(['post_events'])
for message in consumer:
event = json.loads(message.value)
if event['follower_count'] > CELEBRITY_THRESHOLD:
# Celebrity: store in outbox, skip fan-out
store_celebrity_outbox(event)
else:
# Regular fan-out in batches
fanout_to_followers(event)
# Commit offset after processing
consumer.commit()
Best Practices for Feed Systems at Scale
1. Embrace Asynchrony
Never block the post creation response on fan-out completion. Use message queues (Kafka, SQS, Pulsar) to decouple post creation from feed population. Users tolerate a 500ms delay in seeing a friend's post in their feed—they don't tolerate a 500ms delay in publishing their own post.
2. Set Hard Limits Everywhere
Every query, every cache entry, every merge operation needs a limit. Feed cache entries capped at 800. Celebrity outbox capped at 2000. Merge operations limited to 200 candidates. Pagination limited to 50 items per page. Without limits, a user following 10,000 accounts will crash your feed generation.
# Defensive limit constants
FEED_CACHE_MAX_SIZE = 800
CELEBRITY_OUTBOX_MAX_SIZE = 2000
MERGE_CANDIDATE_MAX = 200
PAGE_SIZE_MAX = 50
FOLLOWED_USERS_FETCH_LIMIT = 2000 # Cap even if user follows more
3. Pre-Compute Affinities Offline
Don't compute user-to-user relationship strengths during feed serve. Run nightly batch jobs that analyze interaction logs and produce affinity scores. Store them in Redis as a flat hashmap. Feed serve just looks up pre-computed values.
4. Monitor Feed Freshness
Track the age of the newest post in each user's feed cache. Alert if feeds are stale. Key metrics: p50/p99 feed latency, cache hit rate, fan-out lag (time from post creation to appearing in feeds), and feed refresh rate.
# Metrics to emit
# - feed_serve_latency_p99: histogram
# - feed_cache_hit_rate: gauge
# - fanout_lag_seconds: histogram (post_created_at to feed_insert_time)
# - feed_staleness_seconds: age of newest entry per user
# - celebrity_merge_count: counter
# - cold_start_rate: percentage of requests triggering warm
5. Use Read-Only Snapshots for Degradation
When the feed pipeline fails (Redis down, fan-out workers backlogged), degrade gracefully. Serve a snapshot from a read-only replica or a simplified chronological query. Never show an error page—show something, even if it's not perfectly personalized.
6. Test with Realistic Social Graphs
Synthetic testing with uniform follower distributions won't catch problems. Generate test graphs that mirror real social networks: power-law distribution (a few users with millions of followers, most with under 100), clusters, and dormant accounts. Test the celebrity path and the cold-start path equally.
7. Cache Invalidation Is a Lie—Use TTLs Instead
When a user deletes a post, you could try to remove it from millions of feed caches—or you could let it expire naturally. Use short TTLs on feed entries (7-30 days) and filter out deleted posts at read time by checking a bloom filter of recently deleted post IDs.
# Deleted post bloom filter (checked during feed serve)
def filter_deleted_posts(posts):
deleted_bloom = get_deleted_posts_bloom_filter()
return [
post for post in posts
if not deleted_bloom.might_contain(post['post_id'])
]
8. Separate Read and Write Paths Completely
The write path (post creation, fan-out) and read path (feed serve) should use different database connection pools, different Redis clusters, and different scaling policies. A spike in post creation shouldn't starve feed reads of connections.
Putting It All Together: Architecture Diagram as Code
# Infrastructure-as-code representation (Terraform-style pseudocode)
resource "feed_system" {
# Write path
post_api_service {
instances = 200
cpu_per_instance = 4
scaling_policy = "cpu_usage > 70%"
}
kafka_cluster {
brokers = 12
topic "post_events" {
partitions = 64
retention_hours = 72
}
}
fanout_workers {
instances = 100
batch_size = 1000
celebrity_threshold = 1000000
}
# Read path
feed_api_service {
instances = 500
cpu_per_instance = 8
scaling_policy = "request_count > 5000/min"
}
redis_feed_cluster {
master_nodes = 20
replica_nodes = 20
max_memory_per_node = "64GB"
eviction_policy = "volatile-lru"
# Each node handles ~5M feed keys
}
postgres_shards {
count = 20
per_shard {
instance_class = "db.r5.4xlarge"
storage = "2TB"
read_replicas = 2
}
}
# Offline/Async
affinity_compute_job {
schedule = "daily at 2am UTC"
framework = "Spark on Kubernetes"
}
bloom_filter_deleted_posts {
update_frequency = "every 10 minutes"
false_positive_rate = 0.001
}
}
Conclusion
Designing a social media feed for 100 million users forces you to confront every hard tradeoff in distributed systems: consistency versus latency, pre-computation versus freshness, push versus pull, and cost versus performance. The hybrid fan-out model—pre-building feeds for regular users while merging celebrity posts at read time—has proven itself across the industry. Redis sorted sets provide the ideal data structure for feed storage, offering sorted retrieval, range queries, and efficient trimming in a single atomic operation. Kafka decouples the write path, letting post creation return instantly while fan-out workers asynchronously populate millions of feed caches.
The most important lesson is that no single approach works for everyone. Your system must treat a user with 50 followers and a user with 50 million followers differently. It must handle cold starts gracefully, degrade without crashing, and rank content with lightweight models that don't add latency. Build with hard limits, monitor freshness obsessively, and always have a fallback path. The feed is your platform's heartbeat—keep it beating under any load.