Design Instagram Feed

Understanding Requirements

Functional Requirements

Non-Functional Requirements

Capacity Estimation

Request Flow

Major Components

Client

CDN / Edge Layer

API Gateway

User Service

1. Registration & Profile

-- Table: users

CREATE TABLE users (
    user_id BIGSERIAL PRIMARY KEY,
    username TEXT UNIQUE NOT NULL,
    email TEXT UNIQUE NOT NULL,
    password_hash TEXT NOT NULL,
    created_at TIMESTAMP
);

-- Table: user_profiles

CREATE TABLE user_profiles (
    user_id BIGINT PRIMARY KEY,
    bio TEXT,
    avatar_url TEXT,
    privacy_setting TEXT,
    updated_at TIMESTAMP
);

2. Authentication

def login(username, password):
    user = db.find_user(username)

    if verify(password, user.password_hash):
        token = generate_token(user.user_id)
        redis.set(token, user.user_id, ttl=86400)  # 1 day expiry
        return token

def authenticate(token):
    user_id = redis.get(token)
    return user_id

3. Follow Graph

-- Who follows a user (used in fanout)

CREATE TABLE followers_by_user (
    user_id text,
    follower_id text,
    followed_at timestamp,
    PRIMARY KEY ((user_id), follower_id)
);

-- Whom a user follows (used in feed generation)

CREATE TABLE following_by_user (
    user_id text,
    following_id text,
    followed_at timestamp,
    PRIMARY KEY ((user_id), following_id)
);

def follow(user_id, target_user_id):
    cassandra.insert_follow(user_id, target_user_id)
    publish_event("FOLLOW_CREATED", user_id, target_user_id)


def unfollow(user_id, target_user_id):
    cassandra.delete_follow(user_id, target_user_id)
    publish_event("FOLLOW_REMOVED", user_id, target_user_id)

4. Caching Layer (Follower Optimization)

followers = redis.get(user_id)

if not followers:
    followers = cassandra.get_followers(user_id)
    redis.set(user_id, followers, ttl=300)

Post Service

A pre-signed URL is a temporary, secure URL generated by the backend that allows a client to directly upload or download a file from object storage (like S3) without exposing credentials.

It contains permissions (e.g., upload/download), an expiry time, and a signed token that authorizes the operation.

This lets clients transfer large files directly to storage, reducing load on backend servers while keeping access secure and controlled.

Data Model and Schema Design

-- Table: posts_by_user
-- Optimized for: "Fetch posts of a user ordered by latest"

CREATE TABLE posts_by_user (
    user_id text,
    created_at timestamp,
    post_id text,
    caption text,
    media_urls list,   -- final processed CDN URLs
    media_path text,         -- original S3 path (source of truth)
    media_type text,
    visibility text,
    status text,             -- PROCESSING / READY / FAILED / DELETED
    updated_at timestamp,
    PRIMARY KEY ((user_id), created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC);

-- PRIMARY KEY breakdown:
-- (user_id)     → Partition Key
-- created_at    → Clustering Column (sorted DESC)
-- post_id       → Uniqueness within same timestamp

In Cassandra, a row is uniquely identified by the full PRIMARY KEY, which combines the partition key and all clustering columns.

If two rows share the same values for this combination, Cassandra treats them as the same row, and the later write will overwrite the earlier one.

So to store multiple distinct posts safely, the key must be unique—for example:
PRIMARY KEY ((user_id), created_at, post_id), where post_id ensures uniqueness even if timestamps collide.

CREATE TABLE posts_by_id (
    post_id text PRIMARY KEY,
    user_id text,
    caption text,
    media_urls list,   -- processed variants
    media_path text,         -- original upload location
    media_type text,
    visibility text,
    status text,
    created_at timestamp,
    updated_at timestamp
);

Post Creation Flow

def create_post(user_id, caption, media_files):
    # Step 1: Upload media to object storage
    media_urls = []

    for file in media_files:
        url = upload_to_object_store(file)
        media_urls.append(url)

    # Step 2: Create post metadata
    post_id = generate_post_id()
    save_post_metadata(post_id, user_id, caption, media_urls)

    # Step 3: Publish event for downstream systems
    publish_event("POST_CREATED", post_id, user_id)

    return post_id

Media Processing Service

def create_post(user_id, media_file):
    # Step 1: Upload original file
    original_path = upload_to_object_store(media_file)

    # Step 2: Save metadata (initial state)
    post_id = save_post_metadata(user_id, original_path)

    # Step 3: Trigger async media processing
    publish_event("MEDIA_PROCESS", post_id, original_path)

    return post_id

def process_media(post_id, original_path):

    # Generate multiple variants
    thumbnail = resize(original_path, "150x150")
    medium = resize(original_path, "720p")
    high = resize(original_path, "1080p")

    # Upload processed files
    urls = upload_all([thumbnail, medium, high])

    # Update metadata
    update_post_media(post_id, urls)

-- Conceptual media variants (stored in DB)

media_urls = {
    "thumbnail": ".../image_150.jpg",
    "medium": ".../image_720.jpg",
    "high": ".../image_1080.jpg"
}

Feed Service

Step 1: Fetch Feed IDs (Fast DB Read)

post_ids = feed_store.fetch(user_id, cursor, limit=20) 

Here's a simple, practical example of cursor-based pagination using created_at as the cursor.

First Request (Initial Load)

SELECT post_id, created_at FROM feed_by_user WHERE user_id = 123 LIMIT 3; 

Response:

posts = [ {post_id: "p1", created_at: 105}, {post_id: "p2", created_at: 100}, {post_id: "p3", created_at: 95} ] next_cursor = 95 

Here, 95 (last item's timestamp) becomes the cursor.

Next Request (Using Cursor)

SELECT post_id, created_at FROM feed_by_user WHERE user_id = 123 AND created_at < 95 LIMIT 3; 

Response:

posts = [ {post_id: "p4", created_at: 90}, {post_id: "p5", created_at: 85}, {post_id: "p6", created_at: 80} ] next_cursor = 80 

Step 2: Hydrate Metadata (Cache First)

posts = post_cache.get_bulk(post_ids)

missing = find_missing(posts)

if missing:
    db_posts = fetch_from_post_service(missing)
    post_cache.set_bulk(db_posts)
    posts.update(db_posts)

Step 3: Apply Ranking / Personalization

ranked = rank(posts, user_id) 

Step 4: Return Response

return { "posts": ranked, "next_cursor": ranked[-1].created_at } 

Key Optimizations

Fallback Path (Pull-Based)

def fallback_feed(user_id):
    following = get_following(user_id)

    posts = []

    for user in following:
        posts.extend(fetch_recent_posts(user))

    return merge_and_sort(posts)

Fanout Service

def handle_post_created(event):
    post_id = event.post_id
    author_id = event.author_id

    followers = user_service.get_followers(author_id)

    for follower in followers:
        feed_store.insert(
            user_id=follower,
            post_id=post_id,
            created_at=event.created_at,
            author_id=author_id
        )

-- Table: feed_by_user
-- Optimized for: "Fetch home feed (infinite scroll)"

CREATE TABLE feed_by_user (
    user_id text,
    created_at timestamp,
    post_id text,
    author_id text,
    PRIMARY KEY ((user_id), created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC);

-- PRIMARY KEY breakdown:
-- (user_id)     → Partition Key (all feed items of a user stored together)
-- created_at    → Clustering Column (sorted by recency DESC)
-- post_id       → Ensures uniqueness within same timestamp

INSERT INTO feed_by_user (user_id, created_at, post_id, author_id) VALUES (?, ?, ?, ?);

Feed Store Partition Hotspot Risk

The issue arises from using user_id as the sole partition key in feed_by_user. For highly active users, this leads to very large partitions over time, as all their feed data accumulates in a single location.

Additionally, frequent fanout writes targeting the same user can create hot partitions, where a single node becomes overloaded with read/write traffic. This can degrade performance, increase latency, and even cause instability under heavy load.

The solution is to introduce partition bucketing, where data is split into smaller chunks using a time-based bucket (e.g., day or week). Instead of a single partition per user, the key becomes a combination like:

PRIMARY KEY ((user_id, bucket), created_at, post_id)

Here, bucket (such as 2026-05-03) distributes data across multiple partitions, ensuring no single partition grows too large or becomes a hotspot. During reads, the Feed Service fetches data from recent buckets (e.g., last few days) and merges results.

This approach slightly increases read complexity but significantly improves write scalability, load distribution, and system stability at scale.

Engagement Service

Responsibilities

Data Model (Cassandra)

-- Likes per post

CREATE TABLE likes_by_post (
    post_id text,
    user_id text,
    created_at timestamp,
    PRIMARY KEY ((post_id), user_id)
);

-- Comments per post (time-ordered)

CREATE TABLE comments_by_post (
    post_id text,
    created_at timestamp,
    comment_id text,
    user_id text,
    text text,
    PRIMARY KEY ((post_id), created_at, comment_id)
) WITH CLUSTERING ORDER BY (created_at DESC);

-- Engagement counters (high write throughput)

CREATE TABLE post_engagement_counters (
    post_id text PRIMARY KEY,
    like_count counter,
    comment_count counter,
    share_count counter
);

Interaction Flow

def like_post(user_id, post_id):
    cassandra.insert_like(post_id, user_id)
    cassandra.increment_counter(post_id, "like_count")

    publish_event("POST_LIKED", user_id, post_id)

def add_comment(user_id, post_id, text):
    comment_id = generate_id()

    cassandra.insert_comment(post_id, comment_id, user_id, text)
    cassandra.increment_counter(post_id, "comment_count")

    publish_event("COMMENT_ADDED", user_id, post_id)

Multi-Device Consistency

Feed Ranking Service

def rank_feed(user_id, candidates):
    features = feature_store.get(user_id, candidates)
    scores = model.predict(features)
    ranked = sort_by_score(candidates, scores)
    return ranked[:20]  # return top posts

Cache Layer

1. Cache Read Flow (Feed Fetch)

def get_feed(user_id, cursor):
    feed = redis.get(user_id)

    if not feed:
        feed = feed_store.fetch(user_id, cursor)
        redis.set(user_id, feed, ttl=300)   # cache for 5 mins

    return feed

2. Cache Write / Update Flow (Fanout)

Option A: Write-through cache

feed_store.insert(user_id, post_id)

redis.prepend(user_id, post_id)   # update cache immediately

Option B: Lazy update (most common)

feed_store.insert(user_id, post_id)

redis.delete(user_id)   # invalidate cache

3. Cache Invalidation Strategy

4. Post Metadata Cache (Important Layer)

posts = redis.get_bulk(post_ids)

if missing:
    posts.update(fetch_from_post_service(missing))
    redis.set_bulk(posts)

5. Follower Cache (Fanout Optimization)

followers = redis.get(author_id)

if not followers:
    followers = user_service.get_followers(author_id)
    redis.set(author_id, followers)

User opens app
   ↓
Feed Service → Redis (feed cache)
   ↓ (miss)
Cassandra (feed_by_user)
   ↓
Redis updated
   ↓
Post metadata cache (Redis)
   ↓
Final response

Notification Service

def handle_like_event(event):
    actor = event.user_id
    post_owner = event.post_owner_id

    notification = {
        "type": "LIKE",
        "actor": actor,
        "target_user": post_owner,
        "post_id": event.post_id,
        "created_at": now()
    }

    notification_queue.publish(notification)

def process_notification(notification):
    save_notification(notification)
    send_push(notification)

-- Table: notifications_by_user

CREATE TABLE notifications_by_user (
    user_id text,
    created_at timestamp,
    notification_id text,
    type text,
    actor_id text,
    post_id text,
    is_read boolean,
    PRIMARY KEY ((user_id), created_at, notification_id)
) WITH CLUSTERING ORDER BY (created_at DESC);

def send_push(notification):
    device_tokens = get_user_devices(notification["target_user"])

    for token in device_tokens:
        push_provider.send(token, format_message(notification))

In-App (Real-Time) Notifications

gateway.send(user_id, notification)

Notification Service: Add Aggregation

The problem is that without aggregation, the system may generate a large number of noisy and redundant notifications (e.g., "User1 liked", "User2 liked", …), which can overwhelm users and degrade experience. It also increases write load and unnecessary push traffic.

The solution is notification aggregation, where similar events within a time window are grouped into a single notification (e.g., "10 people liked your post"). This is typically implemented by maintaining an aggregation window and updating an existing notification record instead of creating new ones. This reduces noise, improves user experience, and lowers system load while keeping notifications meaningful and concise.

Scalability Considerations

Consistency vs Performance

Failure Handling

Conclusion