What Is Database Sharding?

A Database Sharding is a technique for horizontally scaling a database by dividing data across multiple independent database servers called shards.

Each shard stores a subset of the data and independently handles both read and write requests, distributing the workload across multiple database instances.

Instead of storing all data on a single server, sharding spreads it across multiple machines, increasing storage capacity, write throughput, and overall system scalability.

Sharding is commonly used by large-scale systems such as social media platforms, e-commerce applications, online gaming systems, and financial services that manage billions of records.

Why Do We Need Sharding?

Suppose an e-commerce application has grown to hundreds of millions of customers and billions of orders.

Even after adding indexes and read replicas, the primary database still processes every write request. As traffic grows, it becomes the bottleneck, limiting write throughput and increasing transaction latency.

Adding more replicas improves read scalability, but it does not increase write scalability because every insert, update, and delete must still be processed by the same primary database.

A single database server also has practical limits on CPU, memory, storage capacity, and I/O throughput. As the dataset grows, it may eventually exceed the capacity of a single machine.

Sharding addresses these limitations by partitioning data across multiple database servers.

Since each shard stores only a portion of the data and processes its own read and write requests, the application can scale beyond the limits of a single database server while supporting much larger datasets and higher throughput.

How Sharding Works

Sharding works by partitioning data across multiple database servers based on a sharding key, such as a customer ID, region, tenant ID, or account number.

Each shard stores and manages only its assigned subset of the data.

Customer IDs
-----------------------------
1 - 1,000,000      -> Shard 1
1,000,001 - 2,000,000 -> Shard 2
2,000,001 - 3,000,000 -> Shard 3

When a request arrives, the application or a Shard Router uses the sharding key to determine which shard owns the requested data and forwards the request to that database.

Customer ID = 1,456,321
         |
         v
   +--------------+
   | Shard Router |
   +--------------+
         |
         v
     +---------+
     | Shard 2 |
     +---------+

Because each request is routed to only one shard, queries execute on a much smaller dataset and the workload is distributed across multiple database servers.

As additional shards are added, both the application's storage capacity and read/write throughput can scale horizontally.

Partitioning vs Sharding

Partitioning divides data into multiple partitions within the same database server.

Sharding distributes data across multiple independent database servers.

Sharding Types

Different applications choose different sharding strategies depending on how data is accessed.

Range-Based Sharding

Data is divided using value ranges.

Customer IDs
----------------------------
1 - 1,000,000
      |
      v
   +---------+
   | Shard 1 |
   +---------+

1,000,001 - 2,000,000
      |
      v
   +---------+
   | Shard 2 |
   +---------+

2,000,001 - 3,000,000
      |
      v
   +---------+
   | Shard 3 |
   +---------+

This approach is simple but may lead to uneven data distribution if certain ranges receive significantly more traffic.

Hash-Based Sharding

In Hash-Based Sharding, a hash function computes the target shard from the sharding key.

Instead of storing consecutive ranges of data together, records are distributed across shards based on the hash value.

Shard = Hash(CustomerId) % 4

For example:

Customer ID = 1001
Hash(1001) % 4
      |
      v
   Shard 2

Because hash values are typically distributed uniformly, data and requests are spread more evenly across shards, reducing the risk of one shard becoming overloaded.

The trade-off is that hash-based sharding does not preserve the natural ordering of data, making range queries such as CustomerId BETWEEN 1000 AND 2000 less efficient because multiple shards may need to be searched.

Directory-Based Sharding

In Directory-Based Sharding, a separate lookup service (or metadata table) maintains the mapping between a sharding key and the shard that stores the data.

Instead of calculating the target shard using a range or hash function, the application first queries the lookup service and then routes the request to the appropriate shard.

    Customer ID
         |
         v
  +---------------+
  | Lookup Table  |
  +---------------+
         |
         v
     +---------+
     | Shard 3 |
     +---------+

This approach provides greater flexibility because records can be moved between shards by simply updating the lookup table, without changing the sharding algorithm.

The trade-off is that every request depends on the lookup service, making it an additional component that must be highly available and kept synchronized with the actual shard locations.

Sharding Key

A Sharding Key is the column or field used to determine which shard stores a particular record.

Choosing the correct sharding key is one of the most important design decisions because it directly affects data distribution, query performance, and horizontal scalability.

Common sharding keys include:


Customer ID
Order ID
Tenant ID
Region

When a request arrives, the application or shard router uses the sharding key to identify the target shard.

Customer ID = 1456321
         |
         v
   Sharding Key
         |
         v
     +---------+
     | Shard 2 |
     +---------+

A good sharding key distributes data and requests evenly across all shards while ensuring that most queries access only a single shard.

A poor sharding key can cause hotspots, where one shard receives significantly more traffic than the others, leading to uneven resource utilization and reduced scalability.

Sharding in Apache Cassandra

Unlike traditional relational databases, Apache Cassandra performs sharding automatically. Developers do not manually create or manage shards.

Instead, Cassandra distributes data across all nodes in the cluster using a Partition Key. When a row is inserted, Cassandra hashes the partition key to determine which node owns the data.

As the cluster grows, data is automatically distributed across additional nodes, allowing both reads and writes to scale horizontally.

        Spring Boot
             |
             v
     +----------------+
     |  Customer ID   |
     | Partition Key  |
     +----------------+
             |
          Hash()
             |
             v
+------+------+------+------+
|Node 1|Node 2|Node 3|Node 4|
+------+------+------+------+

Partition Key

The Partition Key is the Cassandra equivalent of a sharding key. It determines which node stores a row.

Every read and write operation uses the partition key to locate the correct node without scanning the entire cluster.

For example:

CREATE TABLE customers (
    customer_id UUID,
    name TEXT,
    city TEXT,
    PRIMARY KEY (customer_id)
);

Here, customer_id is the partition key. Cassandra hashes this value and stores the row on the appropriate node.

Composite Partition Key and Clustering Columns

Suppose an application frequently retrieves customers using the following query requirements:

- Filter by customer_id and country.
- Return customers sorted by registration_date.
- Support range queries on registration_date.

A suitable table design is:

CREATE TABLE customers (
    customer_id UUID,
    country TEXT,
    registration_date DATE,
    name TEXT,
    email TEXT,
    PRIMARY KEY (
        (customer_id, country),
        registration_date
    )
);

The columns inside the first parentheses form the Composite Partition Key. Cassandra hashes the combination of customer_id and country to determine which node stores the data.

The remaining column, registration_date, is the Clustering Column. It determines the order of rows within the partition, allowing efficient sorting and range queries. For example:

SELECT *
FROM customers
WHERE customer_id = ?
  AND country = ?
  AND registration_date >= '2025-01-01'
  AND registration_date <= '2025-12-31';

When this query executes, Cassandra first hashes (customer_id, country) to locate the correct partition.

(customer_id, country)
           |
           v
   Composite Partition Key
           |
        Hash()
           |
           v
      +---------+
      | Node 3  |
      +---------+
           |
           v
Rows sorted by registration_date

Because rows within the partition are stored in registration_date order, Cassandra can efficiently return the requested date range without scanning the entire cluster.

Summary

Database Sharding improves scalability by distributing data across multiple independent database servers, allowing both reads and writes to scale horizontally.

Unlike partitioning, which divides data within a single database server, sharding distributes data across multiple servers, overcoming the hardware limitations of a single machine.

Applications typically choose among range-based, hash-based, or directory-based sharding strategies based on their access patterns and scalability requirements.

In large-scale systems, sharding is often combined with replication to provide both horizontal scalability and high availability.