Instead of directing all traffic to a single server, incoming requests are distributed across multiple service instances using a load balancer.
Load balancing can operate at different layers of the network stack.
Layer 4 (L4) load balancers make routing decisions using transport-layer information such as IP addresses and TCP/UDP ports.
Layer 7 (L7) load balancers inspect application-layer data such as HTTP headers, URLs, cookies, and host names to perform intelligent request routing.
In this section, we'll explore how L4 load balancing, L7 load balancing, and Consistent Hashing work, when each approach should be used, and the trade-offs involved in designing scalable distributed systems.
L4 Load Balancing
L4 Load Balancing works at the transport layer (TCP/UDP). It does not understand HTTP methods, cookies, headers, or payloads. It only looks at IP addresses and ports.This makes it extremely fast and capable of handling millions of concurrent connections with minimal overhead.
They maintain fairness in network connections, perform health checks, and distribute packets evenly using techniques like Round Robin and Least Connections.
Because they operate below the application layer, they are protocol-agnostic.
L7 Load Balancing
L7 Load Balancing works at the application layer (HTTP/HTTPS/WebSockets). It can inspect the full request—paths, headers, cookies, JWT tokens, MIME types—and make intelligent decisions.
This enables powerful routing capabilities like:
- Path-based routing (/products/* → Product Service),
- Header-based routing (mobile vs desktop clients),
- Canary deployments,
- A/B testing,
- Protocol upgrades (WebSockets, gRPC),
- Traffic shadowing,
- Request rewriting and transformation.
A canary deployment is a risk-mitigation strategy for releasing new software versions by gradually rolling them out to a small subset of users (the "canary group") before a full release, allowing for real-world testing, performance monitoring, and quick rollback if issues arise, much like canaries in coal mines warned miners of gas.
A/B testing (or split testing) is a method to compare two versions (A & B) of something (like a webpage, email, or app feature) by showing them to different user groups to see which performs better at achieving a goal, like more clicks or sign-ups, using data to make informed decisions instead of guesses.
It involves splitting traffic, running the experiment, collecting data, and using statistical analysis to find the "winner" for optimization.
Consistent Hashing (Stable Routing)
Consistent Hashing is a technique used to map requests or keys to servers in a way that minimizes disruption when nodes are added or removed.Unlike standard hashing (modulo-based), which remaps almost all keys when a node changes, consistent hashing only remaps a small fraction of them.
This property is essential when the system requires:
1. Session stickiness: The load balancer keeps a user connected to the same backend server for the entire session, ensuring consistent experience and avoiding re-authentication or state loss.
2. Cache affinity: Requests for the same data are routed to the server that already has that data cached, improving speed and reducing redundant computation.
3. Shard routing: Incoming requests are directed to a specific shard or partition based on a hashing rule, ensuring that related data is always handled by the correct backend segment.
4. Distributed storage mapping: Data is placed across multiple storage nodes using a consistent mapping strategy, allowing efficient lookup and high scalability without centralized coordination.
5. Real-time personalization: Requests are routed to systems that maintain user-specific context, enabling dynamic, personalized responses such as tailored recommendations or pricing.
Systems like Redis Cluster, Cassandra, Kafka partitioners, and service meshes (Envoy/Istio) rely heavily on consistent hashing.
Why typical load balancing fails for caches? Traditional load balancing algorithms such as Round Robin or Least Connections distribute requests across backend servers without considering where cached data resides.
As a result, requests from the same user may be routed to different application instances or cache nodes, leading to frequent cache misses and unnecessary database queries.
To maintain cache locality, systems often use consistent hashing, where requests are routed based on a hash of a stable identifier.
How Consistent Hashing Works
Traditional hashing uses something like:
serverIndex = hash(key) % numberOfServers
The problem is, if the number of servers changes (scale up/down), almost every key remaps, which destroys caching consistency.
Consistent Hashing solves this by placing both servers and keys on a ring. Clients route a key to the next server clockwise on the ring.
Step 1: Build a Hash Ring
1. Hash each server's ID (e.g., IP/hostname).
2. Place them on a circular ring ranging from 0 → 2^32.
3. Hash each request key (e.g., userId, cartId) and place it on the same ring.
When a key is hashed, it moves CLOCKWISE to the nearest server on the ring.
Step 2: Routing a Request
For example, a userId=42 hashes to a position on the ring.
If the next server clockwise is Server B, all requests for user 42 always go to Server B.
This keeps request–cache affinity stable.
Step 3: Node Failure Scenario
- Server B fails (goes offline)
- In modulo hashing → almost ALL keys remap.
- In consistent hashing → only keys that belonged to Server B are affected.
What happens?
All keys that mapped to Server B now move to the next server clockwise, say Server C.
All keys that belonged to Server A and Server C remain untouched.
Before Failure: A → B → C → A
After Failure: A → C → A
Only B's segment is redistributed. The rest of the ring is perfectly stable. This minimizes cache misses and preserves system performance.
Step 4: New Node Addition (Scale Out)
- Adding Server D to handle more load.
- Server D is placed on the ring based on its hash.
What happens?
Only keys in the segment between previous server and Server D move to Server D.
All other keys stay on their existing nodes.
Old Ring: A → B → C → A
New Ring: A → B → C → D → A
Only a portion of keys previously assigned to A (or C depending on position) now move to D.
Minimal key movement = minimal cache disruption.
Virtual Nodes (VNodes) – Improving Load Distribution
In real systems, servers don't have identical performance characteristics. If we use just one position per server on the ring:
- One server might accidentally get a large portion of the hash space.
- Load may be uneven (hotspots).
- Adding/removing nodes causes larger shifts in key ownership.
Virtual Nodes fix this by placing each server multiple times on the ring, making the distribution smoother and more predictable.
serverIndex = hash(key) % numberOfServersMinimal key movement = minimal cache disruption.
Ring Without Virtual Nodes (Uneven Load Distribution)
Consider three servers: A, B, and C. Each server is assigned a single position on the hash ring based on its hash value.Suppose they are placed as follows:
------ A ---- B -------------------- C ---- (back to A)
Notice that the distances between the servers are highly uneven:
A → B is a small segment.
B → C is a very large segment.
C → A is a medium-sized segment.
In consistent hashing, each server is responsible for the keys that fall between the previous server and its own position on the ring.
Since the segment before C is much larger than the others, Server C ends up owning significantly more keys, while Server B owns very few.
As a result, the traffic and data distribution become unbalanced. Some servers may become overloaded while others remain underutilized, leading to unpredictable performance.
Furthermore, if one server is added or removed, a large continuous portion of the ring must be reassigned, causing significant cache invalidation and data movement.
Ring With Virtual Nodes (Balanced Load Distribution)
To solve this problem, each physical server is represented by multiple virtual nodes (VNodes) that are placed at different positions on the hash ring.For example:
A → A#1, A#2, A#3
B → B#1, B#2, B#3
C → C#1, C#2, C#3
A possible arrangement on the ring might look like:
A#1 → C#1 → B#1 → A#2 → C#2 → B#2 → A#3 → C#3 → B#3 → (back to A#1)
Now each physical server owns several smaller segments distributed across the ring instead of one large continuous segment. For example:
A#1 → C#1 belongs to Server A.
C#1 → B#1 belongs to Server C.
B#1 → A#2 belongs to Server B.
A#2 → C#2 belongs to Server A.
C#2 → B#2 belongs to Server C.
B#2 → A#3 belongs to Server B.
A#3 → C#3 belongs to Server A.
C#3 → B#3 belongs to Server C.
B#3 → A#1 belongs to Server B.
Although the individual segments may still vary in size, each server now owns multiple segments spread throughout the ring.
The total load becomes the sum of many small segments rather than one large segment, producing a much more balanced distribution.
This approach offers several advantages:
1. No server owns a single disproportionately large portion of the ring.
2. Random variations are averaged across multiple virtual nodes.
3. Hot keys are naturally spread across the cluster.
4. Overall load distribution becomes far more uniform.
Even if one virtual node of Server A happens to fall in a crowded region, its remaining virtual nodes are likely to be located elsewhere, balancing the overall workload.
Adding a Node Without Virtual Nodes
Suppose a new Server D is added:--- A --- B --------------------- C ---- (D added here)
If D's hash lands inside the large segment between C and A, it immediately becomes responsible for that entire portion of the ring.
This results in a large number of keys moving to the new server, causing significant cache invalidation, sudden changes in traffic distribution, and temporary performance degradation while caches warm up.
Adding a Node With Virtual Nodes
With virtual nodes, the new server is also represented by multiple positions on the ring.For example:
D → D#1, D#2, D#3
The ring might now look like:
A#1 → C#1 → D#1 → B#1 → A#2 → C#2 → B#2 → D#2 → A#3 → C#3 → B#3 → D#3 → A#1
Each virtual node of Server D only takes ownership of the small segment immediately preceding it.
Instead of one large section of the ring being reassigned, several small sections are redistributed across the cluster.
For example, without virtual nodes, adding a server might require approximately 40% of the keys to move if the new server lands in a large segment.
With virtual nodes, only the small portions owned by D#1, D#2, and D#3 are reassigned, which may result in only about 10–15% of the keys moving, depending on the number of virtual nodes.
This significantly reduces cache misses, minimizes data movement, balances the load more evenly, and allows the cluster to scale predictably with minimal disruption.
Conclusion
L4 Load Balancer receives raw TCP connections and distributes load quickly across edge servers. L4 Load Balancing gives raw speed and massive throughput.L7 Load Balancer inspects HTTP requests and routes to the appropriate microservice based on paths, headers, or version rules. L7 Load Balancing adds intelligence and control.
Consistent Hashing ensures stability and affinity under system churn.
