The application uses the response that arrives first and cancels the remaining requests.
This technique is particularly useful in large distributed systems where most requests complete quickly, but a small percentage experience significantly higher latency due to temporary slowdowns.
What is Tail Latency?
In a distributed system, most requests complete within the expected response time.However, a small percentage become significantly slower because of temporary issues such as network congestion, CPU contention, garbage collection pauses, disk I/O, or resource contention.
These unusually slow requests are known as tail latency because they appear at the end (or tail) of the latency distribution.
Request Count
^
|
12 | █
10 | ██
8 | ███
6 | █████
4 | ██████████
2 | ██████████████████████████████████████
0 +-------------------------------------------------->
50 100 150 200 250 300 500 800 ms
Most Requests Tail Latency
Although these slow requests are relatively rare, they often determine the response time perceived by users.
For example, suppose a Product API retrieves product information by calling three independent services.
Product API
|
+---------+---------+
| | |
v v v
Inventory Pricing Reviews
Inventory 50 ms
Pricing 60 ms
Reviews 800 ms
--------------------
Total 800 ms
Even though the Inventory and Pricing services respond quickly, users still experience an 800-millisecond response because of the delayed Reviews service.
Request Hedging is designed to reduce the impact of these unusually slow requests by sending a duplicate request to another healthy replica when the original request appears unusually slow.
How Does Request Hedging Work?
Rather than sending duplicate requests immediately, request hedging waits for a short period before issuing a second request to another healthy replica.If the original request completes within the expected time, no additional request is sent.
If the request exceeds the configured latency threshold, a duplicate request is sent to another replica. The application returns the response that arrives first and cancels the remaining request.
Time
|
|------ Original Request -------------------------->
|
|----100 ms----|
| +------ Hedged Request ------------>
|
+-----------------------------------------------> First Response Wins
|
v
Cancel Remaining Request
Spring Boot Example
Suppose an Inventory Service is deployed across multiple instances behind a load balancer. Order Service
|
+-------+-------+
| |
v v
Inventory-1 Inventory-2
The application returns the first successful response and cancels the remaining request.
public class RequestHedger {
private final ExecutorService executor =
Executors.newVirtualThreadPerTaskExecutor();
public T execute(
Supplier primary,
Supplier secondary,
Duration hedgingDelay
) {
CompletableFuture primaryRequest =
CompletableFuture.supplyAsync(
primary,
executor);
CompletableFuture hedgedRequest =
primaryRequest
.completeOnTimeout(
null,
hedgingDelay.toMillis(),
TimeUnit.MILLISECONDS)
.thenCompose(result -> {
if (result != null) {
return CompletableFuture.completedFuture(result);
}
return CompletableFuture.supplyAsync(
secondary,
executor);
});
return hedgedRequest.join();
}
}
If the original request is still pending after the configured hedging delay, the requestHedger sends a duplicate request to Inventory-2. Whichever instance responds first provides the result, while the slower request is cancelled.
Because only unusually slow requests trigger a duplicate call, request hedging significantly reduces tail latency while adding only a small amount of additional network traffic.
Advantages
Request hedging significantly reduces tail latency without requiring changes to business logic.It improves response times for latency-sensitive applications and reduces the impact of temporary server slowdowns, network delays, and resource contention.
Since only a small percentage of requests require hedging, the increase in network traffic is usually much smaller than sending duplicate requests for every operation.
Limitations
Request hedging increases overall request volume because duplicate requests may be sent to multiple replicas.If the hedging delay is configured too aggressively, unnecessary requests can increase CPU utilization and network traffic, potentially reducing overall system efficiency.
It is also unsuitable for non-idempotent operations because duplicate execution may produce incorrect business results.
Request hedging is effective only when multiple healthy replicas of the same service are available.
Summary
Request Hedging is a latency optimization technique that reduces tail latency by sending duplicate requests to multiple replicas when the original request appears unusually slow.Unlike retries, which occur after failures, request hedging proactively addresses slow responses and returns the first successful result while cancelling the remaining requests.
It is most effective for read-only, idempotent operations running on replicated services, where reducing tail latency can significantly improve overall application responsiveness and user experience.
