In this article, we'll explore what these terms really mean, how they are measured, the trade-offs involved in optimizing them, and the techniques used to build low-latency, high-throughput systems that scale efficiently.
1. Understanding Performance
Performance describes how efficiently a system uses CPU, memory, storage, and network resources to complete work.In practice, this means how much work the system can perform, how quickly it performs that work, and how consistently it maintains that behavior under varying load.
A system that responds quickly under light load but slows dramatically as traffic increases is not performing well.
Likewise, a system capable of processing massive workloads while consuming excessive CPU, memory, or network bandwidth is not truly performant because it wastes infrastructure resources.
High performance is achieved by eliminating unnecessary work, optimizing critical code paths, minimizing bottlenecks, and making efficient use of the underlying hardware.
Example: Optimizing a Product Search API
Imagine you run an e-commerce platform with a Product Search API.Initially, every search request queries a large product catalog, joins multiple tables to fetch additional metadata, sorts the results, and then serializes the response before sending it back to the client.
Under normal traffic, each request completes in around 150 ms, which appears acceptable.
As traffic increases, however, the database becomes overloaded, CPU utilization rises, queries take longer to execute, and response times increase to 1–2 seconds.
To improve performance, you might:
- Move search queries to an indexed search engine such as Elasticsearch, which is optimized for fast searching and filtering.
- Cache frequently searched products in memory to avoid redundant database queries.
- Perform sorting within the database or search engine instead of the application.
- Return only the fields required by the client to reduce serialization and network overhead.
- Precompute frequently accessed product metadata to eliminate expensive joins.
After these optimizations:
- Average response time drops from 150 ms to around 40 ms.
- The API continues to respond consistently even under 10× higher traffic.
- CPU utilization decreases by nearly 60%.
- Fewer application servers are required, reducing infrastructure costs.
Notice that none of these optimizations required faster hardware.
The improvement came from eliminating unnecessary work, reducing bottlenecks, and optimizing the system's hot path—the portion of the code executed most frequently.
This is the essence of performance engineering, accomplishing the same work using fewer resources and in less time.
2. Understanding Latency
Latency is the delay experienced while processing a single request from start to finish. From a user's perspective, it is the time between sending a request and receiving the first meaningful response.Even a system with enormous computing power can suffer from poor latency if requests spend time waiting on slow I/O, network calls, locks, garbage collection (GC) pauses, or overloaded dependencies.
Response Time = Waiting Time + Processing Time
| Latency (ms) | Perceived By User | Typical Use Case |
|---|---|---|
| 1–5 ms | Instantaneous | High-frequency trading, cache hits |
| 20–50 ms | Smooth | Social media applications |
| 100–200 ms | Slight delay | E-commerce and business APIs |
| 500+ ms | Noticeable lag | Heavy workflows or poorly optimized systems |
Looking only at the average latency can be misleading because it hides these slow requests Instead, one should rely on percentiles to understand how latency is distributed across all requests.
Understanding Latency PercentilesAcross many engineering teams, p95 latency is treated as the primary customer-experience metric because it reflects how the system performs for nearly all users.
p50 latency (Median): 50% of requests complete faster than this value, while the remaining 50% take longer.
p95 latency: 95% of requests complete within this value, while the slowest 5% exceed it.
p99 latency: 99% of requests complete within this value, while only the slowest 1% take longer.
How are percentiles calculated?
The system continuously records request latencies over a specific time window, sorts them, and calculates percentile values for that window.
Examples:
Last 1 minute → Percentiles are calculated using requests received during that minute.
Last 5 minutes → Percentiles are calculated using requests from the previous five minutes.
Last 1 hour → Percentiles represent requests collected during that hour.
Monitoring tools such as Prometheus, Grafana, and Datadog allow you to choose the time range over which these percentile values are calculated.
p99 latency, on the other hand, highlights the worst-performing requests and often serves as an early warning signal for contention, resource exhaustion, or cascading failures.
A system that performs well on average but experiences poor tail latency cannot scale gracefully because a small percentage of extremely slow requests can significantly impact the overall user experience.
Latency Example
The following pseudocode demonstrates one of the most common sources of high latency in distributed systems—a blocking database call placed directly on the application's critical request path.public User getUser(String id) {
// Adds ~120 ms latency under load
return database.query(
"SELECT * FROM users WHERE id = ?",
id
);
}
If the database experiences slow disk I/O, lock contention, or increased request queues, the original 120 ms latency can quickly grow into several hundred milliseconds.
Because this method lies on the application's critical request path, the additional waiting time affects every request. As traffic grows, these delays accumulate, increasing queue lengths, reducing throughput, and ultimately degrading the end-user experience.
To reduce this latency, frequently accessed data can be served directly from an in-memory cache, eliminating expensive database round-trips altogether.
When multiple records must be retrieved, batching them into a single query significantly reduces overhead.
Many modern systems also adopt asynchronous I/O, allowing the application to continue useful work while waiting for the database.
In scenarios with predictable read patterns, data is often precomputed or denormalized, enabling a single lightweight lookup instead of multiple expensive joins.
Latency vs Response Time
Although the terms latency and response time are often used interchangeably, they represent different concepts.Latency refers to the delay experienced while waiting for an operation to progress, such as network transmission, queueing, disk I/O, or waiting for a remote service.
Response time is the total time required to complete a request, including waiting delays, computation, serialization, and transmitting the response back to the client.
Example: Suppose a user requests their profile.
The service spends 40 ms waiting for the database to return the requested data. It then takes another 45 ms to process the result, serialize it into JSON, apply security checks, and send the response back to the client.
The database contributes 40 ms of latency, while the user experiences a total response time of 85 ms.
In other words, latency is one component of response time, whereas response time represents the complete end-to-end duration of the request.
3. Understanding Throughput
While latency describes the experience of a single request, throughput measures the total amount of work a system can complete in a given period of time.A high-throughput system can process a large number of operations—such as HTTP requests, database transactions, messages from a queue, or records written to storage—without becoming a bottleneck.
In practice, engineers increase throughput by introducing more concurrency. This may involve adding additional threads, worker processes, application instances, or replicas so that multiple tasks can be processed simultaneously instead of waiting for one operation to finish before starting the next.
However, increasing concurrency is not free. As more work executes in parallel, contention for shared resources also increases. Threads and processes begin competing for CPU time, memory, locks, disk I/O, and network bandwidth.
Beyond a certain point, adding more concurrency no longer improves throughput and often increases latency. Finding the right balance between throughput and latency is one of the fundamental challenges of performance engineering.
Throughput = Total Work / Time Taken
A system may report impressive throughput numbers while still delivering a poor user experience. Consider a queue processing service capable of consuming 1,000 messages per second. On paper, this appears highly efficient.
However, if each message waits two seconds in the queue before a worker begins processing it, the overall end-to-end latency remains high. Users and downstream services experience the delay regardless of how quickly the worker processes the message once execution begins.
This illustrates an important principle high throughput alone does not guarantee good performance. A well-designed system must balance both throughput and latency to deliver fast, responsive, and scalable applications.
4. Performance vs Latency vs Throughput
These three concepts are tightly interdependent:| Dimension | Focus | What Affects It | Primary Goal |
|---|---|---|---|
| Performance | Overall system efficiency | CPU, memory, I/O, architecture | Doing more with less |
| Latency | Speed of single request | Network, disk, GC, locks | Minimize waiting |
| Throughput | Volume of processed work | Concurrency, batching, scaling | Maximize work rate |
1. Increasing throughput via batching increases latency for individual requests.
2. Reducing latency by serving from cache reduces durability and increases memory pressure.
One of the most important formulas to apply as an engineer is:
Little's Law: L = λ × W
Where:
L = Number of items in system
λ (lambda) = arrival rate (throughput)
W = response time (latency)
This law predicts queue buildup, load behavior, and system collapse points. When your latency increases, queue depth rises, throughput falls, and users experience cascading slowdowns.
5. Designing High-Speed Systems
Building high-speed systems requires optimizing every layer of the application, from algorithms and data structures to databases, networking, concurrency, and hardware utilization.The goal is to maximize performance and throughput while keeping latency consistently low, even under heavy load.
1. Choose Efficient Algorithms and Data Structures: Replacing an O(n²) algorithm with an O(n log n) or O(n) solution usually provides a much larger improvement than hardware upgrades.
2. Eliminate Unnecessary Work: Avoid duplicate database queries, repeated calculations, excessive object creation, and redundant network calls. Cache frequently accessed data and precompute expensive results whenever possible.
3. Reduce Disk and Network I/O: Minimize database round-trips, reduce payload sizes, batch multiple operations together, and avoid unnecessary serialization and deserialization. Whenever possible, keep frequently accessed data in memory.
4. Use Caching Effectively: Frequently requested data can be served directly from an in-memory cache such as Redis or an application cache, avoiding expensive database queries and external service calls.
5. Increase Concurrency Carefully: Choose an appropriate number of threads or workers to improve throughput while minimizing lock contention, and prefer non-blocking or asynchronous operations where appropriate.
6. Remove Bottlenecks: Use profiling, distributed tracing, and monitoring tools to identify the slowest component before attempting optimizations.
7. Scale Horizontally: Horizontal scaling improves throughput and increases fault tolerance, provided the application is designed to be stateless and the workload can be partitioned efficiently.
8. Continuously Measure Performance: Monitor metrics such as latency, throughput, CPU utilization, memory usage, GC pauses, disk I/O, and network utilization.
A cache stampede (also called dogpile effect) happens when:
1. Many requests try to access the same cached data
2. The cache expires or is missing
3. All requests simultaneously hit the database/backend
What's the problem?
- Sudden spike in load on the backend
- Can cause slowdowns or system crash
Example:
1. A popular item's cache expires
2. 10,000 users request it at the same time
3. All 10,000 hit the database → overload
6. Conclusion
Building high-speed systems is not about making individual components faster—it is about designing the entire system to perform efficiently under real-world load.Continuously monitor your systems, optimize the true bottlenecks, and always balance latency, throughput, and resource utilization to deliver the best possible user experience.
