1 Overview
When latency spikes or origin load climbs, the load balancer is often the first suspect. It is visible, it sits in the request path, and its metrics are easy to graph. But in many production incidents, the real culprit is the cache key. A cache key that includes too many request attributes, or too few, can turn a perfectly healthy load balancing tier into a bottleneck. The load balancer distributes requests, but the cache decides which requests can be served without touching the origin at all. If the cache key is wrong, every request becomes a miss, and the load balancer is forced to fan out traffic that should never have reached the backend.
This article explains why cache key design deserves the same scrutiny as load balancing algorithms and connection pooling. It covers how cache keys are constructed, where common advice fails, and how to measure whether a change actually improved hit rate and reduced origin pressure. The goal is not to blame one component but to show that caching and load balancing are coupled. A load balancer cannot compensate for a cache that never hits, and a cache cannot fix a load balancer that is misconfigured. The cache key is the contract between them.
2 Where the common advice breaks down
The most common advice is to cache by URL. That works for static assets with stable paths, but it collapses as soon as the application uses query parameters for personalization, tracking, or pagination. A URL like /products?utm_source=newsletter&session_id=abc123 looks unique to the cache even though the underlying product listing is identical for thousands of users. The result is a cache filled with near-duplicate entries, each with a short lifetime, and a hit rate that never climbs. Teams then add more cache capacity, assuming the problem is size, when the problem is key diversity.
Another common recommendation is to include every header that might affect the response. That is safe in theory but disastrous in practice. Headers such as User-Agent, Accept-Language, and Cookie vary widely across clients, and including them in the cache key fragments the cache into tiny buckets. The cache becomes a write-heavy store with almost no reuse. The opposite advice, to ignore all headers, risks serving a cached response to a user who should have received a different variant. The breakdown is not in the mechanism but in the assumption that a single rule fits all endpoints. Cache keys must be designed per resource, based on which request attributes actually change the response body.
3 How the mechanism actually works
A cache key is a deterministic representation of the request that the cache uses to decide whether a stored response can be reused. At its simplest, the key is the request method plus the normalized URL. More advanced caches allow the key to include selected headers, cookies, or query parameters, and to exclude others. When a request arrives, the cache computes the key, looks it up, and either returns a stored response or forwards the request to the origin. The origin response is then stored under that key, subject to freshness rules and eviction policies.
The critical detail is that the key is computed before the cache knows anything about the response. If an attribute is not in the key, the cache cannot distinguish between two requests that differ only by that attribute. If an attribute is in the key but does not affect the response, the cache stores redundant copies. This is why normalization matters. Sorting query parameters, stripping tracking parameters, and canonicalizing case or trailing slashes can collapse many request variants into one key. Conversely, adding a header like Accept-Encoding is necessary when the origin returns compressed or uncompressed bodies. The cache key is not a security boundary; it is a reuse boundary. Every attribute in the key is a claim that the response may differ when that attribute differs.
4 Trade-offs you are really making
Every cache key decision trades hit rate against correctness. A broad key, such as one that ignores cookies and most query parameters, produces a high hit rate because many requests map to the same entry. That reduces origin load and latency, but it also means a response generated for one user may be served to another. For public, non-personalized content, that is acceptable and often desirable. For account pages, shopping carts, or any response that embeds user-specific data, a broad key is a correctness bug waiting to happen.
A narrow key, such as one that includes session identifiers and all headers, maximizes correctness but destroys reuse. The cache becomes a per-user store, and the hit rate approaches zero for users who make only a few requests. The origin sees nearly all traffic, and the cache adds latency without reducing load. The middle ground is to separate resources by volatility and personalization. Static assets can use a broad key. Public API responses can use a key that includes only parameters that change the response. Personalized fragments can be cached separately with a key that includes a user or tenant identifier, or they can be excluded from shared caching entirely. The trade-off is not binary; it is a spectrum that must be chosen per endpoint.
5 Implementing it correctly
Start by inventorying the request attributes that actually influence the response body. For each endpoint, list the query parameters, headers, and cookies that the origin reads when generating the response. Anything not on that list should be excluded from the cache key. Then normalize what remains: sort query parameters alphabetically, lowercase header names, strip known tracking parameters, and remove empty values. This alone can collapse a large number of variants into a single key without changing correctness.
Next, decide how to handle personalization. If a response contains user-specific data, either exclude it from shared caching or split it into a cacheable shell and a personalized fragment fetched separately. For multi-tenant systems, include a tenant identifier in the key but not a user identifier unless the response truly varies per user. Use explicit cache-control directives to communicate freshness and revalidation to the cache, and version keys when the response format changes. Finally, test with real traffic patterns. A cache key that looks correct in a unit test can still fragment under production query strings. Log the computed key for a sample of requests and inspect the distribution. If the top keys account for a small fraction of traffic, the key is too narrow. If a single key serves wildly different responses, it is too broad.
6 Measuring whether it helped
The primary metric is hit rate, but hit rate alone is misleading. A cache can report a high hit rate while serving stale or incorrect content. Measure hit rate alongside origin offload, which is the fraction of requests that never reach the origin. If hit rate rises but origin offload does not, the cache is absorbing requests that would have been cheap anyway, or the origin is still being hit for revalidation. Also track the distribution of cache keys. A healthy cache has a relatively small number of keys that account for most traffic. If the number of unique keys grows linearly with request volume, the key is too specific.
Latency percentiles for cache hits and misses should be monitored separately. A cache that adds latency on misses without improving hit latency is not helping. For correctness, sample responses from cache and compare them to origin responses for the same logical request. This catches cases where a broad key serves the wrong variant. Finally, watch origin error rates and load balancer queue depths. If the cache key change reduced origin traffic, the load balancer should show lower connection counts and shorter queues. If those metrics do not move, the bottleneck is elsewhere. The goal is not a perfect hit rate but a measurable reduction in origin work without a rise in incorrect responses.
7 Frequently asked questions
Should the cache key include the full URL or just the path? It depends on whether query parameters change the response. If they do, include the ones that matter and normalize the rest. If they do not, exclude them entirely. Including the full URL is the safest default for correctness but often the worst for hit rate.
How do cookies fit into the cache key? Cookies are request attributes like any other. If the origin uses a cookie to vary the response, that cookie must be in the key or the response must not be shared. For session cookies that only identify a user but do not change the response, excluding them is usually safe. For authentication or preference cookies that change content, include them or bypass the cache.
What about the Vary header? The Vary header tells the cache which request headers were used to select the response. It is a signal, not a key itself. If the Vary header lists headers that are not in the cache key, the cache may serve the wrong variant. Keep Vary and the cache key consistent.
How often should the cache key be reviewed? Whenever an endpoint changes how it reads request attributes, or whenever hit rate drops without a traffic change. Cache keys are part of the API contract and should be versioned and tested like any other interface.