1 Overview
A team notices rising latency and origin load, so they add load balancers and more origin servers. Performance improves briefly, then degrades again. The real problem is often a low cache hit rate: the cache is present but not effective, so most requests still reach the origin. Adding load balancers distributes the load but does not reduce it. Fixing the hit rate reduces the work that needs distributing in the first place.
This case study examines a common pattern: a caching layer is deployed, but its hit rate is poor because cache keys are too specific, TTLs are too short, or responses are marked uncacheable. The team treats the cache as a checkbox rather than a system to tune. When hit rate is low, every request becomes an origin request, and load balancers merely spread the pain across more servers.
The article focuses on the mechanism, trade-offs, and measurement of hit rate improvement. It does not promise specific performance numbers or recommend particular products. Instead, it explains why fixing the cache before scaling out is usually the higher-leverage move, and how to reason about the decisions involved.
2 Where the common advice breaks down
Common advice says to add load balancers when traffic grows. That advice assumes the origin is doing necessary work. If most requests are cacheable but are missing the cache, the origin is doing redundant work. Load balancing then multiplies the capacity to do redundant work, which is expensive and does not address the root cause. The cache may be misconfigured, or the application may be setting headers that prevent caching.
Another piece of common advice is to set short TTLs to avoid stale content. Short TTLs increase freshness but reduce hit rate, because entries expire before they can be reused. The cache then behaves more like a pass-through than a store. Teams may also vary cache keys by user, session, or minor headers, which fragments the cache and lowers hit rate. Each variation creates a separate entry, so even popular content may not be reused across users.
A third breakdown is treating cache invalidation as an afterthought. If invalidation is unreliable, teams shorten TTLs or disable caching for safety. That reduces hit rate and pushes load back to the origin. The advice to add load balancers then becomes a way to cope with a cache that is not doing its job. The better sequence is to fix cacheability, key design, and TTLs first, then scale the origin if needed.
3 How the mechanism actually works
A cache stores a response under a key derived from the request. When a request arrives, the cache computes the key and looks for a matching entry. If found and still fresh, it returns the stored response without contacting the origin. If not found or expired, it forwards the request to the origin, stores the response if allowed, and returns it. The hit rate is the fraction of requests served from cache. A low hit rate means most requests are misses, so the origin does most of the work.
Cacheability is determined by response headers and cache configuration. Responses may be marked public, private, or no-store. A private response is typically not stored in a shared cache, which reduces hit rate for personalized content. The cache key may include the URL, query string, host, and selected headers. Including too many headers, such as cookies or user agent, fragments the cache. Including too few may serve incorrect content to some users. The key design is a balance between reuse and correctness.
TTL controls how long an entry is considered fresh. A longer TTL increases the chance of reuse but also the chance of serving stale content. Invalidation can remove entries before expiry, but it adds complexity and coordination. Some systems use versioned keys, where a change in content changes the key, making old entries unreachable and effectively invalidated. The mechanism is simple, but the configuration determines whether the cache reduces origin load or merely adds a hop.
4 Trade-offs you are really making
The central trade-off is freshness versus hit rate. Longer TTLs and broader cache keys increase hit rate and reduce origin load, but they also increase the risk of serving stale or incorrect content. Shorter TTLs and narrower keys improve correctness but lower hit rate, which pushes load back to the origin. There is no universal setting; the right balance depends on how often content changes and how much staleness the application can tolerate.
Another trade-off is complexity versus control. Explicit invalidation gives precise control over when content changes, but it requires reliable signaling from the application to the cache. If that signaling fails, stale content may persist. Time-based expiry is simpler and more robust, but it cannot guarantee immediate freshness. Versioned keys avoid invalidation coordination but require the application to generate new keys when content changes, which can complicate URL design and caching rules.
A third trade-off is storage and memory versus origin cost. A larger cache with more entries can improve hit rate, but it consumes memory or disk and may require eviction policies that affect which entries survive. A smaller cache may evict useful entries and lower hit rate. The decision affects both cost and latency: better hit rate reduces origin bandwidth and compute, but may increase cache infrastructure cost. The goal is to find the point where the reduction in origin work outweighs the added cache cost and complexity.
5 Implementing it correctly
Start by measuring the current hit rate and identifying the main causes of misses. Break down misses by URL pattern, cache key variation, and response cacheability. Often a small number of patterns account for most misses. For example, a query parameter that does not affect content may be fragmenting the cache. Normalizing or ignoring such parameters can improve hit rate without changing correctness. Similarly, removing unnecessary cookies from the cache key can increase reuse for anonymous users.
Next, review response headers. Ensure that cacheable responses are marked public and have appropriate TTLs. For personalized content, consider caching fragments or using a separate cache key that includes only the necessary user-specific attributes. Avoid caching responses that contain sensitive data in shared caches. If the application sets no-store or private by default, the cache will not store those responses, so hit rate will remain low. Work with application developers to adjust headers where safe.
Then, set TTLs based on content change frequency. Static assets can have long TTLs with versioned filenames. Dynamic content may need short TTLs or explicit invalidation. Implement a purge mechanism if immediate updates are required, and test it under failure conditions. Finally, ensure that the cache is not bypassed by load balancer health checks or monitoring probes. Once the cache is effective, adding load balancers can address genuine origin capacity limits rather than masking a cache problem.
6 Measuring whether it helped
The primary metric is hit rate, but it must be interpreted carefully. A higher hit rate is not always better if it comes from caching incorrect content. Measure hit rate alongside error rates, origin request volume, and latency. If hit rate improves but origin load does not drop, the cache may be serving only small responses while large or expensive requests still miss. Segment hit rate by content type, URL pattern, and user cohort to find where the cache is effective and where it is not.
Also measure origin offload, which is the reduction in requests or bytes served by the origin. A high hit rate on small objects may not reduce origin bandwidth much. Conversely, a moderate hit rate on large objects can significantly reduce bandwidth. Track cache eviction rates and memory usage to ensure the cache is not thrashing. If eviction is high, the cache may be too small or the TTLs too long for the working set.
Finally, measure user-perceived latency and error rates. A cache that serves stale content may reduce latency but increase user-visible errors or support tickets. Run experiments where possible, such as A/B tests with different TTLs or key designs, and compare both performance and correctness. The goal is not a perfect hit rate but a balanced improvement that reduces origin load without harming correctness. After confirming the cache is effective, revisit the need for additional load balancers.
7 Frequently asked questions
What hit rate should I aim for? There is no universal target. It depends on content mix, traffic patterns, and tolerance for staleness. A cache that serves most static assets and a portion of dynamic content may be effective even if overall hit rate is moderate. Focus on reducing unnecessary origin requests rather than chasing a number.
Should I cache personalized content? It depends on sensitivity and variability. Caching personalized content in a shared cache can leak data between users. Safer approaches include caching only anonymous content, caching fragments, or using a private cache keyed by user. Each approach has trade-offs in hit rate and complexity.
How do I handle invalidation? Prefer time-based expiry for simplicity, and add explicit purge only where immediate updates are required. Versioned keys can avoid invalidation coordination but require application changes. Test invalidation under failure to ensure stale content does not persist indefinitely.
When should I add load balancers? Add them when the origin is genuinely capacity-constrained after the cache is effective. If the cache is missing most requests, load balancers will not solve the underlying problem. Fix the hit rate first, then scale the origin if needed.