1 Overview
Most teams adopt a caching layer for one reason: the origin cannot absorb the traffic it is being asked to serve. The cache is introduced as a pressure valve, and in the first weeks it works exactly as advertised. Latency drops, origin CPU falls, and the graph looks healthy. The trouble starts later, when the cache quietly stops being treated as a pressure valve and starts being treated as the system of record. Engineers read from it as if it were authoritative, write to it as if writes were durable, and reason about it as if its contents were guaranteed to be there.
That shift is rarely a single decision. It accumulates through small conveniences: a read path that skips the origin because the value is almost always present, a background job that assumes a key exists, a feature flag stored only in the cache. Each choice is locally reasonable and collectively fragile. The cache becomes load-bearing in a way nobody designed, and the first symptom is usually not a crash but a subtle correctness bug that appears only under eviction, failover, or a cold start.
This article is about that failure mode. It is not an argument against caching, which remains one of the highest-leverage tools available. It is an argument for keeping the mental model honest: a cache is a copy with a lifetime, and every design decision should survive the copy disappearing without warning.
2 Where the common advice breaks down
The standard guidance is simple: cache aggressively, pick a TTL, invalidate on write, and move on. Each clause hides an assumption. Cache aggressively assumes your keys are stable and your working set fits. Pick a TTL assumes one number is appropriate for every object. Invalidate on write assumes you can enumerate every key that a write affects, which is often false the moment a single logical entity is rendered into several cached representations.
The invalidation clause fails most often. A write to one record may change a detail page, a list page, a search result, a feed, and an aggregate count. If those are separate cache keys, the write path must know all of them. Teams discover this the hard way when a price change appears on the product page but not in the category listing for another ten minutes. The usual fix is a purge call that fans out to a list of key patterns, which works until the pattern list drifts out of sync with the rendering code.
The TTL clause fails more quietly. A single global TTL is a compromise that is too long for volatile data and too short for static assets, so it wastes both freshness and hit rate. The aggressive-caching clause fails under memory pressure: when the cache evicts, the origin sees a stampede of simultaneous misses for the same popular key, and the load spike can exceed what the cache was meant to prevent. None of these are exotic; they are the normal consequences of applying a one-line rule to a system with many object classes.
3 How the mechanism actually works
A cache is a mapping from a key to a value with an associated lifetime and an eviction policy. The key is derived from the request, which means anything that varies the response must vary the key, and anything that does not must be normalized away. Query parameter ordering, tracking parameters, accept headers, and cookies all influence whether two requests share an entry. A cache that keys on the full URL will fragment on parameters that do not change the response, silently lowering the hit rate while looking correct.
Eviction is the part teams underestimate. Memory is finite, so the cache must discard entries, and the policy decides which. Least-recently-used eviction is common and reasonable, but it means a hot key can be evicted if a burst of one-off keys arrives. When that happens, the next request for the hot key misses and goes to the origin. If many clients request it at once, they all miss together, which is the stampede problem. Request coalescing, sometimes called single-flight, collapses those concurrent misses into one origin fetch and is often the single most valuable safeguard.
Expiry and eviction are different events with different consequences. Expiry is predictable and can be staggered with jitter so that a batch of keys written together does not expire together. Eviction is unpredictable and should be treated as always imminent. Any code path that assumes a key is present is a latent bug, because the only guarantee a cache offers is that a lookup may or may not return a value.
4 Trade-offs you are really making
The central trade-off is freshness against origin load. Longer lifetimes and broader sharing raise the hit rate and reduce the work the origin does, but they widen the window in which a user can see data that no longer reflects reality. Shorter lifetimes narrow that window but push more requests to the origin, which costs latency and capacity. There is no setting that optimizes both; there is only a point on the curve that matches how much staleness your product can tolerate.
The second trade-off is sharing against correctness. A cache entry shared across users is efficient, but it is only safe if the response does not depend on who is asking. The moment a response embeds a name, a permission, or a price tier, sharing it is a data leak waiting for the wrong key. Splitting the key by user restores correctness and destroys the hit rate, because each user now has a private entry that is unlikely to be reused. The honest answer is that personalized responses usually should not be cached at the shared layer at all.
The third trade-off is operational simplicity against control. A short TTL and no purge endpoint is simple and self-healing. A long TTL with a purge API gives finer control but adds a component that can fail, a key inventory that can drift, and a debugging surface that grows with every new cached view. Teams often add purge capability before they need it and then depend on it, which converts a convenience into a correctness requirement.
5 Implementing it correctly
Start by classifying objects by change frequency rather than by endpoint. Immutable assets, reference data, and slowly changing content can carry long lifetimes. Volatile data should carry short lifetimes or no caching at all. Assigning a lifetime per class, and documenting why, prevents the drift that comes from copying a TTL from a neighboring service. Add jitter to expiry so that objects written in the same batch do not all expire on the same second.
Make the key explicit and minimal. Normalize parameters that do not affect the response, include everything that does, and version the key when the response shape changes so that old entries become unreachable rather than wrong. For correctness on writes, prefer changing the key over purging it: a version suffix or content hash means a new write produces a new key, and the old entry expires on its own. This turns invalidation from a distributed coordination problem into a key-naming problem, which is far easier to get right.
Treat every read as a possible miss. The code path should fetch from the origin and repopulate on miss without special casing, and concurrent misses for the same key should be coalesced. Set a timeout on origin fetches so a slow origin does not hold many waiting requests. Finally, decide in advance what happens when the cache is unavailable entirely: if the answer is that the site goes down, the cache is not a cache, it is a dependency, and it should be designed and operated as one.
6 Measuring whether it helped
Hit rate is the first metric everyone watches and the least informative on its own. A high hit rate can coexist with a terrible user experience if the cached entries are the wrong ones, and a low hit rate can be fine if the misses are cheap. Track hit rate by object class and by key, not just in aggregate, so that a single noisy endpoint cannot hide behind healthy averages. A falling hit rate on one class usually points to a key change, a new parameter, or an eviction pattern.
Measure origin load alongside hit rate, because that is the outcome the cache exists to produce. Look at request volume, latency percentiles, and error rates at the origin, and compare them against the cache's own latency. If the cache is fast but the origin is unchanged, the cache is absorbing requests that were never expensive. If the origin is still saturated, the cache is missing the traffic that matters, often because the hot keys are not the ones being stored.
Finally, measure staleness directly. Instrument how long after a write the old value can still be served, and compare that to what the product actually requires. This is the number that determines whether your lifetime choices are defensible, and it is the number most teams never compute. A cache that is fast, cheap, and occasionally wrong is not a success; it is a deferred incident.
7 Frequently asked questions
How long should a TTL be? There is no universal answer, and any specific number offered without context should be distrusted. Derive it from how often the underlying data changes and how much staleness the product tolerates. If a value changes hourly and users can accept an hour of lag, an hour is reasonable. If a value changes constantly and correctness matters, the right lifetime may be seconds or none at all.
Should I invalidate on write or rely on expiry? Prefer expiry with versioned keys for correctness, and reserve explicit invalidation for cases where a bounded, known set of keys must change immediately. Explicit invalidation is powerful but fragile, because it requires the write path to know every cached representation of the data it changed. When that knowledge is incomplete, stale entries survive.
Why does my hit rate drop after a deploy? Deploys often change response shapes, add parameters, or alter key construction, which makes previously cached entries unreachable. That is expected and usually healthy, because it prevents old and new formats from mixing. If the drop persists, check whether a new query parameter or header is fragmenting the key space.
Is a cache a database? No. A cache may discard any entry at any time, so it cannot be the only place a value lives. If losing an entry would lose data or break correctness, that value belongs in durable storage, with the cache holding only a copy.