1 Overview
Round-robin load balancing is often the first algorithm engineers reach for because it is simple, stateless, and easy to reason about. Each incoming request goes to the next backend in a rotating list, which spreads traffic evenly when requests are independent and backends are interchangeable. That assumption holds for many stateless APIs, but it breaks down when backends are not interchangeable because they hold cached state. In a cache-backed system, the backend that receives a request often determines whether the response can be served from memory or must be fetched from an origin. When requests for the same key land on different backends, each backend may cache the same object separately, wasting memory and increasing origin load.
Hash-based load balancing changes the routing decision so that a given key consistently maps to the same backend. Instead of rotating through servers, the load balancer computes a hash of a request attribute, such as a URL path, cache key, or session identifier, and uses that hash to select a backend. The result is that repeated requests for the same resource tend to hit the same cache node, improving locality and reducing duplicate work. This approach is not universally better; it introduces new failure modes around rebalancing and hot keys that round-robin avoids. The goal of this article is to explain when hash-based routing helps, how it works under the hood, and what trade-offs you accept when you adopt it.
2 Where the common advice breaks down
The common advice to use round-robin assumes that any backend can serve any request with equal cost. That is true for stateless compute, but it is false for cache nodes. If you run a pool of reverse-proxy caches or in-memory data stores behind a round-robin balancer, each node will independently cache whatever requests it happens to receive. A popular object may be fetched from the origin once per backend, multiplying origin traffic by the number of nodes. Memory usage also becomes inefficient because the same object is stored multiple times. The more backends you add, the worse the duplication becomes, which is the opposite of what horizontal scaling should achieve.
Round-robin also interacts poorly with cache invalidation. When an object changes, you need to purge it from every backend that might hold a copy. With round-robin, that could be any node, so invalidation becomes a broadcast operation. Broadcast purges are slower and more failure-prone than targeted purges, and they can cause a thundering herd if many nodes miss simultaneously and refetch from the origin. Hash-based routing narrows the set of nodes that can hold a given key, which makes invalidation more predictable. The trade-off is that you must handle membership changes carefully, because adding or removing a node can remap many keys and cause a wave of cache misses if the hash ring is not designed well.
3 How the mechanism actually works
A hash-based load balancer takes a request attribute and runs it through a hash function to produce a number. That number is then mapped onto a set of backends. The simplest form is modulo hashing: compute hash(key) mod N, where N is the number of backends, and send the request to that backend. This gives consistent routing as long as N does not change. The problem is that when N changes, almost every key maps to a different backend, causing a massive cache miss storm. Consistent hashing solves this by placing both keys and backends on a circular hash space, often called a ring. Each backend is assigned one or more positions on the ring, and each key is routed to the first backend clockwise from its hash position. Adding or removing a backend only remaps the keys that fall between the changed backend and its predecessor, which is a small fraction of the total keyspace.
Virtual nodes improve balance. Instead of assigning each backend a single point on the ring, you assign many points, called virtual nodes or vnodes. This spreads each backend across the ring more evenly, so load is distributed better even when the number of backends is small or the key distribution is skewed. The hash function itself matters: it should be fast, deterministic, and produce a uniform distribution. Common choices include cryptographic hashes truncated to a fixed size and non-cryptographic hashes designed for speed. The load balancer must also decide what to hash. Hashing the full URL including query parameters gives fine-grained routing, while hashing only a normalized cache key gives coarser grouping. The choice affects hit rate and how sensitive routing is to request variations.
4 Trade-offs you are really making
Hash-based routing improves cache locality, which can reduce origin load and memory duplication, but it trades away some of the simplicity and resilience of round-robin. With round-robin, any healthy backend can serve any request, so failover is trivial: remove the unhealthy node and the rotation continues. With hash-based routing, removing a node remaps its keys to other nodes, which can cause a temporary spike in misses and load on the remaining backends. If the hash ring is not designed with enough virtual nodes, the redistribution may be uneven, overloading some backends while others sit idle. You also lose the ability to freely add capacity without affecting routing; every membership change requires careful planning and often a warm-up period.
Hot keys are another challenge. If a single key receives a disproportionate share of traffic, hash-based routing sends all of that traffic to one backend. Round-robin would spread those requests across all backends, which can be better for throughput even if it duplicates cache entries. To handle hot keys, you may need to replicate the hot key across multiple backends and route reads to any replica, or introduce a small local cache in front of the hash-based layer. These mitigations add complexity. Finally, hash-based routing can make debugging harder because the path a request takes depends on its key, not just on a simple rotation. You need better observability to understand why a particular request landed on a particular backend and whether that backend had the data cached.
5 Implementing it correctly
Start by choosing a stable key that represents the unit of caching. For HTTP reverse proxies, that is often the normalized URL without volatile query parameters, or a cache key derived from the request. For application-level caches, it is the cache key itself. Avoid hashing client-specific data unless you intend to pin clients to backends, because that can create uneven load. Next, select a hash function that is fast and uniform. You do not need cryptographic strength for routing, but you do need low collision rates and deterministic output across load balancer instances. If you run multiple load balancer nodes, they must all compute the same hash for the same key, so the algorithm and any salt must be identical.
Implement consistent hashing with virtual nodes rather than simple modulo hashing. A common approach is to assign each backend a few hundred virtual nodes on the ring, which gives good balance without excessive memory overhead. When a backend is added or removed, the ring updates and only the affected keys remap. To avoid a cold-start miss storm, you can gradually shift traffic to the new ring or pre-warm the new backend by replaying recent keys. Health checking must be integrated with ring membership: when a backend fails, remove it from the ring so its keys route elsewhere, and when it recovers, add it back carefully. Consider using a bounded load metric, such as least-connections, as a tiebreaker among backends that are close on the ring, to avoid overloading a single node during rebalancing.
6 Measuring whether it helped
The primary metric to watch is cache hit rate per backend and overall. With round-robin, hit rate may be lower because each backend caches independently. With hash-based routing, you expect the same key to hit the same backend more often, so hit rate should improve if the key distribution is uniform. Measure hit rate before and after the change, and segment it by key popularity to see whether hot keys or long-tail keys behave differently. Also track origin request rate; if hash-based routing is working, origin traffic should drop because fewer duplicate fetches occur. Be careful to measure over a representative period, because cache warm-up and diurnal patterns can skew short samples.
Latency is another important signal. Hash-based routing can reduce average latency by increasing cache hits, but it can also increase tail latency if a hot key pins traffic to one backend that becomes saturated. Monitor per-backend request rates and latency percentiles to detect imbalance. If one backend consistently receives more traffic, your hash function or virtual node count may need adjustment. Invalidation efficiency is also worth measuring: count how many backends receive a purge for a given key. With hash-based routing, that number should be small and predictable. Finally, track rebalancing events. When you add or remove a backend, measure the spike in misses and origin load. If the spike is too large, increase the number of virtual nodes or implement a gradual traffic shift. These measurements will tell you whether the added complexity is paying off.
7 Frequently asked questions
Does hash-based routing replace the need for a cache? No. It improves locality so that each cache node sees a more consistent subset of keys, but you still need a cache layer with eviction policies and TTLs. Hash-based routing is a routing strategy, not a caching strategy. It works best when combined with a cache that can store the keys it receives. Without a cache, hashing only changes which backend handles a request, which may not matter for stateless work.
What happens when a backend fails? The load balancer should remove it from the hash ring so its keys route to the next backend clockwise. This causes a temporary increase in misses for those keys, because the new backend may not have them cached. To reduce impact, you can replicate critical keys across multiple backends or use a fallback layer. The key is to ensure health checks are fast and ring updates are atomic across all load balancer instances.
Can I use hash-based routing with sticky sessions? Yes, but be careful. Hashing a session identifier pins a user to a backend, which can create uneven load if some users are much more active than others. For session state, it is often better to externalize the session store and use hash-based routing only for cacheable content. If you must pin sessions, monitor per-backend load and consider rebalancing inactive sessions.
How many virtual nodes should I use? There is no single answer. More virtual nodes improve balance but increase memory and lookup cost. A few hundred per backend is a common starting point. Measure the distribution of keys across backends and adjust until it is acceptably uniform for your workload.