1 Overview
Round robin is the default load balancing algorithm in most tutorials, and for good reason. It is trivial to reason about, needs almost no state, and distributes request counts evenly across a pool. If every request cost roughly the same and every backend had identical capacity, it would be close to optimal. The trouble is that production APIs violate both assumptions constantly.
A single endpoint might fan out to a database, call a downstream service, or trigger a cache miss that takes an order of magnitude longer than a cached read. When those requests land on backends in strict rotation, the slow ones pile up behind each other while other backends sit idle. The result is not a capacity problem in aggregate; it is a placement problem. You have enough total capacity, but the balancer keeps handing work to a backend that is already busy.
This article explains why round robin fails under uneven request cost and backend health, how the alternatives actually work at the connection and request level, and how to measure whether a change helped. The goal is not to declare one algorithm universally correct, but to give you the mental model to pick deliberately.
2 Where the common advice breaks down
The standard advice says round robin is fine because it spreads load evenly. That claim conflates request count with work. Two requests can differ by two orders of magnitude in CPU, memory, or downstream wait time. A backend that receives ten cheap requests is far less loaded than one that receives ten requests each waiting on a slow query. Round robin counts requests, not cost.
The second breakdown is health and readiness. Many simple balancers treat a backend as either up or down based on a probe. A backend can pass its health check while its thread pool is saturated, its connection pool is exhausted, or its garbage collector is thrashing. Round robin keeps sending it traffic because the probe only asks whether the process answers, not whether it can do useful work. This is how a single degraded instance drags down a whole pool.
The third breakdown is connection reuse. With keep-alive, a client may hold a long-lived connection to one backend. Round robin at the request layer can still distribute new requests, but if the balancer is doing connection-level balancing, a heavy client can pin to one backend for the life of the connection. The even distribution you expected never materializes.
Finally, round robin ignores locality. If backends keep warm caches, sticky or locality-aware routing can serve more requests from memory. Strict rotation scatters related requests and forces repeated cache misses, which raises latency without any backend being overloaded.
3 How the mechanism actually works
At its core, round robin maintains a counter and advances it for each new request or connection. The counter modulo the number of healthy backends gives the target. Weighted round robin extends this by giving each backend a weight and cycling through a schedule proportional to those weights. Smooth weighted round robin interleaves selections so a high-weight backend is not hit in a burst. All of these are still open-loop: they decide placement without observing current load.
Least connections is the first closed-loop alternative. The balancer tracks the number of active connections or in-flight requests per backend and sends the next request to the one with the fewest. This adapts to slow requests automatically, because a backend stuck on a long call accumulates connections and stops receiving new ones. Least response time goes further, factoring in observed latency, though it needs enough samples to avoid chasing noise.
Consistent hashing takes a different approach. It maps a key, such as a user ID or cache key, onto a ring of backends so the same key lands on the same backend. This gives locality and stable routing when backends are added or removed, at the cost of uneven load if the key distribution is skewed. Power-of-two-choices picks two backends at random and sends to the less loaded one, which approximates least connections with far less state and scales well across many balancer instances.
Each mechanism trades state and coordination for adaptivity. The right choice depends on whether your bottleneck is request cost variance, connection pinning, or cache locality.
4 Trade-offs you are really making
Choosing an algorithm is choosing what you are willing to be ignorant about. Round robin is ignorant of load, so it is cheap and predictable but fragile under variance. Least connections is ignorant of request cost until connections accumulate, so it reacts after the fact. Least response time is ignorant of the future and can oscillate if latency samples are noisy. Consistent hashing is ignorant of balance and can create hot spots when keys are skewed.
State and coordination are the hidden costs. Least connections requires every balancer to know the active count per backend. In a single balancer that is trivial. Across many balancer instances, each one only sees its own connections unless you share state, so the global picture is approximate. Power-of-two-choices sidesteps this by making a local decision from partial information, which is why it scales well but is not perfectly optimal.
There is also a tail-latency trade-off. Algorithms that chase the currently fastest backend can herd traffic toward it, briefly overloading it and creating new latency spikes. Algorithms that spread evenly avoid herding but tolerate stragglers. Neither is universally better; it depends on whether your latency distribution is dominated by queueing or by a few slow outliers.
Finally, consider failure behavior. Round robin degrades gracefully when a backend dies because the counter simply skips it. Adaptive algorithms can overreact to a transient slowdown and drain a healthy backend. Whatever you choose, pair it with sensible health checks and circuit breaking so the algorithm is not making decisions from bad signals.
5 Implementing it correctly
Start by classifying your endpoints. Static or cheap endpoints with uniform cost are fine on round robin. Expensive or variable-cost endpoints benefit from least connections or power-of-two-choices. If you have cache locality requirements, use consistent hashing for the keys that matter and round robin for the rest. Mixed strategies per route are normal and often better than one global choice.
Tune health checks to reflect readiness, not just liveness. A probe that only checks whether the process accepts a socket will not catch saturation. Consider a lightweight endpoint that verifies the backend can acquire a database connection or complete a trivial downstream call. Set the interval and failure threshold so a genuinely dead backend is removed quickly without flapping on transient blips.
Handle connection reuse deliberately. If clients hold long-lived connections, enable per-request balancing rather than per-connection balancing, or cap connection lifetime so clients periodically rebalance. For consistent hashing, use a ring with virtual nodes to smooth distribution, and plan for rebalancing when the pool changes.
Add circuit breaking and load shedding at the backend or sidecar so a struggling instance stops accepting work before it collapses. The balancer should not be the only defense. Document the chosen algorithm per route, the health-check semantics, and the expected behavior during deploys, because draining and rebalancing are where most incidents happen.
6 Measuring whether it helped
Measure at the request level, not just the pool level. Track per-backend request counts, in-flight requests, and latency percentiles. If round robin was the problem, you should see high variance in in-flight counts across backends and a long tail in latency. After switching, the distribution should tighten and the tail should shrink, though not necessarily to zero.
Watch for herding. If you adopt a latency-aware algorithm, plot per-backend load over time. A healthy adaptive balancer keeps backends roughly comparable without rapid oscillation. If one backend repeatedly spikes and drains, your algorithm is overreacting or your latency samples are too noisy. Increase smoothing or fall back to a less aggressive method.
Track error rates and timeouts per backend, not just globally. A pool-wide error rate can hide one bad instance. Per-backend error attribution tells you whether the algorithm is routing around failures or into them. Also measure connection churn and rebalancing events, since consistent hashing changes can cause cache misses and latency spikes during pool changes.
Finally, compare against a control. Run the new algorithm on a subset of traffic or behind a feature flag, and compare tail latency and capacity headroom. The question is not whether the average improved, but whether you can serve the same load with fewer backends or lower latency at the same cost. That is the only result that justifies the added complexity.
7 Frequently asked questions
Is round robin ever the right choice? Yes. When request costs are uniform, backends are homogeneous, and connections are short-lived, round robin is simple, predictable, and cheap. It is also a reasonable default for stateless endpoints that do not touch shared resources. The mistake is using it everywhere without checking whether those conditions hold.
Does least connections require shared state across balancers? Not strictly. Each balancer can track its own view, which approximates global load. If your balancers are behind a single entry point or you can share counters cheaply, the approximation improves. Power-of-two-choices is a good middle ground when sharing state is expensive.
How does consistent hashing interact with autoscaling? Adding or removing backends changes the ring and remaps some keys. Virtual nodes reduce the fraction of keys that move, but some cache misses are unavoidable. Plan for a warm-up period and consider draining before removal so in-flight work completes.
What about weighted algorithms? Weights are useful when backends have different capacities, but they are static. If capacity changes with load, weights become stale. Combine weights with a load-aware algorithm only if you can keep the weights current, otherwise you are adding configuration that drifts from reality.
Can I just add more backends instead? Sometimes. Over-provisioning masks placement problems but raises cost and does not fix tail latency caused by a single slow request. Fixing placement is usually cheaper than scaling out, and the two are complementary rather than substitutes.