1 Overview
Round-robin is the default most teams reach for, and for good reason. It is trivial to reason about, it spreads connection counts evenly, and it has no state to corrupt. When every backend is identical, every request costs roughly the same, and traffic is steady, round-robin is close to optimal. The trouble is that production rarely looks like that. Backends differ in CPU, memory, and noisy-neighbor exposure; requests differ in cost by orders of magnitude; and traffic arrives in bursts that do not respect your assumptions.
This guide is a field map of load balancing algorithms rather than a recommendation of one. The goal is to help you choose deliberately, understand what each algorithm optimizes, and know when the common advice stops being true. We will cover the mechanism behind the main families, the trade-offs they force, how to implement them without subtle bugs, and how to measure whether the change actually helped. The framing throughout is that a load balancer is a control system, and every control system has a plant it is trying to regulate, a sensor it trusts, and a failure mode when the sensor lies.
Nothing here depends on a specific product. The same ideas apply whether you are configuring a reverse proxy, a service mesh sidecar, or a custom dispatcher. What matters is the feedback signal you use, how fast you react, and how much you are willing to pay in complexity for better tail behavior.
2 Where the common advice breaks down
The most repeated advice is to start with round-robin and only move on when you have a problem. That is reasonable as a default, but it hides the fact that round-robin can actively create the problem it is supposed to avoid. If one backend is slower, it accumulates in-flight requests, its queue grows, and clients time out and retry. Retries add load, which makes the slow backend slower. The algorithm keeps sending it an equal share because it counts requests, not work.
A second breakdown is the assumption that least-connections fixes this. Least-connections helps when request cost is uniform and connections map cleanly to work, but with HTTP keep-alive a single connection can carry many concurrent streams, and a backend with few connections may still be saturated. It also ignores request cost entirely: a backend handling one expensive query looks idle to a connection counter. Similarly, latency-based routing sounds adaptive, but it can oscillate if the measurement window is short relative to request duration, and it can herd traffic onto whichever backend reported a fast sample by luck.
A third issue is treating health checks as a complete picture of capacity. A backend can pass a shallow liveness check while its thread pool is exhausted. Algorithms that trust binary health and ignore saturation will keep feeding a dying node until it fails outright. The common advice assumes homogeneous, stateless, cheap-to-retry backends; when any of those assumptions fail, the default quietly becomes a tail-latency amplifier.
3 How the mechanism actually works
Every algorithm is a function from observed state to a routing decision. Round-robin keeps a counter and increments it, which is stateless per request and perfectly fair in request count. Weighted round-robin adds static weights, letting you express known capacity differences, but the weights do not adapt. Least-connections tracks active request or connection counts per backend and picks the minimum, which approximates least-queued-work when request cost is uniform. Power-of-two-choices samples two backends at random and picks the less loaded, which avoids the coordination cost of a global minimum while still relieving hot spots.
Latency-aware and load-aware algorithms close a feedback loop. They measure a signal such as response time, queue depth, or CPU, smooth it over a window, and bias selection toward backends with better recent performance. The smoothing is the critical part: exponential moving averages or sliding windows prevent a single slow sample from redirecting all traffic. Consistent hashing is a different family entirely; it maps a request key to a backend so that the same key tends to land on the same node, which matters for cache affinity and session stickiness. It trades perfect balance for locality, and it needs a ring with virtual nodes to avoid hotspots when membership changes.
The unifying idea is that you are choosing which signal to trust and how aggressively to act on it. Request-count fairness is cheap and stable. Queue-based signals are more accurate but noisier. Key-based routing buys locality at the cost of balance. No algorithm reads the future; they all react to a lagging indicator, and the lag is what determines whether the system converges or oscillates.
4 Trade-offs you are really making
The first trade-off is simplicity versus adaptivity. Round-robin and static weighting are easy to test, easy to reason about, and hard to misconfigure. Adaptive algorithms require tuning windows, thresholds, and decay rates, and a bad tuning can be worse than no adaptation at all. If your backends are genuinely homogeneous and your traffic is stable, adaptivity buys little and costs operational attention.
The second trade-off is balance versus locality. Spreading requests evenly minimizes the maximum load in theory, but it scatters related work. If your backends hold per-key state, an even spread means cache misses, duplicated computation, and cross-node coordination. Consistent hashing and sticky routing accept imbalance in exchange for locality, which is often the better deal for stateful workloads. The cost is that a hot key can overload one node, and rebalancing on membership change moves keys and invalidates local state.
The third trade-off is tail latency versus throughput. Algorithms that react to queue depth protect the tail by shedding load away from slow nodes, but they can leave capacity idle on fast nodes and reduce overall utilization. Algorithms that pack work evenly maximize throughput but expose the slowest request to the slowest node. You cannot optimize both simultaneously; you choose which percentile you care about. Finally, there is a cost in the balancer itself: tracking per-backend state, sampling latency, and maintaining hash rings all consume CPU and memory, and that overhead grows with fleet size and request rate.
5 Implementing it correctly
Start by making the feedback signal honest. If you route on latency, measure server-side processing time rather than client-observed time, because client time includes network and retries that the balancer cannot fix. If you route on connections, account for multiplexed streams so that one connection does not masquerade as one unit of work. Whatever signal you choose, smooth it over a window longer than the typical request duration, or you will chase noise. A common pattern is an exponentially weighted moving average with a half-life of a few seconds, updated on each completed request.
Second, make health and draining explicit. Liveness checks should be shallow and fast; readiness checks should reflect the ability to accept new work. When removing a node, drain in-flight requests before closing connections, and stop new assignments first. For consistent hashing, use enough virtual nodes per backend to smooth the ring, and decide in advance how key movement on membership change will be handled, because that is where correctness bugs hide.
Third, bound the damage of bad decisions. Cap retries and make them budgeted so a struggling backend cannot trigger a retry storm. Add outlier detection that ejects a backend temporarily after repeated failures, with a probe to bring it back. Keep a deterministic fallback, such as round-robin, for when the adaptive signal is missing or stale. Finally, make the algorithm configurable and observable: log the chosen backend and the signal value for sampled requests so you can reconstruct why traffic went where it did.
6 Measuring whether it helped
The only honest measure is the distribution of outcomes, not the average. Track p50, p95, and p99 latency per backend and in aggregate, plus error rate and retry rate. A change that improves the mean while worsening the tail is usually a regression for user experience. Compare before and after over comparable traffic windows, and beware of seasonality and deploys that coincide with the change. If you cannot hold traffic constant, at least record the mix of request types so you can normalize.
Look at balance directly. Plot the distribution of load across backends, not just the mean. A good algorithm produces a tight distribution for uniform work and a deliberately skewed one for stateful or keyed work. Measure queue depth and in-flight requests per backend, since those reveal saturation before latency does. If you are using consistent hashing, measure key movement on membership changes and the resulting cache miss rate; that is often the hidden cost of the approach.
Finally, measure the balancer itself. Track decision latency, memory for per-backend state, and CPU spent on sampling and smoothing. An adaptive algorithm that consumes a meaningful share of a core at high request rates may not be worth its tail-latency gains. The test is simple: does the change reduce tail latency or error rate at equal or lower cost, and does it stay stable under failure and reconfiguration? If it only looks better on a dashboard average, it has not helped.
7 Frequently asked questions
Is round-robin ever the right answer? Yes, frequently. When backends are homogeneous, requests are cheap and uniform, and traffic is steady, round-robin is stable, predictable, and easy to operate. The mistake is not choosing it; the mistake is keeping it after the workload changes and blaming the backends for a routing problem.
When should I move to least-connections or power-of-two-choices? When request cost varies or backends have different queue depths, and you want a cheap improvement without a full feedback loop. Power-of-two-choices is often the sweet spot because it relieves hot spots with minimal coordination and no global state. Least-connections is a good fit when connections map cleanly to work and you can account for multiplexing.
Do I need latency-aware routing? Only if your backends genuinely differ in performance and that difference changes over time. If they are identical, latency-aware routing mostly adds noise and tuning burden. If they differ, it can protect the tail, but you must smooth the signal and bound retries or you will oscillate.
How does consistent hashing fit in? Use it when locality matters more than perfect balance, such as cache affinity or per-key state. Accept that hot keys can overload a node and that membership changes move keys. Size the ring with virtual nodes, and measure miss rates after changes.
What about health checks? Treat them as a safety net, not a capacity model. Combine shallow liveness with readiness that reflects saturation, and always drain before removing a node.