1 Overview
Load balancing algorithms decide which backend receives each request. That decision shapes latency, connection reuse, failure behavior, and how much capacity you actually need. The common families are static and dynamic. Static methods include round robin, weighted round robin, and hashing. Dynamic methods include least connections, least response time, and power of two choices. Each family makes different assumptions about request cost, backend health, and traffic shape.
The right choice depends on what you are optimizing. If requests are uniform and backends are identical, simple rotation works well. If requests vary in cost, connection-aware or latency-aware selection reduces queueing. If sessions must stick to a backend, hashing or affinity rules matter more than raw distribution. No algorithm fixes bad health checks, slow backends, or skewed capacity.
This guide treats algorithms as mechanisms, not brands. It explains where common advice fails, how selection actually works, the trade-offs you accept, how to implement correctly, and how to measure whether the change helped. The goal is not a single best algorithm. The goal is a defensible choice tied to your traffic, failure modes, and operational constraints.
2 Where the common advice breaks down
The most common advice is to start with round robin and move to least connections when latency rises. That advice is not wrong, but it hides important conditions. Round robin assumes requests are roughly equal in cost and backends are roughly equal in capacity. When either assumption fails, round robin can pile work onto a backend that is already slow, because it ignores current load. Least connections improves on that, but it assumes connection count is a good proxy for load. With long-lived connections, HTTP/2 multiplexing, or streaming, connection count can misrepresent actual work.
Another breakdown is treating health checks as ground truth. A backend can pass a shallow health check while its dependencies are degraded. An algorithm that trusts health status will keep sending traffic to a backend that is technically up but functionally slow. Similarly, hashing for session affinity can create hot spots when the key distribution is skewed or when a backend leaves the pool. The remapping can move many sessions at once, causing a thundering herd.
Finally, common advice often ignores connection reuse. A load balancer that opens a new backend connection per request may distribute evenly but pay handshake and TLS costs repeatedly. An algorithm that favors warm connections can reduce latency, but it may also concentrate load. The breakdown is not in the algorithm itself. It is in assuming the algorithm sees the whole picture.
3 How the mechanism actually works
At a high level, a load balancer maintains a set of backends and a selection function. For each incoming request, the selection function reads some state, chooses a backend, and forwards the request. The state can be as simple as a counter for round robin, or as complex as per-backend in-flight request counts, recent latency samples, and connection pool status. The mechanism is a control loop: observe, choose, forward, and update state based on the result.
Static algorithms do not observe backend load. Round robin increments a counter and picks the next backend. Weighted round robin uses configured weights to bias selection. Hashing computes a hash of a key, such as a client identifier or URL, and maps it to a backend. These are predictable and cheap, but they do not react to runtime conditions. Dynamic algorithms observe state. Least connections picks the backend with the fewest active connections. Least response time picks the backend with the best recent latency, often combined with active request counts. Power of two choices picks two backends at random and chooses the one with fewer active requests, which approximates least connections with less coordination.
Health checks run alongside selection. They mark backends as available or unavailable, and they may drain connections before removal. Session affinity, when used, is usually implemented by hashing or by a cookie that pins a client to a backend. The interaction between selection, health, and affinity is where most real behavior emerges.
4 Trade-offs you are really making
Every algorithm trades simplicity for adaptivity, and adaptivity for predictability. Round robin is simple, cheap, and easy to reason about. It distributes requests evenly when requests are uniform. It fails when requests are not uniform, because it cannot see that one backend is overloaded. Least connections is more adaptive, but it requires shared state or local counters that can drift. It also assumes connection count tracks load, which is not always true. Least response time can reduce tail latency, but it can oscillate if latency samples are noisy or if a fast backend gets overloaded and becomes slow.
Hashing gives you affinity and cache locality. If a backend caches data keyed by the hash, repeated requests can hit warm state. The trade-off is hot spots and costly remapping when the backend set changes. Consistent hashing reduces remapping, but it adds complexity and still cannot guarantee even load. Weighted algorithms let you reflect heterogeneous capacity, but weights are static unless you automate them. Automation introduces feedback loops that can be unstable if not damped.
There is also a connection reuse trade-off. Reusing warm connections reduces handshake and TLS overhead, which lowers latency. But it can concentrate traffic on a subset of backends. Algorithms that spread connections widely may pay more setup cost. The honest position is that no algorithm dominates. You choose which failure mode you can tolerate and which cost you are willing to pay.
5 Implementing it correctly
Start by defining what a request costs and what a backend can handle. If requests are uniform and backends are identical, round robin or weighted round robin is sufficient. If request cost varies, measure in-flight work per backend and use least connections or power of two choices. If sessions or cache locality matter, use hashing with consistent hashing to limit remapping. If latency is the primary concern, use a latency-aware algorithm with smoothing, and cap how quickly selection can shift to avoid oscillation.
Health checks must be meaningful. A shallow check that only verifies the process is listening will not catch dependency degradation. Use checks that exercise the critical path without being expensive. Configure thresholds so a single failure does not remove a backend, and use connection draining so in-flight requests finish before removal. For session affinity, prefer a cookie-based pin with a fallback to rehash, and set a short session TTL so stale pins expire. For connection reuse, size pools per backend and monitor pool exhaustion.
Implement selection as a small, testable component. Log the chosen backend and the reason for the choice. Make weights and thresholds configurable without redeploying. Test failure scenarios: backend slow, backend down, backend returning errors, and backend set changing. Verify that the algorithm degrades gracefully rather than amplifying the failure. Correct implementation is less about the algorithm name and more about the surrounding controls.
6 Measuring whether it helped
Measure before and after with the same traffic shape. The primary signals are latency percentiles, error rates, and backend utilization. Look at p50, p95, and p99 latency, not just averages. A change that improves the average but worsens the tail is usually a regression. Track per-backend request counts and in-flight work to see whether distribution is actually even. Track connection pool usage and connection setup rates to understand reuse. Track retries and timeouts, because an algorithm that increases retries is not helping.
Also measure stability. If selection oscillates, you will see backend load swing and latency variance rise. If health checks flap, you will see backends enter and leave the pool frequently. If session remapping is frequent, you will see cache misses and session resets. These are leading indicators that the algorithm is fighting the system rather than cooperating with it. Compare against a control period and use the same client mix. Avoid drawing conclusions from a single peak or a short window.
Finally, measure cost. Better distribution can reduce overprovisioning, but it can also increase connection churn or coordination overhead. The question is whether the latency and reliability gains justify the operational complexity. If the algorithm is hard to debug or tune, that is a cost too. A simpler algorithm that meets your service level objectives is often the better engineering choice.
7 Frequently asked questions
Is round robin ever the right choice? Yes. When requests are uniform, backends are identical, and you want predictable behavior, round robin is simple and effective. It is also a good baseline for comparison. The mistake is using it when request cost varies widely or when backends have different capacity.
Should I always use least connections? No. Least connections helps when request cost varies and connection count tracks work. With long-lived connections or multiplexing, connection count can mislead. Power of two choices often gives similar benefits with less coordination. Test with your traffic.
How do I handle session affinity without creating hot spots? Use consistent hashing with a short session TTL, and provide a fallback path that can rehash to another backend. Monitor per-backend session counts. If a backend becomes hot, consider moving session state to a shared store so affinity is not required.
What about health checks? They are part of the algorithm. A load balancer that selects perfectly but trusts a bad health check will still send traffic to a failing backend. Use meaningful checks, conservative thresholds, and connection draining.
How do I know if the change helped? Compare latency percentiles, error rates, backend utilization, and retry counts over comparable periods. If the tail improves and stability holds, the change helped. If only the average improves, look deeper.