1 Overview
Load balancing algorithms decide which backend server receives each incoming request. They are the traffic cops of your infrastructure, and when they fail, the outage is immediate and visible. This guide focuses on the algorithms themselves, not on specific products or vendors. The goal is to help you understand the mechanisms well enough to choose, implement, and debug them during your first production incident.
The most common algorithms include round robin, least connections, least response time, and consistent hashing. Each makes different assumptions about your workload and your servers. Round robin assumes all servers are equal and requests are similar. Least connections assumes that the number of active connections is a good proxy for load. Least response time assumes that latency is the best signal. Consistent hashing assumes that request affinity matters, often for caching.
No algorithm is universally best. The right choice depends on your traffic patterns, your tolerance for complexity, and how your backends behave under load. This article walks through where common advice breaks down, how the mechanisms actually work, the trade-offs you are really making, how to implement correctly, how to measure success, and answers to frequent questions.
2 Where the common advice breaks down
Common advice often says to start with round robin and move to least connections when you need more fairness. That advice breaks down when requests have wildly different costs. A round robin balancer will happily send a expensive request to a server that is already struggling, because it does not look at server state. Least connections improves this by tracking active connections, but it can be fooled by long-lived connections that are idle. A server with many idle keep-alive connections may look busy while another server with fewer but active connections is actually more loaded.
Another breakdown occurs with consistent hashing. It is often recommended for cache affinity, but it can create hot spots if your key distribution is skewed. If one key is extremely popular, the server that owns that key becomes a bottleneck. Adding or removing servers also remaps keys, which can cause cache misses and a thundering herd on your backends. The advice rarely mentions that you need a good hash function and a way to handle hot keys.
Finally, advice often ignores health checking. An algorithm that routes to a dead server is worse than no algorithm. But health checks add their own trade-offs: too aggressive and you remove servers that are merely slow; too lax and you keep sending traffic to a black hole. The common advice rarely covers how to tune health checks in concert with the algorithm.
3 How the mechanism actually works
At its core, a load balancer maintains a list of healthy backends and a policy for selecting one. Round robin iterates through the list in order. Weighted round robin assigns each server a weight and cycles through them proportionally. Least connections tracks the number of active connections per server and picks the one with the fewest. Least response time tracks latency, often using a moving average, and picks the fastest. Consistent hashing maps a request key to a point on a hash ring, and each server owns a range of the ring.
The selection happens per connection or per request, depending on the layer. A layer 4 load balancer typically selects a backend when a TCP connection is established, so all requests on that connection go to the same server. A layer 7 load balancer can select per request, which allows more granular balancing but adds overhead. The algorithm also interacts with connection pooling: if the load balancer reuses backend connections, least connections may not reflect actual request load.
Health checks are part of the mechanism. Active health checks send probes to backends and remove them from rotation if they fail. Passive health checks observe real traffic and eject servers that return errors or time out. The algorithm only sees the healthy set, so the quality of health checking directly affects the algorithm's effectiveness. Some balancers also support slow start, where a newly added server receives gradually increasing traffic to avoid overwhelming it.
4 Trade-offs you are really making
Every algorithm trades simplicity for adaptivity. Round robin is simple and predictable, but it ignores server load and request cost. Least connections is more adaptive but requires tracking state and can be fooled by idle connections. Least response time is even more adaptive but can oscillate if latency measurements are noisy. Consistent hashing provides affinity but can create hot spots and complicates scaling.
You are also trading latency for throughput. A simple algorithm may add less overhead per request, reducing latency, but it may not utilize servers evenly, reducing throughput. A complex algorithm may improve utilization but add processing time and potential for bugs. The load balancer itself can become a bottleneck if the algorithm requires heavy computation or frequent state updates.
Another trade-off is between global and local decisions. A load balancer that makes decisions based only on its own view may be fast but suboptimal if multiple balancers are involved. A distributed algorithm that shares state can make better decisions but adds coordination overhead and failure modes. In practice, many systems use a combination: a simple algorithm at the edge and more sophisticated routing within a service mesh.
Finally, you trade operational complexity for performance. Adaptive algorithms require tuning, monitoring, and a deeper understanding of your workload. They can improve performance during normal operation but may behave unpredictably during failures. Simple algorithms are easier to debug but may leave performance on the table.
5 Implementing it correctly
Start by understanding your workload. Are requests uniform in cost? Are they long-lived or short? Do you need session affinity? The answers will narrow your choices. For uniform, short requests, round robin or weighted round robin is often sufficient. For variable costs, least connections or least response time may be better. For caching, consistent hashing can help, but only if you can handle hot keys.
Implement health checks that match your application. A simple TCP check may be too shallow; an HTTP check that exercises a critical path is better but may be more expensive. Set timeouts and thresholds based on your recovery objectives. Use slow start to avoid overwhelming new servers. If you use consistent hashing, choose a hash function with good distribution and consider adding virtual nodes to smooth the ring.
Test your configuration under load. Simulate server failures and see how the algorithm reacts. Does it remove the failed server quickly? Does it redistribute load without overloading the remaining servers? Monitor the load balancer's own resource usage. A load balancer that is CPU-bound will add latency regardless of the algorithm.
Document your choices and their rationale. During an outage, you will need to know why you chose least connections over round robin. Keep the configuration simple enough that you can reason about it under pressure. Avoid changing multiple variables at once; if you must tune, change one thing and measure.
6 Measuring whether it helped
Measure the right things. For load balancing, the key metrics are per-server request rate, active connections, response time, and error rate. You want to see that load is evenly distributed according to your algorithm's intent. If you use least connections, check that no server has significantly more connections than others. If you use least response time, check that latency is consistent across servers.
Look at tail latency, not just averages. A load balancer that makes average latency look good but causes occasional spikes is not helping. Measure the 95th and 99th percentiles. Also measure the load balancer's own latency: time spent in the balancer should be small relative to backend time.
Track cache hit rates if you use consistent hashing. A drop in hit rate after a scaling event indicates that your hashing is causing too many remaps. Track the frequency of health check failures and the time it takes to eject a failed server. If ejection is slow, your users will see errors.
Finally, measure the cost. More sophisticated algorithms may require more powerful load balancers or more monitoring. Compare the cost of over-provisioning with a simple algorithm against the cost of complexity with an adaptive one. The goal is not to achieve perfect balance but to meet your service level objectives at a reasonable cost.
7 Frequently asked questions
What is the best load balancing algorithm? There is no single best. It depends on your workload, your servers, and your operational maturity. Start simple and only add complexity when you have evidence that it helps.
Should I use layer 4 or layer 7 load balancing? Layer 4 is faster and simpler but less flexible. Layer 7 allows per-request routing and content-based decisions but adds overhead. Use layer 4 for raw throughput and layer 7 when you need application awareness.
How do I handle session affinity? Consistent hashing or sticky sessions can provide affinity, but they can create hot spots and complicate failover. Consider externalizing session state instead.
What about global server load balancing? That adds another layer of complexity, often using DNS. Be careful with DNS TTLs and failover times. Test thoroughly.
How many health checks should I run? Enough to detect failures quickly without adding significant load. A common approach is a lightweight check every few seconds with a few retries before marking a server down.
Can I change algorithms without downtime? Usually yes, but test in a staging environment first. Some algorithms require state that may not be present if you switch mid-flight.
What should I monitor during an outage? Per-server load, error rates, and the load balancer's own health. Have a runbook that tells you how to drain a server or switch algorithms quickly.