Load balancing: definition, algorithms and failover

01Load balancing: definition and purpose

Load balancing distributes connections or requests across eligible servers. A load balancer selects a backend for traffic addressed to one logical service. The client uses that service address; routing determines whether application instance A, B, or C receives the work.

Horizontal application scaling introduces multiple instances behind the same service address. A load balancer distributes work among them using a routing policy. Essential session state must be available to every eligible instance: rerouting a request cannot recover a cart stored only in a failed process.

02Layer 4 versus Layer 7 load balancing

“Layer” refers to the kind of network information the intermediary understands. Layer 4 (L4) is the transport layer: TCP/UDP connections or flows identified by network addresses and port numbers. A port identifies a service endpoint on a machine. L4 balancing commonly chooses a backend using this connection information. It can forward a connection without interpreting each application request. Layer 7 (L7) is the application layer. L7 balancing understands an application protocol such as HTTP, the request/response protocol used by web applications. It can route /images to an image service and /checkout to a checkout service, or use a hostname or header.

TLS termination means the encrypted client connection ends at the balancer, which can inspect the decrypted HTTP message. The balancer may then create a separately encrypted connection to the backend. If TLS passes through untouched, a transport balancer does not get the same HTTP routing information. Say where encryption ends and which network segments remain protected.

For GET /images/P7.jpg, an L7 rule can first choose the image pool; a balancing algorithm then chooses A or B inside that pool. Choosing the right service and distributing work among its instances are two separate decisions.

03Worked example: round robin, weights and least connections

After choosing the service pool, the balancer still needs a rule for selecting an instance. The main choices use a fixed schedule, a configured share of capacity, a measurement of current work, or a stable caller identity. Compare them by asking which signal best represents the work in this pool.

Assume A, B, and C are healthy and serve equal-cost short requests. Round robin visits them in order. Requests R1 through R6 go to A, B, C, A, B, C. Step one: R1 arrives and A is next. Step two: R2 advances the cursor to B. Step three: R3 advances to C. Step four: R4 wraps to A; R5 and R6 repeat B and C. Each server receives two requests, but equal counts imply equal work only under our equal-cost assumption. It is easy to operate, but it ignores work already running.

Now A has twice the tested capacity of B and C. A weighted schedule such as A, B, A, C repeats, giving A about half the requests and B/C a quarter each. Weights express capacity assumptions; they do not detect a new slow dependency.

For long-lived connections, suppose A has 10 active connections, B has 2, and C has 5. Least connections sends the next comparable connection to B. But if B's two connections each carry many expensive streams, the count may misrepresent actual load.

Implementations differ, so explain the signal instead of promising an exact universal algorithm.

Power of two choices reduces the need to compare every backend: randomly sample two eligible servers and choose the one with less measured work. With A=10, B=2 and C=5 comparable active connections, sampling A/C chooses C; sampling A/B chooses B. It need not find the global minimum to reduce imbalance. A least-request implementation counts active requests instead of transport connections, which can better match multiplexed HTTP work; neither count captures arbitrary CPU cost.

04Session affinity and persistent state

Affinity, or a sticky session, tries to send a caller back to the same backend. It can improve reuse of a local cache. IP hashing is one way; a routing cookie is another. If many students share one school's public IP through network address translation (NAT), IP hashing can concentrate them on one server.

Hashing a stable key can also place cached objects consistently. That is useful when the same key should reach the same owner, but the design must explain how keys move when servers join or leave and how it handles a key that receives unusually heavy traffic. A balancer cannot divide one expensive request simply by hashing it.

05Health checks and backend failover

At 12:00:00, B's process stops. A health check is a small probe used to decide whether B should receive new work. A readiness check asks whether it can serve new requests; a liveness check asks whether restarting the process might be necessary.

1. Requests already sent to B may fail before a health probe detects the crash. A health system does not make detection instantaneous.
2. After the configured failure threshold, the balancer removes B from new selection. A and C inherit its traffic.
3. Safe retries use a deadline and an operation identity where needed. Retrying a purchase blindly may duplicate it if B committed just before losing the response.
4. When B restarts, readiness remains false until required initialization completes. Reintroduce it gradually so a cold cache does not create a surge of database work.

A shallow probe can say “healthy” while every database query fails. An overly broad probe can remove all servers when one shared optional dependency fails. Design probes around the work each pool must actually serve.

Distinguish the source of health information. Active checks send dedicated probes even when no user traffic arrives. Passive checks infer trouble from real request failures, so an idle backend can remain untested. Thresholds reduce transient ejections but increase detection time. Neither proves future success, and an application error caused by the caller is not automatically evidence that the server is unhealthy.

06Connection draining, overload and balancer redundancy

Draining stops assigning new work while allowing existing requests to finish within a deadline. For a deployment, mark C unready, let short requests finish, then stop it. Long-lived sockets need a reconnect protocol or explicit migration; draining does not preserve in-memory conversation state by itself.

The balancer also needs a replacement if it fails. Active/passive keeps a standby and a way to redirect traffic; active/active runs several balancers. Cached DNS answers can delay redirection. Even a managed balancer needs enough surviving capacity for the failures you plan to tolerate. Existing TCP or TLS connections may break when their balancer fails: another balancer does not automatically inherit them. Clients therefore need reconnect and retry limits.

Interview answer: “For similar short HTTP calls I begin with weighted round robin over ready instances. I move essential session state out of individual servers. I calculate surviving capacity, drain during changes, and make retries safe. For long-lived or uneven work, I change the routing signal after measuring which resource is saturated.”

These policies become routing configuration in a proxy. In NGINX, an upstream group names the available backends, while proxy_pass forwards matching requests to that group. Weights and selection rules then control how the group distributes work.

A concrete starting implementation is an NGINX HTTP proxy with an upstream group, explicit backend weights and proxy_pass; choose least_conn when comparable active connections are a better signal. Its upstream module documents passive failure handling through max_fails and fail_timeout. Dedicated active HTTP checks require the documented health-check module/product support, so verify the installed edition rather than assuming all capabilities follow from the NGINX name. This configuration routes requests. The application and storage design must separately preserve carts, determine which database node may accept writes, and handle retries without losing or duplicating an operation. See the upstream reference and health-check guide.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.