What is a load balancer?
A load balancer is a reverse proxy that sits in front of a group of servers and distributes incoming requests across them. Clients connect to the load balancer. The load balancer forwards each request to a backend server, waits for the response, and returns it to the client. The client never connects directly to a backend server.
A load balancer is like a maitre d' at a restaurant. You do not seat yourself. The maitre d' knows which tables are occupied, which waiter has capacity, and routes you to the best available option. Servers are the tables. Requests are diners.
Load balancers serve three core functions. First: distribution, spreading traffic so no single server is overwhelmed. Second: redundancy, detecting when a server is unhealthy and routing around it. Third: abstraction, hiding the complexity of the backend fleet behind a single IP address.
Layer 4 vs Layer 7
Load balancers operate at different layers of the networking stack. The layer they work at determines what information they can see and what decisions they can make.
Sees only TCP/UDP metadata: source IP, destination IP, source port, destination port. Cannot read the HTTP request. Routes based on network-layer information alone.
Examples: AWS NLB, HAProxy (TCP mode), hardware LBs
Fully decodes the HTTP request. Sees method, URL path, headers, cookies, and body. Can make intelligent routing decisions based on request content.
Examples: AWS ALB, Nginx, Caddy, Traefik, Cloudflare LB
When do you need one?
| Situation | Why a load balancer helps |
|---|---|
| Single server at capacity | Add more servers and distribute traffic across them. Capacity scales linearly. |
| Single point of failure | Health checks detect failures; traffic reroutes to healthy servers in seconds, not hours. |
| Zero-downtime deployments | Rolling deploys: drain one server, update it, return to rotation. Users never see downtime. |
| Multiple services (API + frontend) | Route by path: /api to API fleet, everything else to frontend fleet. One public IP. |
| Geographic distribution | Multiple LBs in regions, GeoDNS routes users to nearest. Latency drops for global users. |
Distribution Algorithms
The algorithm determines which backend server receives each incoming request. The right choice depends on whether servers are homogeneous, whether requests have uniform cost, and whether users need to be pinned to a specific server.
Use the explorer below to see each algorithm in action. Switch algorithms, kill a server, and watch how traffic redistributes. The distribution chart at the bottom shows where requests are actually landing.
Requests cycle through servers in order. Simple, effective for uniform workloads.
server = servers[request_count % len(servers)]
Strengths
- +Zero configuration. Works out of the box for homogeneous fleets.
- +Predictable distribution: N servers each get exactly 1/N of traffic over time.
Weaknesses
- -Ignores server load. A server handling a 5-second request gets the same traffic as one doing 1ms work.
- -Breaks if servers have different hardware specs.
Best for: Identical servers, uniform request cost (most REST APIs, static file serving).
server = min(servers, key=lambda s: s.active_connections)
Strengths
- +Self-balancing: faster servers naturally drain their queue and accept more requests.
- +Handles variable request duration correctly without any configuration.
Weaknesses
- -Requires the LB to track active connection counts for every backend.
- -Overhead of counting connections can outweigh benefit for very short-lived requests.
Best for: Variable request duration (database queries, external API calls, streaming).
server = servers[hash(client_ip) % len(servers)]
Strengths
- +Session affinity without application changes or cookie overhead.
- +Deterministic: same client always hits same server, useful for in-memory caches.
Weaknesses
- -Uneven distribution if traffic comes from a small number of IPs (e.g., corporate NAT).
- -Adding or removing servers remaps all clients. Consistent hashing solves this.
Best for: Stateful applications needing affinity, when adding sticky session cookies is not possible.
# Server A: weight 3, Server B: weight 1 => A gets 75%, B gets 25%
Strengths
- +Handles heterogeneous fleets with different CPU/RAM correctly.
- +Simple extension of round robin with a single weight parameter per server.
Weaknesses
- -Weights must be configured manually and updated when hardware changes.
- -Dynamic load changes are not reflected in static weights.
Best for: Mixed hardware fleets, gradual traffic migration (canary deploys use this: 5% weight to new version).
Health Checks and Failure Handling
A load balancer is only as reliable as its knowledge of which backends are working. Health checks are the mechanism by which the LB continuously verifies that each server is alive and capable of handling requests.
Use the simulator below: run health checks to see the LB probing each server, then kill a server to watch the LB detect failure and remove it from rotation.
Run a health check to see the log...
How health checks work
How often the LB sends a probe. Typically 5-30 seconds. Lower intervals detect failures faster but add probe traffic.
How long to wait for a response before declaring a failure. Typically 2-5 seconds. Must be less than interval.
How many consecutive failures before removing from rotation (healthy to sick), and successes to re-add (sick to healthy).
Active vs passive health checks
Active (probing)
The LB periodically sends a request to a designated health endpoint (typically /health or /healthz). A 200 OK means healthy. Anything else, or no response within timeout, means sick.
GET /health HTTP/1.1
Host: server-a.internal
HTTP/1.1 200 OK
{"status":"ok","db":"connected"}Passive (observing)
The LB watches real traffic responses. If a server returns 5xx errors or connections time out on X consecutive requests, it is marked unhealthy. No extra probe traffic, but slower to detect failures under low traffic.
# Nginx passive health check config proxy_next_upstream error timeout; proxy_next_upstream_tries 3; proxy_next_upstream_timeout 30s;
Design your /health endpoint to check real dependencies: can it reach the database? the cache? If the server is up but its DB connection pool is exhausted, it should return 503, not 200. The LB should route around broken servers, not just dead ones.
SSL Termination, Sticky Sessions, and Global Load Balancing
SSL/TLS termination
Decrypting HTTPS is CPU-intensive. Rather than making every application server handle TLS, the load balancer can terminate TLS at the edge and forward plain HTTP internally. This is called SSL termination.
Certificate management in one place
Renew TLS certificates on the LB only. No need to update certs on every backend server.
Faster backends
App servers do zero TLS work. CPU is fully available for application logic.
Traffic inspection at L7
Decrypted traffic lets the LB read headers, inject request IDs, and do content-based routing.
Re-encryption for compliance
Some environments require encrypted traffic end-to-end. LB terminates and re-encrypts before forwarding.
Sticky sessions (session affinity)
By default, each request may land on any server. For stateful applications that store session data in memory, this causes users to lose their session when routed to a different server. Sticky sessions solve this by binding a user to a specific backend.
Cookie-based
LB sets a cookie (e.g., AWSALB=...) on the first response. Subsequent requests from that client include the cookie; LB reads it and routes to the same server.
The safest approach. Works with L7 LBs. Cookie has a configurable TTL.
IP-hash based
Hash the client IP to consistently map to a server. No cookie needed. Works at L4.
Breaks under NAT (many users share one IP). Distribution can be uneven.
Avoid sticky sessions if you can. They make deployments harder (you must drain stickied servers before removing them), break auto-scaling (new servers receive no traffic from existing sessions), and are a crutch for a stateful architecture. Move session state to Redis instead, and become truly stateless.
Global load balancing
Regional load balancers work within a single data center. For global services, you need to route users to the nearest data center as well. Two techniques handle this.
DNS returns different IP addresses based on the geographic location of the DNS query. EU users get a Frankfurt IP, US users get a Virginia IP. Each resolves to a regional load balancer.
Latency reduction: 50-200ms for global users
Propagation delay: Bound by DNS TTL
Used by: Cloudflare, AWS Route 53, Google Cloud DNS
A single IP address is announced from multiple physical locations via BGP routing. The internet automatically routes traffic to the nearest announcement point. No DNS tricks required.
Failover speed: Sub-second (BGP convergence)
DDoS resistance: Attack traffic absorbed across PoPs
Used by: Cloudflare, Fastly, all major CDNs
CDNs combine Anycast with load balancing: each PoP is a cluster of servers behind a load balancer, and the PoP itself is reached via Anycast. Users get sub-10ms latency to the edge, with the LB distributing load across servers within that PoP.