Chapter 2 — Load Balancing & Reverse Proxies

The why

A load balancer isn't a traffic cop — it's the conductor of an orchestra. It decides who plays when, notices when a violinist leaves the stage, and ensures the audience never hears a missed note.

Core concepts

Layer 4 vs Layer 7: the OSI model in practice

Load balancers operate at two different layers of the networking stack, and the choice between them is one of the most consequential architecture decisions you'll make.

Layer 4 (TCP/UDP — Transport Layer):

An L4 load balancer works with IP addresses and ports. It doesn't inspect the content of packets — it sees source IP, destination IP, source port, destination port. When a TCP connection arrives, it forwards the entire connection to one backend server based on the algorithm (round-robin, least connections, etc.).

# Nginx as L4 load balancer (stream module)
stream {
    upstream database_replicas {
        server db-replica-1.internal:5432 max_fails=3 fail_timeout=30s;
        server db-replica-2.internal:5432 max_fails=3 fail_timeout=30s;
        server db-replica-3.internal:5432 backup;
    }
 
    server {
        listen 5432;
        proxy_pass database_replicas;
        proxy_connect_timeout 5s;
    }
}

L4 pros: Extremely fast (packet-level decisions), protocol-agnostic (works for HTTP, WebSocket, database protocols, gRPC, anything TCP), lower resource usage.

L4 cons: No awareness of HTTP semantics — can't route based on URL path, can't set cookies, can't inspect headers, can't terminate TLS with different certs per domain.

Layer 7 (HTTP/HTTPS — Application Layer):

An L7 load balancer terminates the TCP connection, reads the HTTP request, and makes routing decisions based on the full request content: URL path, headers, cookies, query parameters, even request body (with limits).

# Nginx as L7 load balancer (http module)
upstream app_servers {
    least_conn;
    server app-1.internal:3000 weight=3 max_fails=2 fail_timeout=10s;
    server app-2.internal:3000 weight=2 max_fails=2 fail_timeout=10s;
    server app-3.internal:3000 weight=1 max_fails=2 fail_timeout=10s;
    keepalive 

L7 pros: Content-based routing, TLS termination with SNI, header manipulation, rate limiting, authentication at the edge, caching, compression, request/response modification.

L7 cons: Higher latency (terminates and re-establishes TCP connections), more CPU and memory intensive, protocol-specific (HTTP/HTTPS only, though modern L7 balancers also handle gRPC and WebSocket).

Load balancing algorithms

The algorithm determines which backend server receives each request. The right choice depends entirely on your workload characteristics.

Consistent hashing: the cache-friendly algorithm

Consistent hashing deserves special attention because it solves a critical problem: when you add or remove a server from a distributed cache cluster, you want to minimize the number of keys that need to be redistributed.

Without consistent hashing (simple modulo):

server = hash(key) % N
 
If N changes from 4 to 5:
- Hash("user:123") % 4 = 2  → server 2
- Hash("user:123") % 5 = 0  → server 0 (changed!)
- ~75% of keys are now on the wrong server

With consistent hashing, servers and keys are placed on a ring (0 to 2^32-1). A key maps to the nearest server clockwise on the ring. Adding server S5 only affects keys between S5 and the next server clockwise — typically ~1/N of keys:

// Simplified consistent hashing ring
class ConsistentHashRing {
  private ring: Map<number, string> = new Map();
  private sortedHashes: number[] = [];
  private virtualNodes = 150; //

Health checks: knowing who's alive

A load balancer is only as good as its knowledge of backend health. Sending traffic to a dead server means errors for users. Health checks come in two flavors:

Active health checks: The load balancer periodically probes backends (every 5-30 seconds). If a probe fails N consecutive times, the backend is marked unhealthy and removed from rotation.

upstream backend {
    server 10.0.1.1:3000 max_fails=3 fail_timeout=30s;
    server 10.0.1.2:3000 max_fails=3 fail_timeout=30s;
 
    # Active health check (Nginx Plus only)
    health_check interval=5s fails=3 passes=2;
    health_check uri=/health;
    health_check match=healthy;
}
 
# Define what "healthy" means
match healthy {
    status 200;
    header

Passive health checks (circuit breakers): Instead of probing, the balancer observes actual request outcomes. If a backend returns too many 5xx errors, it's temporarily removed from rotation.

// Good health check — validates dependencies
app.get('/health', async (req, res) => {
  const checks = await Promise.allSettled([
    db.query('SELECT 1'),
    redis

TLS termination at the edge

TLS termination means the load balancer handles the TLS handshake, decrypts the traffic, and forwards unencrypted HTTP to your backend servers. This offloads CPU-intensive cryptographic operations from your application servers.

server {
    listen 443 ssl http2;
    server_name api.example.com;
 
    # Modern TLS configuration
    ssl_certificate     /etc/ssl/certs/example.com.crt;
    ssl_certificate_key /etc/ssl/private/example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    ssl_prefer_server_ciphers on;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 10m;
 
    # OCSP stapling for faster certificate validation
    ssl_stapling 

Rate limiting at the edge

The load balancer is the ideal place for rate limiting — stop abusive traffic before it reaches your application servers and consumes database connections.

# Define rate limit zones
limit_req_zone $binary_remote_addr zone=per_ip:10m rate=30r/s;
limit_req_zone $http_x_api_key zone=per_api_key:10m rate=100r/s;
limit_conn_zone $binary_remote_addr zone=per_ip_conn:10m;
 
server {
    location /api/ {
        # Rate limit: 30 req/s per IP, burst of 20 allowed
        limit_req zone=per_ip burst=20 nodelay;
 
        # Concurrent connection limit: 10 per IP
        limit_conn per_ip_conn 10;
 

Real-world example: How Cloudflare's load balancing works

Cloudflare operates one of the world's largest reverse proxy and load balancing infrastructures. Every request to a Cloudflare-protected site hits their edge first. Key design decisions:

Anycast networking: Cloudflare advertises the same IP address from all 300+ data centers worldwide. BGP (Border Gateway Protocol) routes each user to the nearest data center automatically — no DNS-based geo-routing needed. This is essentially "Layer 3.5 load balancing" that handles DDoS absorption, not just traffic distribution.

Health check philosophy: Cloudflare runs health checks from every data center, not from a centralized location. A backend that's reachable from Virginia but not from Singapore gets routed traffic only from Virginia. This "local health" model prevents regional network issues from taking down the global service.

Failover architecture: Cloudflare supports four steering policies: off (standard), geo-steering, dynamic steering (lowest latency), and proximity steering. During a regional outage, traffic automatically shifts to the next-healthiest origin — this happens within seconds, not minutes.

The lesson for your architecture: Don't build global load balancing from scratch. Use the edge infrastructure that already exists (Cloudflare, CloudFront, Fastly). The engineering effort to build multi-region anycast load balancing rivals the effort to build your entire application.

Common mistakes

Production notes

• Always configure a backup/fallback server. If all primary backends fail health checks, the backup serves a static "under maintenance" page instead of Nginx returning a bare 502 error. This is one line of config that saves your on-call engineer's sanity.
• Use keepalive connections to backends. Without keepalive, every proxied request opens a new TCP connection with the three-way handshake overhead. With keepalive 64 in the upstream block, Nginx maintains a pool of 64 persistent connections to each backend.
• Monitor load balancer metrics separately from app metrics. Track: backend response time per upstream server, 5xx rate by backend, connection queue depth, TLS handshake latency. A slow backend should trigger an alert before users notice.
• Deploy load balancer config as code. Nginx configs, HAProxy configs, and ALB target groups should live in version control and deploy via CI/CD, not manual SSH edits. A bad load balancer config is a global outage.
• Set reasonable timeouts at every layer. Client → LB (60s), LB → backend (30s), backend → database (10s), database query (5s). Each layer's timeout should be shorter than the layer above it to avoid cascading waits.

Performance notes

• TLS session resumption cuts handshake latency by 50-75%. Use ssl_session_cache shared:SSL:10m to cache TLS sessions. Returning visitors skip the full handshake — latency drops from ~100ms to ~2ms for the TLS layer.
• HTTP/2 multiplexing at the edge is a massive win. A single TCP connection carries multiple concurrent requests. Browsers typically open 6 connections per domain — with HTTP/2, all assets stream over one connection, reducing connection overhead by 80%+.
• Use least_time (Nginx Plus) or least_conn with heterogeneous backends. Round-robin with one slow server creates a bottleneck — the slow server gets the same traffic as fast servers, and its latency drags down every Nth request.
• Enable Gzip/Brotli compression at the load balancer. Offload compression from application servers to the LB. Typical text responses (JSON, HTML, CSS, JS) compress to 15-30% of original size. The CPU cost at the LB is negligible compared to the bandwidth savings.
• Proxy buffering is a double-edged sword. It protects backends from slow clients (the LB buffers the response and sends it to the slow client at the client's pace). But for streaming responses (server-sent events, large file downloads), buffering adds latency and memory pressure. Disable it selectively.

Security notes

• Never expose backend servers directly to the internet. All traffic must flow through the load balancer. Security groups should allow port 3000/8000 only from the LB's security group, not from 0.0.0.0/0.
• Use proxy_set_header X-Forwarded-For — but don't trust it blindly. The X-Forwarded-For header can be spoofed by clients if the LB is misconfigured. Always use $proxy_add_x_forwarded_for (which appends, not sets) and configure your app to trust only the last proxy-hop IP.
• Rate limit by authenticated identity, not just IP. An attacker with a valid API key and a botnet of 10,000 IPs bypasses per-IP rate limiting. Layer per-IP rate limiting with per-API-key rate limiting.
• Validate Host headers to prevent cache poisoning. If your Nginx config uses $host in proxy_pass without validation, an attacker sending Host: evil.com can trick your app into generating malicious links. Use an explicit server_name and reject unknown hosts.
• ModSecurity / WAF at the LB layer. SQL injection, XSS, path traversal — these should be blocked at the load balancer before they reach your app. Cloud WAF (AWS WAF, Cloudflare WAF) or self-hosted ModSecurity with Nginx is non-negotiable for production.

Exercises

Beginner

1. Nginx configuration exercise: Given a scenario with two backend services — a user API (2 servers on ports 3001-3002) and a payment API (3 servers on ports 4001-4003) — write an Nginx config that routes /api/users/* to the user API and /api/payments/* to the payment API, with round-robin for both.

2. Health check design: Design a health check endpoint specification (what it checks, what it returns, what status codes) for a service that depends on Postgres, Redis, and an S3-compatible object store. The service should report "healthy" only if Postgres is reachable, "degraded" if Redis is down but Postgres is up, and "unhealthy" if Postgres is down.

Intermediate

1. Implement a weighted least-connections load balancer: Write a TypeScript class that implements a load balancer with weighted least-connections algorithm. It should track active connections per backend, support dynamic weight changes, and handle backend addition/removal without dropping existing connections.

2. Simulate consistent hashing redistribution: Write a simulation that creates a consistent hashing ring with 5 servers and 10,000 keys. Then add a 6th server and calculate the percentage of keys that need to be redistributed. Compare this with simple modulo-based hashing. Experiment with different numbers of virtual nodes (1, 10, 50, 150) and chart the redistribution percentage.

Advanced

1. Design a global load balancing strategy: Your SaaS product has users in North America, Europe, and Asia-Pacific. Latency from Tokyo to Virginia is 150ms. You need: users routed to the nearest healthy region, session continuity (a user doesn't switch regions mid-session), and disaster recovery (if a region goes down, traffic shifts to the next nearest). Design the DNS, load balancing, and data replication strategy. Include tradeoffs between latency, consistency, and cost.

2. Build a rate limiter with sliding window: Implement a sliding-window rate limiter in TypeScript using Redis that limits requests to 100 per minute per API key. Unlike fixed-window (which resets at minute boundaries), the sliding window considers the last 60 seconds. This prevents the "double burst" problem where a client sends 100 requests at second 59 and 100 more at second 1 of the next minute. Use a Lua script for atomicity.

Interview questions

Beginner

1. Q: What's the difference between a load balancer and a reverse proxy? A: A reverse proxy is the broader concept — it sits in front of backend servers and forwards client requests. A load balancer is a specific type of reverse proxy that distributes traffic across multiple backends. All load balancers are reverse proxies. Not all reverse proxies balance load — some might just do caching, SSL termination, or authentication for a single backend.

2. Q: Why would you choose Layer 7 over Layer 4 load balancing? A: L7 gives you content-based routing (route /api/users to one service, /api/payments to another), header manipulation, cookie-based session stickiness, TLS termination with different certificates per domain (SNI), rate limiting, caching, and request inspection. The tradeoff is higher latency (~1-5ms additional) and more CPU usage. Choose L7 when you need HTTP-level intelligence. Choose L4 for raw speed, non-HTTP protocols (database, gRPC without HTTP gateway), or when you're load balancing another load balancer.

3. Q: What happens when a backend server fails a health check? A: The load balancer marks it as unhealthy and stops sending new requests to it. Existing connections may be allowed to complete (depending on draining configuration). The balancer periodically retries the health check. When the backend passes N consecutive checks (typically 2-3), it's marked healthy and rejoins rotation. The key: failure detection must be fast (aggressive health checks) but recovery must be gated (require multiple successes to avoid flapping).

Senior

1. Q: How does consistent hashing minimize disruption when adding or removing cache servers? A: Consistent hashing places servers and keys on a hash ring (0 to 2^32-1). A key maps to the first server clockwise on the ring. When you add a server S_new at hash position H, only keys between H and the next server clockwise are reassigned — typically 1/N of total keys (where N is the number of servers). Without consistent hashing (e.g., hash(key) % N), changing N reassigns nearly all keys. Virtual nodes (multiple hash points per physical server) are critical for even distribution. Without them, a removed server's keys all pile onto a single neighbor, creating a hot spot. With 150 virtual nodes per server, the removed server's keys distribute roughly evenly across remaining servers.

2. Q: You're deploying a new version. How do you drain connections from an old instance without dropping in-flight requests? A: Connection draining has three phases: (1) Remove from rotation — the load balancer stops sending NEW requests to the old instance but allows existing connections to complete. (2) Wait for drain — wait for in-flight requests to finish, up to a maximum timeout (typically 30-300 seconds, depending on your longest legitimate request). Long-lived connections (WebSockets, SSE) need special handling — either wait for them to disconnect naturally or forcibly terminate after a generous grace period. (3) Force termination — after the drain timeout, forcibly close remaining connections and terminate the instance. Critical details: the load balancer must implement this (ALB connection draining, Nginx drain mode), health checks should fail during drain so new instances take over, and you should monitor for elevated 5xx rates during the drain window.

3. Q: Design a rate limiting strategy that handles both authenticated and unauthenticated users differently, and survives a Redis outage. A: Implement a three-tier rate limiting strategy. Tier 1 (edge): IP-based rate limiting at the load balancer level (e.g., 60 req/min per IP). This stops basic DoS and scraping. Tier 2 (authenticated): Per-user rate limiting using Redis counters with sliding windows. Authenticated users get higher limits (1000 req/min) and burst allowances. Tier 3 (cost-based): Assign a "cost" to each endpoint (GET /users = 1 point, POST /payments = 10 points, file upload = 50 points). Users get a point budget per minute. Redis failure strategy: Implement a local fallback rate limiter. When Redis is unreachable, each app instance switches to in-memory rate limiting. Since instances don't share , the effective limit becomes N × per-instance limit (where N is instance count). This is generous but prevents total failure. For stricter fail-closed behavior, reduce the per-instance limit to . Monitor Redis availability and alert on fallback mode activation.

Summary

Load balancers and reverse proxies are the nervous system of a distributed backend. They route traffic, absorb TLS overhead, rate-limit abuse, and shield your application from the internet's chaos. The choice between L4 (fast, protocol-agnostic) and L7 (intelligent, HTTP-aware) shapes your architecture's flexibility. The algorithm — round-robin, least-connections, consistent hashing, or weighted — determines how evenly and predictably load distributes. Health checks must validate real functionality, not just process liveness. TLS termination at the edge offloads crypto but demands care with downstream encryption. Rate limiting at the load balancer layer stops abuse before it reaches your expensive backend resources. Tools like Nginx, HAProxy, and cloud-native ALBs/Cloudflare make these patterns accessible — the engineering is in knowing which pattern to apply and when.

Quick recall

• L4 = TCP-level, fast, protocol-agnostic. L7 = HTTP-level, content-aware routing, TLS termination.
• Round-robin for uniform workloads. Least-connections for variable-duration requests. Consistent hashing for caches.
• Health checks must validate dependency health (DB, cache, queue), not just process liveness.
• Consistent hashing with 150 virtual nodes per server minimizes key redistribution on server changes.
• TLS termination at load balancer. Use TLS re-encryption to backends for sensitive data.
• Rate limiting at the LB: per-IP + per-API-key + cost-based endpoint weighting.
• Connection draining: stop new requests → wait for in-flight → force terminate after timeout.
• Proxy buffering: protects backends from slow clients but hurts streaming endpoints — disable selectively.
• HTTP/2 multiplexing at the edge reduces connection overhead by 80%+.
• Never expose backends directly to the internet. Security groups should only allow LB traffic.

Quiz

1. Which load balancing algorithm minimizes key redistribution when servers are added or removed? A) Round-robin B) Least connections C) Consistent hashing D) IP hash

2. A health check that returns 200 OK when the process is alive but the database is unreachable is: A) A good minimal check B) Sufficient for most use cases C) Worthless — it will route traffic to a server that can't serve requests D) Required by cloud providers

3. What does keepalive 64 in an Nginx upstream block do? A) Limits the server to 64 connections B) Maintains a pool of 64 persistent TCP connections to each backend C) Keeps client connections alive for 64 seconds D) Limits the upstream to 64 servers

4. When should you use Layer 4 instead of Layer 7 load balancing? A) When you need URL-based routing B) When load balancing non-HTTP protocols (database, raw TCP) or when raw throughput matters more than HTTP intelligence C) When you need TLS termination D) Always — L4 is better in every way

5. What's the purpose of virtual nodes in consistent hashing? A) To evenly distribute keys when a server is added or removed, preventing hot spots B) To create virtual servers that don't cost money C) To encrypt hash values D) To increase the ring size

6. Your Nginx config uses $host in proxy_pass, and an attacker sends Host: evil.com. What vulnerability does this create? A) SQL injection B) DDoS C) Cache poisoning — your app might generate links pointing to evil.com D) TLS certificate mismatch

7. During deployment, you remove an instance from rotation and immediately terminate it. What happens to in-flight requests? A) They complete successfully B) They're automatically retried by the load balancer C) They're abruptly terminated — users see errors. You need connection draining first D) They're buffered and replayed