Design a Load Balancer

Requirements

Functional: distribute incoming requests across a pool of backend servers; stop sending traffic to unhealthy servers. Non-functional: the load balancer itself must not become a single point of failure; adding/removing backend servers shouldn't require downtime.

Architecture

Algorithm choice

• Round robin: cycles through servers in order — simple, works well when all servers and all requests are roughly equal cost.
• Least connections: sends the next request to whichever server currently has the fewest active connections — better when request processing time varies a lot.
• Weighted variants of either: account for servers with different capacities (e.g. a bigger instance gets proportionally more traffic).

Layer 4 vs. Layer 7

A Layer 4 (transport-layer) load balancer routes based on IP/port only — fast, but can't make decisions based on request content. A Layer 7 (application-layer) load balancer reads the actual HTTP request (path, headers, cookies) and can route /api/* to one pool and /static/* to another, or keep a user's session pinned to the same server. L7 costs more CPU per request but is far more common today because that routing flexibility is usually worth it.

Follow-up: how do you make the load balancer itself highly available?

Run multiple load balancer instances behind a floating/virtual IP (or DNS round-robin across them), with a protocol like VRRP so if the active instance fails, a standby takes over the IP within seconds — the load balancer is just as much a single-point-of-failure risk as any backend server, and needs the same redundancy treatment.