Load balancers and proxies

Once you have several servers, something must decide which one gets each request. A load balancer sits in front of them, accepts every request, and forwards it to a healthy server. Clients see a single address and never know how many servers exist.

It gives you three things: capacity (work is shared), availability (a dead server is skipped), and freedom to add or remove servers without telling anyone.

A balancer can go at every boundary between layers.

• Between clients and web servers, which is the most common place.
• Between web servers and application servers.
• Between application servers and the database or cache tier.

You do not need all three. Add one where a layer has more than one machine and traffic must be shared.

The choice is made by an algorithm. Each has a strength and a weakness.

• Round robin: take turns in order. Simple and fair when requests cost the same, but it ignores how busy each server is.
• Weighted round robin: stronger servers get more turns. Good for a mixed fleet.
• Least connections: send to the server with the fewest open connections. Adapts to slow requests.
• Least response time: prefer the server that answers fastest.
• IP hash: the client address always maps to the same server. This keeps a user on one server, but spreads load unevenly.

Start with round robin or least connections, and move to something smarter only when you can show a problem.

A layer 4 balancer decides using only network facts: addresses and ports. It is very fast and does not look inside the request. A layer 7 balancer reads the actual HTTP request, so it can route by path (images to one pool, API calls to another), by header, or by cookie, and it can end the TLS connection. It uses more CPU but is far more flexible.

Both run health checks: they probe each server regularly and stop sending traffic to one that fails to answer. That is how a crash becomes invisible to users.

A single balancer is itself a single point of failure, the very thing we were removing. The standard fix is a pair. One is active and the other watches it with a heartbeat. If the active one stops, the standby takes over its address.

Sticky sessions, where a user is pinned to one server, can seem handy but undo much of the benefit: that server becomes special. Prefer stateless servers and keep sessions in a shared store.

A proxy is a middleman that forwards requests on someone's behalf. The two kinds differ in whose side they are on.

• A forward proxy sits in front of clients. It hides who the clients are, can filter sites, and can cache popular pages for a whole office.
• A reverse proxy sits in front of servers. It hides the servers, ends TLS, compresses and caches responses, and shields them from direct attacks.

A load balancer is essentially a reverse proxy that also shares load. The tradeoff of any proxy is an extra hop of latency and one more thing that must be kept running, so add one for a reason you can name.