HAProxy Timeout Ladder: Which Timeout Is Causing Your 503s

HAProxy returns 503 when it has a request in hand and no backend it is willing to use. The misleading part is that the full 503 rate rarely maps to one timeout. On a typical frontend with defaults in place, timeout client, timeout server, timeout connect, timeout http-request and timeout http-keep-alive all interact, and the one you tuned is frequently not the one firing. Reading show stat with the right column tells you in seconds which layer gave up.

The ladder is about who waits for whom. Each timeout is a deadline for a different party: the client, the server, the connection attempt, the header exchange, or an idle keep-alive slot. When two of them disagree, the shorter one wins and the logs blame whichever subsystem closed last. On a 2 GB VPS serving a few hundred requests per second, a plainly provisioned KVM instance is enough as long as these values are set deliberately rather than inherited.

Map each timer to the party that waits

Write the five timeouts down next to the state they govern. If you cannot say which component is blocked in each state, you will tune by intuition and lose. The usual defaults of 50s for client and server, 10s for connect, and 5s for keep-alive are tuned for a generic web app; they are wrong for APIs with 90-second background jobs and equally wrong for a static site behind a CDN.

Directive	State	Typical 503 signature
timeout connect	Backend TCP handshake	sC=1 in stats, rising fast
timeout server	Backend is silent	sH=1 with sC=0
timeout client	Client stops sending	cR=1, no server-side errors
timeout http-request	Partial headers	400/408, sometimes 503 upstream
timeout http-keep-alive	Idle idle keep-alive socket	No error; pool churn instead

timeout connect: the backend is not answering SYNs

A 503 caused by timeout connect means the backend box never completed a TCP handshake within the window. It is not slow application code; it is a host that is down, firewalled, or accepting connections slower than the listen backlog allows. In show stat, the sC column increments per attempt, and a healthy backend shows zero on that counter for the whole minute.

Ten seconds is a sane default for a LAN backend; two seconds is fine inside a single host. Raising it to 30 to “fix” 503s only stretches every failure into a longer user-visible stall, because failed health checks and retries all inherit the same window. If half your connects are slow, the problem is the backend’s SYN queue, not your HAProxy value.

timeout server: the request is in flight and nothing comes back

This is the most common misattribution. The connection succeeded, the request went out, and the backend stayed silent past the deadline, so HAProxy cuts the connection and returns 503 to the client. The distinguishing signal is sH=1 with sC=0 in the stats row: the server was reached but produced no response headers in time.

Set this above your slowest legitimate response, not above your slowest observed one. If your p99 is 1.2s, a 60s value means a stuck backend consumes a connection slot for a minute before the client learns anything. Log %ts with a high precision to compare server response times to this timeout; if the 503 timestamps coincide with %ts near the limit, this directive is the culprit.

timeout client: the client went quiet

timeout client applies in both directions: while waiting for the client’s request headers and while streaming the response back. A slow mobile client on a bad link can trip this and the connection is closed normally, but if the backend is also slow the pooled connection can be freed in a state that produces an upstream 503 on retry. Look at cR for client-side aborts and cross-check against your access log’s total request duration.

Thirty seconds is generous for header arrival and tight for a large upload. Keep the two purposes separate: use timeout http-request for header deadlines and leave timeout client for the streaming phase. Conflating them is how people end up with 60-second uploads timing out at the 30-second client limit and blaming the app server.

timeout http-request and keep-alive: the two that look harmless

timeout http-request limits how long a client may take to send a complete header block. A default of five seconds works until a client sends a POST with a large header set over a slow link and gets a 408 instead. That 408 is not a 503, but it burns a frontend connection count, and with maxconn near its limit the next request genuinely gets 503 with no backend error at all.

defaults
  mode http
  timeout connect 3s
  timeout client  30s
  timeout server  30s
  timeout http-request 10s
  timeout http-keep-alive 5s
  timeout check 2s

frontend web
  maxconn 2000
  bind *:80
  default_backend app

timeout http-keep-alive never causes a 503 directly. It closes idle keep-alive sockets, and if it is shorter than the client’s own keep-alive plan, clients reconnect constantly and each reconnect consumes a frontend slot. On a VPS with maxconn 2000 and ulimit -n still at 1024, that churn produces real 503s at the socket level. Check ulimit -n before touching any timer.

Ordering rules that prevent contradictions

Three invariants keep the ladder coherent. Keep-alive must be shorter than client, so an idle socket is reclaimed before the client deadline fires. Connect must be much shorter than server, so a dead host fails fast instead of sitting in a retry loop. And the health check timeout must be shorter than server, otherwise a single sick backend holds a check slot while the pool marks it up.

• timeout http-keep-alive < timeout client (typically 5s vs 30s)
• timeout connect < timeout server (3s vs 30s)
• timeout check < timeout server (2s vs 30s)
• timeout http-request close to timeout connect, never above client
• maxconn under 80 percent of the process fd limit

Diagnosing with stats instead of guessing

Enable the stats socket and query it directly. The four counters that matter are ereq (request errors), econ (connection errors), sC (server connect retries) and sH (server aborts). A row where econ climbs while sH stays flat points at connect; the reverse points at server. That distinction, made in one command, saves a tuning round trip.

echo "show stat" | socat stdio /var/run/haproxy/admin.sock \
  | cut -d, -f2,5,13,14,60,61
haproxy -c -f /etc/haproxy/haproxy.cfg

Make one change at a time and reload rather than restart; a reload keeps the stats counters, a restart resets them and wipes the evidence. After a fix, watch the 503 rate for a full keep-alive cycle, meaning at least twice the longest timeout you touched. On a well-sized KVM virtual machine the rate should fall to the background level within five minutes and stay there.