{"id":406,"date":"2026-06-12T03:59:00","date_gmt":"2026-06-12T03:59:00","guid":{"rendered":"https:\/\/virtualserversvps.com\/blog\/?p=406"},"modified":"2026-11-03T22:04:07","modified_gmt":"2026-11-03T22:04:07","slug":"haproxy-timeout-ladder-tcp-http-establish","status":"publish","type":"post","link":"https:\/\/virtualserversvps.com\/blog\/haproxy-timeout-ladder-tcp-http-establish\/","title":{"rendered":"HAProxy Timeout Ladder: Which Timeout Is Causing Your 503s"},"content":{"rendered":"\n

HAProxy returns 503 when it has a request in hand and no backend it is willing to use. The misleading part is that the full 503 rate rarely maps to one timeout. On a typical frontend with defaults in place, timeout client<\/code>, timeout server<\/code>, timeout connect<\/code>, timeout http-request<\/code> and timeout http-keep-alive<\/code> all interact, and the one you tuned is frequently not the one firing. Reading show stat<\/code> with the right column tells you in seconds which layer gave up.<\/p>\n\n\n\n

The ladder is about who waits for whom. Each timeout is a deadline for a different party: the client, the server, the connection attempt, the header exchange, or an idle keep-alive slot. When two of them disagree, the shorter one wins and the logs blame whichever subsystem closed last. On a 2 GB VPS serving a few hundred requests per second, a plainly provisioned KVM instance<\/a> is enough as long as these values are set deliberately rather than inherited.<\/p>\n\n\n\n

Map each timer to the party that waits<\/h2>\n\n\n\n

Write the five timeouts down next to the state they govern. If you cannot say which component is blocked in each state, you will tune by intuition and lose. The usual defaults of 50s for client and server, 10s for connect, and 5s for keep-alive are tuned for a generic web app; they are wrong for APIs with 90-second background jobs and equally wrong for a static site behind a CDN.<\/p>\n\n\n\n

Directive<\/th>State<\/th>Typical 503 signature<\/th><\/tr><\/thead>
timeout connect<\/td>Backend TCP handshake<\/td>sC=1 in stats, rising fast<\/td><\/tr>
timeout server<\/td>Backend is silent<\/td>sH=1 with sC=0<\/td><\/tr>
timeout client<\/td>Client stops sending<\/td>cR=1, no server-side errors<\/td><\/tr>
timeout http-request<\/td>Partial headers<\/td>400\/408, sometimes 503 upstream<\/td><\/tr>
timeout http-keep-alive<\/td>Idle idle keep-alive socket<\/td>No error; pool churn instead<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

timeout connect: the backend is not answering SYNs<\/h2>\n\n\n\n

A 503 caused by timeout connect<\/code> means the backend box never completed a TCP handshake within the window. It is not slow application code; it is a host that is down, firewalled, or accepting connections slower than the listen backlog allows. In show stat<\/code>, the sC<\/code> column increments per attempt, and a healthy backend shows zero on that counter for the whole minute.<\/p>\n\n\n\n

Ten seconds is a sane default for a LAN backend; two seconds is fine inside a single host. Raising it to 30 to “fix” 503s only stretches every failure into a longer user-visible stall, because failed health checks and retries all inherit the same window. If half your connects are slow, the problem is the backend’s SYN queue, not your HAProxy value.<\/p>\n\n\n\n

timeout server: the request is in flight and nothing comes back<\/h2>\n\n\n\n

This is the most common misattribution. The connection succeeded, the request went out, and the backend stayed silent past the deadline, so HAProxy cuts the connection and returns 503 to the client. The distinguishing signal is sH=1<\/code> with sC=0<\/code> in the stats row: the server was reached but produced no response headers in time.<\/p>\n\n\n\n

Set this above your slowest legitimate response, not above your slowest observed one. If your p99 is 1.2s, a 60s value means a stuck backend consumes a connection slot for a minute before the client learns anything. Log %ts<\/code> with a high precision to compare server response times to this timeout; if the 503 timestamps coincide with %ts<\/code> near the limit, this directive is the culprit.<\/p>\n\n\n\n

timeout client: the client went quiet<\/h2>\n\n\n\n

timeout client<\/code> applies in both directions: while waiting for the client’s request headers and while streaming the response back. A slow mobile client on a bad link can trip this and the connection is closed normally, but if the backend is also slow the pooled connection can be freed in a state that produces an upstream 503 on retry. Look at cR<\/code> for client-side aborts and cross-check against your access log’s total request duration.<\/p>\n\n\n\n

Thirty seconds is generous for header arrival and tight for a large upload. Keep the two purposes separate: use timeout http-request<\/code> for header deadlines and leave timeout client<\/code> for the streaming phase. Conflating them is how people end up with 60-second uploads timing out at the 30-second client limit and blaming the app server.<\/p>\n\n\n\n

timeout http-request and keep-alive: the two that look harmless<\/h2>\n\n\n\n

timeout http-request<\/code> limits how long a client may take to send a complete header block. A default of five seconds works until a client sends a POST with a large header set over a slow link and gets a 408 instead. That 408 is not a 503, but it burns a frontend connection count, and with maxconn<\/code> near its limit the next request genuinely gets 503 with no backend error at all.<\/p>\n\n\n\n

defaults\n  mode http\n  timeout connect 3s\n  timeout client  30s\n  timeout server  30s\n  timeout http-request 10s\n  timeout http-keep-alive 5s\n  timeout check 2s\n\nfrontend web\n  maxconn 2000\n  bind *:80\n  default_backend app<\/code><\/pre>\n\n\n\n
timeout http-keep-alive<\/code> never causes a 503 directly. It closes idle keep-alive sockets, and if it is shorter than the client’s own keep-alive plan, clients reconnect constantly and each reconnect consumes a frontend slot. On a VPS with maxconn 2000<\/code> and ulimit -n<\/code> still at 1024, that churn produces real 503s at the socket level. Check ulimit -n<\/code> before touching any timer.<\/p>\n\n\n\n
Ordering rules that prevent contradictions<\/h2>\n\n\n\n
Three invariants keep the ladder coherent. Keep-alive must be shorter than client, so an idle socket is reclaimed before the client deadline fires. Connect must be much shorter than server, so a dead host fails fast instead of sitting in a retry loop. And the health check timeout must be shorter than server, otherwise a single sick backend holds a check slot while the pool marks it up.<\/p>\n\n\n\n