Building Resilient API Clients — Timeouts, Retries, TLS, and Circuit Breakers
Configure an outbound HTTP client in PHP with explicit connect/request timeouts, retry logic using exponential backoff for transient failures, mandatory TLS certificate verification, mapping of HTTP status codes and network errors to typed exceptions the caller can handle meaningfully, and the circuit-breaker pattern for avoiding cascading failure when a downstream dependency is unhealthy.
Calling another service over the network is fundamentally less reliable than calling a function in your own process -- the network can be slow, the remote service can be down, a single request can hang indefinitely, and a transient blip that a simple retry would fix can otherwise take down a request that had nothing really wrong with it. A resilient API client treats all of this as the normal, expected cost of doing business over a network rather than as exceptional edge cases, and handles it with a small, consistent set of techniques: bounded timeouts, smart retries, real TLS verification, typed error handling, and -- at scale -- circuit breakers that stop hammering a service that's already struggling.
Timeouts: Never Call the Network Without a Bound
An HTTP call with no timeout can hang forever if the remote server accepts the TCP connection but never responds -- a genuinely common failure mode (an overloaded server, a stuck backend, a network partition mid-response), and one that will eventually exhaust your own server's request-handling capacity as calls pile up waiting on a dependency that will never answer. Every outbound call needs an explicit CONNECT timeout (how long to wait for the initial TCP+TLS handshake) and a separate total/request timeout (how long to wait for the full response after that), because these two failure modes have different likely causes and different sensible bounds.
The right timeout value depends on the call: a user-facing request that a person is actively waiting on needs a short, tight timeout (a few seconds at most, because making the user wait longer than that is itself a failure); a background job processing a queue can often afford a longer one. There is no universally correct timeout value -- only a deliberately chosen one appropriate to the call's context, versus the far more common mistake of no timeout at all.
• Connect timeout: bounds how long to wait for the TCP/TLS handshake to complete. • Request/total timeout: bounds how long to wait for the full response after connecting. • No timeout at all means a hung remote server can hang your own request-handling capacity indefinitely. • Choose the timeout value deliberately per call context (user-facing vs background) -- there's no one-size value.
Retries with Exponential Backoff
Not every failure should be retried, and retrying the wrong kind makes things worse, not better. Transient failures -- a connection timeout, a 503 Service Unavailable, a 429 Too Many Requests -- are worth retrying because the same request often succeeds moments later. Non-transient failures -- a 400 Bad Request (the request itself is malformed and will fail identically every time), a 401/403 (retrying with the same bad credentials changes nothing), a 404 -- should NOT be retried, because retrying them just wastes time and load for a guaranteed identical failure.
When you do retry, back off: retrying immediately in a tight loop against a struggling service adds MORE load right when it can least handle it, potentially turning a brief blip into a sustained outage. Exponential backoff waits progressively longer between attempts (1s, 2s, 4s, 8s...), and adding random jitter to each wait prevents many clients that all failed at the same moment from all retrying in lockstep and re-creating the exact spike that caused the failure in the first place.
Retries must also respect a Retry-After header when the server sends one (common on 429 and 503 responses) -- it's the server telling you exactly how long to wait, and overriding that with your own shorter backoff defeats the purpose of the server sending it at all.
• Retry transient failures (timeouts, 503, 429); never retry non-transient ones (400, 401/403, 404) -- they'll fail identically every time. • Exponential backoff: wait progressively longer between attempts (1s, 2s, 4s...) instead of retrying immediately. • Add jitter (randomness) to backoff so many simultaneously-failing clients don't retry in lockstep and re-spike the same failure. • Honor a Retry-After header when the server sends one -- it overrides your own default backoff schedule.
TLS Verification: Never Disable It
TLS certificate verification confirms the server you connected to is actually who it claims to be, not an attacker intercepting the connection (a man-in-the-middle). When a client hits a certificate error -- often during local development against a self-signed cert, or in a misconfigured environment -- the fastest 'fix' is to disable verification entirely (CURLOPT_SSL_VERIFYPEER => false or equivalent). This is never an acceptable fix outside a genuinely isolated local sandbox, because it silently removes the entire guarantee that the connection is even talking to the real server, and it is exactly the kind of setting that gets copy-pasted from a local dev script into a production deployment where it becomes a serious, quiet vulnerability.
The correct fix for a legitimate certificate problem (an internal CA, a self-signed cert in a genuinely trusted internal environment) is to install and trust that specific certificate authority, not to turn verification off entirely -- keeping the guarantee intact for every OTHER connection the client makes while accommodating the one legitimate exception.
• TLS verification confirms you're actually talking to the real server, not an attacker in the middle. • Disabling it 'to make a dev error go away' is never acceptable outside a fully isolated local sandbox. • For a legitimate internal/self-signed cert, trust that specific CA -- don't disable verification globally. • This exact shortcut is one of the most common causes of real production security incidents in API clients.
Mapping Errors to Typed Exceptions
A raw HTTP client typically returns either a response object (with whatever status code, even an error one) or throws a low-level transport exception (DNS failure, connection refused, timeout) -- neither of which tells the CALLING code, in its own vocabulary, what actually went wrong or what to do about it. A well-designed client wraps these into a small hierarchy of typed exceptions specific to the calling application's domain -- ApiTimeoutException, ApiAuthenticationException, ApiNotFoundException, ApiServerException -- so calling code can catch exactly the failure modes it knows how to handle (maybe retry on timeout, maybe show a specific 'not found' message on 404) without needing to know the underlying HTTP client's own exception classes or manually inspect status codes at every call site.
• A raw client's status codes/transport exceptions don't tell calling code what to DO -- translate them into a typed exception hierarchy. • Calling code then catches specific, meaningful exception types instead of manually branching on status codes everywhere it makes a call. • This isolates the rest of the application from which underlying HTTP client library is actually being used.
The Circuit Breaker Pattern
Retries and timeouts handle an individual request's resilience, but neither stops your application from continuing to hammer a downstream service that is already completely down -- every request still waits out its full timeout and retry schedule before failing, multiplying load on a struggling dependency and slowing down every caller in the process. A circuit breaker wraps calls to a dependency and tracks their success/failure rate; once failures cross a threshold, it 'opens' -- for a cooldown period, calls fail IMMEDIATELY without even attempting the network call, giving the struggling downstream service room to recover instead of continuing to be hit by a full load of doomed requests.
After the cooldown, the breaker moves to a 'half-open' state and lets a small number of test requests through; if they succeed, it closes again (normal traffic resumes); if they still fail, it reopens and waits another cooldown. This three-state cycle (closed -> open -> half-open -> closed or back to open) is exactly why circuit breakers prevent cascading failure: a struggling downstream service that would otherwise be kept struggling by continuous retry traffic instead gets a real chance to recover, and callers fail fast instead of each independently waiting out a timeout that was never going to succeed.
• Closed (normal): calls pass through and are tracked; too many failures trips the breaker open. • Open: calls fail immediately, no network attempt at all, for a cooldown period -- protects the struggling dependency AND fails fast for callers. • Half-open: after cooldown, a small number of test calls decide whether to close (recovered) or reopen (still down). • Prevents cascading failure: without it, every caller independently retries and times out against a service that's already down, multiplying load and amplifying the outage.
A Guzzle Client Configured for Resilience
connect_timeout and timeout are set separately because a slow handshake and a slow response body are different failure modes worth distinguishing. verify: true (or omitted -- true is the default) keeps certificate verification on; it should never be explicitly set to false outside an isolated sandbox. http_errors: false lets the calling code inspect the status code itself and decide how to map it, rather than Guzzle throwing generically on every 4xx/5xx.
$client = new \GuzzleHttp\Client([
'connect_timeout' => 2.0, // seconds to establish TCP+TLS
'timeout' => 5.0, // seconds for the full request/response
'verify' => true, // TLS verification ON — never false in production
'http_errors' => false, // handle status codes ourselves instead of exceptions-on-4xx/5xx
]);Code Examples
The client distinguishes non-transient failures (404, 401/403) that throw immediately with a typed, meaningful exception from transient ones (connection timeout, 429, 5xx) that retry with exponential backoff and jitter, honoring a server-provided Retry-After header when present. Calling code can now write catch (ApiNotFoundException) or catch (ApiAuthenticationException) with meaningful, specific handling, instead of inspecting a raw status code at every call site throughout the application.
$status === 429, $status >= 500 => $this->retryOrFail($attempt, $status, $response, $path),
The breaker tracks consecutive failures and, once past the threshold, moves to Open and rejects every call immediately (no network attempt) until the cooldown passes, then allows exactly one half-open test call to decide whether to close again or reopen. Expected result: Calls fail instantly (CircuitOpenException) during the cooldown window instead of each waiting out a full timeout against a downstream that's already down throw new CircuitOpenException('Circuit open — failing fast without calling downstream');
Common mistake: Applying retries and a circuit breaker at the SAME layer without coordinating them -- retrying 4 times inside a call that the breaker also wraps means each 'one call' to the breaker actually generates 4 real network attempts, delaying when the breaker ever sees enough distinct failures to trip open.
<?php
declare(strict_types=1);
final class ApiTimeoutException extends RuntimeException {}
final class ApiAuthenticationException extends RuntimeException {}
final class ApiNotFoundException extends RuntimeException {}
final class ApiServerException extends RuntimeException {}
final class ResilientApiClient
{
private const MAX_ATTEMPTS = 4;
public function __construct(private \GuzzleHttp\Client $http) {}
public function get(string $path): array
{
$attempt = 0;
while (true) {
$attempt++;
try {
$response = $this->http->get($path, [
'connect_timeout' => 2.0,
'timeout' => 5.0,
]);
} catch (\GuzzleHttp\Exception\ConnectException $e) {
if ($attempt >= self::MAX_ATTEMPTS) {
throw new ApiTimeoutException("Timed out after {$attempt} attempts", previous: $e);
}
$this->backoff($attempt);
continue; // retry: transient
}
$status = $response->getStatusCode();
return match (true) {
$status === 200 => json_decode((strin
…<?php
declare(strict_types=1);
enum CircuitState { case Closed; case Open; case HalfOpen; }
final class CircuitBreaker
{
private CircuitState $state = CircuitState::Closed;
private int $failureCount = 0;
private ?int $openedAt = null;
public function __construct(
private int $failureThreshold = 5,
private int $cooldownSeconds = 30,
) {}
public function call(callable $fn): mixed
{
if ($this->state === CircuitState::Open) {
if (time() - $this->openedAt >= $this->cooldownSeconds) {
$this->state = CircuitState::HalfOpen; // try a test request
} else {
throw new CircuitOpenException('Circuit open — failing fast without calling downstream');
}
}
try {
$result = $fn();
$this->onSuccess();
return $result;
} catch (Throwable $e) {
$this->onFailure();
throw $e;
}
}
private function onSuccess(): void
{
$this->failureCount = 0;
$this->state = CircuitState::Closed; // recovered (or was already healthy)
}
private function onFailure(): void
{
…Circuit breaker state machine
Open fails fast and protects the downstream; half-open cautiously tests recovery before fully trusting it again.
failures < threshold
+-------------------+
| |
v |
+---------+ threshold exceeded +--------+
| CLOSED | ------------------------>| OPEN |
+---------+ +--------+
^ |
| success cooldown elapsed
| |
| v
| +-------------+
+------------------------- | HALF-OPEN |
test call fails --------> (reopens) |
+-------------+Important Rules
Set an explicit connect timeout and a separate request timeout on every outbound call -- never call the network unbounded.
Retry only transient failures (timeouts, 429, 5xx) with exponential backoff plus jitter; never retry non-transient ones (400, 401/403, 404).
Honor a server's Retry-After header when present instead of overriding it with your own backoff schedule.
Keep TLS verification on in every environment except a genuinely isolated local sandbox; fix real certificate problems by trusting the specific CA, not by disabling verification.
Map raw status codes and transport exceptions into a typed exception hierarchy so calling code can handle specific failure modes meaningfully.
Wrap calls to a dependency in a circuit breaker so repeated failures stop generating full-timeout network attempts and instead fail fast during a cooldown.
Netflix's Hystrix and the popularization of circuit breakers
Netflix's Hystrix library (now succeeded by resilience4j and similar libraries in other languages, including PHP ports of the same pattern) popularized the circuit-breaker pattern industry-wide after Netflix's own experience with cascading failures: a single slow or failing internal service, left unprotected, could exhaust connection pools and thread capacity in every service that called it, then in every service that called THOSE services, turning one dependency's bad day into a platform-wide outage. Wrapping every inter-service call in a circuit breaker (plus bounded timeouts and bulkheads limiting concurrent calls to any one dependency) was the fix that let Netflix's architecture degrade gracefully -- one struggling dependency, not the whole platform.
Setting CURLOPT_SSL_VERIFYPEER => false (or Guzzle's 'verify' => false) to make a local certificate error go away, then shipping that configuration to production.
Keep verification on everywhere; for a legitimate self-signed or internal CA, install and trust that specific certificate instead of disabling verification globally.
Disabling TLS verification removes the guarantee that the client is even talking to the real server, turning every request into a potential target for a man-in-the-middle attack -- and this exact shortcut, copy-pasted from a dev fix, is a recurring real-world cause of production security incidents.
Retrying a 401 Unauthorized response with the same credentials several times before giving up.
Treat 401/403 as non-transient and fail immediately with a typed authentication exception -- retrying with identical bad credentials produces the identical failure every time.
Retrying a failure that cannot possibly succeed on a later attempt wastes time, adds needless load on the downstream service, and delays the caller from finding out about a real problem (like an expired credential) that needs to actually be fixed, not retried.
Why should a 400 Bad Request or 401 Unauthorized response generally NOT be retried by an API client? A. Retrying these status codes is against the HTTP specification B. These are non-transient failures — the same request will fail identically on every retry, so retrying just wastes time and load C. Only 5xx status codes are valid HTTP responses D. Retrying them causes the server to ban the client automatically
B. These are non-transient failures — the same request will fail identically on every retry, so retrying just wastes time and load — A malformed request or bad credentials will produce the exact same failure on every retry since nothing about the request changes; only transient failures (timeouts, 429, 5xx) are worth retrying because the same request might succeed on a later attempt.
What problem does the circuit-breaker pattern solve that retries and timeouts alone do not? A. It encrypts traffic to the downstream service B. It stops the client from continuing to send full-timeout requests to an already-failing downstream service, failing fast instead and giving that service room to recover C. It automatically fixes bugs in the downstream service D. It replaces the need for TLS certificate verification
B. It stops the client from continuing to send full-timeout requests to an already-failing downstream service, failing fast instead and giving that service room to recover — Timeouts and retries still let every caller independently wait out a timeout and retry against a service that's already down, multiplying load on it; a circuit breaker detects sustained failure and fails immediately during a cooldown, protecting the downstream and giving callers a fast, predictable failure instead.
Combine the ResilientApiClient and CircuitBreaker examples correctly: rewrite the usage so the circuit breaker wraps ONE full attempt at a time (not the whole internal retry loop), meaning the breaker should see each individual HTTP attempt as a separate call so its failure count reflects real, distinct network-level failures rather than being hidden behind the client's own internal retries.
In ResilientApiClient::get()'s while(true) loop, replace the direct $this->http->get(...) call with $response = $this->breaker->call(fn() => $this->http->get($path, [...])); -- this way each individual attempt inside the retry loop is what the breaker tracks, so 5 consecutive individual failed attempts (whether from one call's retries or across several calls) trips it open, and a CircuitOpenException thrown from inside the loop should be caught and re-thrown immediately without further retries, since retrying while the circuit is open would defeat its purpose.
Refactor ResilientApiClient to accept the CircuitBreaker in its constructor and call $this->breaker->call(...) around the single low-level $this->http->get(...) call inside the retry loop, rather than wrapping the whole get() method from outside.