Resilient Outbound HTTP Clients

Calling another service is not ordinary function invocation. The caller crosses a network, enters another failure domain, waits behind queues it cannot inspect, and receives bytes that may not satisfy the promised contract. A convenient fetch() scattered through business code hides those facts without removing them.

A resilient outbound HTTP client makes that boundary explicit. It owns transport reuse, deadline enforcement, retry classification, request identity, response limits, decoding, and telemetry. Business code supplies an operation and receives a typed result; it does not improvise timeout and retry behavior at every call site.

This article stays at that boundary: every attempt is bounded, every repeat is justified, every response is treated as untrusted input, and every outcome is observable without exposing secrets. Circuit and concurrency policy remains above the client.

Make the Boundary a Real Component #

An outbound dependency should have a client named after the capability it provides, such as InventoryClient, rather than a generic HTTP helper exposed throughout the application. The capability client translates domain operations into HTTP details and translates HTTP outcomes back into a small error taxonomy. That gives one place to enforce policy and one contract for callers to test.

Separate three layers even if they live in one module:

1. A long-lived transport owns connection pools, DNS behavior, TLS sessions, proxy configuration, and low-level socket limits.
2. A request executor applies deadlines, retry rules, body limits, and attempt telemetry.
3. A capability adapter constructs routes and validates domain-specific response schemas.

Creating a transport per request forfeits keep-alive and TLS reuse, raises handshake latency and ephemeral-port use, and magnifies bursts. Keep it for the process lifetime and close it during graceful shutdown.

Each dependency still needs an explicit origin, authentication method, response limit, and operation policy. A payment lookup and an analytics export need not share a body cap or deadline. Validate configuration at startup rather than on the first live request.

Keep credentials and mutable headers out of logs and generic exception messages. Build authorization headers at the last responsible moment, allow-list safe telemetry fields, and prevent callers from overriding security-sensitive headers such as Host or a request signature. The boundary is where outbound trust begins.

Reuse Connections Without Letting Pools Run Wild #

Connection reuse removes repeated TCP and TLS setup, but a pool is also a queue and resource limit. An unbounded pool can overwhelm the dependency; a tiny one serializes independent work. Expose transport settings deliberately rather than treating keep-alive as a magic switch.

For HTTP/1.1, a connection usually carries one active request at a time, so maximum connections strongly influences concurrency. HTTP/2 multiplexes streams over fewer connections, but peer stream limits, flow control, and head-of-line effects inside the application still matter. A protocol upgrade does not make capacity infinite.

Useful transport controls include:

Pool waiting must count against the request deadline. If a request waits 400 ms for a connection and then receives a fresh 500 ms attempt timeout, the client silently expanded its budget. Record pool wait separately when the transport exposes it; rising wait time is often the earliest sign that local concurrency or downstream capacity is saturated.

Respect platform DNS behavior instead of pinning addresses forever or resolving on every attempt. A retry may establish a fresh connection after a known unusable one, but it must not create a reconnect storm.

Spend One Deadline Across the Whole Operation #

A deadline is an absolute latest completion time covering pool wait, connection setup, upload, server work, download, parsing, and retry delay. Every phase consumes the same finite budget.

If an incoming request already has a deadline, the outbound client should receive the remaining budget rather than invent a new one. Reserve time for local response handling and for the caller to produce its own result. For a background operation without an upstream deadline, choose an operation-specific maximum at the capability boundary.

Let $D$ be the absolute deadline and $t$ the current monotonic time. The remaining budget is

Before starting attempt $i$, require enough budget for its minimum useful execution and any deliberate delay. An attempt timeout can be bounded as

If that value is non-positive, fail with a deadline error instead of launching work that cannot produce a usable answer. Use a monotonic clock for elapsed durations; wall-clock adjustments must not grant extra execution time or cause premature expiry.

Cancellation should propagate through upload, response streaming, and retry sleep. Aborting locally does not prove the server stopped; it may have committed the request before the connection closed. Retry safety therefore depends on operation semantics, not an abort exception. The capability client owns attempt count so nested controller, helper, and transport retries cannot multiply work beyond the deadline.

Classify Before Retrying #

Retries are appropriate only when the operation is safe to repeat, the failure is plausibly transient, and enough deadline remains. All three conditions are required. Retrying every non-2xx response turns overload into more load and can duplicate side effects.

HTTP methods offer hints, not proof: endpoints can violate their intended semantics. A POST is repeatable only when the server implements an idempotency key with defined scope, canonical request, retention, and stored outcome. Policy should identify operations rather than infer safety from the verb alone.

A practical classification looks like this:

Use capped exponential backoff with jitter so clients do not synchronize after an outage. Bound attempts and Retry-After by local policy and the remaining deadline. A fleet-level retry budget can protect a recovering dependency from repeated traffic.

Return the classified final cause with attempt context, not a vague “request failed after retries.” Callers should distinguish deadline exhaustion, rejection, unavailability, malformed responses, and cancellation without parsing prose.

Work Through a Bounded Inventory Client #

Consider an inventory service with two operations. Reading a stock record is safe to retry. Reserving stock uses POST, but the server accepts an idempotency key and returns the original result when the same key and canonical request are repeated. Both operations share one executor while declaring different policies.

This TypeScript sketch injects a process-long transport, an abort-aware sleep, and a telemetry callback. Pool construction is omitted, but its connection and idle limits must be finite. readBoundedJson streams and counts decoded bytes, cancels above the cap, then parses and schema-validates JSON.

type AttemptPolicy = Readonly<{
  operation: string;
  maxAttempts: number;
  repeatable: boolean;
  responseLimitBytes: number;
}>;

type FailureKind =
  | 'cancelled'
  | 'deadline'
  | 'transport'
  | 'rejected'
  | 'unavailable'
  | 'response-too-large'
  | 'invalid-response';

class ClientFailure extends Error {
  constructor(
    readonly kind: FailureKind,
    message: string,
    readonly status?: number,
  ) {
    super(message);
  }
}

const retryableStatuses = new Set([408, 425, 429, 500, 502, 503, 504]);
async function executeJson(input: Readonly<{
  fetch: typeof globalThis.fetch;
  sleep: (ms: number, signal: AbortSignal) => Promise<void>;
  recordAttempt: (event: Record<string, unknown>) => void;
  url: string;
  init: RequestInit;
  policy: AttemptPolicy;
  deadlineMs: number;
  callerSignal: AbortSignal;
}>): Promise<unknown> {
  for (let attempt = 1; attempt <= input.policy.maxAttempts; attempt += 1) {
    const budget = input.deadlineMs - performance.now();
    if (budget <= 25) throw new ClientFailure('deadline', 'deadline exhausted');

    const started = performance.now();
    const signal = AbortSignal.any([
      input.callerSignal,
      AbortSignal.timeout(budget - 25),
    ]);
    let status: number | undefined;

    try {
      const response = await input.fetch(input.url, {
        ...input.init,
        signal,
        redirect: 'error',
      });
      status = response.status;
      if (response.ok) {
        const result = await readBoundedJson(
          response,
          input.policy.responseLimitBytes,
        );
        input.recordAttempt({
          operation: input.policy.operation,
          attempt,
          durationMs: performance.now() - started,
          outcome: 'success',
          status,
        });
        return result;
      }

      const kind = retryableStatuses.has(status) ? 'unavailable' : 'rejected';
      throw new ClientFailure(kind, `dependency returned ${status}`, status);
    } catch (error) {
      const failure = error instanceof ClientFailure
        ? error
        : new ClientFailure(signal.aborted ? 'cancelled' : 'transport', 'request failed');
      input.recordAttempt({
        operation: input.policy.operation,
        attempt,
        durationMs: performance.now() - started,
        outcome: failure.kind,
        status,
      });

      const canRetry = input.policy.repeatable
        && attempt < input.policy.maxAttempts
        && (failure.kind === 'transport' || failure.kind === 'unavailable');
      if (!canRetry) throw failure;

      const backoff = Math.min(400, 40 * 2 ** (attempt - 1));
      const jittered = Math.floor(Math.random() * backoff);
      if (input.deadlineMs - performance.now() <= jittered + 25) {
        throw new ClientFailure('deadline', 'no budget for another attempt');
      }
      await input.sleep(jittered, input.callerSignal);
    }
  }
  throw new ClientFailure('transport', 'attempt policy exhausted');
}

The success path records telemetry only after bounded decoding; the catch path records all other attempt outcomes. Error bodies need a smaller independent cap. For reserve, create one idempotency key and stable serialized body outside executeJson, then reuse both across attempts. The server must reject the same key with different content. getStock can declare repeatable: true; an unsafe operation must use one attempt.

Bound Responses and Redirects #

The peer controls response headers, body bytes, compression ratio, content type, and timing. Treat all of them as untrusted. A Content-Length larger than policy can be rejected early, but its absence or presence is not sufficient: stream and count decoded bytes because transfer encoding and compression can obscure the final size.

Set separate limits for successful and error representations. Error pages from a proxy can be unexpectedly large or HTML rather than JSON. Read only enough to extract an allow-listed diagnostic, then discard the rest so error handling cannot become the memory incident.

Validate Content-Type before schema decoding, reject unsupported character encodings, and use strict runtime schemas with bounds on arrays and strings. A syntactically valid JSON document can still allocate excessive objects or violate domain invariants. Validation errors should include a safe contract identifier, not the entire body.

Redirects require policy because they can change both destination and method semantics. A capability client normally knows its origin and should reject redirects unless the API explicitly documents them. Blindly following a redirect can send credentials to another host, bypass an egress allow-list, or transform a request in surprising ways. Likewise, construct paths with URL APIs and validate identifiers instead of concatenating attacker-controlled absolute URLs.

Request uploads need bounds too. Streaming a large body avoids buffering but does not make it free; cap bytes, ensure cancellation stops the producer, and decide whether a partially transmitted body is repeatable. A one-shot stream cannot simply be retried unless it can be recreated from stable input.

Integrate Circuits Without Hiding Their State #

A circuit breaker can use the client’s classified outcomes to avoid calls that are unlikely to succeed. The integration point should be explicit: check permission before the first attempt, report each logical operation’s final outcome, and expose a distinct circuit-open result to the caller. Do not disguise an open circuit as a network timeout.

The breaker usually belongs to dependency policy above the low-level executor because it needs service-level knowledge: which operations share a failure domain, which errors count, how probes are coordinated, and what degradation is available. Counting every retry as an independent business failure can open a circuit too aggressively; counting only final outcomes can hide attempt load. Record both, but define breaker transitions against one documented unit.

Half-open probes must remain bounded and should not allow every process to flood a recovering service simultaneously. Authentication failures, invalid requests, oversized bodies, and schema violations normally should not trip availability circuits because repetition will not repair them. A 429 may indicate client-specific quota rather than service failure, so grouping keys matter.

This division keeps the client composable. It emits typed facts and accepts a gate; it does not become a complete tutorial on bulkheads, global retry budgets, or fleet-level outage control. Those policies coordinate many clients and belong at a broader system boundary.

Verify Behavior and Operate the Client #

Client tests should control time and transport rather than rely on a live flaky endpoint. A scripted fake transport can return a reset, 503, delayed body chunk, malformed JSON, or successful response in a deterministic sequence. Use a fake monotonic clock and injected randomness so deadline and backoff assertions do not sleep in real time.

At minimum, verify these invariants:

• One logical idempotency key is reused across all attempts, while attempt identifiers remain distinct.
• A non-repeatable operation makes one attempt after ambiguous transport failure.
• Pool wait, backoff, streaming, and parsing cannot exceed the absolute deadline.
• Caller cancellation interrupts transport and retry sleep without starting another attempt.
• Retry-After is parsed conservatively and cannot exceed local caps.
• Bodies over the decoded-byte limit are cancelled and never fully buffered.
• Redirects, unexpected content types, invalid schemas, and ordinary 4xx responses are not retried.
• Telemetry records an outcome even when decoding or cancellation fails.

Add integration tests with a local controllable server for socket reuse, truncated responses, slow headers, slow body chunks, connection resets, and graceful shutdown. Property-based tests are useful for response chunk boundaries and retry schedules: total attempts must never exceed policy, and simulated time must never pass the deadline before another attempt starts.

In operation, emit one span for the logical call and child spans or events for attempts. Record dependency name, operation, status class, failure kind, protocol, attempt count, pool wait if available, request and bounded response sizes, and durations by phase. Avoid raw URLs with identifiers, headers, query strings, bodies, and idempotency keys. Keep metrics low-cardinality; trace IDs provide detail without turning every customer value into a label.

Dashboards should separate first-attempt success from retry-assisted success. A stable overall success rate can conceal a dependency that now requires two attempts for every request, doubling load and consuming latency budget. Watch deadline exhaustion, cancellation, retry volume, connection creation versus reuse, pool queue time, response-limit rejections, and schema failures. Alert on user impact and sustained changes, not individual transient resets.

Operational changes deserve controlled rollout. Tightening a body cap can break a legitimate large response; increasing attempts can amplify outages; changing pool limits can shift queueing downstream. Canary configuration by dependency and preserve a fast rollback. The strongest client is not the one that retries most. It is the one whose finite resource and semantic choices remain visible under failure.

The final boundary is simple to state: reuse transport resources, spend one deadline, repeat only proven-safe operations, constrain every byte received, and report classified outcomes. With those invariants centralized, capability code becomes easier to read and system-level resilience mechanisms receive facts they can safely act on.