Asynchronous APIs for Long-Running Operations

Some requests cannot finish within a useful HTTP exchange. A data export may scan millions of records, a media conversion may take minutes, and an infrastructure change may wait for several control loops. Keeping the connection open couples completion to proxy timeouts, client connectivity, and one server process. Returning success immediately is worse because the caller cannot tell whether work was merely received or actually completed.

An asynchronous API should turn the work into a resource. The submission request creates or identifies a durable operation resource; later requests observe, cancel, or retrieve its outcome. 202 Accepted then has a precise meaning: the server accepted responsibility for attempting the operation, not that the requested business change succeeded.

The central thesis is that a long-running operation is a public state machine, not a background implementation detail. Its identity, transitions, terminal outcome, authorization, and retention are part of the API contract. Queue and worker internals can change without forcing clients to change, provided that resource remains coherent.

Give Accepted Work a Durable Identity #

The operation resource closes the information gap between submission and completion. Without it, a caller that loses its connection after receiving no response cannot distinguish rejection, acceptance, and completion. With a stable operation identifier, it can resume observation from another process or device.

A submission commonly returns:

HTTP/1.1 202 Accepted
Location: /v1/operations/op_01J8M7A6Q9
Retry-After: 3
Content-Type: application/json

{
  "id": "op_01J8M7A6Q9",
  "kind": "customer-export",
  "state": "queued",
  "createdAt": "2026-04-19T10:00:00Z",
  "links": {
    "self": "/v1/operations/op_01J8M7A6Q9",
    "cancel": "/v1/operations/op_01J8M7A6Q9/cancellation"
  }
}

Location names the canonical observation endpoint. Retry-After is advice, not a lease or completion promise. The body lets clients begin without reconstructing URLs. A correlation or trace identifier may also be returned, but it must not replace the operation ID: tracing data is an observability aid, while the operation is durable product state.

Persist the operation before returning 202. If the API acknowledges first and only then attempts to enqueue volatile work, a crash can create an operation the caller believes exists but the system cannot find. The persistence and dispatch mechanism may use a transactional outbox or another durable handoff, but that is an implementation choice beneath the resource contract. The externally visible invariant is that every accepted operation can be queried and will eventually reach a terminal state or an explicitly detectable operational failure.

Not every delayed response deserves 202. If work normally completes quickly and only rare instances are slow, the API can still choose asynchronous semantics consistently, or offer an explicit Prefer: respond-async style option. Do not unpredictably alternate between a final domain response and an operation shape without documenting both paths.

Define a Small, Monotone State Machine #

Clients should not infer state from progress text. Define a finite set with legal transitions. A useful baseline is queued, running, succeeded, failed, and cancelled. Some domains need cancel_requested, pausing, or waiting_for_input, but every additional state expands client logic and compatibility obligations.

Transitions should be monotone: once terminal, an operation does not return to running. If automatic retry occurs internally, expose it as attempt metadata while the public state remains running, or define a documented nonterminal state. Reusing a failed operation record for a new execution destroys its audit history; create a new operation linked to the previous one instead.

Terminal states need stable meanings:

Record timestamps as server facts, not client estimates. startedAt can remain absent while queued; completedAt appears only in terminal states. Include an opaque version or entity tag so clients and support tools can identify which representation they observed. The operation type and API version determine which result and metadata fields are legal.

Be honest about liveness. “Eventually terminal” requires reconciliation that finds abandoned queued or running records after worker crashes. A stale-operation monitor can re-dispatch safe work or mark an internal failure according to policy. Leaving an operation eternally at 63 percent is a contract failure even if the queue implementation considers the job lost.

Make Submission Idempotent #

Clients retry submissions when connections fail. If each retry creates another export, charge, or infrastructure change, asynchronous acceptance magnifies ambiguity. The creation endpoint should support an idempotency key scoped to the authenticated principal and operation kind.

For the same key and canonically equivalent request, return the same operation resource, whether it is queued, running, or terminal. For the same key with different input, reject the request with a conflict rather than silently attaching the caller to unrelated work. Store the key, a canonical request fingerprint, the operation ID, and enough response metadata atomically with acceptance.

Idempotency has a retention contract. If keys expire after 24 hours but operations remain visible for 30 days, a retry after 25 hours can create new work. Document key lifetime and make it at least as long as clients are expected to retry submission. Never derive authorization from possession of the key; it may be logged by callers and must not act as a bearer credential.

Distinguish three outcomes at creation:

• 202 Accepted: a new or previously accepted operation is not yet represented as the final domain result.
• 200 OK or 303 See Other: policy may direct a duplicate request to an already completed operation or result, if documented consistently.
• A synchronous 4xx: validation or authorization failed before acceptance, so no operation was created.

Validate all cheap, deterministic preconditions before accepting. Required fields, authorization, request size, and obviously invalid ranges should fail synchronously. Conditions that can change during execution, such as resource availability, may still produce a terminal operation failure. The contract should state which checks happen at each boundary so clients know whether to inspect the submission response or operation error.

Avoid using a user-selected business name as the operation ID. Generate an unguessable or access-controlled identifier, and store any business deduplication key separately. Two legitimate exports with identical parameters may be distinct operations even if their computation can share an internal result.

Work Through an Export Operation #

Suppose an administrator requests a customer export for a date range. The resulting file belongs in a separate exports resource; the operation describes the attempt to create it. This separation keeps the operation small and allows result retention to differ from audit retention.

POST /v1/customer-exports HTTP/1.1
Idempotency-Key: tenant-42-export-2026-04-19-a
Content-Type: application/json

{
  "from": "2026-01-01",
  "through": "2026-03-31",
  "format": "csv"
}

While active, polling returns a representation such as:

{
  "id": "op_01J8M7A6Q9",
  "kind": "customer-export",
  "state": "running",
  "createdAt": "2026-04-19T10:00:00Z",
  "startedAt": "2026-04-19T10:00:02Z",
  "progress": {
    "completedUnits": 42000,
    "totalUnits": 100000,
    "unit": "customers",
    "messageCode": "WRITING_EXPORT"
  },
  "links": {
    "self": "/v1/operations/op_01J8M7A6Q9",
    "cancel": "/v1/operations/op_01J8M7A6Q9/cancellation"
  }
}

On success, the operation points to the result instead of embedding a large file or expiring download credential:

{
  "id": "op_01J8M7A6Q9",
  "kind": "customer-export",
  "state": "succeeded",
  "completedAt": "2026-04-19T10:04:12Z",
  "result": {
    "resourceType": "customer-export",
    "resourceId": "exp_01J8M7DK2P"
  },
  "links": {
    "self": "/v1/operations/op_01J8M7A6Q9",
    "result": "/v1/customer-exports/exp_01J8M7DK2P"
  }
}

The result endpoint can authorize the download and mint a short-lived storage URL when requested. Persisting a signed URL in the operation would expose a credential and guarantee that old representations become stale.

On failure, return a stable machine-readable error with an operation-specific code, safe detail, and retry guidance. HTTP status for GET /operations/{id} can remain 200 because retrieval succeeded and the representation says the work failed. Returning 500 for every poll of a failed operation confuses resource state with transport failure. If the operation itself has expired, 410 Gone is more informative than pretending it never existed, provided the retention policy promised prior visibility.

Report Progress Without Making False Promises #

Progress is optional. Inaccurate precision is worse than an honest indeterminate state. A percent is meaningful only when the denominator is known and work units have reasonably stable cost. Counting rows can be useful for an export, while “73 percent deployed” may be fiction if the remaining reconciliation step has unbounded duration.

Prefer structured progress: completed units, total units when known, unit name, current phase code, and last update time. Keep display text localized by the client using stable phase codes; server-authored English prose is hard to translate and unsafe to parse. Progress should never decrease unless the contract explicitly models discovered work, and clients must not treat 100 percent as success before the state becomes succeeded.

Polling guidance protects both parties. Return Retry-After or a documented minimum interval, support conditional requests with ETag, and answer 304 Not Modified when nothing changed. Clients should apply jitter and back off for long operations. The server may rate-limit abusive polling, but the normal contract should make polite behavior easy.

Long polling or server-sent events can reduce empty polls, yet the operation resource remains authoritative. Streams disconnect, browser tabs sleep, and intermediaries time out. An event should carry the operation ID and version so a reconnecting client can fetch current state rather than reconstruct it from missed events.

Do not expose internal queue position as a promise unless the scheduler can maintain its meaning. Priority changes, retries, and heterogeneous task durations make “third in line” unstable. A coarse queued state and optional nonbinding estimate are more honest. If estimates are provided, label them as estimates and never make client correctness depend on them.

Model Cancellation as a Request and a Race #

Cancellation is not deletion. Deleting the operation record erases the only place where the caller can discover whether work stopped. Model cancellation as an action or subresource, for example PUT /operations/{id}/cancellation, and make repeated requests idempotent.

The server can synchronously acknowledge that cancellation was requested, but it often cannot promise immediate cancellation. A worker may be between checkpoints, an external system may not support abort, or the operation may complete concurrently. That is why cancel_requested can transition to either cancelled or succeeded. The API must not report cancelled until it has established the domain’s cancellation condition.

Define that condition per operation type. For an export, cancellation may mean no downloadable result will be published and temporary objects will be removed. For an infrastructure change, compensating already-applied steps may be impossible or unsafe; the operation might reject cancellation after a commit point. Return a conflict with a stable code such as OPERATION_NOT_CANCELLABLE rather than accepting a request the system will ignore.

Workers need cooperative checkpoints and cleanup, but the API should not expose worker process details. Persist the cancellation request durably, authorize it like the original operation, and record who requested it. A client losing the cancellation response can safely repeat the same action and then poll.

Cancellation has side-effect semantics. If the requested business outcome partially occurred, cancelled must not imply a rollback unless the contract guarantees one. Include a result or remediation link when partial artifacts remain. This is especially important for operations that orchestrate external systems whose completion cannot be recalled.

Use Callbacks as Hints, Not the Source of Truth #

Some clients cannot poll efficiently and request a callback. Treat that callback as a notification that the operation changed, not as the sole copy of its result. Delivery can be delayed, duplicated, reordered, or lost; the recipient should authenticate the event and fetch the operation resource by ID.

Accept callback configuration only from authorized clients, validate destinations against server-side policy, and prevent requests to internal network addresses. Sign notifications, include an event ID, operation ID, operation version, and event type, and retry delivery under a bounded policy. Those details are part of a webhook delivery contract, but the long-running API need not expose its internal queue or block operation completion on callback success.

Operation success and notification delivery are separate states. A completed export stays succeeded even when its callback endpoint is unavailable. Expose notification attempts in an appropriate delivery log or support channel rather than changing the business operation to failed. Conversely, receiving a callback does not grant access to the result; the subsequent authenticated fetch still enforces tenant and resource authorization.

If clients register a callback at submission, bind it to the idempotent request. Reusing the same idempotency key with a different destination should conflict unless the API explicitly supports updating notification preferences. Otherwise a replay could redirect sensitive completion signals.

Events and polling can coexist cleanly: callbacks reduce latency and request volume, while polling repairs missed delivery. This redundancy is useful because both mechanisms converge on one durable representation instead of maintaining competing truths.

Secure, Retain, and Expire Operations #

Operation IDs often appear in logs, URLs, and notifications, so they must not confer authorization. Check the authenticated principal and tenant on every read, cancellation, result fetch, and callback configuration change. Avoid sequential guessable identifiers unless access controls and rate limits make enumeration harmless; opaque IDs provide useful defense in depth.

Minimize stored request data. An operation may need a request fingerprint and safe summary, not a permanent copy of secrets or an enormous payload. Redact error details before exposing them to clients. Keep internal diagnostics linked by an observability ID, with access controlled separately from the public representation.

Define retention for at least four things: operation metadata, idempotency records, result resources, and notifications. They have different purposes. A downloadable export might expire after hours, the operation audit after weeks, and a legal record after a domain-specific period. Publish expiresAt fields where clients need them and make expiry behavior deterministic.

When a result expires before the operation, retain a succeeded operation whose result link reports expiry; do not rewrite history to failed. When operation metadata expires, 410 Gone can communicate known deletion during a documented tombstone window. Eventually returning 404 may be reasonable after the tombstone itself expires. Ensure idempotency retention does not permit an unexpected duplicate while clients can still reasonably replay a submission.

Deletion must cover temporary artifacts, callback data, and indexes, not only the primary operation row. Retention workers should be idempotent and observable. A tenant deletion workflow needs to find operations and results even if they are stored in separate systems.

Test Transitions and Operate for Stuck Work #

Tests should generate legal and illegal transition sequences against the state machine. Assert that terminal states are immutable, timestamps appear in order, result and error are mutually exclusive, cancellation races end in documented states, and duplicate submissions resolve to one operation. Property-based state-machine tests are especially effective because they explore repeated polling, retry, cancellation, and expiry in combinations hand-written examples miss.

Integration tests should interrupt the handoff after operation persistence, terminate a worker during execution, repeat a submission after an ambiguous response, deliver callbacks out of order, and expire results independently from metadata. Use a controllable clock. Verify that reconciliation finds abandoned operations and that no accepted record remains nonterminal forever without an alert.

Operational metrics should include acceptance rate, queue-to-start delay, execution duration by operation kind, age of the oldest nonterminal operation, state transition counts, reconciliation actions, cancellation latency, callback delivery outcomes, and retention backlog. Cardinality belongs in traces or queryable logs, not metric labels containing operation IDs.

Alert on violated invariants: operations with no dispatch record, running without a heartbeat beyond policy, terminal records missing a result or error, progress versions moving backward, and cleanup that cannot keep pace. Runbooks should distinguish a slow dependency from lost dispatch, stuck workers, and a broken observer path. Pausing new acceptance may be safer than accumulating work whose completion objective can no longer be met.

Compatibility tests protect clients as the representation evolves. Add optional fields; do not rename states casually or change a terminal meaning. Clients should ignore unknown fields, but they cannot safely ignore an unknown state unless the contract provides a fallback category such as terminal: false. Version new operation kinds or media types when semantics truly change.

The tradeoff is additional durable state and lifecycle work. For genuinely long operations, that complexity already exists; hiding it behind a held connection merely makes it unobservable. A well-designed operation resource gives acceptance, execution, cancellation, notification, result, and expiry distinct meanings. Clients can disconnect and recover, operators can find stuck work, and implementations can evolve without turning a queue identifier into the public API.