Designing Idempotent and Versioned APIs

A public API outlives the code that first implements it. Mobile applications remain installed for years, partner integrations deploy on their own schedules, proxies retry requests, and a timeout can hide a successful write. Changing a handler is easy; changing every caller at the same instant is usually impossible.

Good API design therefore starts with two promises. Repeating a request should have a predictable effect, and deploying a new representation should not silently break an old consumer. Idempotency governs repeated intent. Versioning governs change over time. They meet at the HTTP contract: methods, validators, status codes, headers, schemas, and documented failure behavior.

This is not a claim that a network can provide exactly-once execution. The narrower goal is to make the public interface safe under retries and gradual evolution, while allowing the implementation behind it to change.

Start with HTTP method semantics #

HTTP already supplies a vocabulary for intent. GET, HEAD, PUT, DELETE, OPTIONS, and TRACE are defined as idempotent: sending the same request multiple times should have the same intended effect as sending it once. POST and PATCH are not idempotent by definition, although an API can add conditions that make a particular operation retry-safe.

Safety and idempotency are different. A safe method is read-only from the caller’s perspective. An idempotent method may change state on the first request; repetitions do not introduce additional intended changes. Logging a second DELETE, updating metrics, or refreshing a cache does not violate that semantic promise because the requested resource state still converges.

Choose the method from resource semantics, not from the convenience of a framework route. If a client can name the final resource and submit its full desired state, PUT /imports/{clientImportId} is often simpler than POST /imports. If the server assigns the identity, POST is appropriate, but creation needs a separate idempotency contract.

Patch documents also matter. JSON Merge Patch sets fields to desired values and is often idempotent for an identical document. A JSON Patch operation such as replace is idempotent, while add to an array or a domain command such as increment may not be. Do not label all PATCH requests retry-safe merely because they share a verb.

Give non-idempotent operations a retry contract #

For creation and commands, accept an Idempotency-Key generated once per user intent and reused across network attempts. Scope it by authenticated principal and operation, bind it to the validated request, and retain the completed result for a documented period. A key reused with a different payload is a conflict, not a new command.

The handler should expose protocol behavior clearly even if storage is delegated to a repository. The following example validates the boundary, derives a stable operation scope, and maps repository outcomes to HTTP responses:

type CreateOrder = Readonly<{
  customerId: string;
  currency: 'USD' | 'EUR' | 'VND';
  lines: ReadonlyArray<{ sku: string; quantity: number }>;
}>;

type HttpResponse = Readonly<{
  status: number;
  headers?: Readonly<Record<string, string>>;
  body?: unknown;
}>;

const createOrderSchema = {
  type: 'object',
  additionalProperties: false,
  required: ['customerId', 'currency', 'lines'],
  properties: {
    customerId: { type: 'string', minLength: 1 },
    currency: { enum: ['USD', 'EUR', 'VND'] },
    lines: {
      type: 'array',
      minItems: 1,
      items: {
        type: 'object',
        additionalProperties: false,
        required: ['sku', 'quantity'],
        properties: {
          sku: { type: 'string', minLength: 1 },
          quantity: { type: 'integer', minimum: 1 },
        },
      },
    },
  },
} as const;

export async function postOrder(request: RequestContext): Promise<HttpResponse> {
  const key = request.header('idempotency-key');
  if (!key || key.length > 128) {
    return problem(400, 'IDEMPOTENCY_KEY_REQUIRED', 'Supply a key up to 128 characters.');
  }

  const input = request.validate<CreateOrder>(createOrderSchema);
  const result = await request.services.operations.execute({
    scope: `${request.principal.accountId}:POST:/orders`,
    key,
    input,
    run: (transaction) => transaction.orders.create(input),
  });

  switch (result.kind) {
    case 'created':
    case 'replayed':
      return {
        status: 201,
        headers: {
          location: `/orders/${result.order.id}`,
          'idempotency-status': result.kind,
        },
        body: toOrderV2(result.order),
      };
    case 'payload-mismatch':
      return problem(409, 'IDEMPOTENCY_KEY_REUSED', 'The key belongs to another payload.');
    case 'in-progress':
      return {
        ...problem(409, 'OPERATION_IN_PROGRESS', 'The original request is still running.'),
        headers: { 'retry-after': '2' },
      };
  }
}

Validation must happen before canonical hashing so equivalent input gets one identity. Decide whether absent optional fields and explicit defaults are equivalent. Never include volatile headers such as tracing IDs in the payload fingerprint. Authorization, however, must run on every attempt: replaying a stored success must not let a caller bypass current access control.

Return the original status, selected headers, and representation when replaying. Do not turn a successful retry into 200 with an unrelated shape if the original contract promised 201. For long operations, 202 Accepted with an operation resource such as /operations/{id} is usually better than holding concurrent retries open.

Prevent lost updates with ETags and preconditions #

Idempotency keys protect repeated intent; they do not resolve two different writers racing to modify one resource. Use optimistic concurrency. Return an ETag with a representation, require If-Match on mutations, and reject a stale validator with 412 Precondition Failed.

A strong ETag may be derived from an immutable row version. It need not hash the whole JSON response, and it should not include formatting details that differ between encoders.

function orderEtag(version: bigint): string {
  return `"order-${version.toString()}"`;
}

export async function getOrder(request: RequestContext): Promise<HttpResponse> {
  const order = await request.services.orders.find(request.params.orderId);
  if (!order) return problem(404, 'ORDER_NOT_FOUND', 'Order does not exist.');

  return {
    status: 200,
    headers: { etag: orderEtag(order.version) },
    body: toOrderV2(order),
  };
}

export async function patchOrder(request: RequestContext): Promise<HttpResponse> {
  const ifMatch = request.header('if-match');
  if (!ifMatch) {
    return problem(428, 'PRECONDITION_REQUIRED', 'Send the ETag in If-Match.');
  }

  const patch = request.validate<{
    shippingAddress?: AddressInput;
    note?: string | null;
  }>(updateOrderSchema);

  const currentVersion = parseOrderEtag(ifMatch);
  const updated = await request.services.orders.updateIfVersion(
    request.params.orderId,
    currentVersion,
    patch,
  );

  if (updated.kind === 'missing') {
    return problem(404, 'ORDER_NOT_FOUND', 'Order does not exist.');
  }
  if (updated.kind === 'stale') {
    return problem(412, 'STALE_ORDER_VERSION', 'Reload the order and reconcile your change.');
  }

  return {
    status: 200,
    headers: { etag: orderEtag(updated.order.version) },
    body: toOrderV2(updated.order),
  };
}

The repository must make the comparison and update atomic, for example UPDATE orders SET ..., version = version + 1 WHERE id = ? AND version = ?. Reading the version and updating in separate unprotected statements reintroduces the lost-update race.

Use If-None-Match: * with PUT when a client wants create-only semantics at a known URI. Use If-Match: * when a mutation should proceed only if the resource exists. Reserve 409 Conflict for domain conflicts or idempotency-key misuse; a failed HTTP validator has the more precise 412 response.

Evolve schemas additively before introducing a version #

Versioning is not a substitute for compatibility discipline. Most API evolution should happen within a version: add optional response fields, accept old and new inputs during migration, preserve field meaning, and make consumers ignore unknown fields. Removing a field, changing its type, reinterpreting a value, changing units, or making an optional input required is breaking.

Keep a canonical domain model and place version adapters at the edge. Do not fork the entire business service for every public representation.

type OrderV1 = Readonly<{
  id: string;
  total: number;
  status: 'open' | 'paid';
}>;

type OrderV2 = Readonly<{
  id: string;
  amount: { currency: string; minor: number };
  status: 'pending' | 'paid' | 'cancelled';
  links: { self: string };
}>;

export function toOrderV1(order: Order): OrderV1 {
  return {
    id: order.id,
    total: order.totalMinor / 100,
    status: order.status === 'paid' ? 'paid' : 'open',
  };
}

export function toOrderV2(order: Order): OrderV2 {
  return {
    id: order.id,
    amount: { currency: order.currency, minor: order.totalMinor },
    status: order.status,
    links: { self: `/orders/${order.id}` },
  };
}

Storage changes need the same expand-migrate-contract rhythm. Add the new column without removing the old one, deploy code that can read both and writes both, backfill in bounded batches, switch reads, verify parity, and only then remove the old column in a later deployment.

ALTER TABLE orders ADD COLUMN total_minor BIGINT;

UPDATE orders
SET total_minor = ROUND(total_amount * 100)
WHERE id > :after_id
  AND id <= :batch_end
  AND total_minor IS NULL;

ALTER TABLE orders ALTER COLUMN total_minor SET NOT NULL;
-- Drop total_amount only after every deployed reader uses total_minor.

This deployment sequence matters even when only v2 exposes total_minor: old application instances and background jobs may still read the old storage shape while a rolling deployment is in progress.

Choose one explicit version negotiation strategy #

Introduce a new major version only when a useful contract cannot remain compatible. URI, header, and media-type versioning can all work; consistency and operational visibility matter more than ideology.

A version-selection middleware should reject unsupported values rather than silently falling back. Silent fallback can parse a request under the wrong schema and create valid but unintended state.

type ApiVersion = '1' | '2';

export function selectApiVersion(request: RequestContext): ApiVersion {
  const fromPath = request.params.apiVersion;
  const fromHeader = request.header('api-version');

  if (fromPath && fromHeader && fromPath !== fromHeader) {
    throw new HttpProblem(400, 'VERSION_CONFLICT', 'Path and header versions differ.');
  }

  const selected = fromHeader ?? fromPath ?? '2';
  if (selected !== '1' && selected !== '2') {
    throw new HttpProblem(406, 'VERSION_UNSUPPORTED', `Version ${selected} is unavailable.`);
  }

  return selected;
}

export async function orderHandler(request: RequestContext): Promise<HttpResponse> {
  const version = selectApiVersion(request);
  const order = await request.services.orders.require(request.params.orderId);
  return {
    status: 200,
    headers: { vary: 'Api-Version' },
    body: version === '1' ? toOrderV1(order) : toOrderV2(order),
  };
}

Version request and response schemas together. Idempotency records should also remember the negotiated version because the same logical result may have different representations. Prefer requiring a new idempotency key when the caller intentionally changes major version; otherwise define whether replay returns the originally stored representation or re-renders current domain state.

Deprecate with evidence and test the whole contract #

Deprecation is a migration program, not a banner in documentation. Announce a replacement, publish a last-supported date, emit machine-readable Deprecation and Sunset headers, link to a migration guide, and identify consumers by authenticated client identity. Contact active owners and measure remaining traffic before shutdown.

During migration, keep behavior deterministic. A v1 request should not sometimes receive a v2 body based on an account flag. Shadow traffic can compare old and new implementations, but it must suppress duplicate side effects. Telemetry should record version, route template, status, error code, retry/replay state, ETag failures, client identity, and deprecation warning delivery without logging secrets or full idempotency keys.

Contract tests should pin more than happy-path JSON. Verify required and optional fields, unknown-field policy, content type, status codes, Location, ETags, idempotent replay, payload mismatch, stale updates, and errors. Run old consumer examples against the new server before deployment.

describe.each(['1', '2'] as const)('orders API v%s', (version) => {
  it('replays a timed-out create without creating a second order', async () => {
    const request = {
      method: 'POST',
      path: '/orders',
      headers: { 'api-version': version, 'idempotency-key': 'checkout-7f4' },
      body: validOrderInput,
    };

    const first = await api.inject(request);
    const retry = await api.inject(request);

    expect(first.statusCode).toBe(201);
    expect(retry.statusCode).toBe(201);
    expect(retry.json()).toEqual(first.json());
    expect(await orders.count()).toBe(1);
  });

  it('rejects a stale conditional update', async () => {
    const fetched = await api.inject({
      method: 'GET',
      path: '/orders/ord_42',
      headers: { 'api-version': version },
    });

    await changeOrderDirectly('ord_42');
    const response = await api.inject({
      method: 'PATCH',
      path: '/orders/ord_42',
      headers: { 'api-version': version, 'if-match': fetched.headers.etag! },
      body: { note: 'Leave at reception' },
    });

    expect(response.statusCode).toBe(412);
    expect(response.json()).toMatchObject({ code: 'STALE_ORDER_VERSION' });
  });
});

Use a stable problem shape such as { code, message, details?, traceId }. Clients should branch on code, never parse prose. Classify retries explicitly: 400, 401, 403, 404, 409 payload mismatch, and 412 are normally terminal until the request changes; 408, 429, 502, 503, and 504 may be retried with capped exponential backoff, jitter, the same idempotency key, and respect for Retry-After. A generic 500 is ambiguous for a mutation and therefore requires the same-key retry or a status lookup, not blind recreation.

Takeaways #

• Model resources and choose HTTP methods by semantics. PUT and DELETE naturally converge; POST and some PATCH operations need an explicit retry contract.
• Treat an idempotency key as public operation identity: scope it, bind it to validated input, preserve the original response, and reuse it after ambiguous failures.
• Use ETags and If-Match for competing writers. Return 428 when a precondition is required and 412 when it is stale.
• Prefer additive schema evolution and tolerant consumers. Put public-version adapters around one canonical domain model and migrate storage with expand-migrate-contract deployments.
• Choose URI, header, or media-type versioning deliberately, reject unsupported negotiation, and include the selected version in caching, observability, and replay rules.
• Deprecate using dates, headers, migration guidance, client-level usage data, and contract tests. A version is removable only when real consumers have moved.
• Make retry and error behavior part of the API contract. Stable error codes, Retry-After, bounded backoff, and outcome lookup turn failure from guesswork into protocol.