The Saga Pattern for Distributed Transactions

The payment service says a charge succeeded. The inventory service never answers. The order API times out, so the client submits again. Ten minutes later, two messages arrive out of order and a support agent asks a deceptively simple question: did this order happen?

Inside one database, a transaction gives that question a crisp answer. Across independently deployed services and independently owned databases, there is no single commit point unless every participant accepts a distributed transaction protocol. Most cloud-native systems deliberately avoid that coupling. They still need a way to carry a business operation through partial failure without pretending that the network is reliable.

A saga is that way. It is a durable sequence of local transactions. Each step commits in its own service and publishes enough outcome for the workflow to continue. If a later step cannot complete, the saga runs compensating actions for earlier steps. This trades isolation and automatic rollback for explicit business semantics, recoverable progress, and operational visibility. The trade is worthwhile only when it is designed rather than hidden behind a chain of HTTP calls.

A saga is a protocol, not a long database transaction #

Consider placing an order: Orders creates a pending order, Inventory reserves stock, Payments authorizes money, and Orders confirms the purchase. Each service owns its data and commits locally. No database lock spans the workflow, which may last seconds, hours, or even days.

The diagram shows a failure path, but the happy path follows the same protocol: after authorization, the orchestrator confirms the order and marks the saga complete. Every arrow represents a durable fact or an idempotent command, not an ephemeral callback that disappears when a process restarts.

Sagas are commonly coordinated in two ways. In choreography, services react to events and publish the next events. In orchestration, one durable coordinator decides which command comes next. Two-phase commit (2PC) is different: a coordinator asks transactional participants to prepare, then tells all of them to commit or abort.

Choreography can be elegant when the flow is short: OrderPlaced leads to StockReserved, which leads to PaymentAuthorized. It becomes difficult when cancellation, deadlines, branching, and human intervention turn the event graph into an implicit state machine. Orchestration makes that state machine visible and is usually the safer default for a business-critical workflow. It does introduce a coordinator, but that coordinator owns policy, not participant data; Inventory and Payments remain autonomous.

Compensation moves the business forward #

A compensation is not a database rollback. Once Inventory commits a reservation, other transactions may observe it. Once Payments records an authorization, audit and fraud systems may consume that fact. History cannot be erased safely. A compensation performs a new local transaction whose business meaning counteracts an earlier one: release reserved stock, void an authorization, issue a refund, or create a correcting ledger entry.

That distinction changes the design. Compensation may be lossy: refunding a shipment does not unship it, and cancelling a hotel room may incur a fee. It may fail and require retries. It must also be idempotent because the orchestrator can crash after the participant succeeds but before the success is recorded. ReleaseReservation(reservationId) should return success when the reservation is already released, not subtract inventory twice.

Define the forward action, compensation, retry policy, and point of no return together:

Some workflows need semantic locks to prevent users or other workflows from treating intermediate data as final. An order in PENDING_PAYMENT cannot be shipped; reserved inventory is unavailable to competing orders; an account transfer may expose TRANSFER_PENDING instead of a misleading final balance. These are domain states enforced by business rules, not database mutexes. They remain meaningful across process restarts and make isolation anomalies explicit.

Ordering also matters. Compensations usually run in reverse order, but not mechanically. If payment authorization is uncertain because a request timed out, first query Payments using the operation key; blindly issuing a refund can create a second error. Some effects are pivot steps: before the pivot, failures can compensate; after it, the workflow must retry forward to completion or enter manual recovery.

Make the orchestrator a durable state machine #

An orchestrator should be restartable from persisted state. It does not hold a database transaction open while calling services. Instead, it loads one saga instance, performs one idempotent remote operation, records the outcome with optimistic concurrency, and schedules the next advance. A queue consumer, timer, or recovery scanner can call advance repeatedly.

The following TypeScript sketch coordinates inventory and payment. The ports hide transport details, while stable operation keys make every command safe to replay. Relative imports include .js because the source targets strict ESM.

// src/order-saga/order-saga.ts
import { VersionConflictError } from './errors.js';
import type {
  Clock,
  InventoryClient,
  OrderClient,
  PaymentClient,
  SagaScheduler,
  SagaStore,
} from './ports.js';

type SagaState =
  | 'RESERVING_INVENTORY'
  | 'AUTHORIZING_PAYMENT'
  | 'CONFIRMING_ORDER'
  | 'RELEASING_INVENTORY'
  | 'CANCELLING_ORDER'
  | 'COMPLETED'
  | 'CANCELLED'
  | 'MANUAL_REVIEW';

interface OrderSaga {
  id: string;
  orderId: string;
  customerId: string;
  items: ReadonlyArray<{ sku: string; quantity: number }>;
  amountCents: number;
  state: SagaState;
  version: number;
  attempts: number;
  deadline: string;
  reservationId?: string;
  paymentAuthorizationId?: string;
  failure?: string;
}

type StepResult =
  | { kind: 'advanced'; saga: OrderSaga }
  | { kind: 'retry'; reason: string }
  | { kind: 'idle' };

export class OrderSagaOrchestrator {
  constructor(
    private readonly store: SagaStore<OrderSaga>,
    private readonly inventory: InventoryClient,
    private readonly payments: PaymentClient,
    private readonly orders: OrderClient,
    private readonly scheduler: SagaScheduler,
    private readonly clock: Clock,
  ) {}

  async advance(sagaId: string): Promise<void> {
    const saga = await this.store.load(sagaId);
    if (!saga || this.isTerminal(saga.state)) return;

    if (this.clock.now() >= new Date(saga.deadline)) {
      await this.persist(saga, this.onDeadline(saga));
      return;
    }

    const result = await this.runStep(saga);
    if (result.kind === 'idle') return;

    if (result.kind === 'retry') {
      const attempts = saga.attempts + 1;
      if (attempts >= 8) {
        await this.persist(saga, {
          ...saga,
          state: 'MANUAL_REVIEW',
          attempts,
          failure: result.reason,
        });
        return;
      }

      await this.store.save(
        { ...saga, attempts, failure: result.reason },
        saga.version,
      );
      await this.scheduler.after(saga.id, this.backoff(attempts));
      return;
    }

    await this.persist(saga, { ...result.saga, attempts: 0 });
    if (!this.isTerminal(result.saga.state)) {
      await this.scheduler.now(saga.id);
    }
  }

  private async runStep(saga: OrderSaga): Promise<StepResult> {
    try {
      switch (saga.state) {
        case 'RESERVING_INVENTORY': {
          const result = await this.inventory.reserve({
            operationKey: `${saga.id}:reserve`,
            orderId: saga.orderId,
            items: saga.items,
          });
          if (!result.accepted) {
            return {
              kind: 'advanced',
              saga: { ...saga, state: 'CANCELLING_ORDER', failure: 'out_of_stock' },
            };
          }
          return {
            kind: 'advanced',
            saga: {
              ...saga,
              reservationId: result.reservationId,
              state: 'AUTHORIZING_PAYMENT',
            },
          };
        }

        case 'AUTHORIZING_PAYMENT': {
          const result = await this.payments.authorize({
            operationKey: `${saga.id}:authorize`,
            customerId: saga.customerId,
            amountCents: saga.amountCents,
          });
          if (!result.authorized) {
            return {
              kind: 'advanced',
              saga: { ...saga, state: 'RELEASING_INVENTORY', failure: 'payment_declined' },
            };
          }
          return {
            kind: 'advanced',
            saga: {
              ...saga,
              paymentAuthorizationId: result.authorizationId,
              state: 'CONFIRMING_ORDER',
            },
          };
        }

        case 'CONFIRMING_ORDER':
          await this.orders.confirm({
            operationKey: `${saga.id}:confirm`,
            orderId: saga.orderId,
            paymentAuthorizationId: saga.paymentAuthorizationId!,
          });
          return { kind: 'advanced', saga: { ...saga, state: 'COMPLETED' } };

        case 'RELEASING_INVENTORY':
          await this.inventory.release({
            operationKey: `${saga.id}:release`,
            reservationId: saga.reservationId!,
          });
          return { kind: 'advanced', saga: { ...saga, state: 'CANCELLING_ORDER' } };

        case 'CANCELLING_ORDER':
          await this.orders.cancel({
            operationKey: `${saga.id}:cancel`,
            orderId: saga.orderId,
            reason: saga.failure ?? 'saga_cancelled',
          });
          return { kind: 'advanced', saga: { ...saga, state: 'CANCELLED' } };

        default:
          return { kind: 'idle' };
      }
    } catch (error) {
      return {
        kind: 'retry',
        reason: error instanceof Error ? error.message : 'unknown_error',
      };
    }
  }

  private onDeadline(saga: OrderSaga): OrderSaga {
    if (saga.state === 'RESERVING_INVENTORY') {
      return { ...saga, state: 'CANCELLING_ORDER', failure: 'deadline_exceeded' };
    }
    if (saga.state === 'AUTHORIZING_PAYMENT' && saga.reservationId) {
      return { ...saga, state: 'RELEASING_INVENTORY', failure: 'deadline_exceeded' };
    }
    return { ...saga, state: 'MANUAL_REVIEW', failure: 'deadline_after_pivot' };
  }

  private async persist(previous: OrderSaga, next: OrderSaga): Promise<void> {
    try {
      await this.store.save(next, previous.version);
    } catch (error) {
      if (!(error instanceof VersionConflictError)) throw error;
      await this.scheduler.now(previous.id);
    }
  }

  private backoff(attempt: number): number {
    const exponential = Math.min(1_000 * 2 ** attempt, 60_000);
    return exponential + Math.floor(Math.random() * 500);
  }

  private isTerminal(state: SagaState): boolean {
    return state === 'COMPLETED' || state === 'CANCELLED' || state === 'MANUAL_REVIEW';
  }
}

This example intentionally advances one step at a time. If the process dies after reserve succeeds but before save, the next run sends the same ${saga.id}:reserve key and receives the original reservation. The participant should store the key and response atomically with its local transaction. An inbox table deduplicates incoming commands; an outbox table publishes the resulting event without a dual-write gap.

Optimistic concurrency prevents two workers from advancing the same saga version. It does not make remote effects atomic with coordinator state, so idempotency remains mandatory. Avoid holding worker threads while waiting for backoff: persist nextAttemptAt or schedule a delayed message instead.

Treat uncertainty as a first-class outcome #

Retries are correct only for transient failures and only when operations are idempotent. Use exponential backoff with jitter, cap the number of automatic attempts, and distinguish a rejection such as payment_declined from an unavailable dependency. Retrying a business rejection wastes capacity; abandoning a transient timeout too early creates unnecessary compensation.

A timeout means unknown, not failed. The request may have reached the participant even though its response did not return. Each participant should expose status by business identifier or operation key so recovery can ask, “What happened to authorization saga-42:authorize?” The answer can move the workflow forward without guessing. When status cannot be established, route the saga to manual review rather than issuing contradictory financial commands.

Deadlines express business policy: inventory may be held for fifteen minutes, while a travel booking may tolerate hours. Store the absolute deadline in saga state. When it expires, choose a transition based on completed effects and the pivot boundary, as the example does. A technical HTTP timeout is much shorter and only controls one attempt; it must not silently become the business deadline.

Recovery needs a durable sweeper that finds nonterminal sagas with stale heartbeats or due retries. Re-enqueueing them must be safe. Keep terminal records long enough for deduplication, audit, and support, then archive according to retention policy. A dead-letter queue is a transport safety net, not the workflow record; operators need the saga state and business context to decide what to do.

Observability, versioning, and tests are part of correctness #

Give every saga a globally unique sagaId and carry it in logs, traces, commands, events, and support screens alongside the business key such as orderId. Record state transitions as structured events with fromState, toState, step, attempt, duration, and sanitized failure class. Useful metrics include age of the oldest active saga, count by state, retries per step, deadline breaches, compensation rate, and manual-review backlog. Alert on stuck progress and age, not merely on individual HTTP errors.

Long-running instances survive deployments, so workflow definitions need versions. Persist definitionVersion when the saga starts. Keep handlers for active versions or write an explicit migration that maps old states and payloads to the new model. Renaming AUTHORIZING_PAYMENT in code without migrating stored rows can strand every in-flight order. Event and command schemas also need additive evolution: tolerate unknown fields, retain defaults, and do not repurpose established meanings.

Test the state machine as a transition system, not only through the happy-path API. Table-driven tests should cover every state and participant outcome. Then inject failures at each boundary: before the remote call, after the participant commits, before coordinator persistence, after persistence but before scheduling, and during compensation. Verify duplicate and out-of-order delivery, concurrent workers, expired deadlines, exhausted retries, old workflow versions, and restart from every persisted state. Property-based tests can assert invariants such as “a cancelled saga never retains an active reservation” and “a completed saga has exactly one payment authorization identity.”

In integration tests, use real persistence for inbox, outbox, and optimistic version checks. Fake clients are useful for deterministic transitions, but they will not reveal transaction-boundary mistakes. A small failure-injection suite often catches more than a large collection of end-to-end happy paths.

Know when not to use a saga #

Do not use a saga to excuse arbitrary service boundaries. If the operation requires strict invariants across data that always changes together, keep that data in one transactional boundary, perhaps inside a modular monolith. A database transaction is simpler, faster, and more isolated. Similarly, use 2PC only when the databases and infrastructure support it reliably, participants are controlled, transactions are short, and the availability trade is acceptable.

A saga is also a poor fit when no honest compensation or forward-recovery policy exists and partial visibility would violate a hard requirement. Financial ledgers often use append-only balancing entries, but some regulatory operations must be accepted atomically by a system of record. Physical actions may cross a point where software cannot undo them. In those cases, redesign the workflow around reservation, approval, escrow, or a human checkpoint before the irreversible step.

Do not build an orchestrator for a single fire-and-forget notification, and do not choose choreography merely to avoid naming the owner of a complex process. The coordination style should follow the business lifecycle and failure modes, not an architectural fashion.

Takeaways #

• A saga is a durable sequence of local commits, not an ACID transaction stretched across services.
• Compensations are new business actions. They must be explicit, idempotent, observable, and allowed to fail safely.
• Semantic locks represent intermediate truth with domain states instead of long-lived database locks.
• Persist the state machine, use stable operation keys, query uncertain outcomes, and separate attempt timeouts from business deadlines.
• Version workflows and schemas because deployed code changes while saga instances are still alive.
• Test every transition and crash boundary, then give operators audited recovery actions.
• Prefer one local transaction when the data and invariant naturally belong together. A saga is a deliberate cost paid for autonomy, not a default distributed-systems pattern.