Event-Driven Architecture Without the Hype

A checkout service publishes OrderPlaced, three teams subscribe, and suddenly the architecture diagram looks wonderfully decoupled. Six months later, nobody can explain why one customer received two emails, why another order never reached fulfillment, or which version of OrderPlaced a consumer expects. The broker is healthy. The system is not.

Event-driven architecture is useful, but it is not a synonym for modern, scalable, or loosely coupled. It exchanges temporal coupling for operational and semantic complexity. Producers no longer wait for consumers, yet every participant must agree on what a message means, what happens when it is repeated, and how to recover when processing stops halfway through.

The productive question is therefore not, “Should we use Kafka?” It is, “Which business facts deserve asynchronous distribution, and are we prepared to operate the resulting workflow?”

Events Are Facts, Commands Are Requests #

An event records something that has already happened: OrderPlaced, PaymentAuthorized, or ShipmentDispatched. Its name is normally past tense. A producer publishes the fact and does not prescribe who must react. Zero, one, or many consumers may use it.

A command asks a specific capability to do something: AuthorizePayment, ReserveInventory, or SendReceipt. It has an intended owner and can be accepted or rejected. Putting a command on a queue may be sensible, but transport does not turn it into an event.

Confusing the two creates hidden coupling. An “event” named UpdateSearchIndex is really a remote instruction; the publisher knows both the recipient and the desired implementation. Conversely, treating PaymentAuthorized as a command invites consumers to interpret a durable business fact as a retryable request.

Events also differ from change-data capture. A row-level message such as “column status changed from 2 to 3” exposes storage details. A domain event such as OrderCancelled captures business meaning and can remain stable when tables change. Change-data capture is valuable for replication and integration, but consumers should not have to reverse-engineer intent from database mutations.

Before publishing an event, write one sentence that is true after the transaction commits. If the sentence contains “please,” a target service, or an implementation detail, it is probably a command.

The Contract: Envelope and Schema Evolution #

A useful event has a stable envelope and a domain-specific payload. The envelope carries identity, causality, time, and schema metadata consistently across event types. The payload says what happened in business terms.

export type EventEnvelope<TType extends string, TPayload> = Readonly<{
  id: string;
  type: TType;
  schemaVersion: number;
  occurredAt: string;
  producer: string;
  correlationId: string;
  causationId?: string;
  aggregateId: string;
  sequence: number;
  payload: Readonly<TPayload>;
}>;

export type OrderPlacedV2 = EventEnvelope<
  'commerce.order-placed',
  {
    orderId: string;
    customerId: string;
    currency: 'USD' | 'EUR' | 'VND';
    totalMinor: number;
    lines: ReadonlyArray<{
      sku: string;
      quantity: number;
      unitPriceMinor: number;
    }>;
  }
>;

export function makeOrderPlaced(
  order: {
    id: string;
    customerId: string;
    currency: OrderPlacedV2['payload']['currency'];
    totalMinor: number;
    lines: OrderPlacedV2['payload']['lines'];
    version: number;
  },
  context: { correlationId: string; causationId?: string },
): OrderPlacedV2 {
  return {
    id: crypto.randomUUID(),
    type: 'commerce.order-placed',
    schemaVersion: 2,
    occurredAt: new Date().toISOString(),
    producer: 'checkout-service',
    correlationId: context.correlationId,
    causationId: context.causationId,
    aggregateId: order.id,
    sequence: order.version,
    payload: {
      orderId: order.id,
      customerId: order.customerId,
      currency: order.currency,
      totalMinor: order.totalMinor,
      lines: order.lines,
    },
  };
}

id identifies this publication and supports deduplication. correlationId connects an end-to-end business flow, while causationId points to the message or request that directly caused this event. aggregateId and sequence make per-entity ordering explicit. occurredAt is business time, not broker ingestion time.

Schema evolution is where long-lived event systems either become platforms or archaeology sites. Prefer additive changes: add an optional field, define its default interpretation, and keep old fields until known consumers migrate. Never silently change a field from cents to decimal currency, local time to UTC, or “created” to “approved” while retaining its name.

Consumers should validate at the boundary and explicitly upcast supported old versions into one internal representation:

type OrderPlacedV1Payload = {
  orderId: string;
  customerId: string;
  totalCents: number;
};

type CurrentOrderPlaced = OrderPlacedV2['payload'];

export function normalizeOrderPlaced(
  version: number,
  payload: OrderPlacedV1Payload | CurrentOrderPlaced,
): CurrentOrderPlaced {
  if (version === 1) {
    const old = payload as OrderPlacedV1Payload;
    return {
      orderId: old.orderId,
      customerId: old.customerId,
      currency: 'USD',
      totalMinor: old.totalCents,
      lines: [],
    };
  }

  if (version === 2) return payload as CurrentOrderPlaced;
  throw new Error(`Unsupported commerce.order-placed v${version}`);
}

Keep compatibility fixtures in consumer tests and assign an owner to every schema. A registry can enforce syntax, but only people can review semantics. Version the event when meaning changes incompatibly; do not create OrderPlacedV7 merely because an optional field was added.

Delivery, Ordering, and Idempotency #

Most production brokers provide at-least-once delivery: a message is retried until acknowledged, so duplicates are expected. At-most-once delivery avoids duplicates by accepting possible loss. “Exactly once” usually means exactly once within a limited broker or transactional boundary; it does not make an email provider, payment gateway, and database participate in one magical atomic transaction.

Design for at-least-once delivery and make side effects idempotent. On the producer side, the transactional outbox closes the gap between committing domain state and publishing its event. The order and outbox row are written in one database transaction; a relay publishes unsent rows later.

import type { DatabaseTransaction } from './database.js';
import type { EventBus } from './event-bus.js';

export async function placeOrder(
  transaction: DatabaseTransaction,
  input: PlaceOrderInput,
): Promise<string> {
  return transaction.run(async (database) => {
    const order = await database.orders.insert(input);
    const event = makeOrderPlaced(order, {
      correlationId: input.correlationId,
      causationId: input.requestId,
    });

    await database.outbox.insert({
      eventId: event.id,
      topic: 'commerce-events',
      partitionKey: event.aggregateId,
      body: JSON.stringify(event),
      createdAt: event.occurredAt,
    });

    return order.id;
  });
}

export async function publishOutboxBatch(
  transaction: DatabaseTransaction,
  bus: EventBus,
): Promise<void> {
  const rows = await transaction.outbox.claimUnpublished(100);

  for (const row of rows) {
    await bus.publish(row.topic, row.partitionKey, row.body);
    await transaction.outbox.markPublished(row.eventId);
  }
}

The relay can still crash after publish and before markPublished, so consumers must deduplicate. Record the event ID and business update in the same local transaction. A unique key on (consumer, event_id) turns a repeated delivery into a harmless no-op.

import type { Database } from './database.js';

export async function handleOrderPlaced(
  database: Database,
  event: OrderPlacedV2,
): Promise<'processed' | 'duplicate'> {
  return database.transaction(async (transaction) => {
    const firstDelivery = await transaction.inbox.tryInsert({
      consumer: 'fulfillment-order-projector',
      eventId: event.id,
      receivedAt: new Date().toISOString(),
    });

    if (!firstDelivery) return 'duplicate';

    const payload = normalizeOrderPlaced(event.schemaVersion, event.payload);
    await transaction.fulfillmentOrders.insert({
      orderId: payload.orderId,
      customerId: payload.customerId,
      status: 'awaiting-inventory',
      sourceSequence: event.sequence,
    });

    return 'processed';
  });
}

This protects the database effect. External effects need their own idempotency key: pass event.id to a payment or email provider that supports one, or store an intent before dispatch and reconcile uncertain outcomes. Never retry a timed-out charge blindly; timeout means “outcome unknown,” not “failed.”

Ordering is similarly bounded. Global ordering is expensive and rarely necessary. Partition by aggregateId when events for one order must remain ordered, then use sequence to reject stale events or detect gaps. Do not assume events for different aggregates have a meaningful total order. If a consumer requires CustomerCreated before OrderPlaced, it needs buffering, retry, or a read-through fallback rather than wishful timing.

Choreography, Failure Modes, and Recovery #

Choreography lets services react to facts without a central workflow owner. It works well when reactions are independent: analytics, search indexing, and customer notifications can all observe OrderPlaced without blocking checkout.

As the workflow gains dependent steps, deadlines, and compensation, pure choreography becomes difficult to understand. The business process is spread across subscriptions, and adding one event can trigger an undocumented cascade. An orchestrated saga makes sequence and state explicit, but introduces a coordinator that must itself be durable and available. Neither style is universally superior: use choreography for loose, independent reactions; use orchestration when the workflow has a clear owner and meaningful state transitions.

Common failure modes deserve designs, not just alerts:

• Poison messages always fail because of malformed data or an unsupported schema. Retry only a bounded number of times, then quarantine them with the error and original bytes.
• Transient dependency failures need exponential backoff with jitter. Immediate retries amplify an outage.
• Consumer lag can turn a technically healthy pipeline into a business incident. Measure age of the oldest unprocessed event, not only queue depth.
• Partial side effects arise when a database write succeeds but an external call has an unknown result. Persist intent and reconcile rather than guessing.
• Replay storms overload downstream systems or repeat non-idempotent effects. Replays need rate limits, isolated consumer groups, and an explicit scope.
• Dead-letter queues become data graveyards unless they have an owner, retention policy, inspection tooling, and a tested redrive path.

Compensation is not rollback. Cancelling an order after inventory reservation creates new facts; it cannot erase the time when inventory was unavailable to someone else. Model compensating actions explicitly and make them idempotent too.

Observability and When Synchronous Wins #

Traditional request tracing follows a call stack. Event workflows cross processes and may pause for hours, so observability must travel in the envelope. Log event.id, type, correlationId, causationId, aggregateId, schemaVersion, consumer name, attempt number, and processing duration as structured fields. Propagate W3C trace context in message headers when possible, but keep durable business correlation separate from vendor-specific tracing.

Useful metrics include publish rate, processing rate, retry count, dead-letter rate, consumer lag by partition, oldest-message age, handler latency, and outcome counts by event type. Dashboards should describe the business flow as well as the broker: “orders awaiting payment for more than ten minutes” is more actionable than “partition 7 has 4,812 records.” Alerts need a runbook that says whether to pause, skip, quarantine, replay, or compensate.

Auditability also requires restraint. Events are durable and widely copied, so avoid secrets and unnecessary personal data. Prefer stable identifiers over entire customer profiles. Define retention and deletion behavior before regulated data enters an immutable log.

Finally, sometimes a synchronous call is simply better. Use one when the caller cannot proceed without an immediate answer, when a user needs validation now, when consistency must be observed before returning, or when the interaction is a straightforward query. Checking inventory before presenting a delivery promise may warrant a bounded synchronous request; publishing InventoryCheckRequested and polling for an event would add latency and more failure states without useful autonomy.

A hybrid is often strongest: perform the minimal synchronous decision required for correctness, commit local state plus an outbox event, then distribute secondary work asynchronously. The checkout can synchronously validate the command and return an order ID while email, analytics, and projections catch up.

Choose event-driven communication when time decoupling, independent consumers, buffering, or replay creates concrete value. Choose a synchronous API when directness, immediate feedback, and a simpler failure model matter more. Architecture improves when both are ordinary tools rather than identities.

Takeaways #

• Model events as immutable business facts and commands as requests with an owner.
• Treat the envelope and payload as a long-lived API; evolve schemas additively and test old fixtures.
• Assume at-least-once delivery. Use an outbox for producer atomicity and an inbox or idempotency key for consumer effects.
• Define ordering only where the domain requires it, usually per aggregate, and expose sequence numbers.
• Pick choreography for independent reactions and orchestration for stateful, deadline-driven workflows.
• Design quarantine, replay, reconciliation, compensation, and observability before the first incident.
• Keep synchronous calls for immediate decisions and queries. Event-driven architecture earns its complexity only when asynchrony provides measurable value.