Message Queues, Streams, and Durable Logs

Message infrastructure is often described with overlapping product vocabulary. A broker may call a destination a queue, a topic, or a stream while offering features associated with all three. The useful distinction is not the product label. It is the contract between stored records, consumers, and time.

A work queue transfers responsibility for each task to one worker. A stream lets independent consumer groups observe a continuing sequence while tracking their own positions. A durable log makes the retained ordered history itself the primary abstraction, so consumers can rebuild state by replaying it. These models can share an implementation, but they optimize different workloads and produce different failure modes.

The thesis of this article is practical: choose messaging by asking who owns a record after publication, how long it must remain available, whether old records must be replayable, and which ordering boundary correctness actually requires. Starting from those questions prevents a generic enthusiasm for asynchronous events from becoming an accidental data-retention or processing contract.

Begin With the Consumption Contract #

Imagine a producer writes record R. The first design question is what successful consumption means.

In a work queue, R represents outstanding work. One eligible worker claims it, performs the operation, and acknowledges completion. The broker may redeliver it when the claim expires or the worker disconnects. Other workers generally do not need to see the same task. The queue’s meaningful state is therefore the set of tasks waiting, in flight, delayed, or dead-lettered.

In a stream, R is an observation available to multiple applications. Fraud detection, analytics, and customer notifications may each consume it independently. Each application maintains a cursor, often per partition, and advancing one cursor must not hide the record from another group. The meaningful state includes both retained records and consumer positions.

In a durable log, the ordered sequence is a source from which state can be derived. A projection can start at offset zero, read every retained record, and reconstruct a materialized view. Consumers may still process continuously like stream readers, but replay is not merely an emergency feature; it is part of the data model.

These are ideals rather than mutually exclusive product categories. A retained stream can feed a competing-worker group. A queue can support several subscriptions through copied destinations. The distinction still matters because emulating one contract with another moves complexity somewhere: duplicated storage, custom cursor databases, manual fan-out, or an application-managed archive.

Compare Retention, Replay, and Fan-Out #

Retention determines whether the broker is only a handoff mechanism or also a history. Queue records usually disappear after acknowledgement, apart from audit metadata or a dead-letter copy. That keeps storage proportional to backlog, but a newly deployed consumer cannot ask the queue for tasks completed last month.

Streams decouple retention from acknowledgement. A record may remain for seven days or until a byte limit is reached even if every current group has consumed it. A new group can begin at the oldest retained offset, while an existing group can reset its cursor to replay a range. This makes experimentation, backfills, and independent fan-out inexpensive relative to publishing a separate copy for every destination.

A durable log requires a stronger retention argument. If rebuilding a ledger projection requires the complete sequence, deleting old records after seven days violates the recovery model. The system might retain indefinitely, archive immutable segments to cheaper storage, or compact by key. Compaction keeps a later value for each key, which is useful for reconstructing current configuration but unsuitable when every transition matters. A tombstone also needs enough retention time for all replicas and rebuilds to observe deletion.

Fan-out exposes another difference. Suppose one publication must reach ten independent applications. Ten queue destinations make ten physical or logical copies and ten delivery policies. A stream stores the record once and gives each group a separate position. That is attractive until one group needs a radically different security boundary, region, retention period, or payload. Shared storage couples those requirements, so explicit replication or separate topics may still be the cleaner design.

Replay is powerful but not free. Replayed records can repeat emails, payments, or webhooks unless consumers separate durable state reconstruction from external side effects. Schemas and referenced data may also have changed since publication. A valid replay contract specifies the record schema version, deterministic transformation behavior, side-effect policy, and the starting snapshot or offset.

Define Ordering at the Smallest Correct Boundary #

Messaging discussions often ask whether a system is ordered as though ordering were a global boolean. Ordering is always relative to a scope. A broker may preserve order within one queue or partition while processing across partitions proceeds concurrently. Redelivery can also make completion order differ from delivery order.

Global order is expensive because every producer and consumer must coordinate through one serial position. Most applications need a narrower invariant: updates for one account, order, device, or document must be observed in sequence, while unrelated keys can proceed independently. Partitioning by that key preserves local order and unlocks parallelism across keys.

Choosing a partition key is therefore a correctness and capacity decision. Hashing by accountId keeps one account’s records together, but a single very active account can create a hot partition. Random partitioning spreads load but destroys per-account order. Composite or hierarchical keys can distribute a large tenant, though then operations spanning subkeys need explicit coordination.

Sequence numbers help a consumer distinguish broker order from domain order. A record can carry (entityId, version). The consumer accepts version $n+1$ after version $n$, ignores an exact duplicate, and parks a gap for repair. This works only if one authority allocates versions for that entity or conflicting writers are reconciled elsewhere.

Retries complicate ordering. If task A fails and returns to a queue while task B succeeds, strict FIFO completion is lost. Blocking the entire queue behind A preserves order but creates head-of-line blocking. Per-key queues, partition-level pausing, or a consumer-side reorder buffer localize that cost. The correct choice follows the domain invariant: an image-resize queue may allow overtaking, while a balance projection cannot silently apply version 12 before version 11.

Work an Order-Fulfillment Example #

Consider an online store with three flows:

1. Reserve inventory and request shipment for each accepted order.
2. Feed order transitions to fraud, analytics, and customer-notification applications.
3. Rebuild an order_status projection after its database is lost.

Using one mechanism for all three obscures their contracts. A clearer design uses a queue for commands and a retained partitioned log for facts.

When checkout commits order O-481, an outbox publisher appends an immutable fact:

{
  "eventId": "evt-9017",
  "orderId": "O-481",
  "orderVersion": 1,
  "type": "OrderAccepted",
  "occurredAt": "2026-05-01T10:00:00Z"
}

The topic is partitioned by orderId, so every transition for O-481 shares an ordered partition. Fraud, analytics, notifications, and the status projector use separate consumer groups. A slow analytics job does not move the notification cursor. Retention is long enough for operational reprocessing, and immutable segments are archived because the projection-recovery requirement exceeds online retention.

The fulfillment coordinator consumes OrderAccepted, records its decision durably, and enqueues ReserveInventory with a stable command ID. Competing inventory workers claim commands. If a worker crashes before acknowledgement, the command is delivered again. The inventory API therefore stores the command ID with the reservation result and returns that result on duplicates.

Suppose notifications deploy a bug and advance through offsets 8,200 to 8,900 while suppressing valid messages. Operators fix the code, reset only that group’s cursor to 8,200, and run it in a replay mode that writes intended notifications to a review table rather than sending them immediately. After reconciling already-sent IDs, they release the missing sends. Other groups remain untouched.

Now suppose the projection database is lost. The projector restores a snapshot that records partition offsets and then replays later order facts. It verifies that each order version is contiguous before committing the new state and its corresponding offset in one transaction. A queue of unacknowledged tasks could not perform this rebuild because successfully completed facts would already be gone.

This design deliberately does not claim one universal exactly-once channel. It combines at-least-once delivery, idempotent command handlers, per-order versions, transactional projection checkpoints, and side-effect reconciliation. Those are inspectable local guarantees.

Make Consumer State Explicit #

Consumer progress is application state even when a broker stores it. For a queue, state includes which worker owns a claim, when its visibility timeout expires, how many attempts occurred, and when a task moves to a dead-letter destination. An acknowledgement should mean the business effect is durable, not merely that parsing succeeded.

For a stream or log, a cursor says that every earlier record within its scope has reached a defined processing milestone. Committing the cursor before writing application state can lose work after a crash. Writing state before committing the cursor can repeat work. Consumers address this gap with idempotent updates, a transaction that includes both state and offset where supported, or an inbox table keyed by record ID.

Parallel processing makes cursor movement subtle. If offsets 40, 41, and 42 run concurrently and 41 fails, committing 42 as a single high-water mark may skip 41. A consumer can restrict concurrency per partition, track completed gaps and commit only the contiguous prefix, or subdivide work by key behind a partition reader. More concurrency requires more progress bookkeeping.

Ownership changes also need epochs. During a consumer-group rebalance, an old consumer may finish work after its partition has been assigned elsewhere. Generation numbers or leases allow the state store to reject a stale checkpoint. Without that fencing, two owners can race even though the broker’s assignment appears exclusive.

Monitor state as lag in both records and time. Ten thousand records may be harmless at high throughput, while ten records blocked for an hour may violate a freshness objective. Queue age, oldest in-flight claim, redelivery rate, consumer offset, end offset, and last successfully processed event time answer different operational questions.

Choose Semantics for Failures, Not the Happy Path #

At-most-once processing drops a record rather than risk duplication. At-least-once processing retries until acknowledgement and can duplicate effects. Exactly-once claims are scoped: a broker may atomically write output records and offsets inside its own transaction, but an email provider or external database is outside that boundary.

Design poison-record handling before deployment. Infinite retry blocks an ordered partition and consumes capacity. Immediate dead-lettering preserves flow but can violate a state sequence. A useful policy classifies failures:

• Transient dependency failures retry with bounded, jittered delay.
• Permanent validation failures move to quarantine with the schema error.
• Missing predecessor versions pause or park that key for targeted repair.
• Unknown code defects stop or isolate the affected partition rather than acknowledging data loss.

Dead-letter queues are not completion. They require ownership, alerts, inspection tooling, retention, redrive semantics, and protection against replaying the same invalid record forever. Store the original record, error classification, attempt history, source position, and code version without leaking secrets.

Backpressure differs by model. A bounded queue rejects or delays producers when backlog reaches capacity. A retained stream often continues accepting records while consumer lag grows, converting compute pressure into a storage and freshness problem. Set lag and retention headroom alerts early enough that the oldest needed record cannot expire before recovery.

Broker availability also deserves precision. Mirroring replicas does not guarantee acknowledged records survive every failure; durability depends on acknowledgement quorum, replica synchronization, producer retry behavior, and leader election. Likewise, a highly available broker does not make a non-idempotent consumer safe.

Verify the Contract Before Operating It #

Test messaging systems with crash points, not only end-to-end success. For a queue worker, inject termination before the side effect, after the side effect but before acknowledgement, and after acknowledgement. Verify that no task is silently lost, duplicate delivery is harmless, visibility timeouts exceed legitimate processing time or are renewable, and dead-letter thresholds behave as documented.

For stream consumers, run a deterministic record set containing duplicates, version gaps, malformed payloads, and several partition keys. Stop the process between state writes and cursor commits. Rebalance partitions during in-flight work. Reset a test group to an earlier offset and prove replay yields the same projection checksum without repeating forbidden external effects.

Capacity tests should vary producer rate, record size, key skew, consumer concurrency, and dependency latency. Observe publish acknowledgement latency, queue depth, oldest-record age, partition imbalance, consumer lag time, redeliveries, storage growth, and expired-record risk. An even synthetic key distribution will not reveal a tenant hotspot, so include deliberately skewed traffic.

Operational runbooks should cover a stuck partition, poison record, runaway producer, slow consumer, broker failover, retention exhaustion, cursor reset, and archive restore. Protect administrative operations: deleting a destination or resetting a production group can be more destructive than ordinary data-plane access.

A decision review can then ask concrete questions:

• Is the record outstanding work, a shared observation, or authoritative history?
• Does one worker own it, or does each consumer group need an independent view?
• What ordering key follows from the business invariant?
• How long must online replay remain possible, and what survives beyond that window?
• Where are progress and deduplication stored atomically enough for the required guarantee?
• What happens when a consumer is offline longer than retention?

Queues, streams, and durable logs are not maturity levels on one architectural ladder. They are contracts for different relationships between work, history, and readers. A queue is strongest when completion should retire a task. A stream is strongest when many independent applications need a retained flow. A durable log is strongest when ordered history must reconstruct state. Make those obligations explicit, and product-specific features become implementation choices instead of substitutes for design.