Backpressure: Flow Control in Async Systems

An API reads events at 40,000 messages per second, enriches them through a dependency that can sustain 6,000, and promises to process every message “asynchronously.” For a few minutes the system looks fast because producers never wait. Then memory climbs, garbage collection dominates CPU, timeouts trigger retries, and messages that are already too old to matter occupy the queue ahead of fresh work. Asynchrony did not remove the capacity limit; it hid the limit in a growing backlog.

Backpressure is the mechanism by which a constrained consumer makes upstream production slow down, pause, or choose an explicit overload policy. It is flow control across stages. A sound design bounds both work in progress and waiting work, carries demand in a direction producers can observe, and defines what happens when demand cannot be reduced.

Pressure starts with a rate mismatch #

Consider one pipeline stage with arrival rate lambda items per second, service rate mu, and average number of items in the stage L. If lambda remains greater than mu, no queue implementation can make the system stable. During an overload interval of t seconds, backlog grows approximately as:

For a stable system in steady state, Little’s Law relates concurrency, throughput, and time in the system:

• L = average items in service plus queue
• lambda = completed throughput per second
• W = average seconds an item spends in the stage

If a dependency completes 200 requests per second and each request spends 0.25 seconds there, the expected concurrency is $L = 200 \times 0.25 = 50$. A measured concurrency of 500 at the same throughput is not extra capacity. It is roughly 450 requests waiting, timing out, or contending.

Utilization gives another warning signal:

As rho approaches 1, small service-time variance creates disproportionate queueing delay. Operating permanently at 100% utilization leaves no room for bursts, slow requests, maintenance, or retries. Capacity planning therefore needs a latency budget and headroom, not only an average throughput target.

Push, pull, and demand signals #

In a push interface, the producer decides when to emit. DOM events, EventEmitter, webhooks, and many callbacks behave this way. If the consumer performs asynchronous work inside a listener, the emitter usually does not await it. Thousands of promises can become active before the first one completes.

In a pull interface, the consumer asks for the next value. An async iterator advances only when code calls and awaits next(). That makes demand part of the protocol:

async function* readPages(
  fetchPage: (cursor?: string) => Promise<{
    items: string[];
    nextCursor?: string;
  }>,
): AsyncGenerator<string> {
  let cursor: string | undefined;

  do {
    const page = await fetchPage(cursor);
    for (const item of page.items) yield item;
    cursor = page.nextCursor;
  } while (cursor);
}

for await (const item of readPages(fetchPage)) {
  await persist(item); // The next item is not requested until this completes.
}

Pull is not automatically bounded. A page may contain a million records, fetchPage may prefetch internally, or a consumer may start work without awaiting it. Concurrency must still be explicit. A bounded concurrent mapper can keep at most concurrency operations active and stop pulling while that window is full:

type Tracked<Result> = Promise<{
  tracked: Tracked<Result>;
  result: Result;
}>;

export async function* mapConcurrent<Input, Result>(
  source: AsyncIterable<Input>,
  concurrency: number,
  map: (input: Input) => Promise<Result>,
): AsyncGenerator<Result> {
  if (!Number.isInteger(concurrency) || concurrency < 1) {
    throw new RangeError('concurrency must be a positive integer');
  }

  const active = new Set<Tracked<Result>>();

  for await (const input of source) {
    let tracked!: Tracked<Result>;
    tracked = map(input).then((result) => ({ tracked, result }));
    active.add(tracked);

    if (active.size >= concurrency) {
      const completed = await Promise.race(active);
      active.delete(completed.tracked);
      yield completed.result;
    }
  }

  while (active.size > 0) {
    const completed = await Promise.race(active);
    active.delete(completed.tracked);
    yield completed.result;
  }
}

This mapper emits completion order rather than input order. Preserving order requires holding later results behind the earliest unfinished item, which can cause head-of-line blocking. Production variants should also accept an AbortSignal, cancel or settle remaining tasks after failure, and define whether one failed item stops the entire stream.

Demand may be represented as iterator pulls, Reactive Streams credits, HTTP/2 or QUIC flow-control windows, TCP receive windows, stream return values, or broker prefetch limits. The syntax differs, but the contract is the same: downstream grants finite permission to send.

Bounded queues, watermarks, and Node streams #

A bounded queue turns capacity into an enforceable invariant. When full, push must wait, reject, or apply a documented loss policy. The following queue blocks producers until a consumer creates space. Waiting producers do not place their payloads in an internal unbounded list; each caller retains its one item while awaiting admission.

export class BoundedAsyncQueue<Item> implements AsyncIterable<Item> {
  readonly #items: Item[] = [];
  readonly #readers: Array<(result: IteratorResult<Item>) => void> = [];
  readonly #spaceWaiters: Array<() => void> = [];
  #closed = false;

  constructor(readonly capacity: number) {
    if (!Number.isInteger(capacity) || capacity < 1) {
      throw new RangeError('capacity must be a positive integer');
    }
  }

  async push(item: Item, signal?: AbortSignal): Promise<void> {
    while (this.#items.length >= this.capacity && !this.#closed) {
      await this.#waitForSpace(signal);
    }
    if (this.#closed) throw new Error('queue is closed');

    const reader = this.#readers.shift();
    if (reader) reader({ value: item, done: false });
    else this.#items.push(item);
  }

  close(): void {
    if (this.#closed) return;
    this.#closed = true;
    for (const reader of this.#readers.splice(0)) {
      reader({ value: undefined, done: true });
    }
    for (const wake of this.#spaceWaiters.splice(0)) wake();
  }

  async next(): Promise<IteratorResult<Item>> {
    if (this.#items.length > 0) {
      const value = this.#items.shift() as Item;
      this.#spaceWaiters.shift()?.();
      return { value, done: false };
    }
    if (this.#closed) return { value: undefined, done: true };
    return new Promise((resolve) => this.#readers.push(resolve));
  }

  [Symbol.asyncIterator](): AsyncIterator<Item> {
    return { next: () => this.next() };
  }

  #waitForSpace(signal?: AbortSignal): Promise<void> {
    if (signal?.aborted) return Promise.reject(signal.reason);

    return new Promise((resolve, reject) => {
      const cleanup = () => signal?.removeEventListener('abort', onAbort);
      const wake = () => {
        cleanup();
        resolve();
      };
      const onAbort = () => {
        const index = this.#spaceWaiters.indexOf(wake);
        if (index >= 0) this.#spaceWaiters.splice(index, 1);
        cleanup();
        reject(signal?.reason ?? new Error('aborted'));
      };

      this.#spaceWaiters.push(wake);
      signal?.addEventListener('abort', onAbort, { once: true });
    });
  }
}

A single capacity threshold can make a producer oscillate rapidly between running and paused. Watermarks add hysteresis: pause at the high watermark and resume only after depth falls below the low watermark. For example, a 1,000-item queue might pause intake at 800 and resume at 300. The gap absorbs timing delay and prevents control chatter. Capacity remains 1,000 for in-flight races; the high watermark is the operational trigger, not the hard safety limit.

Node streams already implement this negotiation. Writable.write() returns false when its internal buffer reaches highWaterMark; a manual producer must wait for drain. pipeline() wires the pause, errors, cleanup, and completion path for stream composition:

import { createWriteStream } from 'node:fs';
import { Readable, Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';

async function* records(): AsyncGenerator<{ id: string; score: number }> {
  for await (const row of readDatabaseCursor()) yield row;
}

const toJsonLines = new Transform({
  objectMode: true,
  writableHighWaterMark: 32,
  transform(record: { id: string; score: number }, _encoding, callback) {
    callback(null, `${JSON.stringify(record)}\n`);
  },
});

await pipeline(
  Readable.from(records(), { objectMode: true, highWaterMark: 32 }),
  toJsonLines,
  createWriteStream('scores.ndjson', { highWaterMark: 64 * 1024 }),
);

Do not interpret highWaterMark as an exact global memory ceiling. Each stage has a buffer; object-mode marks count objects rather than bytes; transforms and kernel buffers hold additional data. Bound large payloads by bytes or reject oversized objects separately.

Batching and overload policy #

Batching amortizes fixed costs such as network round trips, transaction commits, compression frames, and broker acknowledgements. A useful batcher flushes on either maxItems or maxWaitMs. Waiting only for a full batch gives low-traffic items unbounded latency; flushing only on a timer can make burst throughput needlessly poor. Batch byte size often matters more than item count when payload sizes vary.

Backpressure works when the producer can slow down. Public traffic, UDP telemetry, webhook senders, and already-published broker records may not obey a local pause. The receiving system then needs an overload policy:

Load shedding should happen before expensive parsing, authentication fan-out, or dependency calls when possible. Use admission control based on queue capacity, concurrency, deadlines, priority, and tenant budgets. A request with 20 ms remaining should not enter a queue whose estimated wait is 500 ms. Per-tenant limits prevent one noisy source from consuming every slot.

Retries are a producer too. If each failed attempt is retried up to r times and each attempt independently fails with probability p, the expected attempt multiplier is:

• A = expected attempts per original operation
• p = probability an attempt fails
• r = maximum retry count

During an outage p approaches 1, so A approaches r + 1. Three retries can nearly quadruple pressure on a dependency that is already failing. Cap attempts, use exponential backoff with jitter, honor Retry-After, enforce a total deadline, and combine retries with a concurrency limiter or retry budget. Circuit breakers can stop clearly futile calls, but their half-open probes also need bounded demand.

Queue systems move pressure; they do not erase it #

A durable broker decouples producer availability from consumer availability. It can absorb a finite burst and replay after a deployment. It does not create consumer capacity. If ingress permanently exceeds processing, retention grows until a byte limit, age limit, partition limit, or financial limit is reached.

Broker-specific demand controls matter. RabbitMQ-style prefetch bounds unacknowledged deliveries per consumer. A prefetch of 1 protects a slow worker but can underuse concurrency; an enormous prefetch transfers the broker backlog into worker memory and can make distribution unfair. Set it near the worker’s useful concurrency, then measure.

Kafka consumers pull records, but max.poll.records is not a complete processing bound. If code polls again before finishing the previous batch, or starts one promise per record, in-memory work still grows. Pause assigned partitions when local capacity is full, keep polling often enough to maintain group membership where required, resume below a low watermark, and commit offsets only according to the desired delivery semantics. Per-partition ordering means concurrency should often be partition-aware.

Cloud queues commonly offer visibility timeouts. If processing exceeds the timeout, the same message becomes visible and creates duplicate work. Extend visibility deliberately for long tasks, make handlers idempotent, cap receive attempts, and route poison messages to a dead-letter queue. A dead-letter queue is not disposal: monitor its age and count, retain enough context to diagnose failures, and provide a controlled redrive process that cannot flood the recovered service.

End-to-end pressure must cross boundaries. Limiting consumers while allowing an HTTP ingress tier to enqueue without bound merely relocates the failure. State a maximum accepted backlog in items, bytes, or age, and make producers see the consequence when it is reached.

Measure and test the overload path #

Throughput alone hides pressure. Observe every stage, and distinguish waiting from active work:

• queue depth in items and bytes, plus oldest-item age
• arrival, admission, completion, rejection, drop, retry, and dead-letter rates
• active concurrency and configured concurrency limit
• service time separately from queue wait and end-to-end latency
• high/low watermark transitions and time spent paused
• timeout budget remaining when work starts
• event-loop delay, heap usage, garbage-collection pause, socket count, and broker lag

Depth is a state; its derivative explains direction. A depth of 10,000 draining at 2,000 per second may be healthier than a depth of 100 growing continuously. Oldest-item age often maps to user harm more directly than count. Alert on sustained growth and age relative to the service-level objective, not one universal depth threshold.

Test flow control under deterministic stress. Use a fake or controlled dependency whose completion you can gate. Fill the queue to its high watermark and assert that upstream stops pulling. Release one operation and assert that active work never exceeds the limit. Verify resume only below the low watermark, cancellation removes waiters, shutdown drains or rejects according to policy, and one failure does not leave producers waiting forever.

Then run load tests with bursty arrivals, variable and long-tail service times, oversized payloads, dependency brownouts, retryable errors, broker redelivery, and a consumer restart. Track a conservation equation over a test interval:

All labels are ASCII and each term should be counted exactly once. If the equation does not balance, work may be duplicated, leaked, or silently lost. Also verify recovery: after the dependency becomes healthy, backlog age should converge to zero without a retry surge causing another collapse.

Takeaways #

• Backpressure is an end-to-end demand protocol, not a larger queue. It makes finite downstream capacity visible upstream.
• Stable service requires long-run arrival rate below service rate. Little’s Law connects throughput, latency, and in-flight work, while high utilization explains sharp queueing delay.
• Pull interfaces, bounded concurrency, bounded queues, and high/low watermarks provide complementary controls. None is sufficient when hidden buffers remain unbounded.
• Node streams propagate pressure through write(), drain, and pipeline(), but each stage and payload size still contributes to the real memory bound.
• Batching improves efficiency; admission control, shedding, deadlines, and fairness define behavior when producers cannot slow down.
• Retries amplify pressure during failure. Bound them with jittered backoff, total deadlines, retry budgets, and concurrency limits.
• Brokers relocate and persist backlog. Prefetch, polling, visibility timeouts, offset rules, idempotency, and dead-letter operations determine whether that backlog stays controlled.
• Measure queue age, wait time, rates, retries, and saturation, then test overload and recovery as first-class behavior.