Background Job Systems: Queues, Retries, and Idempotency

Moving work out of an HTTP request does not make that work reliable. It changes who owns the waiting. The caller may receive a quick response, but the system must now preserve intent across process crashes, choose when another worker may try, distinguish a slow attempt from a dead one, prevent duplicate effects, and explain a job that has been “running” since yesterday.

A robust background-job system is therefore a durable state machine with a delivery mechanism attached. The queue is important, but it is not the source of all semantics. Job identity, attempt history, lease ownership, retry classification, scheduling policy, and effect invariants belong to the application contract whether delivery uses a database table or an external broker.

This article builds that contract around a render_statement job that produces one PDF statement for an account and billing period. The worked example is deliberately ordinary: it is long enough to outlive a request, expensive enough to limit, and visible enough that duplicate or missing results matter.

Make the Job Contract Explicit #

A job describes durable intent. Its record should identify the operation and subject without embedding an arbitrary function call. For the example, the logical key is (account_id, billing_period, format_version). The payload contains immutable inputs or references to versioned inputs; the handler version identifies processing semantics. A job submitted today must not silently execute different business rules after next week’s deployment unless that compatibility is intentional.

Define states from observable facts rather than worker implementation details:

An attempt is not a job. The job has a stable ID across its lifetime; every lease creates a new attempt ID and number. This distinction lets an operator see that job job_842 failed twice, then succeeded on attempt three. Reusing one mutable “last error” field destroys the evidence needed to understand retries and races.

Specify success in domain terms. For render_statement, success means the PDF artifact exists at its deterministic object key, its checksum and metadata are committed, and the statement record points to it. “The handler returned” is insufficient: a process can return after writing an object but before recording it, or record metadata before a failed upload. The handler must make these boundaries recoverable.

Cancellation also needs semantics. It is usually a request not to begin more work, not proof that an in-flight side effect was reversed. A worker should check cancellation at defined safe points and stop cooperatively. The final state may need cancelled with an artifact already produced, depending on which effect committed first. Hiding that race behind a Boolean flag makes APIs lie.

Enqueue Intent Without a Transaction Gap #

The first failure window appears before any worker runs. Suppose an API writes statement_requested = true, commits, and then publishes a queue message. A crash between those operations leaves business state saying work was requested with no deliverable job. Publishing first reverses the gap: a worker may observe a job whose database transaction later rolls back.

When intent and job state share a relational database, insert them in one transaction:

BEGIN;

INSERT INTO statements (
  account_id, billing_period, format_version, status
) VALUES (:account_id, :period, :version, 'pending')
ON CONFLICT (account_id, billing_period, format_version) DO NOTHING;

INSERT INTO jobs (
  id, kind, logical_key, payload, state, run_at, max_attempts
) VALUES (
  :job_id,
  'render_statement',
  :logical_key,
  :payload,
  'ready',
  CURRENT_TIMESTAMP,
  6
)
ON CONFLICT (kind, logical_key) DO NOTHING;

COMMIT;

The uniqueness constraint turns repeated submission into retrieval of the same logical work. It does not make execution exactly once; a worker can still receive or lease the job more than once. Submission deduplication and effect idempotency solve different races.

If an external delivery service is used, a dispatcher can publish committed job IDs from an outbox and mark publication progress. Publication itself may happen twice after an ambiguous response, so the delivery payload carries the stable job ID. The receiving side treats the external message as a wake-up hint and loads authoritative job state before claiming work. This prevents an old duplicate message from resurrecting a succeeded or cancelled job.

Payloads deserve limits and schemas. Store the account and period, not an unbounded serialized object graph. Large input belongs in durable storage referenced by an immutable version or checksum. Encrypt sensitive values, define retention, and avoid secrets that may expire before a scheduled job runs. Validate payloads at submission and again at execution when old records can outlive code versions.

For APIs, return the stable job or operation resource. A client that times out while submitting can repeat the request with the same logical key and discover the accepted job. Generating a new identity for every retry creates duplicate work before the worker system has a chance to help.

Lease Attempts Instead of Assuming Ownership #

Workers must claim work atomically. Reading a ready row and updating it in separate operations allows two workers to run the same attempt. A database-backed queue can select eligible rows with locking and skip rows claimed by peers; other delivery mechanisms offer an equivalent visibility timeout. In both cases, ownership is temporary.

WITH candidate AS (
  SELECT id
  FROM jobs
  WHERE state = 'ready'
    AND run_at <= CURRENT_TIMESTAMP
  ORDER BY priority DESC, run_at ASC, id ASC
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE jobs
SET state = 'running',
    attempt_count = attempt_count + 1,
    lease_owner = :worker_id,
    lease_token = :random_token,
    lease_expires_at = CURRENT_TIMESTAMP + INTERVAL '2 minutes'
WHERE id = (SELECT id FROM candidate)
RETURNING *;

The random lease token fences stale workers. Completion updates must include WHERE lease_token = :token AND state = 'running'. If a worker pauses, loses its lease, and resumes after another worker completes the job, its stale success cannot overwrite current state. Worker identity alone is not enough because one process can hold multiple successive leases.

Set the initial lease longer than ordinary handler time but finite enough to recover from death. For genuinely variable work, heartbeat at a fraction of the lease duration. A heartbeat extends only a lease still owned by the same token. It should also report a bounded progress marker, not reset an unresponsive job forever. Establish a maximum runtime or progress deadline so a live but stuck worker eventually loses ownership.

Lease expiry does not cancel code. The old process may still write to storage after expiry. This is the central reason leases cannot provide exactly-once effects. Fencing tokens can protect resources that enforce monotonically increasing tokens, while application uniqueness and conditional writes protect ordinary databases. External APIs need their own idempotency key or reconciliation logic.

A graceful worker stops leasing new jobs, continues heartbeats for accepted work, and finishes or abandons within a shutdown deadline. On forced shutdown, let leases expire rather than rewriting jobs blindly; another worker may already have recovered them. Record worker version, host, attempt start, heartbeat, and lease loss so operations can distinguish deployment churn from handler failure.

Work Through One Statement Job #

Consider two identical API submissions for account acct_91, period 2026-05, and format version 3. The logical-key constraint returns one job, job_842. Worker A leases attempt 1 with token 31 and begins rendering. It uploads the completed PDF to the deterministic object key:

statements/acct_91/2026-05/v3.pdf

Before committing artifact metadata, Worker A loses its database connection. The lease expires. Worker B acquires attempt 2 with token 32. It first checks the artifact key and validates stored checksum metadata. If the upload is complete, it can reuse the artifact; if the object is absent or marked incomplete, it renders to a temporary key and atomically promotes the result according to the storage API’s guarantees.

Worker B then commits success conditionally:

BEGIN;

INSERT INTO statement_artifacts (
  account_id, billing_period, format_version, object_key, sha256
) VALUES (:account_id, :period, :version, :object_key, :sha256)
ON CONFLICT (account_id, billing_period, format_version)
DO UPDATE SET
  object_key = EXCLUDED.object_key,
  sha256 = EXCLUDED.sha256;

UPDATE jobs
SET state = 'succeeded',
    completed_at = CURRENT_TIMESTAMP,
    lease_owner = NULL,
    lease_token = NULL,
    lease_expires_at = NULL
WHERE id = :job_id
  AND state = 'running'
  AND lease_token = :lease_token;

-- Require exactly one updated job row before commit.
COMMIT;

In a complete implementation, the artifact upsert must reject conflicting bytes for the same logical key rather than quietly replacing a valid statement. Deterministic output is helpful but not assumed: the checksum exposes a mismatch caused by changed input, renderer version, fonts, or locale. If conflict is possible, include every meaning-changing input in the format version or persist an immutable source snapshot.

Now suppose Worker A resumes. Its metadata update uses token 31 and affects zero job rows. It must stop and discard any temporary output. Worker B owns the conclusion. The system may have rendered twice, but it publishes one logical artifact and one successful job state.

This example separates three identities: job_842 names intent, attempt IDs name executions, and the logical artifact key names the domain effect. Trying to use one identifier for all three makes either retries or domain uniqueness ambiguous.

Retry Errors with a Bounded Policy #

Not every exception deserves another attempt. Classify errors near the failing dependency and preserve a stable machine-readable code:

Use exponential backoff with jitter, capped by a maximum interval and job deadline. Count attempts from durable history, not process memory. A deployment restart must not reset a job to its first retry. Jobs also need an age or completion deadline: succeeding a week late can be worse than failing visibly, especially when the result drives customer communication.

Keep a retry budget across dependency layers. If the HTTP client retries three times inside a job that itself runs six attempts, one logical operation can call the dependency eighteen times. Prefer one layer to own retries or make the total explicit. Respect cancellation while waiting; a cancelled job should not wake merely to discover it should stop.

After a terminal error or exhausted policy, retain the job in a failed state with its payload reference, attempt history, first and last error codes, and remediation data. A dead-letter queue is an operational view of those failed records, not a trash bin. Operators need to inspect, group by cause, repair the prerequisite or deploy a fix, then redrive selected jobs. Redrive creates a new attempt and audit entry while preserving the stable job identity.

Bulk redrive is hazardous. Releasing thousands of poison jobs after a fix can saturate the repaired dependency. Rate-limit by job kind and tenant, canary a sample, and stop automatically if the same error recurs. Expire payloads according to data policy and make an expired job explicitly unrecoverable instead of retrying with missing context.

Make Effects Idempotent and Duplicates Harmless #

At-least-once execution is the practical default because every handoff has an ambiguous crash window. Deduplicating delivery by message ID can reduce work, but it cannot prove an effect did not occur. If a payment request times out after reaching the provider, suppressing the queue message does not reveal whether money moved.

Design idempotency at the domain boundary. Database effects use unique constraints, compare-and-set transitions, or an inbox record committed with the mutation. Object creation uses deterministic keys plus conditional creation and checksum verification. External APIs receive a stable key derived from the job and effect, such as job_842:publish-artifact, not a fresh attempt ID. Notifications use a unique business key when policy says only one should exist.

Record the result of an effect, not only that it started. A local effect ledger can map (job_id, effect_name) to pending, confirmed, or uncertain, with the remote request ID and response. On retry, a confirmed effect returns its stored result. An uncertain effect invokes a provider lookup or reconciliation path before sending again. This is more honest than marking every caught timeout as failure.

Idempotency has a scope and retention period. Reusing a key for different canonical arguments must be rejected. Deleting keys too early allows an old delayed attempt to repeat the effect; retaining payloads forever may violate privacy policy. Tie retention to the maximum job, retry, and replay horizon, then preserve only the minimum audit data needed after expiry.

Handlers should be restartable from durable checkpoints only when checkpoints have stable meaning. Recording “47 percent” is useful for display but rarely sufficient for resumption. Recording “partitions 0 through 46 committed with these checksums” can be. Checkpoint writes and the work they claim must form one recoverable boundary, or retries will skip incomplete output.

Schedule Fairly and Bound the Workers #

A single FIFO queue is simple but can let a flood of slow exports delay urgent password emails. Separate job classes when their resource needs, latency objectives, or failure domains differ. Give each class bounded concurrency and queue capacity. Priority can order eligible work, but unbounded high-priority traffic starves everything else; aging or reserved capacity provides eventual progress.

Fairness often belongs at the tenant boundary. Round-robin or weighted scheduling across tenants prevents one bulk import from occupying every worker. Per-tenant rate limits protect downstream APIs that enforce account-specific quotas. Global concurrency protects CPU, memory, database pools, and network connections. More workers increase queue consumption only until one of those shared resources saturates.

Scheduling a future job means persisting run_at in a durable clock domain and promoting it when eligible. It does not promise execution at an exact instant. Publish a lateness objective and measure started_at - run_at. For recurring work, store the schedule separately from generated occurrences. Derive each occurrence from a calendar rule and stable occurrence key so scheduler failover neither skips nor duplicates a period.

Backpressure begins at submission. When retained work exceeds safe capacity, reject optional jobs, coalesce superseded jobs, or delay low-priority acceptance. An infinite queue converts overload into stale work and storage growth. Estimate drain time by job class and expose it to callers when useful. Cancellation and supersession can remove obsolete intent, but workers must still handle a race with jobs already leased.

Worker processes should declare supported handler versions and resource limits. Do not lease a version they cannot execute. Cap payload size, per-job memory, child-process lifetime, output size, and dependency concurrency. Isolate jobs that execute untrusted input more strongly than ordinary application code; a queue boundary alone is not a security sandbox.

Observe, Test, and Operate the State Machine #

The primary operational signals describe flow through durable states: ready count by job kind, oldest eligible age, scheduling lateness, lease acquisition rate, running age, lease expirations, retries by error code, terminal failures, cancellation delay, and completion latency. Queue depth alone is ambiguous; one thousand one-second jobs and one thousand hour-long jobs are different incidents.

Trace every attempt with job ID, attempt ID, logical key hash where safe, worker version, lease token identifier, handler phase, dependency calls, and effect IDs. Keep high-cardinality identifiers in traces or searchable logs, not metric labels. Dashboards should distinguish submission rate from completion rate and show the age distribution. Alert on user impact such as old eligible work or repeated lease loss, not a fixed nonzero queue.

Deterministic state-machine tests are more valuable than tests that merely wait for a worker. Use a controllable clock and explicit worker hooks. Verify:

• two submitters racing on one logical key produce one job;
• two workers racing to claim produce one current lease;
• a stale lease token cannot complete after recovery;
• a crash before effect, after effect, and after success commit converges correctly;
• retryable, terminal, throttled, and unknown errors take distinct paths;
• cancellation before claim and during execution has documented outcomes;
• a scheduler restart produces every recurring occurrence once by logical key;
• redrive preserves history and respects current throttles.

Property-based tests can generate transition sequences and assert invariants: terminal success never returns automatically to ready, attempt numbers increase, at most one unexpired lease is current, and every succeeded job has its required effect record. Integration tests should terminate real worker processes at named barriers, then let another worker recover. Fault injection against storage and dependencies validates the ambiguous windows that mocks often erase.

Operational runbooks should cover stuck leases, rising retry volume, poison payloads, overloaded dependencies, clock skew, and a bad handler deployment. Controls include pausing one job class or tenant, reducing concurrency, disabling redrive, cancelling obsolete work, and rolling back handlers while preserving compatible payload readers. Rehearse draining workers and restoring schedulers before incidents.

The durable principle is that background work is not a disappearing request. It is recorded intent moving through explicit states under temporary ownership. Stable job identity, fenced attempts, bounded retries, idempotent effects, fair scheduling, and inspectable history turn that intent into an operable service. The queue delivers opportunities to work; the state machine determines what those opportunities mean.