Message queues, event logs, delivery guarantees, and backpressure

01What is a message queue, and what does async mean?

A message queue holds work until a consumer can process it. The sender is the producer; a broker is the service that stores and delivers the messages. Asynchronous means the request can finish accepting work before that work finishes executing. Backpressure is the control that slows, defers, or rejects incoming work when processing capacity is insufficient.

An event records something that happened, such as photo-created. A command asks for an action, such as create-thumbnail. A queue commonly distributes commands among workers; a retained log allows independent consumers to replay events. These uses can share infrastructure, but their completion and replay contracts differ.

Synchronous processing makes request latency include the entire downstream task and occupies request-serving capacity throughout it. For example, accepting photo P501, rendering a thumbnail, and publishing its ready state can have very different latency distributions. A burst of slow rendering jobs can exhaust request workers even when upload storage is fast.

A queue stores work so another process can handle it later. We create job J501 and return a status saying the uploaded file and a record that it needs thumbnail processing have been durably saved. This is not the same as “thumbnail ready.” The client receives a photo ID and can check states such as pending, processing, ready, or failed.

The queue decouples when work arrives from when it executes. It can absorb a bounded burst, but it cannot make sustained excess demand disappear. The first design decision is therefore a user-visible contract: acceptance is quick and recoverable; completion is asynchronous and has a separate objective. All job rates and timestamps below are illustrative assumptions.

02Work queue versus event log versus publish/subscribe

A work queue distributes tasks among workers. J501 should be handled by an eligible thumbnail worker, with retry if that worker fails. A visibility lease can temporarily hide the task from other workers, but expiration can lead to another delivery while the first worker still runs.

A retained event log stores an ordered sequence that consumers can replay from a position. Separate consumer groups can independently process the same photo events: one builds thumbnails, another computes usage statistics. Publish/subscribe describes sending events to multiple subscribers; durability, retention, and replay depend on the actual system.

For each system, specify whether messages can repeat, which messages stay ordered, how long history remains available, and what counts as completed work. The names “queue” and “publish/subscribe” do not promise global order or exactly-once effects.

For this thumbnail service, a coherent starting implementation is a PostgreSQL photo/outbox transaction, a retrying relay, an SQS standard work queue, and workers that commit result metadata back to PostgreSQL. The database stores the job's logical state; the queue schedules attempts. A retained partitioned log such as Kafka is useful instead when several consumers need independent replay of photo events. Its order is per partition, so choosing photo ID as a partition key does not provide one global order across every photo.

03Transactional outbox: avoid a lost database-to-broker handoff

Suppose the upload service first commits P501 and then sends J501 to the broker. It crashes between those actions. The photo exists, but no worker learns that processing is required. Reversing the order creates another gap: a job may exist for a photo record that never committed.

A transactional outbox puts the photo record and a row describing the job to be published, including its ID and payload, in one database transaction. They commit together. A relay reads committed outbox rows and publishes jobs. If the broker is unavailable, the intention remains durable for a later retry. This protects the handoff without pretending the database and broker share one local transaction. Outbox reference.

The relay marks an intention published only after the broker confirms the required durable acceptance. A timeout is an unknown outcome, so it retries with the same event ID. Marking the row first would recreate the lost-handoff gap. Monitor oldest unpublished-outbox age separately from broker queue age: work can be stuck before it ever reaches the queue.

The relay needs a way to discover newly committed outbox rows. It can repeatedly query the table, or follow the database’s committed change stream. Change data capture provides the second option; a saved checkpoint records publication progress so a replacement connector can resume.

Change data capture (CDC) exports committed database changes to downstream systems, often by decoding the transaction log. A connector takes a consistent snapshot, continues from its matching log position, and checkpoints progress. If a crash occurs after publishing but before checkpointing, the connector can publish a change again; consumers still need replay-safe writes.

CDC can publish an outbox table without application polling. Capturing every table update instead is a different contract: low-level row changes do not necessarily represent a complete business event. Define transaction boundaries, keys, deletion records and schema evolution. In PostgreSQL, a stalled logical replication slot can retain WAL and exhaust storage, so monitor retained bytes as well as connector lag. CDC does not make an external effect atomic with the source transaction.

Interview check: Why retain both an outbox and CDC? The outbox defines the business event within the source transaction; CDC is one transport for publishing it.

04Consumer acknowledgments, duplicate delivery, and idempotent effects

A consumer acknowledgment tells the broker that a delivery has been handled and can be marked complete under the queue’s contract. The worker must choose that moment carefully: acknowledging before its result is recoverable can lose work after a crash, while acknowledging later allows duplicates that must be safe to handle.

The worker receives J501, whose immutable intent identifies photo P501, its version, and the thumbnail recipe. It creates an immutable output object for this attempt, then commits the authoritative output reference and a receipt for J501 in one database transaction. The receipt is a deduplication record: evidence that this logical operation has a recorded outcome. Only after that commit does the worker acknowledge the queue message.

The object-store write and the database transaction still commit separately. The stable logical identity is photo/version/recipe; immutable attempt-specific object keys avoid two concurrent renders overwriting the same bytes. A protected database transaction chooses the authoritative reference. A database receipt alone cannot make an unrelated external API call atomic.

Cleanup must coordinate with the transaction that makes the output available to readers. An unreferenced object may belong to an active render that has not committed yet. Keep an attempt record protecting it; cleanup first marks an expired attempt abandoned under the same transactional state that publication checks. An abandoned attempt cannot subsequently publish. Delete only abandoned, unreferenced attempt objects, so a scan that observed no reference cannot race with a later valid commit.

05At-most-once, at-least-once, and ordering guarantees

At-most-once handling can avoid repeated attempts by discarding or acknowledging before the effect, but a crash can lose work. At-least-once delivery permits repeats so incomplete or uncertain work can be attempted again. For example, SQS standard delivery explicitly requires duplicate-aware applications.

For J501 we choose at-least-once delivery with an idempotent effect: repeated processing converges on the same recorded thumbnail result. Retain deduplication evidence for the supported retry/replay horizon. Reusing the same job ID for different photo contents must fail or follow a defined versioning rule.

Ordering also needs a scope. If P501 version 3 replaces version 2, a late version-2 job must not overwrite the version-3 ready record. A conditional version check protects that update. Partitioning events by photo can help order their handling, but retries and parallel execution still require a precise rule for applying results.

In a retained log, a consumer’s offset is its recorded position in a partition. Atomically committing that position with new output records prevents one of those facts from advancing without the other. This is useful when consuming one event produces another, but the transaction still has a defined storage boundary.

06Backpressure: calculate queue growth and drain time

If arrivals remain at 400/second, the backlog does not drain. If they stay above 400, it grows. A queue is a buffer, not additional processing capacity.

Backpressure means controlling incoming or concurrent work when downstream capacity cannot keep up. Bound queue size or accepted delay, limit per-tenant demand, and reject or defer new uploads before exhausting durable storage. Retry temporary failures with bounded attempts and randomized backoff. Send permanently invalid images to an explicit failed/dead-letter workflow with diagnostics; retrying malformed bytes forever wastes capacity. Track oldest-job age as well as queue length, because it measures completion delay.

A dead-letter destination retains work that exhausted its retry policy or needs intervention, together with enough failure information to inspect it. An operator or recovery process may correct the cause and replay it deliberately. Moving a job there records an unresolved or failed outcome; it does not complete the thumbnail.

Limit work already taken by workers: fetching millions of jobs only moves the backlog into their memory. Cap concurrent render tasks and extend visibility timeouts while valid long jobs run; duplicates remain possible. After an outage, limit replay speed so old retries leave capacity for new work. If job sizes differ greatly, limit estimated resource use as well as job count.

07Interview answer: define the exactly-once effect boundary

Interviewer: “Can this queue guarantee thumbnails are processed exactly once?”

Candidate: “I would first distinguish rendering the thumbnail from publishing the result that users can see. J501 can be delivered again after a worker commits the ready record but crashes before acknowledging. I would make output publication safe to repeat and record J501’s receipt with the database result, so the retry returns the established outcome.

“I would use an outbox to ensure every accepted photo P501 has a stored job recording the required processing. That relay can also repeat publication, so duplicate handling remains necessary. For overload, I would measure job age and bound admission. A 600-per-second burst cannot be solved indefinitely by workers that process 400 per second.”

This answer traces both handoffs: accepting work into the background system and committing the worker’s result. It defines observable pending, ready, and failed states. The idempotency-retries-and-timeouts and distributed-transactions-and-workflows chapters develop the related boundaries further.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.