Distributed transactions and sagas

01Distributed transaction: definition and local boundaries

A distributed transaction is one transaction whose operations span multiple databases or transactional resource managers, such as an order database and an inventory database. An atomic-commit protocol, such as two-phase commit, coordinates their commit-or-abort outcome. A saga instead coordinates a business operation through committed local transactions and explicit compensating actions. A local database transaction can atomically change its own records, but cannot automatically undo an HTTP request that already succeeded at another service. Crossing independently failing systems therefore requires a protocol for partial completion.

For example, order O81 requests two MUG9 items at $12 each. Inventory starts at five, and hold H81 reserves two units for 120 seconds. Payment action A81 authorizes $24: authorization reserves funds and is distinct from capture. The order can become confirmed only after inventory allocation and the required authorization are established; partial progress remains pending.

If inventory and orders share one database and ownership boundary, a short transaction is the simplest option. Splitting tables into services prematurely creates a harder problem. We study the split because the provider is external and inventory may have a separate owner, not because every application needs distributed transactions.

02Two-phase commit: prepare and commit or abort

Two-phase commit (2PC) makes participating databases agree to commit or abort together. A coordinator records the final decision. First it asks each database to prepare. A database voting “yes” durably saves enough state to finish later and keeps the necessary locks or other protections. If all vote yes, the coordinator durably records “commit” and tells them to commit; otherwise the protocol chooses abort. Each participant must support preparing and honoring that decision.

For O81, suppose the order and inventory databases both support 2PC. They prepare their changes, then follow the same commit-or-abort decision. This prevents one from committing while the other aborts. Their concurrency controls must still provide the required isolation; atomic commit alone does not make all cross-database transactions serializable.

03Saga: local transactions and compensation

A durable workflow stores the business operation’s progress so another worker can continue after a crash. Model that progress as a state machine: named states and allowed transitions, such as awaiting authorization, ready to allocate, or cancellation pending. Each transition records what happened and which action is now permitted.

Our provider does not participate in the database’s prepare/commit protocol, so I choose a durable workflow. The order coordinator records O81’s state, the inventory hold identifier H81, and authorization operation A81. Each transition checks the expected previous state and records the next outgoing intent in the same local transaction.

The workflow needs durably stored state and a service responsible for advancing it. It does not need one process to remain alive throughout: a replacement worker can resume from the stored state.

04Successful saga: reserve, authorize, allocate, confirm

At time 0, inventory conditionally creates H81 for two mugs: available stock becomes three, and H81 expires at time 120. At time 1, the workflow asks the provider to authorize $24 using stable operation A81. At time 2, it records authorization success. It next asks inventory to convert H81 into an allocation for O81, only if the hold still exists and is valid. Inventory performs that check and transition atomically.

If allocation succeeds, a later coordinator transaction records the order as confirmed and publishes its event through an outbox. If the coordinator crashes after allocation but before recording confirmation, retrying the allocation request with O81 returns the existing allocation. It must not remove another two mugs. The same rule applies to authorization A81.

A distributed workflow is therefore a state machine: a set of allowed states and transitions. “Already allocated to O81” is a meaningful result. A vague boolean success loses the identity needed for recovery. The confirmation contract should also specify authorization validity and any later capture/shipping steps; those are separate transitions with their own failure handling.

Success and cancellation can race, so each state change must atomically check that the order or hold is still in a state that allows it. An allocation request names the order and hold; the inventory owner atomically returns the existing allocation, converts a still-valid hold, or rejects expiry/cancellation. The coordinator accepts confirmation only from its expected pending state. If cancellation won locally but allocation had already committed remotely, recovery records that allocation and releases it through an idempotent compensating transition; simply ignoring the late reply would strand stock. Start shipping only after checking that the order is confirmed and remains eligible for fulfillment.

05Unknown outcomes: lost replies and expired reservations

Now let the authorization response disappear. The provider may have processed A81 even though the coordinator received nothing. The workflow records authorization_unknown, queries by A81 or retries under the provider’s idempotency contract, and avoids inventing a new authorization identifier.

06Saga recovery: compensation, retries, and outbox

Suppose voiding A81 times out too. Marking O81 simply “cancelled” and forgetting it would hide unfinished work. Record cancellation requested, authorization cleanup pending, and a stable void operation identifier. Retry or query that operation, and retain enough evidence for an operator to resolve a permanently unclear provider outcome. The client can see that the order will not ship while the authorization release is still processing.

For every step, specify how recovery works if a process crashes: before the local commit there is no recorded intent; after commit a worker can rediscover it; after remote success but before recording the result, a stable key or status query resolves ambiguity. The outbox closes the local database/publication gap, but it does not make the provider part of the local transaction.

Compensation is not always a valid business remedy. Shipping an irreplaceable item twice cannot be made correct merely by scheduling a refund. Protect scarce inventory with conditional allocation and gate irreversible steps carefully. If the business rule forbids exposing partial completion and no compensating action can repair it, reconsider which service owns the data or use participants that can commit the required changes together.

For implementation, a small workflow can use a transactional state table, an outbox and leased workers. A durable workflow engine such as Temporal provides persisted event history and replay, but workflow code must follow its deterministic execution constraints. External calls belong in retryable activities with stable effect identities; the engine does not give a third-party API transactional rollback or unlimited deduplication.

07Orchestration versus choreography and interview explanation

In an interview I would say: “O81 has a durable coordinator record, and each remote step uses a stable operation identifier. Inventory owns the hold’s expiry and conversion. The order remains pending while authorization is uncertain. After the hold expires, a late authorization triggers voiding, not confirmation. Every outgoing step is recoverable from an outbox, and every incoming result is checked against the current workflow state.”

An orchestrated workflow puts these transitions in one explicit coordinator. An event choreography distributes reactions among services; it may reduce central coupling but makes the overall progress and compensation path harder to inspect. Either approach needs ownership of timeouts, retries and terminal outcomes.

Measure the age and count of stuck pending orders, unknown external outcomes, failed compensations and expired holds. Alert on old unfinished work, not only HTTP errors. A successful request log does not prove that the multi-step business operation finished. The design is complete when another worker can recover O81 from persisted facts without guessing what the previous worker intended.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.