Design durable agent workflows

01Problem and scope

A durable agent system combines model-assisted decisions with a persistent workflow and explicitly authorized tools. The database retains accepted decisions and model/tool results when a worker is replaced. The model’s current prompt is neither the permanent task record nor proof that an action is authorized. This design permits research and drafting but requires trusted approval of the exact external purchase proposal. A procurement workflow comparing three approved vendors for ten laptops is the example, including a half-hour approval wait during which no worker process needs to remain assigned.

Workflow, agent and tool definitions

A workflow defines transitions such as research → draft → approval → submit. An agent uses a model to choose information or tools within permitted boundaries. A tool is an application operation with a schema, permission check and result. Model text proposing a tool call is not execution authority. A session event log is the durable history of accepted observations, proposals, approvals and outcomes; it is separate from the model's limited context window.

Clarify autonomy and external authority

Candidate: “May the assistant spend autonomously? Must code run in a sandbox? Which decisions need approval?” Interviewer: “It may research and draft, but the requester must approve the exact purchase order.” Candidate: “I will start with a fixed workflow around model-assisted research, make side effects explicit, and persist waiting state without keeping a worker alive.”

02Functional requirements

Approval must identify the exact action the user reviewed. A canonical proposal uses one defined representation of its action fields, and its hash is a fingerprint of that representation. The trusted approval record binds the approver to that fingerprint; a matching hash identifies content but does not, by itself, grant permission to submit it.

1. Start a task. One logical session per authenticated request identity.
2. Research. Only allowed read tools, recorded inputs/results and bounded budgets.
3. Create a draft. Immutable proposal with vendor, quantity, amount, currency and destination.
4. Obtain approval. Trusted user approval bound to exact proposal hash and scope.
5. Submit an action. The action gateway checks current permissions, budget and approval before durably authorizing submission.
6. Recover a session. Replay recorded decisions/results without repeating confirmed external effects.
7. Cancel a task. Stop future scheduling; disclose already in-flight or completed effects.

Task states and externally verified results

The service starts a task, shows progress, preserves research artifacts, requests a reviewable approval, submits only an authorized exact action, and records a verifiable external result. The requester can cancel or inspect the event history. A worker can disappear without erasing a draft or approval. A task may enter needs-attention when an external action's outcome cannot be safely determined; “keep trying” is not always a valid recovery strategy.

Constraints and exclusions

Exclude unrestricted shell access, blanket autonomous spending and arbitrary website authority. An optional isolated sandbox can run calculations or transform files, but it receives no production credentials through its environment or filesystem. Human approval is not a vague “yes” attached to any future draft; changing vendor, quantity, currency or destination changes the proposal identity and requires a new approval. The system distinguishes task completion from creating a draft, and shows a purchase-order ID only after confirming it with the destination system.

03Non-functional requirements

1. Control latency. Task-start p95 below 500 ms; active progress updates within five seconds when new durable events exist.
2. Scheduling latency. Schedule approved actions within five seconds under admitted load. Model/tool execution latency is separate and may be seconds or minutes.
3. Availability and retention. Target 99.9% task-control availability. Keep waiting tasks for an illustrative thirty-day active horizon; tenant policy and procurement obligations determine event/artifact retention.
4. Durability. Accepted events, approvals and action identities survive one node or availability-zone failure through replicated session authority. Replaceable workers/sandboxes are not the sole copy of accepted artifacts.
5. Disaster recovery. Define and test a separate regional recovery point and restore objective. Reconcile external actions before replaying an old database backup, or recovery itself can duplicate purchases.

Authority and side-effect invariants

An epoch is the session’s worker-ownership version. When a replacement takes over, the authority advances that version and rejects writes from older owners. Action admission is a separate database operation that verifies the exact approval, permissions, cancellation status and reserved budget before recording permission to submit.

Recording action admission allows a dispatcher to send the purchase; the external procurement service then decides whether it creates the order. After dispatch, a local cancellation may be too late to prevent that external action. If the external API has neither idempotency nor lookup, human reconciliation may be necessary instead of blind retry.

04Capacity estimates

Assume 100,000 task starts/day, ten model calls and fifteen tool calls per task. Starts average 100,000 / 86,400 = 1.16/s, model calls 11.6/s and tool calls 17.4/s. With 3,000 input and 500 output tokens per model call, average demand is about 34,722 input tokens/s and 5,787 output tokens/s. Tenfold bursts and long task tails require admission budgets beyond those averages.

Each task emits 200 event records averaging 2 KB: 400 KB/task and 40 GB/day, about 1.2 TB over thirty days before indexes, replicas and artifacts. Large documents, screenshots and draft files belong in object storage with checksums and access rules; repeating a 10 MB vendor PDF in twenty events would waste 200 MB for one task and bloat replay.

Elapsed task time includes waits during which no worker is executing model or tool work. Release orchestration leases during approval/timer waits and wake from durable events. Sandbox cost depends on how many tasks need code and for how long, so allocate one lazily rather than automatically provisioning a container for every read-only task. Track completed-task cost, including failed/repeated calls, rather than only a cheap average model call.

05APIs and contracts

Start a task under trusted identity

The requester calls POST /v1/tasks with {"requestId":"request-81","goal":"Compare approved vendors and draft an order for 10 laptops"}. Return session s81 and a progress route. The authenticated server supplies tenant and user authority; the goal string does not grant capabilities. Event pages use a durable session sequence, not an in-memory websocket offset.

Progress, approval and action APIs

Retry identity and approval changes

An idempotent request can be repeated without creating an additional business effect. Here a retry reuses the same action identity and either retrieves the recorded outcome or asks the destination to resolve that identity. The external action key submit-s81-p8 remains stable across worker crashes; worker attempt IDs are separate. A reused start key with changed goal/settings conflicts. Approval requires current authority, expiry and the exact proposal hash. If a draft changes, p9/h9 is a new reviewable object. External submission retries obey the destination's documented idempotency retention and lookup behavior; no generic HTTP retry policy can manufacture that guarantee for an arbitrary service.

06Data model and access patterns

Recovery replays accepted history; it should not repeat every external call that produced that history. An activity is a recorded unit of model or tool work with an input identity and saved result. Such work can be nondeterministic—running it again may return something different—so the workflow reuses an accepted result when rebuilding its state.

Durable entities

• Session. Session(sessionId,tenantId,userId,state,workflowVersion,epoch,leaseUntil,budget,cancelRequested,lastSequence) owns progress.
• Accepted event history. Event(sessionId,sequence,eventType,payloadOrArtifactRef,producer) is append-only accepted history.
• Nondeterministic activity. Activity(sessionId,activityId,inputHash,state,resultRef,attempts) records nondeterministic model/tool calls.
• Proposal and approval. Proposal(proposalId,canonicalPayloadHash,artifactRef) and Approval(proposalId,hash,approver,scope,expiresAt) bind review to content.
• External action. Action(actionId,proposalId,validatedArgs,state,externalKey,externalResult,admissionEpoch) owns side effects.

Accepted-state transaction

An accepted event is one the session authority has committed, rather than merely a result a worker observed. Materialized session state is the current status computed from those events, such as WAITING_APPROVAL. Both must change together so a replacement worker sees a consistent history and status.

The session authority transaction appends an event and updates materialized session state together. It checks the current worker epoch, so a paused old worker cannot append competing accepted decisions after reassignment. An outbox schedules future work with that same commit. Approval events can be written only by the trusted authenticated approval API, not by a model-generated “approval” field or a sandbox file.

Artifacts, checkpoints and secret isolation

Artifacts use immutable object keys and checksums; record a reference only after bytes exist. Checkpoints contain workflow state and the last applied event sequence, while the durable log retains enough later events to reconstruct accepted transitions. Summaries help model context selection but are not the authority for action history. Credential broker secrets stay outside the event payload and sandbox. Tenant/session keys partition storage and authorization; per-tenant tool/budget records may require a separate guarded reservation when a session admits an external action.

Enforce artifact immutability

Enforce artifact immutability at storage: use create-only writes or store the exact provider object version and verify its digest on read. A canonical payload uses one defined field representation and ordering so the same action produces the same hash. The approval API displays and hashes that stored canonical action payload; it does not approve an editable URL or trust a sandbox's claimed checksum. A sandbox can upload tentative bytes only to its scoped staging area. A trusted artifact-ingestion step validates size/type/digest and current session epoch before accepting a reference into history. Reusing an upload grant must not replace the bytes behind an approved proposal.

Serialize exactly the approved action

07Basic working design

Fixed workflow with narrow tools

Start with one API, one SQL database, a model endpoint and narrow adapters for approved vendor search and procurement. The workflow is RESEARCHING → DRAFTING → WAITING_APPROVAL → SUBMITTING → COMPLETED, with FAILED, CANCELED and NEEDS_ATTENTION outcomes. The model helps choose search queries and summarize vendor facts; deterministic code owns allowed transitions and tool schemas.

Record accepted results and approvals

1. Create session and wake-up. The requester's task transaction creates s81/version 3 and an outbox wake-up.
2. Persist accepted work and proposal. The orchestrator records each accepted model/tool result, builds draft p8, stores its immutable artifact and commits WAITING_APPROVAL.
3. Release the waiting worker. Save the waiting state and return the worker to the pool; another task can use it until an approval event arrives.
4. Resume after trusted approval. The requester later approves p8/h8 through the API; a new wake-up resumes the workflow and validates submission.
5. Recover from SQL state. This baseline already supports a restart because the session state is in SQL, not only in a process variable.

Capacity and external-action recovery

The simple architecture can handle roughly 1.16 task starts/s if its slow calls run asynchronously and waiting sessions consume no worker. It is not necessary to distribute every tool adapter immediately. The hardest requirement is already present: an external purchase may succeed while its response is lost. We need a stable action record and destination idempotency/lookup from the beginning; adding more worker nodes later cannot repair a missing action identity. Backups and test tasks must verify recovery at that boundary, not merely restart the application cleanly.

08Find the baseline flaws

09Improve the design, step by step

1. Persist event/activity history and immutable artifacts

• Trigger: A restarted worker must recover earlier results, including tasks that spent a long time waiting.
• Mechanism: Save model/tool results in a session log with checkpoints. Recovery reads those results instead of repeating completed work; large artifacts live in referenced object storage for recovery and audit.
• Benefit, cost and alternative: Costs are storage, schema evolution and privacy retention. A simple state row may suffice for tiny fixed tasks, but it must still preserve proposals and action outcomes; a conversational summary alone cannot replace them.

2. Separate stateless orchestration from leased work queues

• Trigger: Slow providers and human approvals leave a dedicated worker idle for much of a task.
• Mechanism: Timers and approval events queue ready work. A bounded worker pool claims a session epoch, runs one activity or state change, then releases it. Scale these workers with active demand.
• Benefit, cost and alternative: Costs are duplicate delivery, epoch checks and queue lag. Holding one dedicated process per task is simpler only for short bounded jobs with small concurrency.

The tool gateway decides whether an operation is permitted; the credential broker controls the secret needed to contact its destination. Separating those responsibilities lets an authorized adapter make the approved call without placing reusable production credentials in model context or sandbox memory.

3. Put tools behind a least-privilege gateway and credential broker

• Trigger: Untrusted source text and optional code execution trigger deterministic schema/policy checks, trusted approvals and narrowly scoped adapters.
• Mechanism: A sandbox has no ambient production token; the broker uses credentials outside model-readable state. The gateway can reject a forbidden operation without giving the model or sandbox credentials that could bypass that rejection.
• Benefit, cost and alternative: Costs include adapter development and more explicit authorization. Broad shell/network access is rejected because it bypasses the reviewable tool boundary.

4. Add explicit action reconciliation, versioned replay and tenant budgets

• Trigger: Unknown external outcomes, deployments and runaway loops trigger stable external keys, NEEDS_ATTENTION states, workflow-version pinning and per-task/provider quotas.
• Mechanism: This limits duplicate effects and makes old sessions resumable.
• Benefit, cost and alternative: Costs are destination-specific recovery logic, migrations and intervention. A fully deterministic fixed workflow is preferable when it solves the problem; open-ended agent choices are added only where they materially improve task completion.

Each mechanism has a specific job: saved history supports recovery, authorization permits actions, and the provider’s retry contract prevents duplicate purchases. A sandbox does not replace those checks.

10Detailed architecture

Session authority and leased workers

The task API authenticates the requester, writes session/approval/cancellation events through replicated authority and serves progress/artifacts. A durable outbox and timer/approval wake-up queue schedule work. Stateless orchestrators claim a session epoch, load a checkpoint plus later events, and call a model for a bounded decision. Model output is a proposal, not a direct connection to procurement.

Tool and credential boundary

A tool gateway validates the proposed operation against schema, current tenant/user rights, workflow state, approved proposal hash, budget and cancellation status. Read-only vendor adapters use scoped credentials from a broker; purchase submission uses a separately admitted Action record. An optional sandbox executes calculations or file transformations without access to broker secrets. Accepted artifacts are copied to durable storage before being referenced in the event log.

External dispatch and reconciliation

The action dispatcher sends stable identities to the external procurement API. A reconciliation worker resolves UNKNOWN actions by looking up the original action key at the procurement service or retrying that same key when the destination documents that as safe. It does not ask the model to guess whether po902 exists. Evaluation/audit consumers inspect permitted event metadata and outcomes without governing production authorization.

Ordering and budget partitions

Session partitions own one task's event order. Tenant-wide budget reservations may be another authority with their own reservation/commit protocol; the action is not dispatched until the required reservation succeeds and is recorded. Synchronous user requests end after durable task/approval commits, while research, execution and reconciliation run asynchronously. A worker can die without taking the session or its credential store with it.

11Write path and acknowledgement

Save accepted model/tool results and proposal versions before advancing the workflow. Order approval, submission and cancellation checks in the database; generated text cannot grant permission to act.

Numbered research-to-action trace

1. Create durable session and wake-up. The requester starts request-81. A transaction creates s81 with workflow version 3, read-tool capabilities, budgets and RESEARCHING state, then appends a wake-up outbox event.
2. Claim an epoch and run an activity. Worker W1 claims epoch 41 and loads accepted history. It invokes a model activity with a stable activity identity; the model proposes searchApprovedVendors with validated search arguments.
3. Authorize tools and record results. The tool gateway checks tenant scope and read permission, executes the adapter, and records the accepted result/artifact under the activity identity. A lost unrecorded model result may require a new attempt, but no external purchase has occurred.
4. Persist the exact proposal. The workflow produces proposal p8: vendor V2, ten laptops, total 12,000 in the specified currency, destination office O4. Canonical serialization includes every action-relevant field, yielding hash h8. Store the artifact before committing WAITING_APPROVAL and its event.
5. Obtain trusted approval. W1 releases compute. The requester reads the concrete draft and approves p8/h8. The approval API checks the current approval authority, scope and expiry, then commits a trusted approval event and wake-up.
6. Admit one authorized action. A worker resumes version 3, verifies the exact unchanged proposal and asks the action gateway to admit submit-s81-p8. The gateway checks cancellation, current permissions, approval and budget before atomically creating the admitted Action/outbox.
7. Confirm or reconcile the external result. The dispatcher calls procurement with the stable action key. On confirmed success it records po902 and advances s81 to COMPLETED with a linked result. If the call outcome is unknown, it records UNKNOWN and reconciles; it does not manufacture completion or a new purchase identity.

Changed proposal requires new approval

Changes to p8 create a new proposal/hash and invalidate reuse of the old approval.

12Read and delivery path

Show progress from saved workflow records, including waits and unknown outcomes. Recover an external action through the provider’s lookup or documented same-key retry.

Numbered progress and cancellation flow

1. Authorize durable progress. The requester opens s81 and requests events after sequence 18. The API authorizes tenant/user access and returns ordered durable progress, including WAITING_APPROVAL and the immutable p8 reference if applicable.
2. Display the exact reviewable artifact. The UI loads p8 through an authorized artifact endpoint and displays vendor, quantity, total, currency and destination, not just the model's short summary. The approval hash identifies these exact reviewed terms.
3. Recover accepted history. After a worker crash, replacement W2 claims a higher epoch, loads a valid checkpoint and replays later accepted events. Completed model/tool activities yield their recorded results; they are not automatically executed again.
4. Rebuild context from recorded facts. W2 rebuilds the model's working context from selected history and artifacts. A compaction summary can save tokens, but W2 can still inspect the original approval/action events when deciding what remains to do.
5. Interpret action state. If Action submit-s81-p8 is CONFIRMED, show po902 and never submit again. If UNKNOWN, show the uncertainty and invoke the deterministic reconciliation policy. If WAITING_APPROVAL, release the worker and wait for a trusted event.
6. Report the cancellation boundary. Cancellation is similarly a durable event. The UI reports whether it prevented admission, stopped future research, or arrived after a submitted/unknown action. Reversal of a confirmed purchase is a new explicitly authorized operation.

Pagination and artifact authorization

Event pagination uses the session's committed sequence prefix. Artifact access rechecks current permissions; a shared progress link is not a credential. Large histories use checkpoints and indexed event slices, while retention policy preserves enough action/approval evidence to support reconciliation and the product's audit requirements.

13Correctness deep dive

Local state and remote side effect

The difficult failure occurs when procurement creates the order but our database has not yet recorded the response. The two systems do not share one transaction. The Action row must represent uncertainty, and the destination's actual idempotency/lookup capabilities determine what recovery can prove.

Action state machine

Lost response timeline

What an epoch can fence

The session authority rejects W1's stale epoch for accepted local transitions after W2 takes ownership. The tool gateway also checks current action/session authority before admitting new effects. Yet an already in-flight external call cannot be recalled by changing an epoch; that is why external identity remains necessary.

Cancellation ordering

If cancellation commits before action admission, admission fails. If admission wins first, cancellation may arrive after procurement has acted. The UI must report the confirmed or uncertain side effect rather than promise “nothing happened.” A refund/cancel-order action, if supported, is a separate authorized workflow with its own idempotency key.

Same-authority budget reservation

Separate-authority reservation protocol

If budgets move to another authority, reserve there first under the stable actionId and a payload fingerprint. That durable reservation must remain held until the action authority records an irreversible admitted-or-abandoned decision. Reconciliation releases it only after proving the action was never admitted or definitively had no effect; a timeout/expired worker lease alone cannot release it while an admitted external call may spend. Dispatch requires the reservation's recorded token and the action decision. Cross-authority failure may hold capacity longer, which is preferable to double-spending an assumed released budget.

14Failure and recovery

15Operations, security, and cost

Untrusted evidence and tool security

Quality and recovery metrics

Measure completion quality, evidence accuracy, valid approval handling, duplicate external effects, UNKNOWN-action age, event/queue lag, stale-epoch rejections, intervention rate, token/tool work and time waiting for humans. A task that quickly emits a confident draft but never safely submits is different from a completed approved procurement. Evaluate adversarial documents, altered proposal fields, expired approvals and provider timeouts, not only successful happy-path demos.

Active-work and storage cost

Cost follows actual active work and retained artifacts. Two thousand waiting sessions can be a few database rows each, while two thousand reserved containers consume resources for no computation. Create a sandbox only when an activity needs it, then remove it when the work ends. Checkpointing reduces replay reads, but an overly frequent full-history snapshot multiplies storage; store compact state plus an event position and immutable artifact references.

Workflow-version compatibility

Deploy workflow version 4 with explicit compatibility. Existing version 3 sessions remain on supported code or execute a documented migration; replay must not reinterpret old events under a new operation order. Model/prompt version changes are recorded for new activities, while accepted old results remain historical facts. Test every crash boundary, restored backups, stale workers and expired idempotency windows before claiming duplicate-effect safety.

Temporal replay and versioning qualifications

Temporal is one implementation option for durable timers, recorded Activity outcomes and replay; model/tool I/O belongs in Activities rather than deterministic Workflow code. Activities can execute more than once if completion was not recorded, so destination idempotency is still required. For workflow evolution, use supported Deployment Version routing or patching and replay tests. The official Go versioning page warns that support for the pre-2025 experimental Worker Versioning method was scheduled for removal in March 2026; do not build this September 2026 design around that legacy API or infer compatibility from an old tutorial. Pin the actual server/SDK combination and follow its migration guidance.

16Decision ledger and limitations

Decision table

Checkpoints and evidence retention

A checkpoint accelerates replay but is not a replacement for the evidence needed to reconcile side effects. A context summary optimizes model input but cannot substitute for trusted approvals and Action records. Recovery reads the saved model result. Calling the model again—even with a similar prompt—can produce a different decision because sampling or versions changed.

Safe parallel research versus submission

Parallel research can reduce latency if independent read tools are safe and budgets allow it. Parallel purchases are different: each needs separate authority, stable identity and budget reservation. Broad “agent autonomy” is not a technical guarantee. The design deliberately accepts paused needs-attention states where external outcomes cannot be established. That is more honest and safer than converting every timeout into a new potentially duplicating action.

17Interview closing

Rehearse the architecture and contract

“I designed a durable procurement workflow with model-assisted research. Each session survives worker replacement because accepted events, activity results and proposal artifacts are stored outside the worker. Research uses narrow read tools. Each immutable proposal records the exact vendor, quantity, total, currency and destination. Approval is a trusted event bound to that proposal hash, and the action gateway rechecks current authority and budget before submission.

Defend the critical boundary

“The critical failure is an external purchase that succeeds before the worker saves its response. I retain the original external action identity, represent UNKNOWN explicitly, and reconcile through the destination's documented lookup or idempotent retry. Local epoch fencing prevents stale accepted decisions, but external idempotency handles calls already in flight. Cancellation cannot erase a purchase that already happened.

State the cost and next measurement

“The workload has many model/tool operations but long approval waits, so workers are leased and released rather than pinned to every session. My next tests are response-loss recovery, altered-proposal approval rejection and prompt injection that attempts to bypass the tool gateway.”

Answer the follow-up

If the interviewer asks for unrestricted multi-step autonomy, first identify which decisions benefit from model choice and which external effects require deterministic authority. Expand capabilities one reviewed boundary at a time, with budgets, observability and recoverable action identities. A larger context window alone does not make the workflow durable or authorized.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.