Orchestrating agents in production

A single agent answering a question in a chat window is a demo. An agent that reads a supplier email, checks stock in the ERP, raises a purchase order, waits two days for a buyer to approve it and then books a delivery slot is a distributed system. It has state that must survive a deploy, side effects that must not happen twice, steps that can fail halfway, and a human somewhere in the middle who may answer on Thursday or not at all.

Most agent frameworks make the first case easy and leave the second to you. The result is a familiar failure: the prototype works, then the first production week brings a pod restart mid-run, a retried tool call that creates a duplicate order, and an approval request that nobody can find because it lived in process memory. A better model fixes none of these. They are orchestration problems, and the industry solved most of them years ago for ordinary workflows.

This article sets out how we orchestrate agents for production work: which coordination pattern to pick, why the agent loop belongs inside a durable workflow, how to make retries safe, how to undo work with compensation, and how to wait on people without losing the thread. It is the thinking behind our Monarch Bridge orchestrator and Saga engine accelerators, but none of it depends on our tooling.

Three shapes of orchestration

Nearly every multi-step agent system we see is one of three shapes, or a nesting of them. Anthropic's guide to building effective agents draws a useful line between workflows, where code defines the path, and agents, where the model chooses the path. Microsoft's agent orchestration patterns catalogue makes the same point from the other side: use the least coordination that solves the problem, because every added layer costs latency, tokens and debuggability.

A supervisor is a model that receives the task, delegates to specialist workers (a catalogue agent, a pricing agent, a returns agent) and decides when the job is done. It is flexible and copes with tasks whose shape you cannot predict. It is also the hardest to test, because the route through the workers changes from run to run.

A planner-executor splits thinking from doing. The planner produces an explicit plan, a list of typed steps with arguments, and an executor runs them one by one with ordinary code. The plan is an artefact: you can log it, validate it against a schema, show it to a person for approval, and re-plan from a known point when a step fails.

An explicit graph is a state machine whose nodes may call a model but whose edges are written by engineers. Classification picks a branch; extraction fills a form; a deterministic rule decides whether to escalate. The model contributes judgement inside nodes and never invents control flow.

Figure 1. Supervisor, planner-executor and explicit graph, in increasing order of how much control flow lives in code.

Pattern	Good for	Costs	How you test it
Supervisor	Open-ended research and triage where the route is unknown	Variable token spend, non-repeatable traces, harder authorisation	Outcome evals over many runs; budget and step caps
Planner-executor	Multi-step operations where the steps come from a known vocabulary	A second model call to plan; plans can be wrong in plausible ways	Validate plans against a schema; golden plans for common requests
Explicit graph	Repeatable business processes: intake, claims, reorders, onboarding	Less flexible; new cases need new edges	Unit tests per node, path tests per branch, like any state machine

Our default for business processes is the explicit graph, with a planner-executor inside the one or two nodes that genuinely need open-ended reasoning. We reach for a supervisor only when we cannot enumerate the steps in advance, and even then we cap it with a step budget and a tool allow-list. The more of the path that lives in code, the more of the system you can review, test and explain to an auditor.

The agent loop is a workflow, so run it as one

Whatever the pattern, an agent run is a sequence of steps that read and write external systems over minutes, hours or days. That is the definition of a long-running workflow, and long-running workflows need durable execution: the state of the run is persisted after every step, so a crash, a deploy or a scale-down resumes the run from the last completed step instead of restarting it or losing it.

There are two mature ways to get this. Workflow engines such as Temporal record an event history for every run and rebuild state by replaying it; completed activities are not re-executed during replay, their recorded results are reused. Agent frameworks such as LangGraph persist a checkpoint after each graph step, and the LangGraph durable execution guide is explicit that this only works if your nodes are deterministic and idempotent and side effects are wrapped as tasks. Either way the rule is the same, and it is the most important rule in this article.

Model calls and tool calls are side effects. Record their results; never recompute them on resume. A model is not deterministic. If a run resumes and calls the model again for a step it has already completed, it may get a different answer and take a different branch, and your history no longer describes what happened. In Temporal terms the model call is an activity, never workflow code. In a checkpointing framework the model call sits inside a task whose output is saved.

Figure 2. After a crash the run replays recorded results and continues at the first incomplete step; nothing already done is redone.

Determinism has a cost that teams discover late: you cannot freely change workflow code while runs are in flight. A run that started under version 3 of the graph and resumes under version 4 must still replay correctly. Temporal handles this with explicit versioning markers; with checkpointing frameworks you usually version the graph and let old runs finish on the old definition. Plan for it before your first approval wait stretches over a release.

Retries that do not duplicate work

Durable engines retry failed steps automatically. Temporal's default activity retry policy backs off exponentially from one second with a coefficient of two and keeps going until a timeout or a non-retryable error stops it. That default is right for flaky networks and wrong for anything with a side effect, unless the side effect is idempotent.

The classic failure looks like this. The executor calls create_purchase_order. The ERP creates the order and starts to respond. The worker dies before the response arrives. The engine, correctly, retries the step. The ERP, also correctly from its point of view, creates a second order. Temporal's own error-handling guidance names exactly this case and tells you to make activities idempotent.

We do three things on every write.

Derive an idempotency key from the run, not the attempt. Run ID plus step ID is stable across retries, so the downstream system can recognise the repeat. Payment APIs have done this for years; the Stripe idempotency key documentation is a good model of the contract.
Classify errors before retrying. Timeouts and 503s are retryable. Validation failures, permission denials and "insufficient funds" are not, and retrying them only burns budget and clutters the audit trail. Mark them non-retryable at the tool boundary.
Separate model failures from tool failures. A malformed tool call from the model is not a transient error. Feed the validation message back to the model once or twice, then fail the step to a human rather than looping.

Here is the shape of a write step, sketched in Python against a generic durable engine. The details vary by framework; the structure should not.

@activity(retry=RetryPolicy(max_attempts=5, backoff=2.0,
                            non_retryable=["ValidationError", "PermissionDenied"]))
def create_purchase_order(ctx, draft: PurchaseOrderDraft) -> PurchaseOrderRef:
    key = f"{ctx.run_id}:{ctx.step_id}"          # stable across retries
    existing = erp.find_order_by_key(key)        # downstream dedupe, if supported
    if existing:
        return existing
    return erp.create_order(draft, idempotency_key=key)

If the downstream system has no idempotency support, the step must check before it writes: look up an order with this external reference, and only create one if none exists. That check-then-act is racy in general, but inside a single durable run with one active worker per run it is usually good enough. Where it is not, put a thin adapter in front of the system that stores keys itself.

Compensation when a run fails halfway

An agent that touches three systems cannot wrap them in one database transaction. When step four fails after steps one to three succeeded, you need to undo the earlier work in business terms: release the reserved stock, cancel the draft order, tell the supplier to ignore the earlier message. This is the saga pattern, described by Garcia-Molina and Salem in their 1987 paper "Sagas": a long-lived transaction is a sequence of local transactions, each paired with a compensating transaction that semantically undoes it.

Take an illustrative reorder flow for a distributor. The agent reserves stock at a second warehouse, creates a purchase order for the shortfall, and books an inbound delivery slot. Booking fails because the dock calendar is full for a fortnight. The orchestrator now runs compensations in reverse order: cancel the purchase order (or, if the supplier has already confirmed it, raise a cancellation request and flag it for a buyer), then release the reservation. Each compensation is itself an idempotent, retryable step, because compensations fail too.

Saga for a reorder. The forward path runs left to right: reserve stock, create purchase order, book delivery slot. Book delivery slot fails, marked with a cross. An orange compensation path runs right to left underneath: cancel purchase order, then release reservation. An amber box notes that a confirmed purchase order cannot be cancelled automatically and goes to a buyer for approval. — Figure 3. Each forward step registers its compensation; on failure they run in reverse, and anything that cannot be undone automatically goes to a person.

Two design rules make sagas workable with agents. First, register the compensation at the moment the forward step succeeds, with the identifiers it needs, so the undo does not depend on the model remembering what it did. Second, accept that some actions have no compensation. An email cannot be unsent; a payment can be refunded but not un-made. Put those steps last in the sequence, behind an approval if the stakes warrant it. The order of steps is a design decision about blast radius, not just a reflection of business logic.

Waiting on humans

Human approval is where naive orchestration breaks most visibly. A run that blocks a thread waiting for a click does not survive a deploy, and a run that stores "pending approval" in memory disappears with the pod. In a durable engine, waiting is a first-class state: the run persists, the worker is released, and an external signal resumes it. LangGraph's interrupt mechanism saves the graph state and waits indefinitely for a resume command on the same thread; Temporal does the same with signals and durable timers.

The mechanics are the easy part. What we have learned to get right is the content of the wait.

Snapshot what the human approves. The approval request carries the exact proposed action and the evidence behind it: the draft order, the stock figures, the model's stated reason. The approver approves that snapshot, and the snapshot hash is stored with the decision.

Re-validate on resume. Two days is long enough for prices to change and stock to arrive. When the run resumes, it re-checks the preconditions. If they no longer hold, it does not act on a stale approval; it re-plans or asks again.

Give every wait a deadline and an owner. A durable timer escalates to a second approver, or cancels and compensates, after a fixed period. An approval with no timeout becomes an orphaned run that holds a reservation forever.

This is also where autonomy levels become concrete. In our stack each action type is configured as Assist (the agent drafts, a person does), Approve (the agent acts after sign-off) or Autonomous (the agent acts and reports). The orchestrator, not the prompt, enforces the level: an Approve action physically cannot execute until the signal arrives.

See what the orchestrator did

Durable history gives you a replayable record of each run, but you also want traces that line up with the rest of your telemetry. The OpenTelemetry semantic conventions for generative AI define span names and attributes for model calls, agent invocations and tool executions. We emit one trace per run, one span per step, and child spans for each model and tool call, with the run ID and step ID as attributes so that a support engineer can go from a duplicate-order complaint to the exact retry that caused it.

Record token usage and cost per step, not just per run. The first question after a cost spike is always which step looped, and run-level totals cannot answer it.

Where this approach breaks

Durable orchestration is not free. A workflow engine is one more stateful system to run, upgrade and back up, and its history store grows with every model response you record. For agents that pass large documents between steps, store the payload in object storage and keep a reference in the history, or replay becomes slow and the history store expensive.

Determinism constraints are a real tax on iteration speed. Teams used to editing a prompt and redeploying find that a change to graph structure needs a version gate while old runs drain. If your runs are short (seconds, no human waits, no irreversible writes), a plain queue with idempotent consumers is simpler and we would use that instead.

Compensation only covers what you modelled. When an agent is allowed to choose tools freely, as in a supervisor pattern, you cannot pre-register an undo for every possible action. That is one more reason we keep write tools behind explicit graph nodes and let open-ended agents read and propose rather than act.

Finally, idempotency depends on the systems you call. Plenty of legacy ERPs and supplier portals have no notion of an idempotency key and no reliable lookup by external reference. There the honest answer is an adapter that owns deduplication, or an approval gate on the write, or both. We would rather put a person in front of a non-idempotent write than pretend a retry policy makes it safe.

A checklist before an agent touches production systems

Can you draw the run as a graph, and is every write tool on an edge you wrote rather than one the model chose?
Does every model and tool call run as a recorded step, so a resume never recomputes it?
Does every write carry an idempotency key derived from run ID and step ID, and are non-retryable errors marked as such?
Does every forward step with a side effect register its compensation, and are uncompensatable steps last and gated?
Does every human wait have a snapshot, a re-validation on resume, a deadline and an owner?
Can you go from a business complaint to the exact span, attempt and model output that caused it in under ten minutes?
Do you have a plan for changing the graph while runs started on the old version are still waiting?

If any answer is no, fix that before adding another agent. A second agent multiplies whatever orchestration gaps the first one has.