Why an agent does not fit inside one HTTP request

A long-running agent operation rarely resembles a normal web request. It may need to fetch documents, call a model, validate the result, contact a CRM, wait for human approval, and only then modify a system of record. If all of that is held inside one HTTP connection, the first timeout turns job status into a philosophical question: did it fail, is it still running, or did it complete while the response disappeared?

The worst case is an automatic retry. The interface does not receive a response and submits again, the agent creates another job, and the executor diligently creates a second order. The machine was extremely enthusiastic; accounting remains strangely unimpressed.

A reliable design separates job acceptance from execution. The API persists intent, returns a job identifier, and releases the connection. The job then moves through a queue, a time-bound worker lease, controlled retries, and explicit states. The model can help decide what should happen, while financial and operational changes are performed by deterministic code.

The basic flow

A small business can begin with seven components:

1. An API accepts the command and verifies the user's permissions.
2. PostgreSQL stores a `job` record with a stable idempotency key and the `queued` state.
3. The same transaction writes an event to an outbox table.
4. A dispatcher publishes that event to Redis, RabbitMQ, or another broker.
5. A worker receives the job, acquires a time-limited lease, and renews it with heartbeats.
6. Every external effect goes through a separate executor that validates schema, policy, and a unique `operation_id`.
7. The user sees job status and approves an irreversible step when approval is actually required.

The API response is not “everything is finished” but `202 Accepted` with a `job_id`. The client polls status or receives an event over WebSocket. Resubmitting the same business intent returns the existing job instead of creating its twin.

Idempotency: retry is allowed, duplication is not

Celery warns that a message can be redelivered after a worker failure and that late acknowledgement is safe only for an idempotent task. AWS makes the same practical point in its transactional outbox guidance: reliable delivery can still produce duplicate messages, so consumers must track events they have already processed.

An idempotency key should represent business intent rather than a random UUID generated for each attempt. For “create an invoice for order 841,” it could be `invoice:create:order-841:v1`. A unique database index enforces the rule. If a user double-clicks or a gateway retries the request, the system finds the existing record and returns its state.

Each level benefits from its own identifier:

  • `job_id` for the entire business operation;
  • `step_id` for one stage, such as extracting invoice fields;
  • `operation_id` for an external effect, such as creating the draft invoice;
  • `attempt` for a technical attempt that does not change business meaning.

It is unsafe to assume a model will produce the same answer next time. Idempotency is enforced by storage and the executor, not by the text generator being in a consistent mood.

The outbox closes the gap between database and queue

A classic error looks harmless: save the order, then publish a broker message. If the process crashes between the two actions, the order exists but the job does not. Reversing the order creates the mirror problem: the message is published while the database transaction later rolls back.

The transactional outbox writes the business change and its event in one PostgreSQL transaction. A separate process reads committed outbox rows and publishes them to the broker. It may still publish twice, so an idempotent consumer remains necessary. The outbox does not eliminate duplicates; it prevents an event from disappearing into the gap between two systems.

At small scale, the first pilot may not need a separate broker. PostgreSQL supports `FOR UPDATE SKIP LOCKED`, allowing multiple workers to claim different rows without waiting for rows another worker has locked. The documentation explicitly notes that this is useful for a queue-like table but provides an inconsistent view and is unsuitable for general-purpose reporting. A work queue and an analytics table remain separate concerns even when they share one database.

Leases, heartbeats, and returning unfinished work

A worker should not own a job forever. It records `worker_id`, `lease_until`, and the last heartbeat time. While work continues, the lease is renewed. If the process dies, another worker can return the job to the queue after expiry.

The lease must be longer than a normal heartbeat interval, but not so long that abandoned work remains stuck for hours. For a step that usually takes 30 seconds, a pilot might start with a heartbeat every 10 seconds and a lease of 60 to 90 seconds, then tune the values from measurements. These are starting assumptions, not universal standards.

Before repeating an operation, the new worker checks the step journal and the external system. If an invoice already exists, it records success for the existing `operation_id` rather than creating another one. The system reconciles with reality before acting, a habit that also helps humans on Monday mornings.

Retry only what can recover

A retry is appropriate for a temporary network error, rate limit, or brief dependency outage. A schema violation, forbidden action, or missing mandatory field will not repair itself on attempt five.

A retry policy should include:

  • an explicit list of transient errors;
  • exponential backoff and jitter so all workers do not wake at once;
  • a maximum number of attempts;
  • an overall job time budget;
  • a dead-letter or review queue for exhausted attempts;
  • a safe way to cancel a step that has not started.

Celery also warns that uncontrolled requeue behavior can create an infinite message loop. “Retry everything forever” is therefore not a resilience policy; it is a way to turn one problem into wholesale inventory.

Where the model stops and the executor begins

An LLM can classify a request, propose a plan, extract fields, and prepare tool arguments. Before changing CRM, ERP, or banking data, those arguments need a deterministic action gateway.

It verifies:

  • whether the user's role and job type may use the tool;
  • whether JSON matches a strict schema;
  • whether evidence points to a source document or record;
  • whether amount, quantity, or discount stays within policy limits;
  • whether the same `operation_id` has already run;
  • whether human approval is required.

Irreversible work is easier to control when split into `prepare` and `commit`. The agent creates a draft and evidence, a person reviews the differences, and the executor performs one write after approval. Approval must be bound to a draft version: if data changes, old consent cannot authorize a new operation.

Economics: count accepted work, not model calls

Consider an illustrative workflow with 300 requests per day. Manual routing and draft preparation average eight minutes, or 40 labor hours. If an agent prepares each case and review takes two minutes, approximately 10 labor hours remain. The theoretical opportunity is 30 hours per day before model, infrastructure, support, and exception-handling costs.

However, a repeated call that created a duplicate is not productivity. Better pilot metrics are:

  • cost per human-accepted job;
  • share of jobs accepted without correction;
  • number of suppressed duplicates;
  • age of the oldest queued job;
  • share of expired leases;
  • average and 95th-percentile attempt count;
  • time waiting for approval;
  • cost and time to resolve dead-letter items.

These figures are illustrative, not the result of a particular company. Real economics must be measured on the company's own traffic and include the cost of errors. One duplicate payment can consume the savings from a thousand correctly processed messages.

A two-week pilot

During week one, choose a process with no automatic external action. The API creates a `job`, the worker produces only a draft, and a person accepts or corrects it. Kill a worker mid-step, submit the same command twice, and temporarily make one dependency unavailable.

During week two, add one reversible tool and one approval-gated step. Test lease expiry, backoff, cancellation, a retry after an unknown external response, and recovery from the outbox. Verify separately that old approval cannot apply to a modified draft.

Readiness is not a perfect happy-path demo. The system must demonstrably survive a retry, a worker crash, and a lost response without creating a second business effect. At that point, the agent becomes part of a controlled process rather than a very fast generator of identical task cards.