How one timeout can create two real-world actions
An AI agent may correctly prepare an email, invoice, or CRM record, submit a command to an external system, and fail to receive the response in time. The orchestrator sees an error. The CRM, email provider, or accounting system may already have completed the first command. If the agent simply retries, the business does not get “higher reliability”; it gets two emails, two tasks, or two charge attempts.
This is a standard distributed-systems problem, not a special defect of language models. Agentic workflows make it more visible because the model selects tools dynamically, the chain is longer than a conventional API call, and the operator often sees only the final answer. The more steps involved—finding a customer, creating a document, obtaining approval, sending it, and recording the result—the more opportunities there are for a response to disappear after execution.
AWS describes safe retries through idempotent APIs: the client supplies a unique identifier for its intent, and the service recognises a repeat without creating another side effect. Microsoft separately warns that a broker may redeliver a message, so broker-side duplicate detection is not sufficient; the handler must also be idempotent.
For a small business, the practical conclusion is simple: “retry on error” is safe only when the retry is tied to the same business operation and the system can prove whether it has already been completed.
What counts as one operation
An idempotency key must not be regenerated for every network request. It should identify the business intent, not the delivery attempt. Examples include:
- `create-invoice / order-1842 / version-1`;
- `send-confirmation / booking-731 / English`;
- `update-deal-status / deal-558 / qualified`;
- `create-refund / payment-902 / amount-1500`.
The first and second attempts for the same command use the same key and identical parameters. If an employee changes the amount, recipient, or action type, that is a new intent and requires a new key. AWS explicitly distinguishes a retry of the same request from a new intent with similar parameters.
A hash of the whole JSON document without business context is a weak substitute. An irrelevant field-order change may generate a new key, while two deliberate identical actions might be incorrectly merged. It is safer to create an explicit `operation_id` before calling the model or external API and associate it with the data version, initiator, and target object.
Minimum architecture: intent, ledger, executor
A practical design does not require a large platform. An application database, a background worker, and adapters to external systems are enough.
1. A user or process creates an intent, such as “send booking confirmation.”
2. The application stores the business change and an outgoing-operation row in one transaction.
3. A background executor claims the ready row, checks policy, and calls the external API.
4. The same `operation_id` is passed as the idempotency key when the target supports it.
5. The result, external identifier, and execution time are recorded in the ledger.
6. After a timeout, no new operation is created: the existing one moves to an `unknown` state and is retried or reconciled.
The table can contain `operation_id`, `action_type`, `target_id`, `payload_hash`, `status`, `attempt_count`, `next_attempt_at`, `external_id`, `last_error_class`, `created_at`, and `completed_at`. Storing the full prompt is usually unnecessary; execution needs approved action parameters and a reference to the source-data version.
This ledger separates model reasoning from irreversible action. The LLM may propose a command, but a deterministic layer validates its schema, permissions, limits, and uniqueness. This is especially important for sending messages, changing prices, reserving inventory, creating payment documents, and writing to systems of record.
Why a transactional outbox matters
Without an outbox, there is a dangerous gap between two operations. The application may commit an order to the database and then crash before notifying the worker. Or it may send the message but fail to persist the new status. Either way, business data and the queue diverge.
The transactional outbox closes that gap: the business-record change and the new outgoing-operation row are committed in one transaction in the same database. A separate worker publishes or executes accumulated rows later. Microsoft recommends this pattern to maintain consistency between business data and outgoing messages.
In a small deployment, a PostgreSQL table can serve as the queue. Several workers claim batches of ready rows with locking. PostgreSQL documentation explicitly says that `SKIP LOCKED` is unsuitable for a general consistent view but useful for multiple consumers of a queue-like table: locked rows are skipped, so workers do not wait for one another.
This does not mean PostgreSQL is always better than a message broker. At tens of operations per minute, a table is often simpler to operate. A dedicated broker becomes more attractive for high throughput, multiple independent consumers, long event retention, or complex routing. An idempotent consumer is still required because at-least-once delivery permits duplicates.
Four states instead of a binary success/failure flag
The most dangerous implementation mistake is treating a timeout as proof of failure. At minimum, the workflow needs these states:
- `pending` — the command has not been sent;
- `processing` — an attempt has started and holds a lease or lock;
- `succeeded` — confirmation and an external identifier were stored;
- `unknown` — the response was lost and the external outcome is uncertain;
- `failed_permanent` — parameters are invalid or the action is forbidden, so automatic retry is pointless.
For `unknown`, reconcile first: search by idempotency key, external identifier, order number, or another stable marker. Retry only if the target confirms the result is absent. Using a payment API as its example, Stripe recommends retrying network-uncertain requests with the same key and identical parameters; changed parameters represent a new intent.
Permanent errors—an invalid address, closed deal, missing permission, or breached business limit—belong in an exception queue, not an infinite retry loop. Transient errors—timeouts, throttling, or brief unavailability—can use exponential backoff plus random jitter. Jitter prevents many operations from waking simultaneously and causing another overload.
Where the human remains
Idempotency prevents a technical duplicate, but it does not prove that the original intent was correct. If the model independently decides twice to create two refunds, two distinct keys will correctly allow both. Business constraints therefore need to run before the `operation_id` is committed:
- one active refund per order line;
- one final invoice per shipment version;
- no more than one outgoing message of a given type within a defined interval;
- amounts above a threshold require employee approval;
- changes to banking details or recipient require renewed approval;
- bulk actions require a preview and a batch-size limit.
For risky actions, the employee should see a compact action card rather than the model’s long chain of reasoning: what changes, in which system, for which object, for what amount, who requested it, and how it differs from completed operations. The human decision is recorded alongside the `operation_id`.
Observability without collecting unnecessary content
Management needs a few operational indicators rather than a stream of technical logs:
- percentage of operations completed on the first attempt;
- duplicate attempts safely suppressed;
- number of `unknown` operations and average reconciliation time;
- size and age of the exception queue;
- repeat actions stopped by business constraints;
- share of operations requiring a human and the reasons.
It is also useful to estimate the “cost of uncertainty”: employee verification time, process delay, and potential duplicate cost. This lets the company compare reliability improvements with engineering expenditure. A small company will often gain more by first protecting three expensive actions—payments, legally significant communications, and system-of-record changes—than by building a universal integration bus.
A five-day model pilot
Start with one process that has already produced duplicates or manual reconciliations.
- Day 1: list irreversible actions and all points of network uncertainty.
- Day 2: define the business operation identifier and the rules for a new intent.
- Day 3: add the operation ledger, unique constraint, and `unknown`/`failed_permanent` states.
- Day 4: enable bounded retries with backoff and a manual exception queue.
- Day 5: test “response lost after success,” “one command delivered twice,” “parameters changed,” and “worker crashed after the external call.”
Readiness does not mean a flawless demo. The system must replay every failure point and demonstrate that the external effect occurs once or is explicitly stopped for reconciliation.
What management should take away
Before connecting an agent to CRM, email, inventory, or accounting, ask the team to show the path of one command from business intent to external result. It needs a stable identifier, an atomic ledger entry, an explicit timeout state, a bounded retry policy, and a reconciliation procedure.
If the answer to “what happens when the response is lost after a successful send?” is “the agent tries again,” the automation is not ready for production. The safe answer begins: “it retries the same operation with the same key, or stops for verification.”