Crash-Safe AI Agents: Checkpoints Without Duplicate CRM or ERP Actions

Why “retry the step” is more dangerous than it sounds

An AI agent rarely stops at producing a chat response. In a business workflow, it reads a request, retrieves data, proposes a decision, updates a CRM record, creates an invoice in ERP, sends an email, or assigns a task. As soon as an external side effect is involved, an ordinary retry is no longer harmless.

A common failure looks like this: the agent successfully creates an order but loses the ERP response because the network connection drops. The orchestrator sees an unfinished step and runs it again. Unless the receiving system recognizes the duplicate, the business ends up with two orders, two tasks, or two messages.

Checkpoints make a workflow resumable, but they do not by themselves guarantee that an external effect happens only once. LangGraph documentation explicitly notes that a task may be executed again after a failure, so API calls should be isolated in tasks and designed to be idempotent. AWS describes the same distributed-systems problem: after a timeout, a client may not know whether the request completed, and a safe retry needs a stable identifier for the original intent.

What should be persisted

A reliable agent is not a long prompt with a retry button. It is a state machine in which every state can be reconstructed unambiguously. At a minimum, each workflow instance should keep:

• a `run_id` for the complete execution;
• the `step_id` and workflow schema version;
• source business identifiers such as request, customer, order, and document IDs;
• validated inputs for the current step and a hash of material parameters;
• the model output separately from the decision to perform an action;
• approval status, approver, and decision time;
• a stable `idempotency_key` for every external side effect;
• command status: `planned`, `approved`, `dispatched`, `confirmed`, `failed`, or `needs_reconciliation`;
• confirmation from the external system, such as an order, task, message, or ledger-entry ID;
• attempt count, error class, and the earliest allowed retry time.

The conversation transcript should not be the only state store. It is difficult to resume precisely, hard to migrate, and likely to contain unnecessary personal data. Critical fields should be stored in structured form, while documents should remain in their source systems and be referenced by controlled, versioned identifiers.

Separate reasoning from execution

A practical design has two paths. In the first, the model reads authorized context and proposes what should change, with parameters and rationale. In the second, a deterministic executor validates policy, permissions, and approval before calling CRM, ERP, or a mail gateway.

This separation provides several benefits:

• the model does not receive universal direct write access;
• a proposed action can be validated against a formal schema;
• the same plan can be validated again without executing it again;
• high-impact operations can require human approval;
• completion is proven by the target system, not by the model saying “done.”

For a first pilot, an existing automation service can orchestrate the workflow while an operational database table stores state. Explicit step boundaries and atomic state transitions matter more than the framework name.

Idempotency: one key for one business intent

When an idempotent command is repeated with the same key, it returns the original result instead of creating another object. The key must identify the business intent rather than an individual network attempt. For invoice creation, for example, it could be derived from the request ID, operation type, and approved-plan version.

The core rules are straightforward:

• create the key before the first external call and persist it in the checkpoint;
• use the same key for every retry instead of generating a new UUID;
• persist a hash of the material parameters with the key;
• reject the same key with different parameters as a conflict;
• retain the key for longer than the maximum automatic and manual retry window;
• store the first successful response and return it to repeated requests.

If the target API supports idempotency keys, use its mechanism. If it does not, add an adapter: a command table with a unique index on the key, a lookup by a stable business identifier, or an `upsert` operation instead of unconditional `create`. For irreversible actions—payments, bulk mailings, or legally significant documents—a local flag is insufficient. Reconcile with the target system before retrying.

Where the transactional outbox fits

There is another failure boundary: a service commits a database change but crashes before publishing the corresponding event. Reversing the order is no better—the event may be delivered while the database transaction rolls back. Two independent writes do not become atomic merely because they appear next to each other in code.

The transactional outbox pattern works as follows:

1. Commit the business change and a pending command or event in one database transaction.
2. A separate worker reads unprocessed outbox records and sends them to the recipient.
3. The recipient deduplicates messages using a stable event identifier.
4. The worker marks confirmed delivery; unresolved records move to reconciliation.

Microsoft’s guidance points out that delivery in this design is normally at least once. The outbox prevents event loss, but it does not eliminate the need for an idempotent consumer. For an agent, this is useful when creating tasks, notifications, procurement requests, or any other action that crosses a queue or integration service.

Put the checkpoint on the correct side of the action

It helps to split every high-impact step into four stages:

1. `prepare` — normalize parameters and compute the key;
2. `approve` — validate policy and obtain human approval when required;
3. `execute` — send one idempotent command;
4. `confirm` — persist the external object ID and observed status.

The checkpoint before `execute` must already contain the approved parameters and the stable key. After a response arrives, confirmation must be saved before the workflow advances. If the process crashes between dispatch and confirmation, do not label the action as an ordinary failure. Move it to `needs_reconciliation`: first ask the target system whether an object with this key or business identifier already exists, then decide whether a retry is safe.

Human approval is also part of the state. It must be bound to a specific version of the parameters. If the model recalculates the amount, recipient, or order contents, the previous approval must expire. Otherwise, the workflow is technically “approved” but executes a different action.

Retries need a budget

Automatic retries are useful for short network failures, temporary unavailability, and rate limits. They are harmful when the input is invalid, access is denied, a business rule rejects the operation, or the external result is uncertain.

For every tool, define:

• which error codes are transient;
• the maximum number of attempts;
• exponential backoff with jitter;
• the total step deadline;
• a limit on concurrent retries;
• the transition to manual reconciliation;
• operations that must never be retried automatically.

A retry should not call the model again when the inputs and decision are already persisted, because the command parameters could change. The model should be re-entered only through an explicit workflow transition, such as new source data or a human decision.

Test the architecture before launch

The pilot can be tested on one process without an expensive environment. Choose an action with an understandable failure cost, such as creating a CRM task or an invoice draft, and inject controlled failures:

• drop the connection immediately after dispatch;
• restart the worker between the outbox commit and delivery;
• send the same idempotency key twice;
• send the same key with changed parameters;
• pause at approval and resume a day later;
• delay an old response until after a new attempt begins;
• recover a workflow after deploying a new workflow version.

Success is not merely that the agent “reached the end.” Track three measures: the share of workflows recovered without manual intervention, the number of duplicate external objects, and the number of uncertain operations without confirmation. Before production, duplicates in the test set should be zero, and every uncertain result should appear in an observable reconciliation queue.

Economics for small and mid-sized businesses

The main cost is not the checkpoint database. It is adapting external systems and defining business boundaries. Do not migrate dozens of scenarios to a durable orchestrator at once. Start with one process where a duplicate creates measurable loss or manual cleanup.

The minimum design consists of an operational database, a unique idempotency index, a queue or outbox table, a worker, a manual reconciliation view, and metrics. A dedicated orchestration cluster becomes reasonable when workflows last hours or days, require several approvals, regularly survive failures, and no longer fit a basic job queue.

Measure the economic effect through prevented duplicates, investigation time, and workflow downtime—not only model-call volume. If the architecture prevents a duplicate payment, shipment, or bulk mailing, its value appears before any token optimization.

The next step

Take one agent workflow and create a five-column table: step, persisted state, external side effect, idempotency key, and reconciliation method. Then run the “response lost after successful write” test. If the command creates a second object, or the workflow cannot determine what happened, it is not ready for autonomous execution.

AI-agent reliability does not begin with the promise that “it resumes after a crash.” It begins with a precise answer to a harder question: what has already happened in the business system, and how can we prove it without doing it again?