Why AI should not change an order in the system immediately

Incoming orders often arrive in emails, spreadsheets, PDFs, or messages from account managers. Someone must transfer the customer, items, quantities, prices, address, and delivery date into an ERP or CRM. The work is repetitive, but one incorrect field can result in a wrong invoice, inventory position, or shipment.

A language model is useful for reading varied text: it can extract candidate fields, flag ambiguity, and prepare a draft. But a model response is not a verified record. A model may confuse an item number, pick a figure from a neighboring line, or treat instructions inside a document as commands. The safe pilot objective is therefore not “let AI place orders,” but “let AI prepare a structured draft that the system validates and a person approves.”

OWASP classifies insufficient validation of model output before it is passed to other components as improper output handling. For order intake, the practical implication is simple: do not pass generated content directly into a privileged ERP call. Treat it as untrusted input, like data from an external form.

A workflow that separates extraction from writing

Split the process into four steps.

1. **Receive the document and preserve provenance.** The service accepts an email or attachment and records the sender, time, email identifier, and file hash. Store the file in the company’s protected environment; send only the required text to the model. If the document contains personal or commercially sensitive data, decide separately where recognition runs and which fields may be sent to an external API.
2. **Create a draft.** Ask the model to extract only allowed fields and return an object with a fixed structure: customer, tax or internal ID, items, SKUs, quantities, currency, address, requested date, and links to supporting text fragments. Use null for unknown values rather than a guess. Treat the order text as data, not as a source of new instructions for an agent.
3. **Validate deterministically.** Ordinary code validates the JSON Schema, types, and required fields; checks customers and SKUs against master data; recalculates totals; checks currency and allowed ranges; and looks for duplicates using a stable order key. Schema validation confirms structure and types, not whether an extracted value is true; those checks must be separate.
4. **Show the result to an employee and write through a restricted adapter.** The interface displays the source fragment beside each value, highlights discrepancies, and lets the employee correct the draft. Only after explicit approval does the adapter call the ERP API. The model gets neither a password nor a general ability to perform arbitrary operations.

This design separates probabilistic interpretation from business rules. The model proposes an interpretation; code determines whether it is allowed; an employee approves exceptions; and the accounting system receives only a validated command.

What to validate

Checks should reflect actual company rules, not just formatting requirements.

  • **Customer identity.** Match on an internal identifier and verified details, not just a similar name. If several candidates match, route the order for clarification.
  • **Catalog items.** Check SKU, unit of measure, and item status against master data. Do not let the model create a new product record.
  • **Quantity and price.** Validate numeric ranges, currency, discounts, and arithmetic. The model may transcribe a number, but code should calculate the total.
  • **Dates and address.** Check the requested date against the business calendar and delivery rules. Do not replace a missing date with a “reasonable guess.”
  • **Evidence for each field.** Store the quote or page coordinates from which a value was extracted. If there is no supporting fragment, require manual entry.
  • **Duplicate delivery.** Use the email’s external ID or a normalized-order hash as an idempotency key so that reprocessing after a network failure does not create another order.

The data schema should define a closed set of fields, permitted types, string lengths, and numeric ranges. A validation error should stop the write, not trigger endless attempts to repair the response through another generation.

Permissions, errors, and logs

Give the writing service a separate account with only the permissions it needs. If the integration can create an order draft, do not grant it permission to change prices, master data, or dispatch shipments. Keep connection secrets in a secret manager or protected configuration, not in the system prompt or conversation history.

Keep a decision log: source-document identifier, prompt and model version, extracted fields, validation results, employee edits, final ERP ID, and rejection reason. Do not put the full contract or order text in diagnostic logs when an identifier and selected fields are enough for investigation. Define retention periods and access controls for the log.

Separate failures into understandable categories: the model did not respond; the document could not be read; the schema was invalid; a catalog item was not found; an employee rejected the draft; or the ERP was unavailable. Retrying a read or model request may be safe. Retry an ERP write only with the same idempotency key and after checking the previous request’s status. Otherwise, a timeout can result in a duplicate.

How to measure a pilot honestly

Start with one order type and one intake channel. Assemble a sample of orders already processed manually and record the ground-truth fields. Define metrics before launch:

  • share of documents for which a draft could be prepared;
  • accuracy for each critical field separately—especially customer, SKU, quantity, and price;
  • share of drafts requiring correction and average review time;
  • time from email receipt to confirmed order;
  • duplicates, incorrect writes, and rejected drafts stopped by the system;
  • recognition and review cost per accepted order.

Do not average critical errors together with harmless ones into one “accuracy” figure. An address-formatting issue and a wrong price have different consequences. First estimate the benefit as minutes saved on manual entry minus review, integration maintenance, and exception handling. If employees spend more time correcting drafts, the automation has not yet paid for itself.

It is useful to compare two modes on the same sample: ordinary manual handling and AI draft preparation with review. This measures one specific workflow; it is not a universal performance promise for a model.

Limitations and the next step

This workflow does not eliminate model errors or guarantee protection from prompt injection. An attachment may contain malicious instructions, so document instructions must not expand integration privileges, and final checks must be performed by code outside the model. Human approval is not a cure-all either: the interface should show the source data rather than invite an employee to click “OK” mechanically.

For an initial experiment, choose one frequent, relatively low-risk order type, gather 50–100 anonymized historical examples, and define the required-field dictionary and business rules. Then run them in a mode that cannot write to the ERP. The sample size is a practical starting suggestion for a pilot, not a statistical guarantee. Move to creating drafts only after checking errors on real documents and approving access rights.

A manager should approve the responsibility boundary before choosing a model: AI proposes fields, a validator checks rules, a person approves exceptions, and a separate service writes the result with minimal privileges. That boundary—not model size—is the key architectural decision.