Zero observed errors do not mean zero risk

A team tests an AI system on 50 documents, sees no errors, and starts discussing the removal of human review. The little trainee robot Vnutrik is already imagining an employee-of-the-month badge. Yet a sample of 50 may simply miss rare failures that are expensive for the business. The result is a reason to continue testing, not a promise of error-free handling of requests, invoices, or contracts.

In its work on AI evaluation, NIST distinguishes performance on a specific benchmark from expected performance on similar future tasks. The latter cannot be obtained by merely relabeling the former: the evaluator must explain how cases were selected and quantify uncertainty. For a manager, “everything was correct in our test” does not yet answer “how many errors should we expect in a month?”

Define an error before counting it

Choose the unit of evaluation before running the model. For invoice extraction, it may be an entire document, a single field, or an event such as “the system sent a wrong payment without approval.” Those are different measures. One invoice with the wrong tax identifier but the right total can yield a high field-level accuracy while creating unacceptable operational risk.

Use a short classification:

  • Critical failure: wrong recipient, amount, legally important deadline, or an action taken without authority.
  • Correctable failure: an operator notices an inaccuracy before anything reaches a customer.
  • Delay: the answer is substantively right but arrives too late for the process.
  • Abstention: the system cannot safely handle a document and hands it to a person.

Do not automatically count abstentions as successes. In some processes, a safe handoff is preferable to a confident invention, but it still costs staff time and creates a queue. Measure the share of cases completed without help separately from the rate of critical failures.

What zero failures can tell you

Here is a model calculation, not a universal reliability certificate. Assume independent tests sampled from the same stable population as future work, a failure criterion fixed in advance, and a system that does not change between trials. If there are zero failures in n trials, the exact one-sided 95% upper confidence bound on the unknown failure probability is 1 − 0.05^(1/n). This follows from the binomial model covered in NIST statistical guidance.

The approximate results are:

  • Zero failures in 50 tests: an upper bound of about 5.8%.
  • Zero failures in 100 tests: about 3.0%.
  • Zero failures in 300 tests: about 1.0%.

Under those assumptions, at least 299 independent failure-free trials are needed for a bound strictly below 1%. Even then, this does not prove that the system fails in fewer than one of every hundred cases under all conditions. It is a conditional statistical bound for a defined population and a fixed process. Convenient examples, repeated templates, or documents seen during training defeat the premise regardless of the formula.

If even one failure appears, the zero-failure formula no longer applies. Estimate a different interval and, more importantly, investigate the mechanism: a poor scan, ambiguous identifier, outdated instruction, or integration failure. Retest a repaired process on independent cases. Do not discard difficult examples after seeing the results.

A practical test without a research laboratory

A small business can start with one narrow workflow, such as drafting responses to routine inquiries or extracting invoice fields. Freeze the model version, prompt, RAG retrieval rules, reference data, and settings. Staff who understand the workflow should define the correct outcome and failure types, preferably before they see the model response.

Use actual documents from different weeks and sources, while removing unnecessary personal data from the test environment. Split cases by material conditions: supplier, file format, language, scan quality, unusual request, and exceptions to the standard rule. Put the most costly errors on a separate checklist even if they are uncommon in ordinary traffic. Then measure not just the average success rate but performance in each important group. Three hundred identical, pristine PDFs cannot stand in for poor scans and unusual invoices.

An initial smoke test of 30–50 cases can expose obvious architectural defects. It cannot justify a claim that rare critical failures are unlikely. Stronger estimates require more independent cases in each important group. Where the data are scarce, retain human approval for consequential actions. For RAG, check the retrieved source, document freshness, and access rights as well as the answer. For an agent, distinguish the correctness of a proposed action from whether it was authorized and executed correctly.

The economics: pay for evidence, avoid paying for mistakes

Consider a purely illustrative budget: 300 cases × 4 minutes of expert review = 20 hours. At an assumed fully loaded internal rate of RUB 1,500 per hour, manual assessment costs RUB 30,000. This is not a market quotation or a promised outcome. It excludes gold-standard preparation, process repairs, compute, infrastructure, and retesting. In a workflow with costly failures, that assessment may cost less than one erroneous transaction; in a low-risk workflow, it may be excessive.

Measure cost per accepted business outcome rather than cost per model call. Divide model or API, integration, human review, returns, corrections, and support costs by the number of correctly completed operations. The FinOps Foundation describes this unit-economics approach. If automation saves one minute of typing but adds two minutes of checking, an attractive token price does not describe the business result.

Compare three operating modes on the same cases: a fully manual workflow, AI-generated drafts approved by a person, and partial automation restricted to low-risk cases. For each, measure elapsed time, correction cost, and missed critical failures. A locally hosted model can make sense when documents cannot be sent to an external API or the workload supports a private deployment. Hosting locally does not itself improve accuracy or eliminate maintenance costs. The pilot should answer whether this workflow pays off, not whether the deployment style is fashionable.

A decision for next week

Assign an owner for the workflow and an owner for evaluation. Pick one document type, define three to five critical fields or actions, assemble an initial varied sample, and set an acceptance threshold before running the test. Retain versions of inputs, reference answers, model, and rules so that later comparisons are meaningful. If 50 cases go smoothly, broaden the sample and keep human approval where errors are expensive. If results are poor, inspect failures by group: a cleaner reference list or a more reliable integration may help more than a different model.

A good pilot ends not with “Vnutrik never made a mistake” but with a concrete decision: which operations it may already prepare, which still need a human decision, and what the next level of evidence will cost.