Why high accuracy can still be a poor investment

AI pilot presentations often feature one convenient number: “95% accuracy.” For a manager, that figure is nearly useless without three clarifications: how rare the important case is, which types of error the system makes, and what each error costs.

Google’s official classification-metrics course notes that accuracy—the share of all correct classifications—is a rough measure for balanced data. It can mislead when the target event is rare. If only 1% of requests need special handling, a system that always predicts “ordinary request” achieves 99% accuracy while missing every important case.

In an operating process, the consequence matters more than abstract correctness. Sending an ordinary document to a specialist creates a queue and consumes labour. Missing a complaint, suspicious payment, or contractual exception can create a penalty, lose a customer, or trigger an incorrect action. These errors cannot be averaged as if they were equal.

The economics of AI automation therefore starts neither with token prices nor a model leaderboard. It starts with a confusion matrix and an accountable owner who assigns a cost to each outcome.

The confusion matrix in process language

Binary routing has four outcomes:

  • **a correctly detected important case** is sent to a person;
  • **a false alarm** sends an ordinary case into the manual queue;
  • **a correctly automated ordinary case** follows the safe route;
  • **a missed important case** is automated even though review was required.

Precision asks what share of raised alerts were actually important. Recall shows what share of all important cases the system detected. Changing the decision threshold usually moves these metrics in opposite directions: a more cautious threshold catches more risk but increases manual review.

Defining the “positive” class is itself a management decision. For inbound messages, positive might mean either “safe to answer automatically” or “must go to a specialist.” The second definition is often clearer for risk because recall then measures the share of dangerous exceptions detected. If the team has not agreed on the direction of the metric, the same word “accuracy” can describe opposite outcomes.

Converting errors into money

Measure the manual baseline first, then estimate five components:

  • the cost of manual handling before automation;
  • exception review and sample-audit labour;
  • infrastructure, model, integration, and support;
  • expected loss from false alarms;
  • expected loss from missed important cases.

The last component is the probability of a consequence multiplied by its cost. Not every missed contractual exception causes a loss. But if the probability of a material event is 10% and its average impact is RUB 200,000, the expected cost of one miss is RUB 20,000.

This is not an accounting recognition of a future loss. It is a tool for comparing thresholds and exposing assumptions. If the risk cannot yet be estimated, keep a strict human-review rule and collect evidence in shadow mode.

A model calculation for an exception queue

Consider a service company processing 10,000 inbound documents or messages per month. Only 2%, or 200 cases, require a specialist. Fully manual handling takes two minutes per item. At a fully loaded labour cost of RUB 1,500 per hour, the baseline is about RUB 500,000 per month.

At the selected threshold, a validation sample produces:

  • 190 of 200 important cases routed to a specialist;
  • 10 important cases missed;
  • 500 ordinary cases unnecessarily sent for review;
  • 9,300 ordinary cases correctly handled through automation.

Overall accuracy is 94.9%. Recall for important cases is 95%. Queue precision is approximately 27.5%: most alerts are ordinary, yet that may still be a rational price for avoiding a rare expensive miss.

Assume each exception takes four minutes to review. A queue of 690 items needs 46 hours, or RUB 69,000. Auditing a 2% sample of the 9,300 automated decisions adds about 6.2 hours, or RUB 9,300. The model, infrastructure, and support cost RUB 60,000 per month. If the expected loss per missed case is RUB 20,000, ten misses add RUB 200,000.

The modelled total for the automated mode is approximately RUB 338,300 per month, saving about RUB 161,700 against the manual baseline before one-time integration. If the expected loss per miss is RUB 50,000, the same threshold costs about RUB 638,300 and is RUB 138,300 worse than manual work.

Every number in this example is an assumption, not the result of an actual company or a universal benchmark. The purpose is to show why the same model can pay back in routine request routing and remain unacceptable in payments, legal work, or security.

Architecture for a safe exception queue

A reliable system does not let a model execute an action merely because its score exceeds a threshold. A minimum design includes:

1. **Intake and normalisation.** Validate format, required fields, duplicates, and source.
2. **Classification.** Return a class, confidence score, and technical model version.
3. **Deterministic stops.** Rules outside the LLM block operations based on amount, customer type, personal data, or defined terms.
4. **Risk-zone routing.** A safe zone may allow an automated draft, a middle zone enters a normal queue, and a critical zone reaches a priority specialist.
5. **Human decision.** The interface shows the original item, model proposal, routing reason, and permitted actions.
6. **Execution.** A separate service performs only approved or explicitly pre-authorised actions.
7. **Audit and feedback.** Record the threshold, decision, correction, review time, and observed consequence.

The queue needs an SLA and a size limit. If arrivals exceed the team’s review capacity, the system must fail safely: pause automation, expand manual routing, or disable the action temporarily. An invisible backlog is not a saving; it is deferred manual labour.

Data and integrations required

A pilot needs a labelled dataset that represents rare cases, not a random archive alone. It should cover different channels, seasons, employees, customer types, and policy versions. A rare class cannot be represented by two test examples; one mistake would then move the metric by tens of percentage points.

Minimum integrations include:

  • the document, email, or request source;
  • customer and contract reference data;
  • a task queue with priorities and deadlines;
  • an approval and correction interface;
  • version logs for models, thresholds, and rules;
  • analytics for TP, FP, TN, FN, time, and consequences;
  • a manual route when the model is unavailable.

Have two specialists label the ambiguous portion of the sample. Expert disagreement often reveals an unclear business rule rather than a model defect. Clarify the criterion before demanding consistency from AI.

Where local models, RAG, and agents fit

**A local model** is justified when the classifier handles contracts, personal data, internal requests, or payment information that cannot leave a controlled environment. Locality does not improve metrics automatically; the same testing, access controls, and audit logs are still required.

**RAG** helps when the decision depends on a current policy or the terms of a specific contract. Retrieval must return authorised documents and their versions. It does not replace a risk threshold because the retrieved passage can still be incomplete or misapplied.

**An agent** is usually excessive for simple classification. Start with a model, rules, and a queue. A multi-step agent is warranted only when classification must be followed by checks across several systems. Keep read tools separate from action tools, and require human approval for financial, legal, and external operations.

A pilot with economic gates

**Week 1.** Define the positive class and the cost of every outcome. Measure manual time and the prevalence of rare cases.

**Week 2.** Label a sample and compare the model with a simple rule and the human baseline. Check precision and recall separately for important segments.

**Week 3.** Run in shadow mode. The model proposes a route while the working process remains manual. Collect errors, corrections, and handling time.

**Week 4.** Calculate several thresholds. For each one, show queue volume, misses, review cost, infrastructure, and expected loss.

Production is justified only when four gates are met: a maximum tolerated miss rate is defined, the queue stays within its SLA, the economic benefit remains positive after audit, and a safe manual fallback exists.

Management takeaway

Do not accept a pilot on accuracy alone. Ask for a four-cell confusion matrix, separate costs for false alarms and misses, manual queue volume, and calculations at several thresholds. Select the threshold as a business decision rather than a developer default.

High accuracy looks good on a slide. Positive economics appears only when the system allocates rare expensive errors correctly and people can process the exceptions. Vnutrik has already arranged the folders perfectly; management still has to decide which ones must never be left to the robot.