Measure accepted outcomes

An AI agent may take several steps for one task: read a record, retrieve information, call a tool, check its answer, and retry a failed action. A model bill captures only part of the cost. The business needs the full expense per result that meets its criteria and is accepted by an employee.

Include models, search and other tools, retries, failed attempts, manual corrections, review, and operations. Divide the total by accepted results, not all generated ones. If half of 100 answers are rewritten manually, the business has not automated 100 tasks.

An illustrative example

Assume an agent receives 1,000 tasks in a month and 92% meet a predefined acceptance criterion: 920 tasks. If the combined hypothetical budget for models, tools, support, and review is RUB 26,680, the cost per accepted task is about RUB 29. This is not a provider tariff; it illustrates the formula, and the assumptions must be replaced with the company’s own data.

For each task in a pilot, record attempts and tool calls, tokens or compute, retry reasons, review outcome, and employee minutes. Separate routine cases from exceptions. An answer needing five minutes of review has different economics from one that can be accepted immediately.

How to measure

Choose one narrow, repeatable workflow with a clear endpoint. For example, the agent prepares an application record and an employee approves it before it is written to a CRM. Do not grant irreversible permissions at the start.

Compare at least 50–100 tasks with the current process. Track manual handling time, errors, and rework cost. Mark separately tasks accepted without changes, accepted after correction, and rejected. Use API usage details and your own workflow labels: an organization-wide total does not reveal which process incurred the cost.

Deciding whether to continue

If failures come from poor inputs or unnecessary steps, simplify the workflow, cap retries, and define clear handoff rules. An API may be simpler for a low-volume pilot; a local model may fit steady demand or data requirements. Compare options on the same sample and quality criteria.

A week of measurement gives managers a basis for the decision: cost per accepted task, success share, review time, and risk. Increase the budget only after these measures show value.