Measure outcomes, not request volume

A RAG assistant can handle many conversations and deliver almost no economic value if employees still recheck every answer, correct bad passages, and maintain stale documents. The useful unit is therefore neither a model call nor a generated response. It is a request actually resolved at acceptable quality without creating extra work for the team.

Total cost includes more than generation. It includes preparing and updating documents, indexing and embeddings, retrieval, context and answer tokens, infrastructure, monitoring, exception review, and the time of the knowledge-base owner. Retrieval adds operations and latency to a RAG workflow; retrieved passages increase the model's input. If you select a large model, send excessive context, or allow unlimited retries, variable costs can grow faster than useful outcomes.

A modelled example: 1,000 requests per month

Assume a small service team receives 1,000 routine questions each month. After reviewing a sample, the team estimates that 60% concern approved procedures and product facts that can be searched in its knowledge base. That gives 600 potentially suitable requests. Suppose that during a pilot, 80% of those questions end with an answer the user accepts without a specialist correcting it. The result is 480 independently resolved requests.

If a manual answer to each of those questions took six minutes on average, the theoretical time released would be 48 hours per month: 480 × 6 minutes. This is not a savings promise or a forecast for any particular company; it only demonstrates the formula. In a real pilot, some answers will need review, some questions will fall outside the knowledge base, and some users will ask for clarification. So subtract the cost of reviewing answers, handling exceptions, updating sources, and administering the system.

Compare against the actual “no assistant” process. If employees spend two minutes rather than six on each answer, the potential benefit is much smaller. If a question requires approval, order access, or a decision on an exception, generating text does not complete the process, and those minutes cannot be counted as a resolved request.

A full-cost formula

A pilot needs a transparent worksheet, not a complicated financial model:

  • **Fixed component:** implementation, document preparation and cleanup, source connections, access configuration, chat or CRM integration, and baseline quality evaluation.
  • **Indexing:** processing new and changed documents, embeddings, reindexing schedules, and index storage.
  • **Per conversation:** retrieval, reranking when used, retrieved-context size, input and output tokens, retries, and fallback calls to another model.
  • **Operations:** monitoring, logs, access controls, source corrections, integration support, and incident coverage.
  • **Human work:** answer review, knowledge-base fixes, escalation handling, and process change management.

A simple formula is: monthly TCO = the period’s fixed costs + knowledge-base updates + variable request costs + human review and support. Divide that TCO by the number of requests actually resolved. Then compare the result with the fully loaded cost of manually handling a similar request.

Track at least two measures: cost per accepted automated answer and the team hours actually released after quality checks. If the system produces a correct answer but a mandatory review still takes the same six minutes, the business benefit may be close to zero even when generation itself is inexpensive.

Evaluate quality without fooling yourself

Start with 50–100 real questions using anonymized data and the expected correct action. Include more than easy FAQs: add follow-up questions, outdated instructions, missing documents, conflicting versions, requests for someone else’s data, and cases where the correct action is to refuse or escalate to a specialist.

Break results into separate stages:

  • **Did retrieval find the right source?** Check relevance and coverage of retrieved passages, and note questions for which the correct document is missing entirely.
  • **Is the answer grounded in the source?** Check for claims absent from the retrieved context and ensure citations lead to a current passage.
  • **Does the answer solve the user’s problem?** A correct quotation is not necessarily a clear or complete answer.
  • **Was human intervention required?** Record whether the answer was accepted, corrected, escalated, or ignored.
  • **How much time did it take?** Include latency, retries, manual waiting, and subsequent maintenance.

Microsoft’s documentation separates retrieval-step evaluation from final-answer evaluation: it describes document retrieval, context relevance, groundedness, response relevance, and completeness. Precise retrieval metrics require labelled relevant documents; without such labels, a team can begin with expert review of a small sample. An automated LLM judge can speed up triage, but it is not an infallible source of truth. People should review borderline, costly, or risky answers.

A low-cost pilot plan

1. Choose one narrow stream of recurring questions with bounded error risk. Do not begin with “all company documents.”
2. Assign an owner for source content. Each procedure should have a date, version, accountable owner, and a clear way to retire obsolete material.
3. Capture a baseline: request volume, average response time, repeat contacts, escalations, and actual team time.
4. Build a test set and evaluate retrieval separately from generation. At first, manual labels on a small sample are enough.
5. Bound the context: search only the relevant section, send a few useful passages, and prevent unbounded retries.
6. Run a pilot with visible source citations, a “didn’t help” option, and a simple path to a specialist.
7. After a week or two, compare the same request types against the baseline. Recalculate costs from actual logs, not an assumed average token count.

When the answer is not to buy RAG

If the team has 20 short, stable answers, ordinary search, an FAQ, or a workflow using approved templates may be cheaper and more reliable. RAG is useful when employees need to find information across a meaningful corpus of changing documents and formulate an answer in natural language. It does not remove the need for access control: if retrieval can see a restricted document belonging to another team, a fluent answer merely accelerates a data leak.

Do not expand a pilot until you know what is failing. Low retrieval coverage calls for fixing chunking, metadata, language, or source structure. Good context but poor answers points to the model, prompt, or representation format. An incorrect source document is a knowledge-owner problem, not a reason to buy a more expensive model.

The management decision

Before approving a budget, ask the team to show three numbers: the share of answers accepted without correction, the actual time saved after review, and the full monthly cost of ownership. Start with a small pilot using real anonymized questions. Expand only if the cost per resolved request and quality beat the baseline process while access, sources, and escalation remain under control.