Why tokens are the wrong budget unit

In RAG, a user asks a question, the system retrieves knowledge-base passages, and a model produces an answer with sources. Tokens are prominent on provider bills, so budgets are often estimated as average prompt length multiplied by model price. That leaves out document preparation, index updates, storage, retrieval, reranking, retries, and human review.

The FinOps Foundation recommends connecting spend to a business outcome: a resolved case, accepted document summary, or completed action. For an internal knowledge base, a useful unit is an approved answer: the employee received sufficient information, sources opened correctly, quality rules passed, and no repeat request was needed.

The formula is simple:

**Cost per approved answer = all RAG costs for the period / number of accepted answers.**

The numerator includes infrastructure, data maintenance, and people—not only API calls or GPUs. The denominator cannot be all requests: a failed answer consumed resources without creating value.

Six lines on the real bill

**Data preparation.** Documents must be extracted, OCR-processed, cleaned, chunked, tagged with metadata, and checked for permissions. Correcting sources and assigning owners often costs more than embeddings.

**Indexing and reindexing.** Vectors are created during initial ingestion and after a document, embedding model, or chunking policy changes. A full rebuild can create a material compute and write peak.

**Storage.** Vectors, chunks, payload, filter indexes, replicas, backups, and logs consume capacity. Qdrant reports vector, payload, and payload-index memory separately; every filter index adds RAM and disk.

**Retrieval.** Each question needs a query embedding, permission filtering, vector or hybrid search, and passage reads. In serverless services, cost can depend on operations and search-space size. Pinecone documentation, for example, separates read units, write units, and storage.

**Reranking and generation.** A reranker can improve relevance while adding compute. The LLM then receives instructions, history, and retrieved passages. An unnecessarily high top-k expands context cost without guaranteed quality.

**Quality control.** Low confidence, missing citations, or sensitive questions trigger human review. A repeat query and escalation are part of unit cost, not exceptions outside the AI system.

Fixed and variable costs

Split the monthly bill for management purposes.

Costs that are largely fixed within existing capacity include:

  • connectors and data ownership;
  • server or minimum cloud capacity;
  • redundancy and monitoring;
  • evaluation sets and regression runs;
  • engineering and knowledge-administration time.

Variable costs grow with traffic:

  • query embeddings;
  • reads and retrieval;
  • candidate reranking;
  • input and output tokens;
  • automatic retries;
  • human review and escalation minutes.

On dedicated infrastructure, some variable costs become step-fixed. Hardware is already paid for until growth creates a queue or requires another node. Use realised utilisation, not theoretical accelerator throughput, to price internal requests.

A model calculation for a knowledge base

Consider a company with 10,000 employee questions per month. This is an illustration, not a market quote. Replace every number with actual provider rates and fully loaded labour cost.

Monthly fixed assumptions:

  • source preparation and data-owner work: RUB 25,000;
  • indexing and scheduled updates: RUB 8,000;
  • vector database, storage, and backups: RUB 12,000;
  • monitoring and evaluation runs: RUB 8,000;
  • ten engineering hours at RUB 2,500: RUB 25,000.

The fixed total is **RUB 78,000**.

Variable assumptions per user question:

  • embedding, retrieval, and reads: RUB 0.8;
  • reranking: RUB 1.2;
  • generation and automated validation: RUB 1.5;
  • quick human review: 20% of queries × 3 minutes × RUB 900/hour = RUB 9 averaged across every query;
  • expert escalation: 10% × 8 minutes × RUB 900/hour = RUB 12.

Variable cost is **RUB 24.5 per request**, or RUB 245,000 per month. Total spend is RUB 323,000.

Assume 75% of answers are accepted without another request: 7,500 useful outcomes. Cost per approved answer is **RUB 43.1**. Looking only at generation and retrieval would show RUB 3.5, understating unit cost by more than twelve times.

This is not a universal RAG price. It demonstrates the correct measurement order: human queues and acceptance rates can change economics more than token discounts.

Why volume changes cost nonlinearly

Under the same assumptions, 2,000 requests create RUB 49,000 in variable cost and RUB 127,000 total. At 75% acceptance, unit cost is about **RUB 84.7**, because fixed infrastructure is spread across little demand.

At 50,000 requests, total cost is RUB 1,303,000. With 37,500 accepted answers, unit cost is about **RUB 34.7**. This linear model only holds while technical capacity and the human-review ratio remain stable.

Real systems encounter steps:

  • another vector-database replica or server;
  • a dedicated reranking service;
  • more reviewers;
  • more frequent indexing;
  • larger backup capacity;
  • long-tail queries and traffic peaks.

Build low, planned, and peak scenarios and mark thresholds where a new fixed cost appears.

Acceptance matters more than cheap generation

Suppose the team saves RUB 0.7 per request by shortening context: RUB 7,000 at 10,000 requests. If aggressive trimming reduces acceptance from 75% to 68%, useful answers fall to 6,800 and cost per outcome rises despite a smaller model bill.

Optimise these together:

  • cost per request;
  • answers with correct sources;
  • accepted outcomes;
  • repeat questions within the working window;
  • response time;
  • human review minutes.

A cheap answer that an employee manually rechecks can be more expensive than no RAG. A costlier reranker can be justified if it materially reduces escalation.

How index size enters the model

A vector database is not dimensionless storage. Cost is shaped by chunk count, vector dimension, payload, replication, and filter indexes. Duplicate documents and excessively small chunks increase storage, ingestion, and candidate volume at once.

Qdrant recommends creating payload indexes only for fields used in filters. They speed filtering but consume additional memory and disk. Department, document type, and permission indexes can be valuable; indexing every rare metadata field “for later” is not.

Quantisation reduces vector memory but changes the recall, speed, and compression trade-off. Test it on the company's golden set instead of choosing only by RAM savings.

Practical levers include:

  • deduplicate before embedding;
  • update only changed documents;
  • store content hashes and pipeline versions;
  • choose chunk size by quality, not maximum chunk count;
  • index only filters actually used;
  • separate hot collections from archives;
  • measure full index-recovery cost.

When reranking pays back

A reranker adds a model call or compute service. It should not be enabled blindly on every route. Compare baseline retrieval and reranking on the same questions.

For each configuration, record:

  • source recall in top-k;
  • correct passage in first position;
  • citation quality;
  • generation input tokens;
  • latency;
  • human escalations;
  • cost per approved answer.

Reranking is economical when its added cost is smaller than reductions in irrelevant context, retries, and review. Skip it for simple exact queries and route ambiguous questions through it.

Events worth logging

One trace should connect a question to its business outcome without logging unnecessary sensitive content. Store:

  • use-case and department identifiers;
  • index, embedding, retriever, reranker, and LLM versions;
  • candidate and supplied-passage counts;
  • input and output tokens;
  • time and cost per stage;
  • permission filters applied;
  • automated validation result;
  • escalation, human minutes, and final acceptance.

These events produce a report about accepted-answer cost by process, not merely tokens spent. Allocate shared servers and teams by measured consumption or a documented rule.

A pilot with financial gates

**Gate 1 — retrieval quality.** On 100–300 real questions, the correct source must enter the candidate set at the required recall. Otherwise generation hides a retrieval problem.

**Gate 2 — useful outcome.** Employees record acceptance, repeat, and escalation. Only accepted answers enter the denominator.

**Gate 3 — complete bill.** Infrastructure, data, tokens, and people are captured in one model. One-time implementation cost is shown separately and amortised over an explicit period.

**Gate 4 — comparison with the non-RAG process.** Measure employee search time, error cost, and missed requests. Payback exists only relative to this baseline.

After four weeks, the team can choose what to optimise: sources, chunking, filters, model, reranker, or review queue. Cheaper tokens rarely repair a poor knowledge base.

Management takeaway

Price RAG per approved answer, not per million tokens. Put source preparation, updates, storage, retrieval, reranking, generation, retries, and expert time in the numerator; put only accepted results in the denominator.

Start by adding three fields to the pilot: was the answer accepted, was there a repeat request, and how many review minutes were used? Within a month, the true cost driver becomes visible—and so does the savings opportunity that will not damage quality.