Why GPU price does not decide the business case

Local AI calculations often start with the wrong question: “How much is a GPU server?” The business question is how many approved operations the system completes each month at the required response time. The same accelerator can be economical with a dense document queue and expensive with ten short requests per day.

Total cost includes the server, memory, storage, power, cooling, redundancy, engineering time, monitoring, upgrades, downtime, model tests, and human review. Divide that cost by useful outcomes—an approved invoice, resolved request, verified meeting record, or accepted product card—not by abstract tokens alone.

vLLM documents serving mechanisms including continuous batching, PagedAttention, chunked prefill, prefix caching, and quantization. They do not automatically produce savings. They help when request shapes can share compute and the business accepts the resulting trade-off between latency and aggregate throughput.

Three metrics instead of one speed number

The first metric is time to first token, important for an employee waiting in an interface. The second is time per output token, which shapes response fluidity. The third is aggregate throughput: requests or tokens processed under concurrent load.

MLPerf Inference separates Server and Offline scenarios because interactive arrivals and accumulated batch work are different problems. Server workloads arrive over time and must satisfy latency constraints. Offline workloads optimise completion of a batch. Results cannot be compared without their scenario.

For a business:

  • support chat is measured by latency, consistency, and concurrent users;
  • overnight document work by batch completion time and cost per document;
  • tool-using agents by end-to-end duration including databases and APIs;
  • RAG by retrieval, generation, and citation-verification time.

A single-dialogue benchmark can overstate hardware needs for batch work while understating memory requirements under concurrency. Test the actual distribution of context length, output length, and simultaneous requests.

What makes inference expensive

Inference has two distinct phases. Prefill processes input context and is often compute-heavy. Decode emits tokens sequentially and is frequently memory-bandwidth bound. Long documents with short answers differ from short questions with long reports.

Cost depends on:

  • system-prompt length and prefix repetition;
  • retrieved RAG context size;
  • average and peak output length;
  • concurrent requests;
  • KV-cache required by active sequences;
  • weight quantization and engine support;
  • accelerator idle time.

Continuous batching inserts new work into an active batch to improve utilisation. Chunked prefill divides long inputs so they can be scheduled alongside decoding. Prefix caching avoids reprocessing common beginnings. Yet aggressive batches may damage interactivity, and cache growth may reduce concurrency.

“Tokens per second” without input, output, concurrency, and percentile latency is a marketing number rather than a budget basis.

A modeled small-business calculation

Assume a company considers a local server costing RUB 900,000. Use a three-year horizon and no residual value. The monthly hardware component is RUB 25,000.

Add illustrative monthly costs:

  • power and cooling: RUB 6,000;
  • monitoring, backups, and services: RUB 7,000;
  • eight engineering hours at RUB 2,500: RUB 20,000;
  • repair and downtime reserve: RUB 7,000.

The total is RUB 65,000 per month. This is a model, not a market quote; each company must insert its own hardware, electricity, and labour rates.

At 2,000 approved operations per month, infrastructure costs RUB 32.5 per operation. At 20,000, it costs RUB 3.25. At 200, it costs RUB 325. Hardware is unchanged; utilisation changes unit economics by a factor of one hundred.

Now include review. If each operation needs two minutes at a fully loaded labour rate of RUB 1,500 per hour, review adds RUB 50. At 2,000 operations, total variable cost is therefore RUB 82.5 rather than RUB 32.5. If automation releases ten minutes worth RUB 250, the modeled resource effect is about RUB 167.5 per operation before integration costs.

This reveals two points. At high utilisation, review may cost more than GPU capacity. And accelerating the model is irrelevant when employees still spend excessive time correcting output.

The boundary between API and local deployment

Compare external APIs on the same unit of approved outcome. API cost includes input and output tokens, storage, networking, integration, quality control, and latency impact. Local cost includes full TCO and actual utilisation.

An API is often preferable when volume is low or volatile, models change quickly, data can be redacted, and operating infrastructure adds no strategic value. Local deployment becomes more attractive with steady demand, strict data routing, batchable work, and an existing operations team.

A hybrid design is often more practical than a binary choice. Sensitive documents run locally, rare difficult tasks use an approved external service, and a router selects a path by data class and quality requirement. Calculate each route separately.

Do not compare the strongest API model with a smaller local model by price alone. Quality and correction rates differ. A valid experiment gives both systems the same task set and measures human acceptance.

Data to collect before buying

Two weeks of logging is enough for a first sizing exercise. For each real task, retain anonymised measures:

  • input and output token counts;
  • common and unique prefix length;
  • time to first token and total duration;
  • concurrency and time of day;
  • success, refusal, and retry;
  • human review time;
  • correction reason;
  • business value of the approved result.

Group requests into at least interactive, batch, and long-context classes. Averages are insufficient: peaks and the 95th percentile determine infrastructure, while a long tail often damages economics.

Track idle time explicitly. An accelerator used for 15% of the day may be justified by confidentiality or availability, but that must be deliberate. Otherwise queues, scheduled batch jobs, or sharing the server across processes usually beat purchasing more hardware.

A pilot with four gates

**Gate 1—quality.** On 100–300 real tasks, the model passes thresholds for facts, structure, and dangerous errors. Rejected output does not count as productivity.

**Gate 2—load.** The test reproduces real lengths, concurrency, and peaks. Measure p50 and p95 latency, completed requests per hour, and out-of-memory failures.

**Gate 3—process.** Measure review time and correction share. If the draft saves no time, a faster server will not rescue the project.

**Gate 4—economics.** Compare local TCO, API, and hybrid cost per approved operation. Include 25%, 50%, and 80% utilisation scenarios and a growth reserve.

Purchase only after all four gates. The correct result may be hourly GPU rental, one shared server, CPU inference, or a smaller model. That is an economically sound conclusion, not a failed pilot.

Management takeaway

Local AI pays back not because the company owns a GPU, but because a predictable queue keeps it useful, quality removes human work, and data requirements justify the private environment.

Start with a table of actual requests and cost per approved result rather than a server specification. Two weeks later, management can see the demand range, the price of idle capacity, and the utilisation point at which local deployment beats the alternatives.