Why “the model fits in memory” is not capacity planning

Businesses often begin local LLM selection with two numbers: model size and GPU memory. If the weights load successfully, the server is declared adequate. That is enough for a single-user demonstration. It is not enough for an internal service used by dozens of employees.

A production workload consists of requests that arrive unevenly, vary in length, and consume memory and compute simultaneously. Ten managers may open an assistant within the same minute in the morning. One asks a short documentation question, another pastes a long conversation, and a third runs RAG with several retrieved passages. Even if every request is fast in isolation, together they create a queue.

Local capacity is therefore determined by five interacting variables rather than parameter count:

  • request arrival rate and burstiness;
  • number of concurrently executing requests;
  • input and output length distributions;
  • acceptable p95 time to first token and end-to-end latency;
  • share of answers actually accepted by the user or downstream process.

The final measure connects infrastructure to business value. A fast answer that is rejected still consumes capacity without producing an outcome.

What happens inside the serving engine

Generation has at least two phases. During prefill, the server processes the input context; a long document or large RAG package makes this phase heavier. During decode, the model emits tokens sequentially and stores a KV cache for every active request. More concurrent sequences and longer contexts require more cache memory.

Modern serving engines try to use accelerators more efficiently. llama.cpp enables continuous batching and supports multiple parallel slots, combining tokens from active requests in a shared batch. vLLM uses scheduling and block-based KV-cache management. In the original PagedAttention paper, the authors showed that more efficient cache management could materially increase throughput over the systems evaluated at the time.

Batching does not create free capacity. As demand rises, queue time and tail latency grow. A long prefill can interfere with short interactive requests; insufficient KV cache can cause preemption and recomputation. A configured maximum context length also does not mean that every concurrent user can consume that maximum simultaneously.

The practical conclusion is that single-request tokens per second are insufficient for server comparison. A business must replay its workload mix and count how many requests remain within the agreed service level.

Metrics a manager needs

For an interactive assistant, latency should be separated into:

  • queue time—the period before processing begins;
  • TTFT—the time from scheduling to the first output token;
  • inter-token latency—the delay between subsequent tokens;
  • end-to-end latency—the time until the answer is complete;
  • error and reject rate—failures and overload rejections.

vLLM exposes these as per-request metrics and aggregate histograms. That makes it possible to examine p50, p95, and p99 rather than only averages. For a business, p95 is usually more useful than the mean because it represents a bad but recurring user experience rather than an ideal minute.

Add two product measures to the technical metrics:

  • accepted outcome rate—the share of answers accepted without a repeat request or substantial edit;
  • cost per SLA-compliant accepted outcome—the total cost of an accepted answer that also met the latency target.

The second measure prevents high average speed from hiding overload. A cheap server that leaves a quarter of users waiting too long is only cheap in a spreadsheet.

Building a workload profile

Before buying hardware, export de-identified statistics from a pilot or comparable process for two to four weeks. Each request needs an arrival timestamp, input size, output size, operation type, and outcome. Build distributions rather than one average row.

The minimum profile includes:

  • median and p95 input tokens;
  • median and p95 output tokens;
  • requests per minute during a normal hour and the busiest 15 minutes;
  • maximum number of concurrently active requests;
  • share of RAG, long-document, and tool-call workloads;
  • split between interactive and background tasks.

Interactive and background workloads should be tested separately. A user-facing chat is sensitive to TTFT and token cadence. Overnight document classification can tolerate queueing but values total throughput. Without priorities, a background batch can destroy the p95 of an employee chat.

A load test that answers the cost question

The official `vllm bench serve` tool supports request rate, burstiness, and maximum concurrency. Its documentation explicitly distinguishes maximum-throughput tests, realistic load, stress tests, and SLA validation. llama.cpp provides a server benchmark with virtual users, iterations, batch size, and parallel slots.

Use a matrix rather than a single run:

1. Fix the model, quantization, input length, and output length.
2. Warm the server and run a single-request control.
3. Repeat at concurrency 1, 2, 4, 8, and onward until the SLA fails.
4. At each level, save p50/p95 TTFT, p95 completion time, throughput, errors, queue time, and peak memory.
5. Repeat with p95-long requests and with short requests.
6. Add a burst similar to the start of the workday.
7. Test interference from RAG indexing or a background job.

Capacity is not the maximum setting at which the server still returns an answer. It is the last step at which p95 latency, errors, and memory remain inside predetermined limits with headroom. A production configuration should typically keep a 15–25% reserve for length drift, engine updates, and unusual peaks.

Independent comparisons reinforce the workload dependency. One published study found vLLM stronger under high concurrency while TGI produced lower tail latency in an interactive, moderate-concurrency regime. Those measurements cannot be transferred to a different model and GPU, but the conclusion is durable: the best serving engine is defined by the service profile rather than a universal leaderboard.

Model calculation: one node or two

Consider a hypothetical internal RAG assistant. This is not a market quote; it demonstrates the method.

Assumptions:

  • 18,000 requests per month;
  • 70% of answers are accepted without substantial edits, or 12,600 outcomes;
  • target p95 TTFT is no more than 3 seconds and p95 completion time no more than 25 seconds;
  • total monthly cost of one node is RUB 150,000, including depreciation, electricity, resilience allowance, and operations;
  • load testing shows that one node stays within the SLA up to concurrency 6;
  • the workday peak reaches concurrency 10;
  • one node keeps 80% of accepted answers within the SLA, while two keep 97% within it.

For one node, the number of SLA-compliant accepted outcomes is:

`12,600 × 0.80 = 10,080`.

Cost per outcome is:

`150,000 / 10,080 ≈ RUB 14.9`.

For two identical nodes:

`12,600 × 0.97 = 12,222`,

`300,000 / 12,222 ≈ RUB 24.5`.

The second option costs more per outcome but meets the SLA for nearly the entire workload. That does not mean a second node should be purchased automatically. First test whether a gateway can cap concurrency, background jobs can be moved, context can be shortened, long and short queues can be separated, or the morning burst can be smoothed.

If the business can tolerate waiting, one node may be rational. If the answer is embedded in customer service or a real-time operator workflow, tail latency has a monetary cost through employee idle time, repeated requests, and tool abandonment.

Comparing local deployment with an external API

A local environment has a high fixed cost and a low marginal cost per additional request until it saturates. An API usually turns more of the cost into a variable expense. The break-even point therefore depends on utilization as well as tokens.

Compare both scenarios with the same formula:

`total monthly cost / SLA-compliant accepted outcomes`.

Local cost must include hardware, a standby node or accepted downtime, electricity, administration, monitoring, updates, and evaluation. API cost must include input and output tokens, retries, network failures, a fallback provider, data controls, and human review. Treat data-location requirements and operation without an external connection as separate business constraints.

A bursty service used for only a few hours per week is often better started through an API or hybrid design. Stable daytime demand, sensitive data, and high utilization make a local design more interesting. Low utilization, however, can turn a dedicated GPU into an expensive heater with an excellent benchmark.

Management checklist before procurement

Before requesting a quote, record:

  • production p50 and p95 input and output lengths;
  • peak request rate and concurrency;
  • TTFT and completion-time SLA;
  • share of background tasks and priority policy;
  • accepted outcome rate and acceptable outcome cost;
  • data, availability, and recovery requirements;
  • overload behavior: queue, early rejection, or spillover to an external API.

Ask the vendor to run your de-identified dataset and return raw results, engine configuration, and component versions. Average speed on a synthetic short prompt is not evidence of capacity for your process.

Next step

Collect 200–500 real, de-identified requests and replay one workday peak. Start with one node—or a rented test server—run the concurrency ladder, and identify the first level where p95 crosses the limit. Then calculate the cost of an accepted outcome inside the SLA rather than the cost of a token or request.

This turns a local LLM decision from a debate about graphics cards into ordinary capacity planning: a known workload, measurable service level, explicit reserve, and understandable economics.