Short answer

A cascade of a small and a large LLM can materially reduce inference cost, but only if the company measures the cost of an accepted outcome rather than the share of requests sent to the cheaper model. Retries, validation, human review, and unnoticed errors all belong in the calculation. For a small or midsize business, the safest starting point is usually not a sophisticated learned router. It is a set of risk rules, a small model running in shadow mode, and an explicit escalation threshold.

RouteLLM and FrugalGPT show that routing between models can improve the quality-to-cost trade-off. Their reported savings, however, come from particular datasets, models, and price assumptions. Those percentages should not be copied into the budget of a support team, sales department, or internal RAG system without a company-specific sample and a company-specific cost of error.

What exactly is being routed

There are two similar but economically different patterns.

The first is routing before generation. A classifier or router evaluates the request and sends it directly to one of two models. RouteLLM studies this setup: one model handles each request. The advantage is that double generation is not mandatory. The risk is that an incorrect choice of the weaker model can remain undetected.

The second is a cascade. The small model answers first, then a validator decides whether the result is reliable enough. If not, the request and the required context are sent to the stronger model. FrugalGPT describes this sequential approach: later models are called until a response is considered sufficiently reliable. It is easy to explain operationally, but frequent rejection means paying for both models while the user waits for both.

Processes with different error costs can combine the patterns:

  • route payments, legal wording, personal data, and writes to a system of record directly to the strong model or a human;
  • use the small model for ticket classification, simple field extraction, and drafts of routine replies;
  • escalate ambiguous requests and outputs that fail validation;
  • execute external actions through a deterministic service after approval, not through the LLM itself.

Worked model: savings before and after error cost

The following is not a market price quote. It is a model with explicit assumptions. A company processes 20,000 requests per month. Internal transfer cost is assumed to be RUB 0.35 per small-model call, RUB 2.80 per strong-model call, and RUB 0.05 per request for routing and tracing. These figures are assumed to include GPU rental or depreciation, electricity, and basic operations. Integration development is treated separately.

The router sends 65% of requests to the small model. Of those requests, 18% fail validation and are processed again by the strong model.

Cost before unnoticed errors:

  • small model: 13,000 × RUB 0.35 = RUB 4,550;
  • strong model directly: 7,000 × RUB 2.80 = RUB 19,600;
  • strong model after cascade rejection: 2,340 × RUB 2.80 = RUB 6,552;
  • router and tracing: 20,000 × RUB 0.05 = RUB 1,000;
  • total: RUB 31,702 instead of RUB 56,000 if every request used the strong model.

At this level, the saving is RUB 24,298, or approximately 43%. It looks compelling until the model includes the cost of an incorrectly accepted response.

The small model completed 10,660 requests without escalation. Assume that 0.3% of these results passed automated validation incorrectly and that correcting each case costs an average of RUB 300. The additional loss is roughly RUB 9,594. Real savings fall to RUB 14,704, or approximately 26%.

Under the same assumptions, break-even occurs when unnoticed errors among accepted small-model outputs approach 0.76%. That is the central management lesson: a rare error can consume the benefit of cheap inference when it creates a return, repeated employee work, or an incorrect operation in a business system.

The general monthly cost formula is:

`N × Crouter + N × p × Csmall + N × (1 − p + p × f) × Cstrong + N × p × (1 − f) × e × L`

Here, `N` is request volume, `p` is the share routed to the small model, `f` is the fallback rate, `e` is the unnoticed-error rate among accepted small-model outputs, and `L` is the average loss per such error.

Why benchmark savings are not your TCO

RouteLLM reports substantial cost reductions on MT Bench, MMLU, and GSM8K while retaining a target share of strong-model performance. This is useful evidence that routing can work, but it is not a ready-made business case. A production workload differs in at least five ways.

First, quality must be defined as acceptance of a specific business outcome rather than general capability. Were the account details extracted correctly? Was the right ticket category selected? Did the reply comply with policy? Can the draft be sent to a customer?

Second, the cost of a local model is not linear in tokens. If a separate GPU is reserved for the strong model and remains idle, reducing its request count may not reduce the bill. Savings appear when the company can reduce rented capacity, improve useful cluster utilization, or postpone the next hardware purchase.

Third, a cascade can increase latency. When the weak model fails, the user experiences the sum of the first generation, validation, queueing, and the second generation. That matters more in an interactive assistant than in an overnight batch process.

Fourth, two models mean two prompt versions, two observability profiles, separate tests after weight updates, and possible cache fragmentation. This is recurring operations work, not a one-time threshold setting.

Fifth, the LLM's own confidence is a weak single signal because a model can be confidently wrong. A more reliable decision combines process risk, request features, deterministic output checks, and statistics from a labeled sample.

A minimal architecture without magic

A practical local design contains eight elements:

1. A gateway authenticates the user, enforces access rights, removes prohibited fields, and assigns a trace ID.
2. Risk rules immediately exclude operations that the weak model must not handle.
3. A router evaluates domain, length, tool requirements, output type, and similarity to labeled examples.
4. The small model produces a response or structured result.
5. A validator checks the JSON schema, required fields, source references, business rules, and quality threshold.
6. The strong model receives the original request and required data if the first attempt is rejected. Automatically passing the weak draft forward can anchor the stronger model on the original error.
7. A human handles high-risk and disputed cases.
8. The trace records the selected model, routing reason, token counts, latency, validation result, and final correction without retaining unnecessary personal data.

Data required before switching traffic

The golden set should be stratified by business risk rather than assembled from random conversations:

  • common routine requests;
  • long or multi-step requests;
  • rare categories;
  • cases containing personal or commercially sensitive information;
  • requests that require a tool or a write to an external system;
  • historical errors and employee escalations.

Each example needs an acceptable outcome, a risk class, and an error cost. Experts can compare open-ended outputs pairwise; structured tasks should use schemas, reference data, and reconciliation with the source system.

The threshold should not maximize the share handled by the small model. The team should build a cost-versus-quality curve and track separately:

  • accepted small-model outputs;
  • fallback rate;
  • unnoticed small-model errors;
  • cost per accepted outcome;
  • p95 latency for direct and cascade routes;
  • human-review load;
  • results for every risk class.

What to measure in local inference

The vLLM documentation exposes the metrics needed to replace spreadsheet assumptions with operational economics: running and waiting requests, KV-cache utilization, prompt and generation tokens, time to first token, inter-token latency, end-to-end latency, and queue time. Model reloads, memory preemption, and queue spikes should also be attributed to the accepted outcome.

If both models share one GPU, the pilot must test interference between their queues. If they use separate GPUs, the calculation must include reserved capacity rather than only active seconds. A batch workflow can fill a low-cost queue overnight; a customer-facing chat needs headroom for p95 latency.

A four-week acceptance pilot

During week one, collect 1,000–3,000 representative requests, remove unnecessary personal data, and label risk and acceptance criteria. In week two, run both models and a simple router in shadow mode without changing user-visible responses. In week three, compare two or three thresholds on cost, quality, and p95. In week four, enable the small model only for the low-risk layer with automatic fallback to the strong model.

Four conditions should be met before traffic expands:

  • savings remain positive after error and human-review costs;
  • unnoticed errors stay below the approved limit for every risk class;
  • cascade p95 remains within the SLA;
  • every model update triggers reevaluation and the system can quickly return all traffic to the strong model.

The next management step

Choose one high-volume process with cheap result validation: ticket classification, fixed-field extraction, or template-based drafting. Measure strong-model cost as the baseline, add the cost of error, and run a shadow test. If savings disappear at an unnoticed-error rate below one percent, improve validation and data before investing in a more complex router.

A model cascade is a budget-control mechanism: routine work goes to the cheaper executor, difficult work to the stronger one, while rules and human accountability prevent savings from becoming a correction bill.