Local does not mean free

For a small business, the practical question is usually: when is it cheaper to pay for each call to an external model, and when does it make sense to operate a model inside your own infrastructure? There is no universal break-even point. It depends not only on the API rate, but also on volume, prompt length, latency requirements, and the cost of the people who maintain the system.

A common mistake is to compare the API bill only with the price of a graphics card. A local deployment still has hardware depreciation, electricity, backup capacity, upgrades, monitoring, administrator time, and the cost of downtime. An API may also involve tools, storage, network constraints, and extra checks in addition to token charges. Compare the cost of the same completed business operation, not a model price in isolation.

Start with one repeatable operation

A good candidate has a clear input and a verifiable output: classifying incoming requests, extracting a few fields from a standard document, routing a ticket, or drafting a response from an approved knowledge base. Do not start with “let the model help with everything.” That mixes many scenarios and makes both cost and error rates hard to interpret.

For the pilot, record:
- monthly operation volume and how it is distributed across the day;
- average input and output size, including system instructions and context;
- the share of requests that need human review;
- acceptable latency and the cost of an error;
- minimum availability and data-protection requirements.

Test with real, de-identified examples. A cheap answer that employees must check and rewrite every time is not a cheap solution.

The comparison formula

Let `N` be monthly operations, `C_api` the average full cost of one successful API operation, and `C_local` the monthly cost of the local setup. A simple break-even equation is `N × C_api = C_local`. Below the threshold, variable billing may be more sensible. Above it, a local option deserves a separate test—but only if quality is acceptable and the hardware is used enough.

Include these items in the model:
- API: input and output tokens, retries, a classifier or verifier, tools, and taxes or currency conversion where relevant;
- local: server depreciation, electricity, backup hardware or spare capacity, support, model updates, and downtime;
- both: integration, logging, quality control, and manual handling of exceptions.

A small illustrative example

Assume a local server and support together cost the company RUB 14,000 per month. This is a rounded modeling assumption, not a market quote. If the API option costs an average of RUB 1.50 per successfully processed operation, the mathematical threshold is about 9,334 operations per month: `14,000 / 1.50`.

That is not yet a purchasing decision. If the local model performs worse on the documents in question and staff spend more time correcting results, it can lose even above the threshold. If the server would support only one workflow with 800 monthly operations, fixed costs may be difficult to justify. Recalculate `C_api` using actual token counts and the current rate for the selected model before moving ahead: prices change, and long context can materially alter cost.

When local inference fits

Open tools such as llama.cpp can run quantized models on CPUs and GPUs, provide a server interface, and support parallel requests. This lowers the barrier to experimentation, but it does not remove engineering work: pin the model version, check the license of the actual weights, secure administrative routes, restrict access, collect metrics, and test upgrades.

For a local deployment, pay particular attention to:
- load testing on your own hardware rather than relying only on someone else’s benchmark;
- throughput at the real number of concurrent users;
- memory requirements for weights and context;
- backups, upgrades, and recovery procedures;
- quality on Russian-language prompts and your typical data.

An API is easier to start with and can absorb demand spikes. A local model can provide more control over data and a predictable marginal cost at high utilization. Neither advantage automatically guarantees savings.

A two-week practical plan

1. Choose one repeatable operation and collect 100–300 de-identified examples with reference answers.
2. Measure quality, manual correction rate, latency, and token use through an API.
3. Test one locally runnable model on the same set and measure load on the target hardware.
4. Put variable and fixed costs in one table; use actual volume rather than a “we might grow” forecast.
5. Set stop conditions: quality below threshold, excessive manual review, or no acceptable payback within the business’s planning horizon.

This is a cost-of-ownership exercise for a workflow, not a model contest. If the numbers do not work for one operation, simplify the task, reduce context, or use the model only where it demonstrably helps.