Why a configuration limit is not yet cost control

Once a company launches a second AI service, a shared model key quickly becomes an opaque cash register. The team cannot reliably identify which product created load, which workflow consumed tokens, or whether a limit will hold during a burst of concurrent requests. This is why organizations often place an LLM gateway between applications and models: it issues virtual keys, routes traffic, records usage, and applies limits.

LiteLLM provides a useful example of an important architectural constraint: a budget field alone is not a financial circuit breaker. The official documentation explicitly says budgets require a database. In a database-less deployment, a global budget has no accumulated spend value to compare against and therefore allows requests to continue; key, team, and user budgets are unavailable as well.

For an SMB, the practical lesson is to test the entire admission path rather than merely check for a `max_budget` line: client identity, durable accounting, a concurrency-safe counter, and defined behavior when infrastructure fails.

What the official documentation confirms

LiteLLM describes its gateway as a central layer for authentication, rate limits, budgets, and routing. PostgreSQL stores keys, teams, and spend records, while Redis supports fast counters and coordination across gateway replicas.

Four budget details matter.

  • Without a connected database, current spend cannot be established. The global budget therefore behaves fail-open and requests continue.
  • Budget reservation is enabled by default. Before a request, the gateway estimates its maximum cost, temporarily reserves that amount, rejects the request if the cap would be crossed, and replaces the reservation with actual spend after completion.
  • If reservation is disabled, concurrent requests can all pass against the same older total and collectively exceed the limit.
  • For a hard ceiling, LiteLLM documents `fail_closed_budget_enforcement`. This mode validates spend against the authoritative database; if neither Redis nor the database can confirm the current amount, the gateway returns 503 instead of admitting unverifiable traffic.

Batch jobs are a separate limitation. At submission time, the gateway receives an input file identifier and cannot fully price every prompt inside it. Final spend becomes available when the batch completes. Batch traffic therefore needs its own throughput limit and post-completion reconciliation.

A minimum architecture for a business

A practical design looks like this:

1. The CRM assistant, internal copilot, RAG service, and agents call one gateway rather than providers directly.
2. Each product or business process receives a separate virtual key. One company-wide key prevents attribution and makes it difficult to disable a single faulty workload safely.
3. PostgreSQL holds key owners, teams, limits, and spend logs. It should be backed up and recovered with the same discipline as a financial accounting component.
4. Redis accelerates counters and rate limiting. If a monetary ceiling must be strict, the behavior during Redis failure is configured in advance and tested.
5. Provider secrets remain in the gateway or a secrets manager. Applications get virtual credentials restricted to the models and routes they actually need.
6. Metrics are exported to an external monitoring system: request count, tokens, errors, latency, budget rejections, and the difference between calculated spend and provider invoices.

The same layer is useful in a hybrid setup. A logical route can send ordinary traffic to a local model, difficult requests to an external API, and fallback traffic to another deployment. However, a financial cap does not replace technical quotas. RPM and TPM limits, concurrency limits, maximum output length, and model allow-lists are still required.

Accounting for local models

With a cloud API, cost is usually derived from tokens and a provider price map. For a local model, a zero token price is misleading: the business still pays for GPU or CPU capacity, energy, weight storage, standby redundancy, operations, and support.

Internal allocation therefore needs a separate rate. A simple model is:

`hourly infrastructure cost / useful tokens per hour × request tokens`.

Hourly cost should include rental or hardware depreciation, power, redundancy, and operations labor. Useful throughput must be measured with the real request mix, not a tiny synthetic prompt. When local inference and external APIs sit behind the same gateway, reporting should separate actual provider charges from modeled internal cost.

Another risk is stale pricing data. LiteLLM's documentation recommends keeping the model cost map current. Financial reconciliation remains necessary: daily or weekly, compare gateway calculations with the provider invoice over the same period, models, and token categories, including cached or service-specific token types.

Common failure modes

The first is reusing one key across applications. The cap may trigger, but the business still does not know which workflow to stop. The second is trusting a dashboard limit without a concurrent load test. The third is relying only on money: a long response or an unbounded agent loop can exhaust capacity before reaching the monthly budget.

Failure semantics also matter. A test assistant may reasonably fail open when availability is more important. An agent capable of launching many expensive operations should normally be denied when remaining budget cannot be verified. This is a business control decision, not merely a library default.

Finally, a budget is not a data protection mechanism. The gateway still needs role separation, an administrative audit trail, sensitive-field redaction, and prompt logging disabled by default wherever prompts can contain personal or commercial information.

An illustrative pilot calculation

Assume three internal AI services make 120,000 requests per month. The modeled average cost of a normal request is RUB 0.90, while 5% of requests carry long context and average RUB 9.00. The baseline monthly total is about RUB 156,600:

  • 114,000 ordinary requests × RUB 0.90 = RUB 102,600;
  • 6,000 heavy requests × RUB 9.00 = RUB 54,000.

If a faulty agent creates another 10,000 heavy requests, the increment is RUB 90,000. A per-key budget plus an iteration limit can stop that workload without disabling the other products. This is an illustrative model, not a LiteLLM case result; actual figures depend on provider prices, context length, caching, and the share of local inference.

A two-week pilot

During week one, connect one non-critical service, give it a dedicated virtual key, and collect a baseline without hard blocking. Measure cost and latency distributions, error rate, peak concurrency, and the five most expensive request types.

During week two, add an alert at 70% of the budget, a soft control at 90%, and a hard stop for the test key at 100%. Then simulate a concurrent burst, Redis unavailability, and loss of the database connection. The test target is not a visually appealing chart but observable behavior for every failure mode.

The pilot succeeds when four outcomes are demonstrated:

  • every cost is attributed to a product and owner;
  • limits hold under concurrent load;
  • storage failure produces the chosen fail-open or fail-closed behavior;
  • gateway calculations are reconciled with provider invoices and internal infrastructure cost.

An LLM gateway turns model consumption into a managed service only when the counter, durable storage, and failure policy form one system. A configuration value without that system creates the appearance of control, and that illusion is usually more expensive than the token itself.