Local LLM Prefix Cache: Preventing Cross-Tenant Prompt Timing Leaks

An optimisation that changes the trust boundary

A local LLM server often serves several departments, branches, or customers. To avoid recomputing the same beginning of a long request, the inference engine can use automatic prefix caching: it stores KV-cache blocks for a processed prefix and reuses them when a new request starts with the same tokens.

This optimisation is particularly valuable in business workloads. A large system prompt, tool schema, response policy, or standard document may recur in hundreds of requests. Caching reduces time to first token and computation without changing the model’s output.

A shared cache, however, becomes a shared technical surface. If requests from different trust zones use one key space, the difference between a fast cache hit and a slower miss can reveal that a particular prefix exists. Research on timing side channels shows that adaptive guesses may recover sensitive parts of another request.

The mechanism should not be exaggerated. A prefix cache does not ordinarily return another user’s text directly. The risk comes from a side signal: latency. This is why ordinary API authentication is insufficient when one service credential represents different customers or departments.

How prefix caching works

On the first request, the engine computes internal key-value states for blocks of input tokens. Complete blocks are associated with a hash that incorporates the preceding prefix and the current block. When a second request has the same beginning, the server finds matching blocks and resumes computation only where the requests diverge.

This creates two observable modes:

• a cold request computes the entire prefix;
• a warm request reuses part of the KV cache and usually starts generation sooner.

If a user can submit many requests and measure time to first token precisely enough, they obtain a signal that “this prefix was recently cached.” One signal reveals little. A sequence of adaptive guesses can narrow the possible content.

The most sensitive multi-user deployments are those where:

• one server serves legally independent customers;
• branches must not see one another’s requests;
• HR, sales, and legal teams use different data classes;
• an external portal and an internal assistant share an inference endpoint;
• an API gateway uses one technical credential for all users.

What `cache_salt` protects

In the current vLLM architecture, a request can include `cache_salt`. The value enters the first block’s hash, so the rest of the cache chain can match only requests using the same salt. The documentation explicitly presents this mechanism as prefix-cache isolation in multi-user environments and protection against latency-based inference attacks.

The management meaning is simple: the salt defines a group within which reuse is allowed.

• one salt per organisation preserves cache sharing among its employees;
• one salt per user provides stricter isolation;
• a fresh random salt for every request almost eliminates reuse and much of the cache benefit;
• one global salt for everybody recreates the original shared zone.

A salt is not a customer name. A predictable value such as `company-17` is unsuitable if a user can discover or provide it. vLLM’s API description recommends a random value that is sufficiently long and protected from third parties.

The API gateway is the correct enforcement point

The client application should not define its own cache boundary. Otherwise, an attacker may request another organisation’s salt or omit the field. A trusted gateway makes the decision after authentication.

A minimum flow is:

1. The user authenticates at the corporate API gateway.
2. The gateway obtains a verified `tenant_id` or access class from the server-side session.
3. It derives an opaque value from a protected master secret and the identifier, for example with HMAC.
4. The gateway forcibly inserts `cache_salt` and removes any same-named field supplied by the client.
5. vLLM accepts requests from the gateway, not directly from user networks.
6. Metrics treat requests without a salt as configuration violations.

The master secret must never enter prompts, logs, or client code. The model does not need the salt; the inference engine uses it to address cached blocks.

If one organisation contains nested trust zones, such as a shared handbook and private employee questions, one salt for the entire request may not deliver both maximum reuse and strict isolation. Without a tested multi-level design, choose the narrower boundary and accept a lower hit rate.

Hashing and salting solve different problems

A cryptographically strong hash reduces the risk of intentional collisions, where different blocks receive the same key. vLLM documentation states that recent versions default to SHA-256 and warns that non-cryptographic algorithms may increase collision and information-leakage risks in multi-tenant environments.

Salting solves a different problem. Even a perfect hash produces the same result for identical text. Without a namespace, two customers with the same prefix may reach the same cache and observe the timing difference. Both controls are therefore required:

• a safe hashing algorithm for key correctness;
• a separate salt for each trust zone.

The historical advice to “just upgrade vLLM” is necessary but insufficient. An upgrade fixes known defects and improves secure defaults; it cannot define the company’s tenant architecture.

What `cache_salt` does not protect

Prefix-cache isolation is a narrow control. It does not replace:

• authorisation for the API and agent tools;
• permission filtering for RAG documents;
• separation of logs, traces, and analytics;
• protection of conversation history and message queues;
• encryption of storage and backups;
• personal-data governance;
• updates to the inference engine and dependencies;
• protection against leakage through other shared caches or external connectors.

Even with a correct salt, the server processes the sensitive request. An inference administrator, debug trace, or excessively detailed log may access the data through another path.

Choosing an isolation level

One trusted internal process

If the endpoint serves only one team whose members have equal rights to the same data, one group salt may be reasonable. External portals, test accounts, and service robots should still sit outside that zone.

Multiple departments

The salt can follow an access class or department only when policy genuinely permits prompt sharing inside that group. An organisational label alone does not prove equal rights: HR staff and managers may work with different data scopes.

Multiple customers

For SaaS and managed service providers, the minimum boundary is one salt per customer organisation. If customer administrators must not observe individual users’ requests, a narrower boundary is required.

Highly sensitive operations

For HR, medical, legal, or investigative requests, begin with one salt per user or no cross-request caching. Performance can be reintroduced after testing rather than assumed before it.

Pre-production verification

One configuration line is not an acceptance test. Run a negative test with two synthetic tenants.

1. Send a long unique prefix from tenant A several times and confirm that repetition accelerates.
2. Send the same prefix from tenant B.
3. Use internal engine metrics to verify that B did not reuse A’s blocks.
4. Confirm that the gateway rejects an unsalted request or applies a safe isolated policy.
5. Repeat after upgrading vLLM, changing an adapter, or connecting an external KV-cache backend.

Latency alone is insufficient for acceptance because networking, queues, and batching add noise. Combine time-to-first-token measurements with an internal cache-hit counter. In production, track:

• the share of requests carrying the correct salt group;
• cache hit rate inside each trust zone;
• p50 and p95 time to first token;
• unexpected matches between synthetic tenants;
• changes after engine or configuration upgrades.

The economics of isolation

Stricter isolation reduces the number of matching prefixes and may increase GPU consumption. The choice is not simply “security or speed,” however. Much repeated context remains inside one organisation: its system prompt, documents, and tool schema can still be cached with an organisation-level salt.

Compare three profiles on actual traffic:

• a global cache as an unsafe control point;
• one salt per organisation;
• one salt per user.

For each profile, measure hit rate, time to first token, throughput, and cost per thousand approved answers. Then select the broadest group whose members are genuinely permitted by policy to know one another’s prompt activity.

A concrete next step

Map every user of one local inference endpoint. For each pair, ask whether each party is permitted to know that the other sent a specific prefix. If not, they must not share a prefix-cache namespace.

Next, move `cache_salt` derivation into the gateway, prevent client overrides, enable cryptographic hashing, add a two-tenant regression test, and only then compare performance.

The management conclusion is straightforward: local deployment does not automatically make a shared cache internal and safe. The trust boundary must exist not only in the organisation chart, but also in the cache key. Vnutrik loves reuse; people decide who is allowed to share the acceleration.