Why a conventional cache is not enough

A conventional cache only answers an exact key match. To it, “How do I return an item?” and “What is the return policy?” are different requests. A semantic cache compares vector representations and may reuse a previously generated answer for a similar formulation. In FAQ, internal support and service-desk workloads, this can avoid running the full embedding → retrieval → generation pipeline again.

The cost of a wrong hit is much higher than for a cached image or CSS file. Two similar questions may require different answers because of branch, contract, employee role, document date or the previous conversational turn. A semantic cache therefore cannot be placed in front of RAG as an unconditional accelerator. It is a separate decision layer with hard applicability boundaries.

Redis documentation describes a cache entry as the prompt, its embedding, the response and metadata such as tenant, locale, model version and safety flags. Microsoft gives a simple conversational counterexample: the same follow-up, “What is the second largest?”, means different things after different preceding questions. Vector similarity alone does not prove that an answer is reusable.

What can actually be cached

Smaller businesses should distinguish three mechanisms:

  • exact caching for the same normalized request while the data is unchanged;
  • semantic caching for similar requests within one strictly defined context;
  • inference-server prefix or KV caching, which reuses model computation but not a finished business answer.

Semantic caching is best suited to stable reference answers: opening hours, how to submit a request, the composition of a standard service or a general returns procedure. Poor candidates include inventory, prices, order status, personal data, calculations, legally significant decisions and anything that depends on current CRM or ERP state.

A useful rule is that an answer is a cache candidate if it could safely be printed as a pre-approved card. If the answer requires a transaction, an authorization check or a fresh system query, the cache may at most hold the plan for obtaining the data, not the finished result.

Architecture with two gates

A reliable design combines hard and soft matching.

The application first creates a contextual key. It includes tenant or organization, role and access group, language, channel, system-prompt version, embedding-model version, generation-model version, safety-policy class and knowledge-set version. These fields are checked with exact filters. An entry from another namespace must never participate in vector search.

Only then does the system search for a similar request. The similarity threshold is tuned on real questions rather than copied from a library example. A candidate must pass additional conditions:

  • its TTL has not expired;
  • its knowledge version is still active;
  • the current user does not have narrower rights than the context that produced the answer;
  • the answer was validated and contains no personalized fragments;
  • the request is not classified as “fresh data only”;
  • conversational context is either part of the key or prohibited for this route.

Any uncertainty becomes a cache miss and runs the normal RAG path. A miss adds latency; a false hit serves the wrong or another party’s answer. These errors are not equivalent, so a business system should tune toward safe misses.

Avoiding yesterday’s truth

TTL limits lifetime but does not solve freshness. A policy may change one minute after the cache was filled. Targeted invalidation is required.

Each entry should retain the source document identifiers or the knowledge snapshot version that grounded the answer. When a document changes, is deleted or loses availability, the publishing pipeline emits an invalidation event. The cache removes dependent answers or advances the active knowledge version so old entries become unreachable.

For an initial pilot, versioning a small domain is often simpler than tracking every chunk: `returns-v17`, `hr-policy-v9` or `service-catalog-v24`. Updating the domain switches reads immediately; old entries can be retired in the background. This approach is easier to verify than a complex dependency graph and supports rollback.

There must also be a manual delete-by-ID operation for a known bad answer. Redis demonstrates such targeted deletion in its integration documentation. Flushing the whole cache should remain an emergency control, not the routine update mechanism.

Isolation and logging

A semantic cache is effectively a database of ready-to-serve answers. It inherits the data controls of the RAG system: least privilege for the service identity, encryption, retention limits, backups and auditability.

Tenant and authorization filters should be applied inside the index query, not after nearest-neighbor retrieval. Otherwise the system retrieves another tenant’s entry first and only then tries to discard it, increasing leakage risk and relying on application code to repair the boundary.

Logs should not contain the complete request and response by default. Operations usually need identifiers, versions, similarity distance, hit or miss, rejection reason, latency and a hash of the normalized request. Content sampling should be limited, separately authorized and retained for a defined period.

Measuring economics without fooling yourself

Hit rate is not the primary success metric. A high hit rate may simply mean the threshold is too permissive. At least four measures are needed:

  • safe-hit precision on a labeled set;
  • false-hit rate, especially across intents and authorization boundaries;
  • compute and latency saved per accepted answer;
  • the share of answers withdrawn after source updates.

Consider a model calculation: 20,000 monthly questions, 30% potential repetitions, a full pipeline cost of one notional ruble and a cache lookup cost of 0.05 ruble. If 80% of candidates are confirmed safe hits, gross savings are about 4,560 rubles: `20,000 × 0.30 × 0.80 × (1 − 0.05)`. This is an illustration, not a market promise. Infrastructure, engineering, quality control and error costs must still be deducted.

For a local model, token fees may be zero, but GPU time, queueing and p95 latency remain. If traffic is low and hardware is idle, semantic caching may not justify its complexity. Measure request repetition before buying another platform.

A two-week pilot

Collect 300–500 anonymized questions from one stable process. Label pairs as “reusable,” “similar but different answer,” “fresh data required,” or “must not be cached.” Dangerous near-duplicates are especially valuable: the same product in different regions, the same job title in different legal entities, or the same follow-up after different dialogue turns.

Run the first week in shadow mode. The cache proposes a candidate, while the user still receives the normal RAG answer. Compare outcomes and disagreement reasons. After calibration, enable hits for one low-risk domain only, with TTL, knowledge versioning and an immediate kill switch.

Define the launch gate in advance. One example is zero cross-tenant hits, no more than 0.5% false hits on the control set, 100% removal of test entries after a version change, and measurable p95 improvement. The threshold is a business decision, not a universal constant.

What management should ask for

Ask the team to show five things instead of a hit-rate chart: the exact contextual key, the list of prohibited data classes, the invalidation mechanism, a near-duplicate test, and the emergency disable switch. If any one is missing, the semantic cache is still an experiment.

For a smaller business, a two-week shadow pilot on a single FAQ domain is the sensible next step. It reveals actual repetition and savings without turning an accelerator into a new source of stale or cross-context answers.