A good RAG system does not always answer
An enterprise assistant can sound confident even when retrieval returns a neighboring topic, an expired policy or only half of the required conditions. A system prompt saying “if you do not know, say you do not know” is useful, but it is not a control architecture.
Google's work on sufficient context separates two failure causes: the required evidence is absent from retrieved documents, or the model fails to use context that was actually sufficient. These are different incidents with different remedies.
The harder case is context that is not empty but convincingly wrong. A recent controlled preprint on small local RAG models highlights an important limitation: explicit abstention prompting works reasonably well when evidence is missing, but much worse when a plausible false fact is planted in context. The right not to answer should therefore be a separate pipeline decision, not a sentence in the prompt.
Three outcomes instead of two
A basic chatbot has two states: answer or error. A production RAG system needs at least three:
1. **Answer.** Evidence is sufficient, sources are admissible, and key claims can be linked to specific passages.
2. **Clarify.** The request is ambiguous: department, contract version, date, customer or transaction type is missing.
3. **Escalate.** Evidence is absent, conflicting, stale or the cost of an error exceeds the accepted threshold.
This is not degraded service. A well-designed abstention saves time by giving the user a reason, the missing information and a clear next step instead of a plausible invention.
Why similarity score is not enough
Vector similarity answers a narrow question: is a chunk close to the query in the embedding model's space? It does not prove that:
- the user is authorized to access the document;
- the document is valid for the relevant date;
- the complete set of conditions was retrieved;
- the source is authoritative rather than a discussion;
- multiple documents do not conflict;
- the answer follows only from the retrieved facts.
A threshold such as `score > 0.78` is convenient, but it cannot be transferred blindly across embedding models, collections and question types. It also cannot distinguish one authoritative source from five near-duplicate copies of weak evidence.
An evidence-sufficiency gateway
A practical architecture places a separate answerability gate between retrieval and generation. It receives the question, retrieved passages and metadata, but does not yet produce a polished answer.
Step 1. Deterministic rules
Start with conditions that should not be delegated to an LLM:
- authorization passed for every document;
- a current version and effective date are present;
- mandatory source types were retrieved;
- enough independent documents are available;
- no explicit version conflict exists;
- freshness limits are respected;
- required fields, amounts or identifiers are present.
A returns-policy question, for example, may require current sales terms, the product category and the purchase date. One relevant paragraph without a date does not make the evidence package sufficient.
Step 2. Evidence-package classification
Next, a dedicated classifier or LLM with a strict structured schema evaluates evidence rather than drafting the answer:
- `sufficient` — generation is allowed;
- `needs_clarification` — a query parameter is missing;
- `conflicting` — sources disagree;
- `insufficient` — supporting evidence is absent;
- `restricted` — evidence exists but the user cannot access it.
Return missing premises and passage identifiers with the class. Free-form reasoning without structure is difficult to validate or use for routing.
Step 3. Generate only after admission
The generator receives context only after the gate approves it. Claims must cite evidence IDs. A post-generation check then matches material claims against the retrieved passages.
LangSmith's RAG guidance separates evaluation into answer correctness, answer relevance, groundedness against documents and retrieval relevance. A production gate adds a fifth question: was the task answerable from the admissible evidence package in the first place?
Set thresholds from the cost of error
There is no universal confidence threshold. It depends on the workflow.
Navigation across an internal portal can tolerate broader coverage with a warning. Calculating a benefit, interpreting a contract or changing an ERP record should escalate more often.
Track two linked metrics:
- **coverage** — the share of requests answered automatically;
- **selective risk** — the error rate among requests the system chose to answer.
Raising the threshold usually reduces coverage while improving accepted-answer quality. The target is not maximum automation; it is the point where the cost of the review queue is lower than the expected damage from wrong answers.
Consider a model calculation. A business receives 10,000 monthly requests. At 80% coverage and 3% risk, the system produces 240 wrong accepted answers. After tightening the gate, coverage drops to 55% and risk to 0.7%: roughly 39 errors and 4,500 clarification or human-review routes. The correct choice depends on the cost of one error versus one exception; the percentages must come from the company's own dataset.
Build the right test set
A dataset containing only answerable questions cannot test abstention. At least a third of examples should represent situations where not answering is the correct behavior.
Include five groups:
- answerable questions with complete context;
- questions with no retrieved evidence;
- questions with relevant but insufficient passages;
- questions with stale or conflicting documents;
- questions with a persuasive false passage beside a correct source.
For each example, store the expected route — answer, clarify or human — required evidence, forbidden sources, critical claims and error cost.
Evaluate retrieval and generation separately. Otherwise a pipeline can earn a good score by accident: weak retrieval is masked by parametric model knowledge, or correct retrieval is damaged by generation.
Do not trust a single judge
LLM-as-judge is useful for groundedness and semantic correctness, but it should not be the only safety mechanism. A judge can fail too, especially when it belongs to the same model family as the generator.
Combine:
- deterministic metadata and authorization rules;
- reference answers for critical scenarios;
- mandatory evidence-ID checks;
- an LLM judge for semantic comparisons;
- manual review of a random sample;
- a dedicated hard-negative set.
Microsoft Foundry defines groundedness as alignment between the answer and provided context and supports threshold-based pass/fail evaluation. That is valuable after generation, but it does not decide whether the context was sufficient and admissible before generation began.
What the user should see
Abstention should not look like a technical failure. A good interface explains:
- why no answer was produced;
- exactly what information is missing;
- which sources were checked;
- which clarifying question would help;
- where the request was routed and when a response is expected.
Do not expose the internal similarity score as a confidence percentage. Users can easily mistake it for the probability that the answer is correct. Use meaningful statuses instead: “no current version found,” “contract date required,” or “sources conflict.”
A two-week pilot
Choose one workflow and 100–200 real questions. Label evidence sufficiency and the expected route. Then:
1. add deterministic checks for authorization, version and mandatory sources;
2. implement a structured answerability gate;
3. allow generation only for `sufficient` cases;
4. measure coverage and selective risk;
5. analyze false admissions separately from unnecessary refusals;
6. set the threshold from error cost;
7. run for one week in shadow mode beside an employee.
The goal is not to teach a chatbot to say “I don't know” more often. It is to make the answer-or-escalate decision observable, reproducible and economically manageable.