A thumbs-down is a signal, not a diagnosis

An employee asks a corporate RAG assistant about a product return deadline and marks the answer as wrong. The team could immediately rewrite the prompt, add more documents or blame the model — Vnutrik has already opened a folder for that last option. But none of those steps follows from a single click. The cause could be an outdated policy, failed retrieval, a bad summary of the correct clause, or a question about a different branch.

For a small business, feedback becomes valuable only when a complaint turns into a reproducible case, a specific fix and a repeatable check. Langfuse documentation illustrates the technical chain: user feedback can be linked to the answer's trace, sent to human reviewers and then saved as a test case. This is an example of available tooling, not a requirement to deploy Langfuse. NIST's AI Risk Management Framework also treats ways for users to report problems as part of system measurement and evaluation.

What context to keep with a complaint

If all you retain is “thumbs-down” and the answer text, a week later it may be impossible to tell what the assistant saw. A minimal incident record should include:

  • Request identifier, time, business process and a de-identified user role.
  • Original question and answer only to the extent they may lawfully and appropriately be retained; a protected reference and redacted copy may be preferable to full text.
  • Retrieved document IDs, versions or effective dates, supporting passages and the outcome of access checks.
  • Search-index, model and prompt versions, retrieval settings, and any tool error code.
  • User rating, a short comment and review status: new, confirmed, fixed or not reproducible.

This is our proposed minimum for a project, not a list of mandatory Langfuse product fields. There is no need to dump entire customer contracts into telemetry “just in case.” Langfuse's masking documentation reinforces a technical principle: redact sensitive data at the application side before sending it to an observability system. For a local deployment, separately decide who can inspect traces, how long they remain and whether a source link can reveal the underlying document.

Five distinct causes of “wrong”

Triage should follow the point where the answer pipeline broke, not the strength of the user's reaction.

1. **The correct rule is missing from the knowledge base.** The process owner updates the source, its effective date and its publication owner. A new model cannot invent an absent policy.
2. **The rule exists, but retrieval missed it.** Test document splitting, branch filters, keywords, hybrid search and ranking using the same question.
3. **The right passage was found, but the answer distorted it.** Change the answer format, citation requirement, output constraint or model, then retest against that passage. A tidy citation does not make a false conclusion correct.
4. **The answer came from the wrong role or version.** This is an access-control or freshness defect, not a minor hallucination. Until it is fixed, block this class of answers or escalate to a person.
5. **The question is ambiguous, or the expectation is wrong.** The user might mean a different product, region or date. The assistant should ask rather than confidently pick a random interpretation.

For mixed failures, record a primary cause and secondary factors. Changing the index, prompt and model in one go makes it hard to know which fix worked. Vnutrik's joke stops here: a wrong price, deadline or access rule can be expensive for the business.

From complaint to verifiable change

For every confirmed case, a domain expert specifies expected behaviour: a correct answer tied to a particular source version, or a justified refusal and clarification question. In Langfuse, annotation queues let reviewers score traces and record corrected outputs; datasets let a production example become a repeatable test. These functions neither retrain a model automatically nor prove that a change is safe.

The test must include more than the one broken question. Add neighbouring cases: another branch, an older rule version, a similar product and a user without access. Run the old and new configurations on the same set. Compare correct answers, confidently wrong answers, appropriate abstentions, latency and cost. If improving one case breaks ten nearby ones, hold the release.

It helps to separate roles: the policy owner confirms meaning; the domain expert labels the case; the engineer changes retrieval or prompting; and a release owner approves deployment. One person may cover several roles in a small firm, but the decision and its owner should still be recorded. For high-risk answers, such as payments or employment terms, escalation to a human should remain available even after a fix.

The economics of the feedback queue

Not every thumbs-down deserves the same hour of work. Prioritise recurring and costly errors: a wrong service deadline, contract term, payment detail or access restriction. A rare question about a minor detail can wait. A useful measure is not the number of ratings collected but the share of confirmed issues closed by a verified change without regression on the control set.

Here is an illustrative calculation: over two weeks, an assistant receives 1,000 questions and 60 negative ratings. An expert reviews 40 and confirms 24, of which 10 stem from one outdated policy. Updating that source and its tests may prevent more repeated errors than expensive model fine-tuning. These figures are hypothetical, not a company result. Annotation, integration and maintenance costs should be weighed against prevented mistakes and employee time.

Ratings themselves are biased: annoyed users may complain more often, while silence does not mean an answer was correct. Add a random sample of ordinary answers and expert review of important workflows. User signals help find failures but do not replace independent quality assessment.

Start without a platform programme

For a two-week pilot, choose one process and one knowledge owner. Retain answer IDs, source versions and the minimum safe context. Review negative ratings every few days, assign a cause category and expected behaviour. Turn each confirmed defect into a test, then compare configurations on the same set before release. A spreadsheet and a simple log can suffice at first; a dedicated evaluation platform makes sense when volume and team size make manual tracking unreliable.

Success is less glamorous than claiming an assistant “teaches itself”: the team knows why an answer was wrong, fixes the right layer and can demonstrate that neighbouring answers still work. Vnutrik may collect thumbs-downs by the stack; people remain responsible for repairing the knowledge base.