Why one retrieval method is not enough for RAG

A knowledge base receives two kinds of questions. The first needs an exact match: contract number, part number, surname, error code, policy clause, or abbreviation. The second expresses the same meaning in different words: an employee asks "how do I refund a payment," while the document is titled "procedure for returning funds."

Lexical BM25 search is strong at recognizing specific terms and rare tokens. Vector search compares semantic representations and can find paraphrases. Neither method wins reliably on every dataset. In BEIR, a benchmark spanning heterogeneous retrieval tasks, BM25 remained a strong baseline while more complex methods varied across domains. General leaderboards are not enough; retrieval must be evaluated on the organization's own questions.

Hybrid search runs both strategies, fuses their candidate lists, optionally reorders them with a more precise model, and only then gives a small set of passages to the generative model. Its job is not to make RAG "smarter" in the abstract. It reduces two specific failures:

  • a relevant document never enters the candidate set;
  • the right passage is retrieved but ranks below noise and misses the context window.

Where vector search fails

Embeddings are especially useful for synonyms, conversational queries, and documents that use different terminology. They can nevertheless miss a feature that is business-critical precisely because it must be exact.

A query for part number `AB-1047`, for example, may appear semantically close to descriptions of an entire product family rather than the single matching record. Short codes, software versions, ticket numbers, dates, and surnames carry little "meaning" but require literal matching. Mixed Russian and English, OCR errors, internal jargon, and new terms absent from model training make the problem harder.

Vector retrieval is sensitive to chunk size, embedded fields, and model version. Hybrid search cannot repair poor ingestion, but it provides another route to an exact passage.

Where keyword search fails

BM25 does not understand meaning as a person does. If the question and document share no important terms, the required passage may not enter the candidate list. Synonym dictionaries help but quickly become a separate product that must be maintained across departments, languages, and process versions.

Russian morphology, typos, and long natural-language questions require analyzers and field tuning. Lexical retrieval is not an obsolete fallback; it is a specialized signal to combine with semantic retrieval.

A practical four-stage pipeline

A minimum architecture looks like this.

1. **Query preparation.** The gateway identifies the user, permitted collections, language, and structural filters. Organization, department, document status, validity dates, and access rights are applied before retrieval, not after answer generation.
2. **Two independent retrieval branches.** BM25 and the vector index receive one normalized query and return, for example, 30 candidates each. Important identifiers must survive normalization. The vector branch may receive a human-readable expansion while the lexical branch keeps the original terms.
3. **Result fusion.** Reciprocal Rank Fusion, or RRF, assigns points according to each document's position in each list. A document high in both lists rises; a strong result from only one branch still has a chance. The system retains perhaps 20–40 unique candidates.
4. **Reranking and context assembly.** A more accurate model scores each question-passage pair and reorders the limited set. After deduplication and a token budget check, five to eight passages with provenance enter the LLM context.

The figures 30, 20–40, and five to eight are starting parameters, not universal rules. They should be chosen from retrieval recall, latency, and document size.

Why RRF is a useful pilot default

BM25 scores and cosine similarity measure different quantities. A lexical score of 12.4 cannot be compared directly with a vector similarity of 0.87. RRF avoids the problem by using rank positions rather than raw scores. OpenSearch documentation describes it as a sensible starting point when a team does not yet know how to align score scales and weights.

RRF has limitations. It ignores how much stronger the first result is than the second because only position matters. Its output depends on the number of branches and the rank constant, so teams should not apply one `min_score` across all queries or interpret RRF scores from different questions as absolute confidence.

When raw score margins are informative and a judged dataset exists, score normalization and tuned weights can be better. OpenSearch provides min-max, L2, and z-score normalization with multiple combination methods. That flexibility adds parameters, however, and should be paid for by measured improvement on the control set rather than by a more impressive configuration file.

What a reranker contributes

Fusion broadens coverage but does not always order similar documents correctly. A reranker sees the question and a candidate passage together, so it can model their exact relationship. A cross-encoder is usually more precise than comparing independently created embeddings, but it is more expensive because the model processes every pair.

This is why reranking is a second stage rather than a scan of the full index. If the fused list contains 40 passages, each user query creates 40 pairs. At one thousand daily queries, that is 40 thousand pair evaluations. Model size, passage length, batching, and hardware directly affect p95 latency.

A Russian corpus needs its own validation. The BAAI `bge-reranker-v2-m3` model card, for example, describes the model as multilingual and Apache-2.0 licensed, but this does not establish quality on a company's contracts, part numbers, or procedures. The model and its license should be versioned as part of the retrieval pipeline, then tested locally.

Building a control set

Without judgments, hybrid retrieval can become endless weight tuning. A pilot can start with 100–200 real questions collected from support logs, employee interviews, and routine operations. Secrets and personal data must be removed before annotation.

The set should cover distinct classes:

  • exact identifiers, dates, versions, and surnames;
  • natural paraphrases;
  • questions with several acceptable documents;
  • outdated or conflicting instructions;
  • requests for which the corpus contains no answer;
  • mixed Russian-English terms, typos, and abbreviations.

For every question, a domain expert rates passage relevance, for example from 0 to 3. OpenSearch Search Relevance Workbench uses the same foundational entities: a query set, search configurations, and a judgment list. An automated judge can help expand annotation, but owners of the business process should verify critical examples.

At least four configurations should be compared: BM25, vector-only, hybrid without a reranker, and hybrid with a reranker. Useful metrics include:

  • **Recall@K:** whether at least one required passage appears in the first K results;
  • **nDCG@K:** whether more relevant documents rank higher;
  • the share of answers fully supported by supplied sources;
  • correct abstention when evidence is absent;
  • p50 and p95 retrieval and end-to-end latency;
  • cost or hardware load per accepted answer.

An average metric must not conceal failures. Exact codes, expired documents, and unanswerable questions should be analyzed separately.

Data, filters, and observability

Both branches must search the same corpus version under the same access rules. Otherwise RRF may merge an allowed vector result with a forbidden lexical result. Every passage needs a stable ID, document version, source, ACL, effective date, and deletion state.

Useful telemetry includes the configuration version, branch latency, candidate ranks, reranker output, and passages sent to the model. Sensitive systems can keep identifiers and anonymized error classes instead of query and document text.

If one branch is unavailable, the system should explicitly enter a degraded mode. For critical guidance, the interface can show that retrieval was incomplete.

Modelled economics

Consider a notional corpus of 50,000 passages and one thousand queries per day. Lexical and vector branches return 30 candidates each; after fusion, the reranker scores 40 unique pairs. That produces 40 thousand pair evaluations per day. Rank fusion itself is nearly free; embedding generation, index memory, and reranking drive the cost.

An illustrative calculation: 80 hours at RUB 3,000 equals RUB 240,000. Annotating 150 questions at three minutes each and RUB 1,200 per hour adds RUB 9,000. Four maintenance hours per month at RUB 3,000 equal RUB 144,000 per year. These are model assumptions, not a quotation.

If BM25 already reaches the required Recall@20 and employees rarely paraphrase, hybrid retrieval may not pay back. If search failures cause repeat contacts, manual review of dozens of files, or incorrect actions, the relevant benefit is the cost of avoided errors, not only server milliseconds.

A ten-working-day pilot

1. Collect 100–200 real questions and annotate a control subset.
2. Freeze the current BM25 or vector configuration as the baseline.
3. Add the second branch using the same ACLs and corpus version.
4. Start with RRF and no manually tuned weights; preserve an explanation of each branch's contribution.
5. Add a reranker only for the top 20–40 candidates and cap passage length.
6. Compare four configurations on relevance, abstention, and p95 latency.
7. Run the winner in shadow mode, then expose it to a small user cohort with a fast rollback path.

Management need not choose between "old search" and "AI." The questions are which query classes fail, which second signal recovers them, and what each additional source-supported answer costs.