Why RAG quality is decided before the vector database

When a corporate assistant confidently mixes up table rows or cites a clause that does not exist, teams often investigate the model, embeddings, or prompt. Yet the answer may have been damaged much earlier: the PDF was parsed in the wrong reading order, a scan went through without the right OCR language, a page header was copied into every chunk, or merged table cells became an unrelated sequence of numbers.

Document ingestion is therefore not a helper script. It is part of the production RAG architecture. Its job is not merely to return text. It must identify the document type, choose a parsing route, preserve a link to the original, measure output quality, and keep questionable results out of the live index.

For a small or midsize business, the practical conclusion is simple: first build a controlled pipeline for 50–100 representative documents, then compare answer-generation models. Otherwise the team will optimize generation on top of corrupted evidence.

What tools and research confirm

Docling defines a unified document representation that can retain text, tables, pictures, hierarchy, reading order, element coordinates, and provenance—a pointer to the page and source region. That is more useful than flat Markdown when the user must see where an answer fragment came from.

Docling documentation also exposes separate controls for OCR, table-structure recognition, TableFormer accuracy mode, page and file-size limits, and model prefetching for offline operation. Even a single toolkit therefore assumes several processing routes rather than one universal “parse PDF” button.

Unstructured describes a similar fork with different terminology. PDF strategies include `fast`, `hi_res`, `ocr_only`, and automatic selection: digital text can be extracted quickly, scans need OCR, while tables and complex layouts require a heavier layout-aware mode. Its documentation also records limitations, including cases where reading order in multi-column documents depends on the selected strategy.

The independent OmniDocBench benchmark evaluates parsers across text, tables, formulas, layout, and reading order instead of reducing everything to one number. It contains multiple document types, languages, and layout categories. The business lesson is not to copy a leaderboard winner but to test on the company’s own mix of contracts, manuals, invoices, and scans—and to separate error types.

A reference ingestion architecture

A minimal reliable flow has eight steps:

1. Receive the file into immutable source storage.
2. Check type, size, page count, password status, malicious content, and usage rights.
3. Classify pages as digital text, scan, mixed document, complex table, or unusual layout.
4. Route them to a fast extractor, OCR, or a layout-aware parser.
5. Normalize structure without losing page, headings, tables, and coordinates.
6. Run automated quality checks and selective visual review.
7. Quarantine failures or create chunks with stable metadata.
8. Index only an accepted version and record the processing event.

In compact form:

`source → classifier → parser/OCR → normalized document → quality gate → quarantine or chunker → index`

Keep the original file and normalized output separately. A parser upgrade can then reprocess and compare versions without asking employees to upload the same documents again.

Step 1. Classify the document, not its extension

The `.pdf` extension says nothing about internal quality. One file may contain a digital contract, a scanned appendix, and a table embedded as an image. Routing decisions should therefore be made at page level.

Useful signals include:

  • whether extractable text exists and how much appears on each page;
  • whether the source page count matches the parser result;
  • whether an image covers almost the entire page;
  • whether the page contains columns, tables, forms, stamps, or handwriting;
  • which languages are expected;
  • whether the file is password-protected or restricts extraction;
  • whether it exceeds operational limits.

A normal digital PDF can use fast extraction. A scan needs OCR with the correct language packs. A price list with merged cells or a multi-column manual needs a mode that preserves layout and reading order. A mixed document is best routed page by page.

Step 2. Preserve provenance for every chunk

Do not send only `text` and `embedding` to the index. A practical minimum payload includes:

  • `document_id` and document version;
  • source-file checksum;
  • page number and section path;
  • element type: paragraph, heading, table, or caption;
  • source-region coordinates when available;
  • parser mode and version;
  • language;
  • validation status and confidence tier;
  • access controls and validity period;
  • identifiers of neighboring chunks.

Qdrant can store this JSON metadata as payload and filter on it; the same principle applies to other vector stores. Access rights, current version, and `accepted` status must constrain retrieval before context reaches the model, not after an answer is generated.

Provenance is not only for auditors. Page numbers and coordinates let the interface open the exact region of the PDF. The user can verify the answer quickly, while the developer can determine whether retrieval, generation, or parsing caused the error.

Step 3. Build automated quality gates

There is no universal PDF-quality metric, so the pipeline combines several inexpensive signals. A document should enter quarantine when at least one critical condition is met:

  • the parser returns fewer pages than the source contains;
  • a meaningful page becomes empty;
  • unreadable symbols or repeated blocks exceed a threshold;
  • expected headings, clause numbers, or tables disappear;
  • reading order is visibly broken;
  • a table loses headers or its column count changes sharply;
  • detected language conflicts with the expected one;
  • the result lacks a source-page reference;
  • processing exceeds time or memory limits;
  • a controlled sample fails comparison with ground truth.

Thresholds cannot be copied from another organization. A blank page may be normal in a policy manual, while one missing line may be critical in an invoice. Set limits by document class and by the cost of an error.

A structural fingerprint is particularly useful: page, heading, table, paragraph, and character counts. It does not prove correctness, but it quickly exposes abrupt changes after a parser or OCR-model upgrade.

Step 4. Quarantine instead of silently succeeding

The unsafe pattern is to index everything that did not raise an exception. A parser can successfully return incorrectly read text while technical monitoring stays green.

A quarantine queue should show the reason, a preview of the problematic page, and a concrete action:

  • retry with another OCR language;
  • switch parsing strategy;
  • crop or rotate the page;
  • request a better original;
  • approve the result manually;
  • exclude the file from the knowledge base.

Humans do not need to read the whole archive. They review only failed documents plus a small random sample of accepted ones. That sample detects silent errors and provides evidence for improving routing rules.

Testing the pipeline on company documents

Build a golden set of 50–100 files. Choose the real mixture, not the prettiest examples:

  • digital and scanned PDFs;
  • Russian and mixed-language documents;
  • contracts with numbering and appendices;
  • manuals with columns and image captions;
  • tables with merged cells;
  • poor copies, rotated pages, and stamps;
  • documents that should be rejected.

For each file, record the expected page count, key headings, two or three control phrases, one critical table, and the correct source for several questions. Evaluate four layers rather than only the final RAG answer:

1. text completeness and reading order;
2. preservation of structure and tables;
3. correctness of chunks and metadata;
4. ability to cite the right page.

Every parser, OCR, language-pack, or chunking-rule upgrade should rerun this set. Without regression testing, an improvement on scans can quietly damage digital tables.

Infrastructure and local deployment

The ingestion pipeline can run inside the company network. Fast digital-text extraction usually works on CPU. OCR and layout analysis require more processing time; some models can use a GPU, but hardware should not be purchased before measuring the actual workload.

Separate the responsibilities logically:

  • intake and malware scanning;
  • task queue;
  • CPU workers for fast extraction;
  • resource-limited OCR and layout workers;
  • source and normalized-result storage;
  • metadata database and processing ledger;
  • vector index;
  • quarantine interface.

For an isolated installation, prefetch models and language data, pin versions, and check licenses. Prevent the parser from making arbitrary outbound requests. Documents often contain personal data, commercial terms, and payment details; local OCR is valuable precisely because the file stays inside the controlled environment.

Economics: count pages, not “one RAG system”

The following is a model calculation. Assume the company receives 1,000 documents per month with an average of 12 pages. Of 12,000 pages, 70% contain digital text, 20% need OCR, and 10% need expensive layout and table processing. Replace these shares with measurements from the pilot.

Monthly ingestion cost is approximately:

`pages × processing time for the selected route × compute cost + storage + human review + reprocessing`

Routing is the main economic lever. Sending every page through the heaviest mode increases latency and infrastructure. Processing every page through the fastest route transfers the cost into employee corrections and wrong answers. Track quarantine share and human-review minutes per document separately.

Four metrics are enough for a pilot decision:

  • cost per accepted page;
  • share of documents entering quarantine;
  • critical-error rate on the golden set;
  • time from upload to an accepted searchable version.

More powerful hardware is justified only after the team knows actual pages per hour, peak queue length, and the share of difficult documents.

Limitations and risks

Neither Docling nor Unstructured nor any OCR model guarantees correctness for every PDF. Handwritten corrections, complex diagrams, nested tables, poor faxes, and unusual fonts remain difficult. A benchmark measures quality on its own distribution; it does not replace a company-specific golden set.

Content trust is another risk. A technically well-parsed file can be outdated, unapproved, or uploaded without authorization. The quality gate must therefore validate ownership, version, effective date, and ACL in addition to extraction quality.

Finally, keep text correction separate from source evidence. If a model automatically “repairs” a suspicious table value, preserve both the original and the transformation for comparison.

A low-cost next step

Select one document class and one process—for example, customer-support manuals. In two weeks, a team can:

  • collect 50–100 real files and ground-truth checks;
  • configure two or three parsing routes;
  • retain page and coordinates for every chunk;
  • define five quarantine reasons;
  • measure speed, page cost, and critical errors;
  • connect only accepted documents to a test index.

If answers remain weak, the team will know which layer to investigate. Good RAG starts not with a polished chat box but with a document earning the right to enter the index.