Index updates have a cost too

A RAG knowledge base does not end after its initial import. New versions of manuals, price lists, and policies must be parsed, cleaned, chunked, embedded, and written to the index. Scans may require OCR, and quality control may require spot checks. If every run reprocesses the whole archive, even small changes can create unnecessary work.

There are two update patterns. A full reindex rebuilds the corpus and is easier to reason about, but repeats work on unchanged files. An incremental process compares versions or metadata and processes only added, changed, and deleted documents. It reduces repeated work but requires stable document identifiers, version synchronization, and reliable removal of old chunks.

An illustrative model

Assume a corpus of 1,000 files, each eight pages long. If 10% change in a month, a full run touches 8,000 pages, while a selective run touches about 800. That is one tenth as many pages to recognize and parse, but it does not automatically reduce the entire budget by 90%.

Servers, storage, monitoring, quality checks, and support remain. Real costs also depend on scan share, repeated OCR, document length, embedding model, freshness requirements, and how deletions and renames are detected.

For a useful estimate, measure the full processing cost per new or changed page separately from recurring monthly costs. Then compare full and selective processing on a copy of one corpus.

Building reliable updates

  • Keep a stable document ID, version date, and content hash; a file path can change.
  • When a new version arrives, remove or replace old points so retrieval cannot return stale passages.
  • Store the source, page, version, and access rights for every chunk.
  • Run test queries before and after updates and check that results cite the current edition.
  • For deletion, check the primary index, caches, backups, and search views against agreed retention rules.

Incremental ingestion is not always safer than a full rebuild. A broken change detector can leave stale answers; a missed sync event can omit a new document. Add periodic file-list reconciliation, a job log, and alerts for failed updates.

Choosing an update pattern

A full reindex is convenient for a small corpus, infrequent changes, or an embedding-model migration. Incremental updates fit many sources that change regularly when reprocessing is measurably expensive. A change in vector dimension or embedding model may require a separate migration rather than merely updating changed files.

Start with one folder and a fixed set of test questions. Measure time, cost, error share, and whether old versions are fully removed under both patterns. If savings are meaningful and quality holds, extend the process to other sources. Do not optimize page counts at the expense of retrieval freshness.