Why a vector database suddenly demands so much memory

A local RAG demo looks modest: a few thousand documents, one server and fast search. Then the database receives contract history, manuals, email, multiple versions of every file and access-control metadata. The number of chunks grows into the millions, and expensive RAM unexpectedly appears in the budget.

The reason is simple: a vector is an array of numbers. A 768-dimensional embedding stored as float32 needs 768 × 4 = 3,072 bytes, or roughly 3 KB. One million such vectors consume about 3.07 GB in raw form alone. At 1,536 dimensions, this doubles to 6.14 GB.

That is not the full index. The vectors are accompanied by an HNSW graph or IVF structures, identifiers, payloads, internal metadata, temporary memory for index construction, caches, replicas and growth headroom. Multiplying vector count by dimension is therefore a useful first step and a dangerous final budget.

Components of real memory consumption

For local RAG, memory should be calculated in layers:

  • original vectors or their compressed representation;
  • the approximate-search structure;
  • metadata and access filters;
  • a full-text or sparse index for hybrid search;
  • a hot-data cache;
  • headroom for index construction and updates;
  • a high-availability replica;
  • 20–30% capacity for growth and peaks.

HNSW builds a graph of links between nearby vectors and usually provides strong latency and recall, but the graph also consumes memory. Its size depends on graph connectivity, implementation and object count. IVF partitions the vector space into clusters and can reduce the active search set, but requires tuning the number of clusters and probes.

Milvus documentation describes FLAT, IVF, HNSW and compressed variants. Qdrant and Weaviate separately document quantisation and the option to move parts of the data to disk. The common conclusion is that an index is not selected from a simple “fast versus slow” table. Volume, latency, filters, update behaviour and acceptable quality loss all matter.

What quantisation provides

Quantisation stores vectors at lower precision. Instead of a 32-bit number for every coordinate, the system can use eight bits, a binary representation or a segment code.

Eight-bit scalar quantisation theoretically reduces a float32 representation by about four times. For one million 768-dimensional vectors, the raw vector footprint drops from approximately 3.07 GB to 0.77 GB. Qdrant notes that scalar quantisation may also speed up search because compact data can be processed with efficient CPU instructions.

Product quantisation splits a vector into segments and stores an identifier for the nearest centroid in each segment. In Weaviate's example, a vector with 768 float32 coordinates occupies 3,072 bytes, while PQ with 128 segments requires about 128 bytes plus a small codebook overhead. That is almost a 24-fold reduction in the vector representation itself.

The word “almost” matters more than the number 24. A complete system still includes the graph and metadata, and often retains original vectors on disk for rescoring. Lossy quantisation can also change the order of close candidates.

Why saving gigabytes is not the only objective

RAG does not care about abstract distance accuracy. It cares whether the required chunk reaches the set passed to the language model. If compression saves a server but lowers recall@k for rare contracts or Russian-language phrasing, the generator cannot reconstruct evidence that was never retrieved.

At minimum, compare four modes on the organisation's own corpus:

  • a full float32 index;
  • eight-bit scalar quantisation;
  • more aggressive product or binary quantisation;
  • compressed first-stage retrieval with over-fetching and rescoring against original vectors.

For every mode, measure:

  • recall@k, or the share of queries where a reference chunk appears in the top-k;
  • final answer quality, not retrieval alone;
  • p50 and p95 latency;
  • actual process memory under workload;
  • index build and update time;
  • disk reads;
  • server and backup cost.

Weaviate documents rescoring: candidates are retrieved using compressed vectors, after which distances are recalculated against the originals. This can restore some retrieval quality in exchange for disk capacity, additional computation and slightly higher latency. Nothing magical occurs; the cost simply moves to another line.

A model calculation for five million chunks

Consider a local RAG system with five million 768-dimensional vectors. Assumptions:

  • original format is float32;
  • one vector represents one chunk;
  • average payload with identifiers, ACL data and internal fields is 500 bytes;
  • one serving replica and one backup are required;
  • memory headroom is 25%;
  • specific server pricing is excluded from the illustration.

Raw vectors occupy about 15.36 GB. Payloads add roughly 2.5 GB. Before accounting for the graph, cache and internal structures, the working set is already close to 17.9 GB. With headroom, it reaches 22.4 GB, and HNSW may raise the requirement further.

With eight-bit scalar quantisation, the vector component falls to about 3.84 GB. Payloads do not shrink, so the base set is approximately 6.34 GB, or 7.9 GB with headroom before the graph. The complete process saves less than four times because non-compressible structures remain.

PQ can reduce the vector component further, but the result must be measured on the real index. If original vectors are retained on SSD for rescoring, RAM decreases while disk capacity remains. Replication doubles serving storage, and backups plus blue-green index versions temporarily increase it again.

Calculating economics rather than infrastructure alone

A useful unit is cost per retrieval that leads to an accepted answer. The numerator includes:

  • amortised RAM and CPU nodes;
  • NVMe or network storage;
  • replicas and backups;
  • index builds and reindexing;
  • monitoring, administration and upgrades;
  • the cost of retrieval errors and human review.

The denominator is the number of queries for which the system found sufficient authorised sources and produced an answer that passed validation. If a cheaper index increases manual investigations, its infrastructure saving may be false economy.

Utilisation matters particularly for SMEs. A server with 256 GB of RAM may look like prudent headroom, but with a small corpus and ten requests per minute it becomes an expensive piece of furniture. Keeping originals on NVMe, a compressed index in memory and accepting another 10–20 milliseconds may be more economical.

When each approach fits

Full, uncompressed HNSW is suitable when the corpus fits available RAM, latency is critical and retrieval quality is expensive to lose. It also provides a useful baseline.

Scalar quantisation is a practical first option when memory becomes constrained but predictable quality is still required. It is easier to test and explain than more aggressive techniques.

PQ and binary quantisation are useful for large collections and strict RAM limits, particularly with over-fetching and rescoring. Rare query classes need closer validation.

Disk-based or memory-mapped indexes fit cases where scale matters more than minimum latency and fast local NVMe is available. For an infrequently used internal knowledge base, this is often better than purchasing additional RAM.

FLAT search is appropriate for small collections, filtered subsets and a control baseline. On a small corpus, a complex index may cost more than exhaustive search.

A pilot on one copy of the corpus

A practical comparison takes several days:

1. Freeze the corpus, embedding model and 100–300 real queries with reference chunks.
2. Build a baseline float32 index and measure quality, memory, p95 and construction time.
3. Repeat with eight-bit quantisation.
4. If savings are insufficient, add PQ or binary mode with over-fetching and rescoring.
5. Test access filters: acceleration must never return unauthorised documents.
6. Measure bulk updates and safe switching between index versions separately.
7. Calculate cost per accepted answer and choose the smallest configuration that passes quality and SLA thresholds.

The management conclusion is straightforward: do not buy RAM before measuring. First calculate vector volume and non-compressible overhead, then verify on the organisation's corpus how much quality can be exchanged for a smaller index. A vector database does not have an appetite of its own; it only exposes an architecture that somebody failed to size in advance.