Why embeddings deserve their own cost line

In a local RAG system, most attention goes to the generative model. Before generation can happen, however, documents must be chunked, converted into vectors, loaded into an index and refreshed after every change. Each user query also needs an embedding, and some systems run an additional reranker.

These operations use encoder models. They are smaller than LLMs and can often run efficiently on CPUs, especially after ONNX or OpenVINO export and quantisation. A GPU may process a large batch faster, but its advantage depends on model size, text length, batch size, drivers and actual utilisation. The business question is therefore not merely “which is faster?” It is which setup completes the required workload within its deadline at the lowest cost per accepted result.

The official Sentence Transformers efficiency documentation shows that backend rankings change between a local workstation and a cloud CPU instance, and explicitly recommends benchmarking on deployment hardware with representative inputs. This matters more than any universal table: short product cards and long policy documents create different compute profiles.

Separate three different workloads first

The label “embeddings” often hides three queues with different economics.

Initial corpus ingestion

This is a large but infrequent batch. Total completion time matters: can the archive be processed overnight or over a weekend? Per-document latency matters little, while large batches and high utilisation are valuable.

Incremental updates

New contracts, product cards and instructions arrive every hour. This queue needs a freshness objective—for example, a changed document must become searchable within 15 minutes. The flow is usually uneven, so durable queues, retries and duplicate protection matter.

User-query embeddings

These are small batches with a strict latency budget. Even if a GPU is economical for overnight ingestion, employee queries may be served comfortably by a CPU. The reverse can also be true when online demand is high and an accelerator is already shared with other models.

Choosing one setup for all three queues is like using the same truck as a courier, warehouse and personal car. It may move, but the financial model becomes difficult to defend.

What the tools actually support

Sentence Transformers supports PyTorch, ONNX and OpenVINO. Its documentation covers FP16 on GPUs, optimised ONNX models and INT8 CPU configurations. The published comparisons consider not only throughput but also task-quality ratios against the matching FP32 model on semantic-similarity and retrieval evaluations.

Hugging Face Text Embeddings Inference provides official images for CPUs and several NVIDIA GPU generations, accepts batched inputs and supports air-gapped deployment with model weights mounted locally. Qdrant's FastEmbed also uses ONNX Runtime, supports parallel CPU processing and offers a separate GPU package.

This does not prove that one backend is always cheaper. It means a company can test the same model and the same document set behind a consistent API without rebuilding the entire RAG application.

A minimally honest benchmark

Four candidates are enough for an initial decision:

  • the current PyTorch CPU path as a baseline;
  • ONNX or OpenVINO on CPU;
  • the same model on GPU;
  • a quantised CPU variant if it meets the quality threshold.

Test every candidate on the same corpus. A useful benchmark should reflect the production distribution rather than contain one hundred short sentences:

  • 20–30% short cards;
  • a majority of normal chunks;
  • 10–20% long chunks near the permitted limit;
  • Russian text, tables and mixed alphabets if present in the real data;
  • duplicate and near-duplicate documents.

Warm up the runtime before measurement. Then record batch size, thread count, model version, tokenizer, precision, library, driver and exact hardware. Measure tokens per second in addition to documents per second: one contract and one product card are not equal units of work.

For the online queue, collect p50 and p95 latency separately. For batch work, measure total duration, peak memory and the share of time the device is genuinely busy. If a GPU waits for input 90% of the time, impressive peak speed does not become impressive economic efficiency.

Calculate cost per accepted vector

A basic monthly equation is:

**Pipeline cost = rental or depreciation + electricity + operations + artifact storage + reprocessing cost.**

Divide that cost not by API calls or tokens, but by vectors that pass quality checks and enter the production index.

This captures hidden losses:

  • tokenisation errors and empty inputs;
  • retries after failures;
  • full re-indexing after a model change;
  • retrieval degradation after quantisation;
  • manual review of problematic collections;
  • accelerator idle time between batches.

This follows unit-economics logic: a technical metric becomes useful when tied to a business unit. For RAG, that unit may be an updated document, an accepted vector or an approved answer, but the definition must remain stable.

A model calculation that does not pretend to be a price list

Consider a hypothetical company processing 120,000 new and changed chunks per month, averaging 750 tokens each. The monthly workload is 90 million tokens. Its own benchmark reports:

  • optimised CPU: 12,000 tokens per second;
  • GPU: 100,000 tokens per second;
  • internal CPU resource rate: 60 roubles per hour;
  • internal GPU resource rate: 500 roubles per hour.

These are illustrative assumptions, not market prices or performance promises.

Pure compute time before overhead is:

  • CPU: 90,000,000 / 12,000 = 7,500 seconds, or about 2.08 hours;
  • GPU: 90,000,000 / 100,000 = 900 seconds, or 0.25 hours.

The variable cost of one monthly batch is almost identical:

  • CPU: approximately 125 roubles;
  • GPU: approximately 125 roubles.

Reality changes the conclusion. If the GPU is billed for at least one hour, the batch costs 500 roubles. If the accelerator is purchased and sits idle most of the month, the task must absorb a share of depreciation, power and administration. If the same GPU already serves a local LLM and embeddings run in a free scheduling window, its marginal cost may be lower.

The break-even point is therefore not determined by throughput alone. For variable rental, a first approximation is:

**Volume × CPU hourly cost / CPU throughput = volume × GPU hourly cost / GPU throughput.**

When both sides are equal, completion deadline, minimum billing interval and resource availability decide the outcome. For owned hardware, replace hourly price with fully loaded monthly cost and the share of time used for productive work.

Quality can consume all the savings

Moving from FP32 to INT8 may accelerate CPU inference, but throughput alone is not a sufficient decision metric. Repeat retrieval evaluation on a labelled set: recall@k, nDCG@k, the share of correct documents in top positions and final RAG-answer quality.

Even a small retrieval decline may increase manual corrections or missed evidence. If the faster setup creates more rejected results, its cost per useful unit rises.

Changing the embedding model itself requires particular care. Vectors from different models are generally incompatible, so migration needs a new index, a parallel run and a controlled cutover. Include that migration cost in advance and keep the old index until the new one is verified.

A practical architecture for an SME

The winner is often not “CPU versus GPU” but a hybrid queue:

  • user queries and small updates run continuously on CPU;
  • large imports accumulate in a queue and use the GPU in an available window;
  • a consistent API hides backend selection from the RAG application;
  • model and version identifiers are stored with every vector;
  • retries are idempotent and do not create duplicates;
  • metrics cover tokens, accepted vectors, p95 latency, failures and utilisation.

If no dedicated GPU exists, start with an optimised CPU path. It uses familiar infrastructure and establishes a cost baseline. Add an accelerator only after measuring queue growth, missed service levels or a demonstrable reduction in cost per accepted unit.

A one-week pilot

1. Export 10,000–50,000 representative chunks and preserve their length distribution.
2. Fix one model and one tokenizer.
3. Compare baseline CPU, optimised CPU and an available GPU.
4. Measure throughput, memory, p95 latency and retrieval quality on the same set.
5. Calculate fully loaded cost, including idle time and re-indexing.
6. Choose a backend separately for online queries, incremental updates and large imports.

The valuable output is not a hardware winner but a transparent operating boundary. CPU may be more economical at moderate volume when infrastructure is already paid for. GPU may be justified for large batches, tight deadlines or shared use with other models. The queue and the cost per accepted vector should make the decision—not the badge on the server.