Four-Bit Weights Do Not Mean One-Quarter the Cost: Local LLM Economics

The short answer

Quantization lowers the precision used to represent model weights and usually reduces their memory footprint. It may move a model to less expensive hardware, create room for more concurrent requests, or eliminate a second GPU. But moving from 16 bits to 4 bits does not automatically cut the monthly bill by a factor of four.

Weights are only one part of the system. Total cost also includes the KV cache, activations, runtime buffers, storing several releases, preparing the quantized artifact, testing, electricity, operations, and human review. The selected format also needs efficient kernels on the actual CPU or GPU. Otherwise, the file gets smaller while the request may not get faster.

For a business, the correct comparison unit is neither model gigabytes nor tokens per second. It is the cost of an accepted result at the required service level.

What quantization actually changes

Source model weights are often stored in FP16 or BF16. Weight-only quantization represents them with fewer bits while computation may still use higher precision. The llama.cpp documentation describes conversion from F32 or BF16 to low-bit formats: it reduces model size and may speed up inference, but it can introduce quality loss. To reduce that loss, the project supports an importance matrix built from calibration data.

The Transformers bitsandbytes documentation states that loading a model in 8-bit roughly halves weight memory compared with 16-bit. It also exposes an important qualification: not every module necessarily uses the same lower precision. Normalization layers and other parts of the graph may remain in their original dtype.

Research methods such as GPTQ and AWQ make the trade-off more carefully than naive rounding. GPTQ uses approximate second-order information for layer-wise post-training quantization. AWQ identifies salient weights from activations and protects a small fraction of them. These methods are useful, but paper results do not automatically transfer to a different language, chat template, runtime, or business workflow.

Why file size is not total memory

A rough lower bound for weight storage is:

`parameters × bit width / 8`.

For an 8-billion-parameter model, that is about 16 GB at 16 bits, 8 GB at 8 bits, and 4 GB at 4 bits, before metadata, scales, unquantized tensors, and alignment. This estimates weights, not a production service.

At runtime, memory is also used by:

• the KV cache for active sequences;
• temporary activations and workspace buffers;
• CUDA graphs, kernels, and runtime structures;
• contexts for concurrent users;
• a vision or audio encoder in a multimodal model;
• LoRA adapters and other attached components.

The vLLM tuning guide links insufficient KV cache space to request preemption and recomputation, which harms end-to-end latency. Smaller weights may leave more room for cache and improve concurrency, but the result depends on context length and simultaneous demand. With one short request at a time, free memory alone may save no money.

Hardware compatibility matters more than the format label

The same “4-bit” label can refer to GGUF/K-quants, GPTQ, AWQ, bitsandbytes, INT4 W4A16, and other schemes. They differ in weight packing, scales, calibration, and execution kernels.

The vLLM compatibility table shows that format support varies across NVIDIA generations, AMD, Intel, and CPUs. Choosing a quant before choosing the serving environment is therefore risky. A format may load yet use a less efficient path, require conversion, or fail to support the selected model architecture.

A minimum technical record should include:

• the exact model, weight revision, tokenizer, and chat template;
• quantization format and settings, artifact source, and checksum;
• inference server and version;
• CPU or GPU, memory capacity, and driver;
• input and output lengths, batch size, and concurrency;
• TTFT, inter-token latency, throughput, p95, and errors;
• actual occupied memory after warm-up.

Results from different servers, templates, or request sets are not comparable because several variables change at once.

Measure quality at the business outcome

Perplexity and broad benchmarks are useful for screening, but the business does not pay for them. Quantization may barely affect ordinary prose while increasing errors in critical fields such as a SKU, date, amount, negation, JSON structure, tool call, or evidence citation.

Every candidate needs the same held-out set of real examples. It should include rare, long, and awkward cases, not only clean requests. At minimum, measure:

• results accepted without correction;
• critical-error rate;
• schema and business-rule compliance;
• citation and source-grounding accuracy;
• number of regeneration attempts;
• human review minutes per result.

If the model is part of a RAG or agent workflow, test the whole path. A weaker quant can sometimes be offset by better retrieval or stricter validation, but those controls also add cost and latency.

Worked example: why 8-bit can be cheaper than 4-bit

Consider an illustrative workload of 30,000 documents per month. Correcting a rejected result takes 2.5 minutes on average, and the fully loaded labor rate is RUB 600 per hour. To isolate the difference, integration and routine review costs that are identical across candidates are excluded.

Pilot assumptions:

• BF16: RUB 120,000 monthly infrastructure, 96% accepted without correction;
• 8-bit: RUB 100,000 monthly infrastructure, 95% accepted;
• 4-bit: RUB 80,000 monthly infrastructure, 90% accepted.

Incremental labor is `volume × correction rate × minutes / 60 × hourly rate`.

• BF16: RUB 30,000 labor, RUB 150,000 total;
• 8-bit: RUB 37,500 labor, RUB 137,500 total;
• 4-bit: RUB 75,000 labor, RUB 155,000 total.

This is not a market quote or a forecast for a particular model. It is a sensitivity example. The smallest quant saves RUB 40,000 in infrastructure relative to BF16 but adds RUB 45,000 in correction effort. The 8-bit option wins in this scenario.

A real calculation should also include purchase amortization or rental, electricity, idle time, capacity reserve, artifact storage, engineering work, retries, and incident cost. Economics can improve sharply if the compact model lets the company move from two GPUs to one. If the hardware is already owned and capacity is not constrained, the gain may be limited to concurrency.

A low-cost pilot

There is no need to quantize the entire model estate. Start with one process and three candidates: the original precision, a moderate quant, and an aggressive quant.

1. Freeze 200–500 anonymized work examples and an acceptance rubric.
2. Build candidates from one source revision, or use artifacts with clear provenance.
3. Run them on the same target server with identical generation settings.
4. Measure warm-state memory, TTFT, p95, throughput, and power under a representative load.
5. Blind-review the outputs and count correction minutes.
6. Calculate cost per accepted result and verify SLA headroom.
7. Keep the winner in shadow mode for a week, then pin the release and rollback plan.

Quantization is not a final optimization checkbox. It is a deployment variant that needs its own acceptance test. The rational objective is not the smallest file; it is the required quality and workload on the least expensive resilient setup.

The management decision

Ask the team for a three-row comparison of BF16, 8-bit, and 4-bit: infrastructure, acceptance without correction, and total cost per accepted result. If the table omits human time and p95 on your hardware, the decision has not been costed yet.