The problem may be a second copy of the weights, not the model

Researchers have described an “ingestion tax” in open-model serving. The operating system may already hold checkpoint pages in a file cache that an accelerator can read on unified or coherent-memory systems, while the tensor framework creates its own representation of the same bytes. The mapping and an additional copy coexist. In the middle-capacity regime, one representation fits but two compete for memory, making local inference slower or forcing useful pages out.

The paper, The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks, was published on August 12, 2026. Its authors propose adopting mapped pages as ordinary tensors through DLPack rather than copying weights into framework-owned storage. In their tests, a 65 GB checkpoint reached its first token 6.4 times sooner, Qwen2.5-72B retained the speed of a load-once resident copy, and llama.cpp on an AMD APU used roughly half the memory.

This is not a universal accelerator for every LLM. On a discrete GPU connected through PCIe, the same approach was 39 times slower because every weight access crossed the bus instead of using local VRAM. The practical lesson for businesses is to select a loading strategy using memory topology and full-working-set measurements, not an API name or a single benchmark.

What the ingestion tax is

A conventional model load roughly follows four steps:

1. The OS reads checkpoint files and places their pages in the file cache.
2. PyTorch, MLX, or another framework allocates its own buffer.
3. Weights are copied or transformed into that buffer.
4. Original file pages remain cached while memory is available.

On a server with a discrete GPU, a one-time copy into fast VRAM is usually justified because later tokens repeatedly use local data. On unified-memory computers or systems with a coherent CPU–GPU link, mapped pages may already be in a domain the accelerator can read. A second physical representation may not improve placement, but it still consumes capacity and transfer bandwidth.

The “one copy fits, two do not” regime is especially important. A model can appear to fit installed RAM, then trigger compression, swap, or eviction when the framework allocation is created. Comparing model-file size with installed memory is therefore not a sufficient capacity check.

How file-backed adoption works

The proposed mechanism maps each tensor with `MAP_SHARED`, creates a no-copy GPU buffer, and exports it through DLPack. PyTorch or MLX imports the capsule as ordinary storage. File pages remain clean, shareable across processes, and reclaimable under memory pressure without write-back.

The authors identify three necessary conditions:

  • the kernel reads mapped pages without creating a per-use framework copy;
  • activations remain accelerator-visible and do not round-trip through CPU arrays;
  • dependencies are ordered on the GPU without forcing the host to drain outstanding work.

Zero-copy by itself is not enough. The authors' first implementation removed the copy but ran 2.3 times slower than stock because of synchronization. The improvement appeared only when the full execution contract was satisfied. A `from_dlpack` call or memory mapping alone does not prove faster inference.

Results reported by the authors

For Qwen2.5-72B int8 on a unified-memory machine, the file-backed path reached 7.14 tokens per second. A load-once resident copy achieved 7.23 tokens per second, while per-use ingestion managed 0.94 tokens per second. Mapped weights therefore matched the strongest resident alternative while remaining shareable and reclaimable.

For Qwen2.5-32B, a 65 GB checkpoint reached its first token in 6.97 seconds rather than 37.95 seconds with the stock loader. The paper notes that much of the gap came from removing shard parsing and relayout, not solely from changing storage ownership. The entire 6.4-times improvement should not be attributed to memory mapping.

In an experimental Kimi K3 setup, a dense weight-processing stage fell from 2.62 to 0.35 seconds per token, while full-token latency declined from 4.14 to 1.87 seconds. Streaming routed experts from storage remained a separate bottleneck.

On an AMD Ryzen 7 9700X APU, a Vulkan integration improved llama.cpp from 2.82 to 3.42 tokens per second and reduced peak working set from 8.13 to 4.20 GiB. On NVIDIA GH200, mapped pages and overlapped streaming were within 5% for a workload larger than HBM.

The opposite result appeared on an RTX 5070 Ti: reading mapped weights across PCIe reduced speed from 146.9 to 3.79 tokens per second. The smaller footprint could not compensate. Memory topology decides the byte path.

Where the method can apply

Unified CPU and GPU memory

Apple Silicon and some APUs share a memory pool. When a framework creates a second weight copy in the same DRAM, file-backed storage may free substantial capacity and improve startup. Driver support, correct external-memory import, and full-working-set validation are required.

Coherent CPU–GPU links

Systems such as GH200 let accelerators read CPU memory through a high-speed coherent link. Mapping becomes a placement option when weights or an active set exceed HBM. The decision depends on measured link throughput and the runtime's ability to overlap traffic with computation.

Discrete GPUs over PCIe

When the model fits in VRAM, copying once and reading locally is generally preferable. File-backed access across PCIe turns capacity savings into a performance collapse. Test a resident load, quantization, and multi-GPU partitioning first.

CPU-only inference

Local runtimes have long used memory mapping, but outcomes still depend on file format, OS cache, NUMA, and access pattern. The paper's main contribution is bringing mapped pages into tensor frameworks without an unnecessary copy.

What to measure on an existing server

Before buying more memory or another accelerator, run a short audit:

  • model-file size and actual process resident set;
  • page-cache occupancy before and after model loading;
  • anonymous memory, swap, and compression;
  • the number of physical weight representations;
  • cold-start time and warm time to first token;
  • tokens per second with one and several requests;
  • CPU–GPU bandwidth and whether PCIe is in the path;
  • behavior with two processes when services or model versions must coexist;
  • time spent reading weights, converting formats, and computing.

The test must touch the complete working set. A small buffer can fit in CPU cache and report a rate unavailable to the deployed model. The paper explicitly uses full-working-set comparisons and warns against misleading microbenchmarks.

Economics for a small business

Assume a local checkpoint occupies 65 GB. A stock loader creates another representation of a similar order, while the OS and activations need headroom. A 128 GB server reaches the boundary, and a second process or long context can trigger swap. This is a modelled example, not a universal formula.

If file-backed adoption genuinely leaves one shared representation, the company may postpone a more expensive configuration or run two isolated processes over one set of pages. The saving comes from eliminating duplication, not from “free memory.”

TCO still includes integration work, a non-standard loader, regression tests, engineering time, and compatibility risk after framework upgrades. For a small workload, a stock runtime with a smaller model may be cheaper than optimizing at the memory boundary.

A five-day pilot

**Day 1.** Fix the model, quantization, OS, driver, and runtime versions. Record cold start, TTFT, tokens per second, and peak memory.

**Day 2.** Identify the topology: unified memory, coherent link, or PCIe. Measure bandwidth over the complete working set rather than a small buffer.

**Day 3.** Compare stock loading, a mapped path, and a resident copy when it fits. Keep prompts, token counts, and kernels identical.

**Day 4.** Repeat with two processes and under memory pressure. Check outputs bit-for-bit or against a stable golden set.

**Day 5.** Calculate cost per accepted answer and ongoing engineering cost. Adopt the optimization only when it wins in the target regime and has a safe rollback to the stock loader.

The work is a fresh preprint, and its results are tied to specific machines, formats, and experimental code. It justifies measurement, not a guaranteed speedup. The management takeaway is simple: before buying memory, find out whether the stack stores the same model twice. Vnutrik has already brought the second pile of crates; a good profiler can still stop him at the door.