DiffusionGemma: When 1,500 Tokens per Second Matter for Local Business

The news is the report, not a repeated launch

Google DeepMind has published the DiffusionGemma technical report, detailing an open experimental language model that does not generate strictly one token at a time from left to right. Instead, it iteratively refines a 256-token block. The first report version was submitted to arXiv on July 31, 2026. The model and weights originally appeared on June 10, so the current event is the disclosure of architecture, measurements, and limits—not a second launch of an old release.

The headline number is roughly 1,500 output tokens per second on one NVIDIA H100. This is not a universal speed for every local deployment. The result uses FP8, a single request, 4,096 input tokens, and 1,024 output tokens; the authors explicitly focus on low-concurrency serving. For a business, the important point is a new trade-off: some model quality and more computation per token can be exchanged for exceptionally low latency for an individual user.

DiffusionGemma is released with open weights under Apache 2.0. It is based on Gemma 4 MoE, contains 25.2 billion total parameters, and activates about 3.85 billion per token. Reference implementations exist for Hugging Face Transformers and vLLM, alongside FP8 and NVFP4 quantisations. This is a testable stack, although its creators still describe the release as experimental.

How a diffusion LLM prints a paragraph at once

A conventional autoregressive LLM resembles a typist: the next token appears only after the preceding one. Even when a GPU can execute far more arithmetic, every step reads weights and the KV cache from memory again. With one or a handful of requests, accelerator compute units may remain underused while memory bandwidth is the bottleneck.

DiffusionGemma first encodes the prompt and stores context in a KV cache. It then creates a “canvas” of 256 random tokens and passes the whole block through bidirectional attention several times. More confident positions are committed at every step, while the rest are refined again. Once complete, the block enters history and the next one begins.

According to the report, the model accepts roughly 20 tokens per forward pass on average and usually completes a canvas in about 12 passes with adaptive stopping. The authors contrast this with approximately 3–6 accepted tokens per pass for modern speculative-decoding configurations in their comparison.

Each diffusion pass is heavier: it processes 256 positions, more MoE experts, bidirectional attention, and a dedicated sampler. On H100, one such step was about 3.2 times slower than a single autoregressive step but released far more tokens, producing higher final decoding speed.

Where a business may notice the difference

The strongest fit is an interactive local system serving one user or a small queue where an answer must appear almost immediately.

**Structured extraction.** The report shows that rigid JSON and text closely tied to the source can converge in two or three steps because syntax and much of the content are predictable across the block. Practical uses include local extraction from requests, invoices, delivery notes, and forms. JSON Schema and business validation are still required after the model.

**Inline editing.** Bidirectional refinement naturally supports filling a gap inside text or code. A user fixes the beginning and end, while the model reconstructs the middle using both boundaries. This can be more useful than a conventional chat for contracts, instructions, or source code.

**Short agent responses.** When an agent has already retrieved facts from RAG and only needs a compact final result, low latency improves the working interface. The benefit is clearest in a local application without a large pool of simultaneous users.

**OCR and document normalisation.** Diffusion can revise related positions together, such as a table structure or identifier format. However, published comparisons show lower vision accuracy than standard Gemma 4, so extracted values require deterministic verification.

A long analytical report, legal conclusion, or autonomous decision is a poor first pilot. Maximum depth and accuracy matter more than display speed in these tasks, while DiffusionGemma’s brevity can become a constraint.

The conditions behind 1,500 tokens per second

The report gives an average of 1,456 tokens per second for H100 FP8 at batch size 1, rounded by the authors to 1,500. In the same measurement, Gemma 4 AR delivered 204 tokens per second and its multi-token-prediction version 303. This is a same-device comparison in a specified configuration, not a promise for a laptop or ordinary office GPU.

The speedup changes the bottleneck. Diffusion decoding transfers large quantities of data from memory fewer times but performs more arithmetic. This is attractive on modern GPUs with high compute density. On a CPU, a weaker accelerator, or a system with a different compute-to-memory ratio, the advantage may shrink or disappear.

Concurrency sets another boundary. The authors report that DiffusionGemma beats Gemma 4 with MTP in per-user latency and total throughput up to roughly 32 simultaneous requests. Above that point, the autoregressive model begins to lead in aggregate throughput because its compute cost per token is lower. The implementation has not yet been specifically optimised for large batch sizes, and realistic production-traffic analysis is left for future work.

The implication for an SMB is counterintuitive: a diffusion LLM may be most interesting not for a huge queue, but when an expensive GPU serves one specialist, a local agent, or a few interactive workstations and has spare compute capacity.

Quality: speed has a price

DiffusionGemma does not beat its Gemma 4 starting point on most published benchmarks. The model card lists MMLU-Pro at 77.6% versus 82.6%, AIME 2026 without tools at 69.1% versus 88.3%, and LiveCodeBench v6 at 69.1% versus 77.1%. These are developer-reported results, not an independent audit, and they do not predict performance on a company’s Russian-language documents.

The authors explicitly explain the gap: the model starts from autoregressive weights instead of native diffusion pretraining, supervised fine-tuning is relatively short, and reinforcement learning plus distillation deliberately optimise for fewer passes. Speed is part of the training objective rather than a free improvement.

Known limitations include:

• unusually concise responses that limit long reasoning traces;
• rare local token repetition or “stuttering”;
• occasional missing closing thought tags in multimodal mode;
• declining advantage under high concurrent load;
• factual errors and ambiguity shared with other LLMs.

A production pipeline needs a watchdog: length limits, repetition detection, timeouts, structural validation, and automatic fallback to a conventional model or human queue.

What a local pilot requires

BF16 weights for 25.2 billion parameters have a theoretical parameter footprint of about 50 GB before KV cache, activations, and runtime overhead. The original checkpoint therefore needs a large-memory accelerator or distributed placement. FP8 roughly halves the weight footprint, while a 4-bit version roughly quarters it; actual memory use exceeds these lower bounds because context and runtime also consume memory.

A minimal testable stack includes:

• the official `google/diffusiongemma-26B-A4B-it` checkpoint;
• current Transformers or vLLM;
• an isolated GPU test environment;
• an OpenAI-compatible gateway for comparison with the current model;
• logging of weight, sampler, quantisation, and stopping-parameter versions;
• real anonymised tasks with approved reference outputs.

Do not treat an arbitrary GGUF conversion as proof of support. The June launch announcement still described official `llama.cpp` integration as forthcoming, although its repository now contains an experimental diffusion CLI for several architectures. A production server must separately verify DiffusionGemma itself, API compatibility, quantisation quality, and sampler stability in the selected build.

Economics: count outcomes, not tokens

A TPS record can lead to the wrong investment. If an operator waits 1.5 seconds instead of five but fixes every third result, the business loses. If document processing runs overnight in a large batch, an autoregressive server with batching may be cheaper.

Consider a modelled example: 20 employees perform 50 short operations per day and each saves three seconds of waiting. That releases only 50 team-minutes daily. Even with expensive labour, this saving alone may not justify H100-class hardware. The impact can be larger if latency breaks interactive editing, increases abandoned sessions, or slows a ten-step agent chain.

Compare systems on one task and one SLA:

• p50 and p95 full latency, including prefill and network;
• results accepted without material edits;
• critical errors per 1,000 operations;
• requests per hour at realistic concurrency;
• energy use and GPU-hour cost;
• cost per accepted result;
• requests routed to a fallback model or human.

TPS is an engineering metric. Management needs cost per verified result and time to a completed process.

A ten-day pilot

**Days 1–2.** Choose one short, latency-sensitive task: JSON from a request, code repair, or filling a gap in a document. Record quality and latency for the existing autoregressive model.

**Days 3–4.** Deploy DiffusionGemma in an isolated environment. Use identical inputs, output lengths, and post-processing rules. Do not enable CRM writes.

**Days 5–7.** Run at least 200 normal and edge cases. Measure facts, structure, repetition, and human edits as well as speed. Repeat at 1, 4, 16, and the expected maximum simultaneous users.

**Days 8–9.** Add automatic fallback for repetition, timeout, and schema failure. Check whether switching overhead erases the latency gain.

**Day 10.** Decide using a quality–p95–cost-per-accepted-result matrix. If the gain exists only on an H100 that production will not have, the pilot has failed.

Management takeaway

The DiffusionGemma technical report shows that diffusion LLMs have become a testable engineering alternative rather than only a research idea. Their advantage is narrow: a powerful local GPU, low concurrency, short structured output, and an acceptable quality trade-off.

Do not replace the current model wholesale. Select one latency-critical process, compare both architectures on the same hardware, and keep routing: diffusion for fast predictable responses, a conventional LLM for complex work. Vnutrik can print a paragraph at once; the editor still checks whether an extra line entered the run.