Your server log says 40.7 tok/s. A careful benchmark says 27.8. Here's why both appear.

Published 25 Oct 2026 · Updated 28 Oct 2026

Field note · measured on an 8× GB10 cluster, Kimi K2.7 (~1T-parameter MoE) with speculative decoding, vLLM, July 2026

Three throughput numbers from the same Kimi deployment:

• vLLM's periodic Avg generation throughput log line peaked at 40.7 tok/s.
• The Accepted throughput line from the same logging window read 29.4 tok/s. It is a different counter that measures a different thing (draft tokens accepted), not a corrected version of the first.
• A separate client-side benchmark run, 15 requests with a 95% confidence interval, counting the tokens each request returned, measured 27.8 ± 2.3 tok/s on code.

These are not three views of one moment. The first two come from one short logging window, and the 40.7 is the highest value that line showed. The 27.8 is a mean over 15 requests from a separate run. Quoting the peak as "our speed" overstates it by nearly half.

Why the peak and the mean differ

With speculative decoding, a small draft model proposes several tokens and the big model verifies them in one pass. How many get accepted varies from request to request, and throughput tracks acceptance almost one to one: six identical requests to one server gave 22.6 to 28.4 tok/s as mean acceptance ranged from 1.98 to 2.78. A log line that averages over a few seconds catches the good stretches and the bad ones; its maximum is a lucky stretch, not a sustained rate.

One thing the log line is not doing, at least on our builds: inflating its count with rejected draft tokens. In an earlier session we checked it against delivered tokens directly. The log's average generation throughput read 25.2 tok/s; steps per second times mean acceptance length gave 24.8 tok/s of emitted tokens, and the token counts matched the requested lengths exactly.

The bug that was ours

That check happened because our own benchmark was wrong. It counted streamed chunks as tokens. Without speculation that is fine, one token per chunk. With speculation, vLLM sends every token accepted in a step as one chunk, so chunk counting measures steps per second and under-reports by the acceptance length. It made MiniMax M3's speculative decoding look about 3× slower than it was, and we nearly concluded the feature was broken. Comparing against the server log caught it the same session. Since then our harness counts the token usage the server reports and prints tokens per chunk; anything above 1.0 means chunk counting would be wrong.

Other ways one number fools you

• Run-to-run variance. With speculation on, the same config varied with a standard deviation of 12–19% of the mean on our cluster. A median of 3 can land almost anywhere: one config read 30.2 on one median-of-3 and 22.2 on another the same night. At n=15 its value was 25.4 ± 1.9.
• Restart-to-restart variance. Two server instances of an identical config produced non-overlapping n=15 confidence intervals (prose 26.1 ± 0.8 vs 24.0 ± 0.8). An A/B where each arm ran in a single instance is confounded.
• Aggregate vs. single-stream. One M3 deployment delivered about 24 tok/s to a single user and 170 tok/s in total across 16 concurrent streams, about 10.6 each. Both are true; they answer different questions.
• Prefill vs. decode. An M3 deployment (configured differently from the one in the line above) processed prompts at about 1,520 tok/s and generated at about 27 tok/s. A throughput number without its phase label can be off by more than 50×.

A harness that gets this right (counting delivered tokens, replicating across restarts, separating phases, reporting honest confidence intervals) is fiddly to build, and its failure mode is silent: a broken ruler still produces confident-looking numbers. Our methodology page lists the protocol behind each published number.

Free to read and share. Numbers are from hardware we've measured directly; your stack will differ, so measure your own, and count only what was delivered.