How we measure

Updated 28 Oct 2026 · These rules exist because we broke most of them once and got a wrong answer.

• Only delivered tokens count. We count the tokens each request actually returned (the usage the server reports for that request), not streamed chunks (with speculative decoding those are steps, not tokens) and not the peak of a server log line. We cross-check the client count against the server's own counters. The 40.7-vs-27.8 note is why.
• n ≥ 15 with a 95% confidence interval for speculative-decoding configs. With speculation on, run-to-run spread on this hardware was 12–19% (standard deviation). The same config read 30.2 on one median-of-3 and 22.2 on another the same night; at n=15 it was 25.4 ± 1.9. Without speculation the spread was 1–3%, and n=5 is enough. We adopted this rule on 26 July 2026; entries measured before that are labeled below.
• The restart is the unit of replication. Two server instances of an identical config produced non-overlapping n=15 confidence intervals. For A/B decisions we run more than one restart per arm; a single-instance A/B is only trusted when it finds no difference.
• Prefill, decode, and aggregate are reported separately, labeled. On one of our deployments prompt processing ran at about 1,520 tok/s and generation at about 27 tok/s. A number without its label is not a measurement.
• Both code and prose workloads. Speculative decoding can win on one and lose on the other; a benchmark that ran only one workload has measured half the picture.
• Temperature 0, fixed prompts, warm-up excluded, clocks checked under load. One node whose GPU sat at 494 MHz instead of about 2,400 MHz cut the whole 8-node cluster's single-stream speed to about 13 tok/s, with no throttle reason reported; we had expected about 31, and fixing the clock brought it to 24–26. We check every node's clock under load before trusting a number. A related stuck low-power state has its own free check.
• Same-vs-same controls. When a test script silently ran both arms of an A/B with the same setting, the two arms still differed by about 19%. A difference smaller than what a config shows against itself is not a finding.
• Refuted hypotheses get published. The benchmark entries include what didn't work and what's unfinished.

Protocol behind each benchmark entry

Not every published entry meets every rule above, because some were measured before we adopted them. Here is what each one actually rests on:

• GLM-5.3-Flash-class, 4 nodes: n=15 per workload, mean with 95% confidence interval, delivered tokens, code and prose. The 25 Oct entry for our earlier setup was 47.7 ± 2.7 on code (n=15). Two other measurements of that earlier setup, 50.1 and 47.4 tok/s, were single runs on different days with different settings; they are not replicates.
• Kimi K2-class, 8 nodes: n=15 per workload, mean with 95% confidence interval, delivered tokens, code and prose.
• MiniMax M3-class, 8 nodes: the headline code figure, 34.9 ± 0.7 tok/s (28 Jul), is n=15 with a 95% confidence interval. The prose row and a 27 Jul code figure (34.55 ± 1.13) come from one matched spec-on / spec-off comparison whose ± is the spread our bench reported, with the number of runs not recorded; other rows come from shorter runs.
• GLM-5-class (753B), 8 nodes: measured in July 2026, before the n≥15 rule. The tuned figure is the best of a short series of runs (15.5–16.5 tok/s), with no confidence interval.
• Qwen3.6-35B-A3B, 1 node: measured on 12 July 2026, before the n≥15 rule. Each figure is the median of 5 timed requests after warm-up, temperature 0, 256 output tokens, one server instance. The top figure uses speculative decoding, which under our current rule would need n≥15, so treat it as indicative.