GLM-5-class (753B MoE) on 8× GB10 (DGX Spark-class): 9.1 → 16.5 tok/s

Published 25 Oct 2026 · Updated 28 Oct 2026 · measured 24–26 Jul 2026

8× GB10 · 100 Gbps switch · mixed-precision 4-bit · TP8 · vLLM 0.22.1rc1 (development build) · short-run protocol, stated below

A 753-billion-parameter MoE across eight GB10 nodes. It did not run out of the box at all: it needed four fixes before it produced a token — a library version mismatch across the nodes, and three code patches (a sparse-attention path with no kernel for this chip, forced to dense attention; weights for that path that then had nowhere to load; and an attention kernel that wanted 1 KB more on-chip memory than GB10 has). From there: 9.1 tok/s at the first working launch, 16.5 tok/s with CUDA graph capture turned on — +81% from that one configuration change. Delivered tokens, single stream.

The model's own multi-token-prediction drafter needed a further three-part patch before it would load. On its own, in eager mode with 2 draft tokens, it reached 15.6 tok/s. Graphs plus the drafter together were never measured correctly: the one attempt used a benchmark that counted stream chunks instead of tokens, which undercounts speculative decoding.

Protocol (shorter than our current standard): single stream, 256 output tokens, temperature 0, streamed, one server launch per row, short sets of runs (three each for the last two rows); no confidence intervals, and only the last row splits code from prose. Measured 24–26 Jul 2026 on vLLM 0.22.1rc1. The 16.5 runs alone spanned 15.5–16.5, so do not read much into the gap between the last two rows.

Honest status: headroom remains

This entry is not finished, and we say so: a meaningful slice of this network still runs at 16-bit precision (the attention layers and output head) because its layers resist the standard quantization path — the next lever to measure, worth an estimated double-digit percentage. We publish work-in-progress states because a benchmark database that only shows finished victories is marketing.

What this class teaches

• At 753B across 8 nodes, the collective-communication layer is a first-order cost — the same protocol trap that cost 4.4× on prefill elsewhere (that story) can hit this class too.
• Per-model quantization is not transferable: the recipe that made the 1T model fly does not port here — each checkpoint's layout has to be measured, not assumed.

Free to read and share. Numbers are from hardware we've measured directly; your engine versions and configs will differ — measure your own, and count only delivered tokens.