GLM-5.3-Flash-class (~320B MoE) on 4× GB10 (DGX Spark-class): 98.8 tok/s
Published 25 Oct 2026 · Updated 28 Oct 2026
4× GB10 · 100 Gbps switch · 4-bit · 262k-token context configured · measured 2026-10-28 (first entry 2026-10-25) · per the methodology
A ~320-billion-parameter MoE (about 18B active per token) on half of our cluster: 98.8 tok/s on code and 73.2 on prose, single stream, delivered tokens, with an MIT-licensed speculative drafter. That is double our 25 October entry on code, and 1.8× on prose. knapcio's published results showed us how much headroom we'd left, and we now build on their MIT and Apache-2.0 code; what we changed, and what didn't help. For teams, the same nodes serve 535 tok/s across 128 users with speculation off (why that's a different config).
| metric | result | notes |
|---|---|---|
| decode · code | 98.8 tok/s | ± 2.6 (n=15, 95% CI) · was 47.7 ± 2.7 on 25 Oct |
| decode · prose | 73.2 tok/s | ± 3.8 (n=15, 95% CI) · was 41.3 ± 3.0 on 25 Oct |
| aggregate · 128 users, speculation off | 535 tok/s | prose, 512 tokens each, one run; worst first token 2.8 s · code: 508 tok/s, worst first token 3.1 s |
| aggregate · 64 streams, single-user config | 264–271 tok/s | 3 bursts; this config holds 23 requests at once. 25 Oct config: 420–427, but every 25 Oct answer ran to the 1,024-token cap (65,536 tokens per burst vs ~41k now), so the drop is partly protocol and partly the single-user tuning |
| first working launch | ~19.6 tok/s | September 2026, shorter protocol |
| long context | 97k-token prompt | measured 4 Sep on the vendor's documented config: 36.9 tok/s at 97k vs 39.1 short, within ~6%; not yet re-measured on the 28 Oct stack |
Delivered tokens include the model's reasoning tokens (this model always thinks before answering). Same prompts and protocol on both dates. On 25 Oct, 28 of 30 answers hit the 1,024-token cap mid-reasoning; on 28 Oct, with NVIDIA's checkpoint and its chat template, answers finished on their own at a mean of 650 (code) and 444 (prose) tokens. Shorter answers spend more of the clock on the prompt, which works against the new numbers. But the answers also differ in content: more of each one is answer and less is reasoning cut off at the cap, and that changes how often the drafter's guesses are accepted. These runs can't separate that effect from the software changes, so read the gain as approximate. 4-bit dense layers checked on GSM8K and HumanEval: no measurable loss. Earlier launches of the previous setup gave 50.1 on code on 14 Oct and 47.4 on 15 Oct (the second with a 1M-token context configured; one run each, on a different coding benchmark); the restart is the unit of replication.
Re-measured 28 Oct on a clean rebuild of the serving image: code 98.9 ± 3.2, prose 73.1 ± 3.7 (n=15, 95% CI), 64 streams 258–270 tok/s — the same as above within noise. The same build completed a 235,677-token prompt with the correct answer (73 s to first token).
What stood between the first launch and the 25 Oct entry
- Hand-tuning lost to the shipped recipe. Our own tuned configuration never finished a 97k-token prompt (killed after 75+ minutes); the vendor's documented path did it in 45 seconds — and then needed a small number of chip-specific corrections to run correctly on sm_121.
- A crash that left memory stranded. One launch path failed deterministically mid-startup and left ~86 GB per node held with no owning process — recoverable without a reboot, if you know the sequence (trap index).
What this entry teaches
- Four nodes is enough for this class. A ~320B sparse model serves at chat-ready speed on half the cluster, leaving the other half free for a second model.
- Read the shipped docs before tuning. The biggest single jump came from running the documented configuration exactly, then fixing only what this chip needs.
- Read what others publish on the same hardware. The second doubling came from knapcio's open stack, not from the vendor docs: what we changed on top.
Running a 300B-class model on Spark nodes?
The diagnostic reproduces your setup on our 8-node lab, measures where you really are, and ranks your recoverable headroom in writing — or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your engine versions and configs will differ — measure your own, and count only delivered tokens.