MiniMax M3-class (~400B MoE + vision) on 8× GB10 (DGX Spark-class): 6.4 → 34.9 tok/s
Published 25 Oct 2026 · Updated 28 Oct 2026 · measured 25–29 Jul 2026
8× GB10 · 100 Gbps switch · 4-bit · 512k context · vLLM fork with B12X kernels (development build) · protocol stated below
This is the most instructive entry in the database, because it went through nearly every trap on the index on its way from 6.4 tok/s at the first launch that gave correct output to 34.9 tok/s delivered on code — about a 5.4× journey, every step measured. (The engine's default kernel path was faster than 6.4 but produced garbage.)
| metric | result | notes |
|---|---|---|
| decode · code (spec on) | 34.9 ± 0.7 tok/s | n=15, 95% CI, 28 Jul, Eagle3 drafter, 3 draft tokens; the 27 Jul matched comparison read 34.55 ± 1.13. First correct launch was 6.4 (no speculation, no CUDA graphs) |
| decode · prose (spec off) | 28.9 ± 0.2 tok/s | matched no-spec control, 27 Jul; our record does not say whether the vision encoder was on for this run. Separate no-spec runs that day read 28.3 with vision off and 26.2 with it on. Code and prose need different configs |
| aggregate · 16 streams | 170 tok/s | separate no-speculation profile for concurrent load; ~10.6 tok/s per stream; one run |
| prefill | 699 → 1,520 tok/s | one configuration line; time to first token on a 10k-token prompt 14.6 → 6.7 s |
| prefix caching | up to 67× | 8 agents with a shared 24k-token prefix: 5.1 s, vs 342 s when every agent's context is unique |
| context | 512k | engine checked at 450k tokens; decode falls to ~15 tok/s that deep |
Protocol: single stream, one base prompt per workload with a run-number suffix, 256 output tokens with end-of-sequence ignored, temperature 0 sent with each request, decode time only (time to first token excluded), tokens counted from the API rather than stream chunks. The headline code figure is an n=15 run on 28 Jul with a 95% confidence interval. The prose row and the 27 Jul code figure come from one matched spec-on / spec-off comparison in a single session; there the ± is the spread our bench reported, and the number of runs was not recorded. Measured 25–29 Jul 2026 on a vLLM fork with B12X kernels (development build); newer builds may differ.
The traps this model hit (all documented in the field notes)
- Launch overhead was the difference between 6.4 and 22–23 tok/s (no speculation): the MoE kernel did host-side work that is not allowed inside CUDA graph capture, so the model ran eager until two code fixes and one setting made it capturable.
- Turning on speculative decoding then silently switched graph capture off for the whole model — a separate trap; restoring capture in spec mode, then capturing the verify step too, took speculation from slower than none to +19.8% on code, while prose still lost (the speculative decoding and CUDA graphs note).
- Vision was silently disabled by a default — a vision model shipping text-only, no errors.
- The prefill fix was one configuration line worth 2.2× (the 4.4× protocol tax).
- Code and prose want different serving configs — one profile cannot be optimal for both; we ship two and switch by workload.
Why this matters beyond this model
The ~400B class is the sweet spot of the multi-node Spark question: too big for one or two boxes, small enough that a tuned 8-node cluster serves it at genuinely usable speed. We have not seen measured numbers for this class published elsewhere with the failure path included.
Your 400B-class deployment stuck in the single digits?
This model started at 6.4 tok/s too. The diagnostic finds which of these traps your cluster is in — reproduced on our lab, delivered in writing, or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your engine versions and configs will differ — measure your own, and count only delivered tokens.