Kimi K2-class (~1 trillion parameters) on 8× GB10 (DGX Spark-class)
Published 25 Oct 2026 · Updated 28 Oct 2026 · measured 26 Jul 2026
8× GB10 · 100 Gbps switch · 4-bit · 256k context · vLLM 0.22 development build · per the methodology
A frontier-scale mixture-of-experts model at roughly a trillion parameters is widely assumed to be impractical on desktop-class hardware. Measured, it isn't: a ~1-trillion-parameter Kimi K2-class MoE serves at 27.8 tok/s delivered on code workloads (25.8 on prose), single stream, on eight GB10 nodes connected by an ordinary 100 Gbps switch — not exotic fabric. Native 256k context, and a cold start under three minutes.
| metric | result | notes |
|---|---|---|
| decode · code | 27.8 ± 2.3 tok/s | mean of 15 requests, 95% CI, one launch · three more launches of the same setup: 27.7 ± 1.9, 28.4 ± 1.8, 25.9 ± 1.1 |
| decode · prose | 25.8 ± 1.4 tok/s | same launch · other launches 22.1–24.8 |
| context | 256k native | tuned KV cache |
| cold start → serving | 2 min 45 s | was 10 min 46 s; the weight-loading step alone went from 452 s to 34 s |
Protocol: one prompt per workload, 15 requests each, 256 output tokens, temperature 0, single stream, tokens counted at the client. Speculative decoding (a drafter guessing 3 tokens ahead) makes the spread between launches wider than the spread within one: the same setup on four separate launches gave code means from 25.9 to 28.4, and a further relaunch of the same config gave 23.4 tok/s on code (n=5), about 16% below this entry, so the launch, not the request, is the unit of replication. Measured 26 Jul 2026 on a vLLM 0.22 development build; newer engine builds may differ.
For scale
This entry is a ~1-trillion-parameter model, quantized to 4-bit, spread across eight nodes on an ordinary 100 Gbps switch. On this silicon, multi-node is precisely the part that goes wrong by default.
What stood between out-of-box and this number
- It would not load usably at first — minutes-long stalls with three different root causes that look identical (the loading triage).
- In our July 2026 builds, FP4 on sm_121 produced garbage with the default kernel dispatch — fluent-looking nonsense, no errors. Getting correct FP4 output on this chip is its own discipline.
- Speculative decoding had to be tuned for the workload — the "obvious" configuration was measurably slower.
- The server log peaked at 40.7 tok/s while clients received 27.8 on average — which number to believe is half the work.
Running a frontier-class MoE on Spark hardware?
This entry records a measured tuning journey. The diagnostic gets your cluster the same treatment: we reproduce your config on our 8-node lab, find what stands between you and numbers like these, and put it in writing — or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your engine versions and configs will differ — measure your own, and count only delivered tokens.