DGX Spark / GB10 field notes
The traps we hit bringing up multi-node Grace-Blackwell — and the measured fixes. Everything here is measured on our own 8-node cluster (how we measure). If it saves you a week, and you'd rather hand the next one off, that's what the diagnostic is for.
Field notes, tools and benchmarks
Why is my DGX Spark slow? A checklist
Ten common causes of low tok/s on a Spark or GB10 cluster, in the order to check them: the stuck low-power state, benchmark artifacts, unified memory, FP4 kernels, weight loading, speculative decoding, team vs single-user configs, RDMA, NCCL, and multi-node topology.
database · measuredThe GB10 benchmark database — big models, real numbers, 1–8 nodes
Kimi K2-class ~1T MoE at 27.8 ± 2.3 tok/s on code (n=15) on 8 nodes. Qwen3.6-35B-A3B at 45.2 on one. A ~320B on four, plus a 753B and a ~400B MoE in between. Delivered tokens throughout, with the tuning journey described and the protocol behind each entry stated.
free toolWill it fit — and how fast? The DGX Spark calculator
Model size, quant, node count in — a fit verdict and a rough single-stream speed estimate out, from memory bandwidth plus per-token costs we measured. A rough guide, not a promise: the calculator page states how far our measured results landed from its estimate.
indexThe GB10 trap index — 40 ways this platform bites
Every trap we've root-caused on our multi-node Grace-Blackwell cluster, in one list: the symptom you'll see, and what kind of problem it really is. Fabric, loading, unified memory, spec decode, silent wrong-output bugs, and the monitoring tools that mislead.
speed · spec decode · licensingGLM-5.3-Flash on 4 GB10 nodes: 47.7 → 98.8 tok/s on code, with an MIT drafter
knapcio had already published results twice as fast as ours on the same hardware, which showed us the headroom we'd left. We built on knapcio's MIT/Apache code, swapped the non-commercial drafter for Red Hat's MIT one, and added our own changes. What worked, what didn't, and where we still stand.
serving · concurrencyFast for one user, slow for a team: the speculative-decoding trap
57–104 tok/s for one user, but about 240 tok/s total and four-minute waits at 128 users. Turn speculation off: 535 tok/s total, first tokens under 3 s on prose (3.1 s on code). Same model, same nodes.
quantization · qualityIs 4-bit safe for the dense layers? The probe said check; the exam said yes
About 10% faster per token. GSM8K 247 vs 246, HumanEval 154 vs 154, paired p = 1.0, and exactly what that does and doesn't prove.
networking · multi-nodeYour GB10 RDMA at ~94 Gbps isn't broken: that's the ceiling we measured
One rail measured 93.5 Gb/s; two rails together still totalled 93.5, even to different peers, so the cap appears to sit on the node side. When to stop debugging the wire, and the negotiated-MTU check for when you're well below it.
vLLM · loadingWhy your model hangs "loading weights" on GB10: three causes we measured
A slow file-to-GPU copy path (one clone() took Kimi's weight load from 452 s to 34 s), one storage device feeding eight nodes, and ranks waiting at a barrier rather than loading. The signature of each.
Is your GB10 stuck in low-power mode? A 45-second check
Two of our eight GB10s ran at less than half speed with no error and no throttle reason. A free one-file check tells you in 45 seconds, with healthy vs stuck reference numbers, and the exact power-drain order that fixed ours.
performance · prefillThe NCCL override that cost us 4.4× on prefill
We had forced an NCCL protocol meant for small decode messages, and it applied to prefill's large all-reduces too. NCCL's own automatic choice was fastest. Removing the override: 699 → 1,520 tok/s prefill, TTFT 14.6 s → 6.7 s, decode unchanged.
spec-decode · cuda graphsSpeculative decoding made our model slower. It had switched off CUDA graphs.
With speculation on, the engine silently ran the whole model without CUDA graphs. Restoring capture, then capturing the verify step too, turned speculation into a +19.8% win on code; it still lost on prose.
vLLM · multi-nodevLLM tensor parallelism across Spark nodes: when more nodes help
Going from 4 to 8 nodes was worth +2% with eager execution and +34% with CUDA graphs on the same model. Why launch overhead hides the gain, which TP sizes a model allows, and why expert parallelism lost for single-stream.
networking · RDMAGPUDirect RDMA on GB10: NCCL turns it off. Here's what that costs.
NCCL reports GPUDirect RDMA disabled even when forced on. Measured: large all-reduces still reach the per-node RDMA ceiling, forcing it on changed nothing, and the hang that looked like NCCL was an unset host IP.
quantization · correctnessYour Spark outputs garbage? Check the FP4 kernel path first
On sm_121, some FP4 kernel paths have produced "!!!!!" or repeated tokens with no error. What we saw in July, what changed by October (SGLang 0.5.14's FP4 MoE path measured correct), and how to tell a kernel problem from a model problem.
performance · long contextDoes Spark prefill collapse at long context? The measured curve
1,446 tok/s at 11.7k tokens, 1,245 at 131k, 1,064 at 450k on 8 nodes: prefill slows about 26%, the wait grows, and decode falls by more than half. Plus the two long-context slowdowns that were fixable configuration.
operations · air-gapRunning a Spark cluster with no internet route, and the traps
Engines that find their own IP by dialing out, telemetry that was on by default and logged nothing, and why /etc/hosts alone is not an air gap. From running a cluster whose nodes mostly have no internet route.
memory · operationsWhere your Spark's 128 GB actually goes (and why "out of memory" lies)
CPU and GPU share one pool, so "available" memory overstates what CUDA can get, dropping caches alone may not give the GPU its memory back, and a too-high GPU fraction wedged our server. The checks we use instead.
measurement · benchmarkingYour server log says 40.7 tok/s. A careful benchmark says 27.8.
A peak from a short log window versus an n=15 mean, the chunk-counting bug that made our speculative decoding look 3× slower than it was, and the restart variance that confounds most A/B tests.
Stuck on one of these right now?
The notes are free. If you'd rather not spend weeks finding this out yourself, we reproduce your exact setup on real GB10 hardware and hand back the fix.
Book a 1-day diagnostic →New field notes, by email
A short note when we publish a new trap, benchmark entry, or tuning result. No marketing, just the numbers. Reply "unsubscribe" any time and you're off.
Subscribe by email →We reply from a real mailbox and add you by hand. No tracker, no third-party form.