40 traps · all hit on real hardware · all root-caused
The DGX Spark / GB10 trap index
Every one of these cost real days on a real 8-node Grace-Blackwell cluster before it was root-caused. For each: the symptom you'll see, and what kind of problem it really is — which is usually not what it looks like. If one of these is your week, finding exactly where it lives on your cluster is the hard part, and the diagnosis is what we sell.
Fabric & multi-node
Where "distributed" quietly becomes "slow."
Jumbo frames "configured," but small collectives are still slow.
RoCE negotiates what the whole path allows. Our switch ports sat at their default MTU, so
active_mtu was stuck at 1024; the fix needed the switch and the NICs. Read the negotiated value on every node, not the interface MTU. The RDMA ceiling and MTU note →"I have two rails, so I should see double the bandwidth."
Measured: two rails together deliver exactly one rail's ~93.5 Gb/s per node, even to different peers. Worth knowing before you buy more cables or switch ports.
The GPU sits at ~850 MHz and ~20 W under full load, with no throttle reason, and a reboot doesn't fix it.
A latched low-power state that halves throughput silently. A full power drain in the right order clears it. Free 45-second check →
Prefill is several times slower than the math says it should be.
Check for forced
NCCL_ALGO/NCCL_PROTO settings. Ours, meant for small decode messages, also applied to prefill's large ones: about 4.4× slower per all-reduce, with every network test passing. The NCCL prefill override note →Multi-node launch "hangs loading weights" forever.
Sometimes not a loading problem at all: in one of our cases the nodes were stuck before loading started, waiting on an IP address the engine tried to discover by dialing out. The loading-hang triage note →
A healthy 100G link "only" reaches ~94 Gbps.
That's the per-node RDMA ceiling we measured on GB10 (93.5 Gb/s), and extra rails don't raise it. Stop debugging the wire; the slowness you're chasing lives upstream.
Model loading
Minutes-long stalls that look like "big model, be patient." They aren't.
One CPU core pinned at 100%, the disk idle, weights loading for ~7.5 minutes.
It looks like a single-threaded loader, and we first read it that way. It was the copy path: memory-mapped file pages copied to the GPU at 167 MB/s instead of about 4 GB/s. One
clone() took the load from 452 s to 34 s.Random ranks stall at engine init; every launch a different node is "the slow one."
One rank drags every node down: the whole cluster waits for its worst member, and restarts only move the problem around. Ours disappeared with the copy-path fix above.
Eight nodes loading from one NFS export takes forever.
A distribution-pattern problem, not a storage-speed one. A faster NFS server won't fix it.
After big reads, "used" RAM looks maxed and you're afraid to load.
Much of that memory isn't really spoken for. Knowing what's recoverable can free tens of GB you didn't know you had.
Unified memory
CPU and GPU share one pool. Every intuition from discrete GPUs is wrong here.
Hard-killing the server leaks ~95 GB per node that no process owns.
Memory held below the process level — invisible to the usual tools. Recoverable without a reboot, if you know the sequence.
A 112 GB leak on the master, and a reboot would drop the whole cluster.
There is a no-reboot reclaim path. Knowing it exists is most of the fix.
Loaded a second model because MemAvailable said it would fit. Node OOM-rebooted.
The standard free-memory number is wrong on unified memory. Using it as a go/no-go gate costs you a node.
Raised the GPU memory fraction for more KV cache; a downstream component wedged silently.
On unified memory that fraction isn't taken from where you think — and the failure is silent.
What the silicon actually does
Spec sheets and B200 blog posts do not describe this chip.
In our July 2026 builds, FP4 kernels that scream on B200 produced fluent-looking garbage here.
A kernel-path problem on this chip (sm_121), not your model or your prompt, and every stage reports success. Support changes by release: SGLang 0.5.14's FP4 MoE path measured correct in October 2026. The FP4 garbage-output note →
You're getting ~10% of the marketing TFLOP number and assume misconfiguration.
The marketing figure isn't a target for your workload. Tuning toward an impossible number wastes weeks — know the real ceiling first.
Decode throughput is capped no matter what you tune.
Memory bandwidth sets a hard ceiling on single-stream decode without speculation. Near it, no setting helps; far below it, something specific is in the way. Measure where you are against the ceiling before tuning.
One node halves the whole cluster's throughput.
Tensor parallelism runs in lockstep, so the slowest node sets the pace. One GPU stuck at 494 MHz cut our 8-node decode to about 13 tok/s with no throttle reason shown; fixing the clock brought it to 24–26 (we had expected about 31). A related stuck low-power state survives reboots.
Speculative decoding & CUDA graphs
The biggest wins on this platform, and the easiest to measure wrong.
Turned on speculative decoding; it got slower.
Enabling it had silently switched CUDA-graph capture off for the whole model. Restoring capture, then capturing the verify step too, made speculation +19.8% faster than none on code; on prose it still lost. The speculative decoding and CUDA graphs note →
More draft tokens should mean more speedup. It got slower.
Draft depth is not the lever most people think it is. Past a point, more drafting is pure cost.
The server log shows a great tokens-per-second number; users don't feel it.
A peak from a short log window is not a sustained rate, and a benchmark that counts streamed chunks under-reports speculation by the acceptance length. The 40.7 vs 27.8 tok/s measurement note →
Config A beat config B on Monday; B beats A on Wednesday.
Restart-to-restart variance is bigger than most wins. The restart is the unit of replication — same-instance A/Bs are confounded.
Great single-stream speed, terrible under concurrent load (or vice versa).
These pull in opposite directions. One configuration can't be optimal for both.
The draft model won't load next to a quantized target.
A loader limitation with mixed-precision pairs — fixable, but not with a setting.
Silent wrong-output traps
The worst class: everything green, output quietly wrong.
A multimodal model describes images as word salad after sharding.
Not every part of a model tolerates being split — and the tooling won't warn you.
A coding agent starts strong, then rambles and gives up mid-task.
Often not the model's fault — a default nobody checks is quietly in effect.
Vision requests fail on a vision model.
A silent default, not a broken model — and no error tells you.
Temperature 0 twice, two different answers — is it broken?
Not necessarily broken — but telling harmless variation from real corruption takes a method.
A fix that worked on model A corrupts model B with the "same" architecture.
Recipes don't transfer between checkpoints, even within a family — each one has to be measured.
Ops & monitoring lies
On this stack, several standard observability tools return confident wrong answers.
pgrep says the server is gone. It's serving traffic.
Standard process tools give confident wrong answers on this stack — and automation that trusts them kills healthy servers.
The health endpoint returns dead while the server is mid-stream.
Under load, the usual liveness signal isn't reliable on this stack — supervisors that trust it restart working servers.
Your cleanup script kills itself; your grep finds itself.
A classic automation trap that takes down your own tooling. Easy once you've seen it; baffling until then.
vLLM's usage reporting, on by default in our July 2026 builds, sent hardware details home.
vLLM's usage reporting was on by default and logged nothing; we found it as a live connection. Blocking hostnames is one layer, not isolation. The no-internet operations note →
Env vars set on the master mysteriously don't apply on worker ranks.
Multi-node launchers don't behave the way their docs imply — settings silently go missing between nodes.
A tool's integrations vanish every time it starts.
A startup behaviour silently undoes your changes. You'll fix it again every day until you find it.
A node is down and it's headless in a rack.
Remote recovery is possible without a site visit — and the obvious "it's back up" signal is the wrong one to trust.
Free performance most setups leave on the table
Not bugs — levers.
Stuck at a 32k context window on a box that can hold half a million tokens.
The memory math allows far more — 512k contexts fit on this hardware; we checked the engine at 450k tokens.
Kernels tuned to death, decode still slow.
Check CUDA-graph capture first. It took one model from 6.4 to 22–23 tok/s and another from 9.1 to 16.5, more than any kernel or drafter change we made on either.
Agent workloads re-paying full prefill on every call.
Prefix caching: eight agents sharing a 24k-token context finished in 5.1 s; eight with unique contexts took 342.2 s (67×). Shared context is cheap; unique context is raw prefill.
A plain eager-mode config on a single node.
From 25.5 tok/s with a plain eager-mode config to 45.2 tok/s on the same silicon and model, configuration only (+77%, median of 5 runs). That gap is what a diagnostic finds.
Recognize your week in this list?
Each entry tells you what you're up against. If you'd rather have it found and fixed on real GB10 hardware — with the measured before/after to prove it — book a one-day diagnostic with Anantus. We reproduce your setup and report what we find, and how to fix it.
Book a 1-day diagnostic →