Fabric & multi-node
Where "distributed" quietly becomes "slow."
A single RDMA flow runs at ~13 Gbps on a 100G link.
Look at the switch, not the NICs — the smallest negotiated frame size anywhere in the path wins, and what you configured is not what the fabric negotiated.
"I have two rails, so I should see double the bandwidth."
You won't. A single stream rides one rail; rails don't aggregate per-flow. Plan topology around that, not the spec sheet.
Prefill is several times slower than the math says it should be.
Look at which protocol your collective library picked for large messages — the latency-optimized choice on a bandwidth-bound transfer is a silent multi-× tax.
Multi-node launch "hangs loading weights" forever.
It's often not loading at all — rank 0 is at a barrier waiting for peers that can't find each other. Look at rank discovery, not the loader.
A healthy 100G link "only" reaches ~94 Gbps.
That's line rate after protocol overhead.
Stop debugging — the slowness you're chasing lives upstream.
Full note →
Model loading
Minutes-long stalls that look like "big model, be patient." They aren't.
Weights crawl to the GPU at hundreds of MB/s instead of gigabytes.
Look at how the memory is mapped before the copy — one class of source memory can't be DMA'd and silently falls back to a slow path. Not your disk.
One CPU core pinned at 100%, nineteen idle, GPUs idle, load takes ~8 minutes.
A serialized per-chunk loader. Storage upgrades and resharding won't help; the loader itself is the wall.
Random ranks stall at engine init; every launch a different node is "the slow one."
Look at per-rank allocator state — and remember everyone else waits at the barrier for the worst rank.
Eight nodes loading from one NFS export takes forever.
Stop broadcasting the whole model to everyone. Look at per-rank sharded loading — each node only needs its slice.
After big reads, "used" RAM looks maxed and you're afraid to load.
Look at reclaimable page cache vs. actually-allocated memory — the number you're reading includes cache the kernel will hand back.
The full triage: three loading hangs that look identical.
Unified memory
CPU and GPU share one pool. Every intuition from discrete GPUs is wrong here.
Hard-killing the server leaks ~95 GB per node that no process owns.
Look at driver-held allocations, not process memory — and know the reclaim sequence that doesn't need a reboot.
A 112 GB leak on the master, and a reboot would drop the whole cluster.
There is a no-reboot reclaim path. Knowing it exists is most of the fix.
Loaded a second model because MemAvailable said it would fit. Node OOM-rebooted.
MemAvailable lies on unified memory. Gate on what's actually serving, not on free-memory arithmetic.
Raised the GPU memory fraction for more KV cache; a downstream component wedged silently.
Look at what else shares the pool — on unified memory, "GPU fraction" is taken from everyone.
What the silicon actually does
Spec sheets and B200 blog posts do not describe this chip.
FP4 kernels that scream on B200 produce fluent-looking garbage here.
Look at which kernel actually dispatched on this compute capability — "it runs" and "it runs correctly" are different claims. Output quality is a config bug symptom.
You're getting ~10% of the marketing TFLOP number and assume misconfiguration.
Measure the real sustained ceiling for your precision first — the marketing figure is a different precision's peak. Tuning toward an impossible target wastes weeks.
Decode throughput is capped no matter what you tune.
Compute the bytes-per-token ÷ real memory bandwidth floor before touching any knob. If you're near it, the lever is precision, not settings.
One node halves the whole cluster's throughput; benchmarks won't reproduce.
Look at actual clock frequencies under load on every node — one box silently stuck at a quarter clock poisons every collective, and it can survive reboots.
Speculative decoding & CUDA graphs
The biggest wins on this platform, and the easiest to measure wrong.
Turned on speculative decoding; nothing got faster.
Look at what it silently disabled — on some stacks, enabling spec turns off all graph capture, and the launch overhead eats the entire win. Partial capture recovers it.
More draft tokens should mean more speedup. It got slower.
Look at measured acceptance length — if it's ~2, drafting 7 buys nothing and pays verify cost on all of them.
Spec-decode benchmarks great, but users say it feels the same.
Look at
what your benchmark counts — stream chunks are steps, not tokens.
Full note →Config A beat config B on Monday; B beats A on Wednesday.
Restart-to-restart variance is bigger than most wins. The restart is the unit of replication — same-instance A/Bs are confounded.
Great single-stream speed, terrible under concurrent load (or vice versa).
Spec-decode and high concurrency are opposing modes — you need two serving profiles, not one compromise.
The draft model won't load next to a quantized target.
Look at the loader's assumptions about mixed-precision pairs — it takes surgery, but it works.
Silent wrong-output traps
The worst class: everything green, output quietly wrong.
A multimodal model describes images as word salad after sharding.
Look at which submodules got sharded — not every part of a model tolerates being split across ranks, and the sharder will happily split them anyway.
A coding agent starts strong, then rambles and gives up mid-task.
Look at what sampling temperature is actually in effect when the client sends none — checkpoint defaults can be unusable for agents, and client-side env vars may be dead letters.
Vision requests fail on a vision model.
Look at multimodal input limits — some stacks default image inputs to zero, silently text-only.
Temperature 0 twice, two different answers — is it broken?
No — look at batching-dependent numerics. Greedy ≠ reproducible on a live server. Verify weights offline, not through the engine.
A fix that worked on model A corrupts model B with the "same" architecture.
Look at per-model quant format lock-in — recipes don't transfer between checkpoints, even within a family.
Ops & monitoring lies
On this stack, several standard observability tools return confident wrong answers.
pgrep says the server is gone. It's serving traffic.
The engine renames its own processes. Detect by port and listening socket, never by name.
The health endpoint returns dead while the server is mid-stream.
/health is not a liveness signal under load on this stack. A supervisor trusting it will kill a healthy server that was working fine.
Your cleanup script kills itself; your grep finds itself.
Pattern-match tools match their own command line. Bracket a character or match on ports.
Your "air-gapped" inference stack is phoning home benchmark fingerprints.
Look at actual egress, not /etc/hosts — some components resolve around it. Verify with a capture, not a config.
Env vars set on the master mysteriously don't apply on worker ranks.
Look at what your launch path forwards — many multi-node launchers pass only an allowlist through to remote ranks.
A tool's integrations vanish every time it starts.
Look for a launcher that regenerates its config file on startup, clobbering everything you added.
A node is down and it's headless in a rack.
Wake-on-LAN works on these boxes — and gate your relaunch on GPU + weights being ready, not on ssh answering.
Free performance most setups leave on the table
Not bugs — levers. Each of these was worth more than a hardware upgrade.
Stuck at a 32k context window on a box that can hold half a million tokens.
Look at KV-cache precision and the actual memory math — 512k contexts fit on this hardware, validated deep into the window.
Decode tuned to death, still slow.
The single biggest win we ever measured was a batching limit most people never touch — not the knobs everyone argues about.
Agent workloads re-paying full prefill on every call.
Look at prefix caching — on repeated-prefix workloads we measured up to 67× on the prefill phase.
Out-of-box config on a single node.
Baseline 25.5 tok/s → 45.2 tok/s on the same silicon, same model, config only (+77%). That gap is what a diagnostic finds.