GPUDirect RDMA on GB10: NCCL turns it off. Here's what that costs, measured.
Published 25 Oct 2026 · Updated 28 Oct 2026
Field note · measured on an 8× GB10 cluster, July 2026
On datacenter GPUs, GPUDirect RDMA lets the network card read and write GPU memory directly, with no CPU staging. On GB10 the CPU and GPU share one unified memory pool, and on our cluster NCCL reports "GPU Direct RDMA Disabled" even when we force it on (NCCL_NET_GDR_LEVEL=SYS). Collectives stage through host memory instead. It's natural to assume that is why multi-node GB10 feels slow. Our measurements say it matters less than you'd think.
What we measured
- Large messages still reach the wire ceiling. With NCCL choosing its own algorithm and protocol, a 96 MiB all-reduce across 8 nodes reached 11.98 GB/s of bus bandwidth, the same ~93.5 Gb/s per-node ceiling that raw RDMA tests show (the RDMA ceiling note). The big prefill slowdown we did have came from our own forced NCCL protocol, not from the missing zero-copy path (the NCCL prefill note).
- Forcing GDR on changed nothing measurable. With
NCCL_NET_GDR_LEVEL=5verified in the worker processes, Kimi decode measured code 24.5 ± 1.8 vs 25.9 ± 1.1 tok/s with it off, prose 24.3 ± 1.8 vs 24.8 ± 1.4 (n=15 each, overlapping confidence intervals). We keep it off. - Small messages have a real floor, cause unproven. A 12 KB all-reduce across 8 nodes takes about 74–87 µs in our sweeps, against 2.06 µs raw RDMA latency, and no NCCL setting we tried improved it beyond run-to-run noise. Host staging is one plausible contributor, alongside synchronization across 8 nodes; we have not been able to measure the share, because there is no way to turn GDR on here for comparison.
The practical read: if a multi-node GB10 run is slow, the missing GPUDirect path is rarely the first thing to fix. Check the per-node RDMA ceiling, the negotiated RoCE MTU, and any NCCL algorithm or protocol overrides first. Those are the things we measured making large differences.
"NCCL hangs at init" is usually something else
The hang that looked like NCCL was really a startup stall: NCCL had already connected all 8 ranks. The engine (SGLang) was stuck working out its own IP address by dialing out over a route our nodes don't have. Setting SGLANG_HOST_IP to each node's fabric address fixed it. If your multi-node launch hangs, check whether the NCCL communicator came up before blaming the transport; the loading-hang note covers how to tell these apart.
What this means for your cluster
- Don't expect a GDR setting to unlock bandwidth. On our hardware NCCL disables it anyway, and large transfers already hit the ceiling without it.
- Verify settings reach the workers. One of our GDR A/B tests silently ran both arms with GDR off because the launch script read a different variable name. Check the environment of the worker processes, not the launcher.
- Small-message latency is the real multi-node cost for single-stream decode, and it is mostly a fixed per-collective floor. Fewer collectives per token is the lever that moves it; fewer nodes cuts collective cost but leaves each node more weights to read per token, so measure both ways (the tensor-parallel scaling note).
NCCL hanging or crawling on your multi-node bring-up?
We reproduce your multi-node setup on real hardware, find where the time actually goes, and show you the measured before/after. Or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ, so measure your own.