Your GB10 RDMA at ~94 Gbps isn't broken: that's the ceiling we measured
Published 25 Oct 2026 · Updated 28 Oct 2026
Field note · measured on an 8× GB10 cluster, 100 Gbps RoCE switch, July 2026
If you've connected Grace-Blackwell boxes over a 100 Gbps switch and watched an RDMA test top out around 93–94 Gb/s instead of 100, that number is not a regression. On our nodes it is the ceiling, and it holds no matter how many rails you use.
What we measured
ib_write_bw from our head node to its peers, 65,536-byte messages, 5 seconds per test:
| test | result |
|---|---|
| rail 1 alone | 93.5 Gb/s |
| rail 2 alone | 93.5 Gb/s |
| both rails at once, to the same peer | 46.7 + 46.7 = 93.5 Gb/s |
| both rails at once, to different peers | 46.7 + 46.7 = 93.5 Gb/s |
| latency, either rail | 2.06 µs |
The last bandwidth row is the one that matters. Different cables, different switch ports, different remote machines, and the total is still exactly one rail's worth. So the limit is not the cable, the switch, or the peer. Our working hypothesis is that it sits on the sending node's side (the network adapter's connection to the rest of the GB10, shared by both rails); we measured this from the head node only, not from every node. On our head node, RDMA totals about 93.5 Gb/s however many rails are cabled. NCCL is consistent with that: a large all-reduce across all 8 nodes reached 11.98 GB/s of bus bandwidth, about 95.8 Gb/s, close to the same ceiling.
Two practical consequences:
- Two rails do not add up, even when each rail talks to a different peer. We keep NCCL on a single rail; the second one bought no bandwidth and no latency.
- Read the units.
perftestreports MB/sec, not Gb/s. Our 11,684 MB/sec is 93.5 Gb/s; mislabel that column and you'll think you have terabits.
Rule of thumb: about 93–94 Gb/s on one rail is healthy on these nodes. If you see that, stop debugging the wire; whatever is slow lives in the collective library or the framework. We didn't test faster switch ports, but because the cap held even across different peers, we would not expect a faster switch alone to raise it.
When you're well below the ceiling: check the negotiated MTU
Our fabric once ran with every switch port at the switch's default MTU (about 1,500 bytes), and RoCE negotiated what the path allowed: active_mtu 1024. Small all-reduces fragmented into several times more packets, and a jumbo ping with don't-fragment set lost 100% of packets.
- The fix: MTU 9216 on the switch ports (our Dell switch counts the Ethernet header, so 9216 carries a 9,000-byte payload) and 9000 on both NIC rails on every node.
active_mtuwent to 4096 on all 8 nodes, and the jumbo ping went to 0% loss. - What it bought: Kimi single-stream decode rose from about 28–30 to 31–33 tok/s (+10–11%). Aggregate throughput on a bandwidth-bound model did not change, which is what you'd expect: bigger frames cut per-packet overhead, not the bandwidth ceiling.
- The lesson: what you configured and what RoCE negotiated are different values. Read
active_mtufromibv_devinfoon every node, not the interface MTU, and persist the NIC setting so a reboot doesn't undo it.
The short version: about 94 Gb/s on one rail = healthy, look elsewhere. Adding a rail won't raise it. Far below it, check the negotiated RoCE MTU along the whole path before anything else.
Still stuck below line rate?
If this didn't fix it, book a one-day diagnostic with Anantus. We reproduce your multi-node setup on real GB10 hardware, find where the limit actually is, and either fix it or tell you precisely how, with the measured before/after.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your fabric will differ with firmware, cabling, and switch model, so measure your own.