The NCCL override that cost us 4.4× on prefill
Published 25 Oct 2026 · Updated 28 Oct 2026
Field note · measured on an 8× GB10 cluster, MiniMax M3 (NVFP4) and Kimi K2.7 (NVFP4), vLLM, tensor-parallel 8
Decode was fine. The fabric benchmarked at its ceiling. And yet prefill (the phase that processes your prompt before the first token comes back) ran at 699 tok/s, and a 10,208-token prompt took 14.6 s to its first token.
The cause was not NCCL getting something wrong on its own. It was a setting we had forced. Our serve scripts exported NCCL_ALGO=allreduce:tree and NCCL_PROTO=allreduce:LL, with a comment saying the LL protocol is lowest-latency for small messages. That is true for small messages. But the allreduce: prefix scopes the setting to an operation, not to a message size, so it applied to every all-reduce, including prefill's large ones.
What the override did, measured
On M3, a decode step all-reduces about 12 KB per message; a prefill chunk of 8,192 tokens all-reduces 96 MiB, 120 times per forward pass. We timed one all-reduce at each size across all 8 nodes:
| NCCL setting | 12 KB (decode) | 96 MiB (prefill) |
|---|---|---|
| tree + LL + 2 channels (our old default) | 0.079 ms | 67.1 ms |
| LL forced alone | 0.083 ms | 140.7 ms |
| tree forced alone | 0.074 ms | 58.2 ms |
| 2-channel cap only (new default) | 0.078 ms | 15.1 ms |
| nothing forced (NCCL's automatic choice) | 0.076 ms | 14.7 ms |
Two things stand out. At decode size every setting landed between 0.074 and 0.083 ms, so the override never bought anything even where it was aimed. At prefill size it cost about 4.4× (67.1 vs 15.1 ms). LL carries 8 bytes of payload in each 16-byte unit, which costs nothing at 12 KB and half the wire at 96 MiB. NCCL's automatic choice was the fastest setting we measured; at 96 MiB it reached 11.98 GB/s of bus bandwidth (about 95.8 Gb/s), close to the RDMA ceiling we measured for these nodes.
We kept the 2-channel cap: it was neutral at 96 MiB and faster than automatic at 6 MiB (1.14 vs 3.75 ms).
The lesson: a forced NCCL algorithm or protocol tuned for decode can wreck prefill, because the allreduce: syntax has no size predicate. If your environment or a copied recipe sets NCCL_ALGO or NCCL_PROTO, time an all-reduce at your real prefill message size with and without it. The signature is decode normal, fabric tests normal, but prefill and time-to-first-token far slower than the bandwidth math predicts.
What removing it bought
- MiniMax M3: prefill 699 → 1,520 tok/s; time to first token at 10,208 tokens 14.6 s → 6.7 s; decode 26.6 → 27.3–28.5 tok/s (unchanged within noise).
- Kimi K2.7 on the same nodes: prefill 647 → 1,566 tok/s; first token at 10,208 tokens 15.8 s → 6.5 s; decode unchanged.
- The microbenchmark predicted M3 would reach about 1,508 tok/s (the prefill chunk minus 8.06 s of forced all-reduces plus 1.82 s of unforced ones); we measured 1,520. Two independent methods agreeing within 1% is why we trust the cause.
One measurement trap from the same work: our first prefill benchmark reported 30,000 tok/s. That was prefix caching, because the test prompts repeated and shared prefixes. Prefill benchmarks need prompts that are unique per request and do not overlap across lengths.
Prefill slower than the math says it should be?
Book a one-day diagnostic with Anantus. We measure your actual collective sizes and NCCL settings on real GB10 hardware, check whether a forced setting is taxing your prefill, and either fix it or tell you exactly how, with the measured before/after time to first token.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your fabric and framework will differ, so measure your own.