vLLM tensor parallelism across Spark nodes: when more nodes help

Published 25 Oct 2026 · Updated 28 Oct 2026

Field note · measured on 4 and 8 GB10 nodes over a 100 Gbps switch, vLLM, July–September 2026

Tensor parallelism (TP) splits every layer across every node and all-reduces the results twice per layer, per token. Across a 100 Gbps switch that puts the network on the critical path of every token, so it's reasonable to expect more nodes to stop helping quickly. We measured it, and the answer depends on something other than the network.

The 2×2 that changed our mind

GLM-5.3-Flash (a ~320B MoE), single-stream decode in vLLM, the same model and prompts on 4 and 8 nodes, with and without CUDA graphs:

Run eagerly, doubling the nodes bought +2%, and we first concluded TP8 was worthless. With CUDA graphs it bought +34%. Broken into milliseconds per token, both are consistent: time per token is fixed overhead plus the time each node spends reading its share of the weights. Going from 4 to 8 nodes halves the weight-reading term (23.4 → 11.7 ms). Eagerly, the overhead grew with the node count (27.5 → 38.1 ms, more collective launches) and swallowed the gain. With graphs, overhead was similar on both (14.5 vs 16.5 ms), so the halved reading time showed through.

Node count isn't the biggest lever

On the same model, the vendor's documented configuration (sparse attention plus the model's built-in speculative drafter) reached 39.13 tok/s on 4 nodes, beating our best hand-tuned 8-node result without speculation (35.77). Configuration mattered more than doubling the hardware. A ~35B MoE measured on SGLang scaled from 17.5 tok/s on one node to 39.8 on eight, so small models can gain from more nodes too; measure rather than assume.

What limits your choices

• The model decides the legal TP sizes. TP has to divide the number of attention heads. With 64 heads that means 1, 2, 4 or 8; TP6 is impossible. Memory sets the minimum: the 753B model we served needed about 107 GB per node on 4 nodes, too tight to leave room for the KV cache and runtime, so we ran it on 8. The calculator puts a 753B 4-bit model at roughly 90% of 4-node capacity and flags it as tight; treat anything that close to the limit as too tight to serve.
• Every collective costs a fixed floor. A small (12 KB) all-reduce across 8 nodes takes about 74–87 µs in our sweeps, against 2.06 µs raw RDMA latency, and no NCCL setting we tried improved it beyond noise. On MiniMax M3 that was 120 collectives and about 10 ms of a 38 ms token.
• Expert parallelism lost for single-stream. On Kimi K2.7, TP8 plus expert parallelism across 8 nodes was 18% slower on code and 22% slower on prose than plain TP8 (n=15). Each node holds whole experts, so the unluckiest node on each token gets several of the active experts while others idle, and the slowest node sets the pace. Expert parallelism is for aggregate throughput, not one user.
• Pipeline parallelism: not measured here. We haven't benchmarked pipeline parallelism on this cluster, so we make no claim about it. Note that it needs layers to split evenly: Kimi's 61 layers is a prime number, so it can't be divided evenly across nodes.
• Kernel support depends on the chip. GB10 is sm_121; in the vLLM builds we used, every MLA attention backend except the Triton one gated itself out on this architecture. Check which backends your topology and model can actually use before planning around them.

Getting a model to serve well across four or eight Sparks is a measurement problem: the right node count and parallelism depend on your model's shape, your engine's capture modes, and your fabric.

Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ, so measure your own.