Does Spark prefill collapse at long context? The measured curve
Published 25 Oct 2026 · Updated 28 Oct 2026
Field note · measured prefill and decode across context lengths on GB10 (July and September 2026)
People ask whether inference grinds to a halt at 100k context on a Spark. On the configurations we measured, the prefill rate does not collapse. What grows is the wait, because a longer prompt simply has more tokens to process, and what falls noticeably is decode speed.
The curve
MiniMax M3 (NVFP4) across 8 GB10 nodes, fp8 KV cache, 512k-token context window, one request at a time, with the Eagle3 speculative drafter on (3 draft tokens), so the decode column includes speculation:
| prompt length | time to first token | prefill rate | decode after it |
|---|---|---|---|
| 11.7k tokens | 8.1 s | 1,446 tok/s | 34.9 tok/s |
| 131k tokens | 105 s | 1,245 tok/s | 16.9 tok/s |
| 450k tokens | 423 s (7.0 min) | 1,064 tok/s | 14.8 tok/s |
From 11.7k to 450k tokens the prefill rate fell about 26%. Decode roughly halved by 131k, then fell much more slowly (16.9 to 14.8), which fits this model's design: only some of its layers attend over the full context.
A second data point, GLM-5.3-Flash (sparse attention, NVFP4, fp8 KV) on 4 nodes with the vendor's documented configuration: a 97,184-token prompt prefilled in 45.3 s (2,147 tok/s), and decode at that depth held within 6% of short-context speed.
Physics vs. fixable
Attention cost grows with context, so some slowdown is real. But two of our long-context slowdowns were configuration, not physics:
- A forced NCCL protocol. An override meant for small decode messages also applied to prefill's large ones, and cost about 4.4× on each prefill all-reduce. Removing it took M3 prefill from 699 to 1,520 tok/s at 10k tokens. The NCCL prefill note has the numbers.
- A hand-tuned configuration that never finished. Our own GLM-5.3-Flash setup on 8 nodes was killed after more than 75 minutes on a 107k-token prompt. The vendor's documented configuration did a 97k prompt in 45 seconds on 4 nodes.
The distinction that matters: long-context prefill (prompt ingestion) and long-context decode (token generation) degrade for different reasons. On our runs the prefill rate fell about 26% from 11.7k to 450k tokens, while decode fell by more than half. Measure them separately or you will tune the wrong one.
What keeps a long-context deployment usable
- Check your context limit against what is already reserved. M3 was serving at a 32k limit while the engine had already reserved KV memory for 781,712 tokens. Raising the limit to 524,288 tokens with an fp8 KV cache (31.5 KiB per token per node instead of 63.0 at bf16) needed no change to the memory setting.
- Prefix caching changes the economics when prompts share a prefix. Eight agents sharing a 24k-token context finished in 5.1 s; eight with unique contexts took 342.2 s.
- Benchmark long prompts carefully. A tool that calibrates prompt length with probe requests can turn one 450k data point into half an hour, and repeated prompts hit the prefix cache and report fantasy prefill rates.
Long-context deployment feeling unusable?
We measure your prefill and decode curves separately on real GB10 hardware, separate the physics from the fixable, and hand you the configuration that keeps long context usable, with the numbers to prove it, or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ, so measure your own.