Speculative decoding made our model slower. It had switched off CUDA graphs.

Published 25 Oct 2026 · Updated 28 Oct 2026

Field note · measured July 2026 on 8× GB10, MiniMax M3 (NVFP4) on a vLLM fork, Eagle3 drafter

On paper, speculative decoding is close to free speed: a small draft model guesses several tokens, the big model verifies them in one pass, and every accepted guess is a token you didn't pay full price for. Our draft model was fine and acceptance looked reasonable. The end result was slower than running without speculation at all, and it stayed that way for a long time before we found out why.

What was actually happening

Inference engines use CUDA-graph capture to avoid paying CPU launch overhead for thousands of small GPU operations per token. On GB10 that overhead is large: CUDA graphs alone took this model's decode from 6.4 to 22–23 tok/s.

With speculation enabled, the engine resolved the graph mode to NONE and ran the entire model eagerly. The attention backend declared that it could not capture the multi-token verify batches speculation needs. vLLM's fallback logic then looked for a partial-capture mode, but its check only recognised one way of splitting the model, not the "breakable" capture mode this fork used, so it fell all the way to no capture. Each speculative step cost about 165 ms, close to the model's eager forward time.

The log did say so: Skipping CUDA graph capture, and no Graph capturing finished line. We missed it because nothing failed and nobody compared the startup logs of spec and no-spec runs. Once we grepped every run's log for the capture line, the pattern was obvious: every no-spec run captured, every spec run didn't.

The fix, and what it bought

This was not a setting. We patched vLLM's graph-mode resolution so that, under speculation, it also accepts the fork's breakable-capture mode and drops to partial (piecewise) capture instead of none. That is safe here because breakable capture already runs attention eagerly during capture, which is what the attention backend needs. The patch went onto all 8 nodes.

• Step time: about 165 ms → 108 ms.
• Decode, k=3: code 17.0 → 26.8 tok/s, prose 10.6 → 16.5 tok/s (+56–58%).
• But no speculation with full capture ran at 26.3 tok/s. So this first fix only brought speculation level with no speculation on code, and left it behind on prose. The verify pass was still only partly captured.

These first figures are medians from our harness at the time, measured before we adopted the n≥15 rule in our methodology, so treat differences of a few percent as noise. A sweep with only this first fix agreed: at its best draft length speculation tied no speculation (27.7 tok/s), and drafting 7 tokens instead of 3 made it slower (21.4 tok/s).

Then the verify pass, and a net win on code

The remaining cost was the verify step itself: under speculation its attention still ran outside the graph. Four more source patches, in the attention kernel library and in vLLM (including the sparse-attention layers that make up 57 of this model's 60 layers), let the verify step be captured in full. A matched comparison two days later, on 27 Jul 2026 (same script and prompts, decode time only, tokens counted from the API, Eagle3 drafter with 3 draft tokens):

So speculation now pays on code, where the drafter's guesses are accepted more often (about 2.9 tokens per step, against 2.2 on prose), and still loses on prose. That is why this model ends up with separate serving profiles for code and prose. The ± is the spread our bench reported for that run; this comparison predates our n≥15 rule. One item stays open: with these patches, output with speculation was coherent but not token-identical to greedy output without it, and we did not finish attributing that difference.

The general lesson holds on other stacks too: two features that each help can silently cancel each other, and only the startup log tells you which one won.

Free to read and share. Numbers are from hardware we've measured directly; your engine version and draft setup will differ, so measure your own.