Fast for one user, slow for a team: the speculative-decoding trap

Published 28 Oct 2026

Field note · measured on 4× GB10 (DGX Spark-class) · GLM-5.3-Flash · 1 to 128 concurrent users

Our fastest single-user setup for GLM-5.3-Flash (MIT drafter) on four GB10 (DGX Spark-class) nodes answers one person at about 57 tok/s on prose and 104 on code. Put 128 people on that same setup and the system totals about 240 tok/s, and the unluckiest user waits almost four minutes for a first word. Switch speculative decoding off, and the same four nodes serve those 128 people at 535 tok/s total, with every first word arriving in under three seconds.

These runs use fixed 512-token answers, so the absolute numbers differ from our benchmark entry (natural-length answers with reasoning). Compare within this table. Each cell is one run.

Same model, same hardware, same day. The difference is which question the configuration was tuned to answer: how fast one person gets an answer, or how many people get answers at all.

The measurement

Two server configurations, each started clean, then loaded with 1 to 128 simultaneous requests: greedy decoding, 512 tokens each. The table shows prose; code ran too (team setup at 128: 508 tok/s total, worst wait 3.1 s). Fast is our speculative-decoding setup, tuned for one user. Team is the same engine with speculation off and room for 128 requests at once.

tok/s, delivered tokens. "Worst wait" = slowest time to first token in that round. 4× GB10, GLM-5.3-Flash (NVFP4 experts, 4-bit dense layers), measured 28 Oct 2026.

Why the fast setup stops scaling

1 · Guessing ahead stops being free. Speculative decoding has a small model guess several tokens, and the big model checks them all in one step. With one user, that check costs little more than producing one token, because the step is dominated by reading weights. With several users in flight, each step of this mixture-of-experts model already touches a large share of its experts. Extra guessed tokens then add real work. On prose, by four users, the fast and team setups give each person almost the same speed (29.8 vs 29.0 tok/s). Code, which guesses well, still led at four users (42 vs 29).

2 · It runs out of room before it runs out of compute. The fast setup appears to set aside far more cache per request for its guesses. In our build, 23 requests filled 96% of a cache sized for over two million tokens, so no more could start. Everyone else waited in line. That queue explains the 48-second, 98-second and 238-second worst waits: those users weren't decoding slowly, they hadn't started. The team setup held 128 requests at 41% of its cache.

What to do about it

• Pick the config for the workload. One developer or agent driving the model: speculation, tuned for one stream. A team or a batch job: speculation off, with high concurrency.
• Or run both. Four nodes can run one setup at a time, so switching costs a restart (5–10 minutes for this model). With eight nodes, you could serve one of each and route requests by job type.
• Watch the queue, not the GPU. In the fast setup at 128 users, the throughput looked respectable. The cost showed up only as waiting requests and first-token time.

Engines are adding ways to adjust how far ahead to guess as load rises. In this build, our attempts to switch guessing off automatically under load didn't boot. That's on our list. Until it works, two plain configurations, each measured at its real load, beat one clever one that's wrong half the day.

Free to read and share. Numbers are from hardware we've measured directly; your engine versions and configs will differ — measure your own, and count only delivered tokens.