quantization · correctness · sm_121

When your Spark outputs garbage, check the FP4 kernel path before your prompt

Published 25 Oct 2026 · Updated 28 Oct 2026

Field note · GB10 / sm_121 · observations from July to October 2026; kernel support changes with every release

You load a quantized model, send a normal prompt, and get back repeated tokens, a line of !!!!!, or fluent text that's subtly wrong. Different prompts and sampling settings don't help. The instinct is "the model is broken" or "my prompt is bad." On GB10 the first suspect should be the kernel doing the quantized math.

What we saw, with dates

  • July 2026, published reports we reviewed: the CUTLASS FP4 mixture-of-experts kernels shared by vLLM and SGLang were compiled for sm_120, and a capability check that treats every sm_12x chip alike let them run on GB10 (sm_121), where they produced garbage, typically ! tokens. Some FP4 expert kernels in vLLM also gated on datacenter Blackwell and silently fell back to a path that dequantizes to bf16, so "FP4" did not mean native FP4 math.
  • July 2026, our own run: MiniMax M3 in NVFP4 produced repeated-token garbage on the engine's automatically chosen MoE kernel, from unclamped expert scales turning into infinities. A different MoE kernel (B12X) on the same weights gave correct output.
  • September 2026, a different failure with the same look: on GLM-5.3-Flash, CUDA-graph capture produced wrong tokens with no crash and no NaN, while eager execution was correct. The cause was a caching bug in an attention library under graph replay, not quantization. "Garbage with no error" is not always FP4.
  • 13 October 2026, re-measured: SGLang 0.5.14's automatically chosen FP4 MoE path (cutlass_moe_fp4, now built for sm_121a) was correct on GB10: 0.17–2% relative error against an exact emulation, cosine similarity ≥ 0.9998, no NaN, on one MoE layer of GLM-5.3. flashinfer_cutlass matched it bit for bit. flashinfer_trtllm refused to run (it ships datacenter-only kernels), so don't select it on GB10. Older SGLang (0.5.12.post1) ran the model's sparse-attention indexer on the wrong layers.

So the July "FP4 is garbage on Spark" finding applies to the builds it was measured on, not to every FP4 path today. Our multi-node benchmark entries all run NVFP4 weights.

The tell: output that's structurally broken (repeated characters, !!!!!, the same token over and over) rather than merely low-quality points at a kernel or execution-path problem, not the model or the prompt. If another kernel or execution mode gives coherent text on the same weights, the model is fine.

What to do before you blame the model

  • Find out which kernel actually ran, not which one you configured. Engines pick backends automatically and fall back silently.
  • Run a control. Same weights, a different MoE backend, or eager mode instead of CUDA graphs. If the control is coherent, the model is fine.
  • Check coherence before speed. A wrong kernel can be fast. Compare a few greedy outputs against a known-good run before benchmarking.
  • Note your versions. sm_121 support is changing release by release; a result from three months ago may not hold for your build.

Getting garbage out of a model that should work?

We determine whether it's a kernel or execution-path problem or a real model issue, on real GB10 hardware with a known-good control, and tell you the backend that gives correct, fast output on your versions. Or it's free.

Book a 1-day diagnostic →

Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ, so measure your own.