Where your Spark's 128 GB actually goes, and why the memory numbers lie
Published 25 Oct 2026 · Updated 28 Oct 2026
Field note · measured on GB10 unified memory, July 2026
You have 128 GB. Your model is 70 GB. It won't load, or it loads and something wedges, or the machine reboots when you were sure there was headroom. On GB10 the memory accounting misleads you, because CPU and GPU share one pool and most tools were written for machines where they don't. (nvidia-smi doesn't even report memory on GB10; it shows [N/A].)
"Available" memory is not a safe gate
We loaded a small second model next to a resident large one because free showed 12 GiB available and the new engine's memory budget was 9.13 GiB. The node ran out of memory and rebooted, taking the running model with it. Two things were wrong: "available" counts page cache that CUDA cannot reclaim fast enough during a load, and the engine's memory budget excluded parts of the process (its CUDA context, a vocoder, a tokenizer session, and the peak while weights are staged). On unified memory, free-memory arithmetic is not a safe gate. We now gate launches on what CUDA itself reports (torch.cuda.mem_get_info()) and refuse to load a second model next to a resident one.
"Used" memory after big reads: reclaimable, but not by the usual step
Load a big model and "used" memory spikes and stays high. Much of it is reclaimable, but on GB10 dropping the page cache alone was not enough to give the GPU its memory back. On our head node:
- after
drop_cachesonly: 48 GB free as CUDA sees it; - after also writing
1to/proc/sys/vm/compact_memory: 114 GB free.
In the same situation free -g happily showed 115 GB while a launch died with "Free memory on device (57.65/121.63 GiB) on startup is less than desired GPU memory utilization."
The mental model: treat memory as one shared budget with three claimants: model weights, KV cache, and everything the OS and other processes hold (including cache and driver-held memory). Check what CUDA can actually allocate, not what the OS calls available.
The settings and events that wedge you
- Setting the GPU memory fraction too high. On our head node, which also runs a desktop, an SGLang memory fraction of 0.85 left about 8 GB for everything else; the detokenizer stopped responding and the server wedged. 0.70 worked. A vLLM memory setting of 0.65 ran the same node out of memory during CUDA-graph capture; 0.50 fit. The fraction is taken from the one shared pool, not from a dedicated GPU reserve.
- Hard-killing a server can leave memory held with no owning process. We saw about 95 GB per node after hard-killing an SGLang server, and 112 GB "used" on the head node with only about 10 GB actually committed by processes. It is recoverable without a reboot:
sync,drop_cachesandcompact_memoryfreed the head node's 112 GB in about 3 seconds. Try that before anything more drastic; reloading the driver module is riskier and has left GPUs unusable for us when done in the wrong order. - KV cache is reserved up front. One engine reserved 46.98 GiB of KV cache (room for 781,712 tokens) at startup while serving with a 32k-token limit. The reservation is set by the memory fraction, not by the context limit, so budget for it before you load.
Accounting for 128 GB of unified memory correctly is the difference between confidently loading a model that fits and repeatedly crashing a machine that had the room all along.
Out-of-memory errors on a machine that should have room?
We map where your 128 GB is going (weights, KV cache, reclaimable cache, and leaks) on real GB10 hardware, and hand you a memory budget you can trust. Or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ, so measure your own.