Running a Spark cluster with no internet route, and the traps that come with it
Published 25 Oct 2026 · Updated 28 Oct 2026
Field note · from operating our one 8-node GB10 cluster
Plenty of GB10 owners want their cluster off the internet: regulated data, sensitive models, or principle. We run one 8-node cluster, and most of its nodes have no route to the internet at all. One node is also our desktop machine and is our audited way out; one other node can reach the internet through our office network to download software packages. That is not a certified air gap, but it is close enough to hit every trap below. The platform and the serving stacks assume connectivity in several places.
Trap 1: software that finds its own IP by dialing out
Some multi-node engines work out "which address am I?" by opening a connection toward the internet and reading the local end. With no outbound route, that fails, and the failure is quiet:
- SGLang stalled during multi-node startup, after the GPUs had already connected to each other and before any weights loaded. The fix was setting
SGLANG_HOST_IP(andHOST_IP) to each node's own fabric address. - Ray fell back to binding
127.0.0.1, so worker nodes timed out trying to reach the head. The fix was passing--node-ip-addressexplicitly on every node.
Trap 2: telemetry that is on by default
In July 2026 we found a vLLM worker holding an open HTTPS connection to stats.vllm.ai. vLLM's usage reporting was on by default and logged nothing: on every engine start it sent a hardware fingerprint (GPU, CPU, memory, kernel), the model architecture, quantization and parallel settings, an environment-variable dump, and a persistent ID that links every run. It did not send prompts, completions, or model paths. A local ledger showed 4,808 records over about six weeks. We blocked it four ways on all 8 nodes: vLLM's do-not-track file, environment variables for vLLM, Ray and Hugging Face telemetry, /etc/hosts entries for the known endpoints, and the same variables exported by the serve scripts.
The rule: /etc/hosts entries are one layer, not an air gap. They only cover the hostnames you know about. We found our leak by looking at live connections, not by reading configuration. "Air-gapped" is a property you check at the wire.
Trap 3: updates and dependencies assume a connection
- System and driver updates are built around pulling from the network. On an offline node you have to stage the packages yourself and install them in the right order. On our cluster only one node has a working package manager, so we download packages there and install them on the others by hand.
- Model and tokenizer loading can default to fetching from a model hub; point the engine at local paths and keep a local copy of everything it needs.
- Version pinning matters more offline: you can't quickly pull a fix, so the build you seal has to be verified before it goes behind the gap. Upgrading one package can quietly change another: a FlashInfer upgrade once downgraded our NCCL.
None of this is exotic, but each item cost us a failed launch or a surprise before we wrote it down.
Need a Spark cluster that's genuinely sealed?
We'll check your egress at the wire, get the offline update and serving path working, and show you what your stack tries to send. Or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ, so measure your own.