176B on two desktops: running Qwen3.8-Flash-Next across a pair of GB10s

Updated 31 August 2026: this deployment moved from SGLang to vLLM. Every number below is re-measured, and the reason for the move turned out to be the most interesting thing in the project.

A 176-billion-parameter model does not fit on a desktop. It fits on two, if you are willing to let them share a cable — and if you are prepared to distrust almost every number you measure along the way.

Config, systemd glue and benchmark harness: teriansilva/qwen38-flashnext-2x-gb10.

Why two boxes

Qwen3.8-Flash-Next is 125B main parameters plus a 51B n-gram embedding table and a 4B draft head — but only 6B activate per token. The NVFP4 checkpoint is 126 GiB. A GB10 has ~121.6 GiB usable, so the weights alone overflow one box before you spend a byte on cache. Split across two, per GPU:

That cache holds 2.18M tokens — all eight concurrent sequences resident at a 262k context. It is cheap for a structural reason: only every fourth layer is full attention. The other 36 are Gated DeltaNet, which keeps a constant-size state per request, not per token.

The part that should have been wrong

These same boxes previously ran a dense 27B, where splitting across both machines measured 38–58% slower than two independent copies. The obvious inference — don't split models across these boxes — is wrong.

The dense model pays a full-width all-reduce every decode step. Flash-Next has hidden dim 2560 and three of every four layers are recurrent, with no attention all-reduce at all. The interconnect contributes ~0.27 ms of a ~30 ms token. The architecture, not the hardware, decides whether tensor parallelism is a good idea.

The numbers

Idle server, five prompts per case, exact token counts from the API's own usage, prefill excluded. Same harness for both stacks:

On SGLang, thinking mode cost about 44% of the decode rate. On vLLM that penalty is simply gone. Reasoning still costs tokens — and shares one max_tokens with the answer, so a tight cap returns an empty reply — but it no longer costs throughput.

The bug that moved the stack

Structured-output calls kept returning truncated JSON. Sometimes one token, sometimes five hundred; roughly one call in three, and in a bad window four in five. Plain-text calls looked perfectly healthy.

Two things hid the cause, and both are worth stealing.

First, a proxy rewrote the verdict. Through the OpenAI-compatible gateway, every truncated reply reported finish_reason: "stop" — a normal, successful end-of-turn. Queried directly against the engine, the same replies said "abort". The requests weren't stopping early. They were being killed, and the proxy translated that into success.

Second, the engine log had already said so, in words nobody greps for. I searched for abort, preempt, OOM. The actual line was Grammar accept_token failed … cannot accept special token id 248319.

The root cause is a two-line difference between files that ship in the same checkpoint:

config.json     text_config.vocab_size = 248320
tokenizer.json             vocab size = 248044   → a 276-slot gap

The output layer emits 248,320 logits. Only 248,044 are real tokens. The rest are padding — vocabularies get rounded up so they shard evenly across GPUs — and nothing masked them out of the sampling distribution. So the model could occasionally sample a token that does not exist. Token 248319 is exactly the last padding slot.

Constrained JSON decoding is what turned that into a failure: it correctly refuses a token that isn't in the vocabulary, and SGLang converted that refusal into an aborted request. Without a grammar the phantom token decodes to nothing and generation continues — which is precisely why only JSON traffic broke, and why my plain-text control looked fine.

It also explains a result that had made no sense: greedy decoding was worse than sampling. If a padding slot's logit sits high, taking the argmax picks it more often. I had filed that away as noise. It was the clearest evidence I had.

vLLM does not have the bug — 16 of 16 valid against SGLang's 20–70%. It also fixed a second defect for free: on SGLang a client disconnecting mid-stream left a request the scheduler decoded forever, each orphan permanently holding one of fourteen slots. Throughput bled from 33 to 10 tok/s within an hour.

Things that will bite you

nvidia-smi showing zero compute apps is not proof the GPU is free. A restart died with CUDA-capable device(s) is/are busy seconds after the previous engine stopped, with nvidia-smi already clean. Any drain guard has to wait longer than that check suggests — mine didn't, and it cost a failed boot.

A crashed engine can leave systemd reporting success. The container runs die-on-crash and the unit is Type=oneshot RemainAfterExit=yes, so when the engine died, nothing restarted it and the unit still read active. That combination is a silent outage. It now has a health watchdog.

The memory cliff is real and takes both machines with it — no kernel panic, no log, power cycle required. Never let a loader materialise a temporary copy of the 51 GB embedding table.

Advice about that table inverts between the two stacks. SGLang needed it offloaded to host RAM; vLLM needs it resident, and offloading is what breaks. Same table, opposite instruction.

Four ways a benchmark lied to me

1. I benchmarked a busy server. An early run reported 17.0 tok/s — slower than the model it replaced — because a generation was still streaming in the background.
2. I counted SSE chunks as tokens. Speculative decoding packs several tokens per chunk, undercounting by two to three times.
3. One sample means nothing. Speculative accept rate swings from 0.00 to 0.86 with content, and it dominates throughput.
4. The published headline wasn't reproducible — it came from a synthetic benchmark at temperature=0, which maximises speculative acceptance. Reproduce a benchmark's conditions before comparing yourself to it.

A fifth, added by the migration: engine and gateway disagree about field names. The in-flight gauge and the streamed reasoning field are both named differently by vLLM and SGLang. Check for the wrong one and your idle guard silently passes on a busy server, and your thinking benchmark scores zero and vanishes from the results table instead of failing.

Config

HEAD_IP=10.10.10.1                # RoCE IPs — direct cable, no switch
WORKER_IP=10.10.10.2              # also the ssh target AND the NCCL host IP
IFACE=enp1s0f1np1                 # this pair is cross-wired: head f1, worker f0
MAX_MODEL_LEN=262144              # native context, no rope scaling
MAX_NUM_SEQS=8                    # all eight stay resident in the bf16 cache

Two things worth knowing. The worker address is not merely an ssh target — the distributed runtime binds to it, so it must be the fast one. And I serve the native 262k rather than the 1M the engine advertises: this workload has 400-token prompts, and asking for eight sequences at 1M each demands four times the cache that exists, so most requests get evicted and re-processed. Above 128k, on this hardware, you are trusting something nobody has measured.

Credit where it's due

The hard part — the bring-up and loader work that makes this boot at all — is MiaAI-Lab's dual-Spark recipe, which moved to vLLM shortly before I did. Serving is vLLM; the checkpoint is RadixArk's. My contribution is the config around it, the guards, and the willingness to keep re-measuring until the numbers stopped lying.

Cover art generated locally on an RTX 5090 with Ideogram 4 and animated with SANA-WM — the same workstation that serves this stack's image and video routes.