Running MiniMax-M3 — a 400B model — on two desktops

MiniMax-M3 is a ~400B-parameter mixture-of-experts model. At a useful 4-bit quant it needs roughly 190 GB of memory — more than fits in any single AI desktop we have on the bench. The usual answer is "rent a datacenter GPU." We wanted the other answer: run it locally, on hardware we own.

So we pinned two NVIDIA GB10 desktops together and served the whole model across both of them with llama.cpp's RPC backend. Here's how that went — and a ready-to-adapt config you can download.

The hardware

Two NVIDIA GB10 "DGX Spark"-class boxes (a pair of ASUS Ascent GX10 units). Each is a Grace-Blackwell SoC with roughly 128 GB of unified LPDDR5X at about 273 GB/s, CUDA architecture sm_121. They're joined by a single direct ConnectX-7 link. Two of these together give us a ~256 GB memory pool — just enough.

The setup at a glance

Why this works at all

The instinct is that splitting a model across two machines must be bottlenecked by the network between them. For single-stream decode of a sparse MoE, it isn't. Decode is memory-bandwidth bound — the GPUs spend their time streaming expert weights from local memory, and only tiny per-token activations actually cross the link. That means you don't need exotic datacenter networking: plain TCP over a direct cable is plenty. ~10 tok/s on two desktops that sip power is a very fair trade.

It also means the simplest way to span two machines is the right one. No Ray, no NCCL, no tensor-parallel framework — just rpc-server on one box and --rpc host:port on the other. llama.cpp layer-splits the model in half; the main node holds its shard, runs sampling, and exposes an OpenAI-compatible API.

The gotchas (the part that cost us time)

• Turn auto-fit off. llama.cpp's newer -fit on default busy-waits and hangs across the RPC link — GPU at 0%, one CPU core pinned, the log frozen mid-"fitting params". Place layers explicitly with -ngl 999 --split-mode layer instead.
• Force TCP. RDMA over the ConnectX-7 link was unstable for us (the completion poll spins forever). Setting GGML_RDMA_DEV=none drops it to TCP — reliable, and since per-token payloads are tiny, decode speed is unchanged.
• The first load takes 13–25 minutes — and looks like a hang. On a cold start the worker's half of the weights stream over the link; the memory buffers can report "full" early while the per-tensor copy is still grinding away. This was our single biggest trap: we kept killing and "fixing" a load that just needed to be left alone. Subsequent loads are much quicker.
• Skip the warmup and CUDA graphs over RPC (--no-warmup, GGML_CUDA_DISABLE_GRAPHS=1) — graph capture stalls across the link.

One more lesson from the road to here: we first tried pruned (expert-reduced) variants of M3 on other engines. They were faster to stand up and faster to run, but quality suffered — non-English output in particular degraded badly. The full-weights UD-IQ4_XS is slower, but it's clean. For a model you actually rely on, that's the trade we'd make again.

Grab the config

The launch scripts, systemd units, and the full write-up — sanitized, no secrets — are public here:

👉 https://github.com/teriansilva/minimax-m3-2x-gb10

Fill in your own paths and the address of the link between your two machines, build llama.cpp with the MiniMax-M3 patch, and you've got a 400B-class model answering on your own hardware. Just keep it behind an authenticating gateway — the raw server has no auth of its own.

This is the kind of thing we do at superstatus.io — professionally self-hosted infrastructure, owned end to end. If running your own stack (AI included) sounds better than renting someone else's, that's exactly our wheelhouse.