IMR R&D · infrastructure · June 2026

Two GX10s, one pooled brain

How we link two NVIDIA GX10 superchips into a single memory pool to bench models too big for one box — and the exact configuration that finally made it stable.

2
GX10 GB10 superchips, ARM + Blackwell
128GB
unified memory each (~121 GB usable)
200GbE
ConnectX-7 RoCE/RDMA link between them
~240GB
pooled usable memory across the pair

The topology

ORCHESTRATOR gx10 (loan) llama.cpp llama-server · :8080 holds 30% of weights + KV • serves the OpenAI API + runs the agent loop • kept light on purpose (orchestrator overhead) RPC WORKER gx10-ad87 (ours) llama.cpp rpc-server · :50052 holds 70% of weights + KV • pure compute: receives tensors, returns activations • carries the heavy share (no orchestrator load) 200GbE RoCE RDMA · tensor-split

The config that made it stable

--parallel 1  one KV slot, not four
--flash-attn
q8 KV cache
-fit off  skip slow auto-placement
tensor-split 0.30 / 0.70  loan light
64k context
The earlier hangs weren't the cluster — they were config. Four KV slots, f16 cache, a backwards split and the auto-fit step pushed the orchestrator over its 121 GB and froze it. With the settings above, a 124 GB model loads with the orchestrator sitting light at ~40 GB.

The fit ceiling — what the pool can actually hold

~240 GB pooled wall qwen3.5-122B-A10B Q8 · ~124 GB fits comfortably ✓ (running now) Mistral-Medium-128B Q8 · ~133 GB fits ✓ 235B Q4 · ~143 GB fits, but ragged under load 235B Q8 · ~250 GB → over the wall ✗

Two modes, one slot

SINGLE-BOX — LM Studio watchdog-gated load, one model per box. models that fit ONE GX10 at Q8 + context. CLUSTER — llama.cpp orchestrator + RPC worker, pooled memory. for models too big for one box.
One brain at a time. The cluster serves a single slot, so exactly one workload runs against it — gauntlet or gate, never both at once. Live state (GPU, RAM, resident model, tok/s, context, the agent loop) is pushed every 15s to the Fleet page. The second unit is on loan and ships back, so the cluster is a temporary capability — the standing target stays "biggest model that fits one box."

IMR R&D · infrastructure note · the 2×GX10 cluster, June 2026.