A 75B MoE at 132 tokens per second on three used 3090s, so where are its siblings?
A community rig ran NVIDIA's Nemotron Puzzle 75B-A9B in NVFP4 across three power-capped RTX 3090s at 132 t/s decode, and asked why almost nobody ships models shaped for multi-24GB rigs.
These are the poster’s numbers, not mine, but they are worth repeating: a LocalLLaMA builder ran NVIDIA’s Nemotron Puzzle 75B-A9B, a 75B-total, 9B-active MoE, in NVFP4 under vLLM 0.22.1, pipeline-parallel across three RTX 3090s power-capped to 200W each, and reports 132 tokens per second of decode across three concurrent streams with 256K contexts. The new Marlin fallback kernels run the FP4 path on Ampere, and the model’s hybrid Mamba design keeps the KV cache small enough to make those contexts affordable. A fourth card in the same box runs a speech sidecar, untouched.
The question the poster attached is the real story: 75B-total with a small active expert count is close to a perfect shape for the multi-24GB rigs that enthusiasts actually own, three or four used 3090s, and almost nobody ships models in that size class. The catalog jumps from 30B-class dense models to 100B-plus monsters, and the gap in between belongs to exactly the hardware this community has.
If you have a rig like this, the thread has the full config and is worth your time. And if any lab is listening: the desert between 30B and 100B is not empty because nobody lives there. It is empty because you keep not building houses.