NInfer: 542 tok/s on Qwen3.6-35B-A3B, single request, one RTX 5090
A solo dev open-sourced NInfer, a from-scratch C++/CUDA engine that sustains 542 tok/s on Qwen3.6-35B-A3B over a full 65K-token decode, single request, on one RTX 5090.
NInfer is one developer’s answer to a narrow question: how fast can inference get on a single GPU if you stop trying to be general? On their bench, 542.8 tok/s sustained on Qwen3.6-35B-A3B, single request, across a full 65,536-token decode on one RTX 5090. These are the author’s numbers, not mine, reported as a mean over five seeds after a warmup. Shorter completions go faster (634 tok/s at ~8.7K tokens), and a mix of code, translation, and structured-output prompts all clear 550.
The trick is that NInfer refuses to be llama.cpp. It’s a from-scratch C++/CUDA engine built for exactly two checkpoints (Qwen3.6-27B and the 35B-A3B MoE) on exactly one card: custom quantization, a hand-designed weight layout, fused kernels, a dedicated speculative LM-head draft. That specialization is the whole point and the whole catch. No continuous batching, sm_120a only, and it runs nothing but those two models. The artifacts are on HuggingFace at roughly 5 bits per weight (16.3 and 20.8 GiB), and INT8 KV cache fits the full 262K context on the 5090’s 32GB. Engine and both converted checkpoints are public at github.com/Neroued/ninfer.
Read the numbers honestly and one detail jumps out: speculative decoding earns its keep on structured output (87% acceptance, 661 tok/s) and falls apart on creative writing (38%, 396 tok/s), because a story is exactly where the draft path stops guessing right. The author signs off with a dare: match this on the same model, same quant size, same single 5090. Someone will try.