Spiritbuun's VBR KV cache squeezes more context into 12GB
A llama.cpp fork from spiritbuun adds Variable Bit Rate KV cache quantization, dynamically degrading cache precision only when VRAM pressure demands it.
A 75B MoE at 132 tokens per second on three used 3090s, so where are its siblings?
A community rig ran NVIDIA's Nemotron Puzzle 75B-A9B in NVFP4 across three power-capped RTX 3090s at 132 t/s decode, and asked why almost nobody ships models shaped for multi-24GB rigs.
Unsloth NVFP4 quants for Qwen3.6 claim faster throughput than NVIDIA's own builds
Unsloth published NVFP4 quantized checkpoints for Qwen3.6 27B and 35B-A3B that reportedly beat NVIDIA's reference quants on throughput, with FP8 KV cache support for longer contexts.