Spiritbuun's VBR KV cache squeezes more context into 12GB
A llama.cpp fork from spiritbuun adds Variable Bit Rate KV cache quantization, dynamically degrading cache precision only when VRAM pressure demands it.
Spiritbuun’s buun-llama-cpp fork adds a VBR (Variable Bit Rate) KV cache that works differently from any fixed-tier quant you’ve probably used. Instead of pinning the cache at q4_0 or q8_0 for the whole session, it starts at FP16 and only degrades one layer-side tensor at a time as your VRAM budget actually fills up. The codec ladder runs f16 → turbo8 → turbo4 → turbo3_tcq → turbo2_tcq → turbo1_tcq, and the fork picks where to stop based on what your GPU has left after weights and compute, not on what you guessed when you launched the server.
The practical pitch for 12GB cards is real. A Reddit community writeup reports running Qwen3.6-35B-A3B on an RTX 3060 12GB using the fork plus Apex I-Compact quantization, with meaningful VRAM efficiency gains, though the specific numbers there are second-hand and worth verifying against your own setup. The fork’s own benchmarks are measured on a 3090 with Qwen3.6-27B, so treat the 3060 results as a promising data point, not a guarantee.
The flag you want is --vbr-kv (or equivalent per the repo docs): it derives a KV VRAM budget automatically from whatever is left after weights, advertises the largest context that fits at the floor tier, and degrades on the fly as context fills. No codec to pre-select, no context length to guess. For anyone who’s spent time fiddling with -ctk and -ctv values trying to hit a sweet spot, that’s a genuinely useful default. Whether it holds up on a 12GB card with a 35B MoE is the question worth testing.