Spiritbuun's VBR KV cache squeezes more context into 12GB
A llama.cpp fork from spiritbuun adds Variable Bit Rate KV cache quantization, dynamically degrading cache precision only when VRAM pressure demands it.
A llama.cpp fork from spiritbuun adds Variable Bit Rate KV cache quantization, dynamically degrading cache precision only when VRAM pressure demands it.