Colibrì runs GLM-5.2 (744B MoE) on 25GB of RAM by streaming experts from NVMe
Colibrì is a pure-C, zero-dependency inference engine that runs GLM-5.2, a 744B MoE model, on a consumer machine with 25GB of RAM, by keeping dense weights in RAM and streaming routed experts from disk.
Colibrì is a pure-C, zero-dependency inference engine that runs GLM-5.2, a 744B Mixture-of-Experts model, on a consumer machine with around 25GB of RAM. The trick is splitting the model in two: the dense layers, attention weights, and shared components sit resident in RAM at around 9.9GB after int4 quantization, and the routed experts live on local NVMe as a roughly 370GB int4 model. Per token, the router picks which experts to activate, and the engine streams only those from disk. The ~700B parameters not needed for a given token stay on the drive.
The speed numbers are honest and not good: cold decoding runs around 0.05 to 0.1 tokens per second, because a cold token can require about 11GB of expert reads across 75 MoE layers. Short answers can take several minutes. The project’s own developer described the goal as making the model answer correctly no matter how slow, not making it fast. A learning cache that pins frequently-used experts in spare RAM does help over a session, and the engine reportedly gets faster with repeated use as the hot-expert map fills in. On Apple Silicon with a fast SSD, some community measurements land closer to 1 token per second, which is still well south of usable for most tasks. The engine is Apache-2.0 licensed and available on GitHub (JustVugg/colibri).
This is squarely a proof-of-concept right now. The 4-bit quantization accuracy loss hasn’t been formally measured yet, and the hardware requirement is a big NVMe drive (~400GB free) in addition to the RAM budget. But the architecture is genuinely interesting: MoE sparsity is usually exploited to save GPU memory; Colibrì pushes that logic down to consumer NVMe and RAM, which is a different kind of local-inference math. If disk streaming gets faster with hardware improvements and caching gets smarter, this approach has legs. For now, treat it as a demo of what’s possible, not a daily driver.