A C-based inference engine streams large MoE models from disk, keeping hot experts on GPU and cold ones on NVMe, making frontier models runnable on machines with 8GB RAM minimum.

Colibri is a hierarchical inference engine written in C that runs large mixture-of-experts models on machines with modest resources. Instead of loading an entire model into memory, it keeps the parts used for every token (hot experts) on the GPU via CUDA or Vulkan, warm experts in RAM, and streams remaining experts from NVMe as the model requests them.

The engine supports 13 model families today. Language models include GLM-5.2 and GLM-5.3 (which also run Mistral-Large-4), DeepSeek V4 Flash, DeepSeek V4.1 Flash, Qwen3.6, Qwen3.8-Flash-Next, Kimi K3, and OLMoE. The engine also supports image generation (Qwen-Image-2.1) and decision models (Laya, GLiNER2.5-Decide, Clef). The README reports that a GLM-5.2 model with 744B parameters in int4 quantization became ready in 32 seconds and used 9.9GB resident memory on the test system. The minimum requirement is 8GB RAM and 22GB free disk space; 16GB RAM or more is recommended.

One-step setup

Installation runs through a single script. On Linux and macOS, clone the repository and run ./start-here.sh. On Windows, download the ZIP and double-click START-HERE.bat.

The script detects your machine’s RAM, free disk space, CPU, and available GPUs. It recommends a model that fits your hardware, then either builds the engine (if a compiler is present) or downloads a prebuilt binary. For NVIDIA cards on Linux with the CUDA toolkit installed, it builds for CUDA. On Linux and Windows without CUDA, it uses Vulkan. On integrated GPUs, it only builds for GPU if measurements show it is faster than CPU. If dependencies are missing, the script prints the exact install command and continues with CPU inference; run it again afterward to rebuild for GPU.

The script then downloads the model with resume capability—stop and restart as needed—and starts the inference engine. It prints the browser address (http://127.0.0.1:8000/) and API endpoints compatible with OpenAI and Anthropic clients.

Hardware and backends

Colibri requires no GPU. CPU inference is supported on all platforms. Prebuilt binaries run on CPU and, on Linux and Windows, on Vulkan GPUs without additional setup. CUDA builds are available for NVIDIA cards on Linux when the toolkit is installed. The engine is written in pure C and licensed under Apache-2.0.

Who this is for

Colibri targets developers and researchers who want to run frontier models locally without enterprise infrastructure. The project has gained 48,000+ stars and is trending at 800+ stars per day. The one-step setup detects your hardware and recommends a model that fits; you answer one question and the script handles the rest.

Go deeper

Get the latest AI updates in the Tech AI Magazine app
iPhone · Android