What is kortex?

Kortex is an inference engine for running large language models, written from scratch in Rust. Its main idea is to treat inference as a streaming problem across the whole memory hierarchy, moving data from NVMe storage to RAM to video memory to the GPU's compute units. This lets it run models that are much larger than a single GPU's video memory would normally allow. On a consumer GPU with 20 GB of video memory, the project reports running a 70 billion parameter Llama model at about 2 tokens per second, and a fully loaded 30 billion parameter mixture of experts model at around 160 tokens per second, which the README says is faster than the popular llama.cpp project on the same hardware. Kortex is not built on top of llama.cpp. Its GGUF file parser, tokenizer, quantized math kernels, GPU backend, and streaming pipeline are all original code. llama.cpp is used only as a reference to check that Kortex produces identical output tokens. The project currently only works on Windows, because its streaming path relies on Windows specific direct disk access. The part of the code that runs fully in video memory is built with a portable graphics library and should be easier to move to other systems, but a Linux version is only a planned future step, not something available now. Kortex can load GGUF model files for the Llama 3.x and Qwen3 families, supports several quantization formats that shrink model size, and offers greedy decoding, temperature based sampling, and a chat template mode. It also supports streaming layers from disk when a model does not fit in video memory, and speculative decoding, where a small draft model proposes tokens that a larger model then verifies. There is a benchmarking command and a verification command for checking output correctness. Some things it cannot do yet: it does not support streaming for mixture of experts models, which must currently run fully loaded in video memory, it has no Linux support, and it has no server mode, so it only runs as a single command line tool rather than something with an always on API. The README frames Kortex as most useful for the case where a model is too big for video memory, while suggesting llama.cpp for models that fit. Building it requires stable Rust.