What is bonsai-demo?

Bonsai Demo is a repository that lets you run a family of extremely compressed AI language models on your own computer, whether that is a Mac, a Windows or Linux machine with a graphics card, or just a regular CPU. The models come in two flavors: a one bit family called Bonsai, and a slightly less compressed two bit family called Ternary-Bonsai, each available in several sizes from 1.7 billion up to 27 billion parameters. The newest and largest version, Bonsai 27B, is also the first in the family that can understand images. You can send it photos, screenshots, or PDF files and ask questions about them, not just plain text. It also supports agentic tool calling, meaning the model can request that external tools be run and receive their results back in the same conversation, and it works as a reasoning model where you can choose how much extra thinking effort it spends on a given request. It can also hold very long conversations, well beyond a quarter million tokens. What makes this project notable is how small the compressed models are. The one bit version of Bonsai 27B is squeezed down to a little over one bit per weight, small enough to run entirely on a modern iPhone without needing to swap data in and out of memory. The two bit Ternary version trades a bit of that compactness for higher quality, and is the version this demo downloads and runs by default. Getting started takes just two commands. Cloning the repository and running a single setup script downloads the necessary model files and supporting programs automatically, and a second script starts a local web server you can chat with, complete with support for images and tools, right in your browser. Environment variables let you pick a different model family or size before running the setup, and downloading the largest model currently requires a free Hugging Face account token since those particular files are not yet fully public. The underlying model formats are also being merged into the widely used llama.cpp project, with the one bit format already fully supported there and the two bit format working across several hardware backends. A folder of community submitted speed benchmarks is included so you can compare how the models perform on different machines.