What is esp32-ai?

This project runs a small language model on a microcontroller that costs about $8. The model has 28.9 million parameters and generates text at roughly 9 tokens per second, writing each word to a small screen wired to the chip. Nothing gets sent to a server. Everything runs on the device itself. The previous language model people ran on a chip like this had only 260 thousand parameters, so this one holds about a hundred times more. The hard part is memory. The ESP32-S3 chip has only 512KB of fast memory, which normally forces you to use tiny models. The trick here is to stop putting the model in fast memory at all. Most of a language model's parameters sit in an embedding table, which the model reads from rather than computes on. So you leave that 25 million row table in slow flash storage and pull only the few rows each token needs, about 450 bytes. The small part that does the actual work stays in fast memory. You never load most of the model. It just sits in flash and gets sampled a little at a time. This idea is called Per-Layer Embeddings, from Google's Gemma models. The author adapted it to run on a microcontroller instead of a phone or a GPU. As far as they can tell, nobody had tried it on a chip this small. The model was trained on TinyStories, a dataset of short synthetic stories simple enough that a small model can learn to write coherently. It writes short, simple stories and mostly keeps them coherent. It will not answer questions, follow instructions, write code, or know facts. The interest here is the architecture, fitting a large model onto a tiny chip, rather than what a 28.9 million parameter model can say. The repo includes the firmware, wiring steps, training code, and a writeup of the full method and measurements. The author left the messy commit history on purpose, including a bug in their parameter accounting and the corrected result that followed.