What is minbpe?

minbpe is a small, clearly written Python library that implements the tokenization algorithm used by modern AI language models like GPT-4. Tokenization is the process of splitting text into smaller pieces called tokens before an AI model can process it. The specific algorithm used here is called Byte Pair Encoding, or BPE, which works by repeatedly finding the most common pair of bytes or characters in a text and merging them into a single new token. Over many passes, this builds up a vocabulary of common words and word fragments. The library is written by Andrej Karpathy as a teaching tool. The code is short and heavily commented, meant to be read and understood rather than used at production scale. It shows how the tokenization that powers tools like GPT-4 actually works under the hood, step by step. There are three tokenizer implementations included. The basic one runs BPE directly on text in the simplest possible way. The regex tokenizer adds a preprocessing step that splits text into categories like letters, numbers, and punctuation before merging, which matches the approach OpenAI introduced with GPT-2 and still uses in GPT-4. The GPT-4 tokenizer is a thin layer on top of that which reproduces the exact tokenization output of GPT-4, and the README shows a side-by-side comparison with OpenAI's own tiktoken library to confirm they match. All three tokenizers support the same three operations: training on a body of text to build a vocabulary, encoding text into a list of token numbers, and decoding a list of token numbers back into text. You can also save and load trained tokenizers to disk. The repository is primarily a learning resource. If you want to understand what a language model actually sees when you give it a prompt, or how a tokenizer vocabulary gets built, this code gives you a minimal working example you can run and modify.