What is pruned-bpe-tokenizer?

This project implements a variation of a common text processing technique used to prepare text for AI language models, called Byte Pair Encoding, or BPE for short. BPE breaks text into small reusable chunks, called tokens, that a model reads and generates. This project's twist, called Pruned BPE, tries to make better use of a limited token vocabulary by separating the tokens a model actually needs to see from tokens that were only useful as stepping stones during training. Normal BPE training repeatedly merges the most common pair of adjacent symbols in a large body of text, building up longer and longer tokens, and it keeps every token it ever learned in the final vocabulary. The problem is that many of those learned tokens are only useful as intermediate steps toward building longer, more common tokens, and rarely appear on their own in real text, wasting space in the final vocabulary. Pruned BPE keeps the same training process but afterward analyzes how often each token actually appears and splits them into two files, one of frequently used tokens the model will see, and one of rarely used tokens kept only internally to preserve how longer tokens were built. This frees up room in the visible vocabulary for other, more useful tokens. The repository includes several trainer scripts written in Python and Cython, including single threaded and multi core parallel versions, plus a pure Python version, and a matching tokenizer script to use the trained vocabulary afterward. Training supports checkpoints so you can pause, adjust settings, and resume, and the README describes a two stage training process for when you need an exact final vocabulary size. Two large training text collections are provided, combining English, Chinese, source code, and dozens of other languages, drawn from public datasets, for testing and reproducing the algorithm. The project requires Python 3.10 or newer and is released under the Apache 2.0 license.