What is attention-span?

attention-span is a personal learning project that builds small language models from scratch and documents everything the author learns along the way. It is organized as a collection of training scripts, Jupyter notebooks, and written notes rather than a packaged tool, so it reads more like a study log than a product. The first and main model in the repository is a GPT-124M style model, meaning it has roughly 124 million parameters, built with 12 layers and 12 attention heads. It is trained on TinyStories, a public dataset of short, simple children's stories designed for training small models that can still write coherent narrative text. The notebooks walk through building the model architecture step by step, then take that pretrained model and instruction fine tune it on a synthetic set of instruction and response examples generated from the same TinyStories dataset. The repo includes before and after results from this fine tuning process. Before fine tuning, the model ignored instructions entirely and produced unrelated text. After two epochs of fine tuning on a single A100 GPU, it consistently followed the expected instruction and response format and its ROUGE score, a measure of how closely generated text matches a reference, roughly doubled. The author also notes clear remaining weaknesses, including trouble following multiple required keywords at once and a tendency to repeat similar characters and plots under simple decoding. Beyond the core training script, the project documents a long list of performance optimizations it applies, such as mixed precision training, gradient accumulation to simulate larger batches, and techniques borrowed from Andrej Karpathy's nanoGPT project. The full training run described costs roughly a dollar and a half in cloud GPU time on Google Colab. The full README is longer than what was shown.