What is lingua?

Meta Lingua is a research-focused codebase from Meta for training and running large language models, which are the AI systems behind tools like chatbots and text generators. The project is designed to be minimal and easy to modify, so that AI researchers can experiment with different model designs, training methods, and datasets without fighting through layers of complex infrastructure. The codebase is built on PyTorch, a widely used Python library for machine learning. It includes components for defining model architecture, loading and shuffling training data, distributing training across multiple graphics cards, managing checkpoints so training can be resumed after interruption, and measuring training speed. These components are kept separate and simple so that a researcher can swap one out or modify it without breaking the rest. The project includes several example applications that show how the components fit together. One trains a standard language model using the Llama architecture. Others demonstrate alternative model designs including Mamba, Hawk, minGRU, and minLSTM, which are different approaches to handling sequences of text that some researchers are exploring as alternatives to the standard transformer design. The README includes benchmark results showing how these different architectures compare on reasoning and knowledge tasks at the 1 billion and 7 billion parameter scales. Setting up Meta Lingua requires access to a machine with one or more NVIDIA GPUs and a compute cluster managed by SLURM, which is common in academic and industrial research settings. The setup scripts handle creating the Python environment and downloading training data from Hugging Face. This is a tool aimed squarely at machine learning researchers and engineers, not at end users or application developers.