Generate long-form Russian or English text from a prompt using a 100B-parameter open-weight model
Run batch conditional text generation from a file of input prompts using greedy or sampling decoding
Study Yandex's bilingual pre-training data pipeline and filtering methodology for large language models
Use the model as a research baseline for Russian-language NLP tasks that need a large open-weight model
| yandex/yalm-100b | nabla-c0d3/sslyze | reinderien/mimic | |
|---|---|---|---|
| Stars | 3,756 | 3,756 | 3,754 |
| Language | Python | Python | Python |
| Setup difficulty | hard | easy | easy |
| Complexity | 5/5 | 2/5 | 2/5 |
| Audience | researcher | ops devops | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires multiple high-end GPUs with ~200 GB total GPU memory and 200 GB disk space just to load the model weights.
YaLM 100B is a large text-generating neural network built by Yandex and released for public use by developers and researchers. It works similarly to other GPT-style models: given some text as input, it predicts and outputs the next words. The model can handle both English and Russian, reflecting the bilingual makeup of the data it was trained on. Training this model was a significant undertaking. Yandex ran it on 800 high-end graphics cards for about 65 days, processing roughly 1.7 terabytes of text drawn from web pages, books, news, social media, and Wikipedia. About a quarter of that data came from an English dataset called The Pile, and the rest was Russian text carefully filtered and deduplicated to remove junk, repetitive content, and low-quality pages. Using the model requires serious hardware. The weights alone take up 200 gigabytes of disk space, and running inference requires multiple graphics cards totaling around 200 gigabytes of GPU memory. The repository includes shell scripts to download the weights, pull a pre-built Docker container, and start generating text without having to configure the environment from scratch. Once set up, you can interact with the model in several ways: type prompts directly from the command line for immediate responses, feed it a file of inputs for conditional generation using sampling or greedy decoding, or let it generate text freely without any prompt at all. Each mode corresponds to a ready-made example script in the repository. The code here is not the original training code. It is a lightly modified version of an example from the DeepSpeed project, adapted just enough to load and run Yandex's trained weights. The model weights and a companion vocabulary file are hosted on Hugging Face and can be downloaded via the included script or by cloning the Hugging Face repository directly. The model is released under the Apache 2.0 license, which allows both research and commercial use.
YaLM 100B is a 100-billion-parameter text-generation model from Yandex, released under Apache 2.0, that generates English and Russian text given a prompt, but running it requires serious multi-GPU hardware.
Mainly Python. The stack also includes Python, DeepSpeed, Docker.
Apache 2.0, use freely for any purpose including commercial, as long as you keep the copyright notice.
Setup difficulty is rated hard, with roughly 1day+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.