Distill a large teacher model into a smaller, faster student model.
Train across multiple machines using a custom distributed parameter-splitting implementation.
Serve many simultaneous conversations efficiently using a paged key-value cache and continuous batching.
Speed up generation with speculative decoding that drafts and verifies multiple tokens at once.
| zengxiao-he/tessera | eadmin2/jarvis_ai | greatvishal27-rc/ai-resume_analyzer | |
|---|---|---|---|
| Stars | 56 | 56 | 56 |
| Language | Python | Python | Python |
| Setup difficulty | hard | hard | moderate |
| Complexity | 5/5 | 4/5 | 2/5 |
| Audience | researcher | developer | vibe coder |
Figures from each repo's GitHub metadata at analysis time.
Custom Triton/CUDA kernels and multi-machine distributed training require a compatible GPU setup.
Tessera is a project that builds the full pipeline for training a small language model and then running it efficiently, all from scratch. The central idea is knowledge distillation: you start with a larger, more capable model (the teacher) and train a smaller model (the student) to reproduce its behavior. The result is a model that runs much faster and cheaper while trying to preserve as much of the teacher's quality as possible. On the training side, the project implements its own version of the distributed training technique that splits a model's parameters across multiple processors, along with the matching optimizer. This lets training run across multiple machines without needing a pre-built framework to manage the coordination. The README notes this implementation is verified to produce numerically identical results to single-machine training. The inference (serving) side of Tessera is where much of the complexity lives. It includes a paged key-value cache, which is a memory management technique borrowed from operating systems that lets the server handle many simultaneous conversations without wasting GPU memory. There is also a continuous batching scheduler that keeps the hardware busy by constantly recombining active requests, and speculative decoding, which speeds up generation by having the model draft several tokens at once and then verify them in a single pass. For raw speed, the project includes custom GPU kernels written in Triton and CUDA, which are low-level languages for programming graphics cards directly. These replace standard library functions for attention, normalization, and matrix multiplication with hand-tuned versions. On hardware without a compatible GPU, the code falls back to standard PyTorch operations automatically. The network-facing layer is written in Rust and connects to the Python engine, handling incoming HTTP requests and managing back-pressure when the system is near capacity. There is also a separate reimplementation of the model's core logic in JAX, used purely as an independent check that the PyTorch version produces correct numbers. The project is licensed under Apache 2.0.
Tessera is a from-scratch pipeline for distilling a small language model from a larger one and serving it efficiently with custom GPU kernels.
Mainly Python. The stack also includes Python, PyTorch, Rust.
Apache 2.0: free to use, modify, and distribute, including commercially, with attribution and no warranty.
Setup difficulty is rated hard, with roughly 1day+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.