Run small AI language models like GPT-2 locally without needing a giant framework.
Build custom machine learning tools that need low-level tensor math in C++.
Experiment with quantized models to reduce memory and compute requirements.
Use it as the underlying engine that powers projects like llama.cpp or whisper.cpp.
| kenneth-ge/ggml-xrt | achanana/mavsdk | alange/llama.cpp | |
|---|---|---|---|
| Stars | 0 | — | 0 |
| Language | C++ | C++ | C++ |
| Last pushed | — | 2024-05-20 | — |
| Maintenance | — | Dormant | — |
| Setup difficulty | moderate | moderate | moderate |
| Complexity | 3/5 | 4/5 | 4/5 |
| Audience | developer | developer | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires a C++ build toolchain, CMake, and a Python virtual environment to build the examples.
ggml-xrt is based on ggml, a library that gives programs the low-level building blocks needed to run machine learning models, written in C plus plus. It is designed to work across many kinds of computer hardware and to run models efficiently by using integer quantization, a technique that shrinks how much memory and processing a model needs. According to the README, this codebase does not depend on any outside third party libraries, and it is built so that once a program is running, no additional memory needs to be set aside during that run. The library also includes support for automatic differentiation, which is the underlying math needed to train models rather than just run them, along with two specific optimization methods called ADAM and L-BFGS that are commonly used when training models. The README notes that active development on this project is happening across related projects called llama.cpp and whisper.cpp, both of which are well known tools for running AI language and speech models on ordinary computers. To build the project, you clone the code, set up a Python virtual environment and install its dependencies, then use the CMake build system to compile it. The README gives an example that downloads a small GPT-2 model, a type of AI text generation model, and runs it directly from the command line to generate example text. The README is quite short. It does not describe how the code is organized beyond pointing to an examples folder, and it does not go into detail about how to use ggml inside your own project beyond the sample programs. Anyone wanting to understand the library in depth would likely need to look at the linked llama.cpp or whisper.cpp projects, or the ggml introduction and file format documentation the README links to. This project is aimed squarely at developers already comfortable with machine learning concepts and C plus plus, not at people looking for a beginner friendly tool.
A C++ library providing the low-level math and memory tools needed to run and train machine learning models efficiently on many kinds of hardware.
Mainly C++. The stack also includes C++, CMake, Python.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
Verify against the repo before relying on details.