Run a very large language model that needs several GPUs working together at once.
Pack many smaller model requests onto shared GPUs to raise hardware utilization.
Manage a cluster of GPU nodes serving models through one unified API.
Monitor cluster status and load or chat with models directly from the command line.
| zimingttkx/quantumflow | greatvishal27-rc/ai-resume_analyzer | jsingletonai/dejavu | |
|---|---|---|---|
| Stars | 56 | 56 | 56 |
| Language | Python | Python | Python |
| Setup difficulty | hard | moderate | easy |
| Complexity | 5/5 | 2/5 | 2/5 |
| Audience | ops devops | vibe coder | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires a cluster with multiple GPUs and Python 3.10 or higher to see the full benefit of the scheduling features.
QuantumFlow is a Python framework for scheduling AI inference workloads across a cluster of machines, aimed at running large language models efficiently across many GPUs. The README describes it as a next generation distributed inference platform that treats AI inference tasks the way Kubernetes treats pods, deciding automatically where each task should run instead of requiring someone to assign it to a machine by hand. The project's core feature is intelligent scheduling. It supports multiple scheduling strategies: Gang scheduling, which reserves an entire block of GPUs at once for very large models that need every GPU working together with minimal communication overhead, Pack scheduling, which is better suited to smaller models that can be packed efficiently onto fewer resources, and an adaptive strategy that picks the best approach automatically based on the workload. QuantumFlow also supports multiple inference backends behind one unified interface, including Hugging Face and vLLM, which the README marks as already working, alongside TGI and SGLang, which are listed as planned. Support for domestic Chinese AI accelerator hardware, specifically Ascend NPUs, is also listed as planned rather than finished. The system architecture has four layers: a gateway layer offering REST and gRPC APIs plus a Python SDK and command line interface, a scheduler layer implementing the Gang, Pack, and adaptive strategies alongside a priority queue, a cluster management layer handling node registration, service discovery, and health monitoring, and a worker pool layer that actually runs the models on hardware such as A100, 4090, and H100 GPUs. To get started, you clone the repository and run pip install to install it in editable mode, then start the service either with a bundled script or by running the CLI's serve command, after which a web interface becomes available locally. The command line tool also supports checking cluster status, listing available models, loading a specific model, and chatting with or generating text from a loaded model directly from the terminal. The project is released under the Apache 2.0 license, a permissive open source license.
A Python framework that schedules AI model inference jobs across a GPU cluster, similar to how Kubernetes schedules pods.
Mainly Python. The stack also includes Python, FastAPI, vLLM.
Use freely for any purpose, including commercial use, as long as you keep the copyright and license notice.
Setup difficulty is rated hard, with roughly 1h+ to a first successful run.
Mainly ops devops.
This repo across BitVibe Labs
Verify against the repo before relying on details.