What is spark-moe?

spark-moe lets you run a massive AI language model, GLM-5.2, on a single NVIDIA DGX Spark desktop system, and then talk to it through a familiar API. Instead of needing a whole rack of servers, you download roughly 433 GiB of model files, build the engine, and start a local web service that speaks the same protocol as OpenAI's API. Under the hood, the project is a custom inference engine written in C with CUDA kernels tailored to the DGX Spark's specific chip (the GB10). It loads the model in a compressed format called NVFP4 to fit within the hardware's memory, manages a very large conversation context window (up to 262,144 tokens), and exposes endpoints for chat completions, text completions, and embeddings. You can run it directly or inside a Docker container, and the provided scripts handle downloading, building, testing, and serving. This is for developers or researchers who already own or have access to an NVIDIA DGX Spark (specifically the ASUS Ascent GX10 variant) and want to run a large, cutting-edge model locally without relying on a cloud provider. A concrete use case would be a small team building a prototype chatbot or agent pipeline: they could host this on their Spark box, point their existing OpenAI-compatible client code at it, and test against a powerful local model. The project also includes a "stub" fallback mode, suggesting it can be useful for testing API integration logic even when the full model or GPU isn't present. The notable tradeoff here is the tight hardware coupling. The entire engine and its CUDA kernels are purpose-built for the DGX Spark's architecture, meaning this isn't a general-purpose tool you can deploy on just any machine with a graphics card. It is an optimization play, sacrificing portability to squeeze enough performance out of one specific piece of hardware to run a model that would normally require much larger infrastructure. The README does not go into detail on the model's benchmark performance or accuracy compared to other formats.