What is gguf_inference?

This project is a standalone command line and notebook tool for running image generation models that have been quantized into the GGUF file format, built on top of the ComfyUI GGUF node. It is meant as an alternative to running a full interface like AUTOMATIC1111 or ComfyUI when you only have access to a terminal, a remote server, or a JupyterLab notebook environment. The team built it as part of their own inference engine to keep their infrastructure costs down while supporting more model types over time. The README is upfront that the project's code was planned by one AI model and written by another, with no autonomous agents involved, just a human directing chat based coding tools. To use it, you install PyTorch along with a handful of Python libraries such as transformers, accelerate, safetensors and bitsandbytes, then clone the repository and download a quantized model file, for example a Klein 4B model in one of its GGUF quantization levels. A meta only flag lets you inspect a model file's quantization details before running it. From there, a single command line call generates an image, letting you set the model type, VAE source, text encoder mode, prompt, guidance scale, step count, resolution and output file. Currently supported models are Flux 2 Klein in both its 4B and 9B sizes, and Flux 2 Dev, along with their fine tuned variants. The smallest supported model still needs at least 8GB of video memory to run. The README lists no other known issues at this time. Planned future work includes adding LoRA support, support for older Stable Diffusion model families, and further GPU usage optimizations.