What is haru?

Haru is infrastructure software for companies that run their own large language models on GPU servers instead of relying entirely on outside AI providers. It sits above the raw cloud provisioning tools and gives a product's backend one small, consistent interface for managing a fleet of GPUs, no matter which cloud provider or region those GPUs actually live in. Its main job, at least for now, is keeping AI inference available even when a server or region goes down, while wasting as little GPU capacity as possible. One GPU domain is active and actually answers requests. A second, standby domain keeps the same AI model loaded but puts the inference server into a low power sleep state, moving the model's weights out of GPU memory into regular system memory. That freed up GPU memory is then used to run a lower priority model fine tuning job in the background while the standby domain waits. If the active domain fails, Haru wakes the standby domain, which is fast since the model just needs to move back from system memory rather than reloading from disk, stops the background training job, runs a quick test request against every model to confirm things work, and then switches which domain receives real traffic. If anything goes wrong partway through that process, live traffic never gets rerouted to a domain that is not actually ready. Haru is built as a layer on top of two existing tools, SkyPilot and SkyServe, which handle the lower level work of provisioning cloud GPU machines and running inference services across providers. Haru itself does not talk to cloud provider APIs directly. The project stores its state in a Postgres database, using Neon as its primary supported provider, and every change of state happens as a single database operation so that competing background processes cannot corrupt it. The codebase is organized as several packages: shared data schemas, the core decision making logic, the database layer, provisioning drivers for SkyPilot and SkyServe, a control server with an OpenAI-compatible chat API, and a supervisor program that runs on each GPU machine. This is backend infrastructure meant for teams already running self-hosted AI models, not an end-user application. The README does not state a license.