What is modelrouter?

ModelRouter describes itself as a control plane for AI inference, meaning it aims to sit between an application and the many different AI model providers it might call, deciding on the application's behalf which provider and model should actually handle each request. Instead of an application hardcoding logic like falling back to a second provider when the first one is down, or switching to a cheaper model when a cost budget is tight, that logic would live in ModelRouter as configurable policy. The README explains the problem this is meant to solve: every team building on language models tends to reinvent the same fragile fallback and cost control logic, scattered across services and often untested until something actually breaks. Providers differ in latency, pricing, rate limits, and capabilities, and that inconsistency tends to leak into application code over time. The project compares itself to how an API gateway centralizes concerns like authentication and retries for regular web services, arguing that AI inference traffic needs the same kind of dedicated layer, with added concerns specific to AI workloads such as token based cost and streaming responses. As of this README, the project is explicitly marked as being in a design phase rather than a finished, working system. It lists an extensive set of intended features, including routing based on cost, privacy, and latency, retry and fallback handling, health checks, caching, observability through metrics and tracing, and Java and Android client libraries, alongside a long list of things it deliberately does not intend to do, such as model training, prompt authoring, or acting as a chatbot. It is built in Java using Spring Boot, and its design assumes supporting infrastructure like Redis, PostgreSQL, and Kafka rather than running as a single standalone process. The project is licensed under Apache 2.0.