What is llm-d?

llm-d helps you serve large language models at scale, on Kubernetes, without having to build the infrastructure from scratch. If your team is deploying LLMs and needs them to handle real production traffic efficiently, this project gives you a pre-built path to get there. At its core, it is a distributed inference stack built on top of vLLM, Kubernetes, and an Inference Gateway component. It tackles three main performance problems. First, it routes incoming requests intelligently, using a scheduler that is aware of things like cache state and service-level agreements to pick the best server for each request. Second, it separates the "prefill" stage (processing the prompt) from the "decode" stage (generating the answer), running them on independent instances so each can be optimized independently. Third, it manages a tiered cache of the intermediate computations LLMs produce, storing them locally or across instances to avoid redundant work. The target user is a platform or ML engineering team that needs to run LLMs for a customer-facing application. For example, if you are building an AI-powered customer support tool and need to guarantee low latency across hundreds of concurrent conversations, this framework manages the traffic routing and compute distribution to make that possible. It is built to work across different hardware accelerators, so you are not locked into a specific chip vendor. The project was launched by a coalition of major players including Google, NVIDIA, IBM Research, and Red Hat. It is designed to be modular, meaning teams can swap in their own scheduling logic or caching backends. Some advanced features, like intelligent autoscaling based on traffic patterns and hardware capacity, are noted as planned or in progress. The README does not go deep into the operational requirements for getting it running, but points to a deployer tool and quickstart guide for installation.