What is server?

Triton Inference Server is software from NVIDIA that serves AI models over a network, letting applications send requests to a model and receive predictions in response. The problem it solves is common in production AI deployments: you have a trained model, and you need a reliable, high-performance way for other software to use it. Triton acts as that serving layer, handling the networking, queuing, batching, and scheduling so your application does not have to. One notable feature is that Triton works with models from many different training frameworks. Whether a model was built with PyTorch, converted to the ONNX format, optimized with NVIDIA's TensorRT, or written as a Python script, Triton can serve it. This matters in organizations where different teams use different tools: Triton becomes a common serving interface regardless of how each model was trained. Performance is a central focus. When many requests arrive at once, Triton can group them into batches and run them together on a GPU, which is much faster than processing each request individually. It also supports running multiple model instances at the same time to use all available hardware. For stateful models that need to track context across multiple requests, such as voice recognition systems, it provides sequence batching to keep related requests together. The server exposes its API using standard HTTP/REST and gRPC protocols, following a community specification called KServe. There is also a C API for embedding Triton directly into an application without the network layer, which suits edge deployments where low latency is important. Deployment is typically done through Docker images pulled from NVIDIA's cloud registry, and the README includes Kubernetes deployment examples for major cloud providers. Triton is open source and available under a BSD license. It is also included in NVIDIA's commercial AI Enterprise platform for organizations that want paid support.