What is evaluator?

NeMo Evaluator helps you figure out how well an AI model actually performs. Instead of guessing whether your model is good at math, coding, safety, or general knowledge, you point it at this tool and it runs a structured test, scores the results, and gives you a report. It is like an automated grading system for large language models. At a high level, it comes with 17 built-in benchmarks covering things like multiple-choice questions, math problems, coding tasks, and safety checks. You run a benchmark from the command line, the tool sends your model a series of test questions, evaluates the answers, and produces a report. You can also define your own custom benchmark if the built-in ones do not fit your needs. One notable feature is its built-in proxy that sits between your AI agent and the model it is talking to. This proxy can cache responses, log token usage, limit how many turns an agent takes, and modify requests on the fly. This is useful if you are testing autonomous agents and want tight control over what gets sent to the model and how much it costs you. This tool is designed for teams building or fine-tuning AI models who need rigorous, repeatable evaluation. For example, if you are deciding between two models for a customer support chatbot, you can run both through the same benchmark and use the comparison feature to see exactly where one outperforms the other. It also includes a quality gate feature that gives you a simple GO or NO-GO decision on whether a new model meets your standards across multiple tests. The project is built to scale from a laptop to large compute clusters. It supports Docker containers for running code safely and can deploy across high-performance computing environments using SLURM, a workload manager common in research environments. It also exports results to popular experiment trackers like Weights and Biases or MLflow, so it fits into an existing AI development workflow.