What is ai-evaluation-tools?

This repository is a large curated directory of software for evaluating AI systems, covering more than 300 open source and commercial tools organized into 29 categories. It is not a piece of software you install and run, but a reference list, similar in spirit to other "awesome list" style repositories on GitHub, aimed at people who need to figure out which existing tool fits a specific evaluation problem for a large language model, a retrieval based application, or an AI agent. The list is organized around what a reader is trying to do rather than just alphabetically. A table near the top maps common goals, such as testing an AI agent, tracing a production LLM app, checking for jailbreaks, detecting hallucinations, scoring outputs with another model acting as a judge, comparing foundation models, or benchmarking inference speed, to the relevant section of the list. From there the document expands into dedicated sections covering full evaluation platforms, code based evaluation frameworks, cloud provider services, observability and tracing tools, red teaming and guardrail tools, benchmark suites for coding and agents, multilingual and multimodal evaluation, synthetic data generation, and human annotation tools, among others. Each entry links back to its original source, whether that is a GitHub repository, an official product page, or a research paper, so a reader can verify claims rather than take the list's word for it. Entries are also marked with simple icons showing whether something is open source, open weights, an open core product with a paid layer on top, or fully commercial and closed. The whole catalog is also available in machine readable form as JSON and CSV files for anyone who wants to process it programmatically rather than read it as a document. The project describes itself as reviewed monthly and states a last reviewed date of mid July 2026. It is released under the CC0 license, meaning the content is placed in the public domain with no restrictions on reuse. It has 2 stars and is maintained primarily in Python. The full README is longer than what was shown.