Run a suite of golden test cases against an AI agent and score five reliability metrics.
Fail a GitHub Actions CI check when an agent's quality regresses from a saved baseline.
Detect flaky agents that give inconsistent answers to the same repeated prompt.
Check whether an agent called the expected tools in the expected order.
| nishanttyagi28/agenteval | 0xustaz/streamgate | a-bissell/unleash-lite | |
|---|---|---|---|
| Stars | 1 | 1 | 1 |
| Language | Python | Python | Python |
| Setup difficulty | moderate | hard | hard |
| Complexity | 3/5 | 4/5 | 4/5 |
| Audience | developer | developer | researcher |
Figures from each repo's GitHub metadata at analysis time.
Requires writing YAML golden test cases and wiring an adapter for the agent under test.
AgentEval is a testing framework built specifically for AI agents, the kind of software that uses a large language model to answer questions or carry out tasks. Where normal automated tests check that code always gives the exact same result, AI agents can behave differently each time even with the same input, so ordinary testing tools do not catch every kind of failure. AgentEval fills that gap. The tool works by running an agent against a set of golden test cases written in YAML, a simple text format for structured data. Each case describes a prompt, the expected correct answer, which tools the agent should use, and how much tolerance is allowed. AgentEval then scores the results on five measures: whether the answer is correct, whether the agent made up unsupported facts or numbers, whether it called the right tools, how fast it responded, and how much the run cost. These scores get compared against a saved baseline from an earlier run, and if quality drops too much the tool can fail a check in GitHub Actions, the automation system many software projects use to test code before it is merged. AgentEval also separates true answer failures from infrastructure problems like a provider outage or a broken evaluator, so those do not unfairly count against the agent's quality score. It includes an opt-in mode for detecting flaky behavior, meaning cases where an agent gives a different answer each time it is asked the same question, and can optionally check whether the agent took the expected sequence of steps to reach its answer, not just whether the final answer was right. Results can be viewed in a Streamlit dashboard, a web based interface for browsing data, showing pass and fail rates at both the whole suite level and for individual test cases. This tool is meant for developers building AI agents who want a repeatable way to catch regressions before shipping changes to production, similar to how automated tests are used for regular code.
A CI testing framework for AI agents that scores correctness, hallucination, tool use, latency, and cost against a versioned baseline in GitHub Actions.
Mainly Python. The stack also includes Python, GitHub Actions, Streamlit.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
Verify against the repo before relying on details.