What is simple-evals?

openai/simple-evals — explained in plain English

Analysis updated 2026-06-26

★ 4,486PythonAudience · researcherComplexity · 2/5LicenseSetup · moderate

Mindmap

mindmap
  root((simple-evals))
    What it does
      AI benchmark runner
      Reproduce OpenAI scores
      Multi-model comparison
    Benchmarks Included
      MMLU academic subjects
      GPQA graduate science
      MATH problem solving
      HumanEval code gen
      SimpleQA factual accuracy
    Design Choice
      Zero-shot prompts only
      No elaborate setups
      Plain instructions
    Status
      No new updates post July 2025
      Reference use only
    Audience
      AI researchers
      Model developers

mindmap root((simple-evals)) What it does AI benchmark runner Reproduce OpenAI scores Multi-model comparison Benchmarks Included MMLU academic subjects GPQA graduate science MATH problem solving HumanEval code gen SimpleQA factual accuracy Design Choice Zero-shot prompts only No elaborate setups Plain instructions Status No new updates post July 2025 Reference use only Audience AI researchers Model developers

Click or tap to explore — scroll the page freely

What do people build with it?

USE CASE 1

Reproduce OpenAI's published benchmark scores for GPT-4o or o1 using the exact same methodology they used internally

USE CASE 2

Compare your own fine-tuned model against GPT-4 and Claude on MMLU and GPQA using identical zero-shot prompts

USE CASE 3

Run HumanEval code generation benchmarks on a new open-source model to measure its coding accuracy

USE CASE 4

Study the zero-shot prompting methodology used in modern benchmark evaluations and understand why it differs from older few-shot approaches

What is it built with?

Python

How does it compare?

	openai/simple-evals	ace-step/ace-step	lennylxx/ipv6-hosts
Stars	4,486	4,487	4,488
Language	Python	Python	Python
Setup difficulty	moderate	hard	easy
Complexity	2/5	4/5	1/5
Audience	researcher	developer	general

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · moderate Time to first run · 30min

Requires API keys for each model provider you want to benchmark, the repo is no longer actively maintained as of July 2025.

MIT License, use freely for any purpose, including commercial use, with attribution.

So what is it?

Simple-evals is a lightweight Python library from OpenAI for testing how well language models perform on a set of standardized benchmarks. OpenAI published it to show the methodology behind the accuracy numbers they report when releasing new models, so others can see exactly how those scores are produced. The library runs models through several well-known tests used in the AI research community. These include MMLU, which covers a wide range of academic subjects, GPQA, a set of difficult graduate-level science questions, MATH, which tests mathematical problem-solving, HumanEval, which tests code generation, and SimpleQA, which checks factual accuracy on short questions. The repository also contains reference implementations for HealthBench, BrowseComp, and additional benchmarks. A key design choice in this library is using zero-shot prompts with simple plain instructions rather than elaborate setups. The README explains that some older evaluation methods used extra context or role-playing prompts, which were carry-overs from evaluating earlier models and do not reflect how modern instruction-tuned models actually behave in practice. The repository includes a large results table comparing many models from OpenAI and other providers across these benchmarks, giving a reference point for how different systems compare on the same tests. As of July 2025, OpenAI announced the repository will no longer be updated for new models or benchmark results. It continues to exist as a reference for the three benchmarks mentioned above, and the code can still be used to run evaluations, but active maintenance has stopped. It is intended for researchers and developers who want to reproduce or study published benchmark numbers.

Copy-paste prompts

Prompt 1

Using the openai/simple-evals library, write code to run the MMLU benchmark on gpt-4o and print accuracy scores broken down by subject category.

Prompt 2

How do I add a new model provider to simple-evals so I can benchmark a local LLaMA model on the same GPQA tasks OpenAI uses?

Prompt 3

Show me how to run the HumanEval benchmark from simple-evals on Claude claude-sonnet-4-6 using the Anthropic API.

Prompt 4

Using simple-evals, compare GPT-4o and a custom fine-tuned model on SimpleQA factual accuracy and output a results table.

Prompt 5

Explain the difference between the zero-shot prompting approach in simple-evals and older few-shot evaluation methods, and show me the actual prompt template used for MATH.

Frequently asked questions