Benchmark how well an AI coding agent can build a full emulator under a strict time limit.
Study commit-by-commit transcripts of AI agents building software from scratch.
Test a custom GBA emulator wasm build against an accurate reference for correctness.
Compare AI model performance on a specialized systems-programming task.
| mechanize-work/gba-eval | hannah-wright/saas-landing-page-template | asabeneh/speaksuomi | |
|---|---|---|---|
| Stars | 3 | 3 | 4 |
| Language | HTML | HTML | HTML |
| Last pushed | — | — | 2026-05-29 |
| Maintenance | — | — | Maintained |
| Setup difficulty | moderate | easy | easy |
| Complexity | 4/5 | 1/5 | 1/5 |
| Audience | researcher | vibe coder | general |
Figures from each repo's GitHub metadata at analysis time.
Needs Docker plus git submodules and git LFS to pull the reference emulator and test corpus.
GBA Eval is a benchmark that tests whether AI coding agents can build a working Game Boy Advance emulator from scratch in just 24 hours. The project comes from Mechanize, a company that builds evaluation environments for training and testing large language models, and this repo is a public example of that kind of work. The idea is simple: give an AI agent one day to write a software emulator with WebAssembly support, then grade the result against Mesen2, a well known and highly accurate GBA emulator, used here as the reference. Nine emulators built by different AI models during a May 2026 leaderboard run are published in a companion repository, each with its full commit history and a transcript of the chat log that produced it, so anyone curious can see how the agent worked step by step. Grading is split into three weighted sections. Gameplay replays, which check real button press input and end to end behavior, make up 60 percent of the score. Procedural tests covering the CPU, memory, timers, and DMA account for 20 percent. Audio accuracy, measured by comparing sound output frame by frame against the reference emulator, makes up the final 20 percent. These weights can be changed in a configuration file if someone wants to run the benchmark differently. To actually run a grading pass, Docker is the only required dependency, along with git submodules and git LFS to pull in the reference data and test material. The grading engine itself is written in Rust and can precompute reference data once, then score any candidate wasm file against it. Users can also rebuild the reference emulator from source if they do not trust the bundled version. Licensing is mixed rather than a single blanket license. The project's own code, including the specification, test corpus, and harness, is MIT licensed. Code tied to the Mesen2 reference emulator, including the build script and the compiled reference binary, is GPL-3.0, since it derives from that upstream project. Test ROMs carry their own individual licenses depending on where they came from.
A benchmark that gives AI coding agents 24 hours to build a Game Boy Advance emulator, then grades it against a reference emulator using gameplay, CPU tests, and audio accuracy.
Mainly HTML. The stack also includes Rust, Docker, WebAssembly.
Original code is MIT and free to reuse, but parts tied to the bundled reference emulator are GPL-3.0.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.