What is erdosbench?

ErdosBench is a benchmark designed to test whether AI systems can act like useful assistants for mathematical research. Rather than checking if a model can solve simple math problems, it looks at whether a model can find obstructions to a claim, apply known theorems correctly, spot gaps in a proof, run small scale experiments, and avoid overstating how novel its findings are, using problems written in the style of the mathematician Paul Erdos. This particular repository is a public smoke test version of the full benchmark, released alongside a June 2026 whitepaper. It intentionally contains only 14 of the full 226 problem set, and it does not include the private development, holdout, or answer key data that the complete benchmark uses. The 14 included problems had already been shared publicly in earlier research notes from the ulam.ai team. The repository includes the 14 problem statements in a JSONL data file, a schema describing the required format for a submitted result file, a script that turns the problems into a ready to use prompt for blind evaluation, and a second script that validates a results file has the correct problems and required fields. The validator only checks that the format is correct, it does not check whether the mathematical answers are actually right. To run an evaluation, someone generates a prompt from the problem set, gives that prompt to an AI model or research agent, saves the model's output, and validates the format. Each result must include fields such as a verdict describing the outcome, for example solved, partial progress, or no progress, along with supporting evidence and any remaining gaps in the reasoning. The README is explicit that this public repository is only a smoke test, not the full private evaluation set, and should not be used as evidence for private leaderboard comparisons.