What is gt-bench?

GT-Bench is a research benchmark for testing and improving how AI language models reason about game theory. Game theory is the study of strategic decision making, or how rational players choose actions when the outcome depends on what others do too. The core task is simple and precisely verifiable: given a two player payoff matrix, a grid showing what each player earns for every combination of choices, the model must identify all pure strategy Nash equilibria and briefly explain its reasoning. A Nash equilibrium is a stable outcome where neither player would benefit by switching their choice on their own. Exact scoring means there is no subjective grading, so the benchmark can cleanly measure whether fine tuning a model actually improved that specific skill. The repository generates training, validation, and test datasets, then evaluates a model before and after fine tuning to compare scores. The published experiment uses three training set sizes, 250, 1000, and 5000 examples, on a large language model to measure how data volume affects improvement. Beyond the narrow two player task, a broader suite covers mixed strategy equilibria, larger game matrices, sequential game reasoning solved by backward induction, game descriptions written in plain English that must be mapped to a payoff structure, and repeated round prisoner's dilemma scenarios. Each of these families has its own generator and scoring script, and scoring stays exact throughout. The project is written in Python and was created by a UC Berkeley master's student as part of a Thinking Machines Lab Tinker research grant. It includes scripts to generate datasets, run a baseline model, fine-tune with LoRA through the Tinker service, and score predictions, along with an arXiv-ready research paper describing the results. The full README is longer than what was shown.