What is swe-interact?

SWE-Interact is a benchmark, a set of test tasks used to measure how well AI coding agents perform, rather than a tool you would use to build your own software directly. It contains 75 tasks that test coding agents on multi-turn software engineering work, meaning the agent has to go back and forth with a simulated user over several exchanges instead of getting one instruction and producing one answer. The goal is to evaluate agents in a setting closer to how a real developer actually works with feedback and follow-up requests, rather than a single isolated coding request. The repository holds the task data itself along with example run configurations for a system called Harbor, which is the framework used to actually execute these benchmark tasks. To use it, you install Harbor from its own separate repository, and set up Modal, a service used to provide sandboxed environments where the tasks run safely. Running a benchmark requires setting several credentials in an environment file, including API keys for the AI models involved. One model plays the role of the simulated user asking for changes, and a separate model is used to grade the agent's work against a rubric. The README lists several ready-made run configurations for different coding agents and models, including Codex, Claude Code paired with different Claude models, Gemini through OpenCode, and Kimi, each needing its own specific API key set in the environment file. Once configured, you launch a benchmark run with a single shell command from the repository root, and results are written to a results folder. There is also a simpler single-turn version of the benchmark included for baseline comparison. The project accompanies a research paper describing the motivation for reimagining these coding benchmarks around realistic, user-driven, multi-turn sessions rather than one-shot tasks, and the README asks that the paper be cited if the benchmark is used in research.