What is loss-function-development?

This repository contains a skill for Claude Code (an AI coding assistant) that helps you design better evaluation targets for AI agent loops. The core idea it describes is the difference between a test suite and a loss function. A test suite is a fixed list of checks: once they all pass, you are done. A loss function is a continuous target the agent keeps trying to improve against, more like how machine learning training works, applied to the kind of task-running AI agents that iterate many times to produce a result. The problem the skill addresses is that AI agents are good at finding shortcuts. If you give an agent a finite list of test cases, it can memorize the answers, exploit patterns in your examples, or satisfy the letter of your instructions while missing the point. The skill is designed to help you build an evaluation setup that is harder to game: a large blinded test set the agent cannot see during its run, mechanical scoring instruments that catch constraint violations, and forced variation requirements that stop the agent from just repeating what worked last time. When you invoke the skill with a task description, it analyzes your existing setup, reconstructs or ingests a specification, builds a set of evaluation cases (often by sourcing publicly available reference artifacts), drafts the evaluation harness (scoring scripts, constraint checkers, and probe variants for detecting memorization), and then tries to identify cheats in its own draft before handing you a finished goal.md file. If your agent starts gaming the evaluation mid-run, you can re-invoke the skill in patch mode: it reads the iteration log, identifies which path the agent exploited, and closes it in the evaluation rules rather than in the agent's code. Installation is done by copying the skills folder into your Claude Code skills directory. The repository includes a cheat museum file that documents known exploitation patterns and the evaluation rules that closed them. There are no worked examples included deliberately, because the README explains that a concrete example would cause the skill itself to overfit to that example's shape. Licensed under MIT.