What is dataset-factory?

This project is a tool for building benchmark datasets used to test and evaluate AI agents, meaning software systems that are supposed to complete multi step tasks on their own. Instead of a simple script that spits out random data, it is set up as a full pipeline with several stages: generating records, checking them against strict rules, removing duplicates, filling in coverage gaps, scoring overall quality, and finally packaging everything for release. Each generated record represents one evaluation task, such as a coding problem or a research question, and includes a natural language description, an expected step by step plan, the tools the agent is expected to use, and the criteria for judging success. The tool ships with ten built in categories, things like coding, web research, data analysis, debugging, and document processing, and produces a fixed total of one thousand records split evenly across those categories and across three difficulty levels. New categories can be added without touching any code, simply by dropping in a new configuration file, since each category is defined as a self contained file that the tool automatically discovers. Before generating anything, a validation step checks the configuration itself for mistakes. During generation, duplicate or overly similar records are caught and rejected using three different comparison methods layered together. An independent script, kept separate from the main tool, can then re verify the finished dataset from scratch. Everything can be reproduced exactly by reusing the same random seed value. The tool also generates supporting documents describing the dataset's methodology, its version history, and its overall quality score. This is aimed at researchers and engineers building or evaluating AI agents who need a reliable, well documented, and reproducible test dataset rather than one they build by hand. It is released under the MIT license. The full README is longer than what was shown.