What is product-evaluation-harness?

This project is a tool for comparing how well different AI language models summarize text, aimed at product managers who need to decide which AI model to actually use in a real product. Instead of just eyeballing outputs and guessing which one seems better, it runs a structured pipeline that scores each model's summaries against a fixed set of test cases and produces statistics you can defend to a team or stakeholder. The pipeline works in stages: it runs a set of prepared test questions through different AI models, uses another AI model as a judge to score the outputs against a weighted scorecard covering accuracy, made-up information, formatting, tone, and conciseness, then aggregates those scores into statistics and a visual dashboard. It can also run a formal comparison between exactly two models, calculating whether one is meaningfully better than the other using standard statistical methods, rather than just comparing raw averages that could be due to chance. The dashboard, built with the Streamlit framework, shows a results table, side-by-side model comparisons with charts, a breakdown by test category and difficulty, a chart plotting quality against cost so you can see which model gives the best value, and a detailed viewer for failed test cases that highlights exactly what went wrong line by line. You can try it immediately using generated fake data with no setup beyond installing a few Python packages, or connect it to real AI provider accounts using your own API keys to evaluate real models. The scoring rules and test cases are stored in separate files, so you can add your own criteria or test questions without touching the underlying code. This is aimed at product managers, and to a lesser extent engineers, who need a rigorous, repeatable way to decide which AI model is worth paying for in a production product, rather than relying on gut feeling.