What is frontier-finance?

frontier-finance is a grading tool for testing how well an AI system answers finance-related questions. It works with Samaya AI's FrontierFinance benchmark, a public set of finance questions where each question comes with a checklist of rubrics, specific facts or points a good answer should cover. You bring your own AI system's answers, and this tool checks each answer against its rubrics using a panel of AI language models acting as judges, then reports how often each rubric was satisfied. The grading panel typically uses three judge models from different providers, such as Claude, Gemini, and GPT, voting by majority on whether each rubric was met, so no single AI model's bias decides the outcome on its own. If a judge model fails to respond properly, it gets dropped from that vote rather than failing the whole question, as long as at least one judge still answers. The tool is self-contained and talks directly to each provider's official API rather than relying on any internal company framework. Installation is done with a Python package manager, and you install only the judge providers you plan to use, such as Anthropic, OpenAI, or Gemini, as optional extras. The benchmark's question and rubric data is downloadable separately from Hugging Face with a provided script, no login required, and is licensed under Creative Commons Attribution 4.0, meaning it can be reused with proper credit given. To run a grading pass, you set API keys for whichever judge models you are using, write a YAML configuration file pointing at the rubrics file and your own system's answers, and run the grader command. It produces a metrics summary and a detailed per-question breakdown showing how each rubric was judged, split out by categories like whether a rubric was essential or optional. Unit tests are included and can be run without any API keys.