What is benchmarking?

GC Benchmarking is a command line tool for testing how fast different AI providers respond when you send them the same large language model request. It runs the same model across every provider you have configured with an API key, then measures things like how long it takes for the first word of the reply to appear, total time for the full reply, how many tokens per second come back, how many retries happened, and how often requests failed. By default it is set up to compare a service called General Compute against OpenRouter, another provider that routes requests to different backends, but you can add or remove providers by editing a configuration file without touching any code. To reduce the chance that timing differences are just due to bad luck or caching, the tool interleaves requests between providers within each round, throws away a warm up request before it starts counting, and varies the prompt text slightly so provider side caching does not skew results. If a run gets interrupted partway through, the raw results collected so far are still saved. Setup requires Python 3.10 or newer, a virtual environment, and either an install command or a bundled setup script. You then copy an example environment file and fill in your API keys and base URLs for whichever providers you plan to test. The tool ships with a set of built in workloads that test different prompt sizes, from 256 tokens up to 131,072 tokens, so you can see how a provider handles both small and very large requests. Results are saved as raw and summary CSV files, plus HTML reports with charts, and there is a command to export a finished test session as JSON files for publishing on a separate static website. The README stresses that comparisons are only fair when testing the exact same underlying model, and that OpenRouter's numbers can shift depending on which backend it happens to route through. The authors ask that anyone publishing comparisons involving their own General Compute service contact them first to check the setup is fair. The project is written in Python, has 15 stars, and is released under the MIT License.