Compare how several locally installed Ollama models respond to the exact same prompt.
Test one model's consistency by running the same prompt multiple times at a set temperature.
Automate model comparisons in a script using command-line flags instead of interactive prompts.
Review saved response files with timing and token count data to judge speed versus quality.
| ulyssestenn/omt | asimons81/hermes-dreaming | baskduf/harness-starter-kit | |
|---|---|---|---|
| Stars | 40 | 40 | 40 |
| Language | Python | Python | Python |
| Setup difficulty | easy | moderate | easy |
| Complexity | 2/5 | 3/5 | 2/5 |
| Audience | developer | developer | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires Python 3.7 and a running Ollama instance, uses only the standard library, no pip packages.
Ollama Model Tester is a simple command-line script that helps you compare AI models running locally on your machine. Ollama is a tool that lets you run AI language models (like Llama or Gemma) on your own computer rather than sending data to a cloud service. This script sits on top of Ollama and makes it easy to test how different models respond to the same question. The basic use case is: you have a task in mind, you want to know which of your installed models handles it best, so you run the same prompt through several models and save all the responses to files you can read and compare at your leisure. You can also run the same prompt through one model multiple times to see how consistent or varied its answers are, which is controlled by the temperature setting (higher temperatures produce more varied responses). Running the script asks you a few questions interactively: which model to use, what prompt to send, how many times to run it, and what temperature to use. All of those questions can also be answered via command-line flags, making it straightforward to script or automate. Results are saved in a folder named after the first few words of your prompt. All responses to the same prompt, regardless of which model generated them, land in the same folder. Each model gets its own file inside that folder containing its responses along with timing data and token counts that Ollama reports. The script requires no installation beyond Python 3.7 and a running Ollama instance. It uses only Python's built-in standard library, so there are no packages to install with pip. The README does not mention a license.
A command-line script that runs the same prompt through multiple local Ollama models and saves the responses side by side for comparison.
Mainly Python. The stack also includes Python, Ollama.
Setup difficulty is rated easy, with roughly 5min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
Verify against the repo before relying on details.