What is toolhound?

Toolhound is a diagnostic tool for figuring out why small AI language models fail at calling external tools or functions. Most benchmarks report a single accuracy number, something like a model getting 71 percent of tool calls right, but that number does not explain why the other calls failed, and the reason matters a lot for deciding what to fix. Toolhound runs entirely on an Apple Silicon Mac using the MLX framework and attributes every single failed tool call to one of four specific causes. Those four causes are: a bug in the chat template that mangles how tool related tokens are formatted, a gap in the framework's parser that misses a call the model actually got right, the model producing output that cannot be parsed as a valid call at all, or the model producing a valid call but choosing the wrong tool or wrong arguments. Only the last two are genuinely the model's fault. The first two are framework or template bugs that should be fixed upstream rather than blamed on the model. The tool separates these by using a lenient parser to check whether any reasonable parser could have rescued the output, while a stricter scorer separately judges whether the answer itself was correct. The README gives real examples from testing several small models: one model appeared unable to handle multi-turn tool calls, but the actual cause was its chat template silently dropping a prior call. Another appeared to produce invalid output, but the real problem was the parser missing a valid array format it used. A third model genuinely picked wrong arguments, which is a real model limitation that changing the parser cannot fix. To use it, you need an Apple Silicon Mac running macOS 14 or newer with Python 3.11, since MLX only runs on Apple's own chips. After cloning the repository and installing dependencies, you run a reliability check across several models, then an attribution command that names the specific cause behind each failure. Reports are written as both machine readable JSON and human readable markdown files, each stamped with the exact hardware, software versions, and model revisions used, so the results can be reproduced. It is licensed under Apache 2.0 and is currently a version 2 pre-release.