What is llm-randomness-benchmarks?

This project is a benchmark that tests how well small AI language models can generate real German words when asked to, running entirely on a Mac with Apple Silicon using a local inference library called MLX. Each model is asked, one thousand times, to give a German word at a specific position with no context, and the results are checked against a real German dictionary to see how many are actual valid words and how varied they are. The test is designed to be difficult on purpose. A model can fail in two different ways that the scoring catches separately: producing junk output that is not a real German word at all, such as English filler text or the model's own internal tokens, or collapsing into repetition, where it gives the same valid word over and over instead of a range of different ones. Validation uses a real spelling dictionary that understands German compound words and grammar, and capitalization is checked too, since German nouns must be capitalized to count as valid. The headline score combines both failure modes into one number: the percentage of distinct valid words out of all one thousand attempts, so a model cannot score well just by repeating one correct word constantly. Several supporting numbers are also tracked, including how many outputs were valid dictionary words at all, how many were unique, and average word length. Running it requires Apple Silicon hardware and a recent Python version, along with a command line tool that generates words for each configured model and then compares the results. The included results table shows no tested model scored above 56 percent, and reveals that whether the prompt is asked in German or English changes performance more than switching between different models does. Adding an extra system prompt on top made every single model perform worse, with one collapsing almost entirely into repeating the word "german" hundreds of times. The project has 0 stars and is written in Python.