What is knowledge-cutoff?

knowledge-cutoff is a benchmark that tries to measure what a language model actually knows about the world, month by month, rather than trusting the cutoff date a model's makers advertise. The idea is to ask the model about real world events spread across recent months and see how many it gets right for each month. Since a model's knowledge does not stop sharply on one date but instead fades out gradually, plotting the per-month accuracy reveals roughly where that fade begins. The project is careful to avoid a few ways this kind of test can mislead. Some events are predictable in advance, like an expected election result, so a model could seem to know about them without ever having been trained on the outcome, and the benchmark tags each event by how predictable it was and focuses its main estimate on the least predictable ones. It also separates answers into correct, confidently wrong, or a model simply saying it does not know, since treating an honest, I do not know response the same as a wrong answer would distort the results. To catch a model that just guesses someone is dead for anyone it does not recognize, the dataset includes control questions about people who are definitely alive and about events that never happened at all. Prompts also avoid mentioning today's date so the model cannot infer the answer from context. Using it involves installing with the uv Python tool, adding API keys for whichever AI providers you want to test such as Anthropic, OpenAI, or others, then running a sequence of commands that run the test, grade the answers with an AI judge, and score the results into a cutoff estimate. Results are also published on a live web page and as a dataset on Hugging Face. This is aimed at AI researchers and developers evaluating or comparing language models.