What is infinibench?

InfiniBench is a test suite designed to measure how well AI models can understand long videos like movies and TV episodes. Instead of just asking whether a model can describe a short clip, it evaluates whether an AI can follow a full-length storyline lasting anywhere from tens of minutes to two hours. The benchmark includes over 1,000 hours of video and around 91,000 question-and-answer pairs, making it a tool for checking if models can truly process narratives over a long duration. The benchmark tests AI across eight different skills, split between basic observation and deeper reasoning. Observation skills include recognizing scene transitions, character actions, and chronological order. Reasoning skills involve summarizing plots, understanding context, and linking multiple events together. Questions come in both multiple-choice and open-ended formats. To use the benchmark, teams run their AI models against the dataset, answer the questions, and then compare their results on a public leaderboard. This project is aimed at AI researchers and developers building vision-and-language models. For example, if a startup is creating a tool to automatically generate video summaries or a search engine that can find specific scenes by describing them, this benchmark helps them see how their system handles long-form content. There is also an active challenge where teams can submit their models for evaluation. The findings so far show that current models struggle significantly, with even leading models like GPT-4o scoring under 50% on basic grounding tasks. A notable takeaway from the project's evaluation is that models often rely on pre-trained world knowledge rather than actually watching the video. For instance, some models scored well just by reading metadata like a movie's title. However, when given full video alongside subtitles, performance improved, highlighting that combining visual and text information is crucial for long video understanding.