What is rcrr-bench?

RCRR Bench is a research benchmark that measures how much meaning survives when a document is converted from a scanned page or PDF into Markdown text. Instead of scoring conversion quality by how the output looks, it asks a more direct question: after a page is converted, can an AI reader still correctly answer questions that a human could answer from the original page. The benchmark covers 99 real Japanese investor relations pages pulled from public corporate disclosure filings, paired with 1,410 validated questions with human reviewed gold answers. Fourteen different document conversion systems are compared, including several general purpose vision language model APIs, dedicated OCR products like Azure Document Intelligence and Mistral OCR, open source converters such as Docling and Marker, and the maker's own product, Nebula, tested in two forms: a hosted version that uses a frontier vision model inside a custom pipeline, and a fully self hosted version built by fine tuning an open model called Qwen3-VL-32B on customer controlled hardware. Results are reported as an overall score plus separate scores for text and tables versus charts, along with statistical comparisons showing which differences are meaningful versus which are within the margin of error. The repository includes the full set of per question answers and judge scores for every tested system, along with the source data manifest describing exactly which filings and pages were used and why certain questions were kept or excluded. The project is set up so outside researchers can independently check the published numbers without needing any API keys, simply by installing the requirements and running the included analysis script on the data already in the repository. Anyone who wants to test a new document conversion system can also plug its output into the same scoring scripts to see how it compares to the fourteen systems already measured. The source documents themselves are not redistributed, only referenced by identifier, since they remain the property of the original filers. The code and included data are released under the Apache 2.0 license.