What is arxiv-sanity-preserver?

Arxiv is a website where researchers post scientific papers before they go through formal publishing, and it receives hundreds of new submissions every day across many fields. Arxiv Sanity Preserver is a web application built by AI researcher Andrej Karpathy to make that flood of papers easier to navigate, particularly for machine learning. The application works by regularly fetching new paper listings from Arxiv, downloading the PDFs, extracting the text, and converting each paper into a numerical representation based on the words it contains. Those representations allow the system to calculate how similar any two papers are to each other. The result is a searchable, filterable interface where you can find papers by keyword, sort results by similarity to a paper you already know, see which papers are currently popular, and save papers to a personal reading list. A personalized recommendation system learns from papers you have liked and surfaces other papers it thinks you would find relevant. Running your own instance requires setting up a pipeline of scripts that fetch, download, parse, and index papers. This is not a one-click deployment, the README walks through each script in sequence and notes that the process takes several hours on a machine with 25,000 papers. The web interface is built with Flask, and a separate script can optionally pull Twitter mentions of papers to gauge social interest. The author notes in the README that this original version has been superseded by a rewrite called arxiv-sanity-lite, which is simpler, more reliable, and easier to run. The newer version also supports email digests of new relevant papers and multiple topic tags per account. New users are pointed toward that project instead. The original site was running live and serving over 25,000 papers from machine learning categories at the time the README was last updated.