What is pretraining-tda?

This repository accompanies a research paper on understanding where large language models (LLMs) get their facts. When an AI tells you that "Paris is the capital of France," it likely learned that by reading massive amounts of text during its initial training. This project provides the data and a web tool to help researchers trace a model's answer back to the specific sentences it was trained on. It essentially asks: which passages in the training data were the most influential in making the model give this specific answer? The repo provides pre-computed results from several different tracing methods. These results map test queries, like factual prompts about a person, place, or thing, to "proponents," which are the specific training passages that pushed the model toward its answer. Some methods also identify "opponents," or passages that pushed against the answer. Because these lists of text passages are dense and hard to read in a spreadsheet, the project includes a simple data viewer app. You can load the results directly in your web browser, and the app runs entirely on your computer without sending your data to a server. This tool is primarily for AI researchers and engineers who study model behavior, bias, and factuality. For example, if a model keeps hallucinating a fact, a researcher could use these tools to look at the proponents and see what kind of messy or contradictory training text caused the error. Alongside the tracing results, the repo also provides the raw test queries used to evaluate the AI, including a set of 5,400 factual prompts, and a massive corpus of nearly 20 million Wikipedia sentences used as the search space for the experiments. What's notable about this project is that it acts as a benchmarking package. Rather than just providing one method to trace a model's knowledge, it includes results from multiple different tracing algorithms (like BM25, TRAK, and TrackStar) applied to the same data. This allows researchers to compare how different approaches perform side-by-side using the same test queries and the same viewer tool.