What is lastwordwinscot?

This repository holds the paper, code, datasets, and result files for a research project about how AI models explain their reasoning, known as chain of thought, or CoT. The author set out to study how much of that reasoning is genuine versus after the fact justification, and along the way found a bigger problem with how researchers test this in the first place. The main finding is that many studies which corrupt a step in a model's reasoning to see if accuracy drops are partly measuring the wrong thing. Most benchmark reasoning chains end with a line like "the answer is X". If you damage the text near the end of the chain, you are often just damaging the spot where the answer sits, not the part of the reasoning doing real work. Move that answer line elsewhere, and the part that looks important moves with it. The paper runs several experiments to test this, including comparing five different model families, and finds the effect holds up under repeated attempts to disprove it. The repo is organized so others can reproduce the work. It includes the LaTeX source and PDF of the paper, the figures, eleven curated datasets in JSONL format, eighty one result files with raw numbers, and Python scripts split into dataset building, model inference, analysis, and figure generation. Running the analysis and figure scripts only needs the included data. Rerunning the original model inference requires a Cloud TPU and the JAX machine learning framework, since that is what the experiments were run on. This is aimed at researchers and students interested in how trustworthy AI reasoning explanations really are, not at general software users. The code itself uses MIT licensing, while the paper, figures, datasets, and results use a Creative Commons Attribution license that allows reuse and commercial use as long as the author is credited.