What is mcp_safety_training?

This repository is the companion code for two research papers on making AI models that use external tools safer. It focuses on a specific problem: some requests to a tool-using AI model look completely harmless on the surface but are actually designed to trick the model into misusing its tools in a harmful way, a category the authors call falsely-benign MCP exploits, referring to attacks that use the Model Context Protocol that AI tools rely on to interact with files and other resources. The code trains and evaluates several methods for teaching a model to recognize and refuse these disguised attacks while still responding normally to genuinely harmless requests. It includes standard preference alignment training methods called DPO and SafeDPO, which fine-tune a model using pairs of good and bad example responses, plus two methods that need no training at all: OPAD, which changes how the model generates text on the fly based on stated safety principles, and RAG-Pref, which looks up similar past examples of attacks and safe requests at the moment of generation and feeds them into the model's instructions to guide its response. According to the README, models tested without any of these defenses refused fewer than 35 percent of the disguised attacks, standard DPO and SafeDPO training only raised that to about 48 percent, while RAG-Pref roughly tripled the refusal rate on its own and nearly quadrupled it when combined with DPO or SafeDPO training. The project includes its own training and evaluation dataset hosted on Hugging Face, containing preference pairs for training plus separate held-out sets of disguised attacks and genuinely benign prompts for measuring both refusal of attacks and over-refusal of safe requests. Running the training scripts requires a CUDA capable GPU, Python 3.11, and several machine learning libraries including PyTorch and a flash-attention build matched to that PyTorch version. The project is released under the Apache 2.0 license.