What is appo?

APPO (Agentic Procedural Policy Optimization) is a research project for training AI language models to use tools more effectively. It extends an earlier system called ARPO (Agentic Reinforced Policy Optimization) by adding a technique called procedure-aware branching. The core idea is that when an AI is working through a multi-step task that involves calling external tools like web search, there are certain moments where its decision is particularly uncertain. APPO identifies those moments and explores multiple paths forward from them, then uses the results to better teach the AI which decisions lead to good outcomes. The branching approach is guided by a formula called the Branching Score, which combines how uncertain the model is at a given step with how differently the model would behave if it were retrained on the results of that step. Only the most informative moments are branched, keeping the training process manageable. When branches are explored, they reuse the same tool-calling setup as the main rollout, but their outcomes are mapped back to credit the original trajectory rather than training on the branch steps directly. In practice, using APPO involves three stages. The first is optional: fine-tuning a base model on curated examples of good reasoning to give training a head start. The second stage is the main reinforcement learning loop, where the model generates multi-step responses using tools (web search via a service called Bright Data), branches at identified decision points, and updates its parameters based on the outcomes. Scripts are provided for models ranging from 7 billion to 14 billion parameters, based on the Qwen and Llama families of open-source language models. The third stage is evaluation against benchmarks. The project builds on top of VERL, a framework for large-scale reinforcement learning of language models, and the LLaMA-Factory fine-tuning toolkit. Setup involves creating separate Python environments for each pipeline stage. The research paper citation is marked as coming soon, but the project asks users to cite the ARPO paper if they use the underlying codebase or datasets.