What is toolcua?

ToolCUA is a research project that builds an AI agent designed to operate a computer the way a person would, by clicking, typing, and scrolling on a screen, while also knowing when to call higher-level tools such as file or application APIs instead of clicking through a GUI step by step. The project's core problem is that giving an agent both options at once tends to confuse it. Some models stick to clicking through everything, never using the faster tool calls, while others overuse tools and end up failing the task anyway. ToolCUA's goal is to teach a model to pick the right path at each step. The team trains the agent in three stages. First they build a large set of example task recordings that mix GUI actions with tool calls, generated from existing GUI-only data. Second, they run a fine-tuning stage that teaches the model tool-calling knowledge and when to switch between the two modes. Third, they apply an online reinforcement learning stage inside a simulated desktop environment, using a reward that favors completing tasks in fewer, more efficient steps. The repository ships a released model called ToolCUA-8B, built on top of the Qwen3VL-8B-Instruct model, along with evaluation code and results on a benchmark called OSWorld-MCP. Setup involves installing Python dependencies from a requirements file, downloading the model weights from Hugging Face, and serving the model using vLLM, a tool for running large language models efficiently. Running the evaluations requires setting up the OSWorld or OSWorld-MCP desktop simulation environments separately. According to the results table in the README, ToolCUA-8B outperforms its base model, Qwen3-VL-8B-Instruct, on accuracy, on how often it correctly chooses to invoke a tool, and on completing tasks in fewer steps. This is a research codebase aimed at people building or studying computer-use agents rather than a general-purpose tool for typical end users.