What is tactile?

Tactile is an operating layer for AI agents that helps them control software more reliably by reading the interface's underlying structure rather than just looking at screenshots. The problem it solves: most computer-use agents work by taking a screenshot, guessing where things are on screen, predicting coordinates, clicking, and then taking another screenshot to check what happened. This works but is fragile, since small visual changes can break the whole process. Instead of starting with pixel coordinates, Tactile instructs agents to first read the accessibility tree, the same structured information that screen readers for visually impaired users rely on. This tree tells you what each element is, such as a button, a text field, or a checkbox, what it is named, what state it is in like enabled, disabled, or selected, and what actions can be performed on it directly. When that information is available, the agent can operate the app semantically rather than by guessing coordinates. If the accessibility tree is incomplete, the system falls back to reading text on screen using OCR, and only as a last resort does it use pure visual screenshot reasoning. You would use Tactile if you are building or running AI agents that control desktop applications and want them to be more reliable, faster, and less prone to breaking when an interface changes slightly. A secondary goal of the project is that software built to be agent-friendly through good accessibility structure also ends up being more accessible for human users who rely on screen readers. It is written in Python and works on macOS. The full README is longer than what was shown.