What is apt?

This is the code for a research project called APT, which stands for Action Expert Pretraining, released alongside an academic paper on arXiv. It addresses a problem in training AI systems that control robots by processing both visual input and spoken or written instructions. The type of AI model this work focuses on is called a vision-language-action model, or VLA. A VLA takes in camera images of a scene and a text instruction like "pick up the red cup and place it in the storage box," and outputs a sequence of robot movements to carry out that task. The challenge is that these models are often trained on datasets where text instructions are sparse or noisy, and the action-generation component starts from random weights, leading to unstable training that can corrupt the language understanding part of the model. APT addresses this by splitting the training into two stages. In the first stage, only the action-generation component (called the action expert) is trained, using visual observations and recorded robot movements but with language completely excluded. This gives the action expert a solid understanding of how to move based on what it sees, without any interference from language. In the second stage, a gated mechanism is introduced that injects language features from a large vision-language model (Qwen3-VL) into the action expert layer by layer. The gates are learned values that control how much language influence enters each layer, preserving the visual movement knowledge built in stage one while adding the ability to follow instructions. The result, according to the paper, is better generalization to instructions the model has not seen before, and to tasks that combine familiar instructions in new ways. The repository includes the full training code, data loading utilities for HDF5-format robot trajectory datasets, inference utilities, and evaluation setups for two robot benchmarks: LIBERO (a manipulation benchmark) and a pick-and-place task in the Isaac Sim physics simulator using a UR5 robot arm. Training uses either PyTorch's standard distributed training or the DeepSpeed library for larger multi-GPU setups. The code is structured to support adding new datasets and new model configuration presets.