What is refertrack?

ReferTrack is a research project from a collaboration between SUSTech, Tencent Robotics X, Peking University, and a Futian Laboratory. It tackles a problem called "embodied visual tracking," which means helping a robot follow or keep watch on a specific object as both the robot and the target move through a space. The project comes with a project page, a video demo, and a planned academic paper. The core idea is a two-step approach described as "referring then tracking." First, someone gives the system a language description of a target object, and the system identifies that object in the camera image by drawing a box around it. Then, using that decision, the system works out where the robot should move next to keep tracking the object. This separation of steps is what gives the project its name. To keep track of the target over time, ReferTrack uses something the authors call "temporal-viewpoint-bbox indicator tokens," or TVBI tokens. These tokens feed information about previously selected bounding boxes back into the visual history the system has seen, so the system can preserve clues about how the target has been moving. This helps the robot maintain awareness of the target across different viewpoints and time steps. The authors report that ReferTrack achieves leading single-view performance on a benchmark called EVT-Bench. They also note that the system transfers well from simulation to real legged and humanoid robots, meaning a model trained in a simulated environment can still function on physical robots. The repository is currently sparse. The README lists a set of planned releases that are all unchecked, including model checkpoints, evaluation code, a dataset, training code, and a data engine. None of these appear to be available yet. The repository contains the overview, author list, and links to external resources, but no install instructions, usage details, or code at this time.