What is trackcraft3r?

TrackCraft3R is a research project that takes an ordinary video and figures out how every pixel moves through 3D space over time. Feed it a clip of someone dancing or a car driving down a street, and it produces dense 3D motion trails, tracking where each point in the frame travels, not just left/right/up/down on screen, but also toward or away from the camera. This is useful for anyone working in augmented reality, robotics, visual effects, or 3D scene reconstruction who needs to understand real-world motion from flat video footage. The clever part is how it achieves this. Instead of building a tracker from scratch, the researchers repurpose a video generation model (the kind designed to create video from text) and retrain it to predict 3D trajectories instead of new frames. The system takes a video plus estimated depth and camera movement, then predicts the full set of 3D tracks in a single pass. Training happens in two stages: first, fine-tuning a small portion of the model on synthetic datasets of rendered 3D scenes, then unlocking more components for a second round of training. The primary audience is computer vision researchers and engineers working on 3D tracking, scene reconstruction, or video understanding. A robotics team could use it to help a robot understand how objects move in its environment from camera input. A VFX studio could extract detailed motion data from a live-action clip to blend real footage with computer-generated elements. Notably, this is heavyweight research code. Training requires eight high-end NVIDIA GPUs, and running it on your own video involves a multi-step pipeline: estimating depth and camera motion with a separate tool, packaging that data, running inference, and then visualizing results. The project includes a sample video to walk you through the process, and pre-trained weights are available so you can try it without training from scratch.