What is panovine?

PanoVine is a research codebase from Stanford University for controlling a soft growing vine robot using cameras and machine learning. The project comes from a research paper about teaching a robot made of soft, expanding material to move and manipulate objects by learning from demonstrations, rather than relying on hand coded rules. A vine robot is a type of soft robot that grows and extends the way a plant vine grows, rather than moving with rigid joints like a typical robotic arm. The codebase covers three main stages: collecting demonstration data, training a policy on that data, and running the trained policy on a real vine robot. Data collection uses a script that logs sensor readings such as encoder positions, pressure, torque, and orientation, along with camera video, while a person operates the robot. The recorded logs are then processed into a single combined dataset file using a parallel processing script. From there, a diffusion based policy model is trained on the dataset using a provided configuration file, with support for both single machine and multi GPU training through the accelerate library. Running the code requires setting up a conda or mamba environment from a provided configuration file, which installs the machine learning dependencies needed for data processing and training. Actually collecting data from a real robot, or running the trained policy on one, additionally requires ROS Noetic, a robotics middleware framework, along with drivers specific to the vine robot's motors and sensors. None of that extra setup is needed for offline training on data that has already been collected. The project is released under the MIT license, so it can be freely reused, including for commercial purposes, as long as the copyright notice is kept. It builds on and credits two related open source projects, RoboPanoptes and the Universal Manipulation Interface, for parts of its policy implementation. Given its dependence on specialized robot hardware, this repository is primarily useful to robotics researchers rather than general software developers, though the diffusion policy training code and data processing pipeline could be studied independently of the physical robot.