What is vlm-webcam-vision?

This project takes a live webcam feed and uses a vision language model, a kind of AI that can understand both images and text together, to detect and track hands in real time. As you move your hands in front of the camera, the app draws bounding boxes and labels around each detected hand directly on the video feed, and it smooths the box movement between AI inference frames so the tracking looks steadier rather than jittery. The author describes this as an experimental project built quickly with heavy AI assistance to test how well vision language models can handle real time hand detection, and says clearly that the code favors working functionality over polish. It requires Python 3.12 or newer, a USB webcam, and a separate piece of software called llama-swap running a vision language model locally on your machine, since the hand detection AI itself is not built into this app but called over a local API. You can configure quite a lot without touching the code, using environment variables to switch which AI backend it talks to, which specific model it uses, what detection prompt is sent to the model, which camera to use, how many times per second it asks the AI for a new detection, and the size of the display window. Inside the app itself there is a control panel where you can pick a model, choose from preset prompts like finding hands or detecting general objects, type your own custom prompt, adjust how often frames are analyzed, toggle the smoothing and label cleanup features, tweak camera brightness and exposure, capture a still frame to disk, and view the model's raw debug output. The project has been tested with a small hand tracking model the author fine-tuned themselves, described as around 450 million parameters and only a few hundred megabytes, with detection latency around 130 to 150 milliseconds, but it also works with several other general purpose vision language models as long as they support this kind of bounding box detection. It is released under the MIT license and has one star, reflecting its status as a personal experiment rather than a polished product.