What is instructsam.cpp?

smellslikeml/instructsam.cpp — explained in plain English

Analysis updated 2026-07-26

★ 3C++Audience · developerComplexity · 4/5LicenseSetup · hard

Mindmap

mindmap
  root((repo))
    What it does
      Text-guided segmentation
      CPU-only inference
      Mask image output
    How to run
      Docker container
      CLI tool
      Mount input output
    Under the hood
      Vision encoder
      Qwen3-VL language model
      Mask scoring head
    Tech stack
      C++
      llama.cpp
      Docker
    Model weights
      1.5GB quantized model
      3.3GB full precision model
      Built at compile time

mindmap root((repo)) What it does Text-guided segmentation CPU-only inference Mask image output How to run Docker container CLI tool Mount input output Under the hood Vision encoder Qwen3-VL language model Mask scoring head Tech stack C++ llama.cpp Docker Model weights 1.5GB quantized model 3.3GB full precision model Built at compile time

What do people build with it?

USE CASE 1

Identify and mask specific objects in photos using text descriptions.

USE CASE 2

Batch-process images to extract individual object cutouts without a GPU.

USE CASE 3

Generate visual overlays showing where requested items appear in an image.

What is it built with?

C++llama.cppDockerQwen3-VL

How does it compare?

	smellslikeml/instructsam.cpp	4brm01/picture-poker	bong-water-water-bong/npu-gpu-cpu
Stars	3	3	3
Language	C++	C++	C++
Setup difficulty	hard	easy	hard
Complexity	4/5	1/5	5/5
Audience	developer	general	researcher

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · hard Time to first run · 1h+

Building from source requires a compiled copy of llama.cpp placed next to the repo, using Docker is easier but still downloads multi-gigabyte model weights at build time.

Use freely for any purpose, including commercial use, as long as you include the copyright notice and license file.

So what is it?

InstructSAM is a C++ project that does natural-language image segmentation on a CPU. You give it an image and a text prompt like "Please segment 'box', 'person', 'forklift' in this image," and it returns per-object mask images plus an overlay panel showing the results. On a 12-core laptop, a 640x426 image takes about 1.7 minutes to process end to end, with no GPU required. The quickest way to run it is through Docker. You mount an input image directory and an output directory, pass the image path and your text query, and the container handles the rest. The model weights are converted at Docker build time from an existing project called InstructSAM-2B, so they are not republished separately. If you prefer to build from source, you need a built copy of llama.cpp sitting next to this repo, since the project depends on its libraries for the language model half. Under the hood, the system works in several stages. A vision encoder extracts multi-scale features from the input image. A Qwen3-VL language model then reads the vision data alongside your text prompt, decoding special markers for each object you named. For each phrase, it generates ten candidate mask hypotheses. A detector decoder refines those candidates, a scoring head picks the best one per phrase, and a pixel decoder produces the final masks at full image resolution. The command-line tool writes raw mask data, per-object PNGs, and a side-by-side overlay. The project ships four weight files, all produced during the build process. These include a quantized language model at roughly 1.5 GB, a full-precision version at 3.3 GB, a vision projector at 780 MB, and a grounding model at about 1 GB. There is also a small 80 KB file containing learned mask-query templates. The code is Apache-2.0 licensed. It depends on llama.cpp for the language model stage and is forked from an earlier C++ project called sam3cpp, which contributed the vision encoder and mask decoder structure. The original InstructSAM model and architecture come from CircleRadon on Hugging Face.

Copy-paste prompts

Prompt 1

Help me write a Docker command to run InstructSAM.cpp on a folder of images, searching for 'car', 'person', and 'bicycle', and saving the overlay PNGs to an output folder.

Prompt 2

I want to build InstructSAM.cpp from source instead of using Docker. Walk me through cloning llama.cpp, building it, and linking it so that InstructSAM compiles correctly.

Prompt 3

Write a bash script that loops through a directory of JPG images and runs the InstructSAM.cpp CLI on each one with the text prompt 'segment all vehicles', saving masks to a separate subfolder per image.

Frequently asked questions