Segment Anything

Paper: Alexander Kirillov et al. · 2023 · arXiv:2304.02643 · Full paper

In one sentence: SAM turns image segmentation from a model built for one fixed task into a reusable service: show it what you mean with a point, box, or rough mask, and it returns a plausible object mask—even on many image types it did not train on.

Takeaway

Three things to remember

1. The prompt is the interface. Instead of asking for a fixed class such as “car,” SAM accepts a point, box, rough mask, or—in an exploratory setup—text, then returns a valid region. Paper §§2–3
2. The architecture makes interaction cheap after one expensive step. A large image encoder runs once; a small prompt encoder and mask decoder can then answer a new prompt in about 50 ms in a web browser using the saved image embedding. That number excludes the heavy image-encoding step. Paper §3
3. The model and dataset grew together. A three-stage annotation loop produced 1.1 billion masks from 11 million licensed, privacy-protecting images, giving SAM broad training coverage. Paper §§4–5

The strongest evidence is breadth: the authors test zero-shot transfer across a new suite of 23 segmentation datasets and several downstream tasks. The largest unresolved risk is reliability at the edges—fine structures, small stray components, specialist domains, and the costly image encoder can still matter more than broad generality. Paper §§7–8

Problem

Segmentation did not have a general interface

Image segmentation assigns pixels to an object or region. Before this work, strong systems were usually built around a particular task, label set, or interaction pattern. That makes reuse awkward: a model trained for one benchmark does not automatically become a general component for a new image source or a new segmentation job. The paper’s problem is therefore not simply “draw better boundaries.” It is: can one model expose a flexible interface that many segmentation tasks can use without retraining? Paper §1

One prompt can mean several correct things

A point on a badge could mean the badge, the jacket, or the whole person. A training set usually stores only one of those answers. SAM treats this ambiguity as part of the task: it predicts three candidate masks and confidence scores, rather than blending several meanings into one fuzzy mask. During training, only the candidate with the lowest loss is used for backpropagation. Paper §§2–3

Why this visual matters: it shows why “one prompt, one ground truth” can be the wrong mental model. This is a new teaching illustration, not a reproduced paper figure.

Method

Promptable segmentation

The training task is deliberately broad: given any segmentation prompt, return at least one reasonable mask. Training simulates sequences of point, box, and mask prompts against known masks. At use time, a downstream system can turn its own output into a prompt—for example, an object detector’s box can ask SAM for the object’s exact pixels. This composability is the paper’s main change from a fixed multi-task segmenter. Paper §2

A reusable image representation

SAM has three parts. A Vision Transformer image encoder converts the image into an embedding. A prompt encoder represents points, boxes, text, or masks. A lightweight mask decoder combines both representations to predict masks and their estimated quality. Because the image embedding is reusable, the expensive image computation need not repeat for every click. Paper §3

The diagram separates the one-time image path from the fast prompt path. The paper reports about 50 ms for the prompt encoder and mask decoder after the image embedding exists; it does not claim the whole pipeline is real-time with the largest encoder.

A data engine, not a static labeling campaign

There was no web-scale supply of segmentation masks, so the team used the improving model to help create its next training set. In the assisted-manual stage, professional annotators clicked and refined model output; annotation time fell from 34 to 14 seconds per mask as SAM improved. In the semi-automatic stage, the model filled obvious regions and people added missed objects. In the final stage, a 32×32 point grid, confidence and stability filters, duplicate removal, and overlapping crops generated masks automatically across all 11 million images. Paper §4

This loop is the practical flywheel: better masks improve the model, and a better model makes broader mask collection possible.

Evidence and limits

What the experiments establish

The released SA-1B dataset contains 1.1B masks, and 99.1% were generated fully automatically. To check mask quality, the authors sampled 500 images (50,000 masks), had professionals correct every mask, and compared each automatic/corrected pair: 94% exceeded 90% IoU, while 97% exceeded 75% IoU. This is strong evidence that the automatic pipeline usually produced close geometric matches on that sample, not proof that every mask is correct. Paper §5

For zero-shot single-point segmentation, SAM was evaluated on 23 datasets. Against the strong interactive baseline RITM, the paper reports a mixed per-dataset result: SAM’s mIoU delta ranges from −21.4 to +46.9, which is more informative than saying it simply “wins.” Human ratings are included because one point can validly refer to an object different from the benchmark’s single stored mask. Paper §7.1, Fig. 9

On zero-shot instance segmentation, a supervised ViTDet-H remains ahead by standard mask AP: 51.0 vs. 46.5 on COCO and 46.6 vs. 44.7 on LVIS. SAM’s masks received better human quality ratings in the paper’s study, exposing a tension between benchmark conventions and perceived boundary quality. Paper §7.4, Table 5

What remains uncertain

The authors say SAM can miss fine structures, create small disconnected artifacts, and produce less crisp boundaries than slower zoom-in methods. Dedicated interactive systems may win when given many clicks; specialist tools may win inside their domains. Text-to-mask is explicitly exploratory and not fully robust. Simple prompting for semantic and panoptic segmentation remains unresolved. Paper §8

The dataset is broad but not geographically balanced: Africa, Latin America and the Caribbean, and low-income countries are underrepresented. The reported person-segmentation confidence intervals mostly overlap across perceived gender, age, and skin-tone groups, but the paper also finds an indication of clothing-segmentation bias by perceived gender and warns that larger systems can introduce new biases. Paper §6

Practical meaning

Where SAM is a strong starting point

Interpretation: SAM is most useful as a general mask-producing component when the target object can be indicated spatially and broad reuse matters more than squeezing out the last benchmark point. Likely uses include annotation tools, object extraction, detector-to-mask pipelines, and fast prototypes across unfamiliar image collections. This follows from the paper’s prompt interface and zero-shot tests; it is product interpretation, not an author guarantee.

What to verify before shipping

Test the exact domain rather than relying on the word “anything.” Measure thin structures, small objects, boundary quality, unwanted fragments, and latency including image encoding—not only the 50 ms prompt path. Compare against a specialist model when mistakes are costly. Audit regional and human-representation slices that match the deployment population, and remember that a downstream detector or prompt generator can become the system’s real bottleneck or bias source.

Source notes and terminology

• SAM: Segment Anything Model, the promptable segmentation model.
• SA-1B: the paper’s dataset of 11M images and 1.1B automatically generated masks.
• Mask: a pixel-level selection of an image region.
• IoU: intersection over union, the overlap between two masks divided by their combined area.
• Zero-shot transfer: using SAM on tasks or datasets not used to train it, usually by translating the task into prompts.

Primary source: Kirillov et al., Segment Anything, arXiv:2304.02643, submitted 5 April 2023. Claims and numbers above are grounded in the official arXiv record and the linked paper text. Product implications are labeled as interpretation.