LlamaSeg: Image Segmentation via Autoregressive Mask Generation

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Code: https://github.com/GML-FMGroup/llamaseg
Area: Segmentation
Keywords: autoregressive mask generation, visual tokenizer, open-vocabulary segmentation, referring segmentation, contour fidelity

TL;DR

LlamaSeg uses a frozen VQGAN to turn masks into discrete visual tokens and a LLaMA-style model to generate them from images and text, reaching 55.9 mIoU on ADE20K at 384 pixels, while still trailing specialized discriminative methods in referring segmentation accuracy and latency.

Background & Motivation

Large language models are effective at sequential token generation, whereas segmentation requires accurate regions in a two-dimensional image. Connecting these output interfaces does not automatically teach a language model to generate pixel-level structure. Methods such as LISA emit a special segmentation embedding and delegate mask production to an expert such as SAM, leaving language reasoning and precise segmentation in separate modules. Another approach represents contours as polygon vertices, which supports sequence prediction but requires long coordinate sequences for intricate boundaries. Writing spatial labels as text has a related limitation: a convenient textual representation does not inherently preserve fine geometry.

The tension addressed here is that a unified next-token interface requires discrete sequences, while accurate segmentation depends on local two-dimensional structure and contour details. The authors borrow the representation used in autoregressive image generation, treating a black-and-white mask as an image with simple appearance but demanding geometry. The prediction target is neither a semantic marker interpreted by a specialist nor a manually specified sequence of contour coordinates, but visual codes that an image tokenizer can reconstruct. Changing the representation does not solve language grounding on its own: the model still needs extensive image-text-mask pairs to identify the requested region. The fixed category systems and limited scale of conventional semantic segmentation datasets cannot alone cover varied category names, relational descriptions, and functional descriptions.

The paper therefore develops the SA-OVRS annotation pipeline and a contour evaluation protocol, rather than only replacing an output head. Training data supply text-to-pixel correspondence, discrete mask representations carry shape, and contour metrics expose detail losses that region overlap can conceal. Here, unification primarily means placing segmentation inside standard autoregressive generation; it does not eliminate encoders or decoders, nor imply that one final checkpoint handles every task. Core Idea: use an existing image tokenizer to represent masks as visual tokens, directly learn the next mask token conditioned on images and text, and support grounding with large-scale language-paired data.

Method

Overall Architecture

The inputs are an image and a segmentation instruction; the output is a binary mask corresponding to that instruction. A category name can request regions belonging to the same class, while a referring expression distinguishes a particular instance. During training, the ground-truth mask is encoded and quantized by VQGAN into the target sequence for the autoregressive model. At inference time, no ground-truth mask is provided: the model receives only the image, text, and mask-start marker, then predicts codebook indices sequentially. The indices retrieve codebook vectors that are rearranged into a two-dimensional feature map and reconstructed by the frozen VQGAN decoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["SA-1B images and masks"] --> Data["Language-Paired<br/>Data Construction"]
    Data -->|Training only: ground-truth masks| Tokens["Discrete Mask<br/>Representation"]
    Tokens -.->|Target-token supervision| Model["Conditional<br/>Autoregressive Generation"]
    Data -.->|Training only: image-text pairs| Model
    Input["Image and text instruction"] --> Model
    Model -->|Inference: predicted codebook indices| Decode["Lookup, 2D rearrangement,<br/>and decoding"]
    Decode --> Output["Channel averaging and<br/>binary mask"]

The paper supports two generator configurations: a LLaMA-style backbone trained from scratch, or adaptation of the pretrained Janus Pro multimodal model. The former extracts image and text features with frozen SigLIP2; the latter retains the pretrained language interface and SigLIP image-encoding pathway. Both predict visual mask codes and reconstruct masks, but their encoder configurations should not be conflated. Qwen2-VL and GroundingDINO are used for offline annotation, not called whenever the deployed model segments an image.

Key Designs

1. Language-Paired Data Construction: attach groundable text to pixel masks

SA-1B provides abundant masks, but the model also needs correspondence between masks and language. The first stage uses Qwen2-VL-72B to generate at most 10 candidate open-vocabulary labels per image, then GroundingDINO detects the associated boxes. Labels associated with more than 4 boxes are filtered to reduce matching ambiguity in crowded scenes. The paper also reports a 0.97 threshold for filtering nested detections and a detection confidence requirement above 0.3. The matching IoU threshold is 0.85 for single-box labels and 0.9 for multi-box labels; these are annotation matching rules, not test-time segmentation thresholds. Masks sharing a label are merged into semantic samples so that the model learns category-level regions rather than only individual instances.

The second stage highlights a selected instance with a green contour and asks Qwen2-VL-72B for a referring expression that distinguishes it from other objects. Recoloring contours and cross-verification then check whether the description is unique or could also apply to another instance. Reasoning expressions instead omit the category name and describe function or commonsense attributes, such as identifying a sofa as furniture for both sitting and lying down. The paper explicitly states that reasoning expressions omit the same verification step; all textual annotations should not be described as equally rigorously verified. The detailed counts on page 9 are 1.93M validated instance masks, 1.15M semantic samples, and 850K expressions, comprising 800K referring and 50K reasoning expressions. These are different units and should not be added into a count of independent masks; the abstract's 2M is a rounded scale, with over 5,800 labels covered.

2. Discrete Mask Representation: preserve reconstructable geometry through an image codebook

The authors use a pretrained VQGAN from LlamaGen with a spatial downsampling rate of 16, representing binary masks as three-channel black-and-white images. The encoder first produces low-resolution continuous features, then each feature is matched to its nearest codebook vector by Euclidean distance. The recorded values are codebook indices rather than the vectors themselves; flattening the index grid yields the target token sequence. An input resolution of 256 pixels corresponds to a \(16\times16\) mask grid, or 256 tokens; 384 pixels corresponds to \(24\times24\), or 576 tokens. These grid sizes follow directly from the downsampling rate and explain why higher resolution increases both spatial capacity and sequential generation cost. A visual code is neither a single-pixel label nor a polygon vertex, but a discrete entry into a local latent visual representation.

After generation, each predicted index retrieves a codebook vector, and the vectors are restored to a two-dimensional arrangement for decoding. The decoded output remains a three-channel image; channel averaging and binarization produce the final region. Equations (1) and (2) are corrupted in the cached extraction, so this account retains only the nearest-neighbor quantization and binarization procedures explicitly supported by the prose, without guessing the binary threshold. Freezing the tokenizer preserves reconstruction capabilities acquired from many color images, rather than assuming that black-and-white masks necessarily require specialized retraining. Table 4 tests this assumption: mask-specific fine-tuning degrades reconstruction in the evaluated settings. Reconstruction nevertheless remains lossy, and Figure 7 shows that high IoU does not guarantee intact fine details, making the tokenizer another limit on downstream accuracy.

3. Conditional Autoregressive Generation: make visual codes the actual prediction target

The from-scratch branch uses frozen SigLIP2 with patch size 16, matching the mask tokenizer's downsampling rate. Image and mask tokens therefore correspond to equally sized pixel regions, helping the model associate spatial units in the input and output. The image and text adapters each use two linear layers with GELU to project features into the generator's hidden dimension. The backbone uses one-dimensional RoPE and the explicit sequence layout of text, <BOI>, image, <BOM>, and training mask tokens. The paper also mentions concatenation along the channel dimension, which is not fully consistent with its listed sequence layout; this account follows the explicit sequence for logical ordering without asserting an unverified tensor implementation. Cross-entropy supervises only mask tokens; the conditioning image and text support prediction rather than being reconstructed themselves.

Inference begins after <BOM>, emits one codebook index at a time, and uses preceding predictions as subsequent context. LlamaSeg-B and LlamaSeg-L have 0.77B and 1.5B parameters; they use a LLaMA-style architecture and are not both direct fine-tunings of pretrained language models. The Janus Pro branch instead uses LoRA adaptation while keeping Gen Embedding, Gen Adapter, and Gen Head frozen. It retains the dialogue-role format, starts the mask sequence with <begin_of_mask>, and removes the unconditional generation component. Although causal attention operates on a one-dimensional sequence, preceding mask rows remain in context, allowing the model to learn cross-row spatial relationships. Figure 6 offers qualitative support through attention to preceding rows in the same column and suppression of artificial row-boundary neighbors, but a single visualization cannot establish general topological understanding.

A Worked Example

Consider the paper's illustration asking for the bus on the left; the following grid walkthrough explains the mechanism rather than reporting a new experiment. During training, the image and instruction provide grounding context, while the ground-truth bus mask supplies the supervision sequence. At a resolution of 256 pixels, downsampling by 16 converts the mask into a \(16\times16\) index grid. The model learns to predict subsequent indices from image-text context and preceding mask tokens, rather than stopping after emitting the word bus. At test time, it receives only the image and instruction, generates 256 mask tokens sequentially, and rearranges them into the same spatial grid. The decoder reconstructs the full latent grid, and channel averaging followed by binarization yields the region for the left bus. Grounding errors can therefore originate in conditioning or token prediction, while detail errors can also arise in tokenizer reconstruction; not every failure is a language-reasoning failure.

Loss & Training

The objective is next-token cross-entropy over the mask sequence; the expression below summarizes the prose and is not a reconstruction of a corrupted source equation. Here, \(q_t\) is a ground-truth mask codebook index, \(I\) and \(T\) are image and text conditions, and \(N\) is the number of mask tokens.

\[ \mathcal{L}_{\mathrm{mask}}=-\sum_{t=1}^{N}\log p_{\theta}(q_t\mid I,T,q_{<t}). \]

Pretraining combines SA-OVRS with referring segmentation datasets for 4 epochs, using AdamW with learning rate \(2\times10^{-4}\) and weight decay 0.05. Semantic and referring segmentation are subsequently fine-tuned separately, so their results should not be interpreted as unified evaluation of one final checkpoint. Fine-tuning uses learning rate \(1\times10^{-4}\), zero weight decay, WarmupCosineDecay, and 1% linear warmup. Semantic tasks are fine-tuned for 10 epochs and referring tasks for 20 epochs, using 8 NVIDIA A800 or H20 GPUs. The from-scratch LLaMA branch uses greedy decoding because the objective is a specific target mask rather than diverse image synthesis. The evidence in Table 5 is limited to LlamaSeg-B on the RefCOCO validation set and is not a theorem about every generative segmentation model.

Key Experimental Results

Main Results

The following selection is from Table 1 on page 11; all values are mIoU, higher is better, with closed-set ADE20K and COCO-Stuff followed by three open-vocabulary evaluations. LlamaSeg-L and its explicitly marked 384 pix. variant should be read separately: identical parameter counts do not imply identical inference computation.

Model	Parameters	ADE20K	COCO-Stuff	PC-459	PC-59	PAS-20
Unified-IO-XL	2.9B	45.2	55.2	Not reported	64.0	74.9
Unified-IO2-XXL	6.6B	51.9	55.1	Not reported	Not reported	Not reported
LlamaSeg-B	0.77B	50.5	54.5	28.5	61.7	74.5
LlamaSeg-1B(MLLM)	1B	45.1	50.1	35.6	58.2	71.1
LlamaSeg-L	1.5B	52.0	55.7	35.5	62.5	75.1
LlamaSeg-L (384 pix.)	1.5B	55.9	58.1	36.7	63.8	77.1

Higher resolution improves ADE20K by 3.9 points and COCO-Stuff by 2.4 points over LlamaSeg-L, but PC-59 at 63.8 remains below Unified-IO-XL at 64.0. The paper's broad claim of leading across all benchmarks should therefore not be applied literally to every column. Referring segmentation uses cIoU, which accumulates intersections and unions before taking their ratio, rather than simply averaging instance IoUs. In Table 2 on page 11, RefCOCO val scores are 56.5 for LlamaSeg-L and 54.8 for Unified-IO2-XXL, compared with 72.7 for LAVT and 73.8 for ReLA. This improves generative segmentation but does not surpass specialized discriminative methods; on RefCOCOg test, LlamaSeg-L at 50.4 also trails Unified-IO2-XL at 54.5.

Ablation Study

The following values are from Table 5 on page 14: LlamaSeg-B on RefCOCO validation, with higher cIoU and lower mAHD preferred; mAHD uses an IoU threshold of 0.5.

Decoding strategy	cIoU	mAHD
Greedy search	50.9	14.4
Beam search, B=3	47.3	15.1
Top-K, K=3	49.4	15.3
Top-P, P=0.9	50.2	15.3
Random sampling	49.2	15.7

Greedy search improves cIoU by 3.6 points over beam search and also improves the contour metric; exploring more branches does not produce a better deterministic mask here. The next table is from Table 4 on page 13: tokenizer mask reconstruction only, not full segmentation; fine-tuning uses batch size 128, and mAHD has no IoU-threshold filtering.

Tokenizer training steps	Total IoU	mAHD
0, frozen	96.8	6.1
5000	93.0	12.5
10000	93.5	10.6

These results support retaining the pretrained tokenizer in the tested setup, but do not establish that all mask-tokenizer training is ineffective. Table 6 on page 14 additionally shows that SA-OVRS+Ref. pretraining raises LlamaSeg-B from 48.9 to 50.5 ADE20K mIoU and from 46.0 to 50.9 RefCOCO cIoU. This compares joint pretraining against no pretraining, so the entire difference cannot be attributed to SA-OVRS alone.

Key Findings

The contour metric averages the nearest Euclidean distance from each predicted boundary point to the ground-truth boundary, repeats the process in the reverse direction, and averages the two directional means to obtain dAHD. It is a bidirectional average distance, not the classical maximum-deviation Hausdorff distance; Equation (3) is corrupted in the extraction, so this definition follows the accompanying prose. The authors separately filter predictions at IoU thresholds 0.5, 0.6, 0.7, 0.8, and 0.9 and report average values as mAHD; this is not an unconditional metric that includes every sample. Table 3 on page 12 normalizes resolution to 256 pixels; in the ADE20K IoU-0.5 column, LlamaSeg-L scores 25.54 and Unified-IO-XL scores 75.88. Models may pass the threshold on different samples, so contour quality should be interpreted alongside overall IoU; Table 4 uses no threshold and is not directly comparable to Table 3. Table 7 on page 15 reports 3949.92 ms latency, 0.250 FPS, and 3.69 GB peak memory for LlamaSeg-B, versus 35.89 ms, 27.84 FPS, and 1.78 GB for LAVT. LlamaSeg-B is faster than Unified-IO2-Large at 57906.23 ms, but remains far from real-time segmentation.

Highlights & Insights

• The central change is the output representation, rather than attaching a segmentation expert to a language model. Visual codes put local shape inside the next-token objective while retaining a general image decoder.
• Matching image patch size to mask downsampling gives conditioning and targets corresponding spatial units. This is more directly relevant to dense prediction than merely matching hidden dimensions.
• Apparently simple black-and-white masks do not guarantee that specialized training beats general visual pretraining. Table 4 motivates measuring frozen reconstruction quality before investing in tokenizer retraining.
• Region overlap and boundary quality need separate inspection. The contour analysis is useful, but its threshold-induced sample selection should also be reported.

Limitations & Future Work

• The authors acknowledge slower sequential generation than parallel dense decoding and a substantial referring segmentation gap to specialized discriminative models. The evidence supports feasibility of a unified generation interface rather than superior real-time deployment.
• Tokenizer reconstruction loses details, and a Total IoU of 96.8 does not imply lossless preservation of every thin structure. This is an error source separate from the generator.
• Reader assessment: Table 6 combines SA-OVRS and existing referring data without an equal-budget comparison isolating SA-OVRS, limiting causal attribution of the data contribution.
• Reader assessment: reasoning expressions lack equivalent cross-verification, and the main experiments do not separately report a reasoning segmentation benchmark; 50K reasoning expressions do not establish validated complex reasoning ability.
• Future studies could compare shorter sequences or parallel generation at matched latency and report sample coverage at each IoU threshold. These are research suggestions, not experiments completed in the paper.

Related Work & Insights

• vs LISA / GSVA: these methods connect special embeddings to segmentation experts, whereas LlamaSeg directly predicts visual mask codes. The distinction concerns which component performs pixel generation, not the elimination of visual decoding altogether.
• vs Unified-IO / Unified-IO2: both pursue unified generative interfaces, but LlamaSeg specifically develops mask representation, paired training data, and contour analysis. Superiority claims must remain tied to the actual tables and datasets.
• vs LlamaGen / VQGAN: LlamaSeg reuses discrete image-generation representations without seeking output diversity. Its greedy-decoding ablation shows why synthesis sampling preferences should not be transferred uncritically to deterministic segmentation.
• vs LAVT / ReLA: specialized discriminative models retain substantial referring accuracy advantages, and LAVT also has a latency advantage. The main contribution is exploring native autoregressive segmentation, not replacing all existing segmenters.

Rating

• Novelty: 4/5. Visual-code prediction, language-paired data, and boundary analysis form a complete segmentation approach, although the tokenizer and generator backbone largely reuse existing designs.
• Experimental Thoroughness: 4/5. Semantic, open-vocabulary, referring, reconstruction, decoding, and efficiency analyses are covered, but stricter data attribution and dedicated reasoning segmentation evaluation remain absent.
• Writing Quality: 3/5. The central argument is clear, but some broad superiority claims exceed the tables, and the extracted equations and some tensor descriptions have readability issues.
• Value: 4/5. Useful for unified visual generation interfaces and segmentation data construction, with real-time performance and specialized task accuracy remaining practical constraints.