Skip to content

Kiroshi: An Agentic Perception System for High-Accuracy Image Parsing

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/haipengzhou856/Kiroshi
Area: Segmentation
Keywords: High-Accuracy Segmentation, Image Matting, Multimodal VLM, Agentic Perception, Direct Preference Optimization

TL;DR

Addressing the reliance of interactive matting on tedious manual verification and the inability of multimodal large language models (MLLMs) to capture pixel-level fine boundaries, Kiroshi reformulates image parsing into an explicit plan-and-act agentic loop where an MLLM emits compact text-based grid actions to steer an attention-guided action model, aligned via residual-sampled trajectory Direct Preference Optimization (DPO) for fully automated, high-fidelity matting and segmentation.

Background & Motivation

Extracting fine-grained, semantically meaningful structuresโ€”such as intricate topological boundaries, subtle transparency gradients, and hair-level strandsโ€”is vital for AR/VR, 3D reconstruction, and content manipulation. However, existing high-precision scene parsing approaches predominantly focus on salient object detection or single-target benchmarks (e.g., DIS-5K), making them prone to severe over-matting, region omission, or instance confusion in cluttered multi-instance environments. Interactive matting frameworks (e.g., ZIM, HQ-SAM) partially mitigate this limitation through point or box prompts, but remain heavily reliant on repetitive human inspection and trial-and-error corrections, falling short of full automation.

With the advent of multimodal large language models (MLLMs), referring and reasoning segmentation frameworks have substantially elevated semantic comprehension and cross-modal alignment. Yet, the underlying segmentation decoders (commonly inherited from SAM) face inherent representation and resolution constraints, rendering their predicted masks coarse and incapable of resolving transparent regions or high-frequency boundary details. The core tension lies in the mismatch of granularities: forcing an MLLM to directly predict dense pixel-level coordinates (such as point prompts) leads to an excessively large search space and redundant local visual feedback, causing unstable policy exploration; conversely, conventional dense prediction architectures lack high-level contextual reasoning and self-reflection capabilities.

This paper tackles the challenge by decoupling single-shot prediction into a hierarchical perception agent system: a high-level MLLM policy oversees semantic understanding and patch-level self-reflective decision-making, while a low-level action model executes pixel-level boundary recovery. Core idea: formulate high-accuracy image parsing as an explicit plan-and-act Markov decision process, where an MLLM policy emits compact textual grid actions () to dictate "where to refine" and an attention-masked action model performs localized pixel updates to determine "how to refine," aligned via residual-mined trajectory pairs under DPO for closed-loop autonomous refinement.

Method

Overall Architecture

Kiroshi models high-accuracy image parsing as a discrete-step Markov Decision Process (MDP). Given an input image \(I\), a natural language instruction \(X\), and an initial prompt (such as a bounding box \(B\)), the system maintains an evolving parsing prediction state \(s_t\) (a binary segmentation mask or a continuous alpha matte). At each refinement step \(t\), the policy generator \(\mathcal{P}\) observes the current state \(o_t = (I, X, s_t)\) and emits a region-level action \(\tau_t\) formatted as a decodable textual grid map: $\(\tau_t = \mathcal{P}(I, X, s_t)\)$ The action model \(\mathcal{A}\) subsequently consumes this action, translating it into an attention mask to execute pixel-level mask updates: $\(s_{t+1} = \mathcal{A}(I, s_t, \tau_t), \quad t=0,\dots,T-1\)$ After \(T\) reflective cycles (defaulting to 3 loops), the system outputs the final high-fidelity mask \(\hat{s} = s_T\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image I & Text Instruction X<br/>Generate Initial Box B & Base Mask s0"] --> B["Textual Grid Actions & R-RLE Encoding<br/>MLLM inspects intermediate state and outputs region grid action ฯ„t"]
    B --> C["Grid-Aware Action Model & Attention Masking<br/>Convert grid action into attention mask for localized pixel refinement"]
    C -->|Residual error persists AND step t < T| B
    C -->|Refinement complete at step t = T| D["High-Accuracy Parsing Output<br/>Produce high-fidelity segmentation mask or alpha matte"]
    E["Training: Residual Trajectory Mining & DPO Alignment<br/>Mine positive/negative trajectory pairs based on quality gain under same state"] -.->|Align MLLM decision policy| B

Key Designs

1. Grid-Aware Action Model & Attention Masking: Structuring Interaction Beyond Discrete Points

Existing interactive foundation models (e.g., SAM, ZIM) heavily rely on point-click prompts or bounding boxes. However, empirical evidence shows that interactive matting backbones are notably insensitive to negative point prompts, and point prompts within high-resolution regions introduce severe spatial ambiguity. To constrain the search space while preserving coherent spatial contexts, the action model \(\mathcal{A}\) integrates a DINOv2 vision backbone with a Mask2Former-style Transformer decoder. Across the cross-attention and self-attention operations in each decoder layer, the decoded grid action is converted directly into a spatial attention mask \(\mathcal{M}^b\): $\(\mathcal{M}^b(s) = \begin{cases} 0, & s \in \text{grid} \\ -\infty, & \text{otherwise} \end{cases}\)$ This mechanism forces the attention weights to concentrate strictly on the policy-designated error candidates and semi-transparent boundary regions, filtering out irrelevant background clutter. The pixel decoder then progressively upsamples multi-scale features under this localized constraint, avoiding global re-computation while precisely reconstructing micro-structures such as thin hair strands and boundary cutouts.

2. Textual Grid Actions & R-RLE Encoding: Discretizing Continuous Masks into Compact Language Units

To enable language models to interpret and plan mask refinements without the instability of regressing floating-point coordinates, Kiroshi discretizes the image into a structured spatial grid (defaulting to \(32 \times 32\)). Each grid patch functions as a semantic textual token assigned to one of four states: target foreground (e.g., <human>), background (<bg>), uncertain regions requiring refinement (<unknown>), or other interfering instances (<other>). Directly emitting raw patch grids yields long, redundant token sequences up to 1024 tokens. To address this, the authors introduce Row-wise Run-Length Encoding (R-RLE). Consecutive identical patches along each row are compressed into a single count token (e.g., a row of 64 background patches becomes bg*64). Benefiting from the natural spatial continuity of object silhouettes, R-RLE compresses typical action sequences down to tens of tokens, drastically minimizing MLLM context overhead and enabling efficient learning of error localization.

3. Residual Trajectory Mining & DPO Alignment: Autonomous Preference Optimization Without Dense Human Labels

While SFT provides the MLLM with basic grounding and grid generation capabilities, it does not inherently teach the model which refinement trajectory yields the highest matting quality. To bypass the notorious challenge of dense pixel-level reward engineering, the authors design an autonomous trajectory mining scheme during the action model's rollouts. For a given intermediate prediction \(s_{t-1}\), the system computes the absolute error map against ground truth \(r_t = |Y - s_{t-1}|\). Grid patches where non-zero residual pixels exceed threshold \(\epsilon\) (default 40%) form a candidate pool, from which 75% are uniformly sampled to construct candidate actions. Executing different grid actions produces distinct updated predictions; transitions that improve quantitative metrics (e.g., MSE or SAD) \(\Delta_t = Q(s_t) - Q(s_{t-1}) > 0\) are collected into the positive pool \(\mathcal{D}^+\), while deteriorating transitions are assigned to the negative pool \(\mathcal{D}^-\). Augmenting these with location-grounded reasoning traces generated via Qwen3-VL Plus produces 40,000 paired conversational samples for Direct Preference Optimization (DPO): $\(\mathcal{L}_{\text{DPO}}(\mathcal{P}) = -\mathbb{E}_{(I,X,\tau^+,\tau^-)}\left[\log\sigma\left(\beta\log\frac{\mathcal{P}(\tau^+\mid I,X)}{\mathcal{P}_{\text{ref}}(\tau^+\mid I,X)} - \beta\log\frac{\mathcal{P}(\tau^-\mid I,X)}{\mathcal{P}_{\text{ref}}(\tau^-\mid I,X)}\right)\right]\)$ This closed-loop alignment trains the MLLM to actively detect subtle mask artifacts and execute actions that reliably optimize parsing quality.

Loss & Training

The framework is trained through a progressive two-stage regimen: - Action Model Optimization: Supervised with a weighted combination of \(\ell_1\) loss and Laplacian pyramid loss on synthetic matting datasets (comparable to SmartMat scale) alongside high-resolution segmentation datasets (DIS-5K, UHR-SOD), empowering the model to handle both hard boundaries and soft alpha mattes. - Stage I (MLLM SFT): Supervised fine-tuning using standard cross-entropy loss over RefCOCO series and RefMatte datasets, where <unknown> regions are derived from trimap erosion-dilation, establishing object grounding and R-RLE grid generation. - Stage II (DPO Post-Training): Optimizing the MLLM policy generator using the mined 40k paired trajectory dataset via the DPO objective while keeping the action model frozen. - Inference Pipeline: Operating under an MLLM-as-Judge paradigm with \(T=3\) iterative refinement loops starting from an initial bounding box prompt.

Key Experimental Results

Main Results

To establish rigorous and fair evaluation for multi-instance referring parsing, the authors introduce the HiFiRefMS benchmark, comprising a matting subset (468 images / 1187 masks) and a segmentation subset (500 images / 1366 masks). The table below compares Kiroshi against leading salient, interactive, and MLLM-based methods:

Method Type / Venue Seg: \(\text{max}F_\beta \uparrow\) Seg: \(\text{MAE} \downarrow\) Seg: \(\text{HCE} \downarrow\) Matting: \(\text{MSE} \downarrow\) Matting: \(\text{SAD} \downarrow\) Matting: \(\text{Grad} \downarrow\) Matting: \(\text{Conn} \downarrow\)
IS-Net Salient / ECCV'22 0.749 0.086 1877 0.1134 138.7 93.44 106.8
DiffDIS Salient / NeurIPS'25 0.829 0.058 1386 0.0601 72.37 68.38 52.88
HQ-SAM Interactive / NeurIPS'23 0.745 0.062 1684 0.0984 83.41 92.68 78.49
SmartMat Interactive / CVPR'24 0.759 0.064 1633 0.0438 42.11 37.47 24.06
ZIM Interactive / ICCV'25 0.774 0.057 1539 0.0384 36.41 27.88 17.49
LISA MLLM / CVPR'24 0.703 0.106 2384 0.1208 158.4 113.5 125.7
Text4Seg MLLM / ICLR'25 0.733 0.074 2072 0.0844 96.72 94.85 83.62
SegAgent MLLM / CVPR'25 0.739 0.089 1944 0.0903 87.19 92.33 84.36
Kiroshi (Ours) Agentic / ECCV'26 0.875 0.033 1218 0.0263 26.34 23.71 13.48

Ablation Study

1. Refinement Policy & Calling Iterations Evaluating the transition from vanilla box prompting to DPO-aligned policy across varying inference iterations:

Policy Configuration Iterations Seg: \(\text{max}F_\beta \uparrow\) Seg: \(S_\alpha \uparrow\) Seg: \(\text{HCE} \downarrow\) Matting: \(\text{SAD} \downarrow\) Matting: \(\text{Grad} \downarrow\) Matting: \(\text{Conn} \downarrow\)
Vanilla (Box-only) - 0.764 0.784 1601 32.16 34.84 24.72
DPO Policy 1-round 0.838 0.827 1514 28.20 25.41 18.63
DPO Policy (Default) 3-round 0.875 0.866 1218 26.34 23.71 13.48
DPO Policy 5-round 0.863 0.868 1276 25.87 24.10 13.77

2. Grid Resolution Granularity Examining the impact of grid patch resolution on spatial guidance accuracy and trajectory stability:

Grid Size Seg: \(\text{max}F_\beta \uparrow\) Seg: \(S_\alpha \uparrow\) Seg: \(\text{HCE} \downarrow\) Matting: \(\text{SAD} \downarrow\) Matting: \(\text{Grad} \downarrow\) Matting: \(\text{Conn} \downarrow\)
\(16 \times 16\) 0.862 0.854 1357 26.49 25.43 14.69
\(32 \times 32\) (Default) 0.875 0.866 1218 26.34 23.71 13.48
\(64 \times 64\) 0.820 0.809 1542 29.53 30.02 21.81

Key Findings

  • DPO Trajectory Mining Drives Substantial Performance Leaps: Compared with the vanilla box baseline (0.764 \(\text{max}F_\beta\)), incorporating even a single DPO-guided grid refinement cycle elevates \(\text{max}F_\beta\) to 0.838 and slashes SAD from 32.16 to 28.20. This validates that learning from symmetric within-context positive and negative gains enables the MLLM to focus directly on hard semi-transparent fringes and topological holes.
  • Diminishing Returns Across Iteration Horizons: Expanding from 1 to 3 refinement rounds delivers steady improvements across all metrics (\(\text{max}F_\beta\) reaches 0.875 and HCE drops from 1514 to 1218). However, extending to 5 rounds introduces slight instability (\(\text{max}F_\beta\) slightly retreats to 0.863), establishing 3 loops as the optimal convergence setting.
  • Trade-off in Grid Granularity: Coarse grids (\(64 \times 64\)) lack the spatial acuity required for narrow boundary refinement (\(\text{max}F_\beta\) drops to 0.820), while overly fine grids (\(16 \times 16\)) induce longer token sequences and occasional hallucinations. The \(32 \times 32\) resolution achieves the best balance between descriptive fidelity and sequence compactness.

Highlights & Insights

  • Decoupling "Where to Refine" and "How to Refine": Assigning high-level contextual reasoning to an MLLM policy and sub-pixel high-frequency reconstruction to an attention-masked dense predictor eliminates the need for MLLMs to generate intractable sub-pixel coordinates.
  • Autonomous Within-Context Trajectory Mining: Mining positive and negative trajectory pairs under identical intermediate predictions via quantitative quality differentials (\(\Delta Q\)) bypasses the notorious difficulty of designing dense reward functions in pixel-level vision tasks.
  • Transferable Dense Parsing Paradigm: The combination of R-RLE text grid actions with cross-attention masking is directly extensible to related dense perception tasks, including medical lesion delineation, 3D Gaussian Splatting mask pruning, and localized image inpainting.

Limitations & Future Work

  • Linear Latency Growth with Iteration Steps: Each refinement loop requires an MLLM forward pass and an action model decoding step, leading to an inference latency of 12.94 seconds per image for 3 rounds (compared to 7.42s for 1 round), restricting real-time deployment.
  • Sensitivity to Initial Target Initialization: If the initial bounding box \(B\) severely fails to capture the target under extreme occlusions, subsequent localized grid adjustments may become trapped in suboptimal local extrema without a global reset mechanism.
  • Unified End-to-End Architectures: Currently, the policy generator and action model are trained and invoked separately. Future work could investigate unified architectures driving pixel decoders directly from internal MLLM visual token representations.
  • vs ZIM / SmartMat (Interactive Matting): ZIM and SmartMat depend on continuous manual click interactions that are notoriously insensitive to negative prompts and lack natural language comprehension. Kiroshi replaces human intervention with an autonomous MLLM self-reflection loop, advancing from manual interaction to a fully automated agentic paradigm.
  • vs SegAgent / SAM4MLLM (MLLM-Guided Segmentation): SegAgent attempts to simulate human clicks in continuous pixel space, resulting in huge action spaces and unstable trajectory optimization constrained by SAM's coarse mask decoder. Kiroshi operates on structured patch-level grids and employs residual DPO to deliver sharp hair-level matting fidelity.
  • vs LISA / Text4Seg (Referring Segmentation): LISA utilizes special embedding tokens with SAM, while Text4Seg treats mask prediction as pure text generation; both struggle to handle complex transparency and thin structures. Kiroshi integrates a dedicated grid-aware action model trained with Laplacian pyramid losses, setting a new benchmark for boundary fidelity.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering formulation of high-accuracy image parsing as an agentic plan-and-act MDP using decodable text grids and residual-mined trajectory DPO]
  • Experimental Thoroughness: โญโญโญโญโญ [Introduces the comprehensive HiFiRefMS benchmark, rigorous evaluations across 13 baselines, and extensive cross-dataset analyses]
  • Writing Quality: โญโญโญโญโญ [Clear structural organization, elegant motivation, detailed formalizations, and self-contained architectural illustrations]
  • Value: โญโญโญโญโญ [Provides a practical and principled blueprint for integrating MLLM cognitive reasoning into high-precision pixel-level computer vision]