Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Large Language Models and Reinforcement Learning

Conference: ECCV 2026
Paper: ECCV Official
Cached Full Text: /Users/zy/workspace/paper_cache/ECCV2026/eccv-4591.txt
Area: Multimodal VLM
Keywords: Digital Twin Representation / Visual Reasoning / Reinforcement Learning / GRPO / Multimodal Agent

TL;DR

DT-R1 introduces a unified pure reinforcement learning (GRPO) framework that trains a single LLM to autonomously orchestrate vision foundation models into digital twin (DT) representations, reasoning over continuous geometric and temporal properties via explicit chains of thought and executable Python code to resolve diverse reasoning visual tasks without task-specific architectures or supervised fine-tuning.

Background & Motivation

Embodied artificial intelligence and interactive vision-language systems demand models capable of interpreting complex visual inputs (images and videos) and resolving implicit natural language instructions into heterogeneous output formats, spanning pixel-level segmentation masks, spatial bounding boxes, textual scene summaries, and open-ended visual question answering. Prior research predominantly focused on reasoning segmentation—such as isolating an object indicated by functional descriptions like "segment the container used for holding hot beverages" rather than direct category references. However, real-world deployment requires expanding beyond segmentation masks to a broader suite of Reasoning Visual Tasks (RVTs). Despite recent progress, existing RVT methodologies are fundamentally constrained by three structural bottlenecks. First, prevalent multimodal architectures (such as LISA, VISA, and ScanReason) tokenize visual features into discrete token sequences, discarding fine-grained spatial-temporal continuity and continuous 3D geometric relationships that are indispensable for high-level qualitative and quantitative reasoning. Second, existing RVT paradigms heavily rely on supervised fine-tuning (SFT) over fixed task annotations, producing opaque outputs that lack interpretable reasoning traces and suffering severe degradation when exposed to out-of-domain data distributions. Third, current frameworks remain heavily task- and modality-specific, requiring custom decoder designs and specialized architectures for each task category.

Recent explorations in just-in-time digital twins (JiT) revealed that replacing discrete token embeddings with structured digital twin representations preserves continuous spatial-temporal context for complex reasoning across images and videos. Nonetheless, JiT operates strictly in a heuristic, zero-shot prompting configuration without systematic policy optimization, resulting in brittle tool invocations and an impractical inference latency exceeding 15 seconds per query. These persistent challenges underscore a critical research frontier: can an LLM develop unified visual reasoning capabilities purely via reinforcement learning, autonomously constructing its own digital twin representations and executing symbolic spatial-temporal operations without curated reasoning demonstrations or dedicated architectural decoders?

This paper addresses this question by decoupling perceptual feature extraction from cognitive symbolic deduction, formalizing the physical world into an explicit, multi-level digital twin representation governed by an LLM acting as the central cognitive controller. Core idea: propose DT-R1, an end-to-end reinforcement learning framework that trains an LLM using GRPO with a compound rule-based reward enforcing syntactic structure, valid DAG planning of vision foundation models, Python code executability, and output accuracy, enabling a single model to build digital twins and execute iterative code-augmented visual reasoning across four diverse RVT types and two visual modalities.

Method

Overall Architecture

DT-R1 unifies heterogeneous reasoning visual tasks into a structured, auto-regressive rollout trajectory. Given an implicit text query \(Q\) and visual data \(X = \{I^{(1)}, I^{(2)}, \dots, I^{(T)}\}\) (where \(T=1\) corresponds to an image and \(T>1\) indicates video frames), the LLM generates a multi-stage sequential stream delineated by explicit semantic tokens. Rather than executing disjoint models or appending specialized segmentation/detection heads, the framework progresses across four cohesive phases: initial query intent decomposition within <think> tags, vision foundation model dependency scheduling within <dt_plan>, structured digital twin ingestion inside <dt_rep>, iterative code-augmented spatial-temporal deduction via <think> and <execute>, and adaptive task categorization within <task> followed by target emission inside <answer>.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Implicit Text Query + Image/Video"] --> B["Stage 1: Intent Decomposition<br/>Analyze perceptual requirements in think tags"]
    B --> C["Stage 2: DT Construction Planning DAG<br/>Select models and dependencies in dt_plan"]
    C --> D["External Execution & DT Ingestion<br/>Hierarchical JSON representation injected into dt_rep"]
    D --> E["Stage 3: Iterative Code-Augmented Reasoning<br/>Alternating think deduction and execute OpenCV code"]
    E -->|Condition not met| E
    E -->|Reasoning concluded| F["Stage 4: Adaptive Task Identification<br/>Specify task category in task tags"]
    F --> G["Final Response Emission<br/>Emit masks/boxes/text in answer tags"]

Key Designs

1. Structured Rollout Trajectory and Cognitive Orchestration: Unifying Heterogeneous Visual Tasks into a Discrete Symbolic Stream

To eliminate architectural modifications across varied task requirements (such as pixel masks, bounding boxes, or textual summaries), DT-R1 defines a strict rollout sequence governed by semantic token delimiters: $$ \mathcal{Y} = \Big\langle \texttt{}\mathcal{R}0\texttt{}, \texttt{}\mathcal{G}\texttt{}, \texttt{}\mathcal{D}\texttt{}, \big(\texttt{}\mathcal{R}_i\texttt{}, \texttt{}\mathcal{C}_i\texttt{}, \texttt{}\mathcal{O}_i\texttt{}\big) \Big\rangle $$ Here, }^m, \texttt{}\mathcal{T}\texttt{}, \texttt{}\mathcal{S}\texttt{\(\mathcal{R}_0\) contains initial query analysis, and \(\mathcal{G}\) represents the dependency graph for perceptual models. Upon generating </dt_plan>, text generation pauses while external vision foundation models execute according to \(\mathcal{G}\), inserting the resulting digital twin \(\mathcal{D}\) between <dt_rep> and </dt_rep>. The LLM then conducts \(m\) iterations of explicit reasoning \(\mathcal{R}_i\), interleaved with Python execution cycles (\(\mathcal{C}_i\) and \(\mathcal{O}_i\)). Finally, the model explicitly identifies the task type \(\mathcal{T} \in \{\text{reasoning segmentation}, \text{grounding}, \text{summarization}, \text{VQA}\}\) within <task> tokens and emits the corresponding output format \(\mathcal{S}\) inside <answer>. For segmentation, \(\mathcal{S}\) supplies object instance identifiers paired with mask file paths; for grounding, normalized bounding box coordinates; and for summarization/VQA, natural language text.

2. DAG-Formulated Digital Twin Representation Planning: Lightweight Hierarchical Abstraction Preserving Physical Metrics

Rather than feeding uncompressed point clouds or thousands of spatial visual tokens into the language model, DT-R1 trains the LLM to output a directed acyclic graph (DAG) \(\mathcal{G} = (V, E)\) in JSON notation specifying the prerequisite execution chain of vision foundation models. Nodes \(V\) denote available specialists (SAM2 for instance masks \(\mathcal{M}(t) = \{m_i^{(t)}\}\), DepthAnything2 for continuous metric depth \(Z(t)\), OWLv2 for open-vocabulary detection, and Qwen2.5-VL for multi-level semantics), while edges \(E\) enforce input-output dependencies (e.g., {"DepthStats": ["SAM2", "DepthAnything2"]}). The assembled digital twin \(\mathcal{D}\) captures three physical tiers: - Video-level: Global narrative summaries, environment context, and temporal scope; - Frame-level: Frame-indexed spatial layouts, camera viewpoint descriptions, and temporal timestamps; - Instance-level: Per-object mask file paths, temporal centroid trajectories, visual semantic embeddings, and spatial depth statistics including mean instance depth \(\mu_i^{(t)} = \frac{1}{|m_i^{(t)}|}\sum_{p \in m_i^{(t)}} Z^{(t)}(p)\) and standard deviations. This hierarchical structuring compresses continuous multi-frame spatial-temporal information into an ultra-compact payload (<1MB per video, ~2000 language tokens), bypassing the quadratic context explosion typical of visual tokenizers.

3. Tool-Integrated Python Code Execution: Mitigating Spatial-Temporal Hallucinations via Symbolic Computation

Language models suffer inherent numerical hallucinations when calculating metric Euclidean distances, bounding box overlaps, or multi-frame motion velocities purely through mental arithmetic. When spatial or temporal operations arise in reasoning step \(\mathcal{R}_i\), DT-R1 autonomously synthesizes executable Python code \(\mathcal{C}_i\) leveraging libraries like OpenCV and NumPy inside <execute> tags. The code operates directly over the JSON fields in \(\mathcal{D}\), calculating quantitative spatial metrics (e.g., 3D relative distances from instance mean depths \(\mu_i^{(t)}\)) or evaluating frame-to-frame displacement vectors. The execution engine captures runtime outputs and appends clean execution results \(\mathcal{O}_i\) into <results>, pruning verbose system stack traces and file paths down to the final error message line upon failure, which incentivizes self-correction within subsequent reasoning loops.

4. Dual-Component Rule-Based Reward Design: Closing the RL Loop Without Neural Reward Models

To circumvent the computational overhead and reward-hacking vulnerabilities of parameterized reward models, DT-R1 formulates a pure rule-based objective balancing format integrity and execution precision: \(R(\mathcal{Y}) = \alpha \cdot R_{\text{format}}(\mathcal{Y}) + \beta \cdot R_{\text{accuracy}}(\mathcal{Y})\). - Format Reward \(R_{\text{format}}(\mathcal{Y}) = R_{\text{token}}(\mathcal{Y}) + R_{\text{dag}}(\mathcal{G})\): \(R_{\text{token}}\) inspects the mandatory presence and valid sequential pairing of all special delimiters (+1 for flawless layout, -1 for missing/malformed pairs). \(R_{\text{dag}}(\mathcal{G})\) enforces architectural validity: $$ R_{\text{dag}}(\mathcal{G}) = \mathbb{I}[\text{Valid Format}] \cdot \mathbb{I}[\text{Acyclic}] \cdot \mathbb{I}[\text{Valid Dependencies}] - 0.5 $$ yielding +0.5 if \(\mathcal{G}\) is a strictly acyclic graph referencing legitimate vision models with valid edges, and -0.5 otherwise. - Accuracy Reward \(R_{\text{accuracy}}(\mathcal{Y}) = R_{\text{exec}}(\mathcal{C}) + R_{\text{task}}(\mathcal{T}) + R_{\text{result}}(\mathcal{S})\): \(R_{\text{exec}}\) provides 0 for bug-free Python execution and -0.5 for syntax or runtime failures; \(R_{\text{task}}\) rewards correct classification of the underlying task with +0.25 (0 otherwise); \(R_{\text{result}}\) assigns +1 for accurate predictions and -1 for incorrect outputs (for segmentation and grounding, correct predictions require an \(\text{IoU} > 0.5\) against ground truth annotations; for open-ended VQA and summarization, an LLM-as-a-judge protocol verifies factual consistency against reference answers).

A Worked Example

Consider an intricate video reasoning segmentation prompt: "Segment the part of the puppy that it lifts into the air while standing on the checkered blanket and reaching out as if to grasp something, before standing still and barking." 1. Decomposition (<think>): The model breaks down the temporal constraints (standing on blanket \(\rightarrow\) lifting paw \(\rightarrow\) stationary barking) and identifies that temporal motion tracking, depth estimation, and instance segmentation are simultaneously required. 2. DAG Formulation (<dt_plan>): Generates the tool dependency: {"SAM2": [], "DepthAnything2": [], "DepthStats": ["SAM2", "DepthAnything2"], "SemanticAnalysis": ["SAM2"]}. 3. Representation Ingestion (<dt_rep>): Vision foundation models execute, populating \(\mathcal{D}\) with frame-indexed masks, instance depth averages, and semantic action logs across the video sequence. 4. Symbolic Verification (<execute> & <results>): The LLM writes a Python script computing the vertical coordinate displacement of puppy limbs across temporal intervals, detecting that front_left_paw undergoes maximum upward elevation (+42.6 pixels) during the reaching phase. 5. Output Delivery (<task> & <answer>): The LLM designates the task type as reasoning segmentation and outputs the exact file path to the corresponding binary mask.

Loss & Training

DT-R1 utilizes Group-Relative Policy Optimization (GRPO) to train DeepSeek-R1-Distill-Qwen-7B. For each training instance \((X, Q, T)\), the model samples \(K=8\) candidate rollouts \(\{\mathcal{Y}_1, \dots, \mathcal{Y}_K\}\), updating policy weights based on normalized relative advantages derived from \(R(\mathcal{Y}_k)\). Critically, tokens within <dt_rep> (externally constructed digital twins \(\mathcal{D}\)) and <results> (runtime execution logs \(\mathcal{O}_i\)) are strictly masked out during loss calculation and back-propagation, preventing the policy from fitting non-differentiable external environment artifacts. Optimization uses LoRA (rank 16) with DeepSpeed on 16 NVIDIA RTX 4090 (24GB) GPUs using the TRL library, configured with a batch size of 32, a learning rate of \(5 \times 10^{-5}\), and achieving stable convergence in 8 epochs.

Key Experimental Results

Main Results

DT-R1 underwent comprehensive evaluation across six academic benchmarks encompassing image and video modalities across four RVT categories: in-domain datasets (ReasonSeg, ReVOS) and out-of-domain benchmarks (LLM-Seg40K, JiTBench, RVTBench). The representative performance comparisons are summarized below:

Benchmark / Task	Metric	DT-R1 (Ours)	Prior SOTA (JiT / GroundMore / LISA-13B)	Gain
JiTBench Level 1 (Basic)	J-score / F-score	0.837 / 0.850	0.792 / 0.806 (JiT)	+4.5% / +4.4%
JiTBench Level 2 (Intermediate)	J-score / F-score	0.808 / 0.845	0.766 / 0.801 (JiT)	+4.2% / +4.4%
JiTBench Level 3 (Advanced)	J-score / F-score	0.785 / 0.816	0.747 / 0.777 (JiT)	+3.8% / +3.9%
RVTBench Video Segmentation Level 1	Region Similarity \(\mathcal{J}\)	0.688	0.515 (JiT) / 0.466 (LISA-13B)	+17.3%
RVTBench Video Segmentation Level 4 (Expert)	Region Similarity \(\mathcal{J}\)	0.609	0.448 (JiT) / 0.411 (LISA-13B)	+16.1%
RVTBench Video Grounding Level 1	cIoU / gIoU	0.670 / 0.651	0.515 / 0.497 (GroundMore)	+15.5% / +15.4%
RVTBench Video Grounding Level 4 (Expert)	cIoU / gIoU	0.602 / 0.583	0.466 / 0.448 (GroundMore)	+13.6% / +13.5%
RVTBench Video VQA Level 4 (Expert)	BLEU-4 / CIDEr	0.642 / 0.789	0.462 / 0.712 (Video-R1-7B)	+18.0% / +0.077
ReasonSeg Image Segmentation (Long Query)	gIoU / cIoU	0.812 / 0.834	0.683 / 0.701 (JiT)	+12.9% / +13.3%
ReVOS Video Segmentation (Reasoning Split)	J-score / F-score	0.814 / 0.805	0.618 / 0.652 (JiT)	+19.6% / +15.3%

Ablation Study

A systematic component ablation conducted on JiTBench Level 2 demonstrates the precise impact of each reward formulation:

Configuration	Format Rewards (\(R_{\text{token}}, R_{\text{dag}}\))	Code Exec Reward (\(R_{\text{exec}}\))	Task & Accuracy Rewards (\(R_{\text{task}}, R_{\text{result}}\))	\(\mathcal{J}\)-score	\(\mathcal{F}\)-score	Training Success Rate (%)
Full DT-R1	✓	✓	✓	0.808	0.845	94.2%
w/o Format rewards	✗	✓	✓	0.412	0.438	31.5%
w/o DAG validation	Partial (\(R_{\text{token}}\) only)	✓	✓	0.687	0.712	78.3%
w/o Code execution	✓	✗	✓	0.724	0.756	85.7%
w/o Accuracy rewards	✓	✓	✗	0.523	0.548	62.1%
Only task accuracy	✗	✗	✓	0.298	0.315	22.8%

In addition, individual ablation of vision foundation models reveals significant performance dependencies: removing SAM2 induces a 12.8% decrease in \(\mathcal{J}\) (dropping to 0.684), removing DepthAnything2 lowers \(\mathcal{J}\) by 5.6% (to 0.756), removing OWLv2 degrades it by 8.0% (to 0.732), and removing Qwen2.5-VL incurs a 3.4% penalty (to 0.778).

Key Findings

• Format Constraints Dictate RL Convergence: Omitting format verification causes training stability to collapse to 31.5%, with the model repeatedly generating corrupted graphs and syntax anomalies. Isolating DAG validation alone yields an improvement of over 12% in \(\mathcal{J}\) score (0.808 vs. 0.687), proving that structured graph constraints are fundamental when training language policies to orchestrate multi-tool pipelines.
• Adaptive Reasoning Termination and Exceptional Efficiency: Although the maximum reasoning iteration cap was set to \(m=10\), empirical analysis indicates that reasoning performance plateaus at 5 iterations. DT-R1 dynamically adapts, averaging only 3.8 iterations per sample. Because the digital twin compresses rich continuous video into under 1MB, DT-R1 exhibits an inference speed of 1.54s per query—nearly \(10\times\) faster than JiT's 15.24s.
• Robust Out-of-Domain Generalization: On uncurated out-of-domain benchmarks, DT-R1 achieves a gIoU of 0.641 on LLM-Seg40K (substantially higher than JiT's 0.485 and LISA-13B's 0.392). Similarly, on video grounding within RVTBench, DT-R1 surpasses GroundMore (0.602 vs. 0.466 cIoU on Level 4), confirming that symbolic digital twin reasoning generalizes far better than parameter-bound neural decoders.

Highlights & Insights

• Decoupled Perception and Cognitive Reasoning via Pure RL: By delegating perceptual feature extraction to off-the-shelf foundation models and condensing the environment into a structured digital twin, the LLM functions purely as a symbolic and algorithmic solver. This architectural separation preserves spatial-temporal geometry while bypassing catastrophic forgetting.
• Gradient-Masked Policy Optimization: Masking out externally populated environment tokens (<dt_rep> and <results>) during GRPO gradient computation ensures that policy updates optimize purely for internal model reasoning, tool planning, and code synthesis without suffering gradient contamination from non-differentiable environment data.
• Elimination of Supervised Chain-of-Thought Demonstrations: DT-R1 proves that sophisticated multi-modal tool orchestration and multi-step algorithmic visual reasoning can emerge autonomously through pure reinforcement learning guided by deterministic syntactic and task-level rules.

Limitations & Future Work

• Perceptual Bottleneck and Cascading Foundation Errors: An error analysis on ReVOS reveals that 32% of failure cases stem from incomplete or fragmented segmentation masks produced by SAM2 in occluded or heavily clustered scenes, which misleads subsequent Python calculation.
• Perceptual Foundation Runtime Overhead: Constructing the digital twin representation still requires executing multiple heavy foundation models (SAM2, DepthAnything2, OWLv2) on an RTX 4090 (~0.3 min per video), suggesting the need for lightweight task-distilled specialists or feature caching mechanisms.
• Sensory Modality Boundaries: The current digital twin representation is restricted to visual, depth, and spatial-temporal data, presenting opportunities to integrate auditory and physical sensory streams into unified multimodal representations.

Related Work & Insights

• vs LISA / VISA (End-to-End Decoder Architectures): LISA and VISA inject specialized <seg> tokens into MLLMs and train downstream decoders via SFT, which incurs catastrophic forgetting, lacks explicit reasoning chains, and cannot generalize across varied task outputs; DT-R1 preserves frozen foundation decoders, using pure RL to coordinate symbolic digital twins across four heterogeneous output formats.
• vs JiT (Zero-Shot Digital Twins): While JiT introduced the concept of digital twin representations for visual reasoning, it relies on static heuristic prompting, suffers brittle tool execution, and requires 15.24s per query; DT-R1 introduces policy learning via GRPO, delivering superior accuracy while reducing inference latency to 1.54s.
• vs Seg-Zero / Seg-R1 (RL for Visual Segmentation): Prior RL-based reasoning methods are confined strictly to 2D image reasoning segmentation; DT-R1 extends the RL paradigm across images and videos to encompass segmentation, grounding, summarization, and VQA in a single unified architecture.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First framework to merge GRPO reinforcement learning with structured digital twin representations and code execution for unified visual reasoning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across 6 in-domain and out-of-domain benchmarks, spanning two modalities, four task formats, and extensive reward and model ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual formulation, clear mathematical definitions, and detailed empirical reporting.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly extensible, generalizable blueprint for agentic visual reasoning, freeing multimodal AI from bespoke architectures and costly reasoning chain annotations.