Skip to content

title: >- [Paper Note] Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models description: >- [ECCV 2026][Model Compression][World Models] Proposes DILLO, a distilled language-action world model that predicts future semantic descriptions and safety verdicts directly from policy latents without visual simulation, achieving a 14x speedup. tags: - ECCV 2026 - Model Compression - Language-Action World Models - Knowledge Distillation - Agent Steering date: 2026-09-19 content_hash: 52b630db387ae015

Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/MaxPappa/DILLO
Area: Model Compression
Keywords: language-action world models, cross-modal knowledge distillation, rejection sampling steering, embodied AI safety, latent sufficiency

TL;DR

Addressing the prohibitive latency of conventional visual world models in real-time control, this paper introduces DILLO, a vision-free Language-Action World Model that predicts semantic outcome descriptions and binary safety verdicts directly from policy latents and proposed action chunks via cross-modal distillation, delivering a 14× speedup and substantial proactive steering gains.

Background & Motivation

Deploying AI agents in safety-critical domains such as robotic manipulation and autonomous navigation demands the capability to anticipate action consequences before physical execution. Standard agents, however, operate largely as "black boxes," emitting control signals reactively without foresight. While classical Model Predictive Control (MPC) and model-based reinforcement learning provide foresight by simulating system dynamics forward, applying these principles within high-frequency control loops exposes a severe tension between predictive foresight and control latency.

Existing approaches to agent reliability remain polarized. Reactive post-hoc analysis frameworks (such as AHA, REFLECT, and SAFE) leverage vision-language models (VLMs) to diagnose errors after they occur; while computationally lightweight, retrospective "failure autopsies" cannot prevent irreversible physical damage. Conversely, proactive visual world models (e.g., Dreamer variants and Forewarn) anticipate hazards by hallucinating high-dimensional future states or latent video frames. However, such visual simulation incurs prohibitive compute overhead—recent steering frameworks report latencies of nearly 4 seconds per decision step on an enterprise RTX A6000 GPU, rendering them fundamentally impractical for real-time control loops.

This paper challenges the foundational premise that visual rendering is necessary for proactive failure prevention. A policy's encoder is already explicitly optimized to retain task-critical physical dynamics, including object geometry, relative distances, and contact states. If this internal latent state already contains the critical information required to evaluate an action's viability, generating and re-encoding raw pixels represents redundant overhead. Based on this insight, the authors formalize the Latent Sufficiency Hypothesis, demonstrating that a policy's internal latent representation paired with candidate action chunks serves as a sufficient statistic for anticipating interaction outcomes. Core idea: Shift the proactive control paradigm from "simulate-then-act" to "describe-then-act" via DILLO, a distilled language-action world model that maps policy latents directly to language outcomes and binary verdicts, eliminating visual simulation overhead to enable real-time rejection sampling steering.

Method

Overall Architecture

DILLO (DIstiLLed Language-ActiOn World Model) decouples semantic anticipation from visual simulation through a three-part framework: offline privileged teacher annotation, cross-modal distillation, and online latent rejection sampling. During training, a simulator-privileged VLM Teacher accesses complete ground-truth execution rollouts—including RGB frames, 6-DoF poses of the end-effector and objects, and task completion signals—to generate a natural language interaction preview \(d^T\) and a binary outcome verdict \(c^T \in \{\text{Positive}, \text{Negative}\}\). A lightweight student LLM (DILLO Student) receives only the frozen policy's latent state \(z_t\) and a proposed action chunk \(a_{t:t+k}\), mapped into the LLM embedding space via learnable linear projectors, completely bypassing visual encoders and pixel rendering. At inference time, the base policy encodes the visual observation once and proposes \(N\) candidate action chunks; DILLO evaluates all candidates in a single batched forward pass, filtering negative actions and executing the first verified positive trajectory.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current Observation ot and Action Chunks at:t+k"] --> B["Privileged Teacher Offline Annotation<br/>GT poses and temporal rollouts yield descriptions and verdicts"]
    B --> C["Cross-Modal Distillation<br/>Project policy latents and action chunks into LLM space"]
    C --> D["Three-Stage Progressive Training<br/>Projector alignment → Description distillation → Verdict optimization"]
    D --> E["Latent Rejection Sampling Steering<br/>Single-pass batched evaluation filters negative actions"]
    E -->|Approved action| F["Execute Safe Physical Action"]

Key Designs

1. Cross-Modal Distillation via the Latent Sufficiency Hypothesis: Bypassing Visual Simulation

This design targets the prohibitive multi-second latency of pixel-based world models. The authors hypothesize and empirically confirm that the mutual information between the policy's internal representation tuple \((z_t, a_{t:t+k})\) and future outcome \(y_{t+k} = (d_{t+k}, c_{t+k})\) closely matches the mutual information provided by the full observation sequence \(o_{t:t+k}\):

\[I(o_{t:t+k}; y_{t+k}) \approx I((z_t, a_{t:t+k}); y_{t+k})\]

Whether trained with reinforcement learning (e.g., SAC) or imitation learning (e.g., ACT), the policy encoder \(E_\pi(o_t)\) inherently captures spatial geometries and contact dynamics. Even in failure regimes, \(z_t\) encapsulates the "context of failure" (such as feature misalignment or epistemic uncertainty). DILLO maps this continuous representation into the language space using linear projection modules \(P_z: \mathbb{R}^{D_z} \to \mathbb{R}^{D_{\text{emb}}}\) and \(P_a: \mathbb{R}^{D_a} \to \mathbb{R}^{D_{\text{emb}}}\), concatenating them with a task instruction:

\[X_{\text{in}} = \text{[TASK PROMPT]} \oplus P_z(z_t) \oplus P_a(a_{t:t+k})\]

The student utilizes Gemma-based architectures (Gemma-1B-it or Gemma-VLM-4B-it stripped of its 417M SigLIP vision encoder), performing autoregressive language decoding without processing any visual tokens at inference time.

2. Failure-Aware Dataset Curation: Spectrum of Competency and Perception Noise Injection

Training a discriminator exclusively on converged expert rollouts and trivial failures creates severe blind spots for subtle, compounding execution errors. DILLO establishes a balanced failure distribution using two complementary techniques. First, "Spectrum of Competency Sampling" collects rollouts across policy checkpoints saved at distinct proficiency stages: low-competency checkpoints (20% success rate) provide erratic exploration, mid-competency checkpoints (50% success rate) generate nuanced "near-miss" failures, and high-competency checkpoints (80% success rate) yield hard negatives through rare execution slips. Second, "Perception Noise Injection" introduces Gaussian noise \(\epsilon \sim \mathcal{N}(0, \sigma^2)\) directly into observations during data collection, inducing trajectory drift and compounding physical errors from an otherwise competent policy manifold, faithfully modeling real-world Sim-to-Real covariate shifts.

3. Three-Stage Progressive Curriculum Training: Mitigating Modality Collapse

Mapping continuous control embeddings directly into discrete language tokens often destabilizes pre-trained LLM weights. DILLO addresses this through a phased curriculum: - Stage 1 (Projector Alignment): The LLM backbone \(\theta_{\text{LLM}}\) is kept frozen while only the linear projection layers \(\{P_z, P_a\}\) are optimized via standard next-token prediction on teacher descriptions, ensuring continuous latents become valid semantic prompt embeddings. - Stage 2 (Description Distillation): Low-Rank Adaptation (LoRA) is introduced into the LLM backbone, jointly optimizing LoRA parameters and projectors under a description reconstruction loss \(\mathcal{L}_{\text{Desc}}\) to instill physical transition dynamics. - Stage 3 (Verdict Multi-Task Optimization): The model is tasked with generating the descriptive preview followed by an explicit verdict token \(\hat{c} \in \{\text{Positive}, \text{Negative}\}\), optimized jointly via:

\[\mathcal{L}_{\text{Total}} = \mathcal{L}_{\text{Desc}} + \lambda \cdot \mathcal{L}_{\text{CE}}(c^T, \hat{c})\]

where \(\mathcal{L}_{\text{CE}}\) is cross-entropy against the ground-truth teacher verdict, enforcing semantic alignment between text rationales and task success.

4. Proactive Latent Rejection Sampling: Controller-Agnostic Zero-Visual-Overhead Filter

During online deployment, DILLO functions as an external runtime filter. At decision step \(t\), the base policy computes the latent state \(z_t = E_\pi(o_t)\) once and independently samples a budget of \(N\) candidate action chunks \(\{a^i_{t:t+k}\}_{i=1}^N\) (\(N=5\)). DILLO evaluates all \(N\) proposals in a single batched forward pass, outputting candidate descriptions \(d^i_{t+k}\) and verdicts \(\hat{c}^i_{t+k}\). The controller commits to the first verified positive candidate. If no candidate passes the filter, the system falls back to executing the initial proposal to maintain control continuity. Because no visual simulation or neural rendering is invoked across candidates, decision cycles complete within hundreds of milliseconds.

A Worked Example

Consider a robotic arm tasked with opening a drawer in the LIBERO benchmark: 1. Latent Encoding and Candidate Sampling: At control step \(t\), the camera captures the workspace, and the ACT policy encoder extracts latent \(z_t\). The policy samples \(N=5\) candidate 20-step action chunks \(a^1\) through \(a^5\). 2. Batched DILLO Screening: - Candidate 1: DILLO decodes description "The gripper descends too rapidly, drifting leftward and colliding with the outer drawer rim," predicting verdict Negative. - Candidate 2: DILLO decodes description "The end-effector hesitates in place without closing distance to the handle," predicting verdict Negative. - Candidate 3: DILLO decodes description "The arm advances along the central axis, aligns gripper orientation with the handle, and initiates contact," predicting verdict Positive. - Candidates 4 and 5 are evaluated in the same forward tensor batch. 3. Execution: The steering layer intercepts candidates 1 and 2, executing verified candidate 3 on physical motors. The entire steering evaluation takes 0.26 seconds, while the human operator interface displays the prospective trajectory description in real time.

Key Experimental Results

Main Results

The framework was evaluated on the single-task RL benchmark MetaWorld (Soccer, Sweep-Into, Drawer-Open) and the multi-task imitation learning benchmark LIBERO (Goal, Object, Spatial, LIBERO-10, LIBERO-90). Metrics comprise Text-to-Observation Fidelity (T2O), LLM-judged semantic reasoning score (fact recall via Qwen2.5-32B-Instruct), Safety Verdict Accuracy, and Episode Success Rate (SR, evaluated over 20 episodes per task).

Main steering performance and classification accuracy are summarized below:

Environment & Task Base Policy SR DILLO-1B SR DILLO-4B SR 1B Verdict Acc. 4B Verdict Acc.
MetaWorld Soccer 65.0% 75.0% 80.0% 82.7% 87.4%
MetaWorld Sweep-Into 80.0% 90.0% 85.0% 97.7% 92.1%
MetaWorld Drawer-Open 75.0% 70.0% 90.0% 80.6% 94.7%
LIBERO Goal 86.0% 87.5% 89.0% 80.0% 87.5%
LIBERO Object 82.0% 86.5% 88.0% 79.8% 81.2%
LIBERO Spatial 70.0% 76.0% 73.0% 77.8% 86.7%
LIBERO 10 64.0% 71.5% 74.0% 88.6% 92.5%
LIBERO 90 71.4% 68.6% 70.5% 86.9% 85.7%

Note: DILLO-4B achieves an average safety verdict accuracy of 91.4% across tasks without any visual inputs at test time.

Ablation Study

To validate the Latent Sufficiency Hypothesis, DILLO was evaluated against vision-based baselines: Zero-Shot (ZS) and Few-Shot (FS) models receiving current image \(o_t\), and Few-Shot Captioning (CAP Oracle) which receives both \(o_t\) and the ground-truth future frame \(o_{t+k}\):

Baseline / Method MetaWorld Soccer (1B / 4B) MetaWorld Drawer-Open (1B / 4B) LIBERO Goal (1B / 4B) LIBERO Spatial (1B / 4B)
Pixel Zero-Shot (ZS) 0.398 / 0.540 0.419 / 0.553 0.287 / 0.241 0.341 / 0.194
Pixel Few-Shot (FS) 0.495 / 0.501 0.626 / 0.555 0.262 / 0.422 0.546 / 0.563
CAP Oracle (Ground-Truth Future Frame) 0.477 / 0.626 0.612 / 0.609 0.723 / 0.905 0.794 / 0.902
DILLO (Vision-Free Latent Conditioning) 0.518 / 0.586 0.706 / 0.705 0.727 / 0.710 0.668 / 0.656

Inference latency and hardware benchmarks (evaluating DILLO on an RTX 3090 against Forewarn on an RTX A6000):

Architecture Evaluation Hardware Decision Latency Speedup vs Forewarn Visual Compute Overhead
Forewarn (Visual VLM-in-the-loop) NVIDIA RTX A6000 (48GB) ~3.70 s 1.0× (Baseline) High (Heavy video/visual backbone)
DILLO-4B (Language-Action World Model) NVIDIA RTX 3090 (24GB) 0.38 s ~10× Zero (Pure language decoder)
DILLO-1B (Lightweight World Model) NVIDIA RTX 3090 (24GB) 0.26 s ~14× Zero (Consumer GPU friendly)

Key Findings

  • Latent Embeddings Outperform Raw Pixel Inputs: DILLO significantly outperforms pixel-conditioned ZS and FS baselines in physical dynamic fidelity across benchmarks. On MetaWorld Drawer-Open, DILLO-4B's T2O fidelity (0.705) even surpasses the CAP Oracle (0.609) which had access to actual future images. Because task-trained policy embeddings filter out visual noise and perspective ambiguities, \(z_t\) acts as a cleaner predictive statistic.
  • Enhanced Task Success and Episode Compression: DILLO-4B boosts success rates by up to 15 percentage points (65.0% to 80.0% on Soccer). On LIBERO-10, DILLO-1B increases success from 64.0% to 71.5% while reducing average episode length by 36% (from 390 simulation steps down to 250 steps), curbing erratic recovery loops.
  • Cross-Suite Generalization vs. Gating Calibration: Transferring a LIBERO-90-trained DILLO model to steer a LIBERO-10 ACT policy without retraining improves success from 64.0% to 69.5%, confirming transferable semantic safety representations. However, on LIBERO-90, applying a single global rejection threshold led to slight over-rejection (-4.4%); adjusting gating thresholds based on task difficulty reversed this to a +17.5% gain on hard tasks, highlighting the importance of adaptive gating.

Highlights & Insights

  • Paradigm Shift from Visual Simulation to Semantic Foresight: Bypasses heavy pixel rendering by decoding language previews and verdicts directly from policy latent states, achieving a 14× speedup and bringing proactive world models into real-time robot control.
  • Asymmetric Teacher-Student Distillation: A privileged offline VLM teacher leverages ground-truth 3D poses and multi-view rollouts to eliminate spatial hallucinations, while the online student operates as a lightweight, vision-free language decoder running on consumer hardware.
  • Non-Invasive Safety Architecture: Operates as a plug-and-play outer-loop filter without altering base policy weights, making it universally applicable across RL and imitation learning frameworks while providing human-readable explanations.

Limitations & Future Work

  • Policy Encoder Coupling: The projection layers \(\{P_z, P_a\}\) are bound to the specific latent manifold of the underlying policy; switching policy backbones requires re-aligning the projectors.
  • Dependence on Privileged 3D State Data: Offline teacher distillation requires ground-truth 6-DoF poses. Substituting these with real-world vision estimators (e.g., FoundationPose) may introduce perceptual noise that degrades pseudo-label quality.
  • Spatial Coordinate Conditioning: On benchmarks with extreme spatial diversity (such as LIBERO-Spatial), relying strictly on semantic descriptions without explicit continuous 3D target coordinates limits fine-grained geometric steering.
  • vs Forewarn (VLM-in-the-Loop Steering): Forewarn simulates latent futures and evaluates them using full VLM architectures, requiring 3.7 seconds per decision; DILLO verifies the Latent Sufficiency Hypothesis, stripping away visual computation to achieve a 14× speedup.
  • vs AHA / REFLECT (Post-Hoc Failure Analyzers): Prior VLM frameworks follow an "act-then-describe" paradigm, diagnosing failures after execution; DILLO establishes a "describe-then-act" paradigm that actively screens and blocks unsafe action candidates before physical actuation.
  • vs ECoT (Embodied Chain-of-Thought): ECoT exposes the agent's internal reasoning steps; DILLO is outcome-driven, predicting external physical consequences and interaction outcomes, which decouples safety verification from policy reasoning.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the Latent Sufficiency Hypothesis and establishes the vision-free "describe-then-act" paradigm, redefining efficiency in proactive world modeling.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across single-task RL and multi-task IL suites, combining physical fidelity metrics, LLM-based fact verification, latency benchmarks, and cross-suite transfer.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorously structured narrative, clear theoretical and empirical exposition, and meticulous architectural diagrams.
  • Value: ⭐⭐⭐⭐⭐ Delivers an interpretable, real-time safety layer for embodied AI with zero visual inference overhead.