Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Data

Conference: ECCV2026
Paper: ECCV official page · Paper PDF
Area: Robotics & Embodied AI / Visual Reinforcement Learning
Keywords: sim-to-real, domain adaptation, adaptive imagination, world model, self-consistency

TL;DR

AIDA maps target images into the source state space, rolls out a frozen policy and dynamics model, uses a three-way discriminator to cut off each trajectory once it leaves the target manifold, and applies state→image→state consistency to the retained prefix; with only 50 target trajectories, it obtains the best return ratio on 6 of 7 tasks and the lowest state RMSE on all seven.

Background & Motivation

The sim-to-real problem in visual reinforcement learning comes not only from changing physics but also from the sensitivity of pixel observations to texture, lighting, and sensing conditions. Domain randomization tries to cover these changes in simulation, while domain generalization learns appearance-invariant representations, but both must anticipate what deployment will look like. Domain adaptation can inspect the actual target domain and specialize to its observed shift, yet most existing methods assume abundant target data. That assumption is often the first to fail when collecting real robot experience is expensive.

Methods such as CODAS can map target images into low-dimensional simulator states and directly reuse a policy trained on states, avoiding visual RL training that can be up to roughly 20× slower. With few target trajectories, however, trajectory alignment covers only a narrow state region. A world model appears able to fill the gap through imagination, but longer rollouts compound dynamics and mapping errors until states leave the support of target data. A fixed horizon cannot accommodate variation across tasks or even across starting states in the same episode.

The paper therefore studies a difficult but practical cross-modal setting: the source exposes low-dimensional states, the target exposes only images and actions, the two domains share state space, action space, transition dynamics, and reward, and adaptation cannot collect additional target interactions. Core idea: do not treat every model-generated transition as training data; instead, let a three-way discriminator determine a reliable boundary for each policy-conditioned trajectory and enforce state-image-state consistency only inside that boundary.

Method

Overall Architecture

AIDA has a source-learning stage and a target-adaptation stage. In Stage 1, it trains a policy \(\pi\) with SAC and fits a one-step dynamics model \(f_\omega\) in a simulator that exposes true state. In Stage 2, both are frozen; only a mapper \(q_\phi\) and observation model \(p_\theta\) are learned from a small set of target image trajectories. At deployment, \(q_\phi\) recovers a source-space state from the target image and the frozen policy acts on that state.

Target adaptation starts from trajectory-level reconstruction and adversarial alignment. AIDA initializes imagined rollouts from states inferred from target images, advances them with the frozen policy and dynamics model, and asks a separate three-way discriminator how target-like each transition remains. A rollout stops as soon as confidence falls below a threshold. Each accepted imagined state is then rendered through \(p_\theta\) and mapped back through \(q_\phi\), creating extra cycle supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source-state interaction"] --> B["Cross-modal trajectory alignment<br/>train policy and dynamics"]
    B --> C["Scarce target image trajectories<br/>image→source state"]
    C --> D["Discriminator-gated imagination<br/>per-state adaptive truncation"]
    D --> E["Imagination-data self-consistency<br/>state→image→state"]
    E --> F["Target image→state→action"]

Key Designs

1. Cross-modal trajectory alignment: replace expensive visual-policy learning with image-to-source-state adaptation

The source policy is trained directly on low-dimensional simulator states, while the target-side temporal mapper is \(q_\phi(s_t\mid \hat{s}_{t-1},a_{t-1},o_t)\). It consumes the previous inferred state, previous action, and current image rather than guessing state frame by frame. This matters because a single image may map to a plausible but dynamically inconsistent source state; trajectory context rules out that ambiguity. An autoregressive observation model \(p_\theta(o_t\mid \hat{s}_t,o_{t-1})\) reconstructs target images, and a binary alignment discriminator \(D_\eta\) brings inferred trajectories toward source-state trajectories.

The base objective is

\[ \mathcal{L}_{\mathrm{align}}=\mathcal{L}_{\mathrm{recon}}+\alpha\mathcal{L}_{\mathrm{adv}}. \]

Deployment then reduces to \(\pi(q_\phi(o))\), without relearning control from target pixels. The binary \(D_\eta\) here serves only source-versus-inferred trajectory alignment; it is distinct from the three-way \(D_\psi\) that judges imagination reliability below.

2. Discriminator-gated imagination: choose a different reliable horizon for every starting state

Starting from a target-inferred state, AIDA lets the frozen policy choose actions and recursively predicts states with the frozen dynamics model. A fixed rollout length mixes two errors: compounding multi-step model error and mapper error outside target-data support. AIDA instead trains a three-way discriminator to classify target-inferred, source, and imagined transitions. The explicit imagined class is essential because a binary source/target classifier may assign an OOD transition a probability near 0.5 even when it resembles neither domain, making an unsafe transition appear moderately reliable.

Let \(D_\psi^{\mathrm{tgt}}\) be the probability that a transition belongs to the target-inferred class. The reliable horizon is the first location below threshold \(\delta\), capped by \(K_{\max}\):

\[ K^*=\min\!\left(\min\left\{k\geq0\mid D_\psi^{\mathrm{tgt}}(\tilde{s}_{t+k},a_{t+k},\tilde{s}_{t+k+1})<\delta\right\},K_{\max}\right). \]

Only transitions for \(k=0,\ldots,K^*-1\) enter training. This is more than automatic global hyperparameter tuning: reliable rollout length varies strongly across states within one episode, so the gate decides how far each individual sample can be imagined.

3. Imagination-data self-consistency: turn reliable reachable states into dense semantic supervision

Generating state transitions alone does not improve the image-to-state mapper. For every admitted state \(\tilde{s}_{t+k}\), the observation model first renders a target-style image \(\tilde{o}_{t+k}\) conditioned on the preceding image, then the mapper recovers \(\bar{s}_{t+k}\) from that image and temporal context. AIDA penalizes disagreement with the original imagined state:

\[ \mathcal{L}_{\mathrm{sc}}=\frac{1}{K^*}\sum_{k=1}^{K^*}\left\|\bar{s}_{t+k}-\tilde{s}_{t+k}\right\|_1. \]

This is more targeted than cycle consistency on real samples alone. Policy-conditioned imagination focuses supervision on states the deployed policy is likely to reach, while the discriminator gate prevents erroneous states from training the mapper. The full adaptation objective is \(\mathcal{L}=\mathcal{L}_{\mathrm{align}}+\lambda\mathcal{L}_{\mathrm{sc}}\).

A Worked Example

Consider one target-domain Swimmer frame. The mapper combines the frame with the preceding state and action to infer a low-dimensional state. The frozen policy selects an action, the dynamics model predicts forward, and \(D_\psi^{\mathrm{tgt}}\) checks every step. In the paper's visualization, decoded bodies remain coherent before the red truncation point but become distorted and blurry beyond it. AIDA drops that suffix and renders then re-encodes only the retained states. Starting elsewhere in the same episode can produce a very different horizon, which is precisely what a fixed \(K\) cannot express.

Loss & Training

The source stage trains SAC from low-dimensional states and fits the dynamics model with one-step squared prediction error. During target adaptation, both the policy and dynamics model remain frozen. Training uses only 50 target image trajectories collected beforehand by an expert policy, one sixth of the target data used by prior CODAS work; no method receives further target interaction except the online PAD baseline. All reported results are means and standard deviations over 3 random seeds.

Key Experimental Results

Main Results

Return Ratio is \(r_{\mathrm{ratio}}=r/r^*\), where 1.0 means recovering the source expert's average return. AIDA obtains the best adapted return on 6 of 7 tasks. Walker2d is the exception: BC reaches 0.101 while AIDA reaches 0.058.

Method	HalfCheetah	Hopper	Swimmer	Walker2d	Inv. Pend.	Shadow	Fetch
BC	0.462±0.011	0.121±0.018	0.403±0.042	0.101±0.013	0.319±0.052	0.836±0.033	0.453±0.009
GAN_STACK	0.340±0.087	0.056±0.008	0.261±0.066	0.026±0.004	0.063±0.004	0.645±0.122	0.823±0.053
PAD	0.573±0.075	0.142±0.023	0.347±0.058	0.036±0.005	0.155±0.012	0.395±0.088	0.821±0.044
CODAS	0.711±0.098	0.356±0.086	0.468±0.069	0.038±0.009	0.440±0.038	0.893±0.059	0.897±0.007
AIDA	0.810±0.113	0.440±0.142	0.512±0.047	0.058±0.018	0.561±0.049	0.931±0.027	0.984±0.012

State RMSE directly measures how accurately \(q_\phi\) recovers the true proprioceptive state. The paper reports it for GAN_STACK, CODAS, and AIDA; AIDA is lowest on all seven tasks.

Method	HalfCheetah	Hopper	Swimmer	Walker2d	Inv. Pend.	Shadow	Fetch
GAN_STACK	3.006±0.153	0.862±0.059	1.360±0.272	2.525±0.317	0.328±0.042	0.452±0.029	0.025±0.009
CODAS	1.550±0.081	0.540±0.021	0.482±0.057	2.180±0.405	0.108±0.007	0.425±0.037	0.017±0.004
AIDA	1.403±0.116	0.523±0.023	0.356±0.015	1.784±0.320	0.075±0.004	0.391±0.016	0.015±0.008

Ablation Study

Comparison	Task / evidence	Paper-reported result	Interpretation
Adaptive \(K^*\)	Hopper return curves, 3 seeds	Highest among all settings	Filters low-confidence transitions per state
Fixed \(K=0\)	Hopper return curves, 3 seeds	Baseline	No imagination
Fixed \(K=5,10,20\)	Hopper return curves, 3 seeds	Comparable to or worse than \(K=0\)	Longer horizons indiscriminately admit drift
AIDA with \(\mathcal{L}_{\mathrm{sc}}\)	InvertedPendulum decoding	Angles and positions better match GT	Cycle supervision preserves physical configuration
CODAS without \(\mathcal{L}_{\mathrm{sc}}\)	InvertedPendulum decoding	Angles consistently deviate from GT	Plausible images can still encode wrong state semantics

The ablation figure provides training curves rather than exact converged values, so the table preserves the verifiable ordering and conclusion without estimating numbers from pixels. The discriminator visualization supplies a second check: decoded images are physically coherent before truncation but distorted and blurry afterward, and horizons fluctuate substantially across states in a single episode.

Key Findings

• Relative to the closest baseline, CODAS, AIDA improves Return Ratio by 0.099, 0.084, 0.044, 0.020, 0.121, 0.038, and 0.087 on HalfCheetah, Hopper, Swimmer, Walker2d, InvertedPendulum, Shadow, and Fetch, respectively.
• AIDA exceeds the unlimited-interaction image Oracle on Swimmer (0.512 vs. 0.304) and Fetch (0.984 vs. 0.975), but this does not establish a universal advantage over an oracle: the image Oracle must jointly learn representation and control, whereas AIDA reuses a stable state policy.
• On Walker2d, AIDA has the lowest RMSE but a lower return than BC. Balance-critical control amplifies small state errors, so representation error and control return are not interchangeable metrics.

Highlights & Insights

• The three-way discriminator is not merely a renamed source/target classifier. Its imagined class absorbs states that resemble neither domain, preventing an ambiguous binary probability near 0.5 from masquerading as confidence.
• Adaptive horizon selection changes world-model augmentation from “generate as much as possible” to “consume only the prefix that remains inside support.” The idea transfers naturally to model-based planning, offline RL, and robot video prediction, where a support estimator can control which synthetic data enter training.
• Self-consistency converts unlabeled imagination into representation supervision instead of directly updating the policy with potentially biased synthetic transitions. Freezing control and adapting only the perception interface reduces the error surface that scarce data must constrain.

Limitations & Future Work

• The authors explicitly note that the adapted policy does not consistently recover source-expert performance, especially on balance-critical tasks such as Walker2d. Constrained online adaptation or uncertainty-triggered target collection could close part of this gap while retaining the offline initialization.
• AIDA assumes shared transition dynamics and reward across source and target and addresses only observation shift. Real sim-to-real transfer often also changes friction, latency, and mass; future gates should distinguish perceptual OOD from dynamics OOD or jointly identify target dynamics.
• The 50 target trajectories are few but are collected by an expert policy. The paper does not test suboptimal or mixed-behavior data, nor more extreme budgets. Since the gate learns “target-likeness” from the same limited set, biased coverage may prematurely reject useful novel states.
• Evidence for fixed-horizon selection and self-consistency centers on one Hopper curve and qualitative InvertedPendulum images. Per-task numerical component ablations and sensitivity analyses for \(\delta\), \(K_{\max}\), and \(\lambda\) would make the mechanism claims stronger.

Related Work & Insights

• vs CODAS: CODAS already supplies adversarial trajectory mapping from images to privileged states. AIDA retains that cross-modal backbone but expands scarce-data coverage through gated policy-conditioned imagination and constrains new states with self-consistency. It performs better in the low-data regime at the cost of a dynamics model, observation model, and extra three-way discriminator.
• vs PAD: PAD continually updates a visual encoder at deployment through inverse-dynamics self-supervision and is designed for smaller image-to-image appearance shifts. AIDA requires no deployment interaction and tackles the larger state-to-image gap, but depends on shared dynamics and source-state access.
• vs domain randomization / generalization: DR and DG never inspect target data and must anticipate variation in advance. AIDA specializes from a small nonzero target set; it is suited to scarce-data deployment but offers no zero-shot guarantee for a completely unseen domain.

Rating

• Novelty: ⭐⭐⭐⭐☆ OOD-aware per-state truncation and self-consistency over imagined trajectories form a clear contribution, though they build on CODAS-style trajectory alignment.
• Experimental Thoroughness: ⭐⭐⭐⭐☆ The study covers 5 MuJoCo and 2 Gymnasium-Robotics tasks, five baseline families, and 3 seeds, but lacks cross-task numerical component ablations and hyperparameter sensitivity.
• Writing Quality: ⭐⭐⭐⭐☆ The problem setting, two discriminator roles, and four evaluation questions are clearly organized, although implementation details are deferred to the supplement.
• Value: ⭐⭐⭐⭐☆ The method directly addresses the practical constraint that target data are expensive and further interaction is unavailable, with a reusable principle for quality-controlling model-generated data.