Skip to content

MuSix: Multi-scale Mixture of World Models for Embodied Agents in Evolving Environments

Conference: ECCV 2026
arXiv: 2607.00457
Code: None
Area: Robotics / Embodied AI
Keywords: World Models, Mixture of Experts, Embodied AI, Scale-aware Routing, Test-time Adaptation

TL;DR

This paper proposes the MuSix framework, which explicitly decouples "scale determination" from "model selection" in world models via a two-stage scale-aware routing mechanism driven by experiential distance inspired by Construal Level Theory (CLT). It introduces scale-dependent forgetting rates and gated cross-scale knowledge transfer. This allows low-level immediate knowledge to refresh rapidly while high-level abstract knowledge remains stable, achieving multi-scale adaptive evolution for embodied agents in dynamic environments.

Background & Motivation

Embodied agents in the real world must handle reasoning tasks across multiple scales simultaneously: low-level tasks require perceiving physical interaction details (such as object manipulation and obstacle avoidance) and responding to local dynamic changes in milliseconds, while high-level tasks demand understanding task intent, planning long-horizon action logic, and devising rescue strategies in disaster scenarios. Existing Vision-Language Model (VLM)-based methods, such as SayCanPay and FLARE, have made significant progress in complex instruction following by combining the reasoning capabilities of language models with environmental affordance signals. However, these methods are typically built on static environment assumptions or single model components. When environmental conditions change, either the entire system must be retrained, or it fails to make targeted adaptive adjustments at the appropriate granularity, as a single model cannot simultaneously accommodate fine-grained local predictions and robust high-level abstractions.

The Mixture of Experts (MoE) architecture provides a natural solution to this issue: its modular expert selection can accommodate knowledge of different natures, and its selective activation mechanism enables the system to update only the relevant experts without disturbing other components, effectively mitigating catastrophic forgetting. However, translating this potential into continuous adaptation capabilities in dynamic environments poses two fundamental challenges. First, standard MoE routing operations lack an explicit concept of "scale"—the router's decisions are not bound to any identifiable knowledge granularity, making it impossible to precisely update experts of a specific scale during testing, which leads to the "routing mixture challenge". Second, the decay rates of knowledge at different scales are vastly different: low-level knowledge about local physical dynamics becomes obsolete rapidly as the environment changes, whereas high-level knowledge regarding abstract rules remains relatively stable. However, conventional MoE applies a uniform update strategy to all experts, failing to respect this discrepancy and resulting in the "evolution challenge".

The core insight of this paper is that the common root of these two challenges lies in the absence of an explicit, quantifiable "scale axis" corresponding to cognitive levels in MoE. Drawing inspiration from Construal Level Theory (CLT) in cognitive science—which posits that psychological distance world-maps to the abstractness of human cognition, where near distances trigger concrete, low-level cognitive representations and far distances trigger abstract, high-level representations—the authors operationalize this principle into "experiential distance". This metric quantifies the novelty of the current situation relative to the agent's accumulated experience. Core Idea: This paper proposes MuSix, which organizes the world models of embodied agents into multi-scale groups. Through a two-stage scale-aware routing mechanism (where the Meta-Router first determines weights on a continuous scale space based on experiential distance, and then the Base Router within each scale selects the world models), it achieves explicit scale-aware mixture. Simultaneously, it introduces intra-scale dependent forgetting rates (fast decay at low levels, slow decay at high levels) and gated cross-scale bidirectional knowledge transfer, allowing world models to continuously evolve at their appropriate paces during testing.

Method

Overall Architecture

The core idea of MuSix is to organize world models into multiple scale groups based on knowledge granularity, achieving scale-aware model mixture and continuous evolution through two-stage routing.

The inputs to the framework are the current observation \(o_t\) and action \(a_t\). First, a frozen pre-trained encoder \(\phi\) (CLIP-ViT) maps physical observations to the embedding space. Cumulative multi-step experiences \(E_{<t}\) are modeled as a multivariate Gaussian distribution. The Mahalanobis distance between the embedding of the current observation and this experience distribution defines the experiential distance \(\delta\). The Meta-Router receives \(\delta\) and outputs a weight function \(w_{\text{MR}}(s)\) over a continuous scale space \(\mathcal{S}\). These scale weights are integrated and fused with the outputs of each pre-trained Base Router \(r(s)\) to obtain the final routing weights, which are used to select the top-4 models from \(G=3\) world model groups (15 models in total) to form the mixture of world models \(M\).

During inference, the mixture of world models predicts the next observation, and the prediction error generates a knowledge increment \(\Delta K_{t+1}\). This increment is allocated to corresponding scale groups by the Meta-Router. Each group updates its knowledge state according to a scale-dependent forgetting rate \(\alpha^{(g)}\) (intra-scale adaptation) and then exchanges information with adjacent scales through gated bidirectional transfer (inter-scale adaptation), completing a round of test-time evolution.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Observation $o_t$ + Action $a_t$"] --> B["Experiential Distance Calculation<br/>Mahalanobis $\delta$"]
    B --> C["Meta-Router<br/>$\delta \to w_{\text{MR}}(s)$"]
    C --> D["Continuous Scale Space Weights"]
    D --> E["Low-Scale Group<br/>Sensory / PINN / RBF"]
    D --> F["Mid-Scale Group<br/>Cognitive Map"]
    D --> G["High-Scale Group<br/>Relational / Schema"]
    E --> H["Top-4 World Model Mixture"]
    F --> H
    G --> H
    H --> I["Knowledge Increment Allocation"]
    I --> J["Intra-Scale<br/>$\alpha^{(g)}$ Scale-Dependent Forgetting"]
    J --> K["Inter-Scale<br/>Gated Bidirectional Transfer"]
    K -->|Continuous Test-Time Evolution| A

Key Designs

1. Two-Stage Scale-Aware Routing: Decoupling Scale Determination and Model Selection

The limitation of standard MoE is that the router's output is a flat expert weight vector, which fails to inform the system "which scale level the current input belongs to". This makes it impossible to precisely target specific experts for modification during test-time updates—requiring either a complete model update or no update at all. MuSix explicitly decomposes routing into two stages. In the first stage, a continuous Meta-Router \(\text{MR} : \mathcal{O} \times \mathcal{O}^* \times \mathcal{S} \to \mathbb{R}\) receives the experiential distance \(\delta\) and the current observation \(o_t\), outputting a weight function \(w_{\text{MR}}(s)\) over the scale space \(\mathcal{S}\). In the second stage, for each scale \(s\), a pre-trained Base Router \(r(s) : \mathcal{O} \times \mathcal{A} \to \mathbb{R}^N\) gives unnormalized scores for each world model under that scale. Finally, the router integrates and fuses both components:

\[ \text{router} = \int_{\mathcal{S}} w_{\text{MR}}(s) \, r(s) \, ds \]

In practice, the integral is approximated by Monte Carlo sampling with 16 points. The elegance of this decoupled design lies in the fact that the output of the Meta-Router directly reflects "which scale the current input operates on", enabling precise localization of the scale groups that need to be updated during testing—by allocating updates only to the groups with high \(w_{\text{MR}}\) weights, without interfering with world models at other scales.

2. Experiential Distance and Scale-Aware Loss: Operationalizing CLT

To enable the Meta-Router to automatically select world models of the appropriate scale based on the "novelty" of the situation, a computable metric must first be defined. Grounded in Construal Level Theory, the authors propose experiential distance \(\delta\), defined as the Mahalanobis distance between the current observation embedding \(\phi(o_t)\) and the multi-step experience distribution. The experience distribution is modeled by applying exponentially decaying weights \(w_i = \exp(-\beta(t-i))\) to past observation embeddings, ensuring that more recent observations contribute more to the distribution. When the step count is less than \(T_{\min}\), \(\delta\) is set to \(+\infty\), indicating that all scenarios are treated as novel in the absence of prior experience.

\[ \delta = \sqrt{(\phi(o_t) - \mu_E)^\top \Sigma_E^{-1} (\phi(o_t) - \mu_E)} \]

Based on this, the scale-aware loss \(\mathcal{L}_S\) consists of three terms. The first term, \(\mathcal{L}_S^{(1)} = (R_S \bar{\delta} - \|\mu_S\|)^2\), explicitly aligns the magnitude of the expected scale (the norm of the scale vector output by the Meta-Router) with the normalized experiential distance—low \(\delta\) (familiar situations) corresponds to low magnitude (low scale), and high \(\delta\) (novel situations) corresponds to high magnitude (high scale). The second term, \(\mathcal{L}_S^{(2)} = \lambda_1 \text{tr}(\Sigma_S)\), encourages a concentrated scale distribution to prevent the weights from dispersing across the entire scale space. The third term, \(\mathcal{L}_S^{(3)} = \lambda_2 Z^2\), penalizes divergence in the Meta-Router's output. The overall training loss is defined as \(\mathcal{L} = \mathcal{L}_{\text{TF}} + \lambda_S \mathcal{L}_S\), where \(\mathcal{L}_{\text{TF}}\) is the standard teacher-forcing prediction loss of the world model.

One highly insightful point in the experimental analysis is that, although the scale-aware loss only supervises \(\|\mu_S\|\) (the mean magnitude of the scale vector), the authors observed that each axis in the 3D scale space developed specific responses to different types of inputs—some axes responded to visual novelty, while others responded to task complexity—exhibiting unsupervised specialization. This explains why the 3D scale space consistently outperforms 1D and 2D spaces.

3. Intra- and Inter-Scale Knowledge Adaptation: Evolution of Scales at Their Own Pace

Even with correct scale-aware routing, enabling world models to continuously adapt to environmental changes during testing remains a challenge. Low-level physical dynamics knowledge might become obsolete in a few steps, whereas high-level strategic reasoning can remain valid for dozens of steps. The uniform update strategy of standard MoE fails here—updating all experts at a fast pace destroys stable high-level knowledge, while updating at a slow pace fails to keep up with low-level changes.

MuSix delivers a solution with two complementary parts. At the intra-scale level, each group \(g\) updates its knowledge state using a scale-dependent forgetting rate \(\alpha^{(g)} = \alpha_{\max} \exp(-\gamma(g-1))\). Low-scale groups maintain a larger \(\alpha\) (rapidly forgetting old knowledge and quickly absorbing new increments), whereas high-scale groups maintain a smaller \(\alpha\) (forgetting slowly to maintain long-term stability). At the inter-scale level, a gated bidirectional transfer is introduced: adjacent groups exchange knowledge through learnable gating parameters \(W_+^{(g)}\) (upward transfer) and \(W_-^{(g)}\) (downward transfer), with gating values dynamically modulated by the embedding of the current observation \(o_t\).

\[ K_{t+1}^{(g)} = (1 - \alpha^{(g)}) K_t^{(g)} + \Delta K_{t+1}^{(g)} + \Delta K_{\text{in},t+1}^{(g)} - \Delta K_{\text{out},t+1}^{(g)} \]

Here, the incoming term \(\Delta K_{\text{in}}\) and the outgoing term \(\Delta K_{\text{out}}\) represent the net knowledge inflow and outflow from vertically adjacent groups, respectively, ensuring coordination across scales. The gating parameters are fixed during training, and only the knowledge states themselves are updated during testing—a design that effectively mitigates overfitting risks.

Loss & Training

The total loss is defined as \(\mathcal{L} = \mathcal{L}_{\text{TF}} + \lambda_S \mathcal{L}_S\), where \(\mathcal{L}_{\text{TF}}\) represents the teacher-forcing prediction loss (comprising action prediction, transition prediction, and auxiliary loss with weights of 0.1 / 1.0 / 0.1, respectively). Training employs a three-stage curriculum: only the routers (including both the Meta-Router and Base Routers) are trained in steps 0–500; the world model parameters are integrated in steps 500–1500; and a diversity loss is introduced after step 1500 to prevent expert collapse. The base model utilizes Qwen3-VL-4B-Instruct with LoRA adapters (rank=16). Training is performed for 1 epoch with a cosine learning rate scheduler and a 200-step warmup.

Key Experimental Results

Main Results

Dataset Environment Metric LLM-Planner SayCanPay FLARE Conventional MoE MuSix (Ours)
EmbodiedBench EB-Habitat Avg SR (%) 23.11 34.39 27.11 25.56 40.44
EmbodiedBench EB-Navigation Avg SR (%) 24.96 47.20 34.24 29.28 57.92
HAZARD Fire Rescue Value 38.48 41.07 42.22 41.53 43.71
HAZARD Flood Damage Ratio 6.60 6.80 6.71 7.49 4.65

On EmbodiedBench, MuSix achieves comprehensive leadership, showing the most pronounced advantages in subcategories that require complex reasoning (Cpx) and visual understanding (Vis). In the HAZARD Fire scenario, it achieves the highest Rescue Value, and in the Flood scenario, its Damage Ratio is significantly lower than all baselines (4.65 vs. 6.60–7.49). This demonstrates that multi-scale evolution helps agents avoid unnecessary environmental hazards more effectively.

Ablation Study

Configuration EB-Habitat SR (%) HAZARD Fire Value Description
MuSix Full 40.44 43.71 Full model
w/o Meta-Router 32.22 42.22 Degrades to flat routing, SR drops by 8.22pp, core validation
w/o \(\mathcal{L}_S^{(1)}\) (Scale Alignment) 37.78 39.85 No alignment with experiential distance, Fire Value drops the most
w/o \(\mathcal{L}_S^{(2)}\) (Variance Regularization) 35.56 44.89 Habitat drops by 4.88pp, but Fire slightly increases
w/o intra-scale adaptation 38.89 39.84 Uniform forgetting rate, Fire Value drops by 3.87pp
w/o inter-scale adaptation 37.78 41.67 No cross-scale transfer, EB-Habitat drops by 2.66pp

Key Findings

  • Meta-Router is the largest contributor: Removing the two-stage routing leads to the most severe physical performance degradation (EB-Habitat drops from 40.44% to 32.22%), confirming that "explicit scale determination" is the prerequisite for the entire framework. Without the Meta-Router, knowledge increments can only be uniformly distributed according to the aggregated weights of the Base Routers, preventing scale-targeted updates.
  • Core status of the scale alignment loss: The alignment term of \(\mathcal{L}_S^{(1)}\) affects both Habitat and Fire. Removing it causes the HAZARD Fire Value to drop sharply from 43.71 to 39.85, a degradation that exceeds even the removal of the entire intra-scale adaptation. This indicates that the alignment between routing scale selection and experiential distance is not merely a utility enhancement but directly impacts adaptability.
  • Selection of experiential distance metrics: Comparative experiments show that Mahalanobis distance significantly outperforms cosine similarity (+5.33pp SR) and KL divergence (+2.22pp) because it leverages the covariance structure of the experience distribution to distinguish novelty levels across different directions.
  • Architecture-agnostic nature: Five different configurations of world model groupings (Variants I–V, ranging from 2 to 4 groups) fall within a narrow performance range of 38.89–42.22% on EmbodiedBench, with even the weakest variant significantly outperforming the strongest baseline, SayCanPay (34.39%). This indicates that the performance gains stem from the routing mechanism itself rather than any specific world model architecture.
  • Coping mechanism of high-dimensional scale spaces: The individual axes of the 3D scale space exhibit axis-level specialization in response to different types of situational novelty—even though the loss only supervises the overall magnitude. This suggests that increasing the dimensionality of the scale space provides the capacity required to mine multiple dimensions of novelty.

Highlights & Insights

  • Operationalizing Construal Level Theory (CLT) into a computable loss function represents a highly ingenious interdisciplinary transfer: instead of simply employing CLT as narrative packaging, the authors ground the principle of "psychological distance determining abstraction level" into the explicit alignment of \(\mathcal{L}_S^{(1)}\). This approach goes a step deeper than generic "heuristic inspirations."
  • The "dimension expansion" strategy of two-stage routing is highly generalizable: routing is extended from \(\mathbb{R}^N\) (world model output space) to \(\mathcal{S} \times \mathbb{R}^N\) (scale space \(\times\) model space), swapping an extra dimension for the capability to lock targets by scale. This strategy can be generalized to any MoE scenario that requires a "categorize-first-then-select" paradigm.
  • The elegance of gated bidirectional transfer lies in learning the gating parameters exclusively during the training phase, while updating only the knowledge states during the testing phase—essentially learning a knowledge representation during test time that is far smaller than the full model parameters. This successfully curbs overfitting while achieving continuous evolution.
  • "Using the unsupervised specialization of experience distribution covariance to explain why high-dimensional scale spaces are superior" is a highlight of the experimental design. Instead of settling for the simple "3D is better than 1D" outcome, the authors deeply analyze behavioral differences across axes, offering transferable architectural design intuitions for readers.

Limitations & Future Work

  • The framework inherits the performance caps and limitations of the underlying VLM, where improvements in reasoning quality directly benefit from advances in base model capabilities. The authors candidly acknowledge this—implying that updating the base model necessitates retraining LoRA adapters for all world models.
  • Real-world experiments only cover 8 manipulation tasks on the Franka Research 3 platform; the diversity of platforms and the span of tasks need further expansion.
  • Hyperparameters heavily rely on manual tuning (\(\alpha_{\max}\), \(\gamma\), scale dimensionality \(|\mathcal{S}|\)). Future work could explore allowing the system to automatically discover optimal group counts and architectural designs directly from data.
  • In terms of computational overhead, two-stage routing and 16-point MC sampling introduce roughly a 40% latency increase compared to conventional MoE (3.52s vs. 2.52s), requiring actual deployments to trade off execution speed for adaptive precision.
  • vs SayCanPay: SayCanPay employs learned affordance and cost models to guide LLM planning, but its models lack scale differentiation, requiring full retraining when the environment changes. MuSix's advantage lies in modular, targeted updates enabled by scale-aware routing, though at the expense of higher inference overhead (+89% latency).
  • vs Conventional MoE: The top-level routing of standard MoE lacks interpretability, making it unusable for scale-specific selective updates during testing. MuSix's two-stage routing delivers a tracing-friendly scale axis, though modifying the routing architecture introduces additional training complexity.
  • Theoretical link to CLT: This work represents a milestone in deeply embedding cognitive theories within reinforcement learning routing mechanisms. Distinct from prior styles that "borrow cognitive theories simply as analytical frameworks," this paper directly formulates computable loss functions.

Rating

  • Novelty: ⭐⭐⭐⭐ [The design concept of embedding CLT into MoE routing is highly original in the field of embodied intelligence, and the combination of two-stage routing and scale-dependent forgetting makes substantial contributions.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated comprehensively on 2 simulation benchmarks + real robots. The ablation study covers three loss terms of the two-stage routing and two sub-modules of knowledge adaptation. Analytical experiments (distance metrics, scale dimensions, world model variants) go very deep.]
  • Writing Quality: ⭐⭐⭐⭐ [The methodology is clearly described, and the framework diagram in Figure 2 is easy to understand alongside the text. The experimental analysis is insightful, though the mathematical typesetting is slightly cluttered in some preprint versions.]
  • Value: ⭐⭐⭐⭐ [Provides a highly operational framework for the continuous learning of embodied agents in dynamic environments. The decoupled design of the routing mechanism carries strong generalization value.]