Thermo-JEPA: Learning a Geometry-Grounded Thermal World Model via Cross-Modal Privileged Masking¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://eccv.ecva.net/virtual/2026/poster/3162
Area: 3D Vision
Keywords: World Models, Thermal Vision, Self-Supervised Learning, Cross-Modal Alignment, Zero-Shot Transfer
TL;DR¶
Addressing intra-frame spatial homogenization caused by thermal equilibrium in infrared video, Thermo-JEPA introduces a privileged information learning paradigm that injects confidence-gated spatial topology from paired RGB frames into the EMA teacher's attention mechanism as a soft prior, preventing representation collapse and modal pollution while achieving state-of-the-art pure-thermal dynamics prediction.
Background & Motivation¶
World models have emerged as a foundational paradigm for physical perception and dynamic forecasting, driving breakthroughs across embodied intelligence, robotics navigation, and autonomous driving. However, mainstream world models—predominantly grounded in Joint-Embedding Predictive Architectures (such as V-JEPA)—rely almost exclusively on visible-light (RGB) observations. RGB sensors degrade catastrophically under adverse lighting, extreme glare, dense fog, and severe precipitation. In contrast, thermal infrared (IR) sensors capture radiated heat rather than reflected light, offering genuine all-weather perceptual reliability. Extending generative world modeling into the thermal domain is therefore essential for robust, 24/7 autonomous agents.
Nevertheless, directly transferring standard spatiotemporal masked video modeling to thermal imagery leads to severe representation degradation. The fundamental bottleneck lies in what this work formalizes as intra-frame spatial homogenization: in real-world scenes, adjacent yet distinct physical entities (e.g., a coffee machine, paper cups, and snack packages) frequently exist in near-thermal equilibrium, displaying minimal to zero inter-object temperature gradients. When processed by unconstrained self-attention mechanisms, this lack of high-frequency spatial variation causes the network to blend neighboring patches across semantic boundaries, collapsing rigid physical structures into unstructured, continuous feature distributions. Deprived of clear geometric boundaries, the latent space loses the spatial prerequisites necessary to track physical kinematics, causing downstream dynamics forecasting to collapse.
Prior multimodal methods predominantly resort to dense cross-modal alignment or early-fusion architectures. Forcing the thermal latent space to directly emulate RGB feature distributions triggers severe modal pollution, compelling the network to hallucinate high-frequency visible-light textures that do not physically exist in the infrared spectrum. This distorts intrinsic thermal kinematics and strips systems of pure-IR operational autonomy. To overcome this dilemma, this work adopts the Learning Using Privileged Information (LUPI) paradigm, treating RGB data strictly as a privileged geometric scaffold during pre-training. Core idea: extract intra-frame spatial topology from paired RGB frames and inject it as a confidence-gated soft structural prior into the EMA teacher's attention mechanism, explicitly enforcing entity boundary segregation while strictly isolating gradients to eliminate modal pollution in pure-thermal world modeling.
Method¶
Overall Architecture¶
The Thermo-JEPA architecture comprises three primary stages: privileged intra-frame topology extraction, confidence-gated attention modulation, and masked spatiotemporal prediction. During pre-training, the model takes spatiotemporally aligned RGB-Thermal video pairs (\(X_{\text{IR}}\) and \(X_{\text{RGB}}\)). To prevent temporal causality leakage, topological extraction is strictly confined to independent 2D frames: a frozen DINOv2 backbone computes patch-wise cosine similarity matrices and channel-variance-based spatial confidence maps. Next, the confidence-weighted spatial priors are assembled into a block-diagonal matrix and injected as an additive bias into the multi-head self-attention of the exponential moving average (EMA) thermal teacher. Finally, the student network processes only the unmasked thermal tokens to predict masked representations, supervised by an \(L_1\) alignment loss with strict stop-gradient on the teacher targets. Downstream inference completely discards the RGB branch, operating exclusively on pure thermal video.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Paired RGB-T Video Input<br/>Spatiotemporally aligned X_IR & X_RGB"] --> B["Privileged Spatial Topology Extraction<br/>DINOv2 computes patch affinity & confidence"]
B --> C["Confidence-Gated Attention Modulation<br/>Block-diagonal prior injected into EMA teacher MHSA"]
C --> D["Gradient Isolation & Spatiotemporal Masking<br/>Student latent prediction aligned to teacher targets"]
D --> E["Pure Thermal Inference<br/>Zero-shot action anticipation & trajectory rollout"]
Key Designs¶
1. Privileged Spatial Topology Extraction: Decoupling Pure Geometry from Visible Textures
Direct cross-modal distillation forces thermal representations to mimic visible textures, triggering severe modal hallucination. Thermo-JEPA addresses boundary blurring caused by thermal equilibrium by leveraging RGB frames strictly as privileged information (LUPI) to derive abstract spatial topologies. Specifically, a frozen visible foundation model (DINOv2 ViT-L) processes each frame independently, producing spatial feature tensors \(F_{\text{RGB}}^{4D} \in \mathbb{R}^{B \times T \times N \times d}\). Within each time slice \(t\), the pairwise cosine similarity between spatial patch tokens yields the topological adjacency matrix \(S[b, t, i, j]\):
This formulation extracts relative entity affinities and semantic grouping while stripping away color, illumination, and high-frequency surface textures. Confining the operation strictly to the intra-frame spatial domain guarantees that no temporal information leaks across time steps, preserving the strict temporal causality required by predictive world models.
2. Confidence-Gated Attention Modulation: Enforcing Thermal Autonomy with Adaptive Fallback
Unconditionally injecting RGB topology can backfire under degraded visible conditions (e.g., pitch-black night scenes), corrupting the thermal latent space with spurious visual noise. To resolve this, Thermo-JEPA introduces a spatial confidence gating mechanism based on localized channel variance. Computing feature variance across channels in the DINOv2 embeddings and passing it through a Sigmoid activation yields a spatial confidence vector \(c_{b,t} \in \mathbb{R}^N\), where high values denote salient, well-illuminated structures and low values mark degraded regions. The spatial prior is modulated via an outer product as \(\hat{S}_{b,t} = (c_{b,t} c_{b,t}^T) \odot S[b, t]\). To integrate this prior into joint spatiotemporal multi-head self-attention (MHSA), intra-frame priors are assembled into a block-diagonal matrix \(M_{\text{prior}}^{(b)}\):
where \(\lambda\) is a learnable scalar initialized to zero. Inter-frame attention biases (\(t_i \neq t_j\)) are strictly zeroed out. The modified attention computation in the teacher network is formulated as \(\text{Softmax}\left(\frac{Q_{\text{IR}} K_{\text{IR}}^T}{\sqrt{d}} + M_{\text{prior}}\right) V_{\text{IR}}\). When intrinsic thermal contrast is sharp, thermal self-attention naturally dominates; when visible light degrades, the gating mechanism smoothly zeroes out \(M_{\text{prior}}\), allowing the network to gracefully revert to pure infrared self-attention and preserving thermodynamic autonomy.
3. Gradient Isolation and Spatiotemporal Masking: Preventing Modal Pollution for Robust Kinematic Forecasting
In end-to-end multimodal optimization, backpropagating gradients into privileged RGB branches encourages the student network to discover shortcuts by decoding visible textures, which causes inference failure when RGB inputs are absent. In Thermo-JEPA, the student encoder \(E_\theta\) processes exclusively unmasked thermal tokens. Paired with learnable mask tokens and 3D-RoPE, the predictor \(P_\phi\) infers the masked representations \(Z_{\text{pred}}\). The teacher network \(E_\xi\) is updated slowly via exponential moving average (EMA, momentum 0.999). The pre-training objective minimizes the \(L_1\) distance against teacher targets with an explicit stop-gradient (\(\text{sg}\)):
By completely severing the gradient flow from the teacher to the student, visible-light modal information is strictly contained within the EMA teacher as a soft supervisory signal. The student network is compelled to rely entirely on unmasked thermal inputs to infer geometrically grounded representations, unlocking exceptional zero-shot dynamic forecasting on pure thermal inputs.
Loss & Training¶
Pre-training is conducted on the proposed RGBT-World video dataset (using the 15-hour training split of spatiotemporally aligned video). The backbone architecture is a standard ViT-L/16 (300M parameters) with \(16 \times 16 \times 2\) tubelet patches. Training utilizes the AdamW optimizer with cosine learning rate scheduling (peak learning rate \(1\times 10^{-4}\)) and weight decay 0.04. Spatiotemporal masking ratios are dynamically sampled between 30% and 80% to encourage macroscopic semantic forecasting rather than trivial local interpolation. Training runs on 4 \(\times\) NVIDIA A6000 GPUs. Downstream evaluations adhere strictly to a frozen-backbone protocol, where all pre-trained weights are frozen and only lightweight task probes are trained.
Key Experimental Results¶
Main Results¶
Evaluation encompasses two primary benchmarks: Zero-Shot Action Anticipation on the hold-out split of RGBT-World (assessing kinematic modeling without action supervision) and Diagnostic Probing of spatial and kinematic integrity (evaluating spatial localization mIoU with a \(1 \times 1\) conv probe and 8-frame trajectory displacement errors ADE/FDE with a single-layer Transformer). All evaluated models receive identical pure-thermal inputs at test time under a frozen-backbone protocol.
| Method | Pre-train Params | Pre-train Data | Top-1 Acc (%) | Recall@5 | Spatial mIoU (↑) | Trajectory ADE (↓) | Trajectory FDE (↓) |
|---|---|---|---|---|---|---|---|
| DINOv2 (Frozen) | 1.1B | LVD-142M | 21.4 | 34.2 | 48.2 | 15.6 | 21.3 |
| SigLIP 2 | 1.2B | WebLI | 23.8 | 36.1 | - | - | - |
| InternVideo2 | 6B | Internal | 29.6 | 41.3 | - | - | - |
| VideoPrism | 1B | VP-Corpus | 31.2 | 43.5 | - | - | - |
| Cosmos-World | 7B | Cosmos-Data | 33.5 | 46.8 | - | - | - |
| VideoMAE v2 (IR-Supervised) | - | - | - | - | 55.4 | 12.3 | 17.8 |
| Vanilla V-JEPA2 (IR-Only) | 300M | RGBT-World | 18.6 | 28.4 | 32.1 | 28.5 | 39.4 |
| Contrastive-IR (Early Fusion) | 300M | RGBT-World | 24.2 | 35.5 | - | - | - |
| Thermo-JEPA (Ours) | 300M | RGBT-World | 45.8 | 62.1 | 68.7 | 8.4 | 11.2 |
Ablation Study¶
Ablation models are uniformly pre-trained on RGBT-World for 50 epochs, evaluating the effects of prior sources, confidence gating, and gradient flow on spatial mIoU and zero-shot action accuracy.
| Variant | Prior Source | Confidence Gating (\(C\)) | Gradient Flow | Spatial mIoU (↑) | Zero-Shot Acc (%) | Note |
|---|---|---|---|---|---|---|
| (a) Vanilla V-JEPA2 | None | No | N/A | 32.1 | 18.6 | Severe spatial homogenization collapse |
| (b) Random Topology | Uniform Noise | Yes | Stopped | 15.2 | 12.4 | Random noise severely corrupts latent geometry |
| (c) Edge-based Prior | Sobel Filter | Yes | Stopped | 41.5 | 28.7 | Low-level edges lack semantics and fail under blur |
| (d) Un-gated Soft Attention | DINOv2 | No | Stopped | 61.2 | 38.5 | Absence of gating injects false priors in low light |
| (e) End-to-End Leakage | DINOv2 | Yes | Allowed | 54.8 | 31.2 | Gradient leakage triggers modal pollution |
| (f) Thermo-JEPA (Full Model) | DINOv2 (Variance) | Yes | Stopped | 68.7 | 45.8 | Optimal balance of geometry and thermal autonomy |
Key Findings¶
- Homogenization Collapse is the Fundamental Bottleneck: Vanilla V-JEPA2 trained purely on infrared data achieves only 18.6% action accuracy, underperforming even static image foundation models. PCA feature decomposition reveals that its deep latent dimensions collapse almost entirely into the first principal component (PC1, reflecting global average temperature). Thermo-JEPA maintains robust variance across both PC1 and PC2, confirming that topological grounding successfully prevents dimensional collapse.
- Confidence Gating Enables Adaptive Fallback: Omitting confidence gating (Variant d) causes a 7.3% drop in zero-shot accuracy (from 45.8% to 38.5%). This confirms that injecting raw RGB priors under poor lighting introduces erroneous biases, whereas variance-based gating allows graceful degradation to pure thermal self-attention.
- Gradient Isolation Eliminates Modal Contamination: Permitting gradients to backpropagate through the target representations (Variant e) drops accuracy by 14.6% (31.2% vs. 45.8%). Unconstrained gradients force the student to decode nonexistent visible textures rather than learning intrinsic thermal kinematics.
Highlights & Insights¶
- Privileged Geometric Decoupling: Rather than forcing dense feature matching or cross-modal contrastive alignment, Thermo-JEPA extracts only non-textured, color-free topological graphs from visible light and applies them as soft attention biases, preserving native thermal gradients.
- Causally Isolated Block-Diagonal Prior: Assembling intra-frame priors into a block-diagonal matrix strictly zeroes out inter-frame cross-attention biases. This guarantees that spatial geometry is regularized without causing future temporal leakage in predictive world modeling.
- RGBT-World Video Benchmark: Aggregating 10 public datasets alongside high-resolution aerial recordings (up to 2700×2160 RGB / 640×512 IR), RGBT-World provides 1.75M synchronized frame pairs with sequences extending up to 500 seconds, filling the critical data gap for generative thermal foundation models.
Limitations & Future Work¶
- Dependence on Spatiotemporally Calibrated Data: Pre-training relies on well-calibrated, co-axial, and synchronized RGB-T pairs; dynamic parallax or calibration errors may cause localized geometric misalignments.
- Static Visible Priors in Thermally Dynamic Anomalies: Because the topological prior stems from visible imagery, objects with identical visual appearances but stark thermal discrepancies (e.g., hidden electrical overheating) may receive improper grouping biases.
- Future Directions: Investigating self-supervised alignment under weakly synchronized or unpaired RGB-T video streams, and incorporating physical thermodynamics engines to model heat transfer and thermal inertia.
Related Work & Insights¶
- vs V-JEPA / V-JEPA 2: V-JEPA operates effectively in visible light but suffers from severe spatial homogenization in thermal infrared due to thermal equilibrium; Thermo-JEPA preserves its efficient latent predictive paradigm while resolving structural collapse via gated topological priors.
- vs MultiMAE / OmniMAE: Multimodal masked autoencoders require multi-sensor inputs during both training and inference, creating deployment dependencies and texture hallucinations; Thermo-JEPA adheres to the LUPI paradigm, enabling completely standalone pure-IR inference.
- vs ImageBind / VideoCLIP: Global contrastive alignment aligns global [CLS] embeddings across modalities, failing to preserve fine-grained localized heat dynamics; Thermo-JEPA bypasses feature-level matching and regularizes spatial attention manifolds directly.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the intra-frame spatial homogenization problem in thermal world modeling and introduces an elegant LUPI attention modulation framework.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces the 1.75M-frame RGBT-World benchmark, supported by extensive zero-shot dynamics evaluations, diagnostic probing, PCA analyses, and detailed ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear physical intuition, rigorous mathematical formulation, and well-structured illustrations.
- Value: ⭐⭐⭐⭐⭐ Opens up new frontiers for all-weather embodied AI and physics-grounded predictive world models with high-quality open benchmarks.