PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation¶

Conference: ECCV 2026
arXiv: 2606.26916
Code: https://github.com/sediment1024/PhysRAG
Area: Video Generation
Keywords: Physics-Aware Video Generation, Retrieval-Augmented Generation, Video Diffusion Models, Physics Prior Injection, Learnable Queries

TL;DR¶

PhysRAG introduces Retrieval-Augmented Generation (RAG) to video diffusion models. It first retrieves reference videos most physically relevant to the input prompt from a handcrafted physical video database, encodes their spatiotemporal features using VideoMAE V2, and selectively extracts and injects physical priors into the DiT denoising process via a learnable query bottleneck. It achieves SOTA with an average score of 0.58 on the PhyGenBench physical common sense benchmark, while only increasing inference overhead by 1.24%.

Background & Motivation¶

Text-to-video (T2V) generation has made tremendous progress in recent years. Large-scale DiT models such as Wan, CogVideoX, and HunyuanVideo are capable of generating highly visually realistic videos. However, these models frequently fail when facing scenarios involving physical laws—splashing water violates conservation of momentum, falling objects defy gravity direction, or liquid volumes expand before making contact. The root cause is that existing video diffusion models lack the capability to explicitly model physical dynamics; relying solely on data-driven visual pattern matching is insufficient to guarantee physical consistency.

Existing methods for physics-aware video generation fall into two main categories. Explicit methods rely on deterministic physics simulators or mathematical constraints (e.g., Hamiltonian networks, Newtonian dynamics). While they strictly guarantee physical laws, their generalization capability in complex open-world scenarios is poor, as phenomena like thermodynamics and optics are difficult to describe via closed-form equations. Implicit methods adopt a data-driven approach, implicitly learning physical laws through DPO, reinforcement learning, or 3D constraint alignment. Although showing better generalization, they lack controllability: physical priors are encoded as scalar rewards or inference signals on the prompt side, making them difficult to inject precisely into the generation process. A common blind spot of both paradigms is the failure to utilize explicit spatiotemporal references from real-world physical videos as physical priors.

This paper starts with a cognitive analogy: humans do not learn physical laws by memorizing formulas, but by observing a vast number of daily physical scenarios, retrieving similar experiences from memory and extracting the underlying physical principles when encountering new situations. Inspired by this, PhysRAG proposes to introduce the RAG paradigm to video generation—explicitly retrieving reference videos with the same underlying physical laws and implicitly injecting their physical dynamics into the diffusion model's denoising process via learnable queries. Core Idea: using retrieved real-world physical videos as explicit physical priors, selectively extracting physical dynamics via a learnable query information bottleneck, and injecting them into the DiT to generate videos that conform to physical laws.

Method¶

Overall Architecture¶

The overall pipeline of PhysRAG is divided into offline and online stages. In the offline stage, a two-stage filtering pipeline is used to select approximately 7K high-quality physical videos from the WISA-80K dataset for training, while a PhysRAG database containing 170 videos across 17 physical categories is handcrafted for retrieval during inference. In the online inference stage, given a text prompt, VideoCLIP-XL is used to retrieve the most physically relevant reference video from the PhysRAG database. The pre-trained VideoMAE V2 encoder then extracts spatiotemporal features, which are processed by the Query Inject module—using a set of learnable queries to perform cross-attention with the video features as an information bottleneck to selectively extract physical priors. Finally, the distilled physical tokens are injected into the early layers (layers 0, 1, 2) of the Wan2.2-5B DiT via gated residual connections, guiding the denoising process to generate physically consistent videos.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Prompt"] --> B["Physical Video Retrieval<br/>VideoCLIP-XL + FAISS"]
    B --> C["Video Feature Encoding<br/>VideoMAE V2"]
    C --> D["Query Injection Module<br/>Learnable Query Cross-Attention"]
    D --> E["DiT Denoising<br/>Gated Residual Injection"]
    E --> F["Output: Physics-Aware Video"]

Key Designs¶

1. Two-Stage Data Filtering Pipeline: Extracting High-Quality Physical Videos from Noisy Data at Controllable Cost

WISA-80K is an internet-crawled dataset designed for physics-aware video generation, but the raw data suffers from severe mismatches between descriptions and videos, along with low quality. Direct utilization would introduce noisy physical signals and mislead the model. PhysRAG designs a two-stage filtering pipeline to balance data quality and computational cost.

The first stage (coarse filtering) only uses the text-level Qwen3-VL-4B to score the textual descriptions of videos for quality and physical relevance, retaining the top 10% candidates. This text-only filtering is highly cost-effective and efficiently filters out irrelevant samples, but it introduces false positives—samples where descriptions seem physically meaningful but the video content is mismatched. The second stage (fine-grained alignment verification) uniformly samples keyframes from the candidates retained in the first stage and inputs them along with the corresponding prompts into Qwen3-VL-4B for text-image consistency validation, discarding mismatched samples to yield a retention rate of approximately 90%. Using uniformly sampled keyframes rather than full video sequences is a cost consideration: frame-by-frame inference of Qwen3-VL-4B on the entire scale of WISA-80K would take approximately 2,000 GPU hours (H20), which is computationally prohibitive. Ultimately, a high-quality subset of about 7K videos is filtered from the original 80K.

2. Physical Video Database Construction: Handcrafting 17 Physical Categories for Reliable Retrieval Sources

The performance of the RAG framework heavily depends on the quality and structure of the retrieval database. PhysRAG handcrafts a physical video database containing 170 videos: first, it organizes physical phenomena frequently occurring in open-domain video generation into 17 categories (collision, combustion, explosion, fluid, free fall, thermodynamics, optics, etc.), manually collecting and verifying 10 high-quality videos demonstrating corresponding physical characteristics for each category. The database is organized in a simple hierarchical directory structure, with each category in a separate subdirectory.

This design offers two advantages: first, it constrains the retrieval space to physically meaningful samples, improving retrieval reliability; second, it provides diverse yet consistent references within each category, helping the model extract transferrable physical priors rather than overfitting to individual video instances. During retrieval, VideoCLIP-XL extracts text and video features, and FAISS performs efficient nearest-neighbor retrieval.

3. Learnable Query Injection Module (Query Inject): Information Bottleneck for Selective Physical Dynamics Extraction

This is the core technical contribution of PhysRAG. The pre-trained VideoMAE V2 encoder extracts rich spatiotemporal features \(F_v \in \mathbb{R}^{L \times C_v}\) from retrieved videos. However, directly injecting these dense latent representations into the DiT causes severe issues: besides physical dynamics, retrieved videos contain substantial irrelevant information—background textures, object appearance, lighting conditions, etc. Direct injection introduces visual noise and bias, which degrades generation quality (concat injection scored only 0.540 in ablation studies, Table 3).

The core idea of the Query Inject module is to use a set of learnable queries \(Q \in \mathbb{R}^{N \times C_h}\) (\(N=128\)) as an information bottleneck to selectively focus on components related to physical dynamics in video features via cross-attention:

\[H_{ca} = \text{CrossAttn}(Q, F_v, F_v) \in \mathbb{R}^{N \times C_h}\]

The query \(Q\) acts as the query, while the video feature \(F_v\) acts as both key and value. This operation adaptively aggregates the most physical dynamics relevant to the current generation task. Subsequently, it is refined via FFN + residual connection:

\[H_{ffn} = H_{ca} + \text{FFN}(H_{ca}), \quad \text{FFN}(x) = W_2\,\text{GELU}(W_1 x)\]

The refined features are projected linearly as \(H_{out} = H_{ffn}W_o \in \mathbb{R}^{N \times C_p}\) to obtain compact physical prior tokens. When injecting into the DiT, a lightweight alignment operator \(\mathcal{A}(\cdot)\) first aligns them to the DiT token dimension and sequence length, and then they are concatenated with the original DiT tokens along the sequence dimension and compressed back to the original dimension via a fusion projection \(\phi(\cdot)\):

\[H_{fuse} = \phi(\text{Concat}(H_{DiT}, \tilde{H}_p)) \in \mathbb{R}^{T \times C_d}\]

Finally, they are injected as a gated residual: \(H'_{DiT} = H_{DiT} + \alpha H_{fuse}\), where \(\alpha\) is a learnable scalar gate, allowing the model to adaptively adjust the injection intensity of physical priors. Linear probing experiments (Table 4) verify that the learnable query tokens indeed retain physical category information (accuracy of 67.06% vs. 67.65% for raw features), with only a minor drop, proving that the information bottleneck effectively distills physical information while filtering out irrelevant visual details.

4. Joint Training Strategy: Co-optimization of DiT Backbone and RAG Module

Ablation of training strategies (Table 5) reveals a critical finding: multi-stage training (first SFT fine-tuning the DiT backbone, then freezing the backbone to train the query module alone, i.e., SFT+RAG) only yields 0.551, which is much lower than the 0.578 of joint training, and is even less effective than pure SFT (0.546). This indicates that the effective utilization of physical priors is not a simple case of "learning visual generation first, then overlaying physical information"—the physical dynamics extracted by learnable queries require the DiT backbone to actively adapt to and utilize them during the denoising process, rather than receiving them passively. Joint training enables the DiT and the Query Inject module to co-evolve: queries learn to extract the most useful physical signals for the DiT, and the DiT learns how to use these signals to constrain the generation process. This is also verified by the injection location choice: multi-layer injection (layers 0, 1, 2) achieves the best performance (0.578), whereas single-layer injection (especially at the very early or late layers) drops significantly—injecting at early layers allows the DiT to establish high-level physical representations from the start, while the multi-layer design fully integrates physical priors at different abstraction levels.

A Complete Example: Robotic Arm Etching a Circuit Board¶

Taking the "robotic arm etching a circuit board" scenario in PhyGenBench as an example illustrates the complete workflow of PhysRAG. The input prompt describes a robotic arm executing continuous etching actions on a circuit board. First, VideoCLIP-XL matches the prompt with the features of 17 physical phenomena in the PhysRAG database and retrieves a reference video under the "mechanical interaction" category—this video shows a tool executing continuous pressing and sliding actions on a surface. VideoMAE V2 encodes the retrieved video into a feature matrix \(F_v\) of \(L\) spatiotemporal tokens. 128 learnable queries \(Q\) distill physical prior tokens related to "tool-surface contact mechanics"—including the directionality of applied pressure, spatial constraints of contact points, and continuity of motion—while ignoring specific object appearances and backgrounds in the retrieved video. These physical tokens are aligned and injected into layers 0, 1, 2 of Wan2.2-5B DiT, fusing with the noise latent representation \(H_{DiT}\). The gated residual mechanism allows the model to adaptively determine the contribution weight of the physical prior at each denoising step. In the generation result, the robotic arm executes coherent continuous actions of pushing down, sliding, and lifting, whereas the baseline Wan2.2-5B video shows an almost static robotic arm—since it has never learned such precise mechanic interaction patterns from the training data.

Loss & Training¶

PhysRAG is fine-tuned based on Wan2.2-5B using standard diffusion loss (noise-prediction MSE), jointly training all parameters of the Wan backbone and the Query Inject module. Training configuration: 4 NVIDIA H20 GPUs for 2 days, with an effective batch size of 128 (micro-batch size of 16 per GPU + 2 gradient accumulation steps), a learning rate of \(1 \times 10^{-6}\), weight decay of 0.01, and 20 epochs on a video resolution of \(49 \times 704 \times 480\). The optimizer is AdamW (\(\beta_1=0.9\), \(\beta_2=0.999\)), with BF16 mixed-precision, gradient checkpointing, and DeepSpeed ZeRO-3 CPU offload to reduce memory footprint. Queries are injected into layers 0, 1, 2 of the DiT, increasing the total parameter count from 4.998B to 5.114B (+2.28%).

Key Experimental Results¶

Main Results¶

PhyGenBench Physical Common Sense Evaluation (using GPT-4o as judge): PhysRAG achieves SOTA with an average score of 0.58, outperforming the Wan2.2 baseline of 0.54 (+0.04), beating closed-source commercial models Kling (0.49) and Gen-3 (0.51), and surpassing DiT-Mem (0.56) which also takes a retrieval-augmented approach. Improvements in thermal and material metrics are particularly notable (+0.04 and +0.05, respectively), demonstrating that retrieved real-world physical videos are highly beneficial for complex physical phenomena (thermal conduction, fluid material characteristics).

Method	Scale	Mechanics	Optics	Thermal	Materials	Average
Pika	-	0.35	0.56	0.43	0.39	0.44
Gen-3	-	0.45	0.57	0.49	0.51	0.51
Kling	-	0.45	0.58	0.50	0.40	0.49
CogVideoX	5B	0.39	0.55	0.40	0.42	0.45
Open-Sora V1.2	1.1B	0.43	0.50	0.44	0.37	0.44
DiT-Mem	5B	0.56	0.74	0.48	0.47	0.56
Wan 2.2	5B	0.58	0.60	0.50	0.48	0.54
Wan 2.2 + PhysRAG	5B	0.59	0.66	0.54	0.53	0.58

VBench General Video Quality Evaluation: PhysRAG achieves consistent improvements across both low-level and high-level metrics. Low-Avg increases from 62.10% to 65.48% (the highest among all compared methods), and High-Avg increases from 81.18% to 82.88%. The color score (Color) reaches 93.95%, the highest among all methods. This indicates that injecting physical priors does not compromise general generation quality, but instead improves visual consistency through more reasonable motion constraints.

Method	Color	Style	Obj. Class	Low-Avg	Subj. Consist.	Spatial Rel.	Multi-Obj.	High-Avg
CogVideo	79.57%	22.01%	73.40%	58.33%	92.19%	18.24%	18.11%	42.85%
DiT-Mem	93.67%	21.22%	80.00%	64.96%	95.67%	78.42%	74.38%	82.82%
Wan 2.2	85.92%	21.26%	79.12%	62.10%	95.51%	78.91%	69.12%	81.18%
Wan 2.2 + PhysRAG	93.95%	22.00%	80.50%	65.48%	96.31%	77.33%	75.00%	82.88%

Ablation Study¶

Configuration	Mechanics	Optics	Thermal	Materials	Average	Description
SFT	0.575	0.626	0.511	0.475	0.546	No RAG, pure data quality baseline
SFT+RAG	0.566	0.626	0.511	0.500	0.551	SFT first, then freeze backbone to only train query
PhysRAG (Joint)	0.586	0.660	0.540	0.526	0.578	Backbone + query end-to-end joint optimization
PhysRAG + Random Data	0.558	0.640	0.511	0.466	0.544	Using unfiltered data, worse than Filtered-SFT
Concat Injection	0.541	0.633	0.488	0.500	0.540	Direct concatenation of retrieved features, introducing visual noise
Cross Attention Injection	0.541	0.633	0.522	0.516	0.553	Dense cross-attention, missing the information bottleneck
RAG-Front (Layer 0 only)	0.508	0.500	0.511	0.393	0.478	Single-layer injection, worst performance
RAG-Mid (Layer 15 only)	0.533	0.653	0.500	0.500	0.546	Middle-layer injection is better than ends but worse than multi-layer

Key Findings¶

Joint training is a non-negotiable design: SFT+RAG multi-stage training (0.551) is not only far below joint training (0.578) but its mechanics metric (0.566) is even lower than pure SFT (0.575). Physical priors are not plug-and-play add-ons; the DiT backbone must learn from scratch how to utilize retrieval signals. Freezing the backbone is equivalent to feeding the model extra information it cannot interpret. Joint training allows the DiT and Query Inject module to co-evolve: queries learn to extract the most useful physical signals for the DiT, and the DiT learns how to use these signals to constrain the generation process. The choice of injection location also verifies this: multi-layer injection (layers 0, 1, 2) performs best (0.578), while single-layer injection (especially at the very early or late layers) drops significantly—injecting early allows the DiT to establish high-level physical representations from the start, while the multi-layer design fully integrates physical priors at different abstraction levels.
Data quality determines the upper bound of RAG: PhysRAG + Random Data (0.544) performs even worse than pure Filtered-SFT fine-tuning (0.546), indicating that the RAG mechanism is highly sensitive to training data quality—garbage retrieval sources only amplify noise. Filtering data boosts RAG from 0.544 to 0.578 (+0.034), making it one of the largest single contributors.
Information bottleneck is the soul of Query Inject: Direct concatenation (0.540) and dense cross-attention (0.553) perform significantly worse than Query Inject (0.578). The root cause is that non-physical information in retrieved videos, such as background texture and object appearance, pollutes the DiT feature space, degrading generation quality. The learnable query-drive information bottleneck bridges explicit retrieval with implicit generation.
Multi-layer early injection is the optimal strategy: Single-layer injection (especially at the far end at 0.521 or the beginning at 0.478) performs much worse than multi-layer injection (0.578). The optimal configuration, which injects only into the first three layers, suggests that physical constraints mainly act during the high-level semantic planning stage. This finding offers insights into the hierarchical division of labor in diffusion model denoising: early layers manage "global layout and motion trends", while later layers manage "local detail refinement".
Extremely low computational overhead: PhysRAG adds only 114.25M parameters (+2.28%), 1.81GB of GPU memory (+9.2%), and increases inference latency by 0.82 seconds per sample (+1.24%). FAISS retrieval takes only 0.0065 seconds, which is negligible. The gains stem from the effectiveness of the retrieval and injection mechanisms, rather than parameter compounding.

Highlights & Insights¶

The elegant "human learning physics" analogy runs throughout the paper: It is not just a rhetorical wrapping but genuinely guides the design—from why RAG is used (humans recall similar scenarios) to why an information bottleneck is needed (humans extract physical principles but ignore irrelevant details), each module has a cognitive motivation.
The information bottleneck design addresses a common issue: In retrieval-augmented generation, "how to selectively extract only the useful parts" is a shared challenge. The learnable query bottleneck of Query Inject is a generalizable paradigm that can be directly applied to other retrieval-augmented tasks (e.g., style-guided generation, motion transfer). The key insight is that the most valuable information in retrieved references is "structural" rather than "appearance-based", and the information bottleneck is naturally suited for this "discarding appearance, preserving structure" requirement.
Adaptive cost strategy for data filtering: The two-stage (text \(\to\) multimodal) pipeline is not just simple serialization: it accounts for the non-uniform distribution of false positives and computational costs in each stage. Stage 1 text filtering is cheap but introduces false positives; Stage 2 multimodal verification is expensive but precise. This "coarse-to-fine" strategy is the optimal solution under a budget constraint of 2,000 GPU hours.
Gated residual injection is a low-cost, high-yield design: A single learnable scalar \(\alpha\) allows the model to adaptively control the injection strength of physical priors. Avoiding complex timestep scheduling or hierarchical injection strategies, this minimalist design brings stability and interpretability to training.
Three-layer injection instead of full-layer: Injecting physical priors only into the first three layers yields the best performance, indicating that physical constraints mainly function during high-level semantic planning. This discovery is illuminating for understanding the functional divisions of denoising steps in diffusion models—early layers control the "overall layout and motion trends", whereas later layers refine "local details".

Limitations & Future Work¶

Limited coverage of the physical video database: 170 videos across 17 categories are far from representing the entire raw physical world. Phenomena like fracturing, phase transitions, and electromagnetic induction are not yet covered. Expansion mechanisms for the database (automatic category discovery, crowdsourcing annotation) must be resolved for practical deployment. The handcrafted construction of the database also limits scalability.
Strong dependency on retrieval quality: If the retrieved video does not match the target prompt's physical attributes, the injected physical prior can be misleading. The capability of VideoCLIP-XL as a retrieval model in physical understanding has not been specifically validated, and its performance on fine-grained physical distinctions (e.g., "elastic vs. inelastic collision") remains questionable.
Validated on only a single backbone: All experiments are conducted on Wan2.2-5B, and generalization to other DiT architectures (e.g., CogVideoX, HunyuanVideo) has not been verified. Whether the design of Query Inject is compatible with the internal representations of different DiTs remains to be explored.
The filtered 7K data from WISA-80K is still relatively small: 91.25% of the data was discarded; although quality improved, coverage of long-tail physical phenomena might be lost. Distillation or synthetic data methods could be more sustainable than pure filtering.
Singular physical evaluation metrics: Evaluation on PhyGenBench and VBench relies on GPT-4o as a referee, whose physical judgment accuracy has not been calibrated. The lack of quantitative physical metrics (such as motion trajectory error or collision detection accuracy) makes fine-grained comparison of physical fidelity across different methods difficult.

vs. DiT-Mem: DiT-Mem also guides DiT generation by retrieving video memories, but it retrieves from a general-domain database and modulates DiT using frequency-filtered memory tokens, lacking dedicated modeling of physical attributes. PhysRAG's physical video database and physical prior extraction mechanism outperform DiT-Mem by 0.02 on PhyGenBench, demonstrating that domain-specific retrieval and injection are more effective than general memory modulation for physical scenarios. Insight: Domain-specific retrieval sources and injection mechanisms could be a generalizable paradigm for improving generation controllability in specific scenarios (e.g., medical, remote sensing, industrial).
vs. PhyT2V / PhysCtrl (Explicit Physics Methods): Explicit methods use physical simulators or mathematical constraints to guarantee physical consistency, but offer limited generalization. PhysRAG takes a middle ground—using real-world videos instead of simulators as physical reference sources, preserving the data-driven generalization of implicit methods while gaining the reference controllability of explicit methods. This "using real data to replace simulators" idea can be extended to other difficult-to-model domains (e.g., fluid simulation, soft-body deformation).
vs. VideoDPO / PhysGDPO (Preference Alignment Methods): Preference alignment methods indirectly guide physical consistency via scalar reward signals, which are sparse and hard to pin to specific frames. PhysRAG provides frame-level spatiotemporal references through retrieved videos, yielding much higher signal density and precision than scalar rewards. The two could be complementary: RAG-based dense physical guidance plus DPO-based high-level preference fine-tuning.
vs. RDM / Re-Imagen (Retrieval-Augmented Image Generation): Retrieval augmentation in the image domain is mainly used to improve fidelity and diversity, injecting appearance and style information. PhysRAG extends retrieval augmentation to the video domain, focusing on physical dynamics—retrieving "laws of motion" rather than "what things look like". The combination of VideoMAE + learnable queries serves as a viable blueprint for RAG in the video domain.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First to introduce the RAG paradigm to physics-aware video generation. The derivation from cognitive analogy to method design is clear and convincing, rather than a simple A+B concatenation.
Experimental Thoroughness: ⭐⭐⭐⭐ Main experiments cover both PhyGenBench and VBench benchmarks, and ablation studies span 5 dimensions (injection method, training strategy, data quality, injection layers, computational overhead). However, it lacks validation on multiple backbones other than Wan2.2 and lacks fine-grained physical evaluation metrics.
Writing Quality: ⭐⭐⭐⭐ The cognitive analogy for the motivation is well-written, and the methodology formulas are clear. However, there are LaTeX leftovers in some table references (e.g., LABEL:tab:physgenbench), and the categorization criteria for the 17 classes in the physical database are not detailed enough.
Value: ⭐⭐⭐⭐⭐ Physics awareness is a key bottleneck in transitioning video generation from "looking real" to "moving correctly". The RAG paradigm avoids the generalization dilemma of explicit physics modeling and the control difficulties of implicit methods, representing a promising middle path. The information bottleneck design is also a general contribution to the retrieval-augmentation field.