Latent Visual Diffusion Reasoning with Monte Carlo Tree Search¶

Conference: ECCV2026
arXiv: 2606.27988
Code: https://github.com/XiruiTeng/LVDR_Official.git
Area: Video Understanding / Multimodal VLM
Keywords: Action Quality Assessment, Visual Reasoning, Diffusion Models, MCTS, Interpretability

TL;DR¶

The LVDR framework is proposed, which models the cognitive process of gradually converging from uncertainty to certainty in fine-grained skill assessment as latent space diffusion denoising, and extracts interpretable reasoning trajectories using keypoint-guided Monte Carlo Tree Search. It achieves scoring accuracy surpassing SOTA on four athletic and surgical datasets while outputting transparent decision bases.

Background & Motivation¶

Fine-grained action skill assessment—such as determining whether a gymnastics movement is standard or a surgical procedure is proficient—is far more difficult than simple "action recognition". Action recognition only answers "what was done", whereas skill assessment must answer "how well it was done". This requires the model to not only capture low-level motion dynamics (joint angles, velocity changes, pose coordination) but also reasoning from observed motion features to high-level evaluation judgments. Existing Action Quality Assessment (AQA) methods have made significant progress in prediction accuracy, but they are essentially black boxes—inputting a video and outputting a score, while what body parts and motion patterns they observe and what logic they base their scores on remain completely invisible. In high-risk scenarios like sports training and medical education, experts not only need to know the scores but also "why", in order to provide targeted feedback or verify the reliability of the judgment.

The key challenge of this dilemma lies in: annotating frame-by-frame reasoning trajectories is almost impossible—it is unfeasible to label for each frame "which joint the model is currently attending to and what judgment is being formed". Consequently, previous interpretable methods either rely on manually defined scoring rules (such as NS-AQA using rule engines to analyze visual symbols) or can only provide the contribution weights of each video segment to the final score (such as Interpretability-AQA), failing to model the continuous, step-by-step evolution of reasoning. This work proposes a clever key insight: since reasoning paths cannot be directly annotated, reasoning itself is modeled as a progressive process from uncertainty to certainty. At the beginning of a video, the model's understanding of the overall action is highly "noisy", and as more frames are observed, it gradually "denoises" and converges to an accurate judgment—a logic that naturally aligns with the mathematical framework of diffusion models. Core Idea: Visual reasoning is treated as a diffusion denoising trajectory in the latent space. The model gradually purifies from a noisy initial state along the time dimension to the target semantic distribution, and then a keypoint-guided Monte Carlo Tree Search is utilized to extract the key reasoning steps most relevant to the final judgment, allowing the model to "explain" its decision basis while maintaining high scoring accuracy.

Method¶

Overall Architecture¶

The LVDR framework consists of two main components: (1) a latent visual reasoning module that models the reasoning process as a diffusion denoising trajectory along the temporal dimension; (2) a keypoint-guided MCTS that extracts explicit, interpretable visual evidence from the latent reasoning embeddings.

Given a video of \(T\) frames, the model simultaneously outputs the predicted skill score and the keypoint attention patterns for each frame as the reasoning process. The reasoning module first extracts 3D human keypoints for each frame, which are input to the reasoning state learner along with the frame image to obtain the initial reasoning embedding \(P_i\). The diffusion Transformer takes the sequence of embeddings as a condition and denoises progressively along time: the denoising result of a subsequent frame is bi-directionally conditioned on the denoised embedding of the previous frame, allowing the entire reasoning trajectory to gradually converge from a 'noisy' initial state to the target distribution. Subsequently, MCTS constructs a keypoint decision tree on the latent reasoning embeddings of each frame, repeatedly searching and selecting the keypoint reasoning path that contributes the most to the score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Video of T Frames<br/>I_1,...,I_T"] --> B["Extract 3D<br/>Keypoints K_i per Frame"]
    B --> C["Reasoning State Learner<br/>P_i = θ_R(I_i, K_i)"]
    C --> D["Diffusion Transformer<br/>Conditioned on \hat{P}_{i-1}<br/>Denoises to get \hat{P}_i"]
    D --> E["FC Layer → Predicted Score \hat{s}"]
    D --> F["Complete Reasoning Trajectory<br/>\hat{P}_1 → \hat{P}_2 → ... → \hat{P}_T"]
    F --> G["MCTS Search<br/>Build Joint Decision Tree per Frame<br/>4-Stage Iterative Loop"]
    G --> H["Output Path with Highest Reward<br/>Keypoint Attention Frequency Map"]

Key Designs¶

1. Latent Space Visual Diffusion Reasoning: Modeling Cognitive Convergence as a Denoising Process

The core insight is to describe the evolution of reasoning in the language of diffusion models: at the beginning of the video, information is extremely scarce, and the reasoning embedding is in a "highly noisy" state (corresponding to random noise); as more frames are observed, the model gradually denoises, and the reasoning embedding approaches closer and closer to the "clean" target distribution based on the complete video semantics. This circumvents the fundamental difficulty of the unavailability of frame-by-frame annotations for reasoning trajectories—the model only needs to learn a trajectory from noise to the target distribution, which itself can be interpreted as the evolution of the reasoning process.

Specifically, the target distribution comes from expert commentary text: first, a frozen text encoder (reusing InternVideo2.5) encodes the commentary into a clean text embedding \(z_c\), and a forward diffusion process with a cosine schedule is defined to add noise progressively. During training, timestep \(t\) is randomly sampled, and a denoising autoencoder \(\varepsilon_\theta\) is trained to predict the added noise from the noisy embedding. For each frame, a reasoning state learner \(\theta_R\) takes the frame image and 3D keypoints as input to produce an initial reasoning embedding \(P_i\); a diffusion Transformer \(\theta_D\) takes the denoising result of the previous frame \(\hat{P}_{i-1}\) and the initial embedding of the current frame \(P_i\) as conditions, outputting the denoised reasoning embedding \(\hat{P}_i\) of the current frame. The elegance of this design is that the conditional dependency between frames forces the continuity of the reasoning trajectory—the model cannot make "jump" changes in its judgment, and each update must build on prior reasoning, aligning with how humans progressively form judgments. Finally, \(\hat{P}_T\) is connected to a fully connected layer to regress the predicted score.

2. Keypoint-Guided MCTS: Extracting Explicit Attention Patterns from Latent Reasoning Trajectories

With the reasoning trajectory in the latent space, it is necessary to translate it into a visual form directly understandable by humans—specifically, which joints and motion cues the model is looking at in each frame. Inspired by the cognitive process of a referee's scoring, a referee does not look at all body parts at once but sequentially focuses on different regions (e.g., first looking at the upper limbs, then shifting to the lower limbs), eventually forming a comprehensive judgment. This sequential attention strategy is modeled as a combinatorial search problem.

In sports scenarios, the MCTS tree nodes for each frame are defined as four main joints (elbow, shoulder, hip, knee), with each node represented by a triplet \((a, v, p)\) of the joint's angle \(a\), velocity \(v\), and normalized 3D position \(p\); in surgical scenarios, they correspond to three parts of the instrument (tip, shaft, tail). The tree depth is set to 4 (sports) or 3 (surgery), and a search tree is constructed independently for each frame. The standard MCTS executes four stages iteratively for \(M\) times (defaulting to 150): UCT selection of child nodes \(\rightarrow\) expansion of unvisited paths \(\rightarrow\) prediction of expected rewards using the diffusion model \(\theta_D\)'s embeddings as the simulation policy \(\rightarrow\) backpropagation to update node values. After \(M\) iterations, the path with the highest expected cumulative reward is selected, and the sequence of keypoints on this path constitutes the model's focus of attention in that frame.

During the inference stage, the diffusion module is first executed to obtain latent reasoning embeddings, and then MCTS is applied to extract explicit keypoint attention. Keypoint masking experiments cross-verified the effectiveness of the search: masking keypoints selected by MCTS led to a sharp decrease in performance (Cataract-101 accuracy dropped from 0.71 to 0.53), while masking unselected keypoints had almost no effect. A user study also showed that domain experts deemed 86% of the generated reasoning trajectories as "correct", validating that the keypoints located by MCTS indeed align with human referees' judgments.

Loss & Training¶

The total loss is a weighted sum of the diffusion denoising loss and the score regression loss: L = L_DM + λ * L_score
where L_DM is the standard noise prediction MSE, L_score is the MSE between predicted and ground-truth scores, and λ is set to 1 (ablation studies show λ=1 outperforms 0.5 and 2). The noise schedule selects a cosine schedule instead of a linear schedule (experimentally proven to be significantly better). The visual and text encoders freeze the pre-trained weights of InternVideo2.5. The diffusion Transformer is a 12-layer / 16-head / 512-dimensional hidden-state network. Adam optimizer is used with a 1e-4 learning rate, a batch size of 8, and a single NVIDIA L40S GPU. During MCTS inference, 150 iterations are used by default, taking about 0.094 seconds per frame.

Key Experimental Results¶

Main Results¶

Dataset	Metric	LVDR (Full)	Prev. SOTA	Gain
EgoExo4D (General)	ρ↑	0.88	MAGR (0.73)	+0.15
EgoExo4D (General)	R-l₂↓ (×100)	1.69	MAGR (10.14)	-8.45
JIGSAWS (General)	ρ↑	0.69	MAGR++ (0.55)	+0.14
JIGSAWS (General)	R-l₂↓ (×100)	9.93	MAGR++ (10.20)	-0.27
FitnessAQA (OHP Elbow F1)	F1↑	0.818	MAGR (0.357)	+0.461
FitnessAQA (Squat Inward F1)	F1↑	0.920	MAGR (0.157)	+0.763
Cataract-101	Acc↑	0.71	MAGR (0.62)	+0.09

Ablation Study¶

Configuration	EgoExo4D ρ↑	Cataract-101 Acc↑	FitnessAQA (Elbow) F1↑	Description
Full LVDR	0.88	0.71	0.818	—
W/o diffusion module	0.83	0.67	0.497	Consistent significant drop across all datasets
Mask MCTS-selected keypoints	0.86	0.53	0.282	Drastic drop, confirming keypoint importance
Mask MCTS-unselected keypoints	0.88	0.71	0.818	Barely any impact
Linear noise schedule	0.85	0.67	—	Inferior to cosine schedule
λ=0.5	0.86	0.67	—	—
λ=2	0.86	0.67	—	—

Key Findings¶

The diffusion module is the core guarantee of reasoning quality: removing it leads to consistent performance drops across all datasets (FitnessAQA elbow F1 drops sharply from 0.818 to 0.497), indicating that a unidirectional sequence of frame embeddings is insufficient to model reasoning evolution, and the progressive denoising structure provided by the diffusion process is indispensable.
Keypoints located by MCTS indeed carry the most critical motion cues: masking the selected keypoints causes the accuracy on Cataract-101 to plummet from 0.71 to 0.53, while masking unselected keypoints has no effect, with the two sets of control experiments verifying each other.
User study provides strong qualitative evidence: sports domain experts evaluate 86% of the MCTS reasoning trajectories as "correct" and only 8% as incorrect, showing that the attention patterns learned by the model are highly consistent with the judgments of human referees.
High cross-domain generalizability: the same framework achieves SOTA on four highly diverse datasets (sports videos, surgical videos, fitness videos), verifying the generality and stability of the method.
Controllable inference efficiency: 225 MCTS iterations take 0.129 seconds per frame, and 75 iterations take only 0.048 seconds, demonstrating the potential for expansion to real-time applications.

Highlights & Insights¶

Modeling reasoning mathematically as diffusion is the most critical design: naturally aligning the denoising process with the cognitive process of "from uncertainty to certainty" bypasses the bottleneck of missing reasoning trajectory annotations and yields a temporally structured latent reasoning space—which itself is visualizer-friendly, analyzable, and searchable.
The dual validation mechanism of MCTS is very solid: not only do masking experiments prove that "what is selected is truly important", but a user study also shows that "what is selected is identical to what humans choose". These two sets of evidence cross-validate each other, making the results far more reliable than a single ablation study.
Translating the cognitive psychology intuition of "where a referee looks first and looks next" into the node definitions of the MCTS search tree represents an elegant integration of domain knowledge and algorithmic framework, yielding reasoning results with naturally human-comprehensible semantics.
The method is effective on both athletic (high-dynamic pose changes) and surgical (fine-grained tool manipulation) scenarios without any domain-specific modifications, indicating that the generalization of latent diffusion reasoning exceeds expectations.

Limitations & Future Work¶

Currently, only single-person scenarios are supported; interactive attention patterns in multi-person/team sports (e.g., full-court basketball coordination, team gymnastics) are not yet covered.
MCTS node definitions rely on manually selected joints or instrument parts. In new domains (such as dance, musical instrument playing), domain experts need to redesign the node space, which limits the level of automation.
The quality of the reasoning trajectory heavily depends on the quality of the commentary embeddings from the InternVideo2.5 text encoder—when commentary terminology is specialized or substandard, the target distribution itself may be biased.
The inference speed of 0.129s/frame with 225 MCTS iterations still falls short of the requirement for 30fps real-time scenarios. Although 75 iterations reduce it to 0.048s (~21fps), a more systematic Pareto analysis between steps and accuracy is still needed.

vs MAGR / MAGR++: Traditional AQA SOTA methods, which predict scores via end-to-end regression. They perform well in accuracy but are completely black-box. LVDR achieves higher accuracy (EgoExo4D ρ 0.88 vs 0.73) while providing a transparent reasoning process.
vs NS-AQA / IRIS: Interpretable evaluation based on manually defined rules and scoring checklists. They can provide explanations but cannot learn from data. LVDR is entirely data-driven, automatically learning reasoning trajectories without relying on pre-defined rules.
vs Interpretability-AQA: Uses attention loss + segment-wise weighting to explain the contribution of each video segment, but still produces a single weight value rather than a continuous reasoning trajectory. LVDR models the complete process of reasoning evolving over time, offering higher density of interpretable information.
The paradigm of combining diffusion + MCTS for reasoning interpretability can be transferred to other step-by-step video reasoning tasks (e.g., autonomous driving behavior prediction, surgical step recognition), particularly in scenarios where "reasoning process annotation is unavailable" but "masking validation is feasible".

Rating¶

Novelty: ⭐⭐⭐⭐ [Combining diffusion models with MCTS for AQA interpretability is a brand-new idea, though precedents exist for combining diffusion with reasoning]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Four cross-domain datasets + multiple ablations + keypoint masking + user study + inference speed analysis, establishing a complete chain of evidence]
Writing Quality: ⭐⭐⭐⭐ [Clear framework, rich illustrations, well-organized experiments, but the description of the four MCTS stages is somewhat formulaic, and some paragraphs could be more concise]
Value: ⭐⭐⭐⭐⭐ [Addresses the most practical issue in AQA—the lack of interpretability. The method is general and open-source, offering high practical value]