Latent Visual Diffusion Reasoning with Monte Carlo Tree Search¶

Conference: ECCV2026
arXiv: 2606.27988
Code: https://github.com/XiruiTeng/LVDR_Official.git
Area: Video Understanding / Multimodal VLM
Keywords: Action Quality Assessment, Visual Reasoning, Diffusion Models, MCTS, Interpretability

TL;DR¶

The LVDR framework is proposed to model the cognitive process of gradual convergence from uncertainty to certainty in fine-grained skill assessment as latent space diffusion denoising. It uses keypoint-guided Monte Carlo Tree Search to extract interpretable reasoning trajectories, achieving state-of-the-art scoring accuracy across four sports and surgical datasets while outputting transparent decision-making rationales.

Background & Motivation¶

Fine-grained action skill assessment—such as determining whether a gymnastics movement is standard or a surgical operation is proficient—is far more challenging than simple "action recognition." While action recognition only answers "what was done," skill assessment must answer "how well it was done." This requires the model to not only capture low-level motion dynamics (joint angles, velocity changes, pose coordination) but also progressively infer high-level evaluative judgments from the observed motion features. Existing Action Quality Assessment (AQA) methods have achieved significant progress in predictive accuracy; however, they remain essentially black boxes—taking a video as input and outputting a score, while the intermediate logic of which body parts or movements are being monitored and why the score is given remains completely invisible. In high-risk scenarios such as athletic training and surgical education, experts not only need to know the score but also the underlying "why" in order to provide targeted feedback or verify the reliability of the judgment.

The Key Challenge of this dilemma lies in the fact that frame-by-frame labeling of reasoning trajectories is virtually impossible; it is unfeasible to annotate which joint the model is currently attending to and what judgment is being formed for every single frame. Consequently, previous interpretable methods either rely on manually defined scoring rules (such as NS-AQA using rule engines to analyze visual symbols) or can only provide the contribution weight of each video segment to the final score (such as Interpretability-AQA), failing to model the continuous, step-by-step evolution of the reasoning process. This paper proposes an ingenious Key Insight: since reasoning paths cannot be annotated directly, the reasoning process itself can be modeled as a progressive convergence from uncertainty to certainty. At the beginning of a video, the model's understanding of the overall action is highly "noisy," but it gradually "denoises" and converges to an accurate judgment as more frames are observed—a logic that naturally aligns with the mathematical framework of diffusion models. Core Idea: The visual reasoning process is treated as a diffusion denoising trajectory in the latent space. The model progressively purifies from a noisy initial state along the time dimension to a target semantic distribution, and then keypoint-guided Monte Carlo Tree Search is employed to extract the most relevant key reasoning steps associated with the final judgment. This allows the model to "explain" its decision-making process while maintaining high scoring accuracy.

Method¶

Overall Architecture¶

The LVDR framework consists of two main components: (1) a latent space visual reasoning module that models the reasoning process as a diffusion denoising trajectory along the time dimension; and (2) a keypoint-guided MCTS that extracts explicit, interpretable visual evidence from the latent reasoning embeddings.

Given a video of \(T\) frames, the model simultaneously outputs the predicted skill score and the keypoint attention patterns for each frame as the reasoning process. The reasoning module first extracts 3D human keypoints for each frame, which are input into the reasoning state learner along with the frame image to obtain the initial reasoning embedding \(P_i\). The diffusion Transformer takes the embedding sequence as a condition to perform progressive denoising along the temporal dimension: the denoising result of a subsequent frame is conditioned bilaterally on the denoised embedding of the previous frame, forcing the entire reasoning trajectory to gradually converge from a "noisy" initial state to the target distribution. Subsequently, MCTS builds a keypoint decision tree on each frame's latent reasoning embedding, iteratively searching and selecting the keypoint reasoning path that contributes the most to the score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input video of T frames<br/>I_1,...,I_T"] --> B["Extract 3D<br/>keypoints K_i per frame"]
    B --> C["Reasoning State Learner<br/>P_i = θ_R(I_i, K_i)"]
    C --> D["Diffusion Transformer<br/>Conditioned on P_{i-1}^<br/>Denoises to get P_i^"]
    D --> E["Fully Connected Layer → Predicted Score s^"]
    D --> F["Complete Reasoning Trajectory<br/>P_1^ → P_2^ → ... → P_T^"]
    F --> G["MCTS Search<br/>Constructs Joint Decision Tree per Frame<br/>Four-Stage Iterative Replacements"]
    G --> H["Output Highest Reward Path<br/>Keypoint Attention Frequency Map"]

Key Designs¶

1. Latent Space Visual Diffusion Reasoning: Modeling Cognitive Convergence as a Denoising Process

The Key Insight is to describe the evolution of reasoning using the language of diffusion models: at the beginning of the video, information is extremely scarce, and the reasoning embedding is in a "highly noisy" state (corresponding to random noise); as more frames are observed, the model progressively denoises, and the reasoning embedding becomes increasingly closer to a "clean" target distribution based on the complete video semantics. This bypasses the fundamental difficulty of the lack of frame-by-frame annotations for reasoning trajectories—the model only needs to learn a trajectory from noise to the target distribution, and this trajectory itself can be interpreted as the evolution of the reasoning process.

In practice, the target distribution is derived from expert textual feedback: first, a frozen text encoder (reused from InternVideo2.5) encodes the feedback into a clean text embedding \(z_c\), and a cosine-scheduled forward diffusion process is defined to progressively add noise. During training, a timestep \(t\) is randomly sampled, and a denoising autoencoder \(\epsilon_\theta\) is trained to predict the added noise from the noisy embedding. For each frame, the reasoning state learner \(\theta_R\) takes the frame image and 3D keypoints as input to produce an initial reasoning embedding \(P_i\); the diffusion Transformer \(\theta_D\) receives the denoised result of the previous frame \(P_{i-1}^\) and the current frame's initial embedding \(P_i\) as conditioning inputs, outputting the current frame's denoised reasoning embedding \(P_i^\). The ingenuity of this design lies in: the temporal conditional dependency between frames forces temporal continuity in the reasoning trajectory—the model cannot make "jumping" changes in judgment, as each update must build upon the prior reasoning, aligning with the human cognitive process of gradually forming judgments. Finally, \(P_T^\) is passed through a fully connected layer to regress the predicted score.

2. Keypoint-Guided MCTS: Extracting Explicit Attention Patterns from Latent Reasoning Trajectories

Once the reasoning trajectory in the latent space is obtained, it needs to be translated into a visual format directly understandable by humans—specifically, which joint the model is looking at and what motion cues it is focused on in each frame. Inspired by the cognitive process of human judges, who do not look at all body parts simultaneously but rather sequentially focus on different regions (e.g., first looking at the upper limbs, then shifting to the lower limbs) to form a comprehensive judgment, this sequential attention strategy is modeled as a combinatorial search problem.

In sports scenarios, each tree node in MCTS per frame is defined by four major joints (elbow, shoulder, hip, knee), with each node represented by a triplet \((a, v, p)\) comprising the joint's angle \(a\), velocity \(v\), and normalized 3D position \(p\); surgical scenarios correspond to three parts of the instrument (tip, body, tail). The tree depth is set to 4 (sports) or 3 (surgery), and a search tree is independently constructed for each frame. The standard four-stage MCTS iteration is executed \(M\) times (default is 150): UCT child node selection \(\rightarrow\) unvisited path expansion \(\rightarrow\) simulation policy evaluation using the embedding of the diffusion model \(\theta_D\) to assess the expected reward \(\rightarrow\) backpropagation to update node values. After \(M\) iterations, the path with the highest expected cumulative reward is selected, and the sequence of keypoints along this path represents the model’s focus of attention in that frame.

During the inference stage, the latent reasoning embedding is first obtained through the diffusion module, and then MCTS is applied to extract explicit keypoint attention. Keypoint masking experiments cross-validated the effectiveness of the search: masking the keypoints selected by MCTS led to a dramatic drop in performance (Cataract-101 accuracy dropped from 0.71 to 0.53), whereas masking unselected keypoints had almost no impact. A user study also showed that domain experts considered 86% of the generated reasoning trajectories as "correct," verifying that the keypoints located by MCTS indeed align with the judgments of human referees.

Loss & Training¶

The total loss is a weighted sum of the diffusion denoising loss and the score regression loss:

\[L = L_{DM} + \lambda * L_{score}\]

where \(L_{DM}\) is the standard noise prediction MSE, \(L_{score}\) is the MSE between the predicted score and the ground truth score, and \(\lambda\) is set to 1 (ablation study indicates \(\lambda=1\) outperforms 0.5 and 2). A cosine schedule is selected for the noise schedule instead of a linear schedule (experiments prove it to be significantly superior). The vision and text encoders freeze the pre-trained weights of InternVideo2.5. The diffusion Transformer has 12 layers, 16 heads, and a 512-dimensional hidden state. The training uses the Adam optimizer, a learning rate of 1e-4, batch size of 8, on a single NVIDIA L40S GPU. During MCTS inference, the default is 150 iterations, taking approximately 0.094 seconds per frame.

Key Experimental Results¶

Main Results¶

Dataset	Metric	LVDR (Ours)	Prev. SOTA	Gain
EgoExo4D (Overall)	\(\rho\uparrow\)	0.88	MAGR (0.73)	+0.15
EgoExo4D (Overall)	\(R\text{-}l_2\downarrow\) (\(\times 100\))	1.69	MAGR (10.14)	-8.45
JIGSAWS (Overall)	\(\rho\uparrow\)	0.69	MAGR++ (0.55)	+0.14
JIGSAWS (Overall)	\(R\text{-}l_2\downarrow\) (\(\times 100\))	9.93	MAGR++ (10.20)	-0.27
FitnessAQA (OHP Elbow F1)	F1\(\uparrow\)	0.818	MAGR (0.357)	+0.461
FitnessAQA (Squat Inward F1)	F1\(\uparrow\)	0.920	MAGR (0.157)	+0.763
Cataract-101	Acc\(\uparrow\)	0.71	MAGR (0.62)	+0.09

Ablation Study¶

Configuration	EgoExo4D \(\rho\uparrow\)	Cataract-101 Acc\(\uparrow\)	FitnessAQA (Elbow) F1\(\uparrow\)	Description
Full LVDR	0.88	0.71	0.818	—
w/o Diffusion Module	0.83	0.67	0.497	Consistent and significant performance drop across all datasets
Masking MCTS Selected Keypoints	0.86	0.53	0.282	Significant performance drop, verifying the importance of keypoints
Masking MCTS Unselected Keypoints	0.88	0.71	0.818	Almost no impact
Linear Noise Schedule	0.85	0.67	—	Inferior to cosine schedule
\(\lambda=0.5\)	0.86	0.67	—	—
\(\lambda=2\)	0.86	0.67	—	—

Key Findings¶

The diffusion module is the core guarantee of reasoning quality: removing it leads to a consistent performance drop across all datasets (e.g., FitnessAQA Elbow F1 plummets from 0.818 to 0.497), indicating that a unidirectional frame embedding sequence is insufficient to model reasoning evolution, and the progressive denoising structure provided by the diffusion process is indispensable.
The keypoints located by MCTS indeed carry the most critical motion cues: masking the selected keypoints causes the accuracy on Cataract-101 to plunge from 0.71 to 0.53, whereas masking unselected keypoints yields no impact, with these two control groups mutually validating each other.
The user study provides strong qualitative evidence: sports domain experts evaluated 86% of the MCTS reasoning trajectories as "correct" and only 8% as incorrect, demonstrating that the attention patterns learned by the model are highly consistent with human referee judgments.
Strong cross-domain generalization: the same framework achieves SOTA performance across four highly distinct datasets (sports videos, surgical videos, fitness videos), validating the universality and stability of the proposed method.
Controllable inference efficiency: with 225 MCTS iterations taking 0.129 seconds per frame, and 75 iterations taking only 0.048 seconds, the method possesses the potential for extension to real-time applications.

Highlights & Insights¶

Modeling diffusion as reasoning mathematically is the most critical design: aligning the denoising process with the cognitive transition "from uncertainty to certainty" bypasses the lack of reasoning trajectory annotations while producing a temporally structured latent reasoning space—which is inherently visualizable, analyzable, and searchable.
The dual-validation mechanism of MCTS is robust: it includes both masking experiments proving that "the selected parts are indeed critical" and user studies proving that "the selected parts match human choices." These two bodies of evidence reinforce each other, providing reliability far exceeding a single ablation study.
Translating the cognitive psychology intuition of "where the referee looks first and next" into the node definitions of the MCTS search tree represents an elegant integration of domain knowledge and algorithmic framework, yielding reasoning results with naturally human-comprehensible semantics.
The method is effective across highly distinct athletic (highly dynamic pose transitions) and surgical (fine instrument manipulation) scenarios without any domain-specific modifications, demonstrating that the generalization of latent diffusion reasoning exceeds expectations.

Limitations & Future Work¶

Currently, only single-person scenarios are supported; interactive attention patterns in multi-person/team sports (e.g., full-court basketball plays, gymnastics team events) are not yet covered.
The node definition of MCTS relies on manually selected joints or instrument components. In new domains (such as dance or musical instrument performance), domain experts must redesign the node space, which limits the level of automation.
The quality of the reasoning trajectory heavily depends on the quality of the feedback embeddings from the InternVideo2.5 text encoder—when feedback contains specialized terminology or is of low quality, the target distribution itself may be biased.
The inference speed of 0.129 seconds per frame with 225 MCTS iterations still falls short of 30fps real-time scenarios. Although 75 iterations reduce it to 0.048 seconds (approximately 21fps), a more systematic Pareto analysis between step count and accuracy is required.

vs. MAGR / MAGR++: Traditional SOTA AQA methods regress predicted scores end-to-end, which perform well but are completely black-box. LVDR achieves higher accuracy (EgoExo4D \(\rho\) of 0.88 vs. 0.73) while providing a transparent reasoning process.
vs. NS-AQA / IRIS: Perform interpretable assessment based on hand-crafted rules and score sheets, providing explanations but struggling to learn from data. LVDR is entirely data-driven, automatically learning reasoning trajectories without relying on predefined rules.
vs. Interpretability-AQA: Uses attention loss + segment-wise weighting to explain the contribution of each video segment, but still produces single weight values rather than a continuous reasoning trajectory. LVDR models the complete evolution of reasoning over time, offering a much higher density of interpretable information.
The paradigm combining diffusion and MCTS for reasoning interpretability can be transferred to other step-by-step video reasoning tasks (e.g., autonomous driving behavior prediction, surgical step identification), especially when "reasoning annotations are unavailable" but "masking verification is feasible."

Rating¶

Novelty: ⭐⭐⭐⭐ [Combining diffusion models with MCTS for AQA interpretability is a fresh approach, though paradigms combining diffusion and reasoning have precedents.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Four cross-domain datasets + multiple ablations + keypoint masking + user study + inference speed analysis, providing a complete chain of evidence.]
Writing Quality: ⭐⭐⭐⭐ [Clear framework, rich illustrations, and well-organized experiments; however, the description of the four MCTS stages is somewhat formulaic, and some paragraphs could be more concise.]
Value: ⭐⭐⭐⭐⭐ [Addresses the most practical issue in the AQA field—the lack of interpretability. The method is versatile and open-source, offering high practical value.]