Q-TriM: Question-Guided Tri-Modal Attention for Audio–Visual Question Answering¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Sunghun95/Q-TriM
Area: Audio & Speech / Multimodal VLM
Keywords: Audio-Visual Question Answering, Tri-Modal Attention, Shallow Parallel Fusion, Token Filtering, Out-of-Distribution Robustness
TL;DR¶
Addressing the information loss and error propagation caused by deeply stacked attention layers in audio-visual question answering, Q-TriM introduces a shallow, parallel fusion architecture featuring question-guided Tri-Modal Attention (where Query, Key, and Value come from distinct modalities) and adaptive modality mixture, achieving a state-of-the-art 70.89% overall accuracy on MUSIC-AVQA-R.
Background & Motivation¶
Audio-Visual Question Answering (AVQA) requires intelligent models to jointly interpret continuous video sequences and time-synchronized audio streams under the guidance of natural language questions. With the increasing demand for embodied intelligent assistants, long-form episodic memory systems, and immersive audio-visual dialogue agents, models must not only identify visual sounding sources but also accurately localize transient acoustic events and resolve causal audio-visual correlations across complex scenes. However, modeling fine-grained cross-modal dependencies across three distinct modalities simultaneously introduces severe computational challenges and alignment difficulties.
Conventional AVQA architectures (such as PSTP-Net, TSPM, and the recent state-of-the-art QA-TIGER) predominantly rely on deeply stacked cross-attention and self-attention layers. For instance, QA-TIGER sequentially stacks four cross-attention blocks to capture multi-modal interactions stage by stage. While stacking neural layers facilitates semantic abstraction and compression in accordance with the Information Bottleneck principle, it inevitably incurs non-negligible information loss. In a multi-modal reasoning pipeline, subtle acoustic nuances or fine-grained spatial cues processed in earlier layers can be easily washed out before reaching deeper fusion stages, causing compounding errors across the sequence of attention blocks. Because conventional cross-attention inherently restricts interaction to two modalities at a time (a single Query attending to a paired Key-Value source), existing methods are forced to build deep, multi-stage sequential pipelines.
To eliminate the representational degradation and compounding error inherent in deep stacks, this paper advocates for a shallow and parallel fusion paradigm. By reformulating multi-modal attention from the perspective of conditional joint distributions, the authors demonstrate that assigning Query, Key, and Value directly across three distinct modalities allows three-way interactions to occur within a single stage. The core idea is to replace deeply stacked sequential attention with a single-stage, shallow parallel fusion framework that derives Tri-Modal Attention operators (with distinct Query, Key, and Value modalities) from a text-conditioned joint distribution, coupled with attention-free STE visual token filtering and adaptive modality gating to achieve robust cross-modal reasoning without depth-induced error accumulation.
Method¶
Overall Architecture¶
The Q-TriM architecture comprises three compact, coordinated stages: multi-modal representation extraction with attention-free visual token filtering, single-stage parallel computation of tri-modal and cross-modal attentions, and question-conditioned gated mixture followed by answer reasoning. Given an input video, it is segmented into \(T\) contiguous 2-second clips to extract global frame-level features and patch-level visual tokens via CLIP, while frame-aligned audio features are extracted via ImageBind and question text is encoded into a sentence embedding via CLIP. An attention-free token filtering module first scores and retains the most text-relevant visual patches via a straight-through estimator (STE). Next, four parallel attention paths are computed simultaneously: two standard cross-attentions and two Tri-Modal Attentions. Finally, the four feature streams are dynamically weighted by text-dependent gating coefficients, concatenated with the text query, and processed by a single self-attention layer before being classified by a linear prediction head.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Multimodal Input<br/>Text t / Video Frames V & Patches Vp / Audio A"] --> TF["Question-Guided Token Filtering<br/>Dot-Product Scoring & STE Top-K Patch Selection"]
TF --> TMA["Tri-Modal & Cross-Modal Attention<br/>Four Heterogeneous Q-K-V Operators in Parallel"]
TMA --> MM["Modality Mixture & Gated Reasoning<br/>FiLM Modulation & Question-Conditioned Gating"]
MM --> Out["Self-Attention & Answer Classification<br/>5×D Joint Token Aggregation & Linear Classifier"]
Key Designs¶
1. Question-Guided Token Filtering: Attention-Free Patch Pruning via Straight-Through Estimation
Raw visual frames yield dense patch token sequences containing substantial background clutter and task-irrelevant regions. Processing all spatial patches incurs heavy computational overhead and introduces distracting noise into multi-modal fusion. However, introducing an auxiliary cross-attention layer to perform visual grounding would run counter to the goal of avoiding deep attention stacks. Q-TriM resolves this by designing an attention-free scoring and selection module. The question embedding \(\mathbf{t}\) is first projected into the patch channel dimension as \(\mathbf{t}' = W_t \mathbf{t} + \mathbf{b}_t\), and compatibility scores with all patches \(\mathbf{v}_t^{\mathrm{p}}\) of the \(t\)-th frame are directly computed via scaled dot products: $\(\mathbf{s}_t = \frac{\mathbf{t}' (\mathbf{v}_t^{\mathrm{p}})^\top}{\sqrt{D'}} \in \mathbb{R}^P\)$ The top-\(K\) scoring patches are collected according to a keep ratio \(k=0.5\) (\(K=P/2\)) to yield a discrete hard selection \(X_t^{\mathrm{hard}}\). To enable end-to-end backpropagation through this discrete top-\(K\) operation, the authors employ a Straight-Through Estimator (STE). Soft weights are calculated using temperature \(\tau\) as \(\mathbf{w}_t = \mathrm{softmax}(\mathbf{s}_t / \tau)\), and the top-\(K\) slice \(\mathbf{w}_t^{\mathrm{top}}\) is broadcast to compute a soft-reweighted tensor \(X_t^{\mathrm{soft}} = X_t^{\mathrm{hard}} \odot \mathbf{w}_t^{\mathrm{top}}\). The final filtered patch representation is formed using the stop-gradient operator \(\mathrm{sg}(\cdot)\): $\(\tilde{\mathbf{v}}_t^{\mathrm{p}} = \mathrm{sg}(X_t^{\mathrm{hard}}) + \left( X_t^{\mathrm{soft}} - \mathrm{sg}(X_t^{\mathrm{soft}}) \right)\)$ In the forward pass, this expression strictly outputs the uncorrupted hard tokens \(X_t^{\mathrm{hard}}\), while in the backward pass, gradients flow through \(\mathbf{w}_t^{\mathrm{top}}\) back into \(\mathbf{s}_t\) and \(W_t\). After filtering, a single lightweight self-attention layer is applied independently to each stream (selected patches, global video frames, audio, and text) for intra-modal refinement.
2. Tri-Modal Attention: Single-Stage Three-Way Interaction Breaking the Sequential Stacking Bottleneck
Standard multi-modal attention is restricted to pairwise interactions via Query and Key-Value pairs, which necessitates cascading multiple layers when handling three modalities. The authors reconsider this interaction from a probabilistic perspective by formulating a Conditional Similarity Score (CSS) matrix \(\mathbf{S} \in \mathbb{R}^{T \times T}\) between the \(i\)-th video frame and \(j\)-th audio frame conditioned on question \(\mathbf{t}\), where \([\mathbf{S}]_{i, j} = s(i, j | \mathbf{t})\). Applying a 2D softmax across \(\mathbf{S}\) yields the joint probability distribution \(P_{\mathbf{S}}(i, j | \mathbf{t})\). When the default CSS is formulated as an additive decomposition \(s(i, j | \mathbf{t}) = \mathbf{v}_i \mathbf{t}^\top + \mathbf{a}_j \mathbf{t}^\top\), taking the conditional expectation of video features yields: $\(\mathbb{E}[\mathbf{V} | \mathbf{t}] = \sum_{i, j} P_{\mathbf{S}}(i, j | \mathbf{t}) \cdot \mathbf{v}_i = \mathrm{Attn}(\mathbf{t}, \mathbf{V}, \mathbf{V})\)$ which corresponds to standard cross-modal attention with text as the Query. To capture the strict temporal co-occurrence of audio-visual events, the authors propose a Cross-Reference CSS: \(\bar{s}(i, j | \mathbf{t}) = \mathbf{a}_i \mathbf{t}^\top + \mathbf{v}_j \mathbf{t}^\top\). Here, the \(i\)-th audio representation is used to evaluate the relevance of the \(i\)-th video frame conditioned on the question. The corresponding matrix is the transpose \(\mathbf{S}^\top\). Taking conditional expectations under \(P_{\mathbf{S}^\top}\) derives the novel Tri-Modal Attention (TMA) operator: $\(\mathbb{E}[\mathbf{V} | \mathbf{t}] = \mathrm{Attn}(\mathbf{t}, \mathbf{A}, \mathbf{V})\)$ $\(\mathbb{E}[\mathbf{A} | \mathbf{t}] = \mathrm{Attn}(\mathbf{t}, \mathbf{V}, \mathbf{A})\)$ In \(\mathrm{Attn}(\mathbf{t}, \mathbf{A}, \mathbf{V})\), text acts as Query, audio acts as Key, and video patches act as Value. Its physical intuition is that text and audio jointly determine which temporal timestamps are critical to answering the query, and those acoustic-textual attention weights are directly used to aggregate corresponding visual video patches. All three modalities directly interact within a single attention calculation. Furthermore, the authors prove via Log-Sum-Exp smooth max approximations that a mixed CSS expectation equals the exact uniform mixture of standard cross-attention and tri-modal attention, providing solid theoretical ground for horizontal parallel fusion.
3. Modality Mixture and Gated Reasoning: Text-Conditioned Gating and Global Self-Attention Convergence
From the probabilistic derivation, an optimal multi-modal representation requires synthesizing both cross-attention and tri-modal attention outputs. The model computes four parallel attention outputs: \(r_1 = \mathrm{Attn}(\mathbf{t}, \mathbf{V}^{\mathrm{p}}, \mathbf{V}^{\mathrm{p}})\), \(r_2 = \mathrm{Attn}(\mathbf{t}, \mathbf{A}, \mathbf{A})\), \(r_3 = \mathrm{Attn}(\mathbf{t}, \mathbf{A}, \mathbf{V}^{\mathrm{p}})\), and \(r_4 = \mathrm{Attn}(\mathbf{t}, \mathbf{V}, \mathbf{A})\). To preserve feature stability across disparate question types, Keys and Values are modulated by text using Feature-wise Linear Modulation (FiLM): \(K' = \mathrm{FiLM}(K; \mathbf{t}) = K \odot (\mathbf{1} + W_\gamma \mathbf{t}) + W_\beta \mathbf{t}\) (with zero-initialized weights). Because different questions rely unevenly on visual, acoustic, or audio-visual synergy, the model projects question vector \(\mathbf{t}\) into four dynamic gating scalars: $\((q_1, q_2, q_3, q_4) = W_g \mathbf{t} + \mathbf{b}_g, \qquad g_i = 1 + \frac{1}{2} \tanh(q_i)\)$ The modulated attention streams become \(h_i = g_i r_i\). Finally, the question embedding \(\mathbf{t}\) and the four gated representations are concatenated along the token sequence dimension to form \(\mathbf{H} = [\mathbf{t}; h_1; h_2; h_3; h_4] \in \mathbb{R}^{5 \times D}\). A single self-attention layer \(\mathrm{SA}(\mathbf{H}, \mathbf{H}, \mathbf{H})\) facilitates global message passing across all five streams, and the pooled output is projected through a linear classifier to produce the final answer.
Loss & Training¶
The network is trained end-to-end using cross-entropy loss with label smoothing (smoothing factor 0.1): $\(\mathcal{L}_{\mathrm{CE}} = - \sum_{c=1}^C y_c \log \hat{y}_c\)$ Optimization is performed using AdamW with a step learning rate scheduler that decays the learning rate by a factor of 0.1 every 8 epochs. All attention modules and the classification layer employ a dropout rate of 0.1. A global gradient norm clipping of 0.1 is enforced to stabilize training. The entire model trains efficiently on a single NVIDIA A6000 GPU.
Key Experimental Results¶
Main Results¶
The method is comprehensively evaluated across MUSIC-AVQA-R (probing out-of-distribution robustness and long-tail distributions across Head and Tail categories), MUSIC-AVQA, and MUSIC-AVQA-v2.0 (containing balanced and biased splits).
| Dataset | Evaluation Split / Metric | Q-TriM (Ours) | Previous SOTA (QA-TIGER / TSPM) | Gain over SOTA |
|---|---|---|---|---|
| MUSIC-AVQA-R | Overall Average (%) | 70.89 | 67.99 (QA-TIGER) | +2.90 |
| MUSIC-AVQA-R | Audio QA Count (Head / Tail) (%) | 85.68 / 76.12 | 82.67 / 75.82 | +3.01 / +0.30 |
| MUSIC-AVQA-R | Visual QA Local (Head / Tail) (%) | 89.45 / 78.95 | 84.76 / 75.59 | +4.69 / +3.36 |
| MUSIC-AVQA-R | Audio-Visual QA Count (Head / Tail) (%) | 81.43 / 44.21 | 76.70 / 33.55 | +4.73 / +10.66 |
| MUSIC-AVQA | Overall Average (%) | 77.68 | 77.62 (QA-TIGER) | +0.06 |
| MUSIC-AVQA | Visual QA Average (%) | 85.76 | 85.14 (QA-TIGER) | +0.62 |
| MUSIC-AVQA | Audio-Visual QA Average (%) | 73.78 | 73.74 (QA-TIGER) | +0.04 |
| MUSIC-AVQA-v2.0 | Bias Train \(\to\) Bias Test Avg (%) | 78.40 | 76.93 (QA-TIGER) | +1.47 |
| MUSIC-AVQA-v2.0 | Bias Train \(\to\) Balance Test Avg (%) | 75.31 | 73.91 (QA-TIGER) | +1.40 |
| MUSIC-AVQA-v2.0 | Balance Train \(\to\) Bias Test Avg (%) | 76.92 | 76.57 (QA-TIGER) | +0.35 |
| MUSIC-AVQA-v2.0 | Balance Train \(\to\) Balance Test Avg (%) | 76.29 | 76.43 (QA-TIGER) | -0.14 |
Ablation Study¶
The component contributions are validated on the challenging MUSIC-AVQA-R benchmark.
| Configuration | Audio QA (%) | Visual QA (%) | Audio-Visual QA (%) | Overall Avg (%) | Drop vs. Full Model |
|---|---|---|---|---|---|
| Full Model (Ours) | 74.74 | 83.64 | 62.85 | 70.89 | - |
| w/o Token Filtering | 74.54 | 80.36 | 61.54 | 69.21 | -1.68 |
| w/o Tri-Modal Attention | 73.13 | 81.02 | 61.10 | 68.92 | -1.97 |
| w/o Modality Mixture | 72.08 | 78.21 | 60.96 | 67.85 | -3.04 |
| w/o All Modules (Base) | 70.48 | 80.83 | 58.43 | 66.96 | -3.93 |
Hyperparameter sensitivity on the token filtering keep ratio \(k\) (MUSIC-AVQA-R): - \(k=0.1\): Avg 70.11% (retaining only 10% of patches still outperforms Base by 3.15%, demonstrating strong compression robustness) - \(k=0.3\): Avg 70.56% - \(k=0.5\) (default): Avg 70.89% (optimal accuracy) - \(k=0.7\): Avg 70.39% - \(k=0.9\): Avg 70.50%
Key Findings¶
- Substantial Gains in Out-of-Distribution Robustness: On the debiased MUSIC-AVQA-R benchmark, Q-TriM outperforms prior SOTA QA-TIGER by 2.90% overall (70.89% vs. 67.99%). Most notably, on rare tail-category samples within Audio-Visual QA Count, accuracy jumps from 33.55% to 44.21% (+10.66%), indicating that shallow parallel fusion resists memorizing frequency-driven dataset shortcuts.
- Critical Necessity of Tri-Modal Attention and Modality Mixture: Removing Modality Mixture degrades overall performance by 3.04%, while removing Tri-Modal Attention drops accuracy by 1.97%. On the tripartite Audio-Visual QA category, removing TMA reduces accuracy from 62.85% to 61.10%, validating the necessity of direct three-way interaction across Q, K, and V.
- Robust High-Compression Spatial Pruning: Filtering out 90% of visual patches (\(k=0.1\)) maintains a competitive 70.11% accuracy, confirming that the STE-guided dot-product selector effectively isolates sounding instruments while discarding irrelevant background artifacts.
Highlights & Insights¶
- First-Principles Derivation of Tri-Modal Attention: Rather than heuristically stacking cross-attention layers, Q-TriM derives the assignment of distinct Query, Key, and Value modalities from conditional joint probability distributions, providing a mathematically grounded basis for multi-modal operator design.
- Attention-Free Differentiable Token Pruning: By pairing discrete top-\(K\) selection with straight-through estimation (STE), the model filters visual tokens cleanly without introducing auxiliary attention layers, preventing recursive depth expansion.
- Shallow-and-Parallel Design Philosophy: The work provides compelling empirical evidence that multi-modal fusion does not inherently require deep architectures; horizontal parallelization mitigates inter-modal cue degradation and delivers superior generalization on debiased and out-of-distribution benchmarks.
Limitations & Future Work¶
- Author-Acknowledged Limitations: The exploration of Conditional Similarity Scores remains focused on linear dot products and additive combinations; more complex non-linear similarity kernels remain unexplored. Additionally, evaluations are centered around musical instrument domains (MUSIC-AVQA series) rather than general open-domain video reasoning.
- Observed Limitations: The model relies on frozen representations from deep foundation models (CLIP and ImageBind), whose pre-trained boundaries constrain downstream expressiveness. Furthermore, on the perfectly balanced training and testing split of MUSIC-AVQA-v2.0, Q-TriM (76.29%) slightly trails QA-TIGER (76.43%), where deep Gaussian expert gating still maintains a slight edge.
- Future Directions: The CSS conditional expectation framework can be extended to multi-party dialogue reasoning and embodied navigation. It also presents a lightweight candidate to replace heavy multi-layer Q-Former projection blocks in modern Multimodal Large Language Models (MLLMs).
Related Work & Insights¶
- vs. QA-TIGER (CVPR 2025): QA-TIGER sequentially stacks four cross-attention blocks with Gaussian experts, which risks attenuating subtle inter-modal cues across depth. Q-TriM proves that a single-stage parallel fusion with Tri-Modal Attention achieves a +2.90% improvement on MUSIC-AVQA-R with a much shallower footprint.
- vs. PSTP-Net (ACM MM 2023) & TSPM (ACM MM 2024): Previous methods decouple temporal selection, spatial localization, and question fusion into multi-stage pipelines that suffer from cascading errors. Q-TriM consolidates spatial filtering and spatio-temporal fusion into an integrated, horizontal framework that improves noise resilience.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Mathematically derives heterogeneous Q-K-V Tri-Modal Attention from conditional expectation distributions]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across MUSIC-AVQA, MUSIC-AVQA-R, and MUSIC-AVQA-v2.0 with thorough ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical exposition, well-structured arguments, and clean visual schematics]
- Value: ⭐⭐⭐⭐☆ [Presents a compelling shallow-and-parallel alternative to deep attention stacking in multi-modal reasoning]