A Dual-Transformer Architecture with Cross-Attention for Multi-Camera View Recommendation¶
Conference: ECCV2026
Paper: Official paper page (5242) · Paper PDF
Code: https://gitlab.com/UOC/aiwell/automated-video-editing-research/eccv2026-dualtransformer
Area: Video Understanding
Keywords: Multi-camera view recommendation, Cross-attention, Temporal memory, Video editing, Focal loss
TL;DR¶
The paper separates the encoding of previously shown footage from the evaluation of current camera candidates, letting candidates query temporal memory through cross-attention; with frozen SwinV2-Tiny features and focal loss, it reaches 69.65% [email protected] on TVMCE, a thresholded recommendation metric rather than forced six-way selection accuracy.
Background & Motivation¶
Multi-camera editing requires more than choosing the clearest or most salient current image. A close-up, a wide shot, and a side view may all be plausible, but their suitability depends on what the audience has just seen. TVMCE supplies synchronized raw camera tracks and professional edits, avoiding the mismatch introduced when unrelated frames from edited footage are treated as simultaneous alternatives. The earlier TC-Transformer models historical footage and candidate views together, while Lee et al. add temporal offsets and camera identities to make editing context more explicit.
This paper changes the architecture rather than adding another editing rule or input modality. Its premise is that concatenating historical and candidate tokens mixes two different operations too early: constructing a representation of the past and comparing the options available now. The task boundary matters: the model receives an editing decision point and recommends a camera, rather than jointly determining when every cut should occur. Accordingly, its primary topic is video understanding and editing recommendation, not human understanding or actor-motion prediction.
Cross-attention imposes a simple information-flow constraint: encode the past first, allow each candidate to retrieve relevant context, and then compare the context-enriched candidates. Candidates can compete without participating in the construction of the historical memory itself. Core idea: represent editing history as independent temporal memory, let candidate views query that memory, and refine their relative representations afterward instead of mixing history and candidates in one encoder.
Method¶
Overall Architecture¶
Each decision uses 16 historical visual inputs and 6 synchronized candidate views, producing an independent recommendation score for each candidate. The paper alternates between “past frames” and “past shots”; historical visual inputs is used here without assuming an undocumented sampling scheme. A shared frozen visual backbone extracts image features, which a linear projection maps to 768 dimensions. Temporal memory, candidate queries, and candidate refinement then form the three main stages; identity embeddings, temporal encodings, and the prediction head support those stages rather than defining additional tasks.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["History and candidate images<br/>Frozen backbone and projection"] --> B["Temporal memory<br/>History encoder"]
A -->|"Candidate features and identities"| C["Candidate queries<br/>Candidate encoder and cross-attention"]
B -->|"Historical keys and values"| C
C --> D["Candidate refinement<br/>Self-attention and prediction head"]
D --> E["Six recommendation scores<br/>Thresholding or highest-score selection"]
During training, the downstream projection, embeddings, encoders, attention modules, and prediction head learn from editing labels while the visual backbone remains frozen. At inference, the general model can recommend views without target-video fine-tuning; local adaptation is an additional experiment, not a required recommendation stage. Sigmoid scores support either retaining several candidates above a threshold or forcing a single choice by taking the highest score. Those deployment choices must not be conflated when interpreting the metrics.
Key Designs¶
1. Temporal memory: encode the past without current-candidate interference
The historical branch adds visual features, temporal-offset encodings, and camera-identity information before a one-layer Temporal Transformer Encoder. Temporal offsets use sinusoidal encodings of integer frame distances, clamped at 500, expressing how far an input lies from the current decision rather than merely its sequence index. Camera identities use learned embeddings; Section 3.2 specifically refers to the current camera identity, so a more elaborate per-shot camera trajectory should not be inferred. Self-attention within the historical sequence produces memory with 16 positions rather than reducing the entire past to one average vector.
This branch prepares context rather than selecting the next camera directly. Its outputs become keys and values for cross-attention, and current candidates do not rewrite that memory during the decision. Compared with concatenated encoding, this creates a directional inductive bias: summarize what has already been shown, then evaluate what might be shown next. It does not establish a theoretical inability of a unified encoder to learn similar relationships.
2. Candidate queries: retrieve different historical evidence for different views
Each candidate combines its current visual feature with its own camera-identity embedding, without being appended to the historical sequence. A one-layer Candidate Transformer Encoder first models relationships among the available views. Consequently, the claim that candidates independently query history does not mean they are computationally isolated: they have already exchanged information within the candidate encoder. The encoded candidates provide queries, while historical memory supplies keys and values to an eight-head cross-attention module.
Using \(Q\) for the encoded candidates and \(M\) for memory, the residual update in the paper's Equation (5) is:
Each query weights historical positions according to its own representation and adds the retrieved information back to the candidate. A close-up and a wide shot can therefore consult different historical evidence instead of sharing only one global summary. The residual connection preserves current visual information rather than allowing history to replace it. Attention receives no separate supervision, and conventions such as shot–reverse-shot are not explicitly encoded as rules; any such behavior must emerge indirectly from editing labels.
3. Candidate refinement: compare views again after retrieving context
After cross-attention, a shallow four-head self-attention layer lets the enriched candidates interact once more. This differs from the earlier candidate encoder: the earlier interaction compares currently available views, whereas the later interaction compares views that now incorporate historical evidence. Describing the model as simply querying history and immediately scoring would omit this actual component. The final head uses Layer Normalization, GELU, and a two-layer MLP with a 192-unit hidden layer to produce one logit per candidate.
The model does not impose a six-class softmax whose probabilities sum to one. Instead, it treats each candidate as a binary decision, allowing several scores—or none—to exceed the evaluation threshold. Focal loss addresses the imbalance between one positive candidate and the remaining negative candidates by reducing the influence of easy examples. It changes the supervision, not the number of networks or the presence of a separate editing-style module.
A Worked Example¶
At an already specified cut decision, the system receives 16 historical inputs and the current images from 6 cameras. The historical branch encodes what has been shown, while the candidate branch represents the views currently available. A close-up query may retrieve context associated with the current subject, while a wide-shot query may retrieve other historical evidence; this illustrates the mechanism rather than an attention interpretation verified by the paper. After another round of candidate interaction, the system produces scores and either thresholds them or selects the highest-scoring view. If two candidates exceed 0.5 and one is the annotated camera, that sample's precision is \(1/2\), even if the correct view ranks first. This explains why thresholded precision and Recall@1 describe different aspects of the same model.
Loss & Training¶
Sections 3.3 and 4.1 use focal weighting on binary cross-entropy, decreasing the weight of a prediction as its probability for the correct class increases. The focal parameters are \(\alpha=0.25\) and \(\gamma=2.0\); a BCE variant is also evaluated to separate loss changes from architecture changes. The default visual backbone is SwinV2-Tiny pretrained on ImageNet-1K, with its weights frozen throughout general training. Downstream modules are trained for 30 epochs with AdamW at a learning rate of \(10^{-5}\). Local video adaptation starts from the general TVMCE model, uses different fractions of a target video's local training data, and evaluates on a separate local test split.
Key Experimental Results¶
Main Results¶
TVMCE contains 88 hours of raw footage and 14 hours of edited video spanning concerts, sports, gala shows, and contests. The following selected rows come from Table 1's TVMCE benchmark evaluation, not training-set fitting scores. All values are [email protected] in percent, higher is better; the paper follows the benchmark protocol but does not enumerate global train/validation/test sample counts in the text.
| Method | Backbone and loss | [email protected] ↑ | Interpretation |
|---|---|---|---|
| TC-Transformer, published result | Original setup | 22.50 | Earlier joint-encoding baseline |
| Lee et al., published result | Original setup | 37.16 | Previously reported best result |
| Single Transformer, author reimplementation | SwinV1 + BCE | 47.97 | Matched backbone/loss architecture reference |
| Single Transformer, author reimplementation | SwinV2 + BCE | 54.06 | Reimplemented baseline with a newer backbone |
| Dual Transformer | SwinV1 + BCE | 52.60 | 4.63 percentage points over the matched reimplementation |
| Dual Transformer | SwinV1 + Focal | 56.60 | 4.00 points over the preceding row |
| Dual Transformer | SwinV2 + Focal | 69.65 | Best combination, not an isolated architecture gain |
Lee et al. provide neither public code nor weights, so the single-Transformer rows are this paper's reimplementations rather than the original implementation. The gap between 37.16% and 47.97% already shows that implementation and training choices matter substantially. Attributing the entire difference between 69.65% and 37.16% to cross-attention would therefore be misleading. The matched 47.97% to 52.60% comparison better isolates the architectural change, but still does not separately test cross-attention, the extra encoder, and final candidate self-attention.
Ablation Study¶
The selected rows below come from Table 2, using the same TVMCE benchmark and [email protected] metric. Every configuration retains the Dual Transformer and focal loss, changing only the visual backbone. This is backbone sensitivity analysis, not individual removal of the proposed attention components.
| Visual backbone | [email protected] (%) ↑ | Observation |
|---|---|---|
| SwinV2-Tiny | 69.65 | Best listed configuration |
| MaxViT-Tiny | 58.63 | Another hierarchical visual backbone |
| SwinV1-Tiny | 56.60 | 13.05 points below SwinV2 |
| ConvNeXtV2-Tiny | 56.23 | A competitive convolutional alternative |
| ResNet-50 | 46.74 | Below the stronger backbones above |
| ViT-Base | 25.85 | Weak in this particular setup |
The authors interpret the results as favoring hierarchical window-based features for distinguishing similar views. Backbone pretraining and representation quality can also affect this comparison, however. Table 2 does not provide matched latency measurements, parameter budgets, or repeated-run variation, so it cannot establish an efficiency advantage or statistical significance by itself.
Key Findings¶
- Threshold choice matters. Table 3 selects \(\tau=0.3\) using validation macro F1, reaching 81.41% on validation; fixing that threshold gives test precision/recall/F1 of 74.66%/80.35%/76.52%. These must not be presented as results under the main table's 0.5 threshold.
- Averaging changes the optimum. Table 4's micro evaluation aggregates TP, FP, and FN globally and selects \(\tau=0.4\) on validation, obtaining 76.11% test F1. Macro evaluation instead computes metrics per decision before averaging across samples.
- Forced selection has its own metric. Section 4.4 reports Recall@1 of 76.31%, the fraction of samples whose highest-scoring camera matches the annotation, without a threshold.
- Adaptation does not improve every transfer direction. At 20% of each video's local training data, Table 5 raises same-video [email protected] from 74.44% to 83.65% for video_0000 and from 81.76% to 84.12% for video_0001. Fine-tuning on video_0000 and testing on video_0001 instead gives 76.47%, below its unadapted 81.76%.
[email protected] computes \(TP_i/(TP_i+FP_i)\) for each sample and averages across samples; a true positive must exceed the threshold and match the annotated camera. The text does not specify how a zero denominator is handled when no candidate exceeds the threshold, leaving a reproducibility detail that should not be silently filled in.
Highlights & Insights¶
- The contribution is directional information flow, not another feature list. Temporal and camera embeddings build on prior work; the key architectural change is letting candidates retrieve context from separately encoded history.
- Candidate interaction occurs both before and after retrieval. Relationships among current alternatives and competition after incorporating history need not be modeled by the same layer, although their individual contributions remain unmeasured.
- Evaluation reflects different recommendation interfaces. Thresholded scores suit a recommendation list, whereas Recall@1 better reflects automatic single-camera selection; both help distinguish ranking quality from threshold or calibration effects.
Limitations & Future Work¶
- Author-stated scope. The conclusion proposes audio dynamics and actor movement as future inputs for improving “when to cut.” The present system is primarily visual view selection, not a complete autonomous directing pipeline.
- Reader assessment: limited personalization evidence. Adaptation uses only two videos and local splits within each. Scene content, camera layout, or people may drive the benefit, so the experiment does not isolate content-independent editor style.
- Reader assessment: incomplete reproducibility details. Global split sizes, no-positive-prediction handling, and repeated-run variation are not fully documented. Local training-data percentages should not automatically be interpreted as percentages of entire video duration.
- Reader assessment: mechanistic claims exceed the evidence. Better recommendation metrics do not prove acquisition of a general cinematographic language, and backbone sensitivity does not uniquely identify why one feature extractor wins.
- Internal numerical inconsistencies. The introduction gives 69.95% for SwinV2 and 83.89% for 20% adaptation, while Tables 1–2 and the conclusion use 69.65%, and Table 5 lists 83.65% and 84.12% for the two videos. This note follows the directly verifiable table values.
Related Work & Insights¶
- Versus TC-Transformer (Rao et al., 2022). Both use historical and current candidate footage; this paper changes joint encoding into separate memory and cross-attention rather than redefining the TVMCE task.
- Versus Lee et al. (2025). Camera identities and temporal offsets are inherited ingredients. The matched-backbone BCE reimplementation is the more informative architecture comparison, while the published score provides historical context.
- Versus Gonzálbez-Biosca et al. (2025). Their multimodal concert-editing approach distinguishes when to cut from how to cut. The present work focuses on visual selection among synchronized candidates, so cross-task numbers should not be interpreted as comprehensive editing-quality rankings.
Rating¶
- Novelty: 3/5 — A clear task-specific information-flow change built from established modules and objectives.
- Experimental Thoroughness: 3/5 — Backbone, threshold, and adaptation analyses are useful, but component removal and variance reporting are missing.
- Writing Quality: 3/5 — The central architecture is understandable, with unresolved metric details, sampling terminology, and introductory numbers.
- Value: 4/5 — A practical architectural direction and broader evaluation perspective for camera recommendation, with external generalization still to establish.