CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion

Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/kyutai-labs/casa
Project: https://kyutai.org/casa/
Area: Vision-language model efficiency
Keywords: Cross-attention, vision-language fusion, KV cache, windowed memory, streaming video captioning

TL;DR

CASA revisits cross-attention through controlled experiments, keeping image representations outside the language backbone and replacing window-local visual KV sources for efficient fusion; vanilla CA reaches 6.8 FPS versus 1.2 FPS for token insertion in the controlled Helium1 streaming setup, but complex infographic understanding still shows a substantial performance gap.

Background & Motivation

Most contemporary vision-language models insert visual encoder outputs into the text sequence and process both modalities through the language model's self-attention and feed-forward networks. This requires no separate fusion mechanism, allows image representations to evolve across layers, and lets later text revisit earlier images. However, high-resolution images and continuous video repeatedly expand the sequence. The cost is not just visual encoding: it includes language-backbone computation at every layer and an accumulating KV cache.

Cross-attention keeps images outside the text stream and supplies visual information when text needs it, making it attractive for live video. Yet previous CA models often underperform insertion-based models on charts and documents, motivating visual FFNs, gating, and additional register tokens. A central uncertainty remains: does this gap come from the fusion mechanism, or from different backbones, training data, and implementations? Under mismatched conditions, extra modules might compensate for a training-pipeline gap while also sacrificing CA's original efficiency advantage.

CASA therefore does not introduce an elaborate new module stack. It separates CA and insertion-based self-attention into five testable architectural differences, trains VLMs from text-only LLMs, converts an existing VLM, and evaluates live video applications. Core idea: let text carry information across time while the current image remains a replaceable cross-attention source, then use a modern training pipeline to measure the capabilities retained and the costs avoided.

Method

Overall Architecture

Inputs are images or video frames together with textual instructions. The Qwen2.5-VL vision encoder produces visual tokens that remain outside the language sequence and are projected into visual keys and values at each CA layer. Text undergoes causal self-attention while parallel CA reads the current visual window; the two attention outputs are added before processing continues through the language backbone to generate an answer or live caption.

In streaming use, each new frame switches the visual window instead of permanently appending another image to the language model's KV cache. Earlier visual information survives only through text hidden states previously conditioned on those images, especially post-image delimiter tokens between consecutive frames. The diagram shows the windowed baseline; the joint text-and-vision CA variant is explained in the second design.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image or new video frame"] --> B["Vision encoder"]
    B --> C["Static Visual Sources"]
    T["Text instructions and delimiters"] --> D["Parallel Attention Fusion"]
    C -->|Current visual KV| D
    D --> E["Windowed History Transfer"]
    E --> F["Answer or live caption"]
    E -.->|Text SA in later windows| D

Key Designs

1. Static Visual Sources: avoid sending every image token through the entire language backbone

Here, static means that vision-encoder outputs receive no persistent updates across the depth of the language model. It does not mean that the vision encoder is always frozen or that every layer shares identical projected KV tensors. Vanilla CA uses layer-specific key and value projections, but does not pass image tokens through the language model's self-attention and FFNs. This avoids both occupying the language sequence and retaining the activations associated with repeated visual updates during training. It corresponds to D4 in the paper: insertion updates images through FFNs and self-attention, whereas CA does not.

The authors isolate the effect of removing visual refinement with a variant that updates image tokens through FFNs. Adding those updates improves reading tasks, but substantially increases training memory and computation without eliminating the infographic gap. CASA's efficiency is therefore not free: it preserves the detail supplied by the vision encoder but gives up repeated language-backbone processing of individual visual tokens. It also differs from first compressing an image into a few query tokens, which discards some spatial detail before language-model processing even begins.

2. Parallel Attention Fusion: separate text history from current visual conditioning

Within a Transformer block, textual self-attention and visual cross-attention receive the same text input, use their own projections, and add their outputs. The former accesses what has already been said; the latter accesses what the current image contains. CA is not placed sequentially after SA. Let the current window's image be \(y\), introduced at text position \(K\), and let the current text position be \(T>K\). The following restates only the two attention branches, omitting normalization, residuals, and FFNs rather than pretending to specify a complete block:

\[ \operatorname{SA}_{\mathrm{text}}(x_T)=\operatorname{MHA}(x_T\mid x_{1:T}), \qquad \operatorname{CA}(x_T)=\operatorname{MHA}(x_T\mid y_{1:N}). \]

Here, \(\operatorname{MHA}(q\mid s)\) denotes multi-head attention with query \(q\) and keys and values derived from sequence \(s\); \(N\) is the number of visual tokens in the image. The cached source contains text-extraction damage in its equations. These expressions are a readable restatement based on the prose and attention definitions; refer to the original paper for the exact notation. No missing full positional-encoding formula is reconstructed.

This fusion can vary along D1-D3. D1 shares query, key, value, and output projections between CA and SA in the same block, avoiding additional parameters and enabling merged projection computations. D2 introduces \(\operatorname{CA}_{t+v}\), whose CA keys and values contain the current image and preceding text in the same window. D3 replaces some SA layers with CA instead of always adding parallel attention. D2 also concerns the joint temporal positional information that image and text receive under insertion. Its essential local visibility pattern is:

\[ \operatorname{CA}_{t+v}(x_T)=\operatorname{MHA}\!\left(x_T\mid [y_{1:N},x_{K+1:T}]\right),\qquad T>K. \]

Joint CA does not recover global insertion-based attention because it still operates within the current window. Nor can vanilla CA replace every SA layer: doing so would remove all text-to-text communication. The default replacement experiment replaces every second SA layer, with an every-fourth-layer configuration also evaluated. These are ways of adjusting parameter count, attention depth, and visibility, not three mandatory enhancement modules that must be enabled together. The cached symbol for the parameter-sharing variant is missing, so this note uses its descriptive name instead of guessing the original notation.

3. Windowed History Transfer: replace current visual KV and carry earlier images through text hidden states

Adjacent image positions define windows, and text in a window directly accesses only that window's image. Causal text SA still spans earlier text, so hidden states produced after seeing an image can transmit compressed visual information to later positions. This is the central D5 trade-off: without keeping all historical image KV, the model cannot retrieve arbitrary details from old frames. No dedicated memory tokens are added. For consecutive video frames, the post-image delimiters already present in Qwen's chat template act as gist tokens that supply historical information through later SA.

This memory need not be a readable textual summary; it consists of visually conditioned delimiter hidden states. Video-QA results support the usefulness of this route, but do not establish lossless storage or preservation of arbitrary details. If the model has seen \(k\) frames, each containing \(N\) visual tokens and generating \(T\) text tokens, the KV-entry accounting in Table 1 can be summarized as:

\[ \#\mathrm{KV}_{\mathrm{CA}}\sim kT+N, \qquad \#\mathrm{KV}_{\mathrm{insertion}}\sim kT+kN. \]

This compares storage terms as the number of frames increases, not complete memory usage in bytes. It omits layer count, hidden dimensions, precision, and fixed model overhead. The \(\operatorname{CA}_{t+v}\) variant additionally requires text KV for the current window. The important change is that historical visual storage no longer accumulates with \(k\), while text history still grows. Near-constant memory is consequently an observation over the measured horizon, not a claim of strictly constant total memory for an infinite stream.

A Worked Example

Consider the paper's 2 FPS live-captioning setting, in which a new frame arrives every 0.5 seconds. The first encoded frame becomes the current visual CA source while the model processes text and post-image delimiters. When the second frame arrives, it replaces the current visual source; the two frames are not both appended to a historical visual cache inside text SA.

Text following the second frame sees it directly through CA, while SA accesses text hidden states left after the first frame. This may retain clues about action continuity, but a later request to read an unremembered number in the corner of the first frame has no old image tokens to query. This illustrates the mechanism, not a numerical-reading example reported by the paper. Offline video evaluation also includes a different setting that places all frames in one window, with different computation and memory requirements.

Loss & Training

The available main text introduces no separate new loss function; the contribution is architectural analysis and controlled training comparisons. Image training uses FineVision and a subset of LLaVA-OneVision-1.5, with LLaVA-Video-178K added for subsequent video training. CA layers are initialized from the corresponding backbone's SA layers. Training from Helium1-2B jointly updates the language backbone and CA, whereas conversion from Qwen2.5-VL-3B uses a smaller learning rate for the language backbone while learning CA.

Both routes use the Qwen2.5-VL vision encoder, fine-tuning its last four layers during image training and freezing it during subsequent video training. Multimodal sequence packing uses FlashAttention-2 block-wise attention. Image insertion positions define CA block boundaries, preventing packed text from accessing another window's image and aligning training with windowed inference.

Training uses 64 H100 GPUs, batch size 64, and two gradient-accumulation steps. Image training allows up to 2,048 text tokens and 20,480 visual tokens per GPU. Because insertion requires more memory, its sequence limits are halved and batch size doubled to keep the number of processed tokens comparable. Images have a maximum pixel budget of \(896^2\) and at most 1,024 tokens per image. Video uses a \(504^2\) budget, 2 FPS, clips of at most three minutes, and up to 46,080 visual tokens per sequence.

The Helium1 route runs for 40k steps, Qwen image conversion for 25k steps, and subsequent video training for 15k steps. Live captioning separately starts from the image-trained \(\operatorname{CA}_{t+v}\) Qwen model and fine-tunes on Live-WhisperX-526K; it should not be conflated with the video-QA training sequence. The cache ends after the references and does not include Appendix B.1, so exact learning rates, optimizer settings, and unspecified loss-masking details are not supplied here.

Key Experimental Results

Main Results

The following selection comes from Tables 2 and 3. Higher is better throughout; OCRBench retains its original score scale rather than being converted into a percentage. The Helium1 rows compare fusion mechanisms trained under the same conditions with 0.13T tokens each. The original Qwen row uses the authors' reproduced results. These groups must not be merged into a ranking under an equal pretraining budget.

Experimental setting	Model	ChartQA	DocVQA	InfoVQA	OCRBench	TextVQA
Matched Helium1 training	InsertHe-2B, 2.7B total parameters	79.2	88.3	58.9	746	75.1
Matched Helium1 training	Vanilla CA, 3B total parameters	77.5	86.4	55.1	740	75.4
Qwen architecture conversion	Qwen2.5-VL-3B, reproduced	83.1	92.4	75.1	796	79.6
Qwen architecture conversion	Vanilla CA, 25k steps	81.8	87.9	58.2	796	76.5

With Helium1, CA is 1.7 points lower on ChartQA and 1.9 lower on DocVQA, but 0.3 higher on TextVQA. CA is therefore not fundamentally inferior on every visual task. However, Qwen conversion reduces InfoVQA from 75.1 to 58.2, a 16.9-point gap that should not be hidden behind a general statement about retaining most capabilities.

Ablation Study

Table 6 isolates visual-token FFN updates. This experiment freezes the vision encoder and uses roughly one third of the training-token budget of the main Qwen experiment. The original model is a reference; the three trained variants use matched reduced conditions. Its CA row must not be mixed with the main-results row above.

Model	AI2D	ChartQA	DocVQA	InfoVQA
Original Qwen2.5-VL	79.6	83.1	92.4	75.1
Token insertion, fine-tuned on the same data	78.9	83.8	91.3	68.8
CA with visual FFN updates	76.2	80.6	86.9	59.2
Vanilla CA	73.7	78.3	83.1	52.1

FFN updates raise DocVQA from 83.1 to 86.9 and InfoVQA from 52.1 to 59.2, gains of 3.8 and 7.1 points, respectively. InfoVQA nevertheless remains 9.6 points below insertion fine-tuned on the same data. Conversely, merely fine-tuning the original Qwen on the study's data lowers InfoVQA from 75.1 to 68.8, a 6.3-point difference. Data distribution is therefore another source of the gap, rather than everything being attributable to architecture.

Efficiency and Video Analysis

Table 1 measures Helium1 models on one GPU. Vanilla CA trains at 1,817 text tokens/s versus 1,501 for insertion, while controlled streaming inference reaches 6.8 versus 1.2 FPS, approximately a 5.7-fold difference. FPS here means generating five text tokens per frame while incorporating new frames; it is not an end-to-end guarantee at arbitrary resolutions or on arbitrary devices. Parameter-sharing CA reaches 7.7 FPS. The cached table header does not clearly preserve a unit for its memory measurements, so none is invented here.

The video comparison below comes from Table 4. VideoMME uses the no-subtitles setting. Reproduced Qwen results allow up to 360 frames, while both CA variants allow up to 180, so their input budgets differ.

Model and visual window	VideoMME, no subtitles	NExT-QA	PerceptionTest	MVBench	MLVU
Qwen2.5-VL-3B, reproduced	58.8	78.9	67.0	66.2	65.2
CA t+v, all frames in one window	59.0	79.8	63.8	60.6	67.9
CA t+v, one frame per window	56.9	78.8	62.6	59.5	66.2

The frame-wise setting loses 2.1 points on VideoMME relative to the single-window setting, illustrating the cost of relying on text to retain earlier visual information. Live captioning is evaluated on LiveSports3K with GPT-4o as judge. In Figure 4(a), the 3B CA model achieves a 39.4% win rate at 17k training steps and 39.0% at 20k, compared with 43.2% for LiveCC-7B-Base. These are caption judgments under a particular protocol, not QA accuracy or evidence that the smaller model universally surpasses 7B systems.

Highlights & Insights

• Separating fusion into D1-D5 explains costs more precisely than comparing two aggregate model scores. Parameter sharing and layer frequency adjust relatively lightweight computation, whereas visual refinement and full historical visual KV drive important training and streaming costs, respectively.
• Existing delimiter tokens can support compressed visual history without becoming a new architectural module. This suggests that efficient memory need not require an extra memory network, but the details lost under compression still need explicit measurement.
• Matched training from an LLM, conversion from a VLM, and insertion fine-tuning on the same data jointly constrain the conclusion. They support reconsidering CA, not claiming that every performance gap in prior work has disappeared.

Limitations & Future Work

• Fine-grained infographics remain a clear weakness. The authors explicitly retain a role for insertion in document understanding, infographics, and long-range retrieval; adding visual FFNs only partly closes the gap while surrendering some efficiency.
• Historical visuals are compressed into text hidden states rather than remaining losslessly accessible. A bounded historical window or on-demand visual retrieval is a possible extension, but would increase KV storage and per-step access costs again.
• Near-constant memory is not strict constant-space operation because text history and delimiters still accumulate. Deployment limits require a strategy for managing textual context during very long runs.
• Main image experiments, reduced-budget ablations, and video frame limits differ, so cross-model scores are not all equal-compute comparisons. Short LiveSports3K clips and an LLM judge also do not comprehensively test factuality or long-term memory in extended live captioning.
• The available cache omits Appendices A-D and contains damaged equations and some variant symbols. Missing training details, timing specifications, and additional qualitative examples are limits of the material read here, not evidence that the original paper failed to provide them.

Related Work & Insights

• Compared with Flamingo / OpenFlamingo: CASA continues the established route of injecting visual conditions through CA, but focuses on controlled reassessment with modern backbones and public training data rather than claiming to invent CA. Its parallel attention layout should also be distinguished from earlier gated designs.
• Compared with StreamChat: both support continuously updated visual key-value sources. CASA directly tests the benefit and cost of visual FFNs, showing that visual refinement is not a cost-free default enhancement.
• Compared with Q-Former compression: compression reduces the number of tokens inserted per frame but does not stop them accumulating across frames. Table 5 reports high-resolution task-group scores of 74.2 for insertion with 1,156 tokens, 62.7 with 128 queries, and 56.0 with 32 queries, indicating the detail loss from early compression. Compression and CA can be combined rather than treated as mutually exclusive choices.
• Compared with LiveCC: CASA uses its video-transcription data and caption evaluation to test efficiency, but changes fusion and caching rather than introducing a new caption-supervision signal. Follow-up evaluation should jointly consider speed, factual consistency, and recall of historical events.

Rating

• Novelty: 4/5. CA has a long history; the main contribution is the five-part decomposition, controlled evaluation, and conversion of a pretrained VLM.
• Experimental Thoroughness: 4/5. Image, video, architectural ablations, and live applications are covered, with remaining limits in long-term memory, judge reliability, and cross-budget fairness.
• Writing Quality: 4/5. The main argument and cost analysis are clear, though local CA, joint CA, and different window settings require careful distinction; cache extraction damage is not treated as an authorial writing flaw.
• Value: 4/5. Highly relevant to memory- and latency-constrained streaming VLMs, with more cautious trade-offs required for fine-grained document understanding.