DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/laychou666/DASH
Area: VLM Efficiency / Omnimodal Large Language Models
Keywords: Token Compression, Dynamic Semantic Chunking, Audio Boundaries, Tri-Signal Fusion, Audio-Visual Understanding

TL;DR

DASH treats discontinuities between adjacent audio representations as semantic chunking cues and combines boundary, representational uniqueness, and attention scores to retain tokens, allowing Qwen2.5-Omni-7B to reach 44.9% WorldSense accuracy at 25% target retention and 14.9T FLOPs, compared with OmniZip's 44.7% at 35% retention and 21.4T FLOPs.

Background & Motivation

Omnimodal large language models feed audio, video, and text into a shared language model, with a single video potentially producing tens of thousands of tokens. Pruning can reduce prefill attention computation and subsequent KV-cache requirements without retraining the model. However, compression is not simply a matter of choosing the tokens with the highest attention: a speaker change, the end of an explanation, or the beginning of an action may occupy very few tokens while determining how successive events relate to one another.

OmniZip already uses audio to guide video compression and exploits both spatial and temporal redundancy through interleaved compression. DASH argues that its fixed grouping can still split a coherent semantic unit. The paper describes static groups of 50 audio tokens and four video frames. Such granularity does not follow content changes, while attention concentrates on only a few positions, making low-attention tokens that connect the narrative particularly vulnerable under aggressive pruning. Both the grouping boundaries and the retention criterion therefore need attention, not just another adjustment to the compression ratio.

Pauses, topic transitions, and speaker changes often manifest as lower similarity between adjacent audio embeddings, providing an inexpensive structural cue. Yet changes in sound need not coincide with visual transitions, so audio boundaries should not be interpreted as ground-truth shot boundaries. Core idea: use audio to define variable-length semantic units, organize video with a soft temporal prior, and combine structural importance, content representation, and model attention when deciding which tokens to retain.

Method

Overall Architecture

DASH sits between the modality encoders and the LLM and introduces no learnable parameters. It takes encoded audio and video sequences and produces compressed audio-visual tokens for LLM inference; text is not the compression target of this work. It first detects dynamic semantic boundaries in audio, projects them onto the video timeline, and removes boundaries that would create excessively short segments. Tri-signal audio scoring then produces a retention mask, whose segment-level retention fractions control video compression strength.

There are two distinct forms of audio guidance: boundaries determine where video is grouped, while retention fractions determine how aggressively each group is compressed. The final video tokens are still chosen through spatial density and cross-frame similarity in visual features, rather than by copying the audio mask. Each segment is compressed independently from the original encoder embeddings; previously compressed outputs do not recursively determine later boundaries or features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Encoded audio and video"] --> Chunk["Dynamic Semantic Chunking"]
    Chunk --> Map["Audio-Guided Video Segmentation"]
    Map --> Score["Tri-Signal Fusion Scoring"]
    Score --> Video["Boundary-Aware Video Compression"]
    Input -->|Original visual features| Video
    Score -->|Retained audio tokens| Output["Compressed audio-visual tokens<br/>to the LLM"]
    Video --> Output

Key Designs

1. Dynamic Semantic Chunking: find representation changes instead of cutting at fixed intervals

For each audio token \(a_t\), DASH computes its cosine similarity with the preceding token. A new boundary is accepted only when similarity falls below \(\tau_a=0.4\) and at least \(C_{\min}=30\) tokens have elapsed since the previous accepted boundary. The following presents the conditions from the paper's Equations (1) and (3) in readable notation; \(t_{\mathrm{last}}\) is the most recently accepted boundary position:

\[ \mathrm{sim}_t=\frac{\langle a_{t-1},a_t\rangle}{\|a_{t-1}\|\,\|a_t\|}, \qquad m_t^{\mathrm{boundary}}= \mathbf{1}[\mathrm{sim}_t<0.4]\, \mathbf{1}[t-t_{\mathrm{last}}\geq 30]. \]

The first token is always a boundary, with boundary probability set to 1. In the paper's setting, 30 tokens correspond to approximately one second of audio. This lower bound prevents minor fluctuations from creating chunks of only two or three tokens. It also limits temporal resolution: a very short event can cause a clear representation change yet remain within the current chunk because the spacing requirement is not met. Dynamic chunking seeks coherent units suitable for compression, not precise speech or event segmentation labels.

Alongside the binary boundary mask, DASH retains a continuous boundary probability expressing how dissimilar adjacent representations are. This quantity is used both in token scoring and to decide which of two nearby boundary candidates is more important. Equation (2) has missing symbols in the cached extraction, so this note does not reconstruct its exact clipping expression. The recoverable facts are that the probability lies in \([0,1]\) and increases as similarity decreases. These facts explain the downstream mechanism but do not replace checking the original equation for reproduction.

2. Audio-Guided Video Segmentation: project a structural prior, not a shot-boundary label

Let \(N_a,N_v\) be the total audio and video token counts, and let \(b_i^a\) denote an audio boundary. DASH projects boundaries according to relative temporal position in the two sequences. The central mapping from Equation (4) is:

\[ b_i^v=\left\lfloor b_i^a\frac{N_v}{N_a}\right\rfloor. \]

Projected boundaries are deduplicated, sorted, clamped to \([0,N_v]\), and augmented with endpoints 0 and \(N_v\). With \(K\) tokens per video frame, the subsequent interleaved spatial-temporal compressor needs at least two frames, or \(2K\) tokens per segment. Direct projection can create shorter segments. DASH therefore sorts internal candidates by descending audio boundary probability and inserts them into a set initially containing only the endpoints. A candidate is accepted only if both resulting neighboring segments contain at least \(2K\) tokens.

This greedy procedure reconciles two requirements: semantic cues favor segmentation at content changes, whereas the compressor needs enough context to compare redundancy. Prioritizing strong boundaries is more justified than dropping an arbitrary boundary, but remains a heuristic without a global optimality guarantee. Temporal co-registration is a prerequisite for the mapping. Continuous dialogue over rapidly changing visuals may still lack useful audio boundaries, so visual features must provide corrective evidence when the final mask is constructed.

3. Tri-Signal Fusion Scoring: do not let sparse attention monopolize the token budget

Audio retention combines three signals. The boundary term normalizes boundary probabilities by their maximum to emphasize transitions. The attention term comes from the audio encoder and is similarly normalized, reflecting where the model already focuses. A third term, called uniqueness by the authors, adds evidence from the representation distribution. Their default weights are 0.4, 0.3, and 0.3. The fused score drives Top-K selection rather than simply taking the union of three independently selected sets.

\[ s_t=0.4\,s_t^{\mathrm{bnd}}+0.3\,s_t^{\mathrm{uniq}}+0.3\,s_t^{\mathrm{attn}}, \qquad N_{\mathrm{keep}}=\left\lfloor(1-\rho_a)N_a\right\rfloor. \]

Here \(\rho_a\) is the audio compression fraction, so \(1-\rho_a\) is the retention fraction. The paper's narrative emphasizes comparisons within segments, but the Top-K budget following Equation (10) uses the total audio token count \(N_a\). The text does not fully specify how integer budgets are assigned to individual chunks, so it does not justify inventing a rule that retains exactly 25% in every segment. Indeed, downstream video budgeting relies on differences in actual audio retention across segments. When boundary information is unavailable, such as for very short sequences, the method falls back to attention-only selection.

For uniqueness estimation, DASH first computes per-channel variance and retains the half of channels with the lowest variance. It then applies \(\ell_2\) normalization and computes a global center. The multi-scale Gaussian bandwidth set is \(\{0.125,0.25,0.5,1.0,2.0\}\), intended to avoid excessive sensitivity to a single scale or hard distance threshold. Low-variance selection reflects the authors' assumption that stable semantic dimensions are more useful than transient artifacts; it is not channel importance learned through additional supervision.

A reproduction concern must remain explicit: cached Equation (7) describes Gaussian similarity to the global center, while Equation (8) has an incomplete correspondence between variables. If center similarity is merely normalized directly, high scores indicate proximity to the center, which does not naturally imply greater uniqueness. This note therefore does not invent an inversion, reciprocal, or density-peak factor. Figure 1 also labels a cross-modal contextual merging stage, but the method text does not define it sufficiently to reconstruct a separate retrieval or merging algorithm. The clearly supported core remains tri-signal scoring and audio-guided video compression.

4. Boundary-Aware Video Compression: allocate budgets with audio, select tokens with vision

For each video segment, DASH measures the fraction of tokens retained by tri-signal scoring in the corresponding audio interval and treats it as a proxy for information density. It adjusts the base video compression ratio by adding \(\lambda_r=0.1\) times 0.5 minus the segment's audio retention fraction. Higher audio retention therefore produces milder video compression, while lower audio retention produces stronger compression. This is a limited budget adjustment, not a claim that all audible content must remain uncompressed or that silent footage contains no visual information.

Frames near projected boundaries receive a mild retention increase to protect transitional context. Equation (12) has a missing factor in the cached text; this note preserves only the direction established by the prose and does not invent a protection coefficient. The method then applies OmniZip's ISTC: even frames undergo spatial pruning with DPC-KNN to reduce locally dense, redundant tokens, while odd frames undergo temporal pruning based on similarity to the previous frame. The main additions thus concern where to segment and how much budget to allocate, rather than a newly invented visual redundancy compressor.

A Worked Example

Consider a conceptual audio-video clip in which a person explains an operation continuously, pauses, and proceeds to the next step. This is an illustration, not a measured example from the paper. Similar adjacent audio representations can produce one long chunk during the explanation. A new boundary is detected at the pause only if cosine similarity falls below 0.4 and at least 30 tokens have elapsed since the last boundary. The position is projected using the ratio of total video and audio token counts, and the split survives only if both sides satisfy the two-frame minimum.

Tri-signal scoring can then retain a low-attention transition token because its boundary score is high. If an explanation-dense segment retains more audio tokens, the corresponding video segment receives milder compression. Spatial or cross-frame similarity removes redundant static background content, while visual context near the transition receives modest protection. This example explains how the mechanisms interact without assuming that every pause is a shot change or fabricating final token counts.

Loss & Training

DASH is training-free, with no additional loss, training data, or learnable parameters. Experiments use Qwen2.5-Omni-7B and 3B on NVIDIA H20 96GB GPUs, with FlashAttention, deterministic decoding, and rule-based multiple-choice parsing. VideoMME inputs are capped at 768 frames and the other datasets at 128. The implementation describes 50 audio tokens and 288 video tokens per time window; this window-level count should not be substituted directly for the per-frame quantity \(K\) above.

The 25% and 35% figures are target retention ratios. Actual retention can differ with modality proportions, segment constraints, and content complexity. The paper reports less than 40 ms of additional boundary-detection and tri-signal-scoring overhead, but does not provide a complete input-length distribution or latency variance. That number should not be treated as a fixed cost for inputs of arbitrary length.

Key Experimental Results

Main Results

The following selects Full Tokens, OmniZip, and DASH from the paper's Tables 1 and 2. AVUT, VideoMME, and WorldSense entries are accuracy percentages. AVUT Avg. is preserved as reported rather than recomputed from the six subtasks. VideoMME corresponds to the original table's "wo" column. AVUT/VideoMME FLOPs ratios and absolute WorldSense FLOPs are kept separate to avoid conflating measurement scopes; the latter counts only multimodal audio and video tokens.

Model	Method	Target retention	AVUT/VideoMME FLOPs ratio	AVUT Avg.	VideoMME wo	WorldSense Avg.	WorldSense FLOPs (T)
7B	Full Tokens	100%	100%	64.5	66.0	46.8	73.2
7B	OmniZip	35%	29%	60.6	66.0	44.7	21.4
7B	DASH	35%	29%	61.5	66.7	Not reported	Not reported
7B	DASH	25%	20%	60.9	66.0	44.9	14.9
3B	Full Tokens	100%	100%	62.2	62.6	46.4	37.4
3B	OmniZip	35%	26%	58.7	61.9	44.1	9.9
3B	DASH	35%	26%	59.9	62.6	Not reported	Not reported
3B	DASH	25%	18%	58.8	61.7	44.6	6.7

"Not reported" means that the original Table 2 does not contain that configuration; no values are estimated from curves. Relative to OmniZip at 35%, DASH at 25% improves 7B WorldSense accuracy by 0.2 percentage points while reducing FLOPs by approximately 30.4%. For 3B, accuracy improves by 0.5 points and FLOPs fall by approximately 32.3%. However, DASH remains 1.9 and 1.8 points below the respective Full Tokens baselines on WorldSense, so this is not universally lossless compression. At 25% retention, 3B VideoMME accuracy is also 0.2 points below OmniZip.

The efficiency results below come from the original Table 3. Speedup factors are relative to Full Tokens at the same model size, not fractions of remaining execution time. DASH uses 25% target retention and OmniZip uses 35%.

Model	Method	Memory (original unit: G)	Prefill speedup	End-to-end speedup	WorldSense accuracy
7B	Full Tokens	35	1.0×	1.0×	46.8
7B	OmniZip	25	3.4×	1.4×	44.7
7B	DASH	26	3.5×	1.7×	44.9
3B	Full Tokens	25	1.0×	1.0×	46.4
3B	OmniZip	16	3.3×	1.3×	44.1
3B	DASH	16	3.8×	1.4×	44.6

The maximum 3.8x prefill speedup belongs to 3B, whereas the maximum 1.7x end-to-end speedup belongs to 7B; they must not be combined into a single configuration. DASH also uses 1G more memory than OmniZip on 7B, showing that fewer tokens do not guarantee monotonic improvements in every resource metric.

Ablation Study

This table combines the original Tables 4 and 5, all evaluated on WorldSense with Qwen2.5-Omni-3B. TSF denotes tri-signal fusion, DSC dynamic semantic chunking, and ADVS audio-guided video segmentation. Every configuration targets 25% retention except the OmniZip reference at 35%.

Analysis	Configuration	Target retention	Accuracy (%)
Reference	OmniZip	35%	44.1
Components	Static + TSF	25%	44.4
Components	DSC + ADVS, attention-only selection	25%	44.4
Components	Full DASH	25%	44.6
Boundary algorithm	Random	25%	43.8
Boundary algorithm	Dot Product	25%	43.6
Boundary algorithm	Change Rate	25%	44.0
Boundary algorithm	Cosine	25%	44.6

Full DASH improves on each partial configuration by 0.2 percentage points, supporting complementarity between chunking and fusion. However, gains over the 44.1 reference are not same-retention isolated component gains because that reference uses 35%. Table 4 does not report static grouping plus attention-only selection at 25% retention; this missing baseline is not fabricated here.

Key Findings

• The boundary metric matters: cosine similarity exceeds dot product by 1.0 percentage point and random boundaries by 0.8 points. Scale normalization is a plausible explanation, but the experiments do not directly measure boundary detection accuracy.
• Figure 4 reports relative stability for boundary weight \(w_b\) between 0.3 and 0.5, with the best result at 0.4 and degradation above 0.5. No complete numerical table accompanies the plot, so per-weight accuracies are not interpolated here.
• In Figure 2, fusion and attention-only selection retain 866 tokens in common and differ on another 631; \(631/(631+866)\) is approximately 42.2%. This describes the selection difference in that visualization, not an average improvement over all samples or an accuracy gain.

Highlights & Insights

• Compression must account for temporal structure as well as repeated content. Separating boundary protection from token salience can preserve positions that explain relationships between events, rather than retaining only conspicuous entities.
• One modality can supply both a grouping prior and a budget prior without deciding the other modality's final token choices. This division leaves the visual compressor room to correct misleading or misaligned audio cues.
• The most useful evidence is the accuracy-compute trade-off at low retention, not an absolute claim of outperforming the uncompressed model. For deployment, that is more informative than presenting only a higher theoretical compression ratio.

Limitations & Future Work

• The authors explicitly avoid assuming exact audio-visual boundary coincidence, and actual retention can deviate from the target. The text does not provide targeted stress tests for silent actions, background music, voiceovers, or audio-video misalignment; these are evaluation gaps, not demonstrated failure cases.
• Experiments cover only the 3B and 7B variants of Qwen2.5-Omni, not different model or encoder families. Multiple-choice QA results do not establish benefits for free-form generation, fine-grained temporal localization, or streaming inference.
• Some gains are only 0.2 percentage points, without repeated-run variance or confidence intervals. The authors reproduce OmniZip and DASH, while Random, FastV, and DyCoke results are taken from OmniZip; not all comparisons are rerun in one common experimental pipeline.
• Global-center uniqueness, per-segment Top-K budgeting, and the contextual merging stage shown in the figure still need clarification, while damaged cached equations further limit exact reproduction. Useful next steps are matched-budget single-signal ablations, audio-video misalignment tests, and precise actual-retention statistics, rather than merely expanding the main result table.

Related Work & Insights

• Compared with OmniZip: DASH retains audio-guided compression and ISTC, with its main additions being dynamic chunking, soft temporal projection, and tri-signal scoring. It is a structured enhancement of an existing pipeline, not a wholly new audio-video compressor.
• Compared with H-Net: H-Net learns dynamic chunking routes. DASH borrows the adjacent-representation cosine-change cue, detects boundaries without training on existing audio features, and projects those boundaries onto video.
• Compared with FastV and DyCoke: FastV primarily uses attention within LLM layers, while DyCoke emphasizes temporal video compression. Their omnimodal adaptations here help assess the value of cross-modal structure, but do not establish an overall ranking on their original tasks.
• Transfer opportunities: Long-video retrieval and streaming multimodal assistants could benefit from content-driven grouping and boundary retention. Online boundary confirmation, global-center computation, and audio-video delay would need to be addressed before presenting the current offline strategy as a streaming solution.

Rating

• Novelty: 4/5. Semantic chunking, cross-modal soft priors, and multi-signal scoring form a practical combination, although the underlying compressor and routing cue build on prior work.
• Experimental Thoroughness: 3/5. Three benchmarks, two model sizes, and component ablations are useful, but cross-family validation, complete matched-budget baselines, and statistical uncertainty are missing.
• Writing Quality: 3/5. The main argument is understandable, but some diagrammed modules and scoring definitions remain insufficiently specified; local equation-extraction damage separately limits verification precision.
• Value: 4/5. Training-free integration and measured latency gains are relevant to audio-video deployment, but compression is not lossless on every task and does not beat OmniZip on every resource metric.