Skip to content

AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

Conference: ECCV 2026
arXiv: 2606.30811
Project: https://hkust-longgroup.github.io/AVTok/
Code: None (The paper promises subsequent open-source release)
Area: Video Generation / Audio-Video Generation
Keywords: Audio-Video Generation, Unified Tokenization, 1D Discrete Latent Space, Dual-Stream Transformer, Autoregressive Generation

TL;DR

AVTok proposes the first unified audio-video tokenizer. It employs a two-stream Transformer architecture (shared encoder-decoder + modality-specific learnable queries) to jointly encode audio-video pairs into a single 1D discrete latent space. Coupled with the VFAL progressive training strategy and representation alignment learning, it achieves state-of-the-art performance in both reconstruction quality and downstream autoregressive generation tasks (A2V / V2A / joint generation), while using significantly fewer parameters and computation than dominant dual-branch approaches.

Background & Motivation

Audio-video generation (A2V, V2A, joint audio-video generation) has gained significant attention in recent years. However, dominant methods typically adopt a dual-branch architecture: configuring independent tokenizers and generation modules for audio and video separately, and inserting additional cross-modal interaction modules in between. This design suffers from two fundamental issues: first, there exists a representation gap between the embedding spaces learned by independently trained tokenizers, leading to semantic misalignment between the generated audio and video; second, the dual-branch architecture incurs massive computational overhead (e.g., Ovi's tokenizer + generator total nearly 1B + 17B parameters), making training and deployment costly and difficult to scale.

The core insight of this paper is: if a unified tokenizer can jointly encode audio-video pairs into the same latent space, it can fundamentally eliminate the representation gap and bypass the heavy dual-branch generation architecture. However, the challenge lies in the fact that raw video is 3D spatiotemporal data while raw audio is 1D waveform—with completely different data structures, how can they be unified? Fortunately, recent 1D video tokenization works (LARP, AdapTok) have demonstrated that learnable queries can compress video into 1D discrete tokens, which formally aligns with the natural 1D structure of audio. Following this path, AVTok tokenizes both video and audio into 1D discrete latent representations, utilizing a two-stream shared architecture + unified codebook to achieve joint encoding, becoming the pioneering work in this direction.

Core Idea: Use a two-stream Transformer (shared encoder-decoder + modality-specific queries and normalization layers) to jointly encode audio-video pairs into a 1D discrete latent space of a unified codebook, and solve the modality information imbalance with a "video-first, audio-later" progressive training strategy.

Method

Overall Architecture

The goal of AVTok is to take an audio-video pair (video frame sequence + audio) as input and output a compact set of 1D discrete tokens, which can both reconstruct the original audio-video with high quality and be directly fed into autoregressive generation models for downstream tasks. The entire pipeline consists of five stages: (1) Patchify video frames and audio Mel spectrograms separately into embedding sequences; (2) Pass through a two-stream (one forward pass each for video and audio streams) shared encoder, extracting global information via modality-specific learnable holistic queries; (3) Map the latent vectors corresponding to queries to a unified codebook using an SVQ quantizer to obtain discrete tokens; (4) Symmetrically reconstruct video frames and Mel spectrograms through a two-stream shared decoder; (5) Revert the Mel spectrograms back to waveforms using a HiFi-GAN vocoder. During training, CAV-MAE+ is introduced for representation alignment, alongside a lightweight AR prior model (GPT-2) for next-token prediction to shape an AR-friendly latent space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Audio-Video Pair<br/>Video Frames + Mel Spectrogram"] --> B["Patchify<br/>Video Patch Embeddings + Audio Patch Embeddings"]
    B --> C["Two-Stream Encoder (Shared Parameters)<br/>Video Stream LNᵛ+Queries / Audio Stream LNᵃ+Queries"]
    C --> D["SVQ Quantizer<br/>Unified Codebook → 1D Discrete Tokens"]
    D --> E["Two-Stream Decoder (Shared Parameters)<br/>Video Stream LNᵛ+Queries / Audio Stream LNᵃ+Queries"]
    E --> F["Reconstruct Video Frames + Mel Spectrogram"]
    F --> G["HiFi-GAN Vocoder → Waveform"]
    C -. -->|Rep. Alignment during Training| H["CAV-MAE+ ℒ_rep"]
    D -. -->|AR Prior during Training| I["GPT-2 NTP ℒ_prior"]

Key Designs

1. Two-Stream Transformer: Modality-Specific Queries + Shared Encoder-Decoder to Resolve Representation Gap and Modality Conflict

Directly concatenating audio and video patches and feeding them into a single-stream Transformer (the vanilla version) can theoretically achieve cross-modal fusion. However, experiments show that this naive design performs poorly—video information density is far higher than that of audio, and during training, the video stream suppresses the learning of the audio stream, deteriorating the reconstruction quality of one or both modalities. AVTok's solution draws on the two-stream philosophy of CAV-MAE: all attention and MLP layer parameters are shared between the encoder and decoder, but independent LayerNorm parameters (\(LN_1^v, LN_2^v\) and \(LN_1^a, LN_2^a\)) as well as independent learnable queries (video holistic queries \(\mathbf{Q}_L^v \in \mathbb{R}^{1024 \times d}\), audio holistic queries \(\mathbf{Q}_L^a \in \mathbb{R}^{128 \times d}\), and their respective patch queries) are allocated for the video and audio streams.

The video stream and audio stream undergo two separate forward passes through the same encoder: each pass only feeds the patch embeddings and corresponding queries of a single modality, yet the shared attention layers implicitly fuse cross-modal information (since the parameters are identical, gradients from both forward passes accumulate). The quantizer \(\mathcal{Q}\) is also shared, utilizing a single SVQ codebook to ensure that both video and audio tokens reside in the same discrete space. During decoding, two separate forward passes are likewise executed, each reconstructing its corresponding modality using its own patch queries. This "shared parameters + isolated LN/queries" design is highly ingenious: the shared parameters provide a channel for implicit cross-modal interaction, while the isolated LNs and queries preserve the uniqueness of each modality, preventing the modality suppression issue inherent in single-stream models. Furthermore, the two-stream design allows for encoding the conditional modality individually as needed in downstream tasks (e.g., executing only the video stream to obtain conditioning tokens for V2A), which is impossible for the vanilla single-stream version.

2. VFAL Progressive Training: Video-First-Audio-Later Training to Solve Information Density Imbalance

Even with the two-stream architecture, joint training of audio and video from scratch remains sub-optimal: visual information is rich and dense, while audio information is relatively sparse, causing gradients to be dominated by the video during joint training. AVTok proposes the VFAL (Video-First-Audio-Later) three-stage progressive training strategy:

  • Stage 1 (Video Reconstruction, 75 epochs): Trains only the video stream—including the encoder and decoder (with video-specific LNs), video learnable queries, the quantizer, and the AR prior model. The audio stream is completely discarded. The goal is to first establish a robust video token latent space. At this stage, \(\lambda_1=1.0, \lambda_2=0.0, \lambda_3=0.0, \lambda_4=0.06\), and \(\mathcal{L}_{prior}\) is calculated solely using video tokens \(\mathbf{x}^v\).
  • Stage 2 (Audio Reconstruction, 35 epochs): Freezes the encoder, decoder, and all video stream parameters (shared parameters + video LNs + video queries), and trains only the audio-specific LNs (in both encoder and decoder: \(LN_1^a, LN_2^a\)), audio learnable queries (\(\mathbf{Q}_L^a, \mathbf{Q}_P^a\)), the AR prior model, and the MLP projector \(h_\phi\) for representation alignment. At this stage, \(\lambda_1=0.1, \lambda_2=1.0, \lambda_3=0.5, \lambda_4=0.06\). The intuition is: since the Mel spectrogram can be treated as a grayscale image, the frozen shared parameters already possess the "image" processing capabilities; the audio stream only needs to learn its own "reading" mechanism.
  • Stage 3 (Fine-tuning, 10 epochs): Fine-tunes only the decoder (including LNs of both modalities) to further unify the audio-video reconstruction quality. At this stage, \(\lambda_1=1.0, \lambda_2=0.01, \lambda_3=0.5, \lambda_4=0.06\).

This multi-stage strategy forces the model to learn in a "difficult-first, easy-later" sequence. Each stage has clear objectives, preventing gradient competition during multi-modal joint training. Ablation studies demonstrate that removing VFAL causes the reconstruction rFVD to degrade from 12.80 to 13.19 and rFAD from 5.93 to 9.38, with downstream generation metrics also dropping significantly.

3. Representation Alignment Learning: Boosting Cross-Modal Semantic Alignment with Pre-trained AV Foundation Models

Since cross-modal interaction in the two-stream architecture relies solely on shared parameters implicitly, the authors found that this implicit fusion is insufficient for the model to fully learn the semantic correspondence between audio and video. To address this, AVTok introduces a pre-trained audio-visual foundation model CAV-MAE+ (\(\mathcal{M}_F\)) as an "alignment teacher": the patch-wise continuous latent vectors \(\tilde{\mathbf{Z}}^v, \tilde{\mathbf{Z}}^a\) output by the encoder (i.e., the patch tokens excluding the queries) are linearly interpolated to match the length of \(\mathcal{M}_F\)'s output, mapped via a small MLP projector \(h_\phi\), and aligned with the patch features \(\mathbf{Z}_F^v, \mathbf{Z}_F^a\) extracted by \(\mathcal{M}_F\) from the same input by maximizing their cosine similarity. The representation alignment loss \(\mathcal{L}_{rep}\) is formulated as:

\[\mathcal{L}_{rep} = -\mathbb{E}\left[\sum_{i\in\{a,v\}}\frac{1}{N_i}\sum_{k=1}^{N_i}\text{sim}(\mathbf{Z}_F^{i}[k], h_\phi(\text{interp}(\tilde{\mathbf{Z}}^{i})[k]))\right]\]

Here, \(\mathcal{M}_F\) and \(h_\phi\) are updated during training, and \(\mathcal{M}_F\) is discarded during inference. Performance check: removing \(\mathcal{L}_{rep}\) degrades rFAD from 5.93 to 8.48, indicating that the alignment loss is particularly crucial for audio reconstruction. Ablation studies also reveal that replacing CAV-MAE with the stronger CAV-MAE Sync yields consistent improvements.

4. Cross-Modal AR Generation Prior: Shaping an AR-Friendly Discrete Latent Space via NTP Objectives

AVTok's holistic queries form an unordered set, and the parallel processing of the Transformer encoder does not naturally generate a sequential order, yet downstream autoregressive generation requires an ordered token sequence. Following LARP, AVTok attaches a lightweight GPT-2 as an AR prior model \(\mathcal{M}_P\) during training, calculating the negative log-likelihood loss for next-token prediction (\(\mathcal{L}_{prior}\)) for both concatenation orders (\(\mathbf{x}^v \|\mathbf{x}^a\) and \(\mathbf{x}^a \|\mathbf{x}^v\)). The gradients are backpropagated to the tokenizer's encoder and quantizer, prompting the latent space to spontaneously form a sequential structure suited for AR generation. \(\mathcal{M}_P\) exists only during training and is discarded during inference. Ablation studies show that removing \(\mathcal{L}_{prior}\) yields the best reconstruction metrics (rFVD 10.63 vs. 12.80), but downstream generation quality plummets (gFVD 266.82 vs. 150.26)—this is a classic trade-off of the AR prior: it sacrifices a degree of reconstruction fidelity for downstream generation tasks to secure a sequentialized latent space structure.

A Complete Walkthrough: VFAL Three-Stage Training

Taking a 16-frame "playing guitar" audio-video clip from VGGSound as an example (128x128 resolution, ~4 seconds, 22kHz mono audio), the VFAL training workflow unfolds as follows:

Stage 1 (Video Reconstruction, 75 epochs): The input video frames \(\mathbf{V} \in \mathbb{R}^{16 \times 128 \times 128 \times 3}\) are patchified (\(f_T=4, f_H=8, f_W=8\)) into 1024 d-dimensional patch embeddings \(\mathbf{E}^v \in \mathbb{R}^{1024 \times d}\). The encoder receives the concatenation of \(\mathbf{E}^v\) and 1024 learnable holistic queries \(\mathbf{Q}_L^v\) (totaling 2048 tokens). After processing by 12 layers of shared Transformer (using video-specific LNs), the outputs corresponding to the first 1024 queries are mapped via SVQ to obtain 1024 discrete tokens \(\mathbf{x}^v\). The decoder concatenates 1024 patch queries \(\mathbf{Q}_P^v\) with the dequantized tokens, reconstructing 1024 patch embeddings through 12 decoder layers (video-specific LNs), which are then reshaped back to video frames to calculate \(\mathcal{L}_{rec}^v\) and \(\mathcal{L}_{prior}\) (employing unidirectional NTP on \(\mathbf{x}^v\) only). After 75 epochs, AVTok learns to compress visual information such as "guitarist strumming and moving fingers" into 1024 discrete tokens, with video rFVD converging from high initial values to around 13-14.

Stage 2 (Audio Reconstruction, 35 epochs): Freeze all shared parameters and video-specific components from Stage 1. Convert the audio of the same clip to a Mel spectrogram \(\mathbf{A} \in \mathbb{R}^{80 \times 384}\), patchified (\(f_M=16, f_L=16\)) to get 120 audio patch embeddings \(\mathbf{E}^a \in \mathbb{R}^{120 \times d}\). The encoder receives the concatenation of \(\mathbf{E}^a\) and 128 audio holistic queries \(\mathbf{Q}_L^a\), reusing the shared attention parameters from Stage 1 but with newly initialized audio-specific LNs. The shared codebook of the quantizer outputs 128 discrete tokens \(\mathbf{x}^a\), and the decoder reconstructs the Mel spectrogram, which is then reverted to "guitar string vibration sounds" via HiFi-GAN. At this point, \(\mathcal{L}_{rec}^a\) serves as the primary loss (weight 1.0), and \(\mathcal{L}_{rep}\) is enabled for the first time—patch features extracted by the CAV-MAE+ teacher guide the learning of encoder patch tokens for semantic alignment. Because the shared attention layers already possess the generalized capability to extract information from "visual patches" (the Mel spectrogram is essentially processed like a grayscale image), the audio stream successfully piggybacks on this shared representation within 35 epochs, with audio rFAD dropping from around 20 to approximately 6.

Stage 3 (Fine-tuning, 10 epochs): Fine-tune only the decoder (incorporating both video and audio stream LNs) to jointly optimize all four losses. After 10 epochs, video rFVD further improves from the level after video-only training (due to implicit cross-modal fusion, rFVD drops to 12.80), and audio rFAD stabilizes at 5.93. The resulting 1152 tokens (1024 video + 128 audio) serve as the unified 1D representation of the "playing guitar" audio-video content—in downstream tasks, these 1152 tokens can be fed directly into a Llama-like AR model for A2V (predicting video tokens given audio tokens), V2A (vice versa), or cJAVG (predicting all 1152 tokens given class tokens).

Loss & Training

The total loss is a weighted sum of four components:

\[\mathcal{L} = \lambda_1 \mathcal{L}_{rec}^v + \lambda_2 \mathcal{L}_{rec}^a + \lambda_3 \mathcal{L}_{rep} + \lambda_4 \mathcal{L}_{prior}\]

Here, \(\mathcal{L}_{rec}^v\) includes L1 reconstruction loss + LPIPS perceptual loss + GAN adversarial loss (ViT-based Discriminator) + SVQ quantization loss, with weights (1.0, 1.0, 0.3, 0.1); \(\mathcal{L}_{rec}^a\) consists of multi-scale Mel spectrogram reconstruction loss + deep feature matching loss + GAN adversarial loss (Multi-Scale Sub-Band CQT Discriminator + Multi-Period Discriminator) + SVQ quantization loss, with weights (15.0, 2.0, 1.0, 0.1). Discriminators are updated every 5 steps at 70% of the tokenizer's learning rate, and LeCam regularization is utilized to stabilize training. The four loss weights \(\lambda_{1,2,3,4}\) are dynamically adjusted across local VFAL stages (see Key Design 2). The optimizer is Adam (\(\beta_1=0.9, \beta_2=0.95\)), base lr=0.0001, scheduled with a cosine decay, and warm-up epochs for the three stages are set to 8, 3, and 1, respectively, with a constant batch size of 112.

Key Experimental Results

Main Results

Reconstruction Comparison (Table 1, TAVGBench + VGGSound test sets): Since unified audio-video tokenization is a completely new task, there are no direct open baselines for comparison. Hence, AVTok is compared separately with state-of-the-art video-only and audio-only 1D tokenizers. AVTok comprehensively outperforms all video-only baselines (including LARP) in video reconstruction, and closely approaches dedicated audio codecs in audio reconstruction.

Type Method Config #Tokens PSNR↑ rFVD↓ LPIPS↓ SI-SDR↓ rFAD↓ MR-STFT↓
VO OmniTokenizer 17×128×128 1280 23.84 90.99 0.203 - - -
VO AdapTok 16×128×128 2048 23.87 22.23 0.180 - - -
VO LARP 16×128×128 1024 24.53 14.24 0.137 - - -
AO WavTokenizer 98304×1 (W) 164 - - - 24.27 6.82 1.589
AO SpectralCodec 80×384 (M) 384 - - - 29.30 5.56 1.514
AV Vanilla (Single-stream) 16×128×128 / 80×384 1152 24.50 14.87 0.140 35.45 10.26 2.114
AV AVTok 16×128×128 / 80×384 1152 25.62 12.80 0.126 23.09 5.93 1.523

Note: Vanilla is the single-stream baseline implemented in this work (directly concatenating audio-video patches and feeding them to the shared encoder). AVTok's video PSNR is 1.09dB higher than LARP, and its audio rFAD is close to SpectralCodec (5.93 vs. 5.56), demonstrating that joint encoding is not only feasible but also that cross-modal information can mutually enhance both modalities.

Generation Comparison (Table 2, VGGSound test set): AVTok + AR generation model (Llama-like Transformer, ~208M tokenizer + 632M generator) is compared with dedicated approaches on three tasks: A2V, V2A, and cJAVG, achieving highly competitive results with a total parameter size far smaller than other methods.

Task Method Generation Paradigm Tokenizer Params Generator Params gFVD↓ gFAD↓ DeSync↓ IB-Score↑
A2V TempoTokens Diffusion 83.7M 1.9B 786.61 - 1.359 0.132
A2V AVTok-A2V AR 208.4M 632.0M 150.26 - 1.317 0.143
V2A MMAudio Flow Matching 298.5M 1.3B - 17.09 0.813 0.291
V2A AVTok-V2A AR 208.4M 632.0M - 49.47 1.239 0.249
cJAVG JavisDiT Flow Matching 448.7M 8.9B 1040.28 268.51 1.330 0.195
cJAVG Ovi Flow Matching 988.6M 17.3B 972.65 129.02 0.814 0.172
cJAVG AVTok-cJAVG AR 208.4M 632.4M 138.80 56.58 1.319 0.206

Note: AVTok achieves a massive lead in A2V gFVD (150.26 vs. 786.61) and cJAVG gFVD (138.80 vs. 972.65+). Specifically for cJAVG, it significantly outperforms Ovi (17.3B) and JavisDiT (8.9B) despite having a total parameter footprint that is an order of magnitude smaller. Although its V2A gFAD (49.47) is lower than MMAudio (17.09), it outperforms methods like V-AURA and SpecVQGAN, while MMAudio's generator contains twice the parameters.

Ablation Study

Config Recon. rFVD↓ Recon. rFAD↓ A2V gFVD↓ V2A gFAD↓ cJAVG gFVD↓ cJAVG gFAD↓
Vanilla (Single-stream) 14.87 10.26 - - - -
AVTok (Full) 12.80 5.93 150.26 49.47 138.80 56.58
w/o VFAL 13.19 9.38 209.33 61.02 193.28 80.78
w/o \(\mathcal{L}_{rep}\) 12.90 8.48 182.15 54.16 184.20 75.09
w/o \(\mathcal{L}_{prior}\) 10.63 3.47 266.82 67.84 249.47 90.11

Key Findings

  • The two-stream architecture is the foundation, but the VFAL training strategy is the key multiplier: Removing VFAL leads to a comprehensive collapse in both reconstruction and generation (with rFAD degrading from 5.93 to 9.38, and A2V gFVD from 150.26 to 209.33), demonstrating that even with a correct architecture, the benefits cannot be unlocked without a proper training sequence.
  • The classic trade-off of AR priors is quantitatively validated: Removing \(\mathcal{L}_{prior}\) yields the best reconstruction quality (rFVD 10.63 vs. 12.80 in the full setup), but plummets downstream generation performance (gFVD generally increases by 80-110 points), proving that \(\mathcal{L}_{prior}\) essentially trades off baseline reconstruction fidelity to construct a sequentialized latent space suitable for generation.
  • The number of video tokens significantly impacts audio reconstruction: When the number of video holistic tokens is halved from 1024 to 512, not only does video rFVD degrade from 12.80 to 23.85, but audio rFAD also rises from 5.93 to 14.90. This indicates that cross-modal information carried by the video stream benefits audio reconstruction, whereas reducing audio tokens has a negligible impact on video reconstruction.
  • Massive gains in generation efficiency: AVTok-cJAVG incurs a total inference latency of 12.76s / 3.48 TFLOPs, whereas Ovi requires 87.28s / 14.99K TFLOPs and JavisDiT takes 32.24s / 2.60K TFLOPs. AVTok maintains an order-of-magnitude advantage in efficiency.

Highlights & Insights

  • "Shared parameters + isolated LN/queries" is the essence of two-stream designs: Instead of using simplistic dual encoders, the same parameter set is recycled sequentially by both modalities—shared attention layers provide implicit cross-modal interaction paths, while isolated LNs and queries safeguard modality-specific features. This design pattern can extend to any scenario involving heterogeneous modality fusion (e.g., video + text, image + depth map). The core principle remains: "share what can be shared, isolate what cannot."
  • VFAL's "difficult-first, easy-later" philosophy has general applicability: First train the modality with higher information density to anchor a rugged representation space, then allow the sparse-information modality to "piggyback" on this space, followed by joint fine-tuning. This rationale is generalizable to any multimodal joint training scenario featuring information density imbalances (e.g., RGB + depth, point cloud + image, video + caption).
  • Representation alignment using an external foundation model is a low-cost approach to enhance multimodal fusion quality: CAV-MAE+ functions merely as a "crutch" during training and is discarded during inference, adding zero computational overhead to the final model. This pattern (external frozen model + small projector + alignment loss) has precedents in works like DeRA and REPA; AVTok proves its effectiveness in the multimodal tokenization domain.
  • Unifying audio and video via 1D discrete tokenization is conceptually elegant: It transforms the "3D vs. 1D" modality discrepancy into "both use 1D, just with varying patch dimensions and numbers," paving the way for constructing unified multimodal large models capable of both audio-visual understanding and generation.

Limitations & Future Work

  • Resolution Constraints: Constrained by compute, AVTok was evaluated only on 16-frame 128x128 videos with ~4 seconds of 22kHz audio. Scaling to longer durations and higher resolutions requires overcoming the bottlenecks of positional encodings and computational complexity.
  • Implicit Synchronization Modeling: Currently, AVTok only captures cross-modal temporal sync implicitly via synchronous inputs, shared parameters, and AR priors. The model still suffers from audio-video out-of-sync issues during generation (DeSync metric 1.317-1.319, weaker than Ovi's 0.814). Explicitly modeling temporal alignment (e.g., introducing a sync loss or time-aware attention mechanisms) is a direct and crucial future direction.
  • Complex VFAL Three-Stage Training Pipeline: Although effective, this pipeline requires manually configuring the active modules, epochs, and loss weights for each stage. It incurs high tuning overhead and potential sequential cascading errors. Exploring end-to-end single-stage training (perhaps via dynamic loss scheduling or curriculum learning) is a promising avenue to simplify the process.
  • Limited Data Scale: The training data is limited to 640K pairs (TAVGBench subset 460K + VGGSound 180K), which is far smaller than datasets used by mainstream video generation models. Scaling to larger datasets and diverse scenes is highly likely to boost generalization.
  • Audio Reconstruction Quality Behind Dedicated Codecs: AVTok's SI-SDR (23.09) lags behind SpectralCodec (29.30), indicating a compromise in audio fidelity during unified tokenization. Stronger audio-specific decoders or post-processing modules may be necessary.
  • vs. LARP / AdapTok / DeRA (1D Video Tokenizers): AVTok directly inherits LARP's query-based holistic tokenization architecture and AR prior training mechanism, extending the core concept from unimodal video to joint audio-video encoding. Unlike AdapTok's adaptive temporal causal modeling or DeRA's spatial-temporal decoupling, AVTok's core contribution lies in its "two-stream shared" architectural design rather than the token organization itself.
  • vs. CAV-MAE / CAV-MAE+: Both are representative works in audio-visual pre-training, employing two-stream architectures + contrastive learning for self-supervised representation learning. AVTok borrows their two-stream concept for tokenization rather than representation learning, and AVTok's tokenization is holistic (global queries) rather than patch-wise local. Additionally, CAV-MAE+ serves as the alignment teacher in AVTok, creating an interesting closed-loop system: "using an AV representation model to train an AV tokenizer."
  • vs. Ovi / JavisDiT (Joint Audio-Video Generation): These methods represent the current mainstream paradigm—employing dual-branch VAEs + dual-branch DiTs, which are parameter-heavy but produce high-quality outputs. AVTok operates at a fraction of their parameter size, even outperforming them in A2V and cJAVG metrics, though a performance gap remains in V2A audio quality (gFAD) and audio-video synchronization (DeSync). This suggests that the unified AR paradigm possesses an inherent advantage in video generation (as AR's chronological sequence modeling naturally fits video dynamics) but lags behind diffusion/flow-matching approaches in terms of fine-grained audio precision.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Proposing "unified audio-video tokenization" itself is pioneering. The designs of the two-stream shared architecture and the VFAL training strategy are both clever and practical.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensive comparisons are conducted across reconstruction and three downstream generation tasks. Ablations thoroughly cover architecture and training components. The appendix supplies analyses on efficiency, scaling, token counts, and choice of external models. Deducted one star because the main experiments lack directly comparable unified tokenizer baselines (which is a domain gap, not the authors' fault).
  • Writing Quality: ⭐⭐⭐⭐ Well-structured. The motivations and diagrams (Fig.2 representation gap, Fig.3 methodology overview) allow readers to quickly grasp core ideas. The appendix provides sufficient details to ensure good reproducibility.
  • Value: ⭐⭐⭐⭐⭐ Presents the first viable path for the pipeline "unified multimodal tokenization → unified multimodal generation." Boasting a clean concept, robust results, and massive efficiency advantages, it holds potential to steer future research in audio-video generation and general multimodal large models.