AnyCap: Omni-Modal Captioning with Instruction Alignment¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/qishisuren123/AnyCap
Area: LLM Reasoning
Keywords: Multimodal captioning, instruction alignment, omni-modal generation, revision-based refinement, keypoint density
TL;DR¶
Addressing the lack of fine-grained instruction controllability in existing multimodal captioners and the risk of catastrophic forgetting caused by direct fine-tuning, this paper proposes AnyCap—a plug-and-play revision-based alignment framework that jointly serves images, videos, and audio using a single model, supported by a 300k instruction triplet corpus AnyCapData and the decoupled AnyCapEval benchmark.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) are accelerating the transition toward omni-modal artificial intelligence, where text captions serve as a unified linguistic bridge connecting perception and semantic reasoning. In downstream operational tasks such as multimodal cross-modal retrieval, visual question answering, and text-to-media generative synthesis, real-world users increasingly demand fine-grained, instruction-driven control over the descriptive content, spatial perspective, narrative genre, and stylistic form of generated captions. However, the majority of existing models are pretrained as general-purpose perceptual backbones; when exposed to open-ended generation requests, they predominantly yield coarse, generic summaries or fabricate unrequested peripheral details, falling far short of following granular natural language instructions. While earlier works attempted to enforce control via discrete bounding boxes, spatial masks, or reserved special tokens, these rigid interfaces cannot capture the expressive flexibility and compositional richness inherent in natural language directives.
Compounding this difficulty, direct end-to-end instruction fine-tuning of foundation models introduces severe catastrophic forgetting, which rapidly degrades their underlying general language understanding and reasoning competencies. Furthermore, each perceptual modality exhibits fundamentally divergent descriptive imperatives: static images emphasize spatial entity attributes and local textures, temporal videos require dynamic event progression and camera motion tracking, and acoustic audio relies strictly on transient sonic events and prosodic cues. Meeting these heterogeneous demands typically forces researchers to maintain separate, siloed architectures for each modality. At the same time, the research ecosystem severely lacks high-quality omni-modal instruction corpora and reliable evaluation benchmarks: existing open datasets are predominantly single-modality and short-form, while conventional n-gram metrics (e.g., BLEU, CIDEr) lack semantic nuance and unconstrained LLM-as-a-judge scorers suffer from high random variance and verbosity bias.
Overcoming these structural tensions requires abandoning the conventional paradigm of unconstrained from-scratch generation in favor of a lightweight, cross-modal, plug-and-play alignment strategy that preserves foundation parameters intact. Core idea: Decouple omni-modal instruction alignment into a "base draft generation followed by cross-modal revision" paradigm, leveraging unconstrained base captions as semantic scaffolding while a lightweight single model jointly consumes raw multimodal features, initial text, and dynamic natural language instructions to learn targeted revisions, delivering high-fidelity instruction adherence across image, video, and audio without retraining base models.
Method¶
Overall Architecture¶
AnyCap operates under a modular, plug-and-play revision-based refinement paradigm. The input consists of an arbitrary multimodal artifact \(m \in \{m_{\text{img}}, m_{\text{vid}}, m_{\text{aud}}\}\) along with a fine-grained natural language user instruction \(q\). First, an arbitrary, fully frozen foundation multimodal model (such as GPT-4o, InternVL2.5, or Qwen2.5-VL) processes the input \(m\) to produce an unguided, initial base caption \(y_0\). Subsequently, the lightweight AnyCap module \(M_a\) simultaneously takes the raw multimodal observation \(m\), the user instruction \(q\), and the initial base draft \(y_0\) to generate an instruction-compliant, refined caption formulated as \(y_c = M_a(m, q, y_0)\).
Architecturally, AnyCap extracts modality representations via frozen modality-specific encoders: InternViT is employed for spatial image patches \(m_{\text{img}}\) and temporal video frames \(m_{\text{vid}}\), while Efficient Audio Transformer (EAT) processes acoustic audio clips \(m_{\text{aud}}\). Modality-specific multi-layer perceptron (MLP) projectors transform these diverse perceptual features into a shared semantic space. Simultaneously, the user instruction \(q\) and base draft \(y_0\) are tokenized into textual embeddings. These visual/audio tokens and textual tokens are concatenated and fed into an autoregressive language decoder (initialized from InternLM), which performs causal next-token prediction to synthesize the final revised caption \(y_c\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Media Input (Image/Video/Audio) + User Instruction q"] --> B["Cross-Modal Feature Encoding & Unified Projection<br/>Extract features via InternViT/EAT and map via dedicated MLPs"]
B --> C["Frozen Base MLLM Generates Initial Draft<br/>Produce unconstrained semantic baseline y_0"]
C --> D["Tri-Conditional Revision-Based Alignment Generation<br/>Jointly ingest multimodal tokens, draft y_0, and instruction q"]
D --> E["AnyCapEval Decoupled Benchmark Assessment<br/>Evaluate Content via KPD and Style via discrete rubrics"]
E --> F["Output High-Fidelity, Instruction-Aligned Refined Caption y_c"]
Key Designs¶
1. Cross-Modal Feature Encoding & Unified Projection: Harmonizing Heterogeneous Sensory Inputs
To circumvent the inefficiency of maintaining disparate architectures across sensory modalities, AnyCap unifies spatial, temporal, and acoustic representations within a single language modeling substrate. For visual modalities (static images \(m_{\text{img}}\) and sequence videos \(m_{\text{vid}}\)), AnyCap utilizes a frozen InternViT backbone to extract patch-level spatial and spatiotemporal representations. For auditory inputs (\(m_{\text{aud}}\)), it integrates a frozen Efficient Audio Transformer (EAT) encoder to capture spectral dynamics and acoustic event temporalities. Modality-specific two-layer non-linear MLP projectors map each heterogeneous feature stream into the identical embedding dimension of the language model. This unified projection allows the shared language decoder to process all three sensory modalities within a single parameter set, avoiding architectural proliferation and cross-modal interference.
2. Tri-Conditional Revision-Based Alignment Generation: Grounded Editing over Draft Scaffolding
Generating long, constrained multimodal descriptions from scratch frequently suffers from attention dilution, leading to severe hallucination or instruction omission. AnyCap shifts the formulation to conditional editing: the unconstrained base caption \(y_0\) supplies an initial semantic scaffold, freeing the revision model to focus its attention on decoding the user's specific instructional focus \(q\) and cross-checking raw perceptual features \(m\). The objective optimizes standard causal next-token prediction on the ground-truth sequence \(y_c\). Crucially, to mitigate the risk of "over-editing"—where an already satisfactory caption is unnecessarily rewritten or degraded—the training pipeline explicitly incorporates approximately 40% identity-mapped negative editing samples \((q, c, c)\) wherein the input caption \(c\) already adheres to instruction \(q\). This trains the network to exercise restraint when no correction is warranted while selectively pruning or expanding details when instructions demand divergence.
3. Large-Scale Triplet Corpus AnyCapData: 28 Structured Instruction Dimensions
To establish systematic instruction coverage across modalities, the authors construct AnyCapData, encompassing 300k high-quality samples distributed across images (125k), videos (100k), and audio (75k). AnyCapData formalizes 28 distinct, fine-grained instruction categories derived from extensive literature and user requirement analysis. Content-oriented controls include scene background detailing (Bkg), event progression (Evt), entity highlighting (Ins), instance action tracking (IAct), instance appearance (IApp), relative spatial position (IPos), camera movement (Mov), viewpoint perspective (Per), and regional localization (Reg). Style-oriented controls span concise summarization (Brf), fine granularity (Det), literary genre like poetry and narrative (Gen), strict word/sentence length limits (Len), and thematic phrasing (Thm). Each sample is organized as a data-aware triplet \((q, c, a)\), where \(q\) is a tailored, non-generic instruction grounded in the content, \(c\) is a compliant gold caption, and \(a\) is a suboptimal candidate exhibiting either instruction non-compliance or synthetic hallucinations, delivering explicit contrastive supervision for corrective learning.
4. AnyCapEval Decoupled Benchmark Assessment & Keypoint Density Metric: Eliminating Evaluator Variance
Traditional machine translation metrics cannot measure factual faithfulness, while unconstrained LLM evaluators exhibit severe verbosity bias and score volatility. AnyCapEval addresses these deficiencies by orthogonally decoupling evaluation into Content and Style dimensions, preventing polished prose from masking factual hallucinations. For Content evaluation, the framework introduces the Keypoint Density (KPD) metric. Given ground-truth required keypoints \(\mathcal{K}_{r,q} = \{k_1, k_2, \dots, k_n\}\) specified by instruction \(q\), GPT-4o operates strictly as a deterministic binary extractor verifying which keypoints are present in the prediction \(y_c\), normalized by length: $\(\text{KPD}(y_c; q) = \frac{N_{\text{match}}}{L_{\text{words}}(y_c)} \times 100\)$ where \(N_{\text{match}}\) denotes verified keypoint counts and \(L_{\text{words}}(y_c)\) is total word length. KPD rewards dense, concise coverage while strictly penalizing verbose padding. For Style evaluation, AnyCapEval establishes discrete 5-tier rubrics (0 to 4) with strict length tolerance gates (e.g., brief and detailed captions must fall within \(\pm 30\%\) of reference word counts, otherwise capping scores at 1). Human correlation studies confirm substantial agreement (Spearman \(\rho \ge 0.88\), Kendall \(\tau \ge 0.82\), IAA \(\ge 0.80\)).
Loss & Training¶
AnyCap is instantiated in two parameter scales, AnyCap-2B and AnyCap-8B, with language decoders initialized from InternLM2. The external modality encoders (InternViT and EAT) remain frozen throughout training; only the MLP projectors and language decoder weights are updated. Optimization minimizes the negative log-likelihood of the ground-truth target sequence \(Y_c = (t_1, t_2, \dots, t_T)\) conditioned on the projected multimodal representations \(H_m\), instruction tokens \(X_q\), and initial caption tokens \(X_{y_0}\): $\(\mathcal{L}(\theta) = - \sum_{i=1}^T \log P_\theta(t_i \mid H_m, X_q, X_{y_0}, t_{<i})\)$ Training is conducted across 32 NVIDIA A100 (80GB) GPUs using AdamW in bfloat16 mixed precision. Key hyperparameters include a global batch size of 256, a cosine learning rate schedule with a peak learning rate of \(1 \times 10^{-6}\), warmup ratio of 0.03, weight decay of 0.01, drop path rate of 0.1, and maximum sequence length of 4096. AnyCap-2B converges within 6 hours across 3 epochs, while AnyCap-8B requires 21 hours, exhibiting remarkable training efficiency.
Key Experimental Results¶
Main Results¶
AnyCap was evaluated across image, video, and audio modalities on the AnyCapEval benchmark, serving both proprietary models (GPT-4o, GPT-4o mini) and open-source models (InternVL2.5, Qwen2.5-VL, InternVL3.5 series, MiniCPM-o). Content is measured by KPD (\(\uparrow\)) and Style by rubric scores (\(\uparrow\)).
| Modality / Foundation Backbone | Setup | Content Score (KPD \(\uparrow\)) | Gain over Base | Style Score (0-4 \(\uparrow\)) | Gain over Base |
|---|---|---|---|---|---|
| Image Modality | |||||
| GPT-4o (Proprietary) | Vanilla Base | 2.89 | — | 2.26 | — |
| + AnyCap-2B | 4.11 | +1.22 (+42.2%) | 2.46 | +0.20 (+8.8%) | |
| + AnyCap-8B | 4.54 | +1.65 (+57.1%) | 2.65 | +0.39 (+17.3%) | |
| InternVL2.5-8B (Open-Source) | Vanilla Base | 3.04 | — | 2.12 | — |
| + AnyCap-2B | 4.13 | +1.09 (+35.9%) | 2.36 | +0.24 (+11.3%) | |
| Qwen2.5VL-7B (Open-Source) | Vanilla Base | 3.31 | — | 2.22 | — |
| + AnyCap-2B | 4.36 | +1.05 (+31.7%) | 2.50 | +0.28 (+12.6%) | |
| InternVL3.5-38B (Large-Scale) | Vanilla Base | 3.56 | — | 2.30 | — |
| + AnyCap-2B | 4.07 | +0.51 (+14.3%) | 2.36 | +0.06 (+2.6%) | |
| Video Modality | |||||
| GPT-4o | Vanilla Base | 3.55 | — | 2.15 | — |
| + AnyCap-2B | 5.30 | +1.75 (+49.3%) | 2.30 | +0.15 (+7.0%) | |
| + AnyCap-8B | 5.74 | +2.19 (+61.7%) | 2.32 | +0.17 (+7.9%) | |
| InternVL2.5-8B | Vanilla Base | 3.52 | — | 1.93 | — |
| + AnyCap-2B | 5.07 | +1.55 (+44.0%) | 2.19 | +0.26 (+13.5%) | |
| Audio Modality | |||||
| GPT-4o | Vanilla Base | 1.59 (Evt) | — | 1.18 (Avg) | — |
| + AnyCap-2B | 1.79 (Evt) | +0.20 | 1.28 (Avg) | +0.10 | |
| + AnyCap-8B | 1.88 (Evt) | +0.29 | 1.30 (Avg) | +0.12 | |
| MiniCPM-o (Open-Source Audio) | Vanilla Base | 1.42 (Evt) | — | 0.71 (Avg) | — |
| + AnyCap-2B | 1.55 (Evt) | +0.13 | 1.21 (Avg) | +0.50 (+70.4%) |
Ablation Study¶
The table below contrasts AnyCap's revision-based paradigm against standard alignment strategies (Full Supervised Fine-Tuning SFT, Direct Preference Optimization DPO, and Test-Time Self-Critic prompting SC) and examines the effect of training data ratios between \((q, \text{uncontrolled } a, c)\), \((q, c, c)\), and \((q, \text{hallucinated } a, c)\):
| Alignment Paradigm / Data Ratio | Backbone & Target Modality | Relative Content Gain (%) | Relative Style Gain (%) | Notes & Empirical Behavior |
|---|---|---|---|---|
| SFT (Supervised Fine-Tuning) | InternVL2.5-8B (Image) | +9.2% | +3.5% | End-to-end retraining induces catastrophic forgetting |
| DPO (Preference Optimization) | InternVL2.5-8B (Image) | +9.5% | +4.4% | Preference pairs improve adherence marginally |
| Self-Critic (Prompting SC) | InternVL2.5-8B (Image) | +3.3% | +0.5% | Test-time self-revision prone to hallucination loops |
| AnyCap-2B (Revision-Based) | InternVL2.5-8B (Image) | +35.9% | +11.3% | Plug-and-play revision sharply outclasses retraining |
| AnyCap-8B (Revision-Based) | InternVL2.5-8B (Image) | +40.1% | +16.0% | Higher capacity expands stylistic control ceiling |
| Data Ratio 1:1:1 (2B-111) | 3-Modality Average | +25.8% | +10.0% | Balanced composition, moderate over-editing |
| Data Ratio 1:2:2 (2B-122) | 3-Modality Average | +25.1% | +9.1% | Standard balance |
| Data Ratio 2:2:1 (2B-221) | 3-Modality Average | +30.8% | +11.3% | Optimal ratio: strong uncontrolled correction + 40% (q,c,c) |
| Data Ratio 1:0:2 (2B-102) | 3-Modality Average | +26.4% | +3.9% | Omitting \((q,c,c)\) leads to severe stylistic over-editing |
Key Findings¶
- Augmenting mid-sized open-source models with AnyCap-2B consistently elevates them to rival or surpass unassisted proprietary baselines; for instance, InternVL2.5-8B with AnyCap-2B attains an image style score of 2.36, surpassing unassisted GPT-4o (2.26).
- An explicit breakdown of absolute keypoints versus response length (Table 12) demonstrates that AnyCap does not artificially inflate scores via verbosity padding. In fact, it effectively excises irrelevant details: on GPT-4o's perspective-controlled outputs, average response length dropped from 95.8 to 49.6 words while absolute keypoint recall remained stable or increased (1.42 to 1.55 on IPos).
- In the paradigm ablation, AnyCap completely outperforms full-parameter SFT (+35.9% vs +9.2% on image content) and DPO (+9.5%), demonstrating that isolating the alignment mechanism into an external revision model is far superior to forcing the base model to simultaneously balance perception and control.
- Training composition ablations demonstrate that maintaining approximately 40% \((q, c, c)\) non-editing instances is essential for regularizing revision behavior; completely removing these identity pairs (ratio 1:0:2) causes style gain to collapse from +11.3% to +3.9% due to uncontrolled rewriting.
Highlights & Insights¶
- Plug-and-Play Cross-Modal Revision: By shifting the learning burden from unconstrained generation to conditional editing anchored on a draft caption, AnyCap avoids retraining massive base models, preventing language degradation while offering universal pluggability across open-source and API backbones.
- Orthogonally Decoupled Benchmark Formulation: AnyCapEval separates objective factual compliance (evaluated deterministically via Keypoint Density) from subjective linguistic quality, removing evaluator variance and establishing a reliable benchmark for controlled captioning.
- Unified Tri-Modal Parameter Efficiency: Ingesting images, temporal videos, and audio spectra through a single 2B/8B language backbone with frozen specialized encoders proves that cross-modal alignment does not require fragmented, modality-specific models.
Limitations & Future Work¶
- Uncovered Sensory Modalities: The current framework is restricted to visual (image, video) and acoustic signals; emerging physical modalities such as 3D point clouds, tactile sensing, and robotic proprioceptive states remain unaddressed due to the scarcity of paired instruction data.
- Sequential Two-Stage Latency: Because the pipeline requires an initial forward pass by the foundation model followed by a second forward pass by AnyCap, it introduces sequential latency that may challenge strict real-time edge streaming deployments.
- Dependency on Base Perception Quality: While AnyCap leverages raw multimodal features to correct hallucinations, if the initial base caption suffers from catastrophic misrecognition (e.g., misidentifying the primary scene entity entirely), the revision module can occasionally inherit semantic misdirection.
Related Work & Insights¶
- vs Aligner (Ji et al., 2024): Aligner operates purely in the text domain with a static safety alignment target; AnyCap scales revision-based refinement to omni-modal perception with dynamically user-specified natural language instructions.
- vs ControlCap / FlexCap (Zhao et al., 2024; Dwibedi et al., 2024): Earlier controllable captioning works relied on bounding boxes or length embeddings; AnyCap accepts arbitrary natural language instructions spanning 28 diverse content and style dimensions.
- vs Native Omni-Models (Qwen2.5-Omni, MiniCPM-o): Native omni-modal foundation models exhibit weak instruction adherence in audio captioning; AnyCap provides an external, plug-and-play capability boost that increases MiniCPM-o's audio style score by 70.4%.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering plug-and-play revision-based instruction alignment across vision and audio within a unified model.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 28 instruction types, 3 modalities, multiple proprietary/open backbones, human studies, and downstream video synthesis benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly structured, rigorous formulation, and lucid narrative presentation throughout.
- Value: ⭐⭐⭐⭐⭐ Substantial practical utility for multimodal interactive agents, controllable dataset curation, and text-to-media generative synthesis.