ChronusOmni: Improving Time Awareness of Omni-Modal Large Language Models

Conference: ECCV2026
Paper: Official paper page · PDF
Area: Video Understanding
Keywords: Audiovisual Temporal Grounding, Temporal Interleaved Tokenization, Omni-Modal LLMs, GRPO, Cross-Modal Alignment

TL;DR

ChronusOmni interleaves absolute-time text, video frames, and corresponding audio for Ola-7B, then learns six temporal tasks through audiovisual dense-captioning SFT and task-reward GRPO, reaching 79.85 audio-to-time [email protected] on ChronusAV and 34.5 zero-shot temporal-grounding mIoU on LongVALE.

Background & Motivation

Video temporal grounding typically asks when an action occurs, but real videos also contain off-screen speech, dialogue, environmental sounds, and music. Understanding the images alone does not answer what is visible when a particular sentence is heard. Combining everything into one audiovisual caption also makes it difficult to establish whether the model actually aligned the two modalities in time. A shot may show Character A while Character B speaks off screen, followed by a cut to B's mouth movements: semantic relatedness does not imply temporal coincidence.

Existing systems expose time through different interfaces, including frame indices, learned time embeddings, and timestamps rendered as image watermarks. Frame indices do not directly specify seconds, while watermarks require the model to read text from pixels. Meanwhile, datasets such as LongVALE use joint audiovisual descriptions, making it harder to test audio-to-visual and visual-to-audio matching separately. This paper treats time, vision, and audio as three mutually queryable elements: V2T, A2T, T2V, and T2A represent explicit temporal grounding, while V2A and A2V represent implicit cross-modal grounding. A visual or audio query can be a content description; the answer is an interval or a textual description of the other modality, not generated video or sound.

The data interface and training objective must change together. The former exposes dependable temporal correspondences, while the latter distinguishes a nearly correct boundary from a completely incorrect one. Core idea: connect synchronized audiovisual tokens with directly readable absolute-time text, establish supervision through separate modality captions, and refine six-way grounding using temporal-overlap and caption-quality rewards.

Method

Overall Architecture

The input is a video with its audio track and a query; the output is either a start/end interval or a visual/audio description of the target segment. On the data side, ChronusAV supplies aligned triplets. On the model side, Temporal Interleaved Tokenization organizes the inputs, followed by Temporal-Aware SFT and Task-Reward GRPO. Inference uses the trained model without annotation teachers or an online reinforcement-learning procedure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    source["Long videos with audio"] --> dataset["ChronusAV Aligned<br/>Annotation"]
    source --> tokens["Temporal Interleaved<br/>Tokenization"]
    dataset --> sft["Temporal-Aware SFT"]
    tokens --> sft
    sft --> grpo["Task-Reward GRPO"]
    dataset --> grpo
    grpo --> output["Trained model<br/>Interval or modality caption"]

The backbone is Ola, built on Qwen-2.5-7B. OryxViT encodes 64 uniformly sampled video frames. Whisper-Large-V3 encodes speech, while BEATs encodes sounds and music; their features are concatenated along the channel dimension and downsampled by a factor of 10. Separate two-layer MLP adapters project visual and audio features into the language model. The approach does not train a new audiovisual encoder stack: its main changes concern input organization and language-model post-training.

Key Designs

1. ChronusAV Aligned Annotation: describe what is seen and heard separately within the same interval

The dataset draws English videos with audio from Panda-70M, restricting duration to 60–600 seconds. Visual scenes are first split and semantically similar segments merged; the same boundaries then segment the audio. Videos must contain 5–30 segments. Gemini-2.5-Flash annotates visual clips and Gemini-2.5-Pro annotates audio clips, producing triplets of a time interval, a visual caption, and an audio caption. Modality separation matters because a visual answer that borrows information from the audio could reward label leakage rather than genuine cross-modal temporal alignment.

The resulting collection contains approximately 47K videos and 677K segments. The test split holds out 2,000 videos, selects one segment per video, and generates six directional questions from each triplet, yielding 12K QA pairs that are all human-verified and refined. A separate annotation-quality study checks 1,000 randomly sampled segments, finding visual and audio captions mostly accurate in 96.1% and 93.5% of cases. This supports annotation quality but does not mean that every training annotation was reviewed by a person.

The six queries reuse the same underlying triplet, allowing explicit and implicit temporal abilities to be measured separately. A2T retrieves an interval from an audio description, whereas A2V must describe the simultaneous visual content instead of guessing a likely scene from sound-related common knowledge. One caveat is that segmentation starts with visual events and reuses their boundaries for audio. Shared boundaries establish synchronization, but they do not guarantee that every sound event's natural boundaries coincide with those of a visual event.

2. Temporal Interleaved Tokenization: give frames and audio a shared absolute-time coordinate

TIT writes each sampled instant as ordinary text in the form second{t} and places the visual tokens for that instant beside the audio tokens spanning the interval until the next sampled instant. Videos of different lengths can all have 64 sampled frames while having different temporal spacing, so sequence position alone does not specify elapsed seconds. Text timestamps reuse the language model's existing number-processing interface and avoid reading time from image watermarks.

Written as a complete sequence, the organization described by the paper's Equation (2) is:

\[ I=[T_1,V_1,A_1,T_2,V_2,A_2,\ldots,T_n,V_n,A_n]. \]

Here \(T_i\) contains text tokens for absolute time \(t_i\), \(V_i\) represents the frame at that instant, and \(A_i\) represents audio from \([t_i,t_{i+1})\). Rather than appending an entire audio stream after an entire video stream, TIT makes local time-image-sound adjacency explicit. The model still has to learn the associations: TIT is not a hard boundary constraint and does not structurally guarantee that every predicted interval stays within the video duration.

3. Temporal-Aware SFT: learn complete event descriptions before query-conditioned retrieval

The first stage uses dense audiovisual captioning on ChronusAV. Given the full video and audio, the model outputs temporal boundaries, visual captions, and audio captions for the events. Compared with a single global summary, this requires events to remain distinct while preserving modality identity and temporal correspondence. Training also mixes in audiovisual speech recognition from How2 and audiovisual question answering from AVSD, maintaining supervision for pre-existing understanding capabilities.

However, when time is generated as text, maximum-likelihood training mainly rewards the correct tokens rather than the geometric distance between continuous intervals. Gold prefixes during training can also encourage memorized output patterns that do not translate into precise boundaries during free generation. SFT therefore establishes the temporal-description interface; it is not the final objective for improving temporal precision.

4. Task-Reward GRPO: score intervals by overlap and captions by target-modality content

The second stage directly trains six-way question answering. GRPO updates the policy using relative rewards among candidate responses to the same question. V2T and A2T produce intervals, so they use temporal IoU rather than exact timestamp-string matching. A prediction close to the correct boundaries can therefore receive nonzero feedback. Following the prose definition of Equation (3), the reward is:

\[ R_{\mathrm{IoU}}=\frac{|I_{\mathrm{pred}}\cap I_{\mathrm{gt}}|}{|I_{\mathrm{pred}}\cup I_{\mathrm{gt}}|}. \]

Lengths here refer to temporal duration. An additional format reward requires responses of the form second{start}-second{end}. Equation (4) is damaged in the cached extraction: the incorrect-format branch can be read as 0, and the intended output format is clear, but the correct-format score and the weighting with IoU cannot be reliably recovered. Consequently, no reconstructed combined-reward formula is supplied here.

T2V, T2A, V2A, and A2V produce text captions, using METEOR between the prediction and the corresponding modality's reference caption as their reward. This avoids forcing captioning into interval prediction while allowing one temporal interface to support all six directions. METEOR remains a text-matching proxy, however: it does not directly verify that every described event occurred within the requested interval. Claims about implicit temporal grounding must therefore be assessed alongside dataset construction and independent evaluation.

A Worked Example

The A2T example in Figure 4 uses a 218-second video. The query describes a female voice and light music near the end, with the reference interval \([201.7,217.9]\) seconds. Encoders extract the audiovisual content, TIT places it on an absolute timeline, and the language model predicts the boundaries. ChronusOmni returns \([201.6,217.9]\), whereas Qwen3-Omni returns \([302,317]\), outside the video duration. This is an illustrative comparison, not a guarantee that the proposed model never produces out-of-range predictions.

The V2A example in the same figure asks about the audio accompanying a plastic-bag-opening scene at 120.0–131.9 seconds. A correct answer must cover the speech, packaging noise, and other sounds in that interval without importing later mixing sounds. The paper illustrates omissions and extraneous later events in baseline answers. However, ChronusOmni describes a loud bang where the reference mentions a bell, showing that improved temporal coverage does not imply perfect sound-category recognition.

Loss & Training

Both stages update only the LLM; all encoders and adapters remain frozen. SFT uses 10K How2 samples, 30K AVSD QA pairs, and 30K ChronusAV videos, totaling 70K samples for one epoch. GRPO uses 4,000 QA pairs drawn from the ChronusAV training split for 1,000 steps. For each selected training video, one segment is sampled and a question from one of the six tasks is constructed; this is not six questions for every training video.

The available cache does not contain the appendix referenced by the main text. It also does not specify the learning rate, GRPO group size, KL coefficient, or complete reward-combination details. These reproduction parameters are not inferred from generic GRPO implementations.

Key Experimental Results

Main Results

For ChronusAV grounding, [email protected] is the fraction of samples whose single predicted interval has an IoU of at least 0.7 with the reference. The captioning columns below use METEOR. All values preserve the original reporting scale; METEOR is not treated as accuracy. FT means the baseline was also finetuned on the proposed training data. Qwen3-Omni is the 30B-A3B model; the other models below are 7B.

Model	V2T [email protected]	A2T [email protected]	T2V METEOR	T2A METEOR	V2A METEOR	A2V METEOR
Qwen3-Omni	21.80	33.10	1.62	1.20	1.25	1.49
Qwen2.5-Omni (FT)	34.20	35.30	1.78	3.90	2.91	1.87
ARC-Hunyuan-Video (FT)	38.05	56.55	1.60	1.69	1.11	1.48
ChronusOmni	45.95	79.85	2.12	4.50	3.27	2.12

These results come from Table 2. Against the strongest finetuned baseline for each metric, V2T [email protected] improves by 7.90 percentage points and A2T by 23.30 percentage points. Implicit V2A and A2V METEOR improve by 0.36 and 0.25, respectively. These differences cannot be attributed entirely to TIT, because the backbone and post-training recipe also differ; the ablations provide complementary evidence.

LongVALE is evaluated zero-shot. Omni-TVG retrieves an interval from a joint audiovisual event caption, Omni-DVC generates boundaries and captions for all events, and Omni-SC describes a specified interval. The following grounding and captioning metrics are selected from Table 3:

Zero-shot model	Omni-TVG [email protected]	[email protected]	[email protected]	mIoU	Omni-DVC METEOR	Omni-SC METEOR
ARC-Hunyuan-Video	38.1	24.6	13.4	26.9	2.5	6.0
Qwen3-Omni	35.1	26.3	15.8	26.1	2.7	8.0
ChronusOmni	49.7	32.5	17.6	34.5	5.2	11.7

Table 3 reports a +41.6% relative improvement for [email protected], but the actual second-best value in the table is 38.1, which gives approximately +30.4%. The text's aggregate claim of a 28.8% average gain is therefore not treated here as independently verified. Raw scores are retained instead of repeating the questionable aggregate improvement.

Ablation Study

The following values are from Table 6, using the same ChronusAV tasks and metrics as the main comparison.

Configuration	V2T [email protected]	A2T [email protected]	T2V METEOR	T2A METEOR	V2A METEOR	A2V METEOR
Without TIT	6.80	13.00	1.67	2.99	2.75	1.79
Without SFT	37.20	69.25	1.81	1.89	1.96	1.84
Without GRPO	15.45	19.00	1.07	0.35	0.73	0.66
Full model	45.95	79.85	2.12	4.50	3.27	2.12

Key Findings

• TIT is especially important for strict boundary localization: removing it reduces V2T and A2T [email protected] by 39.15 and 66.85 percentage points. GRPO also matters strongly for captioning; without it, T2A METEOR falls from 4.50 to 0.35.
• Charades-STA cannot be called zero-shot because AVSD training data contains its videos. The paper performs an additional epoch of GRPO finetuning and reports 54.2 [email protected]. ActivityNet is zero-shot, reaching 22.1 [email protected], but its [email protected] and [email protected] are not the best results.
• General capabilities are not preserved without any regression: Video-MME falls from 62.7 to 62.4, and LibriSpeech WER rises from 3.2 to 3.5; Daily-Omni improves from 60.1 to 72.3. Temporal training is more beneficial for joint audiovisual reasoning than a claim that every single-modality capability improves would suggest.

Highlights & Insights

• Time acts as a shared interface rather than another perceptual modality that must be learned from scratch. Placing timestamp text beside corresponding audiovisual content exposes grounding information in a form the language model can directly consume.
• Separate annotation matters more than merely including audio. Distinct visual and audio captions allow simultaneous cross-modal associations to be tested without a joint caption hiding which stream supplied the evidence.
• Representation and objectives play complementary roles: TIT exposes time coordinates, SFT teaches event structure, and GRPO refines boundaries and target descriptions. The principle may transfer to other multi-sensor sequences with explicit timelines, provided reliable task-specific rewards are defined.

Limitations & Future Work

• The authors identify real-world interaction and hour-long videos as future work. Current data mainly covers 60–600-second English videos; the effects of fixed 64-frame sampling and audio downsampling on brief events and longer inputs are not isolated experimentally.
• Visually driven segmentation may truncate sound events, and automated captions still contain errors. Independent audio boundaries, audiovisual time-shift perturbations, and cross-language testing would provide stronger evidence than QA generated using a single segmentation policy.
• METEOR is both a reward and a reported captioning metric, so gains may partly reflect adaptation to reference wording. Human temporal verification, event-coverage metrics, or counterfactual time-swapping tests could separate genuine grounding from language matching.
• The available main text does not sufficiently detail how inputs are rearranged without TIT, isolate individual reward components, or report variability across random seeds. Component ablations support usefulness but do not replace budget-matched and mechanism-isolating comparisons.
• The missing appendix and damaged format-reward equation limit reproduction from the available cache. This is a limitation of the material accessible here, not proof that the complete published paper omits those details.

Related Work & Insights

• Versus Ola: ChronusOmni inherits the audiovisual encoders and language backbone, adding temporal organization and temporal-task post-training. Its gains should not be framed as acquiring omni-modal perception from scratch.
• Versus ARC-Hunyuan-Video: That model renders timestamps as frame watermarks, whereas ChronusOmni supplies them directly as text. The distinction is how time enters the model, not simply whether time labels are present.
• Versus LongVALE and TriSense: Joint audiovisual captions suit joint event understanding; ChronusAV's separated captions make A2V and V2A easier to evaluate independently, while still requiring checks against annotation leakage.
• Versus Time-R1: Both exploit reward-based temporal optimization, but ChronusOmni extends the setting to audio grounding and cross-modal captioning. The transferable principle is feedback that reflects continuous temporal error rather than simply increasing QA volume.

Rating

• Novelty: 4/5. Six-way tasks, separated annotation, and temporal interleaving form a clear audiovisual grounding approach, although SFT and GRPO are not new algorithms.
• Experimental Thoroughness: 4/5. Same-data finetuned baselines, cross-dataset evaluation, and three component ablations are included; reward decomposition, budget matching, and variance reporting remain gaps.
• Writing Quality: 3/5. Task definitions and the central argument are clear, but the LongVALE relative gains contain a numerical inconsistency; damaged cached equations are treated separately as a source-material issue.
• Value: 4/5. The work offers a practical temporal interface and diagnostic data for omni-modal models, with deployment value still dependent on longer-video, open-captioning, and real-interaction validation.