OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Dataset: OmniHuman
Area: Video Generation
Keywords: audio-video joint generation, hierarchical annotation, identity binding, human interaction, perceptual evaluation

TL;DR¶

OmniHuman organizes 1 million human-centric videos into scene-, interaction-, and individual-level supervision, paired with OHBench for diagnosis at the same levels; fine-tuning LTX-2 on 20% of the data improves audiovisual alignment and most human-centric metrics, but not every metric.

Background & Motivation¶

Audio-video joint generation can place appearance, speech, and motion in the same clip, but a believable human activity requires more than clear images. Maintaining a face in a distant shot, assigning alternating dialogue to the correct speaker, and preserving continuous hand-object contact are not determined by single-frame aesthetics. Traditional face and talking-head datasets emphasize clear frontal faces and lip movements, with less coverage of environmental acoustics, complex camera motion, and precise manual operations. OpenHumanVid and SpeakerVid-5M broaden human-centric data, but the authors argue that fine-grained supervision still fails to jointly align scenes, interactions, and individual attributes. The proposed bottleneck is incomplete data structure rather than simply insufficient video volume, motivating a dataset contribution instead of a new generation network.

Evaluation has a corresponding blind spot: a clip may look attractive while a silent listener keeps moving their lips or two faces gradually blend together. An overall quality score cannot readily distinguish failures in environmental sound, speaker assignment, and physical relationships between people and objects. The authors therefore place scenes and cameras at the global level, person-person and person-object behavior at the interaction level, and identity, anatomy, lips, and speech at the individual level. The same hierarchy guides what to collect and what to inspect after generation, matching the granularity of supervision and diagnosis. This design does not establish missing data as the cause of every failure; the fine-tuning experiment tests the complete dataset on one backbone.

Existing detection, tracking, source separation, and multimodal models make large-scale automatic annotation practical. Instead of asking one multimodal model to freely describe everything, the pipeline first establishes identity tracks, transcripts, and synchronization relationships, then incorporates these constraints into captions. This specifically addresses cross-modal errors in which a description sounds plausible but attributes speech to the wrong visible person. Core Idea: construct hierarchical human-centric data using checkable identity and audiovisual correspondences, then evaluate scene quality, interaction realism, and individual fidelity at matching levels.

Method¶

Overall Architecture¶

The input consists of raw videos and audio tracks from open-world scenes; the outputs are audiovisual samples with spatiotemporal tracks and semantic descriptions, plus an evaluation protocol, not a new generator. The data pipeline cleans clips, tracks people, estimates poses, binds speech segments to visual identities, and finally generates hierarchical captions with identity anchors. Samples retain scene and background sound information, frame-level positions and poses, and each person's attributes and speech, allowing downstream tasks to select appropriate fields. OHBench returns multiple scores across global, interaction, and individual levels rather than only an aggregate ranking that obscures failure sources. The training validation fine-tunes existing LTX-2 on these data; inference evaluation runs generation models and diagnoses their resulting video and audio. Qwen3-Omni in annotation and Gemini in evaluation are processing and assessment tools, not newly added branches of LTX-2.

Key Designs¶

1. Progressive audiovisual cleaning: establish usable signals before judging content quality

TransNetV2 first segments videos at shot boundaries, reducing abrupt scene changes within a training sample. OCR and logo detection locate subtitles, watermarks, and other contamination, followed by subject-centered cropping rather than directly training on contaminated clips. Audio checks reject missing tracks, abnormal durations, high silence ratios, and low-volume samples; video and audio are standardized to 30 FPS and 44.1 kHz. Standardization facilitates temporal alignment, but matching sample formats alone does not guarantee lip synchronization, which is checked separately. Demucs performs four-source separation, keeping vocals as the target track for speech analysis and mixing the remaining tracks into background audio. Environmental sound is therefore not simply discarded as noise, because its correspondence with human activity is part of the intended supervision.

Visual filtering proceeds from lightweight to heavier analysis: frame-level CLIP aesthetic scores, residual text and watermark checks, then DOVER assessment of composition and clarity. UniMatch optical flow rejects static, excessively shaky, or abnormal motion, preventing visually clear but dynamically uninformative clips from dominating training. Perceptual hashing and SigLIP cosine similarity check temporal semantic continuity, addressing discontinuities missed by framewise quality assessment. Each stage rejects samples, so the final distribution also inherits preferences of these upstream models. The main text does not provide every filtering threshold or stagewise retention ratio, preventing exact reproduction of individual inclusion decisions. These operations are described in Section 3.1, pages 5–6.

2. Identity and audiovisual binding: connect visible people to active speakers

YOLOv11 detects people, NMS refines detections, and MOTRv2 associates them across frames to produce bounding-box and confidence trajectories. Tracking tolerates a maximum loss span of 5 frames to handle brief occlusions without immediately resetting identity. DWPose-L and dedicated hand processing estimate 134 whole-body keypoints per track, covering the body, face, feet, and both hands. Keypoints retain two-dimensional coordinates and confidence; the extracted keypoint formula is damaged, so this note does not reconstruct it as an exact author equation. The pipeline measures facial clarity through Laplacian variance and rejects videos with persistently blurry faces. It selects the frame with the highest average facial-keypoint confidence as a reference face and associates identities when ArcFace embedding similarity exceeds 0.55. This describes within-video identity correspondence, not a complete explanation of how all 80,000 identities were deduplicated across videos.

On the audio side, 3D-Speaker produces speech intervals with speaker indices, but acoustic clustering alone cannot identify the corresponding visible face. SyncNet therefore compares face-box sequences with speech intervals, and greedy matching assigns each audio segment to the visual track with the highest synchronization response. Retention additionally requires a synchronization score above a preset threshold and a temporal offset within 3 frames; the main text does not specify that score threshold numerically. Audio without a matched visual subject remains background audio, avoiding forced assignment of voice-over speech to a visible person. FunASR-Nano then transcribes retained speech segments, associating each person with explicit speaking times and content. The resulting supervision connects continuous body trajectories with visually identified speech intervals, distinguishing an active speaker from a silent listener. This step appears in Sections 3.2–3.3, pages 6–7.

3. Identity-aware hierarchical captioning: constrain long descriptions with structured evidence

The first Qwen3-Omni stage extracts global video type, shot scale, camera motion, background, and lighting, alongside each person's appearance, actions, and expressions. Audio descriptions include timestamped speech and emotion, music type and relative volume, and textual descriptions of sound events. The annotation enforces mutually exclusive speech and music categories and removes repeated speech or lyrics to reduce conflicting supervision; this is not a claim that real audio never contains both. Reference faces and person IDs require the multimodal model to associate descriptions with specific subjects rather than ambiguous pronouns alone. Only the second stage assembles these fragments into a long-form caption, combining scene, human, and audio information about the same event. To prevent invented dialogue, it inserts [speech_m] and [lyrics_m] placeholders, then replaces them with transcribed content from the first stage.

Identity insertion also has two steps: generate a fluent caption, then append the corresponding REF identity anchor to nouns or pronouns referring to each person. The target is not generic fluency but ownership of actions, appearance, and dialogue in multi-person sentences. Before retaining a clip, the pipeline checks that structured subject counts match tracking outputs and that speaker counts and transcripts remain within an acceptable edit-distance margin. These checks reduce internal contradictions but cannot eliminate upstream errors: a placeholder faithfully inserts an incorrect transcript if ASR has already misrecognized speech. Each person ultimately has an identity, boxes, poses, attributes, speech metadata, and descriptions, forming structured supervision for downstream tasks. This mechanism is described in Section 3.4, pages 7–8.

4. Hierarchical perceptual evaluation: separate interaction failures from overall image quality

The OHBench text lists 331 single-person, 128 two-person, and 20 person-object interaction videos, claiming domain gaps relative to training data. This selection aims to assess generalization, but the main text does not fully operationalize the domain gaps or train-test isolation procedure. At the global level, IQ uses MUSIQ for imaging distortions, while DD uses RAFT flow for motion intensity; neither alone determines whether behavior is natural. Gemini rates background plausibility BP from 1–10, considering stability in static shots and parallax under moving cameras. Audio aesthetics AbS averages four Audiobox dimensions: content enjoyment, usefulness, production complexity, and production quality. KL and FD measure distributional and perceptual differences between generated and real audio, respectively, with lower values preferred; T-A uses CLAP for audio-text semantic similarity. ImageBind evaluates audiovisual event synchronization, called A-V in the prose and V-A in the tables; the complete conversion from temporal offset to the reported score is not specified.

At the interaction level, Gemini rates interaction naturalness IN, listener realism LR, and emotion similarity ES on 1–10 scales. LR checks both correct speaker assignment and natural silence from the listener, so it should not be interpreted as expression quality alone. Dual-person identity consistency IC* averages ArcFace cosine similarities between each generated face and its corresponding ground-truth reference. Person-object evaluation separately measures object consistency OC and contact naturalness CN, examining state continuity and the spatial, temporal, and physical plausibility of contact. At the individual level, ArcFace measures identity fidelity IC, Gemini assesses attribute consistency AC, and SyncNet measures lip synchronization Sync. Speech quality SQ uses the DNSMOS OVRL score, while WER compares SenseVoice transcripts against target text, with lower word error rates preferred. These evaluators have different scales and meanings: high DD does not imply better interaction, and an aggregate score should not be invented without defined weights. Definitions appear in Sections 4.1–4.3, pages 9–11; the authors place human-agreement evidence in supplementary material, so reading the main text alone does not independently verify that claim.

Loss & Training¶

The paper introduces no new generation loss; its utility experiment randomly samples 180,000 single-person and 20,000 two-person clips from OmniHuman to fine-tune LTX-2. The resulting 200,000 samples are 20% of the 1 million clips, testing the dataset as a whole rather than ablating individual annotation components. Section 5.3 does not report a complete learning rate, training schedule, optimizer, or parameter-update scope, and this note does not invent them. I2AV evaluation targets 5-second outputs, using the closest supported duration when necessary; all models use the available resolution closest to 720P. These are evaluation inference settings, not proof that all fine-tuning samples share that duration or resolution.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 2, page 11, for I2AV under the approximately 5-second, approximately 720P evaluation conditions above. It covers image quality, motion, audio, and alignment, retaining one closed-source reference and an open-source reference with strong T-A performance.

Model	IQ ↑	DD ↑	AbS ↑	KL ↓	FD ↓	T-A ↑	V-A ↑
Veo3	0.737	0.878	5.40	0.79	0.87	0.39	0.37
Ovi	0.701	0.691	3.75	0.97	0.90	0.40	0.23
LTX-2	0.720	0.601	3.69	1.04	1.06	0.28	0.27
LTX-ft	0.721	0.665	4.13	0.77	0.93	0.35	0.30

KL falls from 1.04 to 0.77, a relative improvement reported as 25.9%; improvement here means reduced discrepancy, not increased KL. T-A rises from 0.28 to 0.35, a 25.0% relative increase, although the fine-tuned model remains below Ovi at 0.40. IQ only changes from 0.720 to 0.721, indicating that gains are more visible in motion, sound, and cross-modal relationships than in static clarity. The table retains the label Veo3; Section 5.1 instead says Veo3.1, a version-label inconsistency that should not be silently resolved.

Ablation Study¶

The main text contains no component-removal experiment; the following before/after fine-tuning analysis is drawn from Table 3, page 12, also for I2AV. Dual-person and object-interaction metrics apply to their corresponding subsets, so their averages do not imply uniform improvements across every scene type.

Config	IC* ↑	IN ↑	LR ↑	OC ↑	CN ↑	IC ↑	Sync ↑	SQ ↑	WER ↓
LTX-2	0.562	6.47	8.94	6.95	6.98	0.656	6.91	2.89	0.29
LTX-ft	0.589	6.53	8.92	7.12	7.32	0.696	7.11	2.98	0.27

CN rises from 6.98 to 7.32, a reported relative improvement of 4.9%, supporting improved contact modeling. IC increases from 0.656 to 0.696, a reported relative improvement of 6.1%; dual-person IC* also rises from 0.562 to 0.589. However, LR falls from 8.94 to 8.92, contradicting the statement in Section 5.3 that every metric improves. The source table labels LR as -0.1%, which also differs from approximately -0.22% calculated from the displayed values; the raw values are retained without treating that percentage as verified.

Key Findings¶

Disaggregated metrics are more diagnostic than overall rankings: LTX-2 already has strong LR, while its audio quality and T-A still benefit from data fine-tuning.
Table 4, page 12, additionally evaluates S2V, showing that OHBench can select metrics for different generation tasks; it does not establish full validation of every claimed supported task.
Section 5.2 reports more facial and mouth distortion in long shots and greater identity drift in dual-person scenes, but the main text provides no corresponding stratified error rates.

Highlights & Insights¶

The most reusable principle is to bind identities before composing captions. Anchoring actions and dialogue to the same person directly targets attribution errors more than simply making generic captions longer.
Placeholders separate narrative organization from exact transcription. The language model can organize a description without being allowed to reinvent dialogue from context.
Matching the dataset and evaluation hierarchy connects what supervision to add with what improves afterward; this is a design insight, not an established causal attribution to individual components.

Limitations & Future Work¶

The data emphasize single-person, two-person, and English content, with English reportedly accounting for up to 80%; conclusions do not directly extend to group interaction or broad multilingual coverage.
Person-object evaluation contains only 20 clips, limiting rare-tool and complex-occlusion coverage, and confidence intervals are not reported.
Automatic annotations and model judges can inherit upstream biases, particularly in ASR, identity recognition, and social judgments; independent human audits and subgroup error analysis are needed.
Figure 4 labels single-person, two-person, and person-object proportions as 60.0%, 30.0%, and 10.0%, inconsistent with the textual counts of 331, 128, and 20; this note does not silently correct them.
Section 3.5 claims all clips are at least high definition, yet Figure 3 includes a <720P category; its duration and resolution subplot references are also swapped, preventing a firm resolution-floor claim.
Controlled ablations of hierarchical annotations, filtering, and data scale are missing, so the source of gains remains unresolved; future comparisons should fix training budgets and include more backbones.
Human-agreement evidence is deferred to supplementary material and was not verified here; dataset licensing, participant consent, and actual online availability were also not checked through network access.

Compared with OpenHumanVid and SpeakerVid-5M: the emphasis is joint coverage of environmental sound, person-object interaction, and identity-aware captions, rather than only more portrait or speech clips.
Compared with VBench and audiovisual benchmarks: OHBench retains useful image and motion metrics while adding listener behavior, contact naturalness, and background plausibility, avoiding an equation of aesthetics with believable behavior.
Compared with LTX-2 and Ovi: these are evaluated generators, whereas OmniHuman contributes data and evaluation resources; LTX-ft should not be described as a new model trained from scratch.
Research direction: audit correct speech content separately from correct speaker assignment, then run fixed-budget additions of distant-shot and dual-person data to test whether missing coverage causes the observed errors.

Rating¶

Novelty: 4/5. The main contribution is unified hierarchical data and interaction diagnosis rather than a new architecture.
Experimental Thoroughness: 3/5. Coverage includes many models and fine-tuning, but lacks component ablations, uncertainty intervals, and complete training details.
Writing Quality: 3/5. The pipeline is clear, but statistics, version labels, and the claim of universal improvement contain inconsistencies.
Value: 4/5. The work offers concrete data organization and failure-localizing evaluation dimensions for human-centric audiovisual generation.