Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars¶

Conference: ECCV2026
Paper: ECCV Paper
Full Text: Conference PDF
Area: Video Generation
Keywords: talking avatars, human-object interaction, scene perception, motion residuals, lip synchronization

TL;DR¶

InteractAvatar uses parallel perception-planning and video-synthesis streams to make avatars follow text instructions and interact with objects in the reference scene while speaking to audio, reaching VLM-QA 29.07 and PI 0.850 on GroundInter in its joint text-and-audio generation mode.

Background & Motivation¶

A talking avatar is not necessarily an avatar that can execute an action. Audio-driven models can map pronunciation rhythms to mouth movements and add some upper-body or full-body motion, but audio does not specify which object to pick up, where it is, or how the hand should approach it. Consequently, picking up an apple from a table while speaking requires more than increasing motion amplitude: the action must depend on spatial relationships in the input image. The paper calls this task Grounded Human-Object Interaction (GHOI). Grounding here requires environmental perception, preservation of the initial scene, high-level textual control, and plausible interaction with the designated object.

Providing a skeleton sequence improves control but transfers planning to the user, who must obtain an action and align it with the table, hands, and objects in the reference image. Subject-consistent video models solve a different problem: they can borrow a person's appearance to synthesize a new scene without continuing the world depicted in the reference. Thus, generating an attractive video of someone drinking is different from making the person in the input image pick up the cup already present there. The authors identify scene-action grounding and a conflict between controllability and visual quality as the central difficulties. Text alone may select the wrong target or produce little interaction, while rigid pose constraints can restrict object deformation and contact details.

Instead of asking a single video stream to learn all spatial planning implicitly, the method assigns motion structure to a separate generative stream. However, it does not treat that stream's final output as an immutable condition; the two streams continuously exchange features during generation. The planning branch can therefore learn where the action should go while the video branch retains the ability to synthesize a convincing contact process. Core Idea: teach the motion stream to understand the reference scene through detection-like training, then guide parallel video generation with layer-wise motion residuals instead of a one-time skeleton input.

Method¶

Overall Architecture¶

Inputs are a reference image containing a person and nearby objects, an interaction instruction, and optional speech or external motion controls. The output is an avatar video that preserves the scene; joint generation also produces a structural motion sequence. Motion here is neither a 3D mesh nor a raw coordinate array, but an RGB video containing rendered human skeletons and object bounding boxes. PIM Scene-Aware Planning generates these structural frames, AIM Audio and Motion Control synthesizes the final appearance, and M2V Layer-Wise Residual Alignment connects them. Both modules use similar Diffusion Transformer (DiT) architectures, with a shared pretrained VAE for visual encoding and T5 for text encoding. Solid edges below denote conditioning or layer-level feature transfer, while the dashed edge denotes training supervision; PIM does not finish the entire motion sequence before AIM starts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Reference image + text"] --> PIM["PIM Scene-Aware Planning"]
    TRAIN["Detection / continuation / generation<br/>without initial motion"] -.->|Training supervision only| PIM
    MOTION["Motion noise or clean external motion"] --> PIM
    PIM -->|Corresponding motion-layer features| ALIGN["M2V Layer-Wise Residual Alignment"]
    ALIGN -->|Injection at each layer| AIM["AIM Audio and Motion Control"]
    COND["Reference image + text + video noise<br/>Optional audio"] --> AIM
    AIM --> VIDEO["Talking and interacting video<br/>within the reference scene"]
    PIM --> PLAN["Motion sequence in joint generation"]

Key Designs¶

1. PIM Scene-Aware Planning: make object localization and interaction generation the same kind of visual generation task

Ordinary actions and human-object interactions need different structural information: the former use human poses, while the latter also maintain object-box trajectories. PIM marks this distinction with ACTION and HOI task embeddings, concatenating the task embedding with text embeddings for cross-attention. Rendering people and objects in one RGB motion representation reuses the visual priors of a video foundation model without introducing a separate coordinate-to-pixel interface. Motion frames have a short side of only 256 pixels because they represent locations and structure rather than detailed skin, clothing, or object textures. The reference image is prepended to the motion sequence, but treating it as an ordinary initial motion timestep could let photographic texture and positional encoding interfere with structural generation. Modified rotary position embeddings (RoPE) assign it a temporal position of -1 and adjust its spatial indices, making it a global environmental condition rather than a motion frame to continue. The reference image and initial motion frame therefore serve different purposes: the former describes the scene, while the latter, when supplied, specifies the action's starting configuration.

Simply feeding the reference image into the model does not ensure that it identifies the intended object. The continuation task provides the image, initial motion, and text and asks for subsequent motion, allowing the model to proceed from known poses and boxes. Generation without initial motion masks that frame and predicts the entire structural sequence from the image and text, forcing the model to establish its own starting configuration. Randomly truncating the target to 1 frame creates a detection-like task: output the structure associated with the text-specified person and object without generating later actions. This task still uses flow matching, with loss restricted to the first frame, rather than adding a separate detection head and detection objective. Detection data teach the model to locate the right object, while full-sequence generation without initial motion connects localization to action planning. Table 2 labels the latter bridging configuration Det.+Cont.; it means full-sequence generation without reference motion, not ordinary continuation that always receives the first motion frame.

2. M2V Layer-Wise Residual Alignment: let structural guidance evolve alongside video generation

If PIM is frozen and produces a complete skeleton sequence before AIM is trained to render it, structural control becomes a fixed information bottleneck. Skeletons and rectangular boxes cannot fully describe finger contact, object deformation, or changing occlusion, leaving the renderer to infer details from sparse conditions. InteractAvatar instead keeps corresponding layers in both streams and transfers newly developed motion information from PIM to AIM during joint generation. For noninitial layers, the residual is the difference between consecutive PIM block outputs; the first layer uses its own output as the injected feature. This differs from copying the same last-layer feature into every video block or directly adding complete motion features at every layer. Layer-wise increments provide structural guidance at the corresponding representation depth instead of repeatedly imposing the same global condition. The streams perform coordinated denoising rather than operating as a sequential system in which one finishes before handing its result to the other.

Because motion has a lower resolution than video, M2V first spatially aligns residuals with bilinear interpolation, then projects them through a zero-initialized linear layer before injection into the corresponding AIM block. Zero initialization prevents the additional motion path from abruptly changing the pretrained video model at the beginning of training. The model gradually learns which structural features should control actions, reducing training fluctuations. The authors also associate this design with preventing skeletal-line ghosting in generated frames, a concrete risk when structural and photographic images share a visual representation. Joint training supplies a richer feature channel than a completed explicit motion sequence, which the authors argue benefits hand plausibility and object deformation. It remains a learned video-generation mechanism, not an explicit contact solver or a motion planner with physical guarantees.

3. AIM Audio and Motion Control: establish local speech conditioning before adding global structure

AIM extracts audio features with pretrained Wav2Vec using a contextual window around each video timestep rather than isolated phonetic fragments. Aggregated frame-level features enter the video stream through cross-attention to model coarticulation and temporal relationships. Early audio pretraining also spatially weights the injection output with a face mask, concentrating audio influence on the face to simplify learning. The mask is absent from later training and inference, so it should not be described as a required deployment input. The authors emphasize learning audio before motion because audio is a local, weaker, heterogeneous signal, whereas motion is a global, stronger constraint expressed in the same visual modality as video. Training both from the beginning, or learning motion before adding audio, can make the model depend on motion and neglect lip synchronization. Multimodal competence therefore depends on training order, not merely on adding another encoder.

The framework also accepts external skeletons or interaction motion as driving signals without requiring a separate model. In this case, PIM receives clean motion with diffusion timestep set to 0 and acts as a motion-feature encoder. Features travel through the same layer-wise residual path into AIM, giving external driving and autonomous planning a common interface. The paper specifies a 4:1 data ratio between joint generation and external driving in its discussion of their co-training. This explains how one model can either plan its own actions or follow supplied motion, rather than treating every mode as text generation. Combining text, audio, and motion changes inference conditions; the multiple Ours rows in the main table are not independently trained models.

A Worked Example¶

Consider the paper's example of picking up an apple from the table and displaying it: the reference image fixes the initial person, table, and apple locations. PIM must ground the apple mentioned in the instruction to the actual image object rather than create an apple elsewhere. In autonomous generation, the skeleton reaches, grasps, and lifts while the object-box trajectory changes together with the hand position. This illustrates the workflow without introducing frame counts, grasp timings, or trajectory coordinates not specified by the paper. AIM synthesizes appearance in parallel with the structural sequence, receiving motion residuals throughout its layers to fill in grasp appearance, occlusion, and object surfaces. When speech is supplied, audio cross-attention governs speaking rhythms while text and scene conditions continue to determine object interaction. If the user instead supplies complete driving motion, PIM takes the timestep-0 encoding path rather than treating the input as noisy motion to generate.

Loss & Training¶

Both streams use unified flow matching: visual inputs become VAE latents, noise is added, and the network learns the corresponding vector field. Single-frame detection-like supervision and motion-sequence supervision share this objective, changing only conditions, target length, and the region over which loss is computed. Equations (1), (4), (6), and (7) have damaged operators or indices in the text extraction; this note explains the surrounding prose without inventing exact author equations. Training uses SpeakerVid-5M, OpenHumanVid, and HOIGen-1M, with Gemini-2.5 Pro for detailed action and interaction captions, DWPose for skeletons, and DINO for detection. PIM and AIM are initialized from wan2.2-5B but serve distinct objectives rather than sharing all parameters. PIM is pretrained for 30,000 steps on 3–10-second structural clips with a 256-pixel short side; AIM receives 5,000 pretraining steps on 3–6-second videos with a 704-pixel short side. Both use a pretraining learning rate of \(10^{-5}\), followed by 4,000 joint fine-tuning steps at \(2\times10^{-6}\). Final training retains image-to-video samples to preserve identity, dynamics, and general generative priors rather than overfitting to low-motion talking patterns. Section 3.3 separately specifies 30% audio-conditioned samples, 15% ground-truth-motion-conditioned samples, and 60% joint video-motion generation samples. These sum to 105% and are not a clearly explained mutually exclusive partition corresponding to the earlier 4:1 ratio; the text does not fully specify overlapping sampling, so they should not be normalized into an assumed recipe.

Key Experimental Results¶

Main Results¶

GroundInter uses 400 reference images generated with jimeng4.0, covering 100 common objects, with 1–3 objects and 1–3 action descriptions per image, yielding 600 test cases. CosyVoice synthesizes speech from dialogue scripts, and annotations include object masks, detection results, and human keypoints. VLM-QA uses 30 binary questions about objects, humans, and interactions; the reported values follow this question-based scoring scheme and should not be rewritten as percentage success rates. HQ multiplies hand dynamics by Laplacian sharpness, while OQ multiplies object dynamics by DINO consistency; the definitions can be summarized as:

\[ \operatorname{HQ}=\operatorname{HandDynamics}\cdot\operatorname{HandSharpness},\qquad \operatorname{OQ}=\operatorname{ObjectDynamics}\cdot\operatorname{ObjectDINOConsistency}. \]

PI checks contact between detected object boxes and human keypoints; its detailed aggregation protocol is absent from the supplied main text, and it is not a 3D-verified physical-contact rate. CLIPre compares the original prompt with a VLM-generated video caption, DINOref measures reference-image preservation, and Syncconf measures audiovisual synchronization confidence. The following selection is from Table 1, page 10; higher is better for every column, but task groups have different input conditions and do not constitute a strictly matched-input ranking.

Mode / Method	VLM-QA	HQ	OQ	PI	DINOref	Syncconf
Audio-driven / Wan-S2V	24.65	0.336	0.063	0.619	0.870	5.43
Audio-driven / Ours (TA2V)	27.32	0.931	0.133	0.803	0.857	6.04
Text-driven / VACE	26.74	0.908	0.118	0.705	0.817	Not applicable
Text-driven / Ours (T2MV)	29.05	0.975	0.150	0.852	0.835	Not applicable
Text and audio / Ours (TA2MV)	29.07	0.973	0.147	0.850	0.839	5.92

T2MV uses self-generated motion without speech; TA2MV combines speech with joint motion generation, while TA2V is the mode placed in the paper's audio-driven group. For example, TA2V raises PI from Wan-S2V's 0.619 to 0.803 but lowers DINOref from 0.870 to 0.857, so improvement is not universal across quality dimensions. The authors also evaluate 50 real-scene cases: Table 5, page 14, reports VLM-QA 28.49 and PI 0.794 for Ours, versus 26.23 and 0.781 for HuMo.

Ablation Study¶

The next table selects PIM and connection-design ablations from the upper part of Table 3, page 13; this ablation setting should not be conflated with the inference-mode results in Table 1. Cascade uses a separately trained and frozen PIM, Last-layer supplies the last-layer feature to every AIM block, and Addition is the layer-wise addition baseline.

Config	VLM-QA	HQ	OQ	PI	Syncconf
w/o PIM	26.19	0.711	0.104	0.685	5.41
Cascade	28.19	0.876	0.137	0.746	5.27
Last-layer	28.31	0.880	0.141	0.769	5.32
Addition	28.46	0.903	0.134	0.777	5.14
Ours	28.89	0.925	0.144	0.780	5.43

Removing PIM lowers VLM-QA from 28.89 to 26.19; replacing parallel coupling with Cascade lowers HQ from 0.925 to 0.876, indicating that both planning and its information-transfer mechanism matter. The following training-order ablation comes from the lower part of the same table; A denotes audio, M motion, and I retained image-to-video training samples.

Training Config	VLM-QA	HQ	PI	Syncconf
AM	28.23	0.912	0.770	4.23
M → AM	28.84	0.920	0.778	3.98
A → AM	28.02	0.906	0.755	5.49
AI → IAM (Ours)	28.89	0.925	0.780	5.43

Key Findings¶

Learning audio before adding motion produces Syncconf 5.49, versus 3.98 when motion is learned first, supporting the argument that weaker conditions need an established pathway.
The full strategy's Syncconf of 5.43 is slightly below A → AM at 5.49, but VLM-QA, HQ, and PI are higher, revealing a trade-off across objectives.
The full perception-training configuration in Table 2, page 12, reaches VLM-QA 28.89 and PI 0.780; detection and generation without initial motion support scene grounding rather than merely sharpening frames.

Highlights & Insights¶

Detection can be trained as single-frame visual generation. Understanding the initial layout and continuing an action then share representations and objectives instead of relying on separately optimized localization and generation modules.
RGB motion is useful not only for visualization but for directly reusing video-model priors. The resulting risk of skeletal graphics leaking into appearance is specifically addressed by the zero-initialized residual pathway.
The dual-stream benefit involves continuous intermediate-feature exchange, not merely extra parameters. Cascade and last-layer controls show that when and where features enter the rendering stream also affect interaction quality.
Multimodal training has an order-dependent effect. A transferable hypothesis is to establish useful gradients from weaker local conditions before adding strong structural controls, although its generality requires testing in other tasks.

Limitations & Future Work¶

The authors explicitly limit the method to single-person scenes and do not design it for interactions involving multiple people; the results do not establish collaborative manipulation or object handovers.
The main benchmark uses generated reference images, with only 50 additional real-scene cases. Coverage of real occlusions, complex backgrounds, and long-tail objects remains limited.
HQ, OQ, and PI are visual proxies affected by motion level, detection errors, and keypoint errors. Independent evaluation of contact timing, object permanence, and geometric consistency is a reader-proposed next step.
Two streams increase parameter count, while inference-time comparisons are relegated to supplementary material; the supplied main text does not support precise claims about speed, memory, or real-time operation.
Damaged equations, unclear relationships between sampling ratios, and incomplete metric aggregation details limit reproduction from the main text alone; this note does not turn these uncertainties into assumed facts.

vs Wan-S2V / HY-Video-Avatar: these audio-driven methods prioritize speech animation and appearance preservation; this work additionally requires text-directed hand-object actions within the reference scene, placing greater weight on dynamic interaction.
vs UniAnimate-DiT: the paper supplies PIM-generated motion to this pose-driven baseline; the proposed model still benefits from joint feature learning, so the distinction is not simply the availability of skeleton control.
vs HuMo / HunyuanCustom: subject-appearance consistency and scene continuity are different constraints. The latter is part of this task definition rather than treating any interaction video with the same person as a success.
vs VideoJAM: the method follows joint appearance-motion modeling but extends structure to human skeletons and object trajectories and adds detection-like training for scene grounding.
Research implication: before changing the low-level generator, examine whether its control representation explicitly contains the target object, initial state, and temporal evolution; this is a reader interpretation, not an additional experimental result.

Rating¶

Novelty: 4/5. Detection-like motion planning and layer-wise dual-stream alignment provide a task-specific design rather than simply concatenating existing controls.
Experimental Thoroughness: 4/5. Comparisons cover multiple baseline families, connection mechanisms, training order, and real scenes, but real-world scale and efficiency evidence remain limited.
Writing Quality: 4/5. Task boundaries and module responsibilities are clear, while sampling details remain ambiguous; extraction damage is a separate source-readability limitation.
Value: 4/5. The representation and training strategy are reusable for controllable interactive avatars, but do not yet establish multiperson or physically reliable interaction.