3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement

Conference: ECCV2026
Paper: Official paper page · PDF
Project: https://robinhood256100.github.io/web-disp
Area: 3D Vision / Human Image Animation
Keywords: ground adaptation, SMPL-X, motion retargeting, visibility masks, camera trajectories

TL;DR

3STC-HIA adjusts human altitude and orientation along a user-defined path in a unified 3D scene, then guides a pretrained video model with visible scene point clouds for training-free human and camera control, reducing Trajectory100 translation error from RealisMotion's 1.305 m to 0.478 m without a corresponding improvement in yaw accuracy.

Background & Motivation

Human image animation must do more than make a reference subject perform an action: it also needs to place that subject in a scene and determine where the camera observes it. Early 2D skeleton conditions transfer framewise poses effectively but do not explicitly describe depth or the ground surface. When a person moves up a slope or the camera circles the scene, skeletons, appearance, and background can consequently disagree. Introducing 3D human representations such as SMPL-X clarifies body structure, but a plausible body does not automatically imply plausible contact with its environment.

Uni3C places humans and point clouds in a shared space for viewpoint control, while RealisMotion further separates local actions from global motion paths. This paper targets a situation that neither fully addresses: a user draws a top-view path, expects the person to adapt automatically to terrain elevation, and asks the camera to follow a separate trajectory. Treating the drawn path as displacement on a horizontal plane can cause floating or ground penetration. Simply supplying rendered scenery as an ordinary condition can also be overruled by an image-to-video model's dependence on its reference frame, suppressing the intended perspective changes.

The objective is therefore not to train a larger action model, but to make geometric controls constrain an existing generator at inference time. Core idea: retarget human motion using local ground elevation, inject point-cloud latents only where scene geometry is visible from the current viewpoint, and weaken this constraint during denoising so that the path, terrain, and camera jointly shape the generated video.

Method

Overall Architecture

The inputs are a reference image containing a person and a scene, an SMPL-X action sequence, a user-drawn 2D motion path, and a camera control signal consisting of intrinsics and per-frame extrinsics. The output shows the reference subject moving along the requested path under the specified camera movement. The paper uses action for local posture and motion for the combination of posture, global displacement, and direction. This distinction explains how the same running action can be reused along different routes.

After reconstructing and aligning the human and scene in 3D, ground-adaptive motion retargeting determines the person's position and orientation at each frame. The system renders human and scene conditions along the camera path, uses a viewpoint-adaptive visibility mask to select supported scene regions, and applies decaying scene latent fusion throughout generation to inject their geometric information. The video backbone is pretrained RealisDance-DiT.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Reference image + actions<br/>Human path + camera path"] --> Align["Human and scene<br/>3D reconstruction and alignment"]
    Align --> Retarget["Ground-adaptive<br/>motion retargeting"]
    Retarget --> Render["Render human and scene<br/>along the camera path"]
    Render --> Mask["Viewpoint-adaptive<br/>visibility mask"]
    Mask --> Fusion["Decaying<br/>scene latent fusion"]
    Render --> Fusion
    Fusion --> Output["Pretrained video model<br/>completes denoising and decoding"]

The 3D preprocessing is a prerequisite for control, not a newly trained network contribution. Depth-Pro estimates monocular depth; the paper also uses SfM annotations or multi-view stereo reconstruction for metric alignment before constructing the scene point cloud. A single reference image should therefore not be interpreted as a guarantee that all geometric preparation is independent of additional scale information. Source actions come from video motion recovery or a text-to-motion model. Corresponding body keypoints provide a least-squares rigid alignment to the scene, and GeoCalib subsequently calibrates the gravity direction.

Key Designs

1. Ground-adaptive motion retargeting: change the route while preserving the action's vertical dynamics

The user draws a curve in the world's X-Z top-view plane. The system samples it uniformly, associates the samples with action frames, and substitutes these positions for the pelvis's horizontal coordinates. Vertical placement cannot simply remain fixed: at every path location, the system queries a cylindrical point-cloud neighborhood with a radius of 0.15 m, selects the lowest 5% of points by altitude, and averages their Y coordinates to estimate local ground height. Selecting low points is intended to reduce interference from people or objects above the ground, but it remains a geometric heuristic rather than a semantically verified ground estimate.

The estimated ground height does not directly replace pelvis height. Instead, the system adds the difference between the current ground elevation and the first frame's ground elevation to the original action's per-frame pelvis height. Terrain-induced elevation and the action's own vertical dynamics can thus coexist, rather than flattening running or jumping into constant-height movement. Orientation comes from the forward tangent of the updated 3D trajectory: its X and Z components determine yaw about the Y axis, and a Gaussian kernel with standard deviation 3 smooths the sequence over time. The method adjusts altitude and yaw; it should not be interpreted as providing ankle inverse kinematics, full-body tilt adaptation to slopes, or rigorous contact dynamics.

2. Viewpoint-adaptive visibility mask: fuse only scene evidence supported by the current view

The camera trajectory changes both the projected body and the visible scene points. The system renders the human mesh and scene point cloud using the same camera parameters, obtaining their images and per-pixel depths to the nearest visible surfaces. Missing visible points receive infinite depth. These depth maps define a binary mask separating visible scene regions from occluded or human-occupied regions, which is then reshaped for latent-space use. The cached piecewise mask equation is incomplete, so its exact inequalities and boundary conditions are not reconstructed here.

Unlike a fixed background mask taken from the reference image, this mask must be evaluated for each frame as the camera and person move. It prevents scene information from indiscriminately overwriting the animated subject while preserving viewpoint-specific background geometry where point-cloud evidence exists. The scene point cloud still contains some reconstructed points belonging to the reference subject. The authors cite Uni3C to argue that their impact can be neglected, but this paper does not provide a dedicated quantitative ablation of that factor. The prior should therefore not be described as an entirely clean static background.

3. Decaying scene latent fusion: establish perspective first, then release appearance details

Wan-VAE encodes the rendered human, rendered scene, and reference image, while umT5 encodes the text prompt to condition RealisDance-DiT. Beyond concatenating conditions, the method intervenes in the sampling process: the rendered scene latent is forward-noised to the corresponding noise level of the current generated latent and fused with the model prediction through the visibility mask. Matching noise levels matters because mixing a clean scene encoding directly into a highly noisy state would combine representations from incompatible stages of generation.

The fusion adds constraints in visible scene regions while leaving other regions to the generator. Its strength decreases over denoising: early steps emphasize the background perspective associated with camera displacement, whereas later steps allow natural human appearance and finer details to emerge. The weight equation remains readable in the cache:

\[ \lambda(t)=\lambda_{\max}\exp\bigl(\beta(t-T)\bigr). \]

Sampling proceeds from larger to smaller timesteps, so positive \(\beta\) makes the weight decrease. The paper gives \(\lambda_{\max}\in[0.7,0.95]\) and typically about 0.1 to 0.3 for \(\beta\) in an example with \(T=50\) steps. These are reported ranges, not universal optima established by a full sensitivity study. The cached fusion update itself is corrupted; the mechanism is described from the surrounding prose without inventing a latent update equation.

A Worked Example

Consider the paper's illustrated case of a person running along a slope while the camera moves. The running action is first aligned to the reference scene, and the user's top-view curve determines the pelvis's X-Z position at each frame. At each path point, the system queries the lowest 5% of points within a 0.15 m neighborhood to update ground height, adds the terrain elevation relative to the starting point to the original running action, and adjusts orientation along the path tangent.

The human and scene are then rendered from the requested camera positions, and background visibility is recomputed. Early denoising injects scene latents from these views in the visible regions, allowing the background to change with the camera instead of remaining locked to the first frame. Later steps reduce the injection strength so that the generator can complete clothing, body appearance, and motion details. This example explains how the modules connect; it does not imply that the paper reports additional slope-angle measurements or contact-error numbers.

Loss & Training

3STC-HIA is a training-free inference-time guidance method. It introduces no new training loss and does not fine-tune a separate module for each control task. Training-free refers only to the proposed control procedure: Depth-Pro, the motion recovery models, Wan-VAE, and RealisDance-DiT rely on existing training.

The backbone uses Wan2.1's DiT and Flow Matching formulation. The paper illustrates sampling with classical DDIM, but its update equation is corrupted in the cache and cannot establish the full implemented sampler configuration. The available text does not provide sufficient resolution, runtime, memory, or random-seed information to reproduce deployment cost or stability statistics.

Key Experimental Results

Main Results

RealisDance-Val and Trajectory100 each contain 100 video clips and use the first frame as the reference image. The former evaluates video quality. The latter evaluates trajectories recovered from generated videos after alignment to ground-truth trajectories. TransError is the average per-frame Euclidean distance in m; RotError is the average per-frame absolute yaw difference in degrees.

The following table selects the main comparators and metrics from the paper's Table 1. Higher background consistency, motion smoothness, and aesthetic quality are better; lower FID/FVD are better. Methods support different control signals, so this comparison does not establish universal superiority under equally demanding tasks.

Method	Background consistency ↑	Motion smoothness ↑	Aesthetic quality ↑	FID ↓	FVD ↓
RealisDance	95.97	98.59	57.35	25.13	305.84
RealisMotion	95.87	97.61	56.92	24.10	519.66
Uni3C	93.47	97.95	57.61	23.86	516.36
3STC-HIA	96.08	98.83	59.47	27.92	573.41

The paper's Table 2 specifically evaluates human trajectory control on Trajectory100:

Method	Translation error / m ↓	Yaw error / degrees ↓
Tora	5.923	0.342
RealisDance-DiT	1.815	0.158
RealisMotion	1.305	0.095
3STC-HIA	0.478	0.162

Ablation Study

The following columns come independently from the paper's Table 3. Its own full-model baseline is preserved rather than mixed with the main results.

Configuration	Background consistency ↑	Motion smoothness ↑	Aesthetic quality ↑	FID ↓	FVD ↓
Without ground-adaptive retargeting	94.82	93.28	56.42	41.65	685.42
Without viewpoint-adaptive guidance	90.23	96.35	53.28	58.47	832.15
Full model	95.64	97.42	57.86	32.78	521.37

Source inconsistency: Tables 1 and 3 both identify RealisDance-Val, but their full-model background consistency values are 96.08 and 95.64, and their FVD values are 573.41 and 521.37, respectively. The available cache does not explain this discrepancy. Ablation effects must therefore be computed within Table 3, not by subtracting its component-removal results from Table 1's full model.

Key Findings

• Relative to RealisMotion, translation error decreases by 0.827 m, but yaw error increases by 0.067 degrees. The evidence supports improved position control, not a simultaneous lead in orientation accuracy.
• Within the ablation table, removing ground-adaptive retargeting lowers motion smoothness by 4.14. Removing viewpoint-adaptive guidance lowers background consistency by 5.41 and increases FVD by 310.78. The respective effects are more concentrated on human motion and scene stability.
• The method does not win every reported main metric: its FID/FVD are worse than those of RealisDance, RealisMotion, and Uni3C. The authors attribute this to different priorities between control fidelity and distribution matching, but no additional experiment directly verifies that explanation.
• Different control combinations are mainly illustrated qualitatively in Figure 8. The cache provides neither camera trajectory errors nor an independent quantitative table for each combination, so flexible composition is not equivalent to proven absence of interference.

Highlights & Insights

• Height correction uses the terrain elevation relative to the starting point instead of replacing the action's absolute height. This preserves source vertical dynamics and offers a lightweight geometric baseline for transferring actions to new terrain.
• Visibility determines where constraints should apply, while the decay weight determines when they should be strong. Separating spatial support from generation time is better matched to viewpoint changes than continuously locking the entire background.
• Rendering converts 3D controls into conditions that a video model can consume, with additional guidance supplied in latent space. This gives a concrete way to reuse pretrained 2D generative capabilities, although the outcome still depends on upstream geometry quality.

Limitations & Future Work

• The authors identify complex human-scene interactions and coordinated multi-person motion as future work. Current results should not be extrapolated to mutual occlusion, contact, or shared-object manipulation among multiple people.
• Mechanistically, the lowest-5% height heuristic may be affected by low outliers, sparse point clouds, layered surfaces, or missing ground. The paper does not report how empty neighborhoods are handled, nor does it quantify foot contact, penetration depth, or foot sliding.
• Geometric preparation depends on scale alignment, body-keypoint matching, and gravity calibration. Errors can propagate into trajectories and masks, yet sensitivity to monocular depth and scale errors is not systematically evaluated.
• Main and ablation full-model results disagree, and confidence intervals, runtime costs, and detailed camera-control errors are missing. Follow-up work should first clarify the evaluation protocol and then add separate terrain-contact and camera-trajectory metrics to distinguish plausible appearance from strict geometric compliance.

Related Work & Insights

• Compared with RealisDance-DiT: this is the reused video backbone that supplies action-driven animation. The contribution lies in scene-aligned motion retargeting and inference-time visible-scene guidance, not a new video foundation model.
• Compared with RealisMotion: both separate actions from global motion paths, while this method additionally incorporates local terrain elevation and user-defined camera trajectories. The translation-error advantage is clear, but yaw accuracy is weaker than RealisMotion's.
• Compared with Uni3C: both exploit human and scene priors in a unified 3D space. This paper adds ground-adaptive human paths and viewpoint-visible scene fusion to the control pipeline. Uni3C has better main-table FID/FVD, illustrating that broader control support does not automatically improve every quality metric.
• Compared with WORLD-FORGE: the paper follows the idea of exploiting video-model priors through training-free geometric guidance. A transferable strategy is to constrain only visible, supported regions and progressively relax those constraints; replacing a hard binary mask with geometric confidence is a useful direction to test.

Rating

• Novelty: 4/5. Combines terrain-adaptive human paths with viewpoint-visible scene guidance, with the contribution concentrated in geometric control integration and inference mechanisms.
• Experimental Thoroughness: 3/5. Includes two benchmarks and component ablations, but inconsistent reported baselines and limited physical-contact and camera-control measurements weaken the evidence.
• Writing Quality: 3/5. The overall pipeline is clear, but the main/ablation discrepancy is unexplained and several cached equations cannot be reliably recovered.
• Value: 4/5. Offers a practical training-free approach for previs and controllable human video, while complex contact and real deployment costs remain unverified.