HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video¶

Conference: ECCV 2026
Paper: ECCV 2026 official page / project page
Area: 3D Vision
Keywords: uncalibrated sparse views / feed-forward 3D Gaussian reconstruction / volumetric human video / high-resolution novel view synthesis / scale-synchronized camera calibration

TL;DR¶

HiReFF builds on a VGGT backbone with Gaussian, mask, and high-resolution side-tuning branches to feed-forward reconstruct human 3D Gaussians from four uncalibrated 90°-spaced videos, rendering 360° dynamic humans at 2072×2072 and streaming reconstructions at 3.01 FPS on a single RTX 4090.

Background & Motivation¶

Holographic communication, AR/VR, and sports broadcasting all want volumetric human video to become as casual to capture and as instant to watch as a video call, yet deployable solutions either depend on a fully calibrated camera rig plus hours of per-scene optimization (3DGS-Avatar, HumanNeRF and the like), or use a feed-forward network to buy speed on already calibrated multi-view input (GPS-Gaussian). The line that actually removes calibration is the recent family of uncalibrated feed-forward geometry models: DUSt3R, VGGT, MapAnything, and π³ regress camera parameters and point clouds directly from sparse unposed images, and AnySplat goes further by predicting 3D Gaussians for photorealistic novel view synthesis. But these models almost all stop at single-frame inference with 518×518 output, and lifting them onto dynamic human volumetric video does not work out of the box.

The difficulty is that the constraints fight each other. With four views 90° apart, adjacent views barely overlap: the camera head fluctuates heavily during training, the rendering loss has high variance, and convergence suffers; supervising only those four input views is far from enough, yet adding new supervision viewpoints requires ground-truth cameras — and VGGT-style predictions carry no metric scale (the paper's example: the predicted human may be only 0.8 m tall), so the ground-truth cameras cannot simply be plugged in. Foreground is the second trap: human reconstruction obviously wants only the foreground, but cropping the input to the person visibly degrades camera estimation — the paper's empirical observation is that larger input regions yield more accurate camera predictions — so "remove background" and "estimate cameras" collide head-on. Resolution is the hardest wall of all: the compute of a geometry transformer grows with the square of input resolution, and pushing the backbone input from 518×518 straight to 2K exceeds 80 GB of VRAM.

This paper's judgement is that high-resolution rendering does not require higher geometric or camera accuracy, only richer appearance attributes, so resolution can be factored out and solved separately. HiReFF attaches a Gaussian head and a mask head to VGGT's pretrained weights for feed-forward reconstruction of foreground human 3D Gaussians, resolves scale ambiguity and camera jitter with scale-synchronized camera calibration, keeps only foreground Gaussians alive while preserving full-image input through Gaussian-wise foreground masking, and injects 2K image detail into the Gaussian head through a side pathway. Core idea: treat "high resolution" as a problem of Gaussian-attribute precision rather than of geometric precision — the backbone stays at 518×518 and only handles cameras and Gaussian positions, while a second pathway feeds 2K image features into the Gaussian head for appearance detail, buying 2K rendering at a small VRAM cost; in parallel, ground-truth cameras aligned in scale render additional viewpoints to indirectly supervise a frozen camera head, supplying supervision while suppressing its jitter.

Method¶

Overall Architecture¶

The input is four uncalibrated high-resolution RGB videos \(\{V_i\}_{i=1}^4\) with adjacent viewpoints roughly 90° apart, whose frame \(t\) is written \(I_i^t\). For each time step \(t\) the model outputs two things: a set of anisotropic 3D Gaussians \(\{(\mu_g^t,\sigma_g^t,r_g^t,s_g^t,c_g^t)\}_{g=1}^{G}\) (position, opacity, rotation, scale, and spherical-harmonic coefficients), and per-view camera parameters \(\{p_i^t\in\mathbb{R}^9\}_{i=1}^4\) that stay pixel-aligned with the input. The Gaussian count \(G=4\times H_{\text{HR}}\times W_{\text{HR}}\) is exactly the number of pixels across the four high-resolution input views (about 17.17 million at 2K), i.e. one Gaussian per input pixel. Notably, this representation contains no parametric body model at all (no SMPL / SMPL-X): the body is carried entirely by Gaussians positioned through depth back-projection, which is the direct reason it can cope with loose clothing and complex deformation. Dynamics are frame-by-frame feed-forward: each time step is inferred independently, with no explicit deformation field or canonical space, and temporal consistency comes from the frame-smooth camera trajectory and from cross-frame stability during training.

In terms of data flow, the four frames first go through VGGT's Alternating-Attention (AA) Transformer for features, then several heads decode in parallel: the camera head emits 9-D camera parameters per view per frame, the depth head emits per-pixel depth, and the two combine in a back-projection to place Gaussian centers (for pixel homogeneous coordinates \(p=[u,v,1]^{\top}\), \(\mu=R^{\top}(d\,K^{-1}p-t)\) with intrinsics \(K\) and extrinsics \([R|t]\)), keeping every Gaussian aligned with an input pixel; the Gaussian head (a DPT) decodes features into Gaussian attributes. Beyond that, a high-resolution side pathway injects 2K image features into the mid-level of the Gaussian head, a mask head removes background Gaussians point by point, and at training time an additional four novel viewpoints are rendered with scale-aligned ground-truth cameras and compared against ground-truth images.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Four uncalibrated videos<br/>90° apart / 2K"] --> B["AA Transformer backbone<br/>518×518 features"]
    A --> C["High-resolution Side-tuning<br/>HR side features into Gaussian head"]
    B --> D["Camera head / depth head / Gaussian head"]
    C --> D
    D --> E["Gaussian-wise Foreground Masking<br/>mask head filters foreground Gaussians"]
    E --> F["Scale-synchronized Camera Calibration<br/>frozen camera head + 8-view supervision"]
    F --> G["2K 360° dynamic human Gaussians"]

Key Designs¶

1. High-resolution Side-tuning: keep the backbone at 0.5K and feed 2K detail in through a side pathway

The VRAM cost of a geometry transformer grows with input resolution, and feeding 2K images straight into the AA Transformer goes out of memory (over 80 GB in the paper's measurement), while existing feed-forward models (AnySplat, VGGT) also operate at 518×518, which caps rendering resolution at 0.5K. The opening is an observation: high-resolution rendering quality depends mainly on the high-dimensional Gaussian attributes \(\Theta_g\) produced by the Gaussian head \(F_G\) (color/spherical harmonics, scale, rotation, opacity), not on the position \(\mu_g\) or the camera position \(p_i\). If that is true, the backbone does not need to see 2K, only to understand geometry. So the LR image \(I_{\text{LR}}\) still goes through the AA Transformer, its intermediate Gaussian-head features \(\mathcal{U}(F_G^{\text{mid}}(I_{\text{LR}}^t))\) are upsampled, added to features extracted from the HR image by a supplementary network \(F_a\), and the sum is fed to the decoding MLP \(F_D\):

\[\Theta^t = F_D\Big(F_a\big(I_{\text{HR}}^t\big)\oplus\mathcal{U}\big(F_G^{\text{mid}}(I_{\text{LR}}^t)\big)\Big)\]

Here \(F_a\) is an MLP modified by EdgeNeXt, supplying low-level, fine-grained high-frequency information that complements the AA Transformer's high-level semantic features; the mask head and depth head outputs are likewise upsampled to high resolution before rendering. Training renders novel views directly at 2K and compares against 2K ground truth, with a patch-wise perceptual loss to contain memory. The payoff is concrete: holding high-resolution supervision fixed and raising the input from 0.5K to 2K costs only 33.9% more training VRAM (44125 MiB → 59095 MiB), whereas dropping side-tuning and raising the backbone input to 2K runs out of memory.

2. Gaussian-wise Foreground Masking: reconstruct on the full image but let only foreground Gaussians survive

Human reconstruction only needs the foreground, but pre-segmenting the input and removing the background severely degrades camera parameter estimation — especially at 90° spacing where views barely overlap. The paper moves "foreground" from the input end to the output end: the input is always the complete image with background (preserving the context the camera head needs), and an extra mask head \(F_m\) modulates the predicted Gaussians one by one, multiplying each Gaussian's parameters by its predicted foreground probability so that background Gaussians are switched off:

\[(\Theta^{t}_g,\mu^{t}_g)_{g=1}^{H}=F_m\big((\Theta^{t}_g,\mu^{t}_g)_{g=1}^{G}\big)\odot(\Theta^{t}_g,\mu^{t}_g)_{g=1}^{G}\]

(⚠️ Equation (2) of the original is garbled in the cached text; the form above restates the prose description — refer to the original for exact notation.) The mask is supervised with an L1 loss on the four input views only. It works on two levels: the camera head still receives the full image context, avoiding the "trade background for accuracy" dilemma; and the mask loss effectively teaches the Gaussian decoder to only generate Gaussians inside the foreground, so after training the decoder no longer reconstructs regions beyond the mask boundary. Because the input views are nearly orthogonal, a single-view mask may not cover all background, so a few extraneous Gaussians can survive masking — but experiments show they vanish during training, leaving a clean foreground reconstruction.

3. Scale-synchronized Camera Calibration: freeze the camera head and supervise it indirectly with scale-aligned ground-truth cameras

This is the paper's subtlest step, handling three entangled problems at once. First, four input views 90° apart provide too little supervision, so additional novel-view supervision is required, and rendering those viewpoints needs ground-truth cameras. Second, the ground-truth cameras cannot be used directly: VGGT-style predictions have no metric scale, Gaussian positions \(\mu^t\) are self-consistent only with the predicted cameras, and plugging in ground-truth translations would render a completely misaligned scene. Third, wide viewpoint intervals make the camera head fluctuate violently during training, inflating rendering-loss variance and hurting convergence. HiReFF's answer is to freeze the camera head during training, project the Gaussians to the image plane with the ground-truth intrinsics \(\hat{K}^t\) and rotations \(\hat{R}^t\), but first rescale the ground-truth translations into the predicted scale — taking view 1 as reference, it computes a global scale factor from the ratio of predicted to ground-truth translation magnitudes over the non-reference views:

\[\bar{s}=\frac{1}{V-1}\sum_{i=2}^{V}\frac{\|\hat{\mathbf{t}}_i\|}{\|\mathbf{t}_i\|},\qquad \hat{\mathbf{t}}_i'=\bar{s}\,\hat{\mathbf{t}}_i\]

(⚠️ Equation (3) is likewise garbled in the cache; the form above restates the prose about \(V\), the reference view, and "scaling the true camera parameters by adjusting the translation to match the scale of the predicted translation" — refer to the original.) Once the ground-truth cameras are brought into the predicted coordinate frame's scale, they can render \(\{\mu_g^t,\Theta_g^t\}\) for comparison against ground-truth images. The key point is that the camera head is not bypassed, only not updated: Gaussian centers \(\mu_g^t\) are still back-projected from predicted camera parameters, so the rendering loss flows back along "features → camera head → back-projection → render" into the AA Transformer, indirectly optimizing the camera head's input features. Since the emitted camera parameters still come from the camera head, inference needs no ground-truth cameras at all. The extra supervision viewpoints are sampled from frontal, top, and bottom positions outside the input viewing sphere (\(V_a=4\)), giving \(4+V_a=8\) supervised views in total. The benefit is twofold: the extra views restore supervision, and freezing plus indirect optimization removes camera-head jitter, stabilizing predictions across frames and mitigating scale drift between per-frame reconstructions.

A Worked Example¶

Walk one time step \(t\) through the pipeline. Four 2072×2072 high-resolution frames enter: each is downsampled to 518×518 for the AA Transformer while also passing through \(F_a\) on the HR pathway; the backbone then decodes about 4.29 million (2072²) candidate Gaussians per view, \(G\approx 17.17\) million across the four views, each carrying position, opacity, rotation, scale, and spherical harmonics. The mask head scores all 17.17 million Gaussians with a foreground probability, and after point-wise multiplication only the ones belonging to the human remain (denoted \(H\) in the paper). At render time the Gaussian centers come from back-projecting the camera head's predicted pose, while projecting them onto the image plane uses the ground-truth cameras — whose translations have first been multiplied by the global factor \(\bar{s}\) to bring them into the predicted frame's scale. Images are then rendered at 2K over the four input views plus four extra viewpoints (frontal, top, bottom) and compared to ground truth with L1 plus perceptual loss. Everything runs independently per time step, which is what makes it streaming: the previous frame's result never enters the next frame's computation, and stability is maintained only through the inter-frame continuity constraint on the camera trajectory during training.

Loss & Training¶

The objective combines a rendering loss, a mask loss, and a depth distillation loss. The rendering loss supervises synthesized images over \(4+V_a\) views as a weighted sum of L1 and perceptual terms:

\[\mathcal{L}_{\text{render}}=\sum_{i=1}^{4+V_a}\|\hat{I}_i-I_i\|_1+\lambda_P\sum_{i=1}^{4+V_a}\text{Perceptual}(\hat{I}_i,I_i)\]

The perceptual term measures distance in VGG feature space (following Animatable Gaussians), and computation graph surgery is used to lower its memory consumption. The mask loss is an L1 between predicted and ground-truth masks over the four input views. The depth distillation loss keeps the active depth head consistent (MSE) with a frozen reference depth head initialized from VGGT weights, which prevents the depth head from overfitting to the input viewpoints. The total loss is the weighted sum \(\mathcal{L}_{\text{total}}=\lambda_{\text{render}}\mathcal{L}_{\text{render}}+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}}+\lambda_{\text{depth}}\mathcal{L}_{\text{depth}}\).

For initialization, the AA Transformer, camera head, and both depth heads take VGGT pretrained weights, while the 3D Gaussian prediction head, the side-tuning supplementary network, and the mask head are zero-initialized; the renderer is based on gsplat. HR resolution is defined as 2072×2072 and LR as 518×518, with \(V_a=4\) extra supervision views. Loss weights are \(\lambda_P=0.1\), \(\lambda_{\text{render}}=1.0\), \(\lambda_{\text{mask}}=5\times10^{-2}\), and \(\lambda_{\text{depth}}\), whose exponent is lost in the original ⚠️ (refer to the original). Training runs on 8 A800 GPUs with automatic mixed precision.

Key Experimental Results¶

Main Results¶

Training is done mainly on DNA-Rendering (153 actors, 439 motion sequences of 48-view synchronized 2448×2048 video with camera parameters), with ZJU-MoCap and MVHumanNet used to improve generalization; validation uses 20 motion sequences with distinct identities from DNA-Rendering. Metrics are PSNR / SSIM / LPIPS at 2072×2072, with methods that natively render lower resolutions (GPS-Gaussian, AnySplat) bilinearly upsampled to 2K; all methods are evaluated under a 4-view input setting with roughly 90° separation. The "Aligned" column marks whether camera parameters were further optimized for 200 steps to align with ground truth — ours requires no optimization at all.

Method	Cam. Pose	Views	Aligned	PSNR↑	SSIM↑	LPIPS↓
GPS-Gaussian	✓	8	–	26.1039	0.9172	0.1384
4DGT	×	1	×	17.1689	0.8395	0.2719
NoPoSplat	×	2	×	22.6296	0.8876	0.1736
NoPoSplat	×	2	✓	23.4321	0.8939	0.1588
AnySplat	×	4	×	23.7844	0.9040	0.1737
AnySplat	×	4	✓	25.5875	0.9140	0.1598
Ours	×	4	×	26.5138	0.9164	0.1277

Ablation Study¶

Table 2 reports training VRAM (input resolution / supervision resolution / side-tuning / VRAM) and Table 3 reports inference VRAM and frame rate on a single RTX 4090.

Input Res.	Sup. Res.	Side-tuning	Training VRAM (MiB)
518	518	✓	40503
518	2072	✓	44125
2072	2072	✓	59095
2072	2072	×	OOM (>80GB)

Input Res.	Ren. Res.	Inference VRAM (MiB)	Frame Rate (RTX 4090)
518	2072	10852	4.40 FPS
1036	2072	10886	4.02 FPS
2072	2072	14052	3.01 FPS

The paper also ablates Gaussian-wise foreground masking (Fig. 7, qualitative only — no metric numbers are given): without masking the background gets reconstructed, with masking the foreground is clean and the few surviving extraneous Gaussians disappear during training.

Key Findings¶

Best across the board under four uncalibrated views: ours leads all metrics among methods taking uncalibrated input, and does so without any camera optimization. Against the most directly comparable AnySplat (the unaligned row), PSNR improves by 2.7294 (23.7844 → 26.5138), SSIM by 0.0124 (0.9040 → 0.9164), and LPIPS drops by 0.0460 (0.1737 → 0.1277); even after AnySplat's 200-step camera optimization, our margin remains clear.
The GPS-Gaussian comparison needs care: it receives 8 calibrated views, ground-truth cameras, and pre-segmented foreground images, and its SSIM of 0.9172 is on par with our 0.9164 — but it cannot reconstruct from so few views. The paper further argues that part of its metric advantage comes from that segmentation: all methods' results are multiplied by a common mask before evaluation, so the instrument region our method reconstructs thanks to its unmasked input (absent from the ground-truth mask) earns no metric credit, while GPS-Gaussian's artifacts (extra feet) get suppressed by the mask — an evaluation bias that works against this paper.
The cost bottleneck is render resolution, not backbone input resolution: on the training side, holding high-resolution supervision fixed and raising the backbone input from 0.5K to 2K costs only 33.9% more VRAM (44125 → 59095 MiB), while dropping side-tuning and raising the input runs out of memory. On the inference side at 2K rendering, raising the input from 0.5K to 1K costs only 8.6% (4.40 → 4.02 FPS), showing the backbone input resolution barely affects speed; what actually slows things down is the output resolution of the Gaussian renderer (4.40 → 3.01 FPS). ⚠️ Note the paper's text claims 3.01 FPS is "only 24% lower" than 4.40 FPS, but converting frame rates gives an actual 31.6% drop — the two numbers disagree, and the tables should be trusted here.
Qualitatively, ours surpasses the baselines in global shape, garment detail, and facial fidelity; AnySplat and NoPoSplat estimate cameras poorly under the wide 90° baseline, causing misaligned cross-view reconstructions, and 4DGT fails to produce plausible results from a single-frame video. On temporal consistency, garments with complex patterns stay stable across consecutive time steps.

Highlights & Insights¶

The "high resolution doesn't need higher precision" decoupling: the authors lift the HR problem off the backbone's resolution and only add appearance features on a side pathway, so 2K input costs almost no extra VRAM. This is the most compute-efficient and most reusable step in the paper — any feed-forward reconstruction task whose backbone cannot afford high resolution but whose output needs high-resolution detail can copy it.
Freezing the camera head while keeping the gradient path alive: freezing weights is not the same as cutting gradients. The rendering loss flows back through "predicted camera parameters → back-projection → Gaussian centers" into the AA Transformer, bringing in extra-view supervision while suppressing camera-head jitter and incidentally stabilizing cross-frame predictions. This "freeze the parameters, optimize indirectly" trick transfers to other uncalibrated feed-forward tasks where predicted quantities lack metric scale yet extra supervision is needed.
Scale alignment touches only the translation: rotations and intrinsics are left alone, and a single global scale factor brings the ground-truth cameras into the predicted frame — an engineering trick that is trivially simple yet clears the "no metric scale" hurdle.
Masking at the Gaussian level rather than the image level: moving the foreground problem from the input end to the output end preserves the image context the camera head needs, and the mask loss teaches the decoder to generate Gaussians only in the foreground — a classification head's supervision incidentally constrains the geometry decoder, which is more interesting than a plain segmentation step.

Limitations & Future Work¶

Limitations the authors admit: regions occluded from multiple viewpoints show isolated points (for instance the area only the frontal view captures, where insufficient side information leaves discrete points), and they plan to add hand and face priors to improve geometric and topological accuracy there.
Limitations I spot: four views at 90° is a narrow and highly regular configuration, and how much the model depends on that equal-angle layout is untested, so behavior under non-uniform or fewer viewpoints is unknown; quantitative evaluation covers only 20 motion sequences from DNA-Rendering, which is small, while the other two datasets serve training-side generalization only, so "cross-dataset generalization" lacks quantitative evidence; "uncalibrated" means the network is not given camera parameters, yet training and evaluation still lean on DNA-Rendering's own calibration quality and its ground-truth intrinsics/rotations — poor calibration there would shift the entire supervision signal. The masking ablation is qualitative only, with no numbers for the no-mask variant, even though masking is precisely the design that affects camera estimation; quantifying it is what would prove "Gaussian-wise" beats "image-wise." Finally, depth distillation treats the VGGT depth head as a teacher, so failures of VGGT in some domains could propagate to the student.
Improvement directions: add hand/face parametric priors to fix geometry; jitter the viewpoint layout during training to build robustness to non-uniform views; extend side-tuning to 4K or arbitrary resolutions; add cross-dataset quantitative evaluation and numeric mask ablations.

vs AnySplat: the same "uncalibrated images → 3D Gaussians" feed-forward route, but it is single-frame inference capped at 518×518 and does not handle dynamic humans. HiReFF adds a mask head, scale-synchronized calibration, and high-resolution side-tuning on top, pushing the task to dynamic, 2K, and 360°.
vs VGGT / MapAnything / π³: these uncalibrated geometry foundation models yield camera parameters, depth, and point clouds but not photorealistic imagery. This paper uses them as a backbone kept at 518×518 and adds heads and a side pathway to supply the two missing pieces — photorealistic rendering and high resolution.
vs GPS-Gaussian: also a generalizable human Gaussian method, but it needs calibrated cameras and more views (8) at lower resolution. Ours reaches comparable or better metrics under uncalibrated, 4-view, 2K conditions, at the cost of depending on VGGT features.
vs Forge4D: Forge4D was the first to do human novel view synthesis from uncalibrated video, but only for frontal views; HiReFF targets full 360° coverage and additionally solves high resolution and streaming efficiency.
vs per-scene optimization methods (3DGS-Avatar / HumanNeRF, etc.): they need calibrated multi-camera rigs and long per-scene optimization, whereas ours is fully feed-forward and streams at 3.01 FPS.

Rating¶

Novelty: ⭐⭐⭐⭐ Puts "uncalibrated + sparse + high-resolution + streaming" into one feed-forward framework, with scale-synchronized calibration and HR side-tuning as targeted rather than boilerplate solutions.
Experimental Thoroughness: ⭐⭐⭐ Solid main comparison and clear cost analysis, but quantitative evaluation covers only 20 motion sequences, and both cross-dataset evidence and the mask ablation lack numbers.
Writing Quality: ⭐⭐⭐ Method motivation is clearly argued (especially empirical observations like "larger input regions give better cameras"), but the 2K speed percentage contradicts its own table and several equations are garbled in the source.
Value: ⭐⭐⭐⭐ For real scenarios like holographic communication and live broadcasting, "uncalibrated rigs + 4 views + 2K + real time on one GPU" is a practical combination, and the side-tuning and frozen-camera-head tricks transfer well.