Identity-Preserving Human Reconstruction from a Single Image via 3D Token Inference

Conference: ECCV 2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: 3D human reconstruction, identity preservation, sparse voxel representation, 3D token inference, 3D ID Adapter

TL;DR

This paper introduces the Identity-Preserving Large Human Reconstruction Model (IPRM), a feed-forward framework that operates on an SMPL-based sparse voxel space to infer invisible regions via visibility-based cross-attention while strictly preserving visible identity tokens, paired with a parallel 3D ID Adapter to eliminate identity drift during decoding, achieving photorealistic 3DGS and mesh reconstruction in approximately 0.6 seconds.

Background & Motivation

Reconstructing photorealistic, clothed 3D human models from a single in-the-wild image holds tremendous commercial and practical value across virtual reality, animation, digital avatar creation, and gaming. While early paradigms relied on implicit surface functions or regression of parametric meshes, they generally lacked detailed clothing fidelity and high-resolution texture. Recent approaches leveraging 2D multi-view diffusion models can hallucinate plausible unobserved regions; however, their generative nature across independent 2D views inherently struggles with cross-view 3D consistency, frequently causing distorted body geometry, severed fingers, and multi-face artifacts, all while incurring hefty multi-step diffusion sampling overheads of up to several minutes per subject.

To accelerate reconstruction to feed-forward speeds, Large Reconstruction Models (LRMs) have recently been tailored to human digitization. Most notably, methods like LHM sample dense feature points directly on the surface of parametric body models (such as SMPL) and update all 3D geometric tokens conditioned on global 2D image features. Nonetheless, this "2D-condition token inference" formulation encounters two fundamental bottlenecks. First, SMPL surfaces deviate significantly from real-world human geometry when subjects wear loose clothing such as dresses and wide-leg trousers, leading to severe projection misalignments and heavy point-sampling computation. Second, global bidirectional attention across all tokens inevitably overwrites the already observed 2D identity cues during occluded-region inference, culminating in substantial identity drift in facial features, hand morphology, and fine garment patterns.

To resolve these core challenges, this paper shifts away from fragile surface point sampling and indiscriminate global 2D conditioning toward an identity-aligned, geometrically robust 3D representation. Core idea: anchor monocular 3D human reconstruction within an SMPL-based sparse voxel space, employ visibility-based cross-attention to infer unobserved regions while explicitly freezing visible identity tokens, and inject token-wise identity guidance during decoding via a parallel 3D identity branch (3D ID Adapter), achieving photorealistic, identity-preserving feed-forward human reconstruction in under one second.

Method

Overall Architecture

The IPRM architecture operates through a two-stage feed-forward pipeline. In the first stage, given a single in-the-wild human image \(I\) and an estimated coarse SMPL-X model, IPRM defines an active sparse voxel space bounding the SMPL geometry and projects 2D image features extracted by DINOv2 into the occupied voxels. By assessing voxel surface normals and ray casting occlusions, all voxel tokens are partitioned into visible identity tokens \(T_V\) and invisible tokens \(T_I\). An Identity-aware 3D Token Inference Module then executes visibility-based cross-attention to propagate identity cues from \(T_V\) to \(T_I\) while keeping \(T_V\) strictly intact, followed by global refinement through an Adaptive 3D Human Feature that merges visible and hallucinated invisible body priors into a complete 3D feature volume \(V_{\text{3D}}\). In the second stage, an encoder-decoder network decodes these 3D voxel tokens into either 3D Gaussian Splatting (3DGS) attributes or an explicit polygonal mesh. To prevent identity drift during representation conversion, a parallel identity branch extracts 3D identity condition tokens to guide the target decoder via a 3D ID Adapter at the token level.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single input image + coarse SMPL-X mesh"] --> B["SMPL sparse voxel space construction<br/>DINOv2 feature back-projection & visibility classification"]
    B --> C["Visible identity tokens & invisible tokens to infer"]
    C --> D["Visibility-based cross-attention inference<br/>Freeze visible identity tokens and update invisible regions only"]
    D --> E["Adaptive 3D Human Feature refinement<br/>Visible and invisible human global prior guidance"]
    E --> F["Complete 3D sparse voxel features"]
    F --> G["3D ID Adapter guided feature decoding<br/>Parallel 3D identity branch injects key/value guidance"]
    G --> H["Feed-forward 3DGS / Mesh output<br/>Total inference time approx. 0.63 seconds"]

Key Designs

1. SMPL-based sparse voxel tokenization: enhancing topological robustness and computational efficiency

Rather than densely sampling surface points on parametric meshes—a practice vulnerable to loose clothing topology deviations and computationally burdensome—IPRM establishes a sparse voxel grid surrounding the SMPL geometry. By indexing only the active voxels occupied by SMPL, the model captures the coarse volumetric structure of the subject while providing spatial tolerance for garments extending beyond the skin surface. Given DINOv2 image features \(F\), features are projected onto the active voxels to form a serialized token sequence \(T = (z_i, f_i)_{i=0}^{N-1}\), where \(z_i \in \{0, 1, \dots, n-1\}^3\) denotes the 3D voxel coordinate index and \(f_i\) is the projected feature vector (with grid resolution set to \(n=128\)). Subsequently, back-face culling is computed using the angular relationship between voxel normals and camera rays, supplemented by an optional ray-marching occlusion check. This yields a precise visibility mask \(M\) that cleanly categorizes tokens into visible identity tokens \(T_V\) and invisible tokens to be inferred \(T_I\), decoupling known observations from unknown regions directly in 3D space.

2. Identity-aware 3D token inference module: strictly preserving visible identity while completing occluded geometry

In standard self-attention or global conditioning schemes, updating all 3D tokens concurrently degrades the fidelity of the already-aligned input view, corrupting fine facial details and clothing patterns. IPRM introduces visibility-based cross-attention blocks where invisible tokens \(T_I\) serve as queries and visible tokens \(T_V\) act as keys and values. Because attention flows unidirectionally from known observations to occluded volumes, the visible identity tokens \(T_V\) remain mathematically unaltered throughout the inference process. To guarantee structural continuity across visible-invisible boundaries and infuse global anatomical priors, an Adaptive 3D Human Feature is incorporated. Starting from a visible global human vector \(\mathcal{P}_V\) derived from an image encoder, the network predicts an invisible global human feature \(\mathcal{P}_I\) using cross-attention conditioned on the newly inferred \(T_I\). The concatenation \(\mathcal{P}_{3D} = \mathcal{P}_V \parallel \mathcal{P}_I\) then collectively conditions all 3D tokens:

\[ \mathcal{P}_I = \text{Cross-Attention}(\mathcal{P}_V, T_I) \]

3. 3D ID Adapter with parallel identity branch: mitigating identity drift during representation decoding

When translating complete 3D voxel tokens \(V_{\text{3D}}\) into explicit 3DGS primitives (position offsets, opacity, scale, rotation, and spherical harmonics) or Signed Distance Fields (SDF) for mesh generation, standard decoders frequently suffer from identity drift due to latent ambiguities. Conventional 2D ID Adapters condition on 2D image tokens via cross-attention, but cross-domain alignment across 2D and 3D modalities remains sub-optimal. IPRM proposes a 3D ID Adapter backed by a lightweight parallel identity branch. This branch takes single-view visible voxel features \(V_S\) as input and is supervised solely to reconstruct the input identity view with visible Gaussian primitives. In the target reconstruction branch, inspired by ReferenceNet, the keys and values of the self-attention layers for visible tokens are directly replaced by corresponding keys and values computed from the identity branch, while the target branch queries remain unchanged (invisible tokens retain their own keys and values). Performing token-level identity injection strictly within the native 3D token domain ensures that the visible facets of the reconstructed avatar remain faithfully locked to the input identity.

Loss & Training

IPRM uses a decoupled two-stage training strategy: 1. Stage 1 (3D Token Inference Training): Ground-truth 3D voxel features \(V_{\text{3D}}^{\text{GT}}\) are generated offline by projecting and averaging multi-view image DINOv2 features into the SMPL voxel space. The identity-aware 3D token inference module is supervised via a 3D Mean Squared Error (MSE) loss between predicted \(V_{\text{3D}}\) and \(V_{\text{3D}}^{\text{GT}}\), optimized using AdamW with an initial learning rate of \(2 \times 10^{-3}\) for 200,000 iterations. 2. Stage 2 (Encoder-Decoder and 3D ID Adapter Joint Training): Freezing the token inference module, the model fine-tunes the 3D encoder-decoder alongside the parallel identity branch. For 3DGS decoding, rendered views are supervised with a combination of \(L_1\) loss, structural similarity loss (D-SSIM), and perceptual loss (LPIPS). For mesh decoding, signed distance values at voxel vertices produce isosurfaces supervised by \(L_1\) loss on rendered normal and depth maps. The identity branch is supervised exclusively on the input view, reinforced by an identity-view consistency loss between both branches. Stage 2 employs an initial learning rate of \(5 \times 10^{-4}\) for 100,000 iterations.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on Synthetic Data and the THuman2.1 benchmark for 3DGS representation, assessing overall rendering quality (PSNR, SSIM, LPIPS), input-view identity preservation (PSNR(I)), face consistency error (FC↓), per-sample inference runtime, and peak GPU memory usage.

Method	Paradigm	Synthetic PSNR↑	Synthetic SSIM↑	PSNR(I)↑	FC↓	THuman PSNR↑	THuman SSIM↑	Inference Time	GPU Memory
GTA (NeurIPS 2023)	2D Prior	17.03	0.919	17.66	0.051	19.61	0.834	0.68s	≈8GB
SIFu (CVPR 2024)	2D Implicit	16.68	0.917	19.22	0.060	19.44	0.831	0.65s	≈9GB
DreamGaussian (ICLR 2024)	SDS Optimization	18.54	0.917	19.47	0.056	-	-	2 min	≈8GB
SiTH (CVPR 2024)	Diffusion Gen.	-	-	-	-	18.46	0.820	45.12s	≈21GB
PSHuman (CVPR 2025)	Diffusion+Carving	17.56	0.921	21.44	0.037	20.85	0.864	1 min	≈40GB
Trellis* (CVPR 2025)	General 3D Diff.	21.67	0.921	21.99	0.058	21.33	0.886	4.20s	≈11GB
LHM-0.5B (ICCV 2025)	2D-Cond. 3D Token	25.18	0.951	26.64	0.035	-	-	2.01s	≈18GB
IPRM (Ours)	3D Token Inference	27.04	0.954	28.90	0.033	26.74	0.948	0.63s	≈9GB

For polygonal mesh reconstruction on THuman2.1 (evaluated by Chamfer Distance CD↓, Point-to-Surface P2S↓, and Normal Consistency NC↑): - Optimization-based PSHuman (which runs per-case iterative optimization) achieves CD: 0.4399, P2S: 0.4077, NC: 0.8504. - IPRM, operating in a purely feed-forward manner in 0.63s without per-case optimization, achieves CD: 0.4458, P2S: 0.4331, and NC: 0.8504, virtually matching the quality of optimization-based methods while vastly outperforming feed-forward baselines like ICON (CD 0.6146), ECON (CD 0.6725), and GTA (CD 0.5791). - On the CustomHumans dataset, IPRM delivers a PSNR of 29.33 dB and an FC of 0.035, outperforming LHM-0.5B (28.31 dB, FC 0.039).

Ablation Study

Ablations on Synthetic Data isolate the impact of each core architectural component and representation choice on input-view identity preservation and reconstruction fidelity:

Config	PSNR(I)↑	SSIM↑	LPIPS↓	Note & Analysis
Full Model (IPRM)	28.90	0.954	0.03	Default configuration with \(n=128\) sparse voxels
w/o Feature Project	17.01	0.911	0.10	Initializing voxels with positional encodings only causes collapse
Lower Voxel Res (Voxel \(n=64\))	27.36	0.930	0.05	Coarser resolution impairs thin geometry; runtime improves slightly (0.48s vs 0.63s)
w/o Visibility Cross-Attn [Self-Att]	27.87	0.952	0.04	Self-attention overwrites visible identity tokens, dropping PSNR(I) by 1.03 dB
w/o Adaptive 3D Human Feature	28.54	0.953	0.03	Absence of occluded human prior degrades seamless boundary transitions
w/o Condition (No ID Adapter)	25.80	0.950	0.04	Decoding without identity guidance leads to severe identity drift
w/ 2D Image Condition (2D ID Adapter)	26.42	0.952	0.04	2D-to-3D cross-attention yields limited gain due to cross-modal alignment friction

Key Findings

• 3D ID Adapter is the primary driver of identity preservation: Removing conditioning during decoding drops input-view PSNR(I) to 25.80 dB. Standard 2D image conditioning provides a modest bump to 26.42 dB, whereas the proposed 3D ID Adapter boosts PSNR(I) to 28.90 dB (+3.10 dB total gain), proving that token-wise injection within the native 3D feature domain effectively suppresses identity drift.
• Visibility-based cross-attention protects input observations: Replacing unidirectional cross-attention with standard bidirectional self-attention contaminates known visible features during unseen region generation, reducing PSNR(I) by over 1.0 dB.
• Sparse voxels confer superior tolerance to loose garments: Unlike surface point sampling on SMPL, which tears and collapses when garments deviate from the underlying body topology, the volumetric sparse voxel representation encloses outer silhouettes robustly, producing smooth geometry on dresses and wide garments.

Highlights & Insights

• Strictly causal "visible-to-invisible" reasoning in 3D: Rather than relying on black-box 2D diffusion or indiscriminate self-attention, IPRM decouples known and unknown 3D regions via explicit ray visibility, enforcing a unidirectional information flow that preserves the original observation.
• Symmetric 3D identity branch and ReferenceNet-style key/value injection: By training a parallel identity branch purely on the input view and substituting keys and values into the target branch's self-attention layers, IPRM eliminates cross-domain alignment gaps between 2D images and 3D token representations.
• Sub-second feed-forward pipeline supporting dual representations: The method executes in approximately 0.63s (0.2s for token inference, 0.4s for decoding) using only ~9GB GPU memory, delivering both 3DGS for real-time rendering and watertight meshes for downstream rigging without any per-case optimization.

Limitations & Future Work

• Geometric truncation under extreme poses and out-of-body props: Because active voxels are seeded by the estimated SMPL-X model, highly unconventional poses, voluminous ball gowns, or handheld accessories (such as musical instruments or large backpacks) that fall outside the local voxel grid may experience truncation or incomplete reconstruction.
• Dependency on upstream human foundation representations: The Adaptive 3D Human Feature relies on pretrained human-centric encoders (e.g., Sapiens), whose performance may decline when presented with heavily occluded or out-of-distribution attire.
• Future directions: Integrating generalized 3D diffusion priors with parametric SMPL bounds could allow seamless handling of extreme loose clothing and accessories. Furthermore, given its sub-second latency, IPRM can serve as a strong initialization for downstream per-case optimization workflows.

Related Work & Insights

• vs LHM (Large Animatable Human Reconstruction Model): LHM samples dense surface points on SMPL and updates them via global 2D image conditioning, which causes identity drift on facial regions and struggles with loose garments. IPRM replaces surface points with SMPL-based sparse voxels, uses visibility-based cross-attention to shield visible tokens, and adds a 3D ID Adapter, outperforming LHM across all benchmarks.
• vs PSHuman: PSHuman combines multi-view 2D diffusion with per-case mesh carving, taking ~1 minute per subject and requiring ~40GB GPU memory. IPRM is purely feed-forward, taking 0.63s and ~9GB memory while matching PSHuman's mesh reconstruction accuracy (Chamfer Distance 0.4458 vs 0.4399).
• vs Trellis: While Trellis demonstrated the power of structured sparse voxels for general 3D assets, it lacks human-specific anatomical priors and identity constraints. IPRM adapts sparse voxels to SMPL space and introduces identity-aware cross-attention and adapter mechanisms to resolve single-image human reconstruction challenges.

Rating

• Novelty: ⭐⭐⭐⭐☆ Anchors 3D token inference within an SMPL sparse voxel grid using visibility masking and a dedicated 3D ID Adapter.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks spanning both 3DGS and mesh outputs, exhaustive ablations on modules and resolutions, with thorough runtime and memory profiling.
• Writing Quality: ⭐⭐⭐⭐⭐ Clearly structured narrative with rigorous technical motivation regarding the shortcomings of 2D-conditioned point representations.
• Value: ⭐⭐⭐⭐⭐ Achieves high-fidelity, identity-preserving clothed human reconstruction in ~0.63s with low GPU footprint, offering immense utility for practical digital human workflows.