ECCV 2026 3D Vision 4D Gaussian Splatting hand-object interaction reconstruction deformation field interaction-aware regularization progressive optimization

Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: 3D Vision
Keywords: 4D Gaussian Splatting, hand-object interaction reconstruction, deformation field, interaction-aware regularization, progressive optimization

TL;DR¶

Without any object shape, category, or pose prior, this paper drives 4D Gaussian Splatting with three decoupled implicit fields (hand, object, background), augments each Gaussian with a learnable weight \(w\) and radius \(o\) to model occlusion and edge sharpness, conditions the object deformation field on key-frame hand positions, and combines explicit hand / object-rotation / interaction regularizers with a five-phase progressive optimization to surpass 4DGS, Deform3DGS, SC-GS and the specialized HOLD / BIGS on HOI4D and HO3D (PSNR 30.32 dB on HOI4D translation scenes, 3.99 dB above the strongest baseline).

Background & Motivation¶

Accurate reconstruction of hand-object interaction (HOI) underpins VR and robotic teleoperation, and it requires recovering hand shape, object geometry, and the contact relationship between them at the same time. The hardest cases are exactly the most ordinary ones — grasping a cup, twisting a cap — because fingers and object occlude each other, contact boundaries blur in the image, and the motion is fast and includes irregular rotation. The early route treats the manipulated object as known and fits it via 6D pose or template; it is accurate but expensive, since preparing a template or reliable pose annotation for every object is impractical in industrial settings. The other, prior-free route spans SDF and NeRF based methods: the former mostly reconstructs geometry without appearance, while the latter suffers from the inefficiency of backward-mapping ray rendering and becomes prohibitively expensive to train. After 3D Gaussian Splatting (3D-GS) appeared, dynamic Gaussian methods such as 4DGS, Deform3DGS and SC-GS achieved a good fidelity-speed trade-off on generic dynamic scenes, but they generally fail on HOI: feeding every Gaussian into a single deformation MLP forces the model to fit an almost static background while simultaneously producing highly localized, high-frequency hand deformation at the contact instant. The result is misplaced Gaussians, excessive overlap, and a smeared interaction boundary.

A second, finer observation is that the three constituents of an interaction have fundamentally different motion characteristics: the hand is active, fast and local; the object, under passive contact (being held, for instance), is largely carried by the hand; the background is nearly static. A single field that swallows all three is neither reasonable nor necessary. Worse, the deformation field of 4D-GS style methods is content-agnostic — it produces a global offset from canonical coordinates and time and has no idea who is touching what, so the object fails to rotate with the hand when it should and flips implausibly when it should not. Optimization adds another difficulty: HOI exhibits large rotations, large translations and frequent occlusion, so throwing all Gaussians into the optimizer from the start converges slowly and leaves positions misaligned.

The angle taken here is to split hand and object apart at the representation level, then feed interaction back into optimization explicitly as conditions and constraints: hand, object and background each get their own implicit field, the object field takes key-frame hand positions as an extra input so object deformation is conditioned on the hand, every Gaussian gains two learnable scalars (weight \(w\), radius \(o\)) so it can decide for itself whether to be smooth or sharp near occlusion and edges, and three explicit 3D regularizers (hand translation, object rotation, hand-object interaction) plus a penetration term and a coarse-to-fine five-phase optimization schedule are added on top. No object shape, category or 3D box prior is used; the only external input is a coarse MANO estimate from an off-the-shelf hand tracker (the paper reports its overhead as under 3% of total runtime). Core idea: treat HOI reconstruction as a divide-and-condition problem — three decoupled implicit fields each own one motion band, and interaction enters optimization in three forms (the hand-conditioned object field, the hand-rotation prior, and explicit contact regularizers), which cleans up occlusion and interaction boundaries under zero object priors.

Method¶

Overall Architecture¶

The input is a monocular egocentric RGB video plus MANO parameters from an off-the-shelf hand tracker; the output is a complete HOI scene renderable at any timestamp \(t\) from any viewpoint, with hand, object and background available both separately and composited. The pipeline can be summarized as "three implicit fields + three Gaussian sets + one progressive optimization loop": hand Gaussians \(G_H\) and object Gaussians \(G_O\) share a Gaussian parameterization adapted to interaction, denoted \(G_{HO}\) (standard 3D-GS plus two learnable quantities, weight \(w\) and radius \(o\)), while background Gaussians \(G_{BG}\) follow the Deform3DGS formulation. The three fields \(F_H\), \(F_O\), \(F_{BG}\) each deform their own Gaussians from canonical space to the target space at time \(t\), and are finally unified into one shared target space for rasterization during collaborative reconstruction. The split is motivated by two facts: occlusion at the contact region requires finer-grained supervision on each side to be aligned, and the three constituents live in far-apart motion bands — the background needs only low-frequency updates while hand-object interaction demands high-frequency modeling, so a single field inevitably drags them down together.

How interaction actually enters optimization is the key to reading the paper, and it happens in three places. First, the object deformation field \(F_O\) takes key-frame hand positions as input, so object motion is conditioned on the hand. Second, the object rotation loss uses the averaged hand joint rotation as a prior and only acts on contact Gaussians. Third, hand and object are two explicit Gaussian point sets, so a bidirectional Chamfer interaction loss and a penetration loss can be computed directly between them. All three rely solely on the hand tracker, never on an object prior.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: egocentric RGB video<br/>+ tracker MANO parameters"] --> B["Three decoupled fields + hand-conditioned<br/>object deformation field"]
    B --> C["Interaction-aware hand-object Gaussians<br/>learnable weight w and radius o"]
    C --> D["Explicit interaction-aware regularizers<br/>hand / object rotation / interaction / penetration"]
    D --> E["Progressive optimization<br/>init → warm-up → HOI refinement<br/>→ background → collaborative"]
    E --> F["Complete HOI scene at any time t"]

Note that progressive optimization is not a terminal step of the chain but a scheduling framework spanning the whole pipeline: apart from the representation itself, every design below hangs off one of its phases (warm-up applies two 3D losses, refinement activates \(w\) and \(o\), collaboration applies the interaction regularizers).

Key Designs¶

1. Three decoupled fields with a hand-conditioned object deformation field: object deformation should be decided by the hand

Consider first what each field consumes. The hand field \(F_H\) takes the timestamp \(t\) (positionally encoded as \(\gamma(t)\), with noise added to \(t\) following Deform3DGS to counter over-smoothing) together with canonical hand Gaussian coordinates, and outputs offsets in position, rotation and scale. Notably, it does not take MANO parameters as input — hand deformation is learned by the field itself, and MANO appears only in initialization and in the hand loss as coarse guidance. This is a targeted choice: regression-based MANO prediction suffers from error propagation, where an initial hand estimation error cascades into downstream reconstruction errors. Delegating hand deformation to a learnable field and demoting MANO to a coarse 3D guide effectively downgrades tracker error from a hard constraint to a soft prior. The object field \(F_O\) deliberately takes one extra input: at key frame \(k\) (the frame just before hand-object contact), hand and object positions are concatenated and fed in together with the canonical Gaussian coordinates, so that predicting the offset at time \(t\) is aware of where the hand was:

\[\Delta\mathcal{G}_{\text{O}}=\mathcal{F}_{\text{O}}\big(\gamma(\mathbf{x}^{k}_{\text{hand}}\oplus\mathbf{x}^{k}_{\text{obj}}),\,\gamma(t)\big)\]

(⚠️ the equation renders corrupted in the cache; rewritten here from the prose description — refer to the original paper for exact symbols.) This is precisely the compromise between the two failure modes cited in the motivation: a single field is insufficient, yet giving hand and object fully independent fields discards their coupling during contact. The fields stay separate, but the object field carries the hand in its conditioning, so contact-induced deformation has something to latch onto instead of the content-agnostic global offset produced by implicit methods. The background field \(F_{BG}\) follows Deform3DGS and deforms by timestamp \(t\) at low complexity; it is separately pretrained for a fixed number of iterations in the fourth phase so it does not chase the high-frequency jitter of the foreground — anecdotally, this separate modeling visibly improves contrast and dark-region detail. All three fields finally deform into the same target space, where occlusion relationships and lighting coherence are aligned implicitly through the shared photometric supervision.

2. Interaction-aware hand-object Gaussians: let each Gaussian decide whether to be smooth or sharp

At the interaction instant the hand moves violently, surfaces occlude each other and boundaries blur, yet the covariance parameters of standard 3D-GS have no way to express "this region is being occluded", and conventional 4D Gaussians ignore mutual influence between neighboring Gaussians altogether — the outcome is texture drift and soft edges in the interaction zone. The paper's answer is to append two learnable scalars to every Gaussian: weight \(w\in\mathbb{R}^{+}\), which smooths motion and suppresses noise while acting as a soft indicator of structural weakness / occlusion, and radius \(o\in\mathbb{R}^{+}\), which controls edge sharpness, smaller \(o\) meaning a sharper contour. Neither quantity is a post-hoc filter; they rewrite the deformation computation of the HOI refinement phase. First they become neighborhood weights: for Gaussian \(i\), take its \(K\) nearest neighbors, convert distances into spatial proximity weights with an RBF kernel whose bandwidth is \(o_i\), normalize, and modulate by the global importance \(\sigma(w_i)\):

\[w^{\text{spatial}}_{ik}=\exp\left(-\frac{d_{ik}^{2}}{2o_i^{2}}\right),\qquad \hat{w}^{k}_{i}=\sigma(w_i)\cdot\frac{w^{\text{spatial}}_{ik}}{\sum_{j\in\mathcal{N}_K(i)}w^{\text{spatial}}_{ij}}\]

where \(d_{ik}\) is the Euclidean distance between Gaussian centers and \(\sigma\) is the sigmoid (keeping \(w_i\in(0,1)\)). These weights then drive an LBS-style blend of local rigid transforms: each neighbor \(k\) contributes a 6D rotation predicted by the implicit field (converted to a \(3\times3\) \(\Delta R^{t}_{k}\)) and a translation offset, and the deformed position of Gaussian \(i\) at time \(t\) is the weighted combination of those local transforms:

\[\boldsymbol{\rho}^{t}_{i}=\sum_{k\in\mathcal{N}_K(i)}\hat{w}^{k}_{i}\Big(\Delta R^{t}_{k}\big(\mathbf{x}_{i}-\mathbf{x}^{t}_{k}\big)+\mathbf{x}^{t}_{k}\Big)\]

(⚠️ the three equations above are reconstructed from the corrupted formulas and prose in the cache; refer to the original paper.) Who "speaks loudest" in the blend is thus decided jointly by global importance and local neighborhood structure: under occlusion \(w\) is small and \(o\) tightens, so the Gaussian relies on itself rather than being dragged along by neighboring hand or object Gaussians; near edges a small \(o\) degenerates the RBF into an almost hard nearest-neighbor weighting, so the contour is not blurred away. In the ablation, removing this module (the parameters and their training scheme) drops PSNR from 32.96 dB to 28.76 dB (−4.20 dB), the largest single drop reported anywhere in the paper — evidence that the interaction boundary is indeed where the representation earns its keep.

3. Explicit interaction-aware regularizers: write "the hand is grasping" into the loss without any object prior

2D photometric loss alone is insufficient under heavy occlusion and drastic motion — Gaussians drift into floaters or misplaced blobs wherever the image constraint has no say. Three explicit 3D regularizers are therefore added, all depending only on the hand tracker.

The first is the hand loss, which supervises translation only: it pulls each hand Gaussian to the nearest MANO vertex so the Gaussians populate the hand surface, and to sidestep the arm-versus-hand vertex discrepancy it restricts the nearest-point search to a filtered vertex range \(\mathcal{V}_h\), in Chamfer form \(\mathcal{L}^{H}_{\text{trans}}=\frac{1}{N}\sum_{i}\min_{v\in\mathcal{V}_h}\|\mathbf{p}_i-\mathbf{p}_v\|_2\). The second is the object rotation loss, targeting an asymmetry — object translation can be aligned implicitly through spatial proximity, whereas rotation frequently suffers non-physical flipping under passive contact. A global target rotation \(R^{\text{target}}_{\text{hand}}(t)\) is computed by SVD-based rotation averaging over MANO joint rotations, and each object Gaussian's predicted rotation is penalized by its geodesic distance to that target, gated by the contact weight \(\omega_j(t)=\sigma(w_j^{O})\) so that Gaussians with small \(w\) (occluded or not engaged) are exempt:

\[L^{O}_{rot}=\mathbb{E}_{t}\Big[\frac{1}{N}\sum_{j=1}^{N}\omega_j(t)\big\|\log\big(R^{\text{target}}_{\text{hand}}(t)^{-1}R^{\text{obj}}_{t,j}\big)\big\|\Big]\]

Here \(\log(\cdot)\) is the logarithmic map from SO(3) to its Lie algebra so(3), making this a metric-aware angular penalty rather than a Euclidean one; implausible rotations are suppressed while local deformation freedom is preserved. The third is the interaction loss: since hand and object are explicit point sets \(C_H\) and \(C_O\), a bidirectional Chamfer distance between them can be computed directly (with \(\epsilon=10^{-6}\) to avoid division by zero when no contact is detected), pulling the two contact surfaces together. This term is self-supervised and needs no contact annotation.

One more term accompanies them: a penetration loss. Pulling surfaces together alone would let hand and object stick to each other or interpenetrate, so overlapping or overly close hand-object Gaussian pairs are penalized separately — one pulls, the other pushes, and only together do they form a complete contact constraint (⚠️ the exact form of the penetration loss lives in the supplementary material and is not given in the cached main text). This is also where the paper positions itself relative to HOLD, BIGS and MagicHOI: those either depend on object templates / poses or borrow an external diffusion prior to hallucinate occluded geometry, whereas this work uses coarse hand guidance plus explicit contact constraints only.

4. Progressive optimization: separate the background from the hand-object pair and fit coarse-to-fine

HOI scenes involve violent rotation, translation and occlusion, so optimizing every Gaussian at once converges slowly with misaligned positions, and background versus foreground need completely different update frequencies. The optimization is therefore split into five phases. Initialization: hand Gaussians are cold-started from the tracker's MANO vertices (bootstrap only); the object assumes no shape, category or 3D box prior and is initialized by uniform random sampling inside the expanded axis-aligned bounding box (AABB) of the MANO vertices; background Gaussians come from an SfM sparse point cloud. Warm-up: on top of the 2D losses, the hand translation loss and object rotation loss are applied to first steer hand deformation and object rotation onto the right trend, with periodic gradient-based density adjustments (3D-GS adaptive density control) along the way. HOI refinement: \(w\), \(o\) and the KNN + LBS blend of the previous design are switched on, giving Gaussians in the contact region controlled local freedom. Background optimization: background Gaussians are pretrained alone for a fixed number of iterations with density adjustments, yielding a clean background initialization. Collaborative reconstruction: the three fields deform their Gaussians into one shared target space, jointly supervised by the interaction loss and 2D regularization, delivering physically plausible occlusion, smooth edge transitions and lighting coherence.

What is actually designed here is the ordering: the hand comes first because MANO gives it a cold start; the object follows because it has neither prior nor independent motion under contact and only becomes tractable once hand guidance is in place; the background is pretrained in isolation and joins last because it is the easiest thing to corrupt with foreground high-frequency jitter. As for "lighting coherence", the paper provides no dedicated lighting loss; the reasonable reading is that all three Gaussian sets must eventually be rasterized into the same image under the same photometric supervision, so their contributions to each pixel have to be mutually consistent and shadows and shading are implicitly aligned (⚠️ the paper does not argue this explicitly). One easily overlooked detail is the weak prior baked into object initialization: uniform sampling inside the MANO AABB implicitly assumes the object sits next to the hand, which does not hold for interactions such as pushing or throwing where the object leaves the hand's vicinity.

A Worked Example¶

Walk a "hold and rotate a cup" translation-plus-rotation sequence through the pipeline. After initialization, hand Gaussians sit on the MANO vertices while object Gaussians are scattered inside the expanded hand AABB; neither side carries shape or appearance information yet, so the render is essentially noise. During warm-up the hand translation loss pulls \(G_H\) toward the tracked pose, and the object rotation loss starts bending the contacted object Gaussians toward the hand's rotational trend. In HOI refinement \(w\) and \(o\) activate: by design intent, a smaller \(o\) at the contact boundary makes the neighborhood weighting nearly a hard selection so contours sharpen, while a smaller \(w\) under occlusion keeps those Gaussians closer to their own positions instead of being dragged by neighbors. Background optimization then fits the region Gaussians cleanly and dark-region detail appears. Finally, collaborative reconstruction aligns all three sets in a single rasterization, where the interaction and penetration losses bring hand and object into contact without interpenetration. The whole run takes 21,000 iterations and converges in about 1 hour 20 minutes on a single RTX 3090 (24 GB).

Loss & Training¶

The objective combines a 2D photometric loss (inherited from the 3D-GS lineage; ⚠️ the cached main text says only "2D losses", so the exact form should be checked in the original paper / supplementary material) with the hand translation loss, object rotation loss and interaction loss, plus the penetration and momentum losses described in the supplementary material. Key settings given in the main text: the KNN \(K\) is shared by refinement and deformation; key frame \(k\) is the timestamp just before hand-object contact; Adam optimizes both Gaussians and the deformation model; 21,000 iterations reach the best result in 1 h 20 min on one RTX 3090; hand tracking costs under 3% of total runtime. Evaluation follows EgoGaussian's alternate-frame testing protocol, assessing pure translation and translation-plus-rotation separately to probe extrapolation to unseen interactions.

Key Experimental Results¶

Main Results¶

Comparison on HOI4D against three general dynamic Gaussian baselines (2 pure-translation scenes + 2 translation-rotation scenes):

Method	Translation PSNR↑ / SSIM↑ / LPIPS↓	Translation & Rotation PSNR↑ / SSIM↑ / LPIPS↓
4DGS	24.86 / 0.80 / 0.47	23.68 / 0.85 / 0.39
Deform3DGS	26.33 / 0.87 / 0.29	23.57 / 0.89 / 0.28
SC-GS	25.08 / 0.84 / 0.46	17.32 / 0.71 / 0.48
Ours	30.32 / 0.93 / 0.29	24.16 / 0.86 / 0.37
Ours* (full-frame)	33.03 / 0.95 / 0.27	24.02 / 0.85 / 0.39

Comparison on HO3D (translation-rotation scenes; Ours* is full-frame, † means evaluated on hand and object regions only):

Method	PSNR↑	SSIM↑	LPIPS↓
BIGS	3.85	0.24	0.70
HOLD	18.03	0.84	0.26
4DGS	19.44	0.82	0.25
Deform3DGS	9.68	0.36	0.65
SC-GS	20.37	0.80	0.26
Ours	25.19	0.89	0.15
Ours* (full-frame)	25.17	0.89	0.16
BIGS† (hand + object regions)	24.51	0.92	0.07
Ours† (hand + object regions)	28.16	0.95	0.07

One caveat worth stating: every number the paper reports is a view-synthesis metric (PSNR / SSIM / LPIPS) — there are no explicit geometric metrics such as Chamfer distance or F-score. The closest proxy for geometric quality is the HO3D evaluation restricted to hand and object regions († rows): even under an evaluation protocol that favors BIGS, the proposed method is ahead by 3.65 dB. Hand-region quality is emphasized qualitatively (Fig. 3 and Fig. 4) but is never reported as a separate number.

Ablation Study¶

Ablations on HOI4D Scene 1 (the 32.96 dB "Full Model" differs from the 30.32 dB in the main table because the evaluated sequences differ):

Config	PSNR↑	SSIM↑	LPIPS↓	Note
Full Model	32.96	0.95	0.35	full model
w/o Interaction-Aware Module	28.76	0.91	0.40	\(w\), \(o\) and their training scheme removed; −4.20 dB (−12.7%)
w/o HOI Refinement	32.23	0.94	0.39	PSNR −2.2%, LPIPS −11.4%
w/o Object Loss	31.45	0.94	0.38	PSNR −4.6%, LPIPS −8.6%
w/o Hand Loss	32.45	0.95	0.37	PSNR −1.5%, LPIPS −5.7%
w/o Interaction Loss	31.79	0.94	0.40	PSNR −3.5%, LPIPS −14.3%
w/ noise \(\sigma=0.01\)	32.80	0.95	0.35	Gaussian noise on initial object positions; −0.16 dB
w/ noise \(\sigma=0.05\)	32.72	0.95	0.35	stronger initialization noise; −0.24 dB

Key Findings¶

The interaction-aware module (\(w\), \(o\) plus its training scheme) contributes the most: removing it costs 4.20 dB, far more than any single loss term. The interaction boundary is therefore best modeled by the representation itself, while the three 3D regularizers mainly help convergence.
The losses rank as object rotation loss (−1.51 dB, LPIPS +8.6%) > interaction loss (−1.17 dB, LPIPS +14.3%) > hand loss (−0.51 dB, LPIPS +5.7%), which matches the method's own logic: the object is the weakest link, having neither a prior nor independent motion, whereas the hand is backstopped by MANO initialization and is thus least sensitive to extra supervision. Note that the interaction loss does the most damage to LPIPS (+14.3%), suggesting it mainly buys perceptual quality rather than peak signal fidelity.
Robustness to object initialization noise is high: Gaussian noise with \(\sigma=0.01\) and \(\sigma=0.05\) costs only 0.16 dB and 0.24 dB, which the authors use to argue the method does not depend on precise object initialization.
On HO3D every baseline is dragged down by pose error: Deform3DGS fails to converge (9.68 dB), BIGS reconstructs only foreground hand and object without background so its full-frame number collapses to 3.85 dB, and HOLD targets geometry rather than view synthesis (18.03 dB). On this benchmark, robustness to input noise discriminates methods better than peak performance does.
The main text claims a "+9% PSNR gain" in translation scenes, but Table 1's 26.33 → 30.32 dB is a relative gain of about 15%; the two figures disagree and the paper does not explain the 9% (⚠️ refer to the original paper). The rotation-heavy setting moves from 23.57 dB (Deform3DGS) to 24.16 dB, a much smaller margin, indicating that fast irregular rotation remains the hard case.

Highlights & Insights¶

"Interaction" is pinned down to three actionable places: the hand conditioning of the object deformation field (representation), a rotation loss that uses the hand rotation as prior and is gated by a contact weight (optimization), and bidirectional Chamfer plus penetration losses between the two explicit point sets (constraints). This representation-conditioning / loss-gating split is far more concrete than saying "an interaction prior is introduced", and it explains why removing the \(w\), \(o\) representation hurts most.
\(w\) plays three roles at once: a continuous weight that smooths motion and suppresses noise, a contact weight after sigmoid normalization that gates the rotation loss (\(\omega_j=\sigma(w_j^O)\)), and a soft occlusion indicator. One learnable scalar serving both representation and supervision is a parsimonious reuse.
The tension between Chamfer attraction and penetration penalty is the complete form of a contact constraint. Plenty of HOI work implements only the former and ends up fusing hand and object together; this paper explicitly recognizes that "pulling close is not the same as not interpenetrating". That insight transfers directly to any multi-body contact task represented by explicit point sets or Gaussians (dexterous grasping, joint human-object-scene reconstruction).
A transferable recipe: replace object-prior initialization with uniform sampling inside an AABB, then justify it with a noise-robustness experiment. This "weak-prior initialization + robustness verification" pairing suits any setting that lacks object templates but can tolerate an approximate spatial assumption, such as grasping unknown objects in robotics.

Limitations & Future Work¶

Limitations admitted by the authors: the five-phase progressive workflow is long, and a stronger optimizer could unify the phases; extreme cases (very fast motion, complex trajectories) fail (details in the supplementary material), possibly addressable by incorporating more interaction priors.
Limitations I see: the evaluation scale is small — HOI4D contributes only 2 translation and 2 rotation scenes, HO3D only 4 translation-rotation sequences from camera 4, downsampled to half resolution. A 3.99 dB lead at that scale needs more scenes to be confirmed.
Metrics are narrow: there is no Chamfer distance, F-score or hand joint error, so "geometric fidelity" rests on qualitative renderings and the hand/object-region PSNR, even though jointly modeling geometry and appearance is one of the paper's selling points.
Object initialization still carries a spatial prior: uniform sampling inside the expanded MANO AABB assumes the object lies near the hand. Interactions where the object is away from the hand (throwing, distant manipulation) would break it; the noise-robustness experiment only perturbs positions inside that AABB and never tests a misplaced AABB.
The penetration loss is deferred to the supplementary material and its form is not public, and "physical realism" remains an approximation via Chamfer attraction plus a penetration penalty, with no contact force, friction, or deformable-object modeling. Applicability to deformable or articulated objects is untested (articulated-object work is explicitly positioned as orthogonal).
The hand tracker is an offline dependency: when tracking fails (severe occlusion, motion blur) the hand and rotation losses inject wrong guidance in tandem. The paper argues only that tracking is cheap, not that it is robust to failure.
Directions for improvement: let the three fields exchange a light-weight mutual attention or share a canonical space during the collaborative phase, upgrading conditioning from one-way (hand → object) to bidirectional; replace the single key frame \(k\) with a per-frame contact probability so the object field's conditioning stays valid across the whole sequence.

vs 4DGS / Deform3DGS / SC-GS: they process all Gaussians with one deformation field and are inherently blind to interaction. This paper splits into three fields and conditions the object field on the hand, reaching 30.32 dB on HOI4D translation scenes (Deform3DGS: 26.33 dB); the price is a heavier pipeline (five phases, 1 h 20 min training) and a dependency on the key frame \(k\).
vs EgoGaussian: also an egocentric 3D-GS reconstruction, but it requires object pose estimation, reconstructs only the interactive object, and excludes the hand. This paper needs no object pose and reconstructs the full scene (hand + object + background), at the cost of giving up some object-specific precision.
vs HOLD: HOLD is category-agnostic hand-object reconstruction but targets geometry rather than view synthesis, trailing this work by 7.16 dB on HO3D rendering metrics. There is still no like-for-like comparison on geometric accuracy.
vs BIGS: BIGS reconstructs bimanual interaction from monocular video with 3D-GS plus a diffusion prior, but assumes a known object mesh and ignores the background. Evaluated on hand and object regions only, BIGS scores 24.51 dB against this paper's 28.16 dB — the lead holds even under the more favorable protocol.
vs generative-prior routes such as MagicHOI / G-HOP: they are prior-free too but hallucinate occluded object geometry with a diffusion model, trading flexibility for generative regularization. This paper deliberately uses no external generative model and relies on explicit contact constraints, which is lighter but may not match them in geometric completeness under severe occlusion.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of interaction-aware Gaussian parameters, a hand-conditioned object deformation field and a contact-gated rotation loss is new, though each ingredient (RBF neighborhood weights, LBS blending, Chamfer contact losses) has precedent.
Experimental Thoroughness: ⭐⭐⭐ Ablations are complete (including noise robustness), but there are only two datasets and eight scenes in total, geometric metrics are missing, and HOLD/BIGS are not compared under one common protocol.
Writing Quality: ⭐⭐⭐⭐ The motivation builds up cleanly and the three challenges map onto the three contributions; points are lost because several key equations and the penetration loss are deferred to the supplementary material, and the "+9% PSNR" in the text disagrees with the table.
Value: ⭐⭐⭐⭐ The zero-object-prior, full-scene egocentric HOI setting is practically relevant (VR, teleoperation), and decomposing interaction into representation conditioning plus explicit contact constraints transfers to other multi-body interaction reconstruction tasks.