Beyond Prompts: Unconditional 3D Inversion for Out-of-Distribution Shapes¶

Conference: ECCV 2026
Paper: ECCV 2026 official page / project page
Area: 3D Vision
Keywords: 3D generative model inversion / empty-prompt inversion / text-guided 3D editing / rectified flow / out-of-distribution shapes

TL;DR¶

The paper shows that native text-to-3D generative models (TRELLIS) fall into a "sink trap" where they become insensitive to out-of-distribution text conditioning, so it replaces the source prompt used for inversion with an empty prompt and anchors the shape purely in the unconditional prior — cutting the inversion reconstruction error on non-rigid shapes from L1 17.75 to 5.40 and unlocking the first 3D editing pipeline driven only by a target prompt, with no masks and no 2D priors (SigLIP 0.1469, about 1.8x the strongest baseline, averaging 9 seconds per edit).

Background & Motivation¶

Inversion — mapping a real sample back into the noise space of a generative model — is the standard route to editing: once a sample becomes a latent the model can manipulate, editing reduces to "change one prompt and sample again." 2D image editing is lightweight precisely because this route works (Null-Text Inversion and the line of work that followed). 3D has never managed to copy it. SDS-style methods (the DreamFusion lineage) optimize each asset through gradients distilled from a 2D diffusion model, which is slow and expensive; multiview pipelines first generate consistent edited views in image space and then reconstruct back to 3D to restore global consistency; more recent works such as VoxHammer and Nano3D bypass the 3D model's internal latents altogether and edit the conditioning input with a 2D inpainting model, which forces the user to draw masks by hand (VoxHammer additionally renders 150+ views to extract DINOv2 features). In other words, no existing 3D editing pipeline relies exclusively on the native capability of a 3D generative model — all of them bolt on image priors.

This paper asks whether that is simply because 3D generative models are not good enough, and answers no: the culprit is the text conditioning itself. Using SigLIP embeddings to measure how much the geometry changes when the prompt changes (the ratio \(\Delta_{\text{vis}}/\Delta_{\text{txt}}\)), the authors find that TRELLIS responds very unevenly across semantic categories. For subjects such as astronaut, labrador, or scary wolf the ratio drops well below 1, meaning the model emits nearly identical geometry and texture no matter how the prompt is written, as if the language pathway were short-circuited; the authors call this the sink trap. Inversion, however, requires a prompt that describes the source shape. In practice the user only has the observed shape, so that prompt is necessarily approximate — and in these low-density regions "the correct prompt" does not exist at all, which forces the approximate prompt out of distribution and makes the inversion collapse.

A deeper piece of evidence comes from the sampling trajectory itself. Tracking the norm of the predicted velocity \(\|v_t\|\) across timesteps, an approximate prompt inflates it substantially, signalling that the model is trying to escape a low-density region; run the same experiment on 2D flow models such as FLUX.1 and the velocity norm barely moves across prompt types. Since geometric expressivity is clearly sufficient and the guidance term is what breaks, removing that term entirely — inverting with an empty prompt so the trajectory follows only the unconditional velocity field — should be both stable and accurate. Core idea: replace the anchor of inversion, "a text prompt describing the source shape," with "the unconditional generative prior under an empty prompt," decoupling geometric fidelity from linguistic sensitivity first and re-attaching the edit prompt only on top of that stabilized trajectory.

Method¶

Overall Architecture¶

The problem the paper solves is: put an arbitrary 3D mesh into the latent space of a text-to-3D flow model, then modify it according to a new text prompt. The pipeline has only two steps, but the interface between them is what matters. The first, inversion, turns the mesh into a "noise latent plus unconditional embedding" that the generative model can manipulate; the second, editing, starts from that latent and re-samples while looking only at the edit prompt, yielding the modified asset. The input is an arbitrary 3D mesh \(X\) (validated on non-rigid characters and animals from DT4D), voxelized and then encoded by TRELLIS's pretrained VAE into a structure latent \(z_0\); the output is the edited 3D asset. Compared with the standard recipe from the 2D literature, exactly one thing is replaced: the condition used during inversion is not a prompt describing the source shape but the empty prompt \(\varnothing\). Every conclusion that follows — more stable trajectories, higher reconstruction fidelity, editing without masks or 2D priors — grows out of that single choice.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input mesh X<br/>voxelize + VAE encode → z0"] --> B["Empty-Prompt Inversion<br/>forward Euler under ∅"]
    B -->|optional structural refinement| C["Unconditional Embedding Refinement<br/>NTI optimizes the null embedding"]
    B --> D["Inverted noise latent z1"]
    C --> D
    D -->|edit prompt P_edit| E["Same-Source Editing<br/>re-sample from z1"]
    E --> F["Output: edited 3D asset"]

One further scope restriction is inconspicuous but important: the paper performs inversion only on TRELLIS's first-stage structure model \(G_S\) (low-dimensional structure latent to a \(64^3\) coarse voxel structure), while the second-stage SLAT model \(G_L\) stays frozen with a fixed random seed. The reason is that \(G_L\) only changes texture, whereas a geometric edit can only happen in the latent space of \(G_S\); pinning the texture stage down also keeps every comparison attributable to the geometry branch, ruling out texture randomness as a confounder.

Key Designs¶

1. Empty-prompt inversion: anchor the source shape in the unconditional prior instead of an out-of-distribution prompt

Inversion normally takes a pair — a shape plus a prompt \(\tilde C\) describing it — and each forward Euler step of the rectified flow uses the CFG-modified velocity field \(\hat v_\theta(z_t,t,C)=v_\theta(z_t,t,\varnothing)+w\,(v_\theta(z_t,t,C)-v_\theta(z_t,t,\varnothing))\). The problem is that when \(C\) is an approximate prompt sitting in a sparse region of the joint embedding space, the guidance difference on the right becomes a high-variance, unstable gradient. It pushes the inversion trajectory off the training distribution, the error accumulates over 50 steps, and the decoded geometry collapses. The empty prompt removes that difference term entirely, reducing inversion to a plain forward integration along the unconditional velocity field:

\[z_{t_{i+1}} = z_{t_i} + (t_{i+1}-t_i)\,\hat v_\theta(z_{t_i}, t_i, \varnothing)\]

Table 2 is the most direct evidence for this design: on TRELLIS the average L1 of empty-prompt inversion is only 5.40, while an approximate prompt blows up to 76.55. More counterintuitively, even the ground-truth prompt is far worse than the empty prompt (17.75), which means the bulk of the reconstruction error comes not from how well the prompt is written but from the text-conditioning pathway itself being unreliable in the 3D latent space. In the same table, Stable Diffusion v1.4 and FLUX.1 show almost no difference across the three prompt types (PSNR hovering around 15), confirming that this is not a generic diffusion/flow property but something specific to 3D generative models. The same holds for LN3Diff and GaussianAnything (L1 dropping from 1.75/1.82 to 0.05 and from 0.32/0.99 to 0.19), so the phenomenon is not an artifact of a single backbone.

Unconditional inversion also comes with a free structural refinement. NTI optimizes exactly the unconditional embedding along the inversion trajectory, which pairs naturally with an empty prompt: keep the optimized null embeddings from the inversion pass and reuse them during sampling, and the reconstruction sticks even closer to the original. The paper itself notes, however, that under \(\varnothing\) the gap between plain Euler inversion and NTI is already small (Supp. Table 1), so the gain comes mainly from removing the text condition rather than from the optimization — NTI is optional here, not essential.

2. Same-source editing: re-sample from the inverted noise with an edit prompt, no masks and no 2D priors

Once inversion is stable, editing returns to the simple form originally envisaged: swap the condition from \(\varnothing\) to the edit prompt \(P_{\text{edit}}\) (e.g., "cowboy dancing"), re-integrate from the inverted noise latent \(z_1\), and decode. Optionally the unconditional embedding can be optimized once more in the NTI style so that large semantic jumps still preserve the source pose and structure. The pipeline is clean because it stays entirely inside the 3D generative model's own latent space: multiview consistency of the edits is a by-product (verified by the 360-degree renders in Fig. 8) rather than something that needs a multiview diffusion plus reconstruction step, and no 2D image prior or manual mask is required. The cost is one voxelization plus three generative sampling passes, averaging 9 seconds per edit, against more than an hour of per-scene optimization for Vox-E and manual masking plus 150+ rendered views for VoxHammer. This is also the paper's boldest claim: to the authors' knowledge it is the first 3D shape editing pipeline driven solely by a target prompt, without auxiliary models or image priors, and the first to delete the "inversion prompt" that 2D work treats as a given.

3. Diversity ratio R: turning "language responsiveness collapse" into a measurable diagnostic

Throughout the discussion so far, "this class of subjects is unresponsive to prompts" has been asserted qualitatively; this design gives it a definition. Around a semantic anchor (astronaut, labrador, ...) an LLM generates a set of prompts, each producing one asset; every asset is rendered over a 360-degree orbit, and SigLIP image/text encoders compute average pairwise cosine distances: \(\Delta_{\text{vis}}\) measures how far apart the assets are visually, \(\Delta_{\text{txt}}\) how far apart the prompts are linguistically, and their ratio

\[R = \frac{\Delta_{\text{vis}}}{\Delta_{\text{txt}}}\]

characterizes the model's responsiveness to language (⚠️ the definition and values of \(\Delta\) in the original table are partly scrambled in the cached text extraction; the formula is reconstructed from the paper's description — refer to the original). An \(R\ll 1\) means the text moved a lot while the geometry barely moved, i.e., the model operates in a sparse region of the joint embedding space. The authors note that this ratio is closely related to the Lipschitz constant of the generative model with respect to text inputs, the caveat being that cosine distance is not strictly a metric. Table 1 shows TRELLIS with \(R<1\) for most subjects (e.g., Cute puppy 0.515, Surgeon 0.990), while densely covered classes such as Dancing girl and Car exceed 1 (1.205 and 1.124); LN3Diff and GaussianAnything likewise report \(R<1\) for most categories, so uneven text responsiveness is a property of this generation of 3D generative models rather than an isolated case. This diagnostic explains why inversion cannot be anchored on a prompt: users do not have the ground-truth prompt for a shape, and wherever \(R\) is low, any approximate prompt steers the inversion onto a bad trajectory.

A Worked Example¶

Take a human mesh from DT4D in a dancing pose. Voxelize it and encode it with the VAE to get the \(8\times16\times16\times16\) structure latent \(z_0\) of \(G_S\); then run forward Euler integration with CFG scale 5 over 50 steps, running 10 inner Adam iterations (lr \(1\times10^{-4}\)) per step to optimize the unconditional embedding, and obtain the noise latent \(z_1\). At this setting the reconstruction L1 sits in the 5.40 range, whereas inverting the same shape with the approximate prompt "a dancing person" degrades into the 76.55 range, with visibly fused limbs and broken surfaces after decoding. Swap the condition to "cowboy dancing", integrate 50 steps back from \(z_1\), and about 9 seconds later the asset comes back with the original dancing pose and a cowboy as its subject. The same setting fails on extreme poses: when the input pose is itself geometrically far from the distribution (the figure lying on the floor in Fig. 9) and the edit is a large semantic jump, the output geometry becomes implausible — a failure mode the authors explicitly acknowledge.

Loss & Training¶

No model is trained; the whole pipeline is training-free and runs at inference time, using the public TRELLIS weights for both the geometry and texture stages. The only optimization appears in the NTI refinement: at each step of the inversion trajectory a few inner iterations adjust the unconditional embedding so that the latent reconstructed from it matches the corresponding latent on the inverted trajectory. The paper uses Adam with a learning rate of \(1\times10^{-4}\) and 10 inner iterations per sampling step, with 50 steps for both inversion and sampling at CFG scale 5, all on a single L40S GPU. For evaluation, reconstruction uses an L1 metric on the decoded latents (\(\mathcal{L}_1=|\hat z_0 - D(E(z_0))|\), which explicitly discounts the VAE's own reconstruction bias) plus LPIPS between rendered normals over 10 viewpoints, while editing is scored by rendering 30 views per edited sample and measuring SigLIP, CLIP, and LLaVA-1.5-7B (1-5 scale).

Key Experimental Results¶

Main Results¶

Editing quality on 200 non-rigid humanoid/animal characters from DT4D, with 30 rendered views per sample († marks methods that require image-based priors):

Method	SigLIP ↑	CLIP ↑	VLM ↑	Note
TRELLIS.1	0.0797	0.2489	2.65	re-runs only stage two; essentially texture-only updates
VoxHammer	0.0240	0.2207	1.33	needs manual masks; fails on 67% of DT4D samples
Vox-E †	0.0250	0.2430	1.84	per-scene optimization, over 1 hour per edit
Instant3DiT †	0.0405	0.2434	1.99	needs manual masks
Ours	0.1469	0.2829	3.98	9 s per edit on average, no masks

Ablation Study¶

Effect of prompt type on inversion reconstruction quality (2D models are included as controls in the same table; TRELLIS/LN3Diff/GaussianAnything are scored by L1 on decoded latents, the 2D models by PSNR/LPIPS):

Model	Metric	True Prompt	Approximate Prompt	Empty Prompt ∅
Stable Diffusion v1.4	PSNR ↑	15.00 ± 2.43	14.61 ± 3.45	15.66 ± 2.57
Stable Diffusion v1.4	LPIPS ↓	0.56 ± 0.07	0.59 ± 0.08	0.57 ± 0.07
FLUX.1 dev	PSNR ↑	10.32 ± 2.55	10.62 ± 6.12	15.12 ± 3.33
FLUX.1 dev	LPIPS ↓	0.57 ± 0.169	0.58 ± 0.19	0.48 ± 0.13
TRELLIS	L1 ↓	17.75 ± 34.59	76.55 ± 73.35	5.40 ± 14.15
LN3Diff	L1 ↓	1.75 ± 0.98	1.82 ± 1.16	0.05 ± 0.07
GaussianAnything	L1 ↓	0.32 ± 0.45	0.99 ± 0.47	0.19 ± 0.06

Text-responsiveness diagnostic across subjects (the cleanly extracted part of Table 1; ⚠️ the original table is two-column and some rows are scrambled in the cached text extraction — subjects not listed here (Husky / Scary wolf / Astronaut / Labrador) should be read from the original):

Subject	Δvis	Δtxt	R (TRELLIS)	R (LN3Diff)	R (GaussianAnything)
Cute puppy	0.196	0.376	0.515	0.806	0.613
Surgeon	0.377	0.394	0.990	0.970	0.538
Dancing girl	0.512	0.420	1.205	0.626	0.569
Car	0.404	0.367	1.124	0.417	0.452

Key Findings¶

The empty prompt not only beats the approximate prompt, it beats the ground-truth prompt on TRELLIS (L1 5.40 vs 17.75). Most of the reconstruction error therefore comes from the text-conditioning pathway itself rather than from prompt quality — the direct opposite of the 2D lesson that "any reasonable and editable prompt" suffices.
Under \(\varnothing\), Euler inversion and NTI land at similar accuracy (Supp. Table 1), so the jump comes from removing the out-of-distribution guidance term, not from the NTI optimization; NTI is a bonus. Combined with the nearly flat 2D columns in Table 2, this pins the problem on 3D generative models' text branch rather than on the inversion algorithm.
The effect reproduces across models: LN3Diff's L1 drops from 1.75/1.82 to 0.05 and GaussianAnything's from 0.32/0.99 to 0.19, so this is not a TRELLIS-specific quirk.
The editing gap is even larger than the reconstruction gap: SigLIP 0.1469 is about 1.8x the strongest baseline TRELLIS.1 (0.0797), and VLM 3.98 vs 2.65. VoxHammer suffers latent value explosion from inversion drift and fails outright on 67% of DT4D samples; even when it succeeds it often needs manual masks and struggles to change the subject while keeping the original pose.
The efficiency gap is also an order of magnitude: 9 s per edit here (voxelization plus three generative sampling passes) versus over an hour for Vox-E.
The failure mode is explicit: when the input pose is geometrically far from the distribution and the edit is a large semantic jump, the editing trajectory still becomes unstable (Fig. 9) and the output geometry is implausible. The paper gives no quantitative boundary for "how far is far."

Highlights & Insights¶

It deletes the "source prompt" that 2D inversion treats as a given, and the evidence does not come from tuning — an empty prompt beats even the ground-truth prompt. That counterintuitive fact carries the paper, and it also explains why purely prompt-driven 3D editing had not worked before.
Using the velocity-norm sequence \(\|v_t\|\) as an out-of-distribution detector transposes the 2D idea of score-norm OOD detection onto the velocity field of a rectified flow, giving an almost free "is this prompt in distribution?" signal. It transfers directly to any flow-matching latent space, not just 3D.
Inverting only the geometry stage \(G_S\) while freezing the texture stage \(G_L\) looks like a purely engineering choice, but it solves two things at once: it makes the edit genuinely geometric (otherwise only texture would change) and it keeps every comparison attributable.
Because editing happens in the native 3D latent space, multiview consistency is a structural by-product rather than an objective that needs an extra consistency loss — a sharp contrast with the multiview paradigm of "generate first, fix consistency later."
The diagnostic and the method come from the same observation (sparse regions plus an out-of-distribution guidance term), which is why the method is minimal: swap in an empty prompt and change nothing else, with no new module added. Such subtractive contributions are rare.

Limitations & Future Work¶

The authors acknowledge that the method is fundamentally bounded by the generative model's distribution: unconditional inversion recovers the structure of highly out-of-distribution shapes, but the subsequent editing trajectory can still become unstable for extreme poses; nothing in the optimization enforces geometric plausibility, so large semantic shifts can produce unrealistic results.
Their proposed directions are to automatically detect out-of-distribution and low-diversity states (naturally, by turning \(R\) and the velocity norm into online signals), and to apply inversion to new downstream tasks, for example adapting image morphing techniques to 3D shapes for shape analysis.
Limitations I see: the experimental scale is small (200 DT4D shapes, 80 TRELLIS-generated shapes), and the baselines fail en masse on DT4D (VoxHammer on 67% of samples), so there are few head-to-head comparisons — the difficulty of the comparison is not symmetric. Editing quality rests entirely on automatic metrics (SigLIP/CLIP/LLaVA-1.5-7B) with no human preference or user study. The main experiments cover only the first stage of one backbone; the newer TRELLIS.2 and its image-to-3D branch are out of scope. And how far "faithful reconstruction under unconditional inversion" really extends for extremely out-of-distribution shapes is never given a quantitative boundary.
Concrete improvements: combine \(R\) and the velocity norm into an online gate that adapts the inversion mode and edit strength to the current shape/prompt out-of-distribution level; add structure-preserving constraints to editing sampling (correspondences, partial noise write-back, pose anchoring) to address the Fig. 9 failure mode; and scale the diagnostic to more generative models and semantic categories, plotting \(R\) against reconstruction L1 so that "sparse region" becomes a predictable quantity rather than a qualitative statement.

vs Null-Text Inversion (NTI): NTI optimizes unconditional embeddings to improve reconstruction fidelity, on the assumption that any "reasonable and editable" source prompt works. This paper shows that assumption fails for 3D generative models — not only approximate prompts but even the ground-truth prompt is worse than an empty prompt — so NTI's optimization target (the unconditional embedding) naturally merges with the empty-prompt strategy.
vs SDS / Vox-E-style 2D-distillation editing: they back-propagate gradients in image space and optimize per asset, taking hours per edit and requiring image priors; this paper operates entirely inside the 3D generative model's latent space, finishes in seconds, and needs no 2D model.
vs VoxHammer / Nano3D: they never touch the internal latents and instead modify the conditioning input with 2D inpainting, which requires manual masks and fails broadly on non-rigid DT4D data (latent explosion); this paper manipulates the internal latent representation with no masks and no 2D priors.
vs TRELLIS's native editing: the native route re-runs only the second stage \(G_L\), which is texture swapping with almost no geometric change; this paper injects the editing signal into the first stage \(G_S\), which is what makes "change the subject, keep the pose" retargeting possible.
vs multiview inversion (e.g., 3D-LATTE) and image-domain flow inversion / inversion-free editing (FlowEdit, UniEdit, ...): the former inverts in multiview space and leans on 2D backbones for mask selection and geometry optimization; the latter avoids iterative mapping but still needs a source prompt describing the original content. This paper lands exactly in the gap between them: native 3D latent space, no source prompt needed.

Rating¶

Novelty: ⭐⭐⭐⭐ First to delete the inversion prompt from the 3D inversion pipeline, with a sink-trap diagnostic that gives a mechanistic explanation; the conclusion is counterintuitive but well supported.
Experimental Thoroughness: ⭐⭐⭐ The diagnostic and reconstruction conclusions are validated across three 3D generative models, but the editing main results are TRELLIS-only, the datasets are small, and there is no human evaluation or quantitative boundary analysis.
Writing Quality: ⭐⭐⭐⭐ The diagnostic-to-mechanism-to-method logic is clear and figures match the claims tightly; some tables are cramped due to the two-column layout (small tables are hard to read).
Value: ⭐⭐⭐⭐ It supplies a reusable judgment — a 3D generative model's language branch may simply be unusable — plus a training-free, mask-free, seconds-level 3D editing route that later work on 3D editing and inversion can build on directly.