Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models¶

Conference: ECCV2026
Paper: ECCV Paper
Project: Know3D
Area: Multimodal VLM / 3D Vision
Keywords: image-to-3D generation, backside semantic control, diffusion hidden states, parallel cross-attention, conditional flow matching

TL;DR¶

Know3D injects intermediate backside-generation features from Qwen-Image-Edit into TRELLIS.2 to impose language constraints on unseen geometry, achieving ULIP 0.2174 and Uni3D 0.3518 on HY3D-Bench-test, while backside instruction following is demonstrated primarily through qualitative examples.

Background & Motivation¶

Single-image-to-3D models can generate detailed objects, but the input itself cannot resolve a fundamental ambiguity: a front image does not specify the back. Visible regions can be constrained by image evidence, whereas occluded regions depend mainly on shape priors learned from training data. This creates two distinct failure modes: a completion may look plausible but contradict user intent, or it may violate basic structural knowledge about the object. For example, improving front-view texture fidelity does not automatically provide control over a component on the hidden side. Compared with internet-scale image-text data, high-quality 3D assets and text-3D pairs remain limited, so scaling an existing geometry model does not immediately supply equally rich semantic knowledge.

A vision-language model (VLM) has broader object knowledge, but transferring that knowledge to geometry generation requires more than attaching a text prompt to the input. Autoregressively generating 3D representations with a language model introduces a representation space different from its original semantic tasks; the paper argues that geometry quality still trails dedicated 3D generators. Directly injecting VLM hidden states presents another difficulty: abstract semantic features lack explicit image-space structure and may align poorly with the spatial conditioning required by a generator. Generating a backside image and using it as a second input is intuitive, but an incorrect component count or pose can then become evidence that the geometry model tries to reproduce faithfully. The question is therefore not how to replace a geometry model with a language model, but how to obtain an intermediate representation with both semantic content and spatial organization.

Qwen-Image-Edit offers a natural connection: its VLM interprets images and instructions, while its diffusion Transformer turns that interpretation into spatially organized image-generation states. Those states need not be decoded into final pixels before they become useful for 3D generation. Core Idea: first adapt an image editor to generate the opposite view without changing the subject's pose, then inject its intermediate structural-semantic features into a dedicated 3D generator to control otherwise unseen geometry through language.

Method¶

Overall Architecture¶

The inputs are one object image and an optional description of backside components; the output is complete 3D object geometry. The framework consists of semantic-aware backside generation, diffusion hidden-state extraction, and parallel conditioning injection. Qwen2.5-VL processes the image and instruction, while a VAE encodes the input image; together they condition the MMDiT in Qwen-Image-Edit-2511. At a selected timestep in backside denoising, features from multiple intermediate layers are extracted for both the coarse-structure and fine-geometry stages of TRELLIS.2. The original front-image condition remains in place: backside knowledge supplements visible evidence instead of replacing it.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Single image and optional<br/>backside description"] --> BACK["Semantic-aware<br/>backside generation"]
    PAIRS["Training: real front-back views<br/>and component descriptions"] -.-> BACK
    BACK --> HIDDEN["Diffusion hidden-state<br/>extraction"]
    HIDDEN --> INJECT["Parallel conditioning<br/>injection"]
    INPUT -->|Original image condition| INJECT
    INJECT --> COARSE["Sparse structure generation"]
    COARSE --> FINE["Fine geometry generation"]
    INJECT -->|Structural-semantic condition| FINE
    FINE --> OUTPUT["Complete 3D geometry"]

The real front-back pairs in the diagram provide training supervision; ordinary inference does not require the user to supply a real back view. The backside generator is adapted first and frozen during 3D training, so this is not end-to-end joint updating of every foundation model. Extracting intermediate states also does not establish that image denoising can be stopped early; the paper still describes a full iterative denoising process.

Key Designs¶

1. Semantic-aware backside generation: calibrate viewpoint before learning component control

A general image editor can recognize objects and interpret text without reliably preserving correspondence under a camera rotation. When asked for a back view, it may choose the wrong viewpoint or change the original pose of the person or object. Know3D therefore first renders front-back pairs from real 3D assets to train viewpoint changes that preserve the same subject. Azimuths are sampled uniformly for each asset and opposite views are paired; here, the front view means the selected conditioning view, not necessarily the object's canonical front. A VLM observes the paired images and describes salient backside components individually instead of supplying only a generic object-level category label. This supervision granularity lets text refer to specific hidden content while allowing different components to be specified separately.

Every training prompt contains a fixed instruction for a 180-degree camera rotation, with a description sampled from the component set added with probability 0.5. Examples without an additional description teach default completion, while examples with one teach requested completion; both remain conditioned on the input image. The paper's term unconditional backside generation should thus be read as generation without an additional backside semantic constraint, not generation without any condition. Qwen2.5-VL supplies multimodal hidden states, and the front-view VAE latent retains spatial appearance information about the visible subject. Together they condition MMDiT's latent backside-generation process, so text does not act as a separate editor applied to an already generated mesh. The key is to learn both what belongs on the hidden side and how to observe the same subject from the opposite direction within one task.

2. Diffusion hidden-state extraction: use structure formation rather than final pixels alone

After adapting the backside generator, the next question is which representation to pass to the 3D model. One candidate is the fully denoised VAE latent; another decodes that latent into an image and then extracts DINOv3 features. The former emphasizes pixel reconstruction, while the latter adds decoding and visual encoding and may treat incorrectly generated local appearance as reliable conditioning. Know3D selects a third representation: hidden states from multiple intermediate MMDiT layers at a chosen denoising timestep, concatenated into a structural-semantic condition. These are not simply concatenated VLM language tokens; the features have participated in spatial image generation and can therefore carry both object knowledge and layout cues.

The paper expresses this multi-layer aggregation as concatenation; the following relation is sufficient to identify its information sources:

\[ H_{\mathrm{DiT}}=\operatorname{Concat}\left(h^{(1)},h^{(2)},\ldots,h^{(n)}\right). \]

Each hidden state comes from one MMDiT layer at the same selected timestep; the supplied main text does not specify the exact layer count or indices. The timestep ablation compares \(t\in\{0,0.25,0.5,0.75\}\), with \(t=0.25\) producing the best sparse-voxel results. The authors suggest that such states have formed global layout and major semantic components without becoming committed solely to final pixels. This is an interpretation supported by an ablation, not a theorem that the hidden states necessarily encode correct 3D structure. The value of \(t\) should not be treated directly as the percentage of sampler execution completed; the noise-interpolation parameter and inference execution order are different concepts.

3. Parallel conditioning injection: retain front-view priors while adding backside information

TRELLIS.2 retains its original self-attention and image-conditioned cross-attention, with a new branch running in parallel to the existing image-conditioning branch. The concatenated diffusion features undergo linear projection and layer normalization before serving as keys and values in the added cross-attention. Queries come from the 3D backbone features, allowing the generator to select backside information according to its current geometry state instead of overwriting the representation indiscriminately. The new branch passes through a zero-initialized linear layer before fusion with the existing path. This initialization prevents the added condition path from introducing a nonzero perturbation at the start of training, supporting stable adaptation of pretrained generation priors. It does not mean that the original network stays completely frozen: TRELLIS.2 is subsequently adapted through LoRA.

Geometry generation retains two stages: generate a coarse sparse structure from noise, then recover high-fidelity fine geometry conditioned on that structure. Both stages receive the original front-image features and the diffusion structural-semantic features; the coarse stage handles global topology before the fine stage restores local geometry. This order matters because a backside component can change the object's connectivity, which cannot always be resolved by decorating the final surface. The approach also differs from treating the generated back view as a second photograph: its extra input is an intermediate condition that can be interpreted through learning, rather than an observation to reproduce pixel by pixel. The authors illustrate this distinction with an incorrectly generated two-strap bag, but a single example cannot establish robustness to every erroneous condition.

A Worked Example¶

Consider the single-shoulder bag discussed in Figure 7 and on pages 14–15: the input shows a single-shoulder bag, while the back remains unobserved. The backside generator may incorrectly synthesize two straps, yielding an interpretation inconsistent with the front-view semantics when the generated image is considered alone. When that image is encoded with DINOv3 and passed to the geometry model, the authors observe a tendency to fit the incorrect result. In this example, the VAE representation fails to inject effective backside information, producing an implausible structure similar to the original TRELLIS.2 output. Know3D uses intermediate diffusion states, which the paper reports can still produce more reasonable geometry instead of directly copying the erroneous image. The example illustrates how the representation affects error propagation; it is not a quantitative strap-counting accuracy experiment. For language control, the user supplies a component description, which influences the 3D result through the same backside-generation and feature-injection path; Figure 4 demonstrates this capability.

Loss & Training¶

Backside generation and both 3D stages use conditional flow matching (CFM): time-interpolated noise is applied to the target latent, and the model learns the corresponding velocity field. The backside stage is supervised by the real back image's VAE latent, whereas the 3D stages use ground-truth geometry latents; their conditions and target spaces differ. Equations (1)–(5) in the supplied text extraction have damaged concatenation symbols, subtraction signs, and norms, so reconstructed full losses are not presented here as exact author equations. The prose establishes the conditioning sources, the velocity-prediction objective type, and the staged optimization, rather than introducing a separate semantic classification loss.

Backside training uses 5k high-quality TexVerse assets, with 12 uniformly spaced azimuthal views per asset forming 6 annotated front-back pairs. FoV is sampled from 35, 50, 85, 105, and 135 degrees, with elevation between -15 and 45 degrees; these are fixed within each asset but randomized across assets. This stage uses rank-64 LoRA on 32 NVIDIA A800 GPUs with a global batch size of 32. 3D training uses 60k TexVerse meshes, each with 4 unperturbed view pairs and 4 pairs perturbed through FoV scaling and small angular offsets. The adapted Qwen-Image-Edit is frozen, TRELLIS.2 uses rank-64 LoRA, and the newly added conditioning layers are fully trained. The 3D stage also runs on 32 A800 GPUs, with a global batch size of 64; that section does not state the full model's total training steps. The 70k-step figure belongs specifically to the first-stage-only ablations below and must not be reported as the complete system's training budget.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1, page 13, on the test and validation splits of HY3D-Bench; ULIP and Uni3D measure semantic consistency between the input image and generated mesh, with higher values better. Hunyuan3D-2MV receives the original front view and a synthesized back view, whereas the other listed baselines are single-image-to-3D methods; conditioning is not identical across every method.

Method	Test ULIP ↑	Test Uni3D ↑	Val ULIP ↑	Val Uni3D ↑
TRELLIS	0.2143	0.3421	0.2133	0.3464
TRELLIS.2	0.1948	0.3308	0.1967	0.3358
Hunyuan3D-2MV	0.2108	0.3413	0.2111	0.3400
Hunyuan3D-2.1	0.2140	0.3434	0.2122	0.3433
Know3D	0.2174	0.3518	0.2127	0.3512

Know3D improves test Uni3D by 0.0084 over Hunyuan3D-2.1 and test ULIP by 0.0031 over TRELLIS. Its validation ULIP is 0.2127, below TRELLIS at 0.2133, so the result should not be summarized as winning every metric. These scores measure overall image-mesh semantic consistency, not backside instruction following, physical manufacturability, or local topology accuracy directly.

Ablation Study¶

The next two tables come from Tables 2 and 3 on page 14; evaluation uses 100 TexVerse assets excluded from training. Only the first sparse-voxel generation stage is trained, using 60k training samples, batch size 64, 70k steps, and rank-64 LoRA. Crucially, these comparisons directly use ground-truth front and back views as conditions, so they do not measure every benefit of the complete generated-backside pipeline at test time. IoU measures the intersection over union of predicted and ground-truth occupancy sets and is higher-is-better; CD is Chamfer Distance between geometric point sets and is lower-is-better. The tables do not specify CD scaling or units, so the original values are retained without interpreting them as meters or millimeters.

MMDiT timestep \(t\)	IoU ↑	CD ↓
0.0	0.343	2.376
0.25	0.352	2.262
0.5	0.349	2.272
0.75	0.336	2.452

In Table 2, \(t=0.25\) beats the other three settings, but the difference from \(t=0.5\) is only 0.003 IoU and 0.010 CD, with no reported error bars. This supports the importance of extraction timing, not a universal claim that 0.25 is optimal for every foundation model and dataset.

Feature source	IoU ↑	CD ↓
VAE encoder	0.308	2.803
DINOv3	0.342	2.385
MMDiT, \(t=0.25\)	0.352	2.262

In Table 3, MMDiT improves IoU by 0.010 and reduces CD by 0.123 relative to DINOv3; its IoU gain over VAE is 0.044. The table names the first source VAE encoder, while the method comparison describes a fully denoised VAE latent; the original label is retained, and the exact extraction path should be checked for reproduction.

Key Findings¶

Intermediate diffusion features outperform the other two representations in this controlled-conditioning sparse-structure experiment, making the information interface itself a meaningful research variable.
The main comparison supports strong semantic consistency but provides no separate backside text-following score; that capability should be read alongside the qualitative demonstrations in Figure 4.
Figure 7 supports the possibility that intermediate features need not mechanically copy erroneous pixels; it does not measure a systematic error rate or a robustness bound.

Highlights & Insights¶

The diffusion model serves as both a generator and a cross-modal representation interface. Taking features before final pixels are settled provides a concrete way to preserve semantic flexibility.
Added cross-attention retains the original image-conditioning channel. A zero-initialized output layer learns supplementary knowledge from an initially non-perturbing state, which is easier to interpret than replacing the original condition.
Random inclusion of component descriptions unifies default completion and semantic control. The same interface works without backside requirements and accepts explicit constraints when the user has them.

Limitations & Future Work¶

The authors explicitly acknowledge that a multimodal foundation model can still misinterpret instructions and mislead geometry generation; semantic knowledge does not replace reliable spatial verification.
From an evaluation perspective, coarse-stage ablations on 100 assets are not equivalent to complete mesh generation, and end-to-end quantitative tests with synthesized-backside errors are still needed.
The supplied main text reports no backside instruction success rate, user study, or explicit inference latency, so comprehensive validation of controllability and efficiency cannot be claimed.
A testable next step is to measure backside component compliance together with front-geometry preservation and stress-test increasingly incorrect backside conditions; this is a reader proposal, not an experiment completed in the paper.

vs TRELLIS.2: Know3D retains sparse-structure-to-fine-geometry generation and adds a multimodal diffusion conditioning branch rather than introducing a new mesh representation.
vs Hunyuan3D-2MV: The baseline takes a synthesized back image as a second visual input, while Know3D uses hidden features; because the backbones differ, the main-table gap cannot be attributed entirely to representation choice, making Table 3's controlled comparison more targeted.
vs MeshLLM and LLaMA-Mesh: These directions explore direct mesh generation with language models, whereas Know3D retains a dedicated geometry generator and uses the VLM as a source of transferable semantic priors.
Research implication: The choice between final-image conditioning and intermediate-generation-state conditioning could extend to other occluded-geometry completion tasks, but any transfer benefit requires fresh evaluation.

Rating¶

Novelty: 4/5, explicitly compares and uses multimodal diffusion hidden states as an interface for backside geometry control.
Experimental Thoroughness: 3/5, includes main comparisons and feature/timestep ablations but lacks direct instruction-following quantification and complete-system ablations.
Writing Quality: 4/5, clearly connects the problem and method; damaged equations in the supplied extraction and missing implementation details impede exact reproduction.
Value: 4/5, offers a reusable approach to semantic control of unseen regions, bounded by the foundation model's understanding and generation capabilities.