JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising

Conference: ECCV2026
arXiv: 2606.20563
Project Page: https://siang1105.github.io/JanusMesh.github.io/
Code: None
Area: 3D Vision
Keywords: 3D Visual Illusion, Zero-Shot Generation, Cross-Space Denoising, SDF Fusion, View-Constrained Texturing

TL;DR

JanusMesh proposes a zero-shot, two-stage framework that leverages dual-branch denoising and SDF fusion in the TRELLIS structured 3D latent space to generate multi-semantic geometries, followed by view-constrained texture synthesis to assign corresponding appearances to different viewpoints, producing high-quality 3D visual illusion meshes in just 3 to 5 minutes.

Background & Motivation

3D visual illusion—where a single 3D object exhibits completely different semantics from different viewpoints (e.g., a peacock from the front and a pineapple from the back)—is a fascinating yet highly challenging problem. While diffusion models have recently made 2D visual illusions (such as Visual Anagrams) possible, extending them to actual 3D meshes presents severe bottlenecks: SDS-based optimization methods (e.g., Shape from Semantics) require per-shape optimization on the order of 40 minutes and generate heavily oversaturated colors, while naive "direct concatenation" methods (generating two objects separately, then cutting and stitching them in the middle) expose obvious geometric discontinuities and back-face semantic leakage at the seams, disrupting the integrity of the illusion.

The core challenge of this dilemma lies in the fact that 3D visual illusions require a single mesh that is geometrically seamless yet semantically viewpoint-separated. However, existing approaches either slowly search in the optimization space (the SDS scheme) or fail to resolve geometric continuity during spatial stitching. Existing feed-forward 3D generative models (e.g., TRELLIS) can rapidly generate high-quality, single-semantic meshes. However, directly interpolating the latent codes of two objects in the latent space violates the spatial validity of the 3D structure, as the latent space does not possess geometric additivity. This makes zero-shot, efficient multi-semantic fusion on pre-trained generative models an open problem.

The core insight of this work is: rather than forcing a fusion in the latent space, it is better to decode the latent codes back to the voxel space at each denoising step to perform geometric operations (SDF averaging, binarization), and then re-encode them back to continue denoising. This leverages the fast generation capability of feed-forward models while preserving geometric continuity and interpretability in the voxel space. Core Idea: We propose a cross-space dual-branch denoising pipeline. During the denoising process of TRELLIS, the clean latent codes corresponding to two prompts are decoded step-by-step into the voxel space. After being aligned via CLIP-guided orientation search, they are geometrically fused through element-wise Signed Distance Field (SDF) averaging, binarized, and then re-encoded back into the latent space to proceed with denoising, ultimately yielding a single seamless geometry. The second stage utilizes view-constrained texture synthesis to enrich the geometry with appearances corresponding to both semantics.

Method

Overall Architecture

The JanusMesh pipeline is divided into two stages. Given two text prompts \(y_1, y_2\) and target viewing angles \(\theta_1, \theta_2\), the first stage performs dual-branch denoising based on the Rectified Flow model of TRELLIS: both branches start from the same initial noise \(z_t\) and independently predict the clean latent code of the current step conditioned on \(y_1\) and \(y_2\), respectively. At each denoising step, the two predicted clean latent codes are decoded into voxel representations via the Sparse Structure Decoder. After aligning to a unified coordinate system via CLIP-guided orientation search, they are converted into Signed Distance Fields (SDFs) for element-wise averaging, followed by threshold binarization to obtain fused voxels. Finally, they are re-encoded back into the latent space using the Sparse Structure Encoder as input for the next step. This loop continues until all denoising steps are completed, outputting a fused geometry. The second stage renders the fused geometry into depth maps from the target viewpoints, uses depth-conditioned ControlNet (Stable Diffusion) to predict textures for both views, and then back-projects them onto the 3D surface using cosine weight blending to output a dual-semantic textured mesh.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Shared Noise z_t"] --> B["Dual-branch Rectified Flow Denoising<br/>y₁ Branch | y₂ Branch"]
    B --> C["Predict Clean Latents<br/>x¹_{1|t} and x²_{1|t}"]
    C --> D["Decode to Voxel Space<br/>Sparse Structure Decoder"]
    D --> E["CLIP-guided Orientation Alignment<br/>Rotate v₂ to v₁ Coordinate System"]
    E --> F["Element-wise SDF Averaging<br/>+ Threshold Binarization"]
    F --> G["Re-encode to Latent Space<br/>Sparse Structure Encoder"]
    G --> H{"Denoising Steps Finished?"}
    H -->|No| B
    H -->|Yes| I["Fused Geometry"]
    I --> J["View-Constrained Texture Synthesis<br/>Depth ControlNet + Cosine Blending"]
    J --> K["Dual-Semantic Textured Mesh"]

Key Designs

1. Cross-space Step-by-step Denoising Fusion: Doing in Geometry Space What Latent Space Cannot Do

Directly averaging or interpolating two 3D latent codes creates semantic blending and meaningless geometric structures because the latent space lacks spatial additivity. Guided by LookingGlass, this method decodes the Rectified Flow-predicted clean latent codes \(\hat{x}^1_{1|t}\) and \(\hat{x}^2_{1|t}\) at each denoising step into voxel spaces \(v_1\) and \(v_2\) using the Sparse Structure Decoder. After rotating \(v_2\) to the coordinate system of \(v_1\), the two binary occupancy grids are converted into Signed Distance Fields, element-wise averaged, and threshold-binarized:

\[ \text{SDF}_{\text{blend}} = \frac{\text{SDF}(v_1) + \text{SDF}(v_2)}{2}, \quad \hat{x}_{1|t} = [\text{SDF}_{\text{blend}} < \tau] \]

The key to this operation is that the zero-isosurface of the SDF naturally corresponds to the intermediate shape of the two geometries, and the gradual transition of the isosurface ensures a smooth transition of the fusion boundary. The fused voxels are then re-encoded using the Sparse Structure Encoder to generate the input latent codes \(\hat{z}^1_{1|t}\) and \(\hat{z}^2_{1|t}\) for the next step of the respective branches. This "decoding \(\rightarrow\) voxel operation \(\rightarrow\) encoding" cross-space strategy enables zero-shot multi-semantic geometric fusion, and performs correction at each denoising step to prevent error accumulation.

2. CLIP-Guided Adaptive Orientation Alignment: Enabling Natural Silhouette Fusion for Unrelated Objects

Multi-view illusion methods usually assume fixed viewpoints (e.g., \(0^\circ\) and \(180^\circ\)), but the orientations of different objects in their canonical poses vary wildly. Directly fusing a horizontally stretched rhino with a vertically growing pineapple at default angles results in a geometrically chaotic blob. This work proposes to generate two single-semantic voxels \(\hat{v}_1\) and \(\hat{v}_2\) independently using TRELLIS first, and then execute a two-step CLIP search: First, object 1 is rendered into 4 candidate views at \(90^\circ\) intervals along the Z-axis, and the view with the highest CLIP image-text similarity is chosen as the anchor view \(I_1\). Second, object 2 is sampled in 28 rotation combinations at \(90^\circ\) intervals across the X, Y, and Z axes, and the rotation angle with the highest CLIP image-image similarity to \(I_1\) is selected as the fusing orientation \(\theta_2^*\). While this mechanism only requires about 2 minutes of extra time, it yields outstanding results for silhouette-mismatched object pairs.

3. Noise Guidance Strategies: Providing Prior Constraints for Geometrically Conflicting Pairs

Using purely random noise as the starting point of dual-branch denoising lacks spatial structure constraints. For object pairs with massive geometric discrepancies (e.g., "bamboo" and "grapes"), the fusion process is susceptible to geometric interference and convergence difficulties. This paper proposes two optional guidance strategies. Noise Blending Guidance pre-generates \(\hat{v}_1\) and \(\hat{v}_2\) independently, takes half of each to stitch into a guidance voxel \(v_{\text{guide}}\), encodes it into a guidance latent code, and blends it with pure noise as the starting denoising latent code:

\[ z_{\text{init}} = \alpha \cdot \text{Encoder}(v_{\text{guide}}) + (1-\alpha) \cdot z_{\text{noise}} \]

Space Control Guidance further performs interpolation at denoising time step \(t_0\), enabling fusion under strong constraints during the first \(t_0\) steps and free generation during subsequent steps. For silhouette-compatible object pairs, neither strategy is necessary; for highly discrepant pairs, Space Control Guidance yields the target performance; and for pairs with similar silhouettes but different semantics, mild Noise Blending Guidance is sufficient.

4. View-Constrained Texture Synthesis: Bestowing Perspective-Specific Appearances to Fused Geometries

The fused geometries resulting from the first stage contain unnatural geometric aliasing; directly texturing them with TRELLIS would lead to semantically blended textures—yielding a mixture of both semantics from either viewpoint. Therefore, this paper decouples texturing as an independent second stage, using view-aware texture prediction via depth-conditioned ControlNet (backed by Stable Diffusion). At each denoising step, depth maps of the mesh are rendered from target viewpoints \(\theta_1\) and \(\theta_2\). ControlNet predicts clean images \(\hat{x}_{1|t}\) corresponding to each view's text prompt, which are then back-projected onto the 3D surface. Finally, Mesh Texture Aggregation combines the texture contributions using cosine weights based on surface normals, ensuring a natural transition at the seams. View selection uses hard boundaries (taking \(0^\circ\) as an example, \(y_1\) texture is applied for \(270^\circ\)-\(90^\circ\), and \(y_2\) texture elsewhere); however, thanks to the smoothing effect of cosine weight blending, the actual seams are practically imperceptible to the naked eye.

Loss & Training

The proposed method is training-free (zero-shot). The first stage leverages TRELLIS's Rectified Flow model, and the second stage uses pre-trained Stable Diffusion + ControlNet, both without fine-tuning. The first stage employs Interval CFG (applying a guidance scale \(\omega=7.5\) within \(t \in [0.5, 0.95]\)) to avoid oversaturation at extreme noise levels. Denoising takes 25 steps in the first stage and 30 steps in the second stage, completing in 3–5 minutes in total.

Key Experimental Results

Main Results

Quantitative results on 50 randomly sampled prompt pairs (derived from 60 different objects across 5 categories):

Metric	Shape from Semantics	Direct Concat	TRELLIS	DreamBeast	JanusMesh
CLIP Similarity ↑	27.46	29.03	22.18	22.99	28.17
CLIP (opposite) ↓	19.72	20.38	22.68	22.89	19.26
GPT Accuracy (%) ↑	70	76	60	65	84
FID ↓	194.14	187.89	174.13	184.96	185.56
KID ↓	0.051	0.067	0.044	0.094	0.051
Avg. Object Detection Count (→1)	0.64	2.1	0.58	0.76	0.86
Multi-object Rate (%) ↓	2	56	8	18	18
Impact Factor (→1)	0.973	1.129	0.952	0.974	0.994
Generation Time	~40 min	~3–5 min	~2–3 min	~3–4 hr	~3–5 min

JanusMesh comprehensively leads in GPT accuracy (84%), cross-view semantic leakage suppression (CLIP opposite of 19.26%), geometric seamlessness (Impact Factor of 0.994), and generation efficiency (3–5 minutes). While Direct Concatenation achieves the highest CLIP score, this is a systematic artifact; it retains the original appearances from different viewpoints through naive stitching, but suffers from severe geometric discontinuity and semantic leakage (56% multi-object rate).

Ablation Study

Ablation of Geometric Fusion Strategies ("carriage"/"sundae" pair):

Configuration	Key Observation
Union (Logical OR)	Produces conflicting joints
Blur Avg (Average after blurring)	Loses fine geometric details
Minkowski Fusion	Dilates geometric volume
Polar Coord Fusion	Fails for asymmetric objects
SDF Averaging (Ours)	Optimal; the isosurface naturally corresponds to the intermediate shape

Ablation of Noise Guidance Strategies:

Configuration	Applicable Scenarios
No Guidance	Sufficient for silhouette-compatible object pairs
Noise Blending	Best for pairs with similar silhouettes but different semantics
Space Control	Best for pairs with large geometric discrepancies (\(t_0=10\))

Ablation of View-Constrained Texturing: Once the second stage is removed, TRELLIS fails to handle the fused geometry correctly, causing the texturing on both views to exhibit semantic confusion.

Key Findings

• GPT accuracy is the most crucial metric where ours outperforms others: JanusMesh achieves 84%, far exceeding Direct Concat (76%) and Shape from Semantics (70%), indicating that both human eyes and automated evaluation can clearly distinguish target semantics from different perspectives.
• Object detection metrics reveal the quality of geometric fusion: Direct Concat's multi-object rate is as high as 56% (as the detector identifies the stitched body as two separate objects), while JanusMesh reduces this to 18% (on par with DreamBeast), verifying the seamlessness of SDF-based fusion.
• CLIP-guided orientation search is vital for incompatible object pairs: In the user study, 91% of participants agreed that adaptive rotation provides a more natural illusion effect than a fixed \(0^\circ/180^\circ\) setup.
• Flexible choices among three configurations: Cases 1 and 2 (fixed angles + optional noise guidance) can be completed in about 3 minutes, while Case 3 (CLIP search) requires around 5 minutes.

Highlights & Insights

• The cross-space denoising strategy is ingenious and elegant: The paradigm of repeating 'decoding \(\rightarrow\) voxel operations \(\rightarrow\) encoding' during denoising is a general approach to solving the lack of additivity in latent spaces. It is not limited to visual illusions—any scenario requiring the injection of space constraints into 3D generation (such as multi-object composition or component editing) can benefit from this design.
• Using SDF averaging as a geometric fusion operator is an excellent design choice: Compared to Direct Occupancy Average (which easily creates voids), Blur (which loses details), and Minkowski (which dilates the volume), the zero-isosurface of SDF automatically aligns with the intermediate shape, and its gradient semantics maintain a natural transition, rendering it highly interpretable in both differentiable and discrete contexts.
• Flexible reuse of diffusion model priors: This work introduces no trainable parameters, achieving a completely new task solely by orchestrating the execution pipeline of pre-trained models (TRELLIS for geometry, ControlNet for textures). This exemplifies a design philosophy of 'zero-shot orchestration.'
• Noise guidance strategies can be analogized to general tricks for conditional initialization: Whether it is the weighted mixing of Noise Blending or the design of time-step interpolation in Space Control, they essentially inject priors at initial stages—an approach easily transferable to various multi-branch fusion generation tasks.

Limitations & Future Work

• Inheriting TRELLIS fail cases: For certain object categories (e.g., pigs, bats), TRELLIS itself yields subpar generation quality, which JanusMesh cannot bypass.
• Orientation alignment for three-object illusions is not yet automated: Currently, the three-object version is fixed at \(0^\circ/120^\circ/240^\circ\) because CLIP search tends to degenerate into a blurred direction estimate after averaging three silhouettes, failing to align them effectively. Developing an automatic orientation search for three-object scenarios remains an open problem.
• CLIP search precision is limited by angular sampling granularity: Currently, sampling at \(90^\circ\) intervals (yielding 28 combinations) is used. Denser sampling at \(45^\circ\) could further enhance alignment precision but would linearly increase search time.
• Hard perspective boundaries in the texturing stage: Even though cosine-weighted blending masks the switching boundaries, in theory, viewpoint-based hard classification may cause texture jumps in extreme perspective transition zones. A soft switching mechanism based on attention weights or geometric distance could be more robust.

Related Work & Insights

• vs Visual Anagrams / SyncTweedies: 2D methods achieve multi-view illusions by averaging noise predictions under different transformations; JanusMesh extends this from 2D pixels to 3D voxels, replacing noise averaging with SDF averaging.
• vs Shape from Semantics: This optimization-based method optimizes the latent space using SDS, which takes 40 minutes per shape and suffers from severe oversaturation. JanusMesh reduces this to 3–5 minutes while providing higher quality via feed-forward + cross-space denoising.
• vs Direct Concatenation: Naive stitching is fast but suffers from severe geometric discontinuity and semantic leakage; SDF fusion fundamentally resolves seam anomalies.
• vs SpaceControl: Our Space Control Guidance draws inspiration from SpaceControl's execution step control concept but adapts it to dual-semantic fusion scenarios.
• vs LookingGlass: Also utilizes a 'decoding \(\rightarrow\) manipulation \(\rightarrow\) encoding' cross-space strategy, but LookingGlass is used for 2D-to-3D deformation whereas JanusMesh extends it to dual-branch denoising fusion.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to apply a cross-space denoising strategy to 3D visual illusions. The orchestration of dual-branch SDF fusion, CLIP orientation search, and view-constrained texturing is highly creative.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Designs 6 quantitative metrics (including a self-created Impact Factor and object detection score), a user study (50 participants), and complete ablation experiments (fusion strategies \(\times 4\), noise guidance \(\times 3\), texturing and orientation search each \(\times 2\)), providing comprehensive coverage.
• Writing Quality: ⭐⭐⭐⭐⭐ The writing is fluent, the methodology section features clear text-to-figure correspondence, formulas are placed with restraint only where truly necessary, and the ablation logical chain is complete.
• Value: ⭐⭐⭐⭐⭐ Enables zero-shot, training-free generation of high-quality 3D visual illusions in just 3–5 minutes, offering direct application value in creative content generation and 3D design.

Related Papers

• [CVPR 2026] Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching
• [ECCV 2026] Zero-Shot Depth from Defocus: 从对焦堆栈零样本估计度量深度
• [ICLR 2026] pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning
• [NeurIPS 2025] 3D Visual Illusion Depth Estimation
• [ECCV 2026] Agentic Collaborative Cognition for Zero-Shot 3D Understanding