JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising

Conference: ECCV2026
arXiv: 2606.20563
Project Page: https://siang1105.github.io/JanusMesh.github.io/
Code: None
Area: 3D Vision
Keywords: 3D visual illusion, zero-shot generation, cross-space denoising, SDF fusion, viewpoint-constrained texturing

TL;DR

JanusMesh proposes a zero-shot, two-stage framework that utilizes dual-branch denoising and SDF fusion in the structured 3D latent space of TRELLIS to generate multi-semantic geometries, followed by viewpoint-constrained texture synthesis to assign corresponding appearances to each view, generating high-quality 3D visual illusion meshes within 3–5 minutes.

Background & Motivation

3D visual illusions—where a single 3D object exhibits entirely different semantics from different viewpoints (e.g., a peacock from the front and a pineapple from the back)—are both fascinating and highly challenging. While recent diffusion models have enabled 2D visual illusions (such as Visual Anagrams), extending them to real 3D meshes reveals severe bottlenecks in existing methods: SDS-based optimization methods (e.g., Shape from Semantics) require shape-specific optimizations taking around 40 minutes per shape and suffer from severe color oversaturation; conversely, naive "direct concatenation" methods (generating two objects separately and stitching them down the middle) expose obvious geometric discontinuities and back-view semantic leakage at the seams, destroying the overall illusion.

The key challenge of this dilemma is that 3D visual illusions require a single mesh that is geometrically seamless yet semantically separated across viewpoints. Existing approaches either slowly search in the optimization space (the SDS pipeline) or fail to handle geometric continuity during spatial stitching. Existing feed-forward 3D generative models (e.g., TRELLIS) can quickly generate high-quality, single-semantic meshes, but directly interpolating the latent codes of two objects in the latent space disrupts the spatial validity of the 3D structure, as the latent space lacks geometric additivity. This leaves the challenge of performing zero-shot, efficient multi-semantic fusion on pre-trained generative models as an open problem.

The key insight of this work is that rather than forcing fusion within the latent space, it is better to decode the latent codes back to the voxel space at each denoising step to perform geometric operations (SDF averaging and binarization) and then re-encode them to continue denoising. This leverages the fast generation capability of feed-forward models while maintaining geometric continuity and interpretability in the voxel space. Core Idea: A cross-space dual-branch denoising pipeline is proposed. During the denoising process of TRELLIS, the clean latent codes corresponding to two prompts are step-by-step decoded into voxel space. After being aligned via CLIP-guided orientation search, they are geometrically fused through element-wise Signed Distance Field (SDF) averaging and then re-encoded back into the latent space to continue denoising, ultimately yielding a single seamless geometry. In the second stage, viewpoint-constrained texture synthesis is used to assign appearances corresponding to the two semantics to this geometry.

Method

Overall Architecture

The pipeline of JanusMesh is divided into two stages. Given two text prompts \(y_1, y_2\) and target viewpoints \(\theta_1, \theta_2\), the first stage performs dual-branch denoising based on the Rectified Flow model of TRELLIS: both branches start from the same initial noise \(z_t\) and independently predict the clean latent code of the current step conditioned on \(y_1\) and \(y_2\), respectively. At each denoising step, the two predicted clean latent codes are decoded into voxel representations via the Sparse Structure Decoder. After aligning to a unified coordinate system through CLIP-guided orientation search, they are converted into signed distance fields for element-wise averaging, followed by threshold binarization to obtain fused voxels. Finally, the fused voxels are re-encoded back into the latent space using the Sparse Structure Encoder as the input for the next step. This loop continues until all denoising steps are completed, outputting a fused geometry. In the second stage, the fused geometry is rendered as depth maps from the target viewpoints. A depth-conditioned ControlNet (Stable Diffusion) is then used to predict the textures for both viewpoints, which are back-projected onto the 3D surface using cosine weight blending to produce a dual-semantic textured mesh.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Shared Noise z_t"] --> B["Dual-Branch Rectified Flow Denoising<br/>y₁ Branch | y₂ Branch"]
    B --> C["Predict Clean Latents<br/>x¹_{1|t} and x²_{1|t}"]
    C --> D["Decode to Voxel Space<br/>Sparse Structure Decoder"]
    D --> E["CLIP-Guided Orientation Alignment<br/>Rotate v₂ to v₁ Coordinate System"]
    E --> F["Element-wise SDF Averaging<br/>+ Threshold Binarization"]
    F --> G["Re-encode to Latent Space<br/>Sparse Structure Encoder"]
    G --> H{"Denoising Steps Completed?"}
    H -->|No| B
    H -->|Yes| I["Fused Geometry"]
    I --> J["Viewpoint-Constrained Texture Synthesis<br/>Depth ControlNet + Cosine Blending"]
    J --> K["Dual-Semantic Textured Mesh"]

Key Designs

1. Step-by-Step Cross-Space Denoising Fusion: Doing in Geometric Space What Latent Space Cannot Do

Directly averaging or interpolating two 3D latent codes leads to semantic blurs and meaningless geometric structures because the latent space lacks spatial additivity. Borrowing the concept from LookingGlass, this work decodes the clean latent codes \(\hat{x}^1_{1|t}\) and \(\hat{x}^2_{1|t}\) predicted by Rectified Flow at each denoising step into voxel spaces \(v_1\) and \(v_2\) using the Sparse Structure Decoder. After rotating \(v_2\) into the coordinate system of \(v_1\), the two binary occupancy grids are converted into signed distance fields for element-wise averaging and threshold binarization:

\[ \text{SDF}_{\text{blend}} = \frac{\text{SDF}(v_1) + \text{SDF}(v_2)}{2}, \quad \hat{x}_{1|t} = [\text{SDF}_{\text{blend}} < \tau] \]

The key to this operation is that the zero-isosurface of the SDF naturally corresponds to the intermediate shape of the two geometries, and the gradual transition of the isosurface ensures smooth transitions at the fusion boundary. The fused voxels are then re-encoded using the Sparse Structure Encoder to produce the input latent codes \(\hat{z}^1_{1|t}\) and \(\hat{z}^2_{1|t}\) for the next step of the two branches. This cross-space "decode \(\rightarrow\) voxel operation \(\rightarrow\) encode" strategy enables zero-shot multi-semantic geometric fusion, correcting errors at each denoising step to prevent error accumulation.

2. CLIP-Guided Adaptive Orientation Alignment: Naturally Fusing Outlines of Unrelated Objects

Multi-view illusion methods typically assume fixed viewpoints (e.g., \(0^\circ\) and \(180^\circ\)), but the orientations of different objects in canonical poses vary significantly. Directly fusing a horizontally oriented rhinoceros and a vertically growing pineapple at their default angles yields a semantically cluttered geometry after SDF averaging. This work proposes to first independently generate two single-semantic voxels \(\hat{v}_1\) and \(\hat{v}_2\) using TRELLIS, and then perform a two-step CLIP search: first, object 1 is rendered from 4 candidate views at \(90^\circ\) intervals along the Z-axis, with the view yielding the highest CLIP text-image similarity selected as the anchor view \(I_1\); second, 28 rotation combinations are sampled for object 2 at \(90^\circ\) intervals across the X, Y, and Z axes, and the rotation angle with the highest CLIP image-image similarity to \(I_1\) is selected as the fusion orientation \(\theta_2^*\). This mechanism requires only about 2 minutes of additional time but is highly effective for object pairs with mismatched outlines.

3. Noise Guidance Strategies: Providing Prior Constraints for Geometrically Conflicting Pairs

Pure random noise as the starting point of dual-branch denoising lacks spatial structural constraints. For object pairs with massive geometric discrepancies (e.g., "bamboo" and "grapes"), geometric interference and convergence difficulties easily occur during the fusion process. This paper proposes two optional guidance strategies. Noise Blending Guidance pre-generates \(\hat{v}_1\) and \(\hat{v}_2\) independently, concatenates half of each to form a guide voxel \(v_{\text{guide}}\), encodes it into a guide latent code, and blends it with pure noise to serve as the initial denoising latent:

\[ z_{\text{init}} = \alpha \cdot \text{Encoder}(v_{\text{guide}}) + (1-\alpha) \cdot z_{\text{noise}} \]

Space Control Guidance further performs interpolation at denoising timestep \(t_0\), enabling heavily constrained fusion during the first \(t_0\) steps followed by free generation in the remaining steps. For outline-compatible object pairs, neither strategy is necessary; for highly discrepant pairs, Space Control Guidance performs best; while for pairs with similar outlines but different semantics, gentle Noise Blending Guidance is sufficient.

4. Viewpoint-Constrained Texture Synthesis: Assigning Exclusive Appearances to Each View of Fused Geometry

The fused geometry from the first stage contains unnatural geometric blending. Directly texturing it using TRELLIS would result in semantically confused textures, where both views present a mix of both semantics. Therefore, this work separates texturing into a second stage, applying a depth-conditioned ControlNet (backed by Stable Diffusion) for viewpoint-aware texture prediction. At each denoising step, depth maps of the mesh are rendered from the target views \(\theta_1\) and \(\theta_2\). ControlNet predicts the clean image \(\hat{x}_{1|t}\) of the corresponding view based on its specific text prompt, which is then back-projected onto the 3D surface. Finally, Mesh Texture Aggregation blends texture contributions using cosine weights calculated from the surface normals to ensure seamless transitions. Viewpoint selection uses hard boundaries (e.g., \(270^\circ\)–\(90^\circ\) for \(y_1\) texture and the rest for \(y_2\) texture, relative to \(0^\circ\)); however, due to the smoothing effect of the cosine weight blending, the actual seams are visually imperceptible.

Loss & Training

The proposed method is training-free (zero-shot). The first stage employs the Rectified Flow model of TRELLIS, and the second stage uses pre-trained Stable Diffusion + ControlNet, both without fine-tuning. The first stage utilizes Interval CFG (applying a guidance scale of \(\omega=7.5\) within \(t \in [0.5, 0.95]\)) to avoid oversaturation at extreme noise levels. The first stage runs 25 denoising steps and the second runs 30, taking a total of 3–5 minutes.

Key Experimental Results

Main Results

Quantitative results on 50 randomly sampled prompt pairs (corresponding to 60 different objects across 5 categories):

Metric	Shape from Semantics	Direct Concat	TRELLIS	DreamBeast	JanusMesh
CLIP Similarity ↑	27.46	29.03	22.18	22.99	28.17
CLIP (opposite) ↓	19.72	20.38	22.68	22.89	19.26
GPT Accuracy (%) ↑	70	76	60	65	84
FID ↓	194.14	187.89	174.13	184.96	185.56
KID ↓	0.051	0.067	0.044	0.094	0.051
Average Object Detection Count (→1)	0.64	2.1	0.58	0.76	0.86
Multi-Object Rate (%) ↓	2	56	8	18	18
Impact Factor (→1)	0.973	1.129	0.952	0.974	0.994
Generation Time	~40 min	~3–5 min	~2–3 min	~3–4 hr	~3–5 min

JanusMesh leads comprehensively in GPT Accuracy (84%), cross-view semantic leakage suppression (CLIP opposite 19.26), geometric seamlessness (Impact Factor 0.994), and generation efficiency (3–5 minutes). While Direct Concatenation achieves the highest CLIP score, this is a systematic artifact—it preserves the original appearance of each viewpoint via naive stitching, but suffers from severe geometric discontinuity and semantic leakage (56% multi-object rate).

Ablation Study

Ablation on Geometric Fusion Strategies (for the "carriage"/"sundae" pair):

Configuration	Key Observation
Union (logical OR)	Generates conflicting junctions
Blur Avg	Loses fine geometric details
Minkowski Fusion	Dilates geometry volume
Polar Coord Fusion	Fails for asymmetric objects
SDF Averaging (Ours)	Optimal, the isosurface naturally corresponds to the intermediate shape

Ablation on Noise Guidance Strategies:

Configuration	Applicable Scenarios
No guidance	Sufficient for outline-compatible object pairs
Noise Blending	Best for pairs with similar outlines but different semantics
Space Control	Best for geometrically discrepant pairs (\(t_0=10\))

Ablation on Viewpoint-Constrained Texturing: Removing the second stage prevents TRELLIS from correctly processing the fused geometry, resulting in semantically confused textures for both views.

Key Findings

• GPT Accuracy is the most critical surpassing metric of Ours: JanusMesh reaches 84%, significantly outperforming Direct Concat (76%) and Shape from Semantics (70%), which demonstrates that both human eyes and automated evaluation can clearly distinguish the target semantics of each view.
• Object detection metrics reveal geometric fusion quality: Direct Concat suffers from a high multi-object rate of 56% (the detector perceives the stitched entity as two separate objects), while JanusMesh reduces it to 18% (tied with DreamBeast), validating the geometric seamlessness of SDF fusion.
• CLIP-guided orientation search is vital for incompatible pairs: In the user study, 91% of participants found that adaptive rotation produces more natural illusion effects than fixed \(0^\circ/180^\circ\) orientations.
• Flexible choice among three configurations: Case 1/2 (fixed angle + optional noise guidance) can be completed in about 3 minutes, while Case 3 (CLIP search) takes about 5 minutes.

Highlights & Insights

• The cross-space denoising strategy is ingenious and elegant: The iterative "decode \(\rightarrow\) voxel operation \(\rightarrow\) encode" pattern during denoising is a general paradigm to address the non-additivity of the latent space. It is not limited to visual illusions; any scenario requiring spatial constraints in 3D generation (such as multi-object composition or part editing) can benefit from this concept.
• Using SDF averaging as a geometric fusion operator is an excellent design choice: Compared to direct occupancy averaging (which produces holes), Blur (which loses details), or Minkowski (which bloats), the zero-isosurface of the SDF naturally corresponds to the intermediate shape, preserving a natural transition of gradient semantics. It remains interpretable in both differentiable and discrete settings.
• Flexible reuse of diffusion model priors: Ours does not introduce any trainable parameters and accomplishes an entirely new task purely by orchestrating the execution pipeline of pre-trained models (TRELLIS geometry + ControlNet texture)—exemplifying a design philosophy of "zero-shot orchestration".
• Noise guidance strategies are analogous to generic skills of conditional initialization: Whether through the weighted mixing of Noise Blending or the timestep interpolation of Space Control, both essentially inject priors into the initial state. This framework can easily migrate to diverse multi-branch fusion generation scenarios.

Limitations & Future Work

• Inheriting failure cases of TRELLIS: For certain object categories (e.g., pigs, bats), TRELLIS itself exhibits poor generation quality, which JanusMesh cannot bypass.
• Orientation alignment for three-object illusions is not yet automated: The three-object version currently uses fixed \(0^\circ/120^\circ/240^\circ\) orientations because the CLIP search degenerates into a blurred representation after averaging three silhouettes, failing to align them effectively. Designing an automatic orientation search for three-object scenarios remains an open problem.
• CLIP search precision is limited by angular sampling granularity: Currently sampling at \(90^\circ\) intervals (28 combinations); denser sampling at \(45^\circ\) could yield more accurate alignment but would linearly increase search time.
• Hard view boundaries in the texturing stage: Although cosine weight blending conceals switching boundaries, viewpoint-based hard classification could theoretically trigger texture flickering at extreme transition zones; soft switching based on attention weights or geometric distances might prove more robust.

Related Work & Insights

• vs Visual Anagrams / SyncTweedies: 2D methods achieve multi-view illusions by averaging noise predictions under different transformations; JanusMesh extends this from 2D pixels to 3D voxels, replacing noise averaging with SDF averaging.
• vs Shape from Semantics: Uses SDS to optimize latent space, requiring 40 minutes per shape and suffering from severe oversaturation; JanusMesh reduces generation time to 3–5 minutes with higher quality via feed-forward + cross-space denoising.
• vs Direct Concatenation: Naive stitching is fast but exhibits severe geometric discontinuity and semantic leakage; SDF fusion fundamentally resolves seam artifacts.
• vs SpaceControl: Space Control Guidance in Ours borrows the start-timestep control idea of SpaceControl but adapts it to dual-semantic fusion scenarios.
• vs LookingGlass: Sharing the cross-space strategy of "decode \(\rightarrow\) operate \(\rightarrow\) encode", LookingGlass performs 2D-to-3D deformation whereas JanusMesh extends it to dual-branch denoising fusion.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First to apply a cross-space denoising strategy to 3D visual illusions; the execution of dual-branch SDF fusion + CLIP orientation search + viewpoint-constrained texturing is highly creative.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Designs 6 quantitative metrics (including self-created Impact Factor and object detection scores), a user study (50 participants), and comprehensive ablation studies (fusion strategies \(\times 4\), noise guidance \(\times 3\), texturing and orientation search \(\times 2\) each), offering a thorough evaluation.
• Writing Quality: ⭐⭐⭐⭐⭐ Fluent and coherent; the method section is supported by clear diagram-to-text correspondences; mathematical formulations are introduced selectively where strictly necessary, showing a complete logic hook-up in the ablation studies.
• Value: ⭐⭐⭐⭐⭐ Enables zero-shot, training-free generation of high-quality 3D visual illusions within 3–5 minutes, carrying direct application value in creative content generation and 3D design.

Related Papers

• [CVPR 2026] Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching
• [ECCV 2026] Zero-Shot Depth from Defocus: 从对焦堆栈零样本估计度量深度
• [ICLR 2026] pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning
• [NeurIPS 2025] 3D Visual Illusion Depth Estimation
• [ECCV 2026] Agentic Collaborative Cognition for Zero-Shot 3D Understanding