SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: monocular depth estimation, transparent objects, generative opacification, mask refinement, paired rendering

TL;DR

SeeClear uses diffusion to replace transparent objects with opaque appearances that retain contours and surface shading, then applies an unmodified depth model, reducing DA3 RMSE from 24.03 to 21.45 mm on transparent-object regions in ClearGrasp.

Background & Motivation

Monocular depth models typically infer surface positions from contours, texture, shading, and scene semantics, but transparent objects mix these cues together. Texture inside a glass cup may belong to the table behind it, while refraction distorts the background and encourages the model to place cup-wall pixels farther away. For reconstruction or robotic interaction, this is more than a visual defect: an incorrect glass-surface depth can erase a collision region. Specialized methods such as Depth4ToM, D4RD, and DKT adapt depth networks to transparent materials, but generally rely on transparent RGB and depth supervision tied to particular model training procedures. When a general-purpose depth backbone changes, transparent-object adaptation may need to be repeated.

SeeClear instead focuses on the input: if existing models already understand opaque objects, could transparency first be converted into familiar visual evidence? The difficulty is that hiding the background does not establish correct geometry. Filling a cup with a flat color removes the visible table, but also removes shading cues describing curvature and openings. The generator introduces another risk: altered contours or a background blurred by VAE reconstruction can make the depth input worse. The goal is therefore to cover transparent regions while retaining object boundaries and background context, rather than merely producing an attractive image edit.

The authors render transparent and opaque pairs with fixed geometry, cameras, and illumination to teach appearance changes under geometric invariance. Segmentation masks constrain the editing region, CLIP category semantics help preserve object identity, and subsequent mask refinement places the generated object back into the original scene. Geometric consistency here comes mainly from paired data, spatial conditioning, and perceptual supervision, not from solving refraction equations or imposing a hard guarantee of true shape recovery. Core Idea: train a generative opacification front end that turns transparency-induced depth-input distribution shifts into appearance editing, allowing existing depth models to process the opaque surfaces they already handle well.

Method

Overall Architecture

The input is a single RGB image containing transparent objects, and the output remains a scene depth map; ground-truth depth or normals are not required at test time. During training, paired rendering data teaches the generative opacification module to change material while preserving geometry; deployment first localizes transparent objects and then generates opaque appearances per instance. Mask refinement and compositing combine generated object regions with the original background, after which a frozen DA3 or MoGe-2 predicts depth. Thus, the claim of no retraining applies to the depth backbone, not to the entire SeeClear front end.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    DATA["Paired Rendering Data"] -.->|Training supervision only| GEN["Generative Opacification"]
    INPUT["Input RGB Image"] --> SEG["Segmentation and Instance Cropping"]
    SEG --> GEN
    GEN --> MIX["Mask Refinement and Compositing"]
    SEG -->|Segmentation mask| MIX
    INPUT -->|Original background| MIX
    MIX --> DEPTH["Frozen Depth Model"]
    DEPTH --> OUTPUT["Scene Depth Map"]

Key Designs

1. Paired Rendering Data: change material while supervising the same geometric surface

SeeClear-396k uses 11 everyday transparent objects and furniture items across 10 indoor scenes, producing 110 object-scene combinations. Each combination samples 10 camera viewpoints and 2 lighting setups, with 5 scaling magnitudes under 6 anisotropic deformation modes. These modes scale along x, y, z, and the xy, yz, xz combinations, expanding shape variation rather than merely changing backgrounds. Each combination consequently produces 600 rendering sets, for a total of 66,000 configurations. Every set contains transparent RGB, opaque RGB, absolute depth, relative depth, normals, and a mask, yielding 396,000 rendered outputs; 396k does not mean 396k independent transparent scenes.

Transparent and opaque images within a pair share geometry, camera, and illumination, with only the glass-like material replaced by a fixed-color diffuse material. The target image therefore retains shading caused by curvature and occlusion while removing transmitted background and refractive appearance. For the network, the target is not a similar cup, but an interpretable surface for the cup at this position with this shape. The authors organize coverage around failure factors including non-Lambertian appearance, surface-versus-background depth ambiguity, thin boundaries, and multilayer occlusion. Although depth and normals accompany the data, the opacification training described in the main paper neither requires them as test-time conditions nor lists an explicit depth or normal loss. Rendering variations expand viewpoints and appearances, but the underlying object inventory still contains only 11 types, so sample count should not be equated with category diversity.

2. Generative Opacification: replace unreliable transparent texture with spatial extent and global semantics

The implementation first uses Trans4Trans for initial localization, then supplies this as a spatial prompt to SAM 3 to obtain the transparent-object mask. Each instance is cropped around its bounding-box center and resized to 512 × 512, letting the generator concentrate on object appearance instead of arbitrary object sizes and positions in the full image. During training, a pretrained VAE encodes the paired opaque target, noise is added to its latent representation, and a conditional diffusion network learns to predict that noise. The denoising network receives three channel-concatenated inputs: the noisy target latent, the VAE encoding of the transparent RGB image, and the downsampled segmentation mask. The image supplies instance and scene information, while the mask explicitly identifies where transparency should become opacity instead of leaving the edit extent entirely to semantic inference. This explains why masks outperform point heatmaps and bounding boxes: the latter locate an object but cannot precisely delimit its transparent surface.

In parallel, a CLIP image encoder extracts a class token from the transparent image, and a lightweight Transformer Mapper projects it into the denoiser's cross-attention conditioning space. The authors retain only the class token rather than all patch tokens because local tokens may carry background texture seen through the object. Category-level features emphasize object identity and complement the spatial extent supplied by the mask, helping preserve identity without reproducing refractive appearance. This does not mean CLIP directly recovers geometry; it reduces the conditioning representation's dependence on locally contaminated transparent regions. Training also randomly adds or removes small circles, rectangles, and similar shapes near mask boundaries to tolerate common automatic-segmentation errors. At test time, no paired opaque target is available: generation uses the image and predicted mask, and patches from multiple objects can be processed as a batch.

3. Mask Refinement and Compositing: replace edited pixels while preserving original scene evidence

Passing the complete generated image into a depth model creates another problem: VAE encoding and decoding can blur or shift the colors of the background. Hard-pasting with the initial segmentation mask is also insufficient because small differences between predicted contours and generated boundaries leave transparent edges or seams. The Mask Refinement Module, or MRM, therefore examines the opaque candidate, original transparent image, and mask together to learn where to retain generated content and where to retain original pixels. It is a lightweight fully convolutional network with 3 convolutional layers of size 3 × 3 and a 1 × 1 projection layer, using GroupNorm and SiLU. Training inputs are the ground-truth opaque image, transparent image, and perturbed ground-truth mask; at test time, the opaque image becomes the diffusion output and the mask becomes the automatic segmentation result. This training-to-test input difference means MRM must also handle generation errors and should not be treated as a perfect boundary repair mechanism.

MRM outputs a soft mask during training, which is binarized at test time for pixel-level hard compositing. The following is an equivalent expression of the compositing rule described in Section 3.2, not a character-by-character reconstruction of the damaged Equation (5) in the cache:

\[ I^{\mathrm{blend}}=\widehat M_{\mathrm{ref}}\odot I^{\mathrm{pred}}+(1-\widehat M_{\mathrm{ref}})\odot I^{\mathrm{tr}}. \]

The binary mask selects generated object pixels, its complement selects background pixels from the original transparent image, and \(\odot\) denotes element-wise multiplication. Preserving original context avoids background degradation and reduces the risk that overlapping instance patches restore transparency to an already opacified object. Depth estimation runs only after compositing; MRM neither predicts depth nor changes the backbone architecture.

A Worked Example

Consider an open transparent cup on a table as an explanatory example: table texture visible through its body may cause a depth model to interpret the cup wall as background. Segmentation and cropping first identify the cup's image extent, and the generator converts its body into an opaque shaded surface within a 512 × 512 patch. The spatial mask constrains the cup-wall contour, CLIP conditioning preserves cup identity, and paired-rendering training supplies a prior over curved-surface appearance. MRM then selects generated pixels along the object boundary while retaining original RGB for the table and other unedited regions. Finally, the depth backbone estimates surface distances from the new cup shading and unchanged scene; it receives no additional ground-truth cup depth. This process aims to distinguish the nearer cup body from more distant regions inside the opening, but does not guarantee correct completion of every cup. The example explains information flow rather than reporting an additional measured sample; the paper discusses an open container in the text accompanying Figure 4 on page 13.

Loss & Training

The generator uses standard DDPM noise-prediction supervision together with an LPIPS perceptual loss restricted to the transparent-object mask. The perceptual term compares the decoded prediction with the paired opaque target through multilevel VGG features, complementing element-wise latent supervision that is less sensitive to fine boundaries and surface structure. The total objective stated in the text on page 6 is:

\[ \mathcal L=\mathcal L_{\mathrm{LDM}}+\lambda\mathcal L_{\mathrm{LPIPS}}. \]

LPIPS is applied only at low-noise timesteps because high-noise decoded predictions are far from clean images, and premature perceptual constraints can reduce sample diversity. MRM uses binary cross-entropy plus a penalty on intermediate mask values, encouraging clear foreground or background decisions. Equations (2), (4), and (5) have missing symbols in the cache, so this note does not guess the exact noise-loss expression, mask-penalty reduction, or unreported thresholds. The diffusion front end is initialized from Paint-by-Example, uses a base learning rate of \(1\times10^{-5}\) with linear warm-up, and has a reported batch size of 4. Training runs for 328,339 iterations on 4 NVIDIA H100 NVL GPUs; the main paper does not clarify whether the batch size is per-device or global. Inference uses 10 UniPC sampling steps, and the paper reports approximately 7.65 s per image on a single H100. Neither DA3 nor MoGe-2 is fine-tuned; zero-shot means no training on the evaluation datasets, not that the front end was never trained on SeeClear-396k.

Key Experimental Results

Main Results

The following selection comes from Table 1 on page 11 and Table 2 on page 12; RMSE is in mm, lower AbsRel is better, and threshold accuracy is reported as a percentage with higher being better. ClearGrasp ToM denotes transparent-object regions and All denotes the full image; TransPhy3D reports only full-image evaluation because ground-truth object masks are unavailable. Models requiring affine alignment are least-squares aligned to ground truth, with disparity-output models aligned in disparity space before conversion to depth, so these numbers should not all be read as uncalibrated absolute ranging accuracy. Threshold accuracy \(\delta_{1.025}\) is the proportion of valid pixels whose maximum bidirectional prediction-to-ground-truth depth ratio is below 1.025. DKT marked with † was fine-tuned on the respective evaluation dataset, unlike SeeClear.

Dataset and Region	Method	AbsRel	RMSE (mm)	\(\delta_{1.025}\) (%)	Source
ClearGrasp ToM	DA3	0.038	24.03	46.89	Table 1
ClearGrasp ToM	SeeClear + DA3	0.033	21.45	51.27	Table 1
ClearGrasp ToM	MoGe-2	0.101	56.83	5.63	Table 1
ClearGrasp ToM	SeeClear + MoGe-2	0.043	26.62	42.75	Table 1
ClearGrasp ToM	DKT†	0.035	22.43	52.69	Table 1
ClearGrasp All	MoGe-2	0.020	22.30	77.51	Table 1
ClearGrasp All	SeeClear + MoGe-2	0.016	17.87	80.63	Table 1
TransPhy3D All	DA3	0.023	53.25	76.00	Table 2
TransPhy3D All	SeeClear + DA3	0.017	47.46	88.01	Table 2
TransPhy3D All	MoGe-2	0.016	39.78	89.58	Table 2
TransPhy3D All	SeeClear + MoGe-2	0.012	38.91	94.43	Table 2
TransPhy3D All	DKT†	0.014	33.20	89.20	Table 2

Ablation Study

All entries below use ClearGrasp Real, the ToM region, and the DA3 backbone from Table 4 on page 14; lower RMSE is better. Only configurations whose values are corroborated by legible table rows or adjacent prose are included, avoiding guessed column assignments in merged encoder-variant rows. The solid-color baseline fills the same mask separately with red, green, and blue and takes the pixel-wise median of the three depth predictions, rather than selecting an arbitrary single fill.

Config	RMSE (mm)	Note
Full SeeClear	21.45	Mask conditioning, CLIP class token, MRM, and compositing
Bounding box instead of mask	22.96	Coarse extent cannot precisely constrain contours
Point heatmap instead of mask	22.38	More ambiguous spatial extent
Without MRM	21.73	No learned boundary refinement
Without blending	23.60	Patch-level overwriting affects backgrounds and overlapping instances
Original DA3	24.03	No opacification
Solid-color filling	33.09	Hides transmitted texture but lacks surface shading

Key Findings

• Generated surface cues matter more than merely covering transparency: solid-color filling yields 33.09 mm, worse than unmodified DA3 at 24.03 mm, while full generation achieves 21.45 mm.
• Gains depend on backbone and evaluation region: DA3 is stronger on ClearGrasp ToM, whereas SeeClear + MoGe-2 has lower full-image RMSE; these rankings should not be conflated.
• No method dominates every metric: DKT† has higher strict threshold accuracy on ClearGrasp ToM and lower TransPhy3D RMSE than SeeClear + MoGe-2.
• The prose on page 12 describes DA3's \(\delta_{1.025}\) change from 76.00% to 88.01% as “reducing”; Table 2 shows an increase, so this note retains the values and explicitly flags the directional wording error.

Highlights & Insights

• Adaptation targets input appearance instead of the depth network, enabling reuse of existing backbones. The paper verifies this with two backbones, but does not establish equal benefits for arbitrary future models.
• The CLIP class token is useful not because it carries finer texture, but because it avoids reintroducing texture behind transparent objects into generation conditioning. This illustrates why more conditioning information is not always better for a particular task.
• Retaining the original background makes opacification a local edit rather than full-image reconstruction. Avoiding irrelevant changes also matters for depth models that rely on global context.

Limitations & Future Work

• Explicit author-reported failures include large, nearly invisible glass panes with only partially visible frames and severely mislocalized masks; the former can retain depth leakage, while the latter places generated content in the wrong region.
• Reader assessment: plausible generated shading is not observationally verified geometry, and appearance priors can still hallucinate shape when context is weak.
• Reader assessment: approximately 7.65 s per image limits real-time robotic use, and the main paper does not provide a complete latency curve against instance count.
• The authors plan extensions to video and more complex specular, translucent, and multilayer transparent materials with temporal consistency and spatial coherence constraints; these are future directions, not completed results in this paper.

Related Work & Insights

• Compared with Depth4ToM, D4RD, and DKT: these methods adapt depth estimation itself to difficult materials, whereas SeeClear learns a transparent-to-opaque front-end mapping. It offers backbone compatibility at the cost of generation and segmentation dependencies.
• Compared with Marigold, GenPercept, and GeoWizard: these methods use generative priors for geometry prediction, while SeeClear first generates RGB appearance and then invokes a depth backbone. Evaluation should distinguish appearance-adaptation gains from backbone capability.
• Compared with Alchemist and Materialist: these emphasize controllable or physically consistent material editing, whereas SeeClear restricts editing to appearances useful for depth estimation. Its central evaluation criterion is downstream geometric error, not image realism alone.
• Possible extension, reader suggestion: test whether disagreement among depth maps from different opacification samples identifies low-confidence regions, but this requires additional experiments and sample agreement cannot prove geometric correctness.

Rating

• Novelty: 4/5. Generative appearance adaptation clearly separates transparent materials from depth backbones.
• Experimental Thoroughness: 4/5. Real and synthetic data, two backbones, and key ablations are covered, but training conditions and evaluation regions require careful comparison.
• Writing Quality: 4/5. The method and failure cases are explained clearly, with one directional typo in the prose; damaged cached formulas are a text-extraction limitation.
• Value: 4/5. Useful for offline reconstruction and input-adaptation research, with deployment constrained by runtime and segmentation reliability.