Unsupervised Pixel-Level Semantic Left-Right Understanding of In-the-Wild Images¶

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Left-Right Symmetry, Semantic Image Understanding, Unsupervised Learning, Vision Foundation Models, Geometric Prior Transfer

TL;DR¶

Addressing the fundamental challenge of pixel-level semantic left-right prediction on in-the-wild images where 3D geometry is absent, Pix2LR introduces an unsupervised cross-modal framework that swaps the order of prediction and surface aggregation, effectively transferring geometric chirality priors from limited 3D shapes to dense pixel predictions on open-vocabulary natural images.

Background & Motivation¶

Reflective symmetry—particularly left-right semantic symmetry—is an intrinsic and ubiquitous property observed across biological creatures and human-made objects. In the 3D domain, explicit geometric structures such as point clouds and meshes have enabled researchers to successfully resolve intrinsic and extrinsic symmetries, as well as vertex-wise chirality disentanglement driven by vision foundation models. However, extending this capability to 2D in-the-wild images has remained an open and ill-posed problem. Single-view 2D images inherently lack 3D spatial depth, and real-world scenes are heavily challenged by severe object self-occlusion, drastic non-rigid pose articulations, perspective distortions, and arbitrary partial truncations. Under such conditions, conventional 2D geometric symmetry axis detectors fail catastrophically.

Existing 2D symmetry understanding methods exhibit critical limitations. General symmetry axis detection approaches are strictly confined to objects with near-perfect extrinsic reflectional symmetry, collapsing under complex poses or occlusions. Concurrently, semantic correspondence frameworks attempt to fine-tune vision foundation model representations to distinguish left from right, but they rely heavily on annotated sparse keypoint supervision and fail to deliver dense, pixel-wise semantic segmentations. While recent studies confirm that vision foundation models (such as Stable Diffusion and DINOv2/v3) implicitly capture rich left-right semantics during self-supervised pretraining, these embedded representations are inherently noisy and lack global spatial chirality consistency.

The central technical barrier to importing unsupervised 3D geometric constraints into 2D image understanding lies in structural representation incompatibility. Prior 3D methods (such as \(\chi\)) require back-projecting and aggregating multi-view 2D image features onto a 3D mesh surface before learning a chirality projection over mesh vertices, which is fundamentally incompatible with in-the-wild images that have no underlying 3D mesh. This paper's critical angle of attack is to invert this pipeline sequence: by executing prediction before surface aggregation, a lightweight predictor operates directly on 2D dense feature maps to output continuous pixel-level chirality values, which are subsequently aggregated onto 3D shapes only when 3D geometry is available. Core idea: by reversing the sequence of prediction and surface aggregation on 2D feature maps, Pix2LR trains a shared pixel-level left-right predictor on a hybrid dataset of medium-scale 3D animal/human shapes and in-the-wild images, transferring 3D geometric chirality priors to open-world 2D pixel understanding in an entirely unsupervised manner.

Method¶

Overall Architecture¶

Pix2LR adopts a hybrid training paradigm across 3D shape and 2D image datasets, structured into three primary components: feature extraction with horizontal flip alignment, a shared pixel-wise left-right predictor, and joint optimization under 3D geometric and 2D symmetric consistency losses. The framework takes unannotated 3D meshes \(\mathcal{M}\) and in-the-wild images \(I_w\) with foreground masks \(M_w\) as inputs. For both rendered shape views \(I_r\) and natural images \(I_w\), multi-scale dense feature pairs are extracted via frozen vision foundation models (combining Stable Diffusion and DINOv3) across original and horizontally flipped views. A shared predictor \(P_\Phi\) maps dense features directly to continuous chirality values in \([-1, 1]\). Rendered predictions are back-projected onto mesh surfaces to compute 3D geometric losses, while in-the-wild predictions are constrained by a 2D masked flip-symmetry loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Hybrid Input Data<br/>3D meshes M & Natural images Iw"] --> B["Feature Extraction & Flip Alignment<br/>Extract original and un-flipped feature pairs via SD+DINOv3"]
    B --> C["Shared Pixel-Wise Predictor PΦ<br/>Regress continuous chirality maps in [-1, 1]"]
    C --> D{"Modality Routing"}
    D -->|Rendered View Branch| E["Inverted Surface Aggregation<br/>Back-project pixel predictions to vertex values χM"]
    D -->|Natural Image Branch| F["Foreground Masked Flip-Symmetry Loss<br/>Enforce 2D reflectional consistency Limg"]
    E --> G["3D Intrinsic Geometric Losses<br/>Constrain spatial geometry via Ldis + Lvar + Lfif"]
    G --> H["Backpropagate & Optimize PΦ"]
    F --> H

Key Designs¶

1. Feature Extraction & Flip Alignment: Mitigating Gravity Bias via Horizontal Reflection

To harness the latent chirality embedded in vision foundation models, Pix2LR builds an explicit reflection pair for each input image \(I\) (whether rendered view \(I_r\) or natural image \(I_w\)). The mechanism first horizontally flips the original image to obtain \(\bar{I}\), extracts feature maps through frozen foundation models, and horizontally reflects the feature map back to align with the coordinate space of the original image, producing the feature descriptor pair \((F, \bar{F})\). The deliberate choice of horizontal flipping rather than vertical or arbitrary-axis rotation stems from two physical principles. Geometrically, any 2D axis reflection is equivalent to a horizontal reflection combined with an in-plane rotation, inducing an identical 3D chirality transformation on the depicted object. Semantically, pretraining corpora for foundation models overwhelmingly depict upright objects due to gravity bias; upside-down images fall outside the natural data distribution, resulting in severe feature degradation, whereas horizontal flipping strictly preserves upright posture semantics.

2. Inverted Surface Aggregation: Bridging the Cross-Modal Representation Divide

Prior 3D vertex-wise state-of-the-art frameworks (such as \(\chi\)) follow an "aggregate first, predict second" paradigm, where multi-view 2D features are first projected onto 3D vertices, and a downstream network subsequently learns chirality classification on 1D vertex descriptors. This formulation intrinsically prevents the model from processing in-the-wild images lacking 3D mesh representations. Pix2LR overcomes this limitation by inverting the functional ordering: the lightweight predictor \(P_\Phi\) is placed upstream to operate directly on dense 2D feature maps \(F \in \mathbb{R}^{H \times W \times D}\), outputting dense pixel predictions \(S \in [-1, 1]^{H' \times W'}\). For \(N_r\) rendered views, the vertex-wise chirality value \(\chi_v\) for each mesh vertex \(v \in V_M\) is computed via post-hoc surface aggregation across corresponding visible pixels:

\[\chi_v = \frac{1}{\sum_{n=1}^{N_r} |\mathcal{I}_n^v|} \sum_{n=1}^{N_r} \sum_{(x,y) \in \mathcal{I}_n^v} S_r[n, x, y]\]

By moving the predictor before surface aggregation, both in-the-wild images and rendered views share an unobstructed path through \(P_\Phi\), achieving unified representations and enabling seamless cross-modal training.

3. Cross-Modal Joint Loss Optimization: Unsupervised Transfer of Chirality Priors

Without ground-truth annotations on real images, Pix2LR relies on a cooperative unsupervised loss formulation. On the 3D shape surface, aggregated vertex predictions \(\chi_M\) are constrained by three intrinsic geometric losses: the dissimilarity loss \(\mathcal{L}_{\text{dis}}\) forces intrinsically symmetric vertex pairs to adopt opposite signs; the total variation loss \(\mathcal{L}_{\text{var}}\) enforces spatial smoothness across mesh edges; and the fifty-fifty loss \(\mathcal{L}_{\text{fif}}\) penalizes unbalanced left-right surface partitions. On natural images, the model leverages foreground masks \(M_w\) (extracted via SAM) to enforce exact sign-inversion consistency between original and flipped predictions through the image loss \(\mathcal{L}_{\text{img}}\):

\[\mathcal{L}_{\text{img}} = - \frac{1}{N_w} \| (S_w + \bar{S}_w) \odot M_w \|_F\]

This formulation drives an elegant transfer mechanism: while 3D meshes are limited to humans and quadrupeds, their rigorous topological constraints train \(P_\Phi\) to ground an invariant concept of semantic chirality, which the broad semantic space of foundation models then generalizes to unseen categories such as trains, airplanes, and automobiles.

Loss & Training¶

The overall unsupervised objective combines the 3D shape geometric losses and the 2D image consistency loss:

\[\mathcal{L} = \lambda_1 \mathcal{L}_{\text{dis}} + \lambda_2 \mathcal{L}_{\text{var}} + \lambda_3 \mathcal{L}_{\text{fif}} + \lambda_4 \mathcal{L}_{\text{img}}\]

During training, the backbone feature extractors (Stable Diffusion v1-5 and DINOv3) remain entirely frozen, updating only the parameters of the lightweight predictor \(P_\Phi\) via the Adam optimizer. Each batch jointly samples \(N_r\) multi-view rendered images from BeCoS and \(N_w\) natural images from SPair-71k or PF-Pascal, alternating gradients to align 3D geometric manifold constraints with 2D reflectional consistency.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted across two real in-the-wild datasets with sparse keypoint annotations (SPair-71k, PF-Pascal) and a 3D shape rendered dataset with dense pixel annotations (BeCoS). Evaluation metrics comprise per-category average accuracy \(\text{AVG}_{\text{acc}}\) and cross-category consistency accuracy \(\hat{\text{acc}}\).

Table 1: Left-right prediction accuracy comparison across real and rendered datasets (%)

Dataset	Method Configuration	\(\text{AVG}_{\text{acc}}\)	\(\hat{\text{acc}}\)	Strongest Baseline	Performance Margin
SPair-71k (In-the-wild)	GeoAware [35]	70.1	64.0	-	-
SPair-71k (In-the-wild)	\(\chi_{\text{BeCoS}}\) [30]	73.6	73.0	-	-
SPair-71k (In-the-wild)	Ours (BeCoS+Both)	83.8	83.8	73.6 (\(\chi_{\text{BeCoS}}\))	+10.2 / +10.8
PF-Pascal (In-the-wild)	GeoAware [35]	63.2	52.2	-	-
PF-Pascal (In-the-wild)	\(\chi_{\text{BeCoS}}\) [30]	77.3	77.3	-	-
PF-Pascal (In-the-wild)	Ours (BeCoS+PF-Pascal)	86.8	86.8	77.3 (\(\chi_{\text{BeCoS}}\))	+9.5 / +9.5
BeCoS (Rendered)	SD+DINO [36]	83.7	83.7	-	-
BeCoS (Rendered)	GeoAware [35]	73.7	73.7	-	-
BeCoS (Rendered)	\(\chi_{\text{BeCoS}}\) [30]	71.4	71.4	-	-
BeCoS (Rendered)	Ours (BeCoS+Both)	87.4	87.4	83.7 (SD+DINO)	+3.7 / +3.7

Ablation Study¶

The paper evaluates the individual necessity of each loss component and the impact of the underlying 3D shape dataset on global accuracy \(\hat{\text{acc}}\).

Table 2: Ablation analysis on individual loss terms and 3D shape priors (\(\hat{\text{acc}}\), %)

Ablation Category	Configuration	Natural Images (SPair-71k)	Rendered Images (BeCoS)	Note
Loss Terms	Full model (BeCoS)	83.8	87.4	Complete framework
Loss Terms	w/o \(\mathcal{L}_{\text{var}}\) (smoothness)	55.6	53.9	Severe spatial fragmentation (-28.2% / -33.5%)
Loss Terms	w/o \(\mathcal{L}_{\text{dis}}\) (dissimilarity)	56.8	50.3	Chirality collapse without symmetric repulsion (-27.0% / -37.1%)
Loss Terms	w/o \(\mathcal{L}_{\text{fif}}\) (fifty-fifty)	60.9	61.7	Trivial unilateral solutions (-22.9% / -25.7%)
Loss Terms	w/o \(\mathcal{L}_{\text{img}}\) (image loss)	73.8	87.4	Unaffected on rendered shapes but drops 10.0% on natural images
3D Priors	FAUST (human only)	78.2	70.7	Limited topological variety restricts generalization
3D Priors	BeCoS (humans + quadrupeds)	83.8	87.4	Multi-species 3D geometry provides superior prior grounding

Key Findings¶

Crucial Role of 3D Geometric Losses: Omitting either \(\mathcal{L}_{\text{dis}}\) or \(\mathcal{L}_{\text{var}}\) collapses in-the-wild accuracy to near-random performance (55.6% and 56.8%), demonstrating that 2D foundation model representations alone cannot self-organize robust semantic chirality without explicit 3D geometric regularization.
Bridging Power of the 2D Image Loss: Removing \(\mathcal{L}_{\text{img}}\) maintains full accuracy on rendered 3D views (87.4%) but triggers a sharp 10.0% drop on SPair-71k in-the-wild images (from 83.8% to 73.8%), proving that \(\mathcal{L}_{\text{img}}\) is the essential bridge adapting 3D geometric constraints to in-the-wild image distributions.
Robust Out-of-Distribution Generalization: Despite training 3D shapes exclusively on articulated humans and quadrupeds, Pix2LR seamlessly generalizes zero-shot to rigid man-made objects (trains, airplanes, cars) and non-photorealistic artistic domains (paintings and anime).

Highlights & Insights¶

Inverted Prediction Pipeline: Flipping the classical 3D paradigm of "aggregate first, predict second" to "predict first, aggregate second" provides a clean, elegant structural solution that connects unstructured 2D images with structured 3D geometry.
Gravity-Bias-Aware Symmetry Modeling: By recognizing that visual foundation models suffer from severe distribution shifts on inverted images due to gravity bias, restricting reflection to the horizontal axis preserves pretraining feature fidelity while geometrically matching 3D chirality reversal.
Efficient Cross-Modal Prior Transfer: Demonstrates that comprehensive 3D annotations across all semantic categories are unnecessary; grounding a model on a modest, topologically rich 3D prior enables foundation models to broadcast spatial chirality across the entire visual open vocabulary.

Limitations & Future Work¶

Ambiguity in Thin Elongated Structures: Objects with minimal lateral thickness along the left-right axis (such as bicycle handlebars and pedals) present severe projection overlap, hindering precise boundary localization.
Degradation under Multi-Fold and Continuous Symmetries: The binary reflective formulation struggles with rotationally symmetric objects (e.g., bottles or cylindrical tables), where the model arbitrarily assigns left-right boundaries based on camera-facing perspectives.
Future Directions: Integrating instance-level cross-attention to resolve fine-grained thin parts; expanding 3D prior libraries with diverse man-made CAD models; and extending symmetry formulations to continuous and multi-axis representations.

vs \(\chi\) (Wang et al., 2025): \(\chi\) introduced unsupervised vertex-wise chirality prediction on 3D meshes but was structurally locked to 3D inputs due to its early surface aggregation. Pix2LR reverses this sequence, enabling the same network to ingest 2D in-the-wild images and achieving high-fidelity pixel-level predictions.
vs GeoAware (Zhang et al., 2024): GeoAware relies on supervised fine-tuning using annotated semantic keypoint pairs and cannot produce dense pixel-wise segmentations. Pix2LR is completely unsupervised, directly outputs dense chirality heatmaps, and outperforms GeoAware on SPair-71k by 13.7% in average accuracy (83.8% vs 70.1%).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first unsupervised pixel-level semantic left-right prediction framework for natural images; the inverted pipeline design is conceptually elegant.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across real images, rendered shapes, unseen categories, non-photorealistic styles, and extensive ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Problem motivation is sharp, mathematical derivations are transparent, and structural transitions are compelling.
Value: ⭐⭐⭐⭐☆ Highly valuable tool for disambiguating symmetric keypoints, enhancing category-level pose estimation, and guiding symmetric image editing.