Multi-View Foundation Models

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: foundation models, multi-view consistency, cross-view attention, correspondence supervision, semantic preservation

TL;DR

The paper inserts multi-view adapters into 2D foundation models to jointly process images of a static scene, using geometric correspondence supervision and semantic regularization to reduce DINOv2's ScanNet++ localization error from 0.1029 to 0.0247 while retaining 0.9376 cosine similarity to the original features.

Background & Motivation

DINO, CLIP, and SAM learn transferable semantics from large collections of 2D images, but semantic similarity does not imply accurate geometric correspondence. Different parts of an object can have similar semantics, while the same physical point can produce different features when the viewpoint changes. Independent image encoding cannot use visible regions in another image to refine the representation of the current view. This affects cross-view matching, surface normal estimation in a shared coordinate frame, and segmentation applications that propagate an interaction prompt across viewpoints.

Existing feature-lifting methods use NeRF or Gaussian Splatting to construct a shared 3D feature field, but per-scene optimization and feature rendering add overhead. FiT3D and SnD transfer 3D supervision into feed-forward image models, while MEF adapts features using correspondences and a ranking-style objective; they still primarily process each image independently at inference time. At the other end, VGGT and π3 specialize in explicit geometry prediction, whereas this paper aims to preserve a 2D foundation model's native semantic space rather than train a model dedicated to geometric outputs. The challenge is therefore to introduce cross-view information while features are being formed, without allowing geometric training to overwrite existing semantic capabilities.

The authors borrow the idea of model inflation, inserting interactions between views inside the pretrained Transformer rather than appending a separate reconstruction module at the end. Geometric supervision relies only on precomputed correspondences, avoiding a new 3D feature-field fitting process for every backbone. This separates architectural joint observation from training-time semantic preservation and makes the same supervision reusable across feature spaces. Core Idea: exchange information between views inside the foundation model, supervise whether features locate the same 3D point, and anchor them to the original model so that geometric adaptation does not erase their semantics.

Method

Overall Architecture

The input is a set of RGB images of the same static scene; the full version also receives camera parameters, while the pose-free version omits ray conditioning. The outputs remain per-image feature maps, not a point cloud, mesh, or rendered novel view. Original Transformer blocks encode information within each view, and newly inserted adapters exchange information across views, alternating throughout the network. Camera-ray conditioning supplies geometric context to joint attention; during training, correspondence supervision constrains localization and semantic anchoring constrains changes relative to the original model. Dashed edges in the diagram denote training supervision, not ground truth required at inference time or additional inference stages.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Images["Multi-view images<br/>of one static scene"] --> Adapter["Interleaved<br/>multi-view adapters"]
    Camera["Camera parameters<br/>full version"] --> Rays["Camera-ray<br/>conditioning"]
    Rays --> Adapter
    Adapter --> Features["Per-view feature maps"]
    Features --> Output["Matching or downstream probing"]
    Features -.-> Corr["Direct correspondence<br/>supervision"]
    Matches["Training correspondences"] -.-> Corr
    Features -.-> Anchor["Semantically anchored<br/>fine-tuning"]
    Base["Original model features"] -.-> Anchor

Key Designs

1. Camera-ray conditioning: tell image tokens where their observations come from

With image texture alone, cross-view attention must infer both appearance relationships and camera relationships, making repeated textures particularly ambiguous. The full model uses the first view as its reference and computes a dense raymap from the camera parameters of each view. Section 3.1 describes each pixel's ray as 6-dimensional information comprising an origin and a direction, projected into conditioning channels through patch embedding. These conditions are concatenated channel-wise with the initial image features, giving subsequent cross-view interactions access to geometry rather than appearance alone. The raymap is neither predicted depth nor a preconstructed renderable 3D scene; it supplies camera observation geometry. This branch therefore needs camera parameters but does not require training correspondences at inference time.

The paper calls this a Plücker embedding, but cached Eq. (4) describes a concatenation of ray origin and direction. This differs from the common direction-and-moment formulation, so this note retains the verifiable ray-conditioning mechanism without inventing the exact coordinate transformation. The pose-free version simply removes this concatenation while retaining joint multi-view attention and the same training principles. It neither reverts to independent image processing nor adds a separate pose estimator. This distinction matters: the pose-free improvements show that geometric consistency does not come entirely from explicit camera inputs.

2. Interleaved multi-view adapters: exchange cross-view information after every semantic block

Within each original Transformer block, views still undergo spatial self-attention independently, retaining the pretrained within-view representation pathway. The adapter then combines patch tokens from all views into one sequence and applies multi-head self-attention followed by a feed-forward network. With \(M\) views and an \(H\times W\) feature grid per view, the cross-view attention sequence contains \(MHW\) tokens. Every patch can consequently access patches in other views, instead of attempting alignment only after all 2D features have been produced. The interaction outputs are split back into views and passed to the next original block, allowing geometric context to influence subsequent semantic feature formation. Here, 3D-attention means joint attention across views; it does not imply that tokens occupy an explicit 3D voxel grid.

To prevent a randomly initialized adapter from immediately disrupting pretrained features, the authors merge its output into the backbone through a zero-initialized linear projection. The residual relationship in Eq. (3) is:

\[ \mathbf{Z}_i^l=\tilde{\mathbf{Z}}_i^l+\operatorname{ZeroProj}(\mathbf{H}_i^l). \]

The first term is the current 2D block's output, and the second is the multi-view adapter's increment for that view. The zero projection initially prevents the adapter branch from changing backbone outputs, after which training gradually enables cross-view corrections. It does not force the final trained model to reproduce the original features; it controls only the initial disturbance during adaptation. The residual visualization in Figure 1 supports adding a geometric signal to features close to the original semantics, but it does not establish that the representation remains unchanged.

3. Direct correspondence supervision: require similar features to occur at the correct locations

Simply minimizing the feature distance between corresponding points admits an obvious loophole: outputting the same vector everywhere also makes pairwise distances small. The method adopts an existing geometry-aware dense loss, changing the supervision from making two vectors identical to recovering the correct location through their features. For a query point in a source image, its normalized feature is compared by cosine similarity with every location in the target feature map. A temperature-controlled softmax converts similarities into a spatial probability distribution, and SoftArgMax computes the expected location under that distribution. The training loss penalizes the Euclidean distance between this prediction and the ground-truth correspondence, across multiple matched points and view pairs. The correct location must therefore stand out against the entire target image; identical features everywhere cannot explain ground-truth matches distributed across different positions.

Correspondences are established by projecting 3D point clouds; the paper uses COLMAP as an example of obtaining such cross-view matches. The 3D information prepares supervision rather than requiring the network to render a new feature teacher for each backbone. As long as image-coordinate correspondences remain the same, the supervision can adapt DINOv2, DINOv3, CLIP, and SAM despite their different feature dimensions and semantic spaces. Cached Eqs. (6) through (8) have damaged formatting, so this note explains probabilistic localization from the prose without guessing missing normalization or aggregation details. Training and evaluation also differ: training uses differentiable SoftArgMax, while localization evaluation retrieves the most similar feature location in the target image.

4. Semantically anchored fine-tuning: limit the displacement of the original feature space

Correspondence supervision can improve matching, but it has no inherent reason to preserve the original classification semantics. The authors therefore retain an original foundation-model reference for each adapted feature, constraining both cosine direction similarity and closeness of feature norms. The direction term discourages vectors from rotating into unrelated semantic directions, while the norm term discourages scale changes that undermine the intended alignment. The overall objective combines correspondence loss with weighted semantic regularization rather than training for geometry first and projecting features back at inference time. The norm term in cached Eq. (9) has incomplete symbols, so this note does not substitute a familiar regularizer for the authors' exact implementation.

During optimization, the original foundation model is fine-tuned with LoRA, while the new adapter parameters are fully trained. LoRA restricts the degrees of freedom of backbone updates, zero initialization controls the adaptation starting point, and semantic regularization constrains the final representation drift; these roles are distinct. Removing regularization improves localization further but substantially reduces similarity to the original features, demonstrating a genuine conflict between geometric metrics and semantic preservation. The target is therefore not the lowest standalone localization error, but an original feature space that remains reusable after gaining geometric capabilities.

A Worked Example

Consider the 4 scene images used in a training sample, with one tabletop corner visible in several views; this is a workflow illustration, not an additional experiment. The full version first encodes camera-ray information into the corresponding image tokens, then alternates within-view extraction with cross-view exchange at each layer. The corner representation can use appearance and camera context from other views rather than relying only on a small texture region in its own image. During training, the source corner is compared with all target-image locations, and the expected location should approach the target corner obtained by projecting the point cloud. Meanwhile, original model features for the same images provide semantic references, preventing every token from reorganizing solely for matching. Once training finishes, neither correspondence ground truth nor original-model references are required as inference inputs.

With SAM as the backbone, a user can click a target in the first view, find the most similar locations in other views through the joint features, and use those locations as SAM prompts. Outputs still come from SAM's prompt and mask machinery; multi-view features geometrically propagate prompts rather than copying one mask into other images. Section 4.3 also reports further improvement from briefly fine-tuning the prompt encoder and decoder through self-supervision from original SAM outputs while freezing the backbone. The main paper presents this application primarily qualitatively, so it should not be described as having a quantitative cross-view segmentation accuracy table.

Loss & Training

Training uses 230 ScanNet++ scenes and 140K views, sampling 1,000 image sets per scene per epoch. Each set contains 4 images of \(512\times512\) pixels, and each iteration uses 128 sampled correspondences for the correspondence loss. The standard schedule lasts 24 epochs, while SAM and DINOv3 use 12 epochs. Optimization uses AdamW at learning rate \(10^{-4}\) with a linear schedule; both LoRA rank and scaling parameter are 32. The backbones are DINOv2 ViT-S/14 with register tokens, SAM ViT-B, DINOv3 ViT-B/16, and CLIP ViT-B/16. The readable implementation section does not provide numerical values for softmax temperature or regularization weight, so this summary is not a complete reproduction configuration. The surface normal experiment separately freezes the backbone and trains a lightweight multi-view self-attention probe that outputs 3 normal components and an uncertainty parameter. That probe uses an uncertainty-weighted objective, AdamW, 50 epochs, and learning rate \(5\times10^{-5}\); it is separate from foundation-model adaptation.

Key Experimental Results

Main Results

Correspondence evaluation covers 50 held-out ScanNet++ scenes and 115 generalization scenes from 8 datasets, spanning indoor, outdoor, real, synthetic, and object-centric settings. Localization error is normalized Euclidean distance between predicted and ground-truth matches, with lower values better; the readable prose does not specify the denominator, so it should not be interpreted as pixels or meters. The table combines source Table 1 on page 9 and Table 2 on page 11; all compared methods use small DINOv2, and ImageNet results are the paper's zero-shot KNN classification, not CLIP text-prompt classification.

Method	ScanNet++ localization error	Generalization localization error	ScanNet++ base-feature similarity	ImageNet Top-1	ImageNet Top-5
DINOv2	0.1029	0.1404	Not applicable	62.50%	84.37%
MEF	0.0978	0.1232	-0.0249	51.74%	74.44%
FiT3D	0.0858	0.1378	0.7484	48.68%	71.74%
SnD	0.0876	0.1267	0.6125	56.31%	80.37%
Ours, pose-free	0.0262	0.0861	0.9360	58.02%	80.91%
Ours, full model	0.0247	0.0782	0.9376	58.02%	81.21%

The full model reduces localization error relative to original DINOv2 by approximately 76.0% on ScanNet++ and 44.3% on the generalization set; these relative reductions are calculated from Table 2. However, Top-1 changes from 62.50% to 58.02%, a decline of 4.48 percentage points, so the appropriate claim is better semantic preservation than other lifting methods, not completely lossless adaptation. Feature cosine similarity and classification capability are different metrics: FiT3D has higher feature similarity than MEF but lower Top-1 accuracy.

Ablation Study

The following results come from source Table 5 on page 14, with localization error on ScanNet++; the three additive configurations are cumulative, while the final row removes regularization from the full approach.

Config	Localization error	Base-feature similarity	Note
Base DINOv2	0.1029	Not applicable	Original 2D features
+ LoRA Training	0.0347	0.9117	Single-view fine-tuning with correspondence training
+ Multi-View Adapter	0.0262	0.9360	Cross-view interaction without ray conditioning
+ Plücker	0.0247	0.9376	Full model with camera-ray conditioning
- Regularization	0.0204	0.1054	Lower geometric error but substantial semantic-space drift

The adapter reduces error from 0.0347 to 0.0262 while increasing feature similarity, showing that cross-view exchange need not improve geometry only at the expense of semantics. Ray conditioning further reduces error to 0.0247, although its contribution is smaller than the preceding changes from correspondence training and joint processing. Removing regularization reduces similarity from 0.9376 to 0.1054, explaining the final trade-off more clearly than focusing only on the improvement in localization error.

Key Findings

Source Table 3 on page 11 reports shared-3D-frame normal probing on NAVI: angular RMSE is 56.00 for DINOv2 and 32.17 for the proposed method; recall at the 30-degree threshold is 0.434 and 0.693, respectively, in the table's fractional notation. Table 3's caption calls these percentage recalls, but its cells use fractional values; they are preserved here rather than misreading 0.693 as 0.693%. Source Table 4 on page 13 shows SAM's ScanNet++ localization error decreasing from 0.1846 to 0.0244 with feature similarity of 0.9547; CLIP and DINOv3 also improve localization. These cross-backbone results support reusability, but feature similarity cannot replace a full evaluation of every backbone's native tasks. Figure 4 shows a flatter localization-error curve as viewpoint separation increases; this note does not extract unverified angle-specific values from the figure.

Highlights & Insights

• Consistency comes from joint observation. Instead of asking each image to independently infer a shared 3D answer, the method lets views negotiate representations inside intermediate layers, directly addressing large viewpoint changes.
• Supervision is decoupled from the backbone. Correspondence coordinates do not depend on a particular feature space, avoiding feature-field reconstruction when changing backbones; what is reusable is the geometric correspondence data, not universal weights requiring no training.
• Semantic preservation is both an objective and an evaluation criterion. Reporting matching error, base-feature similarity, and classification accuracy exposes the hidden cost of improving geometry while degrading foundation-model capabilities.

Limitations & Future Work

• Static scenes and camera conditions. The main method targets static scenes and its full version needs camera parameters; current results do not establish performance on dynamic objects, severe occlusion, or camera calibration errors.
• Joint-attention cost. Global interactions over \(MHW\) tokens create compute and memory pressure as the view count grows; this is architectural analysis, not a measured speed claim from the paper.
• Remaining semantic loss. The 4.48-percentage-point ImageNet Top-1 decline means preservation is not invariance, and more native downstream tasks should be evaluated.
• The probe is not a learning-free readout. Although the backbone is frozen, the self-attention probe is trained; the results show more accessible geometric information but do not strictly prove that the backbone alone performs all 3D inference.
• Evidence boundary. The assigned cache contains the main paper and references, not the cited supplement; numerical results for view count, latency, regularization strength, context stability, and correspondence density were not verified.
• Possible extensions. Local or sparse cross-view attention and systematic pose-noise evaluation are reasonable reader-proposed directions, not improvements already demonstrated by the paper.

Related Work & Insights

• Compared with FiT3D and SnD: these methods transfer 3D-consistency supervision into image representations, whereas this paper retains joint multi-view inputs at inference time; the comparison concerns both supervision sources and access to other views at test time.
• Compared with MEF: both improve geometric features through correspondences, but this method uses direct position supervision and layer-wise cross-view interaction rather than making a Smooth-AP-style objective its central training mechanism.
• Compared with VGGT and π3: those models focus on explicit geometry recovery, while this paper primarily outputs reusable semantic feature maps and does not aim to replace specialized reconstruction systems.
• Compared with temporal model inflation: interleaving pretrained 2D layers with new higher-order interaction layers is transferable, but camera-ray conditioning and static-scene correspondence supervision should not be equated with a dynamic-video motion model.

Rating

• Novelty: 4/5. The combination of joint multi-view architecture and semantic preservation is clear, although adapters, LoRA, and the geometric localization loss build on existing techniques.
• Experimental Thoroughness: 4/5. Generalization scenes, multiple backbones, semantic classification, and ablations are covered, but supplementary efficiency numbers and dynamic-scene evidence were unavailable for verification here.
• Writing Quality: 4/5. The main mechanism and comparisons are clear, but lossless-semantics language is stronger than the classification results, and ray notation and cached equations require caution.
• Value: 4/5. Useful for multi-view systems needing geometrically consistent semantic features, with practical gains to be weighed against additional view inputs and joint-attention overhead.