CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/jirongli/CapFrame
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Viewpoint Grounding, Camera Pose Optimization, MLLM, Compositional Framing

TL;DR

Addressing the limitation that conventional language-guided 3D methods only determine what object to view rather than how to compose and orient it, CapFrame formalizes the Text-Instructed Viewpoint Grounding task and introduces a Retrieve–Translate–Refine framework that derives geometric pseudo labels from instructions and optimizes the 6-DoF camera pose via differentiable 3DGS rasterization.

Background & Motivation

Recent advances in 3D representation technologies, particularly 3D Gaussian Splatting (3DGS), have unlocked photorealistic, real-time novel view synthesis for applications across virtual reality, digital content creation, and interactive scene editing. However, finding desirable virtual camera viewpoints in complex, sprawling 3D environments still requires laborious manual trial and error. As scene complexity and spatial scale grow, manual positioning scales poorly. Intent-aligned automatic viewpoint recommendation guided by intuitive natural language is therefore crucial for seamless 3D human-computer interaction.

Existing language-guided 3D understanding methods—such as LangSplat, LERF, and various camera trajectory synthesis pipelines—excel at object-centric semantic localization and navigation. Their design answers "what" to observe (which object is present), but overlooks "how" the subject should be framed within a single shot. In practice, human photographic instructions are inherently compositional, specifying explicit orientations (e.g., front or left profile view), in-frame spatial arrangements (e.g., positioned on the right third of the frame), or camera angles (e.g., tilted counterclockwise by 20 degrees). While multimodal large language models (MLLMs) offer strong zero-shot spatial reasoning to interpret such visual concepts, translating unstructured textual descriptions into precise 6-DoF camera parameters in 3D space remains an open challenge.

The fundamental tension stems from the continuous \(SE(3)\) camera pose space: an unconstrained global search is computationally prohibitive and prone to bad local minima, while natural language lacks direct, optimization-ready numerical coordinates for gradient propagation. The core idea of this paper is to decompose viewpoint grounding into a Retrieve–Translate–Refine pipeline, using an MLLM-based question-evaluation ranking over training views for coarse initialization, translating linguistic composition constraints into geometric pseudo labels (orientation and 2D layout boxes), and refining the continuous 6-DoF pose via differentiable 3DGS rasterization.

Method

Overall Architecture

CapFrame bridges abstract natural language instructions and continuous 6-DoF camera poses through three sequential stages: Retrieve, Translate, and Refine. In the Retrieve stage, the pipeline leverages the finite set of training views that reconstruct the 3DGS scene to discretize the search space; candidate poses are pre-filtered via a lightweight vision-language model and ranked with a fine-grained Question-Evaluation (QE) process powered by an MLLM. In the Translate stage, an MLLM extracts key subjects and predicts relative orientation offsets and 2D bounding-box layouts, which are mapped to 3D Gaussian subsets via Grounded SAM back-projection and visual orientation estimators. In the Refine stage, 3DGS renders differentiable subject opacity masks, and the camera pose parameters are continuously updated by backpropagating layout and orientation losses directly to \(SE(3)\) pose Lie algebra.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Instruction + 3DGS Scene and Training Views"] --> B["Semantic-aware Viewpoint Initialization<br/>FG-CLIP Filtering and MLLM Question-Evaluation Ranking"]
    B --> C["Language to Geometric Pseudo Labels Translation<br/>Subject-Gaussian Association + Target Orientation and Layout Parsing"]
    C --> D["Gradient-Based Camera Pose Optimization<br/>Multi-objective Layout and Orientation Losses backpropagated to SE(3)"]
    D --> E["Output: Refined 6-DoF Camera Pose and Aligned Rendered Frame"]

Key Designs

1. Semantic-aware Viewpoint Initialization: Search Space Pruning and Fine-Grained Question-Evaluation Directly optimizing camera poses from scratch in the continuous \(SE(3)\) manifold suffers from severe local minima and slow convergence. Because the training image set \(\mathcal{C}\) covers the 3DGS scene extensively, these viewpoints provide a natural discrete candidate space. To avoid costly MLLM calls across all views, the pipeline first computes global cosine similarities between instruction text embeddings \(f_T\) and image embeddings \(f_{I,i}\) using FG-CLIP, retaining the top \(M \approx |\mathcal{C}|/8\) candidates. To resolve the limitation that holistic embeddings struggle with intricate compositional constraints and cluttered multi-object scenes, CapFrame introduces a Question-Evaluation (QE) strategy: an MLLM (Qwen3-VL) decomposes the instruction into specific attribute questions \(\mathcal{Q}\) (e.g., "Is the subject visible?", "Is the subject located on the right side of the frame?"). The MLLM scores each candidate view across these granular questions, and the average score selects the top \(K=2\) poses \(\mathcal{C}_f\) as initial anchors.

2. Language to Geometric Pseudo Labels Translation: Subject–Gaussian Association and Continuous Geometric Regularizers Because text instructions lack direct numerical targets for optimization, CapFrame converts compositional language into structured geometric pseudo labels. First, the MLLM identifies salient subjects \(\mathcal{O} = \{O_1, \dots, O_n\}\). For each subject in the retrieved views, Grounded SAM generates 2D masks \(M_{ij}^{\text{SAM}}\), which are back-projected using 3DGS rendered depth into 3D points \(\mathcal{P}_{ij}\). A KNN search against Gaussian means identifies the subject-specific Gaussian subset \(\mathcal{G}_{Q,i}\) and its 3D centroid \(\mathbf{C}_{W,i}\). For viewpoint direction, the MLLM infers relative angular offsets \(\Delta d_i = (\Delta\phi_i, \Delta\theta_i, \Delta\gamma_i)\) (azimuth, elevation, roll), while Orient Anything estimates the subject's canonical orientation from images, yielding the target 3D orientation vector \(\mathbf{P}_{W,i}^{\text{orient}}\). For image composition, the MLLM predicts a 2D bounding box \(\mathbf{P}_i^{\text{layout}} = [x_{\min}, y_{\min}, x_{\max}, y_{\max}]\) in normalized image coordinates, which is adjusted for non-zero roll via a 2D rotation matrix around the box center.

3. Gradient-Based Camera Pose Optimization: Differentiable Alpha-Mask Layout and Multi-Vector Orientation Losses Starting from the retrieved pose \(\mathbf{T}_{CW} \in SE(3)\), camera parameters are optimized through Lie algebra updates \(\tau = [\Delta r, \Delta t]^\top \in \mathfrak{se}(3)\) via left composition \(\mathbf{T}_{CW} \leftarrow \exp(\tau) \circ \mathbf{T}_{CW}\). For layout alignment, CapFrame renders only the subject Gaussians \(\mathcal{G}_{Q,i}\) to produce a differentiable soft opacity mask \(M_i^g\) and its soft centroid \(\mathbf{C}_{P,i}\). The layout objective penalizes centroid distance to the target region boundary \(\mathcal{L}_{\text{center}}^i\), maximizes mask coverage inside the bounding box via \(\mathcal{L}_{\text{in}}^i\), and penalizes pixel leakage outside the box via \(\mathcal{L}_{\text{out}}^i\):

\[\mathcal{L}_{\text{in}}^i = 1 - \frac{\sum_{(u,v)\in\Omega} M_i^g(u,v) \cdot \mathbb{I}\big((u,v)\in\mathbf{P}_i^{\text{layout}}\big)}{\sum_{(u,v)\in\Omega} M_i^g(u,v) + \epsilon}\]

For orientation, three cosine alignment losses guide camera rotation: a gravity loss \(\mathcal{L}_{\text{gravity}}\) aligning the camera up-vector with the world up-vector (disabled when roll is explicitly requested), a forward-facing loss \(\mathcal{L}_{\text{forward}}^i\) aligning camera line-of-sight with the subject's target orientation, and a look-at loss \(\mathcal{L}_{\text{look-at}}^i\) aligning the forward vector toward the subject's 3D centroid \(\mathbf{C}_{W,i}\).

Loss & Training

The overall pose optimization objective combines layout and orientation terms:

\[\mathcal{L} = \mathcal{L}_{\text{layout}} + \mathcal{L}_{\text{orient}}\]

where the layout loss aggregates across all target subjects:

\[\mathcal{L}_{\text{layout}} = \sum_{i=1}^n \left( \lambda_c \mathcal{L}_{\text{center}}^i + \lambda_{\text{in}} \mathcal{L}_{\text{in}}^i + \lambda_{\text{out}} \mathcal{L}_{\text{out}}^i \right)\]

and the orientation loss is applied to all asymmetric subjects \(\mathcal{O}_w\) alongside the upright gravity regularizer:

\[\mathcal{L}_{\text{orient}} = \lambda_g \mathcal{L}_{\text{gravity}} + \sum_{O_i \in \mathcal{O}_w} \left( \lambda_f \mathcal{L}_{\text{forward}}^i + \lambda_l \mathcal{L}_{\text{look-at}}^i \right)\]

Pose refinement uses the Adam optimizer with a rotation learning rate of 0.007 and a translation learning rate of 0.005. Hyperparameter weights are set to \(\lambda_c = 1, \lambda_{\text{in}} = 6\), and \(\lambda_{\text{out}} = \lambda_g = \lambda_f = \lambda_l = 2\). Optimization runs for up to 1500 iterations, with early stopping triggered when loss changes fall below \(1 \times 10^{-4}\) or \(1 \times 10^{-5}\) depending on scene scale.

Key Experimental Results

Main Results

Evaluation spans 38 indoor and outdoor real-world scenes across 5 benchmarks (Mip-NeRF 360, Deep Blending, Tanks and Temples, LERF-OVS, and DL3DV-10K), testing 135 curated compositional instructions. CapFrame is compared against Interpolation-based Viewpoint Search (IVS), Sampling-based Viewpoint Search (SVS), and adapted trajectory generation baselines ChatCam and SplaTraj. Metrics include vision-language similarity (CLIP, SigLIP2), blind MLLM judges (GPT-5.4-mini and Gemini-2.5-Flash alignment scores AS and win rates WR), and a 33-participant perceptual user study.

Method	CLIP↑	SigLIP2↑	GPT AS↑	GPT WR↑	Gemini AS↑	Gemini WR↑	User Rating↑	User Preference↑
SplaTraj	0.241	0.276	3.08	3.0%	2.78	3.0%	—	—
ChatCam	0.258	0.283	3.42	3.7%	3.27	5.2%	—	—
IVS (Interpolation)	0.265	0.396	3.49	5.9%	3.19	8.1%	2.53	5.6%
SVS (Spherical Sampling)	0.275	0.435	4.06	14.8%	3.88	16.3%	2.52	16.7%
CapFrame (Ours)	0.282	0.448	4.75	72.6%	4.22	67.4%	4.50	77.7%

Ablation Study

The ablation investigates the quantitative gains of continuous pose refinement over coarse initial views, as well as the impact of removing individual pseudo labels under systematic perturbations.

Table 1: SigLIP2 text-image alignment scores before and after continuous camera pose refinement: | Stage | Mip-NeRF 360 | Deep Blending | Tanks & Temples | LERF-OVS | DL3DV-10K | Note | |---|---|---|---|---|---|---| | Retrieve (Initial Anchor) | 0.121 | 0.274 | 0.307 | 0.503 | 0.393 | Bounded by discrete training views | | Refine (Optimized Pose) | 0.244 | 0.406 | 0.339 | 0.708 | 0.500 | Continuous SE(3) optimization significantly lifts alignment |

Table 2: Sensitivity and ablation analysis under perturbations (relative to unperturbed baseline): | Perturbation Type | Experimental Setting | Rotation Error \(\Delta R\) | Translation Error \(\Delta t\) | \(\Delta\text{CLIP}\) | \(\Delta\text{SigLIP2}\) | Observation | |---|---|---|---|---|---|---| | Input Text Variation | Paraphrased instruction | 9.61° | 0.732 | +0.013 | +0.032 | Robust to semantic paraphrasing | | Initialization Quality | Initialized with Top-3 views | 8.99° | 1.268 | -0.014 | -0.061 | Optimization partly compensates for sub-optimal anchors | | Initialization Quality | Initialized with Top-5 views | 13.28° | 1.801 | -0.018 | -0.165 | Poor initialization increases local minima risk | | Pseudo Label Removal | Without layout labels | 4.08° | 5.701 | -0.024 | -0.236 | Translation drifts severely; framing collapses | | Pseudo Label Removal | Without orientation labels | 33.71° | 2.312 | -0.021 | -0.128 | Rotation error spikes; fails specified viewing angle | | Label Noise | Moderate noise (5-10px / 5-10°) | 8.39° | 0.219 | -0.012 | -0.048 | Resilient to minor pseudo label inaccuracies |

Key Findings

• Layout and orientation supervision are complementary and decoupled: The ablation clearly demonstrates that removing layout constraints inflates translation error \(\Delta t\) to 5.701 while leaving rotation relatively intact, whereas omitting orientation pseudo labels causes rotation error \(\Delta R\) to spike to 33.71°.
• Question-Evaluation (QE) overcomes holistic embedding failure: In cluttered environments, FG-CLIP frequently anchors on irrelevant background patches that match prompt keywords. QE forces the MLLM to verify visibility and coarse placement across explicit sub-questions, securing robust initialization.
• Continuous refinement decisively surpasses discrete search: Compared to dense discrete sampling (SVS/IVS), gradient-based pose optimization adjusts camera frustums continuously on \(SE(3)\), achieving a 72.6% win rate against all baselines under blind GPT-5.4-mini evaluation.

Highlights & Insights

• Geometric pseudo labels as differentiable intermediaries: Rather than attempting direct backpropagation through noisy visual-language features without explicit geometric grounding, CapFrame translates high-level prose into 2D layout boxes and 3D orientation targets, bridging natural language and continuous camera optimization.
• Efficient subject-isolated rasterization: Rendering only the target subject Gaussians \(\mathcal{G}_{Q,i}\) yields a differentiable opacity mask at minimal computation, enabling exact spatial loss computation without reconstructing or modifying the background representation.
• Transferable design pattern: The combination of Grounded SAM back-projection, KNN-based Gaussian association, and MLLM-driven geometric constraint derivation can be directly applied to interactive 3D scene editing, robot viewpoint planning for manipulation, and automated cinematic camera path generation.

Limitations & Future Work

• Cascading dependency on MLLM prompt parsing: The framework depends on MLLMs for view evaluation, entity extraction, and offset reasoning. Inaccurate entity detection or hallucinated bounding boxes can introduce systematic bias into downstream optimization.
• Sensitivity to 2D/3D perception foundation models: In scenes with severe occlusions, reflections, or transparent surfaces, Grounded SAM and Orient Anything may produce noisy masks or incorrect canonical vectors.
• Optimization latency: Iterating up to 1500 gradient steps takes multiple seconds per query, limiting real-time deployment in high-frequency interactive exploration. Future work could investigate direct regression networks or second-order pose optimizers.

Related Work & Insights

• vs SplaTraj & ChatCam: SplaTraj constructs continuous language fields for trajectory exploration, while ChatCam relies on direct CLIP image gradients for anchor placement. Both focus on target visibility rather than photographic framing. CapFrame explicitly enforces precise composition, orientation, and tilt constraints via geometric pseudo labels.
• vs LangSplat & LERF: Prior language-embedded field methods distill high-dimensional language features into every 3D primitive, incurring substantial memory and training overhead. CapFrame demonstrates that zero-shot viewpoint grounding can be achieved on standard 3DGS scenes without modifying pre-trained primitives, relying solely on test-time MLLM reasoning and 2D grounding.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pioneers the text-instructed viewpoint grounding task in 3DGS, providing an elegant bridge from linguistic framing instructions to differentiable pose optimization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across 38 scenes, 135 complex prompts, 5 benchmarks, 2 VLM metrics, 2 blind MLLM judges, and human user studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, cohesive pipeline structuring, and detailed ablation analyses.
• Value: ⭐⭐⭐⭐☆ Highly relevant for virtual reality cinematography, digital asset authoring, and autonomous embodied vision.