Cambrian-P: Pose-Grounded Video Understanding

Conference: ECCV2026
Paper: Official page | PDF
Area: Video Understanding
Keywords: Camera pose, spatial reasoning, interleaved training, frame sampling jitter, pseudo-pose supervision

TL;DR

Cambrian-P adds per-frame camera pose regression to video MLLM training and reconciles geometry learning with question answering through interleaved training and frame sampling jitter, improving VSI-Bench from 69.2 to 73.7 against a matched no-pose baseline; the gains primarily reflect better learned representations rather than inference-time pose-token use.

Background & Motivation

Recognizing objects and actions in a video does not guarantee that a model can organize different views of a room into a coherent space. Relative-direction questions and route planning require distinguishing a camera turn from object motion, not merely identifying what appears in each frame. Camera position and orientation supply the missing relationship: they describe where each two-dimensional projection was observed within a shared scene.

Previous approaches introduce geometry through spatial question-answering data, external 3D encoders, or reconstruction networks. Cambrian-P adopts a lighter interface: train the language model's own hidden representations to predict camera poses instead of requiring a complete scene reconstruction before answering. Simply adding a regression head is insufficient, however. VQA favors fixed, whole-video uniform sampling, whereas pose learning benefits from variable starting points, changing intervals, and longer training. Pose estimation also benefits from strong augmentation that can invalidate the visual evidence needed for an answer. These mismatches can encourage trajectory memorization or unreliable language supervision.

Core idea: use camera pose as compact geometric supervision for video representations, while giving pose learning and VQA compatible sampling and supervision paths, so the model learns cross-frame spatial relationships without requiring explicit predicted trajectories throughout QA inference.

Method

Overall Architecture

The inputs are video frames and a question; the output is a text answer, with additional per-frame camera predictions during training when pose labels are available. The backbone follows Cambrian-S: a SigLIP2-SO400m visual encoder connects to a Qwen2.5 language model through an MLP projector. Added pose tokens extract geometric information from the language model, and a linear projector plus pose head converts their hidden states into camera parameters for joint training with the text objective.

The diagram organizes the method into architecture, supervision, and training orchestration. Interleaved training coordinates the preceding objectives, producing video QA capability and optional pose estimation capability. This is not a pipeline that first predicts a trajectory and then feeds that trajectory back as text to answer the question.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video frames and question"] --> Tokens["Pose Tokens and<br/>Regression Head"]
    Tokens --> Loss["Scale-Aware<br/>Pose Supervision"]
    Labels["GT or pseudo-pose labels"] --> Loss
    Loss --> Training["Interleaved Training<br/>and Sampling Jitter"]
    Input -->|VQA supervision| Training
    Training --> Output["Video QA / pose estimation"]

Key Designs

1. Pose Tokens and Regression Head: learn cross-frame geometry inside the language model

One pose token is appended after each frame's visual tokens. Its placement matters because the language model uses causal attention: this position can access the current frame and preceding visual information. There are two learned queries, not two added tokens per frame. The first frame uses \(c_{\mathrm{first}}\), while every subsequent frame shares \(c_{\mathrm{rest}}\). Distinguishing the first frame establishes the first camera's coordinate system as the reference for all subsequent poses.

The final-layer hidden state at each pose-token position passes through a linear pose projector and a VGGT-inspired pose head. The head contains four self-attention layers followed by a linear prediction layer. It predicts a nine-dimensional encoding: translation \(t_i\) with three components, rotation quaternion \(q_i\) with four components, and horizontal and vertical field of view \(f_i\). These are camera parameters, not human poses or dense depth maps. Since semantic and geometric information pass through the same language model, pose supervision can reshape the shared representations used by VQA rather than merely fit an independent postprocessor.

2. Scale-Aware Pose Supervision: make gradients comparable across video scales

Indoor scans, outdoor driving sequences, and non-metric reconstructions can have radically different translation magnitudes. Directly adding their translation errors would let large-scale scenes dominate the gradient. Non-metric trajectories introduce another problem: their numerical scale is arbitrary rather than physically standardized. Cambrian-P applies weighted L1 supervision to translation, rotation, and field of view, but treats translation scale explicitly. Translation error is normalized by the mean consecutive-frame distance of the ground-truth trajectory, and non-metric samples receive an additional scale alignment.

Let \(N\) be the number of sampled frames, \(t_i\) the target translation, and \(\hat t_i\) the prediction. The following definitions are typeset from the surrounding explanation in cached Section 3.2. The extracted equation layout is damaged; only the clearly recoverable scale definitions are reproduced, without inventing unspecified component-weight values:

\[ \bar d=\frac{1}{N-1}\sum_{i=2}^{N}\|t_i-t_{i-1}\|_2,\qquad s^*=\begin{cases} 1,&\text{metric},\\ \operatorname{stopgrad}\left(\frac{\sum_i\hat t_i\cdot t_i}{\sum_i\hat t_i\cdot\hat t_i}\right),&\text{non-metric}. \end{cases} \]

The translation term compares \(s^*\hat t_i\) against \(t_i\) and divides by \(\bar d\); rotation and field of view retain their respective supervision terms. Stopping the gradient through the alignment coefficient prevents the model from exploiting the alignment operation to reach degenerate solutions rather than learning better trajectories. Metric samples keep \(s^*=1\), preserving their known physical scale. The cache does not specify the numerical safeguards for nearly stationary trajectories, so it cannot establish how every zero-denominator case is handled.

3. Interleaved Training and Sampling Jitter: protect QA evidence while augmenting geometry

Repeated uniform sampling gives the same video the same pose targets at every iteration, allowing memorization of fixed correspondences. Replacing the video with short pose-oriented clips, however, can remove the context required by the original question. The solution is to mix VQA-only, pose-only, and jointly supervised samples within the same batch. Pose-labeled samples produce additional pose-training examples at an augmentation ratio \(\beta\), using random starting frames, dynamic intervals, and augmentations such as color jitter, blur, and grayscale. These additional examples receive only pose loss, avoiding pressure to produce the original answer from incomplete evidence.

The VQA path preserves whole-video coverage through uniform sampling, but lightly jitters its frame indices. For a video containing \(L\) frames, the jitter magnitude is \(\lfloor L\alpha\rfloor\). Each offset is sampled uniformly within the positive and negative bounds, followed by clipping and enforcement of index monotonicity. This changes the exact frame-pose pairs seen across iterations without turning the QA input into a local pose clip. Here \(L\) denotes the video's total frame count to distinguish it from the sampled-frame count \(N\) used above.

The same training interface also accepts pseudo poses. The authors use videos associated with CamS-590K, a subsample of Cambrian-S-3M, run scene-cut detection and Qwen3-VL quality filtering, estimate poses with VIPE, and apply post-filtering before inserting the resulting supervision into interleaved training. This does not require VIPE at QA inference time. Instead, it extends geometric training beyond the limited indoor sources with ground-truth trajectories to in-the-wild video. Filtering thresholds and pseudo-label confidence policies are not fully specified in the available cache.

A Worked Example

Consider the route question in Figure 1: start by the window, face the waste bin, and then navigate to the bathroom. The VQA path samples across the video to retain evidence about the window, waste bin, and bathroom from different viewpoints. Pose tokens simultaneously learn how the camera translates and rotates relative to the first frame. The pose-only path can use shorter, dynamically sampled clips to learn motion, but is not required to answer the full route question.

After training, the QA model should connect those observations before choosing a turn. The cached illustration does not preserve enough spatial layout to determine the correct option reliably, so this note does not invent a left- or right-turn answer. More importantly, Table 10 reports VSI-Bench 72.0 even when pose tokens are removed at QA inference. The learning process therefore should not be conflated with an answer explicitly reading a predicted trajectory.

Loss & Training

The total objective combines cross-entropy over answer tokens with pose regression. The expression below is typeset from the paper's textual description; unavailable labels disable the corresponding loss rather than being fabricated:

\[ \mathcal L=\mathcal L_{\mathrm{NTP}}+\lambda_{\mathrm{pose}}\mathcal L_{\mathrm{pose}}. \]

Training starts from Cambrian-S-7B stage 3 and follows its stage 4 recipe for end-to-end fine-tuning. AdamW learning rates are \(1\times10^{-5}\) for the language model and visual projector, \(2\times10^{-6}\) for the visual encoder, and \(1\times10^{-4}\) for the randomly initialized pose projector and head. Defaults are \(\beta=1\), \(\alpha=0.005\), and \(\lambda_{\mathrm{pose}}=0.2\), with batch size 256 on 64 H200 GPUs.

The main VQA comparisons use only VSI-590K unless stated otherwise. A separate pose-focused experiment adds MapAnything data and changes the settings to \(\beta=20\) and \(\lambda_{\mathrm{pose}}=0.5\). It does not share the ordinary VQA training budget, so its best pose results and the best QA results should not be presented as measurements from a single unified configuration.

Key Experimental Results

Main Results

The following selection comes from Tables 1, 2, and 3. Higher scores are better; gains are score-point differences, not relative percentage improvements. VSI-Bench averages evaluation scores for numerical and multiple-choice questions, so its columns should not all be interpreted as ordinary classification accuracy.

Benchmark / subtask	Comparator	No-pose or comparator score	Cambrian-P	Gain
VSI-Bench average	Cambrian-S-7B dagger	69.2	73.7	+4.5
VSI-Bench absolute distance	Cambrian-S-7B dagger	53.7	60.1	+6.4
VSI-Bench relative direction	Cambrian-S-7B dagger	82.0	89.5	+7.5
VSI-Bench route planning	Cambrian-S-7B dagger	38.7	52.6	+13.9
VSTemporalI-Bench average	Cambrian-P w/o Pose	62.4	68.9	+6.5
VSTemporalI-Bench camera movement direction	Cambrian-P w/o Pose	67.7	87.7	+20.0
MindCube	Cambrian-P w/o Pose	34.3	38.4	+4.1
MVBench	Cambrian-P w/o Pose	51.9	53.5	+1.6

The dagger identifies Cambrian-S fine-tuned only on VSI-590K. The VSTemporalI-Bench experiment additionally uses VLM-3R training data and must be distinguished from the VSI-590K-only out-of-distribution evaluations. Route planning is absent from VSI-590K's training tasks, making its improvement notable evidence of task transfer.

Ablation Study

This selection combines Tables 8, 10, and 11, using the analysis setup of 32 frames and 196 visual tokens per frame, with inference token presence varied in its dedicated block. These are separate ablations, so differences between arbitrary rows do not isolate a single factor.

Experiment block	Configuration	VSI-Bench	ScanNet ATE, lower is better
Components	No-pose baseline	67.3	Not reported
Components	Pose loss + interleaved training, no jitter	71.2	0.144
Components	Pose loss + jitter, no interleaved training	69.4	0.259
Components	Pose loss + interleaved training + jitter	72.0	0.141
Inference	Pose tokens in both training and inference	72.0	Not reported
Inference	Pose tokens in training, removed at QA inference	72.0	Not reported
Loss	Depth supervision only	69.4	0.371
Loss	Pose + depth supervision	71.7	0.156
Loss	Translation supervision only	70.7	0.205
Loss	Rotation supervision only	69.7	0.287

Adding interleaved training to pose supervision with jitter improves the score from 69.4 to 72.0. Adding jitter to pose supervision with interleaved training improves it from 71.2 to 72.0. Both matter, but this table does not provide a complete causal decomposition across every possible combination. Pose supervision exceeds depth-only supervision by 2.6 points, while their combination is 0.3 points worse than pose alone. Denser geometric targets therefore do not automatically yield better QA representations.

Key Findings

The pseudo-pose extension is reported in Table 4. All rows below use 128 frames and the same VSI-590K + CamS-590K training data, allowing a comparison as pose supervision is added progressively.

Pose supervision	Fraction with pose supervision	VSI-Bench	MVBench	Perception Test	EgoSchema
None	0%	70.9	68.0	66.9	71.2
GT	25%	73.7	67.9	67.8	71.7
GT + Pseudo	48%	73.9	69.3	67.9	73.6

Moving from GT to GT + Pseudo adds 1.4 points on MVBench and 1.9 on EgoSchema. This supports the practical value of in-the-wild pseudo poses, but also increases the fraction of supervised samples; it is not a pure label-quality comparison at a fixed label count.

In the pose-focused experiment, ScanNet ATE is 0.078, better than CUT3R's 0.096 and StreamVGGT's 0.127, while offline VGGT reaches 0.035. Cambrian-P's TUM-dynamic and Sintel ATE values are 0.046 and 0.239, respectively, still behind CUT3R's 0.045 and 0.215. ATE measures overall trajectory error, while RPE measures errors in relative pose relationships. All reported pose metrics use Sim(3) alignment, so these results do not establish fully accurate absolute scale without alignment.

Table 9 reports pose gains of +4.7, +2.8, and +2.5 points for frame/token configurations 32/196, 64/64, and 128/64. Frame count and tokens per frame change together, preventing attribution of all differences to sequence length alone. Figure 5 reports a +10.2-point gain for relative-direction questions involving far objects versus +2.3 for near objects. This supports improved global cross-view relationships, but does not directly demonstrate an explicit internal 3D map.

Highlights & Insights

• The supervision interface may transfer better than a reconstruction output. One pose token per frame connects geometry learning to the language model, and QA gains do not require retaining it at inference. Geometry can shape the representation during training rather than becoming a permanent deployment dependency.
• Sampling determines whether supervision remains valid. When a pose-oriented clip lacks the context needed for the question, disabling text loss is more defensible than retaining the original answer. The same principle applies to multi-task training with action localization or local video crops.
• Compact geometry can outperform a denser target. Pose beats depth in these experiments, which does not imply that depth is universally unhelpful. It suggests matching the auxiliary task to the cross-frame relationships required by the downstream question rather than selecting it by supervision density alone.

Limitations & Future Work

• Boundaries exposed by the authors' experiments: With limited data, the randomly initialized pose head struggles to converge. Increasing pose-training iterations or input frames does not guarantee simultaneous monotonic improvements in QA and pose error; the objectives remain sensitive to capacity and training budget.
• Pose superiority is specific: The ScanNet advantage concerns ATE among streaming models, not every metric on every dataset. It also does not establish the long-term robustness of a complete SLAM system.
• Assessment in this note: Few additional architectural components do not imply low total cost. Default training uses 64 H200 GPUs, and the available cache omits the latency appendix referenced in the main text, preventing verification of runtime or added parameter counts.
• Supervision quality remains incompletely characterized: VIPE pseudo poses depend on filtering, but the cache does not fully specify thresholds, noise-stratified evaluations, or numerical protection for nearly stationary trajectories. Confidence-weighted supervision and robust scale normalization are concrete directions for further study.
• Statistical and interpretive limits: The selected results do not include multi-seed error bars. Distance-based gains and the inference-token ablation support a representation-learning account, but cannot alone establish a reliable global world model.

Related Work & Insights

• Cambrian-S: Supplies the visual-language backbone and spatial QA data. Cambrian-P primarily adds a geometric supervision interface and compatible training dynamics, rather than replacing the language model. A data-matched fine-tuning baseline is more informative about its contribution than cross-model rankings.
• VGGT and 3D feature injection: Cambrian-P borrows the pose-head design but predicts poses from language-model states instead of relying on fixed features from an additional 3D encoder. Comparisons with VLM-3R and GeoThinker should focus on how geometry enters the model; scores under different training setups cannot establish universal superiority of one interface.
• CUT3R, StreamVGGT, and VIPE: The first two are specialist streaming geometry comparators; VIPE provides training pseudo labels. Their roles differ, and VIPE is not a required tool at Cambrian-P's QA inference time.

Rating

• Novelty: 4/5. Pose regression is established, but integrating lightweight geometric supervision with an explicit treatment of video-QA training conflicts forms a coherent contribution.
• Experimental Thoroughness: 4/5. Covers QA, pose estimation, components, losses, and pseudo-label scaling, but lacks error bars and a verifiable full efficiency appendix in the available source.
• Writing Quality: 4/5. The training conflicts and ablation logic are clear, although main results, frame-count analysis, and pose-focused configurations require careful separation; damaged equation extraction increases reading overhead.
• Value: 4/5. Offers a transferable multi-task supervision strategy and uses inference-token ablations to clarify where the gains originate.