VOCA: Visual Odometry with Codec Awareness¶

Conference: ECCV 2026
arXiv: 2607.00189
Code: None (Project page https://tum-vision.github.io/voca)
Area: 3D Vision
Keywords: Visual Odometry, Video Compression, KLT Tracking, Motion Vector Prior, Codec Awareness

TL;DR¶

VOCA leverages the built-in motion vectors from the H.264 video encoding process as an initialization prior for KLT optical flow tracking. Built upon the Basalt framework, it achieves high-accuracy causal visual odometry on videos compressed by up to 100x, comprehensively outperforming existing methods on the EuRoC, TUM-VI, and MSD datasets.

Background & Motivation¶

Camera pose estimation is a fundamental component of spatial intelligence systems. From mixed reality headsets to autonomous UAVs, robotic assistants, and self-driving cars, every class of device relies on Visual Odometry (VO) or SLAM to perceive its motion trajectory in real-time. However, the VO/SLAM community has assumed raw, uncompressed images as default inputs since two decades ago—standard benchmark datasets like EuRoC and TUM-VI follow this convention without exception. Real-world camera systems are completely different: a stereo grayscale video stream at 640×480 resolution and 30fps produces more than 1 GB of data per minute, making video compression (H.264, AV1, VP9, etc.) almost mandatory in practical systems due to strict storage and bandwidth constraints. Quantization artifacts introduced by lossy compression—such as blur, blockiness, and reduced contrast—directly violate the photometric constancy assumption that traditional tracking algorithms rely on, leading to a drastic drop in feature matching and optical flow estimation accuracy.

This loss of accuracy under the "compress-then-analyze" paradigm has long been overlooked by the VO community. The only prior work addressing this is MoV-SLAM, which extracts EXPRESS features from H.264 macroblocks for descriptor matching. However, this method heavily depends on the macroblock structure of specific encoders and does not support fisheye cameras, limiting its practical applicability. More fundamentally, the core contradiction lies in the fact that while video encoders produce motion vectors internally to optimize rate-distortion, these vectors are merely coarse approximations of optical flow—the encoder is free to select any visually similar region as a prediction reference, regardless of whether the offset reflects physical scene motion. How to extract true motion cues for tracking from encoding noise is the core challenge in designing codec-aware VO systems.

The authors of VOCA noticed a previously neglected opportunity: although macroblock-level motion vectors in H.264 are noisy, translation-only, and do not always correspond to real physical motion, they cover displacements far exceeding the recovery range of image pyramids—perfectly complementing the shortcomings of pure KLT tracking in large-displacement scenarios. More importantly, these motion vectors are obtained for free as standard outputs of the decoder. Core Idea: Use the macroblock-level motion vectors obtained for free from the H.264 decoder as a translation initialization prior for KLT tracking, combined with dual-mode (pure KLT vs. motion-vector-guided KLT) parallel tracking and consistency validation to suppress anomalies caused by dynamic objects and mismatching. Through an I-frame bridging strategy, temporal consistency of priors between frames is maintained. Built upon the classic VO framework Basalt, this ensures highly accurate and robust causal pose estimation on hundred-fold compressed videos.

Method¶

Overall Architecture¶

VOCA is built on the classic VO system Basalt, with its core modifications concentrated in the front-end feature tracking, while the back-end sliding window bundle adjustment (BA) remains unchanged. The input is an H.264-compressed stereo video stream, and the output is the causal 6-DoF camera poses (using only past frame information). The system pipeline is as follows: the decoder simultaneously outputs reconstructed frames (with compression artifacts) and motion vectors for each macroblock. After extracting feature points from the current frame, two parallel KLT tracking pipelines are executed for each feature point—one uses motion vectors as translation initialization (MV mode), and the other uses traditional previous-frame pixel positions for initialization (OF mode). If both pipelines succeed, consistency verification is performed: consistent tracks are accepted, while inconsistent ones are rejected as potential tracking anomalies. Feature correspondences that pass are ultimately sent to the sliding window BA to solve for the pose.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["H.264 Compressed<br/>Stereo Video Stream"] --> B["Decoder<br/>(ffmpeg 2-pass encoding)"]
    B --> C["Reconstructed Frames<br/>(with compression artifacts)"]
    B --> D["Macroblock-level<br/>Motion Vectors"]
    C --> E["Feature Point Extraction<br/>(Inherited from Basalt)"]
    E --> F["Dual-mode KLT Tracking"]
    D --> F
    F --> G{"Consistency Verification<br/>MV vs. OF"}
    G -->|Consistent| H["Accept Track"]
    G -->|Inconsistent| I["Reject Anomalies<br/>Dynamic Objects/Mismatches"]
    H --> J["Sliding Window BA<br/>→ 6-DoF Pose"]
    I -.-> J
    J --> K["Causal Pose Output"]

Key Designs¶

1. Motion Vector as KLT Initialization Prior: Using the Encoder's Rate-Distortion Optimal Solution to Provide Large-Displacement Starting Points for Tracking

Standard KLT tracking uses the pixel coordinates of feature points from the previous frame to initialize the current frame's search. While effective under slow camera motions, once the actual displacement exceeds the search radius of the image pyramid, the optimization falls into local minima—a problem particularly prominent under fast rotation or accelerated motion in compressed videos. When calculating motion vectors, the H.264 encoder independently searches for the rate-distortion optimal reference block for each \(16\times16\) macroblock, generating a displacement vector \(\mathbf{d}_k\) pointing to the reference frame. Since \(\mathbf{d}_k\) points to the past direction while optical flow \(\mathbf{v}\) points to the future direction, \(-\mathbf{d}_k\) serves as a coarse approximation of the optical flow \(\mathbf{v}\). Basalt's KLT module optimizes an SE(2) transformation \(\mathbf{T}_{\text{2D}} = (\mathbf{R}_{\text{2D}}, \mathbf{t}_{\text{2D}})\) to minimize the normalized photometric error. VOCA initializes the translation component to \(-\mathbf{d}_k\) while keeping the rotation component as identity, allowing the KLT optimizer to refine the pose starting from a point much closer to the true solution. Experiments show that most motion vectors are only 1-2 pixels away from the final converged optical flow solution, whereas the traditional previous-pixel prior can drift by dozens of pixels in large-displacement scenarios, directly causing tracking failure.

2. Dual-Mode Parallel Tracking and Consistency Verification: Detecting Dynamic Objects and Mismatches via Discrepancies between Two Independent Pipelines

Motion vector priors are not a silver bullet. Encoders can match visually similar macroblocks in textureless regions (e.g., white walls or sky) to generate motion vectors that map pixel-wise but are entirely wrong in direction. Moreover, they might correctly track dynamic objects in the scene (e.g., a waving arm), but these large displacements can lead to trajectory drift in the global BA. The key design of VOCA is to run two independent tracking pipelines in parallel: MV mode (motion vector initialization) and OF mode (traditional previous-pixel initialization). If a feature is successfully tracked in only one pipeline, it is directly accepted—this implies one pipeline fell into a failure case while the other succeeded. If both succeed, the difference between the translation vectors of the two results is checked against a predefined threshold: a small difference indicates that both independent search paths converged to the same location, making the track highly reliable. A large difference indicates that at least one pipeline experienced an anomaly, and both are rejected. This design essentially utilizes cross-validation of two independent assumptions (encoder rate-distortion optimization + KLT photometric optimization) to filter anomalies without requiring any supervised data or pre-trained models.

3. I-Frame Bridging: Maintaining Prior Continuity via Constant Velocity Assumptions between Keyframes Lacking Motion Vectors

I-frames (intra-coded frames) in H.264 streams are predicted entirely intra-frame, containing no motion vectors. This poses a systemic challenge because I-frames typically appear at moments of severe scene content changes (when the encoder determines inter-frame prediction is too inefficient), which is precisely when establishing feature correspondences is most difficult. VOCA's solution is simple and practical: for I-frames, it directly reuses the decoded motion vector \(\mathbf{d}_k\) of the previous P-frame, assuming the motion between adjacent frames is approximately constant. Although the constant velocity assumption is rudimentary (especially during scene transitions), experiments show that in most cases, it successfully places the KLT optimizer within the basin of convergence. This is because encoders typically insert P-frames immediately after an I-frame to restore inter-frame prediction. The typical I-frame interval is in the order of dozens of frames (e.g., keyint=1000 with scene cut detection used in this paper), and the prior provided by the constant velocity assumption during this short window is usually superior to a blind search with no prior.

Loss & Training¶

VOCA does not involve any learnable components and requires no training. The entire pipeline relies on two classic optimization problems: the SE(2) non-linear least squares minimizing normalized photometric error in the KLT tracking stage, and the minimization of reprojection error in the sliding window bundle adjustment stage. The difference from Basalt lies solely in the modification of three front-end hyperparameters: relaxing the forward-backward consistency threshold (to compensate for high mismatch rates caused by compression noise), increasing the KLT patch size (to capture more image structure and improve robustness), and inserting the motion vector initialization and dual-mode parallel logic before tracking.

Key Experimental Results¶

Main Results¶

Dataset	Metric	VOCA	Basalt (Baseline)	OKVIS2	ORB-SLAM3	Gain (vs. Basalt)
EuRoC	ATE (cm) ↓	17.35	19.24	22.18	28.59	~10%
EuRoC	RTE (cm) ↓	1.669	1.947	3.084	∞	~15%
TUM-VI	ATE (cm) ↓	9.64	20.34	20.52	8.72	~53%
TUM-VI	RTE (cm) ↓	0.815	2.820	2.120	∞	~71%
MSD	ATE (cm) ↓	4.46	5.51	38.17	R	~19%
MSD	RTE (cm) ↓	0.755	1.478	5.223	R	~49%

(Note: ATE is the absolute trajectory error after SE(3) alignment, reflecting global consistency; RTE is the relative trajectory error with \(\Delta = 6\) frames, reflecting local smoothness, which is crucial for mixed reality experiences. \(\infty\) indicates sequence divergence; R indicates frequent resets. Basalt's baseline for MSD already includes VOCA's general improvements—relaxed consistency thresholds and larger patches—to ensure the incremental contribution of the MV prior is isolated and evaluated. DROID-SLAM as a learned SLAM baseline is compared separately in the appendix.)

Ablation Study¶

Configuration	EuRoC ATE	EuRoC RTE	TUM-VI ATE	TUM-VI RTE	Description
Original Basalt	19.50	1.964	18.00	2.518	Completely no motion vector prior
(A) OF \(\Rightarrow\) MV Fallback	17.20	1.871	10.40	1.380	Pure KLT first, fallback to MV if failed
(B) MV \(\Rightarrow\) OF Fallback	18.40	2.044	10.70	1.121	MV first, fallback to pure KLT if failed
(C) MV \(\parallel\) OF Parallel	18.70	1.710	9.10	0.919	Parallel tracking + consistency verification
(C) + I-frame Bridging (Full)	18.00	1.714	9.20	0.829	Complete VOCA

Key Findings¶

The dual-mode parallel strategy (C) performs significantly better than sequential strategies (A) and (B): On TUM-VI, the ATE of (C) (9.10) is far better than first-OF-then-MV (10.40) and first-MV-then-OF (10.70). This is because the sequential strategies cannot detect anomalies caused by dynamic objects and mismatches. Only when running two pipelines in parallel and performing cross-validation can the system distinguish between "true large displacements" and "erroneous motion vectors".
I-frame bridging is most effective on TUM-VI: Adding I-frame bridging reduces the RTE on TUM-VI from 0.919 to 0.829, which is because handheld camera motion is more random, and the prior value of the constant velocity assumption is greater when I-frames appear.
Robustness under high compression rates is the biggest highlight: Under ~100x compression (500kbps), VOCA's RTE remains essentially unchanged, while Basalt's RTE on TUM-VI degrades from ~1.0 to ~2.8. This shows that the motion vector prior has an inherent resistance to compression artifacts, since the encoder allocates more bits to motion vectors at low bitrates to ensure inter-frame prediction quality.
Comparison with DROID-SLAM reveals the design space of VO: On standard perspective cameras (EuRoC), DROID-SLAM yields a better ATE (9.92 vs 17.00). However, on fisheye cameras (TUM-VI), VOCA leads by a large margin (9.20 vs 45.58), indicating that learned SLAM is less robust to non-ideal camera models than hybrid schemes based on priors and classic optimization.

Highlights & Insights¶

Zero-cost prior signals: Motion vectors are embedded byproducts of the standard codec pipeline and do not add any transmission or computational overhead. VOCA is equivalent to obtaining large-displacement initialization signals from the decoder for free, which is extremely important for embedded systems (UAVs, AR glasses).
Victory of the "Prior + Optimization" paradigm: The most important insight of VOCA is not proposing an entirely new algorithm, but rather demonstrating the immense value of "using encoder byproducts as classic tracking initialization priors"—not replacing KLT, not adding learning modules, but simply altering the initialization strategy to significantly improve tracking robustness on compressed videos.
Generalizability of Dual-Mode Consistency Verification: This design is essentially an unsupervised cross-validation method that can be generalized to any system containing "coarse-grained priors" and "fine-grained optimizers", such as multi-scale prior fusion in depth estimation or validation of different initialization schemes in optical flow estimation.
Compression as an internet-scale perception signal source: VOCA brings VO performance on 100x compressed video to a practical level, meaning the massive amount of compressed video already existing on the internet can serve as perceptual training sources for downstream vision and robotics tasks—which could dramatically shift the training data paradigm of self-supervised VO.

Limitations & Future Work¶

VOCA currently only supports H.264 (the appendix provides a preliminary extension to AV1 as a proof of concept). Its performance across different codecs (HEVC, VP9, VVC) has not been systematically validated; differences in motion vector quality and density across different codecs could affect generalization.
The improvement on the Odyssey+ low-quality camera subset of MSD is limited. This suggests that when the original image signal-to-noise ratio is inherently low (VGA grayscale cameras), the overlay of compression noise and sensor noise significantly diminishes the value of the motion vector prior.
Systematic comparisons with end-to-end VO methods (VGGT, MapAnything, AnyCam) under compressed video settings are an obvious gap—although computationally heavy and non-causal, these methods might exhibit different degradation curves under extreme compression.
The quality of motion vectors depends on encoder configurations (merange, subme, ref for search range, sub-pixel accuracy, reference frame count). A performance gap may exist between consumer-grade default encoding configurations (such as real-time phone encoding) and the carefully configured two-pass encoding used in this work.

vs. MoV-SLAM: The only other VO method that utilizes H.264 encoding information. MoV-SLAM extracts macroblock structures as features (EXPRESS), whereas VOCA reuses motion vectors as tracking priors. MoV-SLAM does not support fisheye cameras and performs extremely poorly on MSD (diverging on almost all sequences). VOCA's KLT framework is more general across sensors and motion patterns.
vs. Basalt: The direct baseline and hosting framework of VOCA. By only introducing three front-end modifications (MV prior, dual-mode validation, I-frame bridging) without changing the back-end BA structure, VOCA achieves a relative reduction in RTE of 15%-71% across three datasets—highlighting the decisive impact of front-end initialization strategies on overall VO system performance.
vs. DROID-SLAM: A representative of learned SLAM. Under standard perspective and compression scenarios, DROID-SLAM leads due to its global optimization. However, under fisheye and compression scenarios, VOCA, as a GPU-free classic method, surpasses it by a wide margin. This reminds the community that in non-ideal sensor setups, classical geometric priors combined with optimization methods are more practical than pure end-to-end methods.

Rating¶

Novelty: ⭐⭐⭐⭐ [Using the long-overlooked motion vectors in video encoders as KLT initialization priors is a clean insight but has not been systematically implemented before. The accompanying designs of dual-mode consistency verification and I-frame bridging demonstrate comprehensive engineering consideration.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spanning three datasets (UAV/handheld/VR headset), five comparative methods (including DROID-SLAM), multiple compression rate sweeps, four sub-equipment categories of MSD, and an AV1 prototype extension—the experimental design is highly complete. It additionally provides reproducible ffmpeg encoding commands and metric derivations.]
Writing Quality: ⭐⭐⭐⭐ [The motivation is clear, the method is self-consistent, and the experiments are logically structured (ranging from main results to ablation studies, compression rate sweeps, and DROID comparisons). The minor drawback is the relatively sparse description of Basalt's existing components; non-VO experts might need to refer to the original Basalt paper to fully grasp the back-end BA details.]
Value: ⭐⭐⭐⭐⭐ [Video compression is an unavoidable aspect in real-world VO systems, yet it has been long overlooked by the community. VOCA offers a simple (three modifications), efficient (zero extra overhead), and reproducible (built on open-source Basalt) solution, and its RTE improvements are of direct value in AR/VR/UAV applications sensitive to local smoothness.]