MAGiSt3R: Multi-Agent Feed-forward 3D Reconstruction from Monocular RGB Videos

Conference: ECCV2026
Paper: ECCV Paper
Project: MAGiSt3R
Area: 3D Vision
Keywords: multi-agent mapping, monocular reconstruction, submap registration, geometry-appearance fusion, pose graph optimization

TL;DR

MAGiSt3R predicts local geometry with VGGT, uses a shared MAGMA network for both intra-agent and inter-agent submap registration, and corrects drift through pose graph optimization, enabling multi-agent indoor reconstruction from uncalibrated RGB videos with mean multi-agent reconstruction errors of 5.37/3.36 in Acc./Comp. on ReplicaMultiagent and a two-agent runtime of 9 FPS.

Background & Motivation

When multiple moving cameras explore an environment together, reconstructing a room independently for each camera is only part of the problem: the system must also recognize shared regions and place independent maps in one coordinate system. Traditional SfM and MVS generally operate on images collected beforehand, whereas navigating cameras need to update maps during acquisition, making online SLAM the closer setting. NeRF and 3D Gaussian Splatting have advanced dense mapping, but many associated multi-agent systems require RGB-D, known calibration, and per-scene optimization at deployment. Online parameter updates add computational overhead, and rendering speed should not be confused with mapping throughput.

3D vision foundation models such as DUSt3R and VGGT offer a different route: predict point maps, depth, or camera parameters directly from RGB without optimizing a neural representation for every new scene. However, accurate geometry within a short clip does not automatically align consecutive clips or maps from different cameras. Submaps have independent reference frames, long sequences accumulate pose errors, and observations from different agents overlap less; RANSAC followed by ICP can therefore merge similar but noncorresponding structures. The paper's incremental goal is not to replace the local reconstruction foundation model, but to supply reliable geometric fusion across submaps and agents.

The authors approach this gap through two questions: which historical observations are useful for alignment, and how can geometry, appearance, and camera information be combined? MAGMA learns to map a registering submap into a reference submap's coordinate system, while the surrounding pipeline generates local geometry, detects loops, and propagates global corrections. An agent here is a camera-bearing observer, not an LLM agent performing language-based planning. Core Idea: reuse one fusion network to learn geometric correspondences between coordinate systems for both incremental single-agent mapping and multi-agent map merging, then constrain long-term consistency with classical pose graph optimization.

Method

Overall Architecture

The inputs are monocular RGB videos independently captured by multiple agents, without externally supplied camera intrinsics or depth. Each agent generates local submaps with VGGT and aligns them to its existing map with MAGMA; after detecting an inter-agent loop, the server reuses MAGMA to merge different maps. The final outputs are dense point maps and camera trajectories in a common coordinate system, not an optimized NeRF or Gaussian Splatting representation. The complete pipeline comprises local submaps and retrieval, MAGMA geometry-appearance registration, inter-agent loop fusion, and backend pose graph optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Multi-agent RGB videos"] --> Local["Local Submaps and Retrieval"]
    Local --> Merge["MAGMA<br/>Geometry-Appearance Registration"]
    Merge --> Cross["Inter-Agent Loop Fusion"]
    Cross -->|Overlap detected; reuse MAGMA| Optimize["Backend Pose Graph Optimization"]
    Merge -->|Intra-agent merging constraints| Optimize
    Optimize --> Output["Global point maps and camera trajectories"]

The diagram shows deployment-time data flow; ground-truth depth and camera parameters serve only as training supervision or evaluation references, not as online mapping inputs. "Feed-forward" means local geometry prediction and learned registration do not require per-scene neural optimization; it does not exclude iterative backend pose optimization. This distinction matters because the complete system is not a single optimization-free network invocation.

Key Designs

1. Local Submaps and Retrieval: retain historical evidence for subsequent alignment

The system processes 10 consecutive RGB frames per window with a stride of 5, so neighboring windows share observations. VGGT predicts depth, confidence, camera parameters, and camera tokens; back-projection with the predicted camera parameters produces point maps. Each submap initially uses its first camera as its reference frame, so the same physical location does not naturally have the same coordinates in different submaps. Points with confidence below 0.25 are removed to prevent clearly unreliable predictions from directly influencing registration. The middle frame of each window is stored as a keyframe, preserving historical observations for retrieval.

Meanwhile, a ViT supplies visual tokens, and DINOv2 SALAD supplies spatial tokens and global descriptors for retrieval. Global descriptors answer which images to select, whereas spatial tokens provide more local appearance information for subsequent registration; these roles should not be conflated. For intra-agent merging, the registering set retains the new submap, while the reference set contains the top 10 correlated views from the merged map and historical keyframes. The network therefore neither ingests the entire map history on every update nor restricts itself to the immediately preceding window. Retrieval matters because it supplies observations that may genuinely overlap, not merely because it reduces the input count.

2. MAGMA Geometry-Appearance Registration: update both registering and reference maps

MAGMA receives a reference set and a registering set: the former defines the target coordinate system, and the latter must be mapped into it. It encodes point maps with a two-dimensional convolution whose kernel size and stride are both 16, spatially aligning geometry tokens with visual patches. Geometry alone cannot reliably distinguish repeated walls, similar furniture, or weakly structured regions, so attention also combines visual tokens with SALAD spatial tokens. The resulting appearance tokens are added to geometry tokens and concatenated with camera tokens to produce unified decoder features. For the reference branch, the relation in the paper's Equation (5) is:

\[ \mathbf{F}_{ref}=[\mathbf{t}_{ref};\boldsymbol{\alpha}_{ref}+\boldsymbol{\gamma}_{ref}]. \]

Here \(\mathbf{t}_{ref}\) denotes camera tokens, \(\boldsymbol{\alpha}_{ref}\) appearance tokens, and \(\boldsymbol{\gamma}_{ref}\) geometry tokens; the semicolon denotes concatenation. The registering branch uses the same organization: geometry describes spatial structure, while appearance helps establish correspondences between different coordinate systems. Two decoders serve the registering and reference sets, with each block containing self-attention, multi-view cross-attention, and an MLP. The registering branch uses its own features as queries and reference features as keys and values, interpreting where the new submap belongs through reference observations. The reference branch is not an immutable container: its representation is also updated, allowing the existing point map to be refined during fusion.

DPT heads predict the registered point maps, refined reference point maps, and their confidence maps. A pose head with 4 self-attention layers and a linear layer jointly processes the two groups of camera embeddings to predict camera parameters. This head uses an asymmetric mask: reference embeddings exchange information within reference tokens, while registering embeddings attend only to reference tokens. The information flow makes the coordinate anchor explicit rather than treating two unaligned sets of camera representations as fully symmetric inputs. MAGMA consequently returns updated point maps and camera parameters, not merely an ICP-style rigid transformation; the inter-agent stage subsequently computes a relative map transformation from those predictions.

3. Inter-Agent Loop Fusion: reuse registration across overlapping maps

The system defines the first camera of the first agent as the global coordinate frame. When another agent creates a new submap, the server compares its global descriptors with historical keyframe descriptors from the first agent. A maximum correlation above 0.6 triggers a loop, indicating shared observations on which cross-map alignment can be attempted. This step proposes overlap candidates; it does not itself complete geometric fusion or guarantee correctness under every form of perceptual ambiguity.

After a loop is detected, the first agent's submap containing the most correlated keyframe becomes the reference set. The second agent retrieves the top 10 correlated views from its current submap and its own historical keyframes to form the registering set. The role assignment differs from intra-agent retrieval: historical retrieval expands the second agent's registering side, while the first agent's reference submap supplies the global anchor. MAGMA registers these point maps and camera parameters to the reference frame, after which the system estimates the local-to-global relative transformation and updates the map. Historical keyframes provide observations beyond the current short window, but the paper does not establish successful fusion at arbitrarily low overlap.

4. Backend Pose Graph Optimization: propagate local constraints across trajectories

Even accurate individual MAGMA merges can accumulate small errors into substantial drift over a long trajectory. Each submap therefore corresponds to an \(SE(3)\) pose graph node, and edges between neighboring submaps use relative transformations estimated by MAGMA during the intra-agent stage. Detected loops add further constraints, with inter-agent loop edges also estimated by MAGMA. The backend applies Levenberg-Marquardt optimization to reconcile relative node poses with these measured edges. Information matrices weight edge uncertainty, giving more reliable constraints greater influence. The resulting submap corrections update every camera pose within the corresponding submap, rather than correcting one keyframe while leaving other frames inconsistent.

MAGMA and this module are complementary: the network supplies better local constraints, and optimization coordinates them globally. The paper explicitly ablates PGO in Table 2, so the gains cannot all be attributed to the local reconstruction foundation model. Equations (13)-(14) are partially corrupted in the cached text; this explanation follows the adjacent prose without reconstructing the authors' exact objective or matrix-product notation.

A Worked Example

Consider two cameras starting at opposite ends of a room and later observing the same region near a doorway; this is an illustrative mechanism example, not an additional experiment. Each camera generates submaps using 10-frame windows and a stride of 5, with MAGMA exploiting correlated historical views to maintain its own map. Before shared observations are detected, the two maps do not acquire a common coordinate frame simply because they depict the same room. Once the second camera's new submap exceeds a correlation of 0.6 with a keyframe from the first camera, the corresponding reference submap is selected. Current and historical correlated views from the second side enter MAGMA, which predicts point maps and camera parameters in the reference frame and supplies a cross-map constraint. Finally, PGO propagates the new constraint to relevant submaps and camera trajectories, rather than merely overlaying a small point cloud at the doorway.

Loss & Training

MAGMA is trained on a mixture of ScanNet, ScanNet++, and Aria Synthetic Environment. These datasets were not natively captured in a multi-agent setting, so the authors create trajectories starting at the first and last frames of a scene, sampling every 2 frames to simulate different observers. Training targets submap fusion rather than optimizing a neural representation inside every test room. Training lasts 200 epochs with a batch size of 5 on 4 A100 GPUs and a learning rate of \(1.5\times10^{-5}\).

Supervision contains three parts: confidence-aware point-map registration loss, camera loss, and geometric consistency loss. Point-map registration normalizes predicted and ground-truth maps using the average Euclidean distance of valid points and includes confidence regularization, preventing scale differences from directly dominating the error. Camera supervision covers quaternions, translation, and a two-dimensional field of view, with the principal point assumed to lie at the image center. Geometric consistency randomly selects two views, reprojects using ground-truth depth and intrinsics together with predicted poses, and compares corresponding geometry in the global frame. Consequently, poses must not only approach their labels but also make cross-view geometry mutually consistent. The respective weights are \(\lambda_{reg}=1\), \(\lambda_{pose}=1\), and \(\lambda_{geo}=0.8\), with confidence regularization coefficient \(\beta=1\). Ground-truth depth supplies training supervision, but deployment still needs only RGB; this does not make the system an RGB-D input method. Cached Equations (4) and (9)-(12) are incomplete, so this note preserves verifiable loss meanings and weights without reconstructing damaged formulas.

Key Experimental Results

Main Results

ReplicaMultiagent is a synthetic two-agent indoor dataset, with reconstruction evaluated on Apart 0, Apart 1, Apart 2, and Office 0. The following values come from the four-scene multi-agent averages in Table 3 on page 12; RICP means RANSAC plus ICP for inter-agent merging. Acc. measures distance from predicted points to ground-truth geometry, while Comp. measures distance from ground-truth geometry to predicted points; both are errors, so lower is better. The cached table header does not clearly specify distance units, so the original values are retained without assigning centimeters or meters.

Method	Inter-agent fusion	Acc., lower is better	Comp., lower is better
DUSt3R	RICP	12.51	9.92
MASt3R	RICP	10.12	8.16
SLAM3R	RICP	14.29	9.06
MASt3R-SLAM	RICP	13.83	9.37
VGGT-SLAM	RICP	11.36	8.44
MAGiSt3R	MAGMA	5.37	3.36

Relative to VGGT-SLAM plus RICP, the complete system reduces Acc./Comp. from 11.36/8.44 to 5.37/3.36. This is a system-level comparison, not a controlled replacement of only the fusion module; the latter requires the ablation below. The paper also explicitly warns that single-agent and multi-agent reconstructions cover different spatial regions, so their errors cannot directly quantify the benefit of adding agents.

Ablation Study

Table 1 on page 11 fixes VGGT as the local reconstruction backbone and compares fusion methods, training settings, and component removals in the ReplicaMultiagent multi-agent setting. ATE RMSE is the root mean square absolute trajectory error, with lower values indicating better tracking; the original numerical scale is retained below.

Config	Fusion-module training setting	Acc.	Comp.	ATE RMSE
A: RICP	No learned fusion training	8.89	6.47	7.66
B: SL(4)	No learned fusion training	13.66	11.67	18.94
C: L2W	Multi-agent	9.71	7.16	9.19
D: MAGMA	Single-agent only	6.89	4.93	5.35
E: Full MAGiSt3R	Multi-agent	5.37	3.36	3.87
F: Without geometric consistency loss	Multi-agent	5.88	3.60	3.97
G: Without spatial tokens	Multi-agent	6.04	3.71	4.19

MAGMA already outperforms these alternative fusion methods when trained only for single-agent operation; multi-agent training further reduces ATE RMSE from 5.35 to 3.87. Removing spatial tokens increases this error to 4.19, showing that local appearance information beyond point maps contributes to registration rather than merely supporting retrieval. Removing geometric consistency loss yields an ATE RMSE of 3.97, supporting its contribution but not a claim that it drives the entire improvement. The RICP row in Table 1 and VGGT-SLAM plus RICP in Table 3 are different experimental configurations; their related backbones do not make 8.89 and 11.36 contradictory values.

Key Findings

Real-world evidence comes from the three-agent AriaMultiagent dataset, which lacks a complete ground-truth reconstruction and is therefore primarily evaluated through camera tracking. The following rows are clearly readable in Table 7 on page 14; all metrics are ATE RMSE, with lower values being better.

Method, uncalibrated RGB input	Room 0	Room 1	Average
VGGT-SLAM + RICP	9.59	8.47	9.03
MA-MASt3R-SLAM	7.59	31.15	19.37
MAGiSt3R	4.68	2.36	3.52

This supports the method's value for real indoor multi-observer videos, but does not validate outdoor operation or large groups of agents. In the two-agent ReplicaMultiagent runtime experiment in Table 8 on page 14, MAGiSt3R achieves 9 FPS, compared with 23 FPS for VGGT-SLAM plus RICP. "Almost 10 FPS" therefore represents an accuracy-speed choice, not the fastest mapping system; some other methods' runtimes in that table are taken directly from their original papers. Figure 5 on page 15 shows self-collected videos with approximately 15% overlap and moving people, but provides qualitative evidence rather than an error table establishing general dynamic-scene robustness.

Highlights & Insights

• The same MAGMA handles intra-agent and inter-agent registration, reusing the ability to transform coordinate systems rather than merely sharing an image encoder. Historical information accumulated by one agent can consequently support cross-map fusion.
• Retrieval and registration have separate roles: global descriptors identify potentially overlapping views, and local appearance plus geometry perform finer alignment. The transferable lesson is not to substitute retrieval similarity for geometric correspondences.
• Neural registration and classical PGO have distinct responsibilities. A useful interpretation is to learn better constraints and then coordinate them globally, rather than assuming that a neural network replaces the entire SLAM backend.

Limitations & Future Work

• The authors explicitly state that the system is not designed for agents experiencing different day/night or adverse-weather conditions, and evaluation beyond 3 agents remains future work.
• Training and primary evaluation focus on indoor environments; city-scale scenes, long-distance trajectories, and larger groups lack direct supporting evidence.
• Reader assessment: a central server participates in fusion, but the main text provides no systematic bandwidth, latency, or disconnection experiments, so 9 FPS is not an end-to-end distributed-deployment guarantee.
• Evidence boundary: AriaMultiagent lacks complete reconstruction ground truth, and the approximately 15% overlap experiment is qualitative; failure rates under low overlap and dynamic conditions still require stricter measurement.
• Extraction boundary: some attention, loss, and PGO formulas and crowded table rows are damaged, and units are not explicit in the readable metric headers. This note does not fill in unverified details.

Related Work & Insights

• vs VGGT / VGGT-SLAM: VGGT supplies local-window geometry priors, while VGGT-SLAM uses SL(4) optimization to handle ambiguities in uncalibrated reconstruction; MAGiSt3R focuses on learned submap fusion and multi-agent collaboration.
• vs SLAM3R: SLAM3R's L2W already embodies local-to-global alignment, whereas this paper combines geometry, appearance, spatial, and camera tokens and explicitly trains multi-agent fusion; the L2W comparison in Table 1 supports the practical relevance of that difference.
• vs MAGiC-SLAM: the latter uses Gaussian Splatting and RGB-D, while this system takes only RGB. Different inputs and representations mean RGB-D tracking errors are references, not a same-input contest.
• Research direction: measuring retrieval false positives, overlap ratios, and fusion errors together could distinguish failures in candidate discovery from failures in coordinate alignment. This is a reader-proposed follow-up experiment, not an established result of the paper.

Rating

• Novelty: 4/5. Learning-based submap registration unifies single-agent and multi-agent fusion with a clear systems goal, while reusing established backbones and optimization.
• Experimental Thoroughness: 4/5. Synthetic and real scenes, fusion ablations, and runtime analysis are included, but scale, communication conditions, and dynamic robustness remain limited.
• Writing Quality: 4/5. Component responsibilities and the relationship between learned registration and the backend are clear; some formulas in the available text extraction are incomplete.
• Value: 4/5. The method offers a practical route to collaborative mapping without known calibration or depth input, with further value dependent on validation at larger scales and in harder environments.