Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Models¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/vita-epfl/Social-Mamba
Area: Time Series
Keywords: human trajectory forecasting, social interaction, state space models, continuous bidirectional scanning, multi-hypothesis prediction

TL;DR¶

Social-Mamba organizes unordered neighbors into a scannable social grid and uses Cycle Mamba for temporal, egocentric, and goal-centric interactions, achieving 0.72/0.92 m ADE/FDE on NBA-Full with 1.9M parameters, although lower computation does not mean the lowest latency in every setting.

Background & Motivation¶

Predicting a person's next trajectory segment requires more than extrapolating their recent velocity. In a crowded passage, an approaching pedestrian can change an avoidance maneuver; on a basketball court, teammates and defenders can alter a cutting route. Social-LSTM incorporates these influences through neighbor pooling, graph networks explicitly represent relationships through message passing, and Transformers exchange information directly across agents. These approaches share a requirement: retain each person's motion history while understanding how other people influence the agent being predicted. Fully connected attention is especially expensive because pairwise interaction computation and storage grow quadratically with the number of agents. How interactions are represented therefore affects both prediction error and the feasibility of real-time execution in crowds.

Mamba's selective state space model accumulates information along a sequence, offering an alternative based on linear-time scanning. However, a set of people has no natural order, and arbitrary neighbor serialization risks treating index order as a social relationship. Information direction presents another obstacle: a standard unidirectional scan only uses previously processed tokens, whereas interpreting current behavior often requires the entire available history. Existing trajectory methods consequently tend to use Mamba for temporal encoding and leave agent interactions to other structures. Even with two Mamba directions, independent states followed by output addition provide no direct recurrent-state transfer between the directions. The paper therefore aims not simply to accelerate attention, but to express social relationships as sequences that suit state space models.

The authors organize nearby trajectories around one ego agent and separate interactions into individual motion, current-state influence, and predicted-endpoint influence. The same neighbor history can then be read under different semantic anchors without constructing a full attention matrix. Importantly, goal-centric interaction does not add the true destination as an input; it aggregates information around the representation at the end of the prediction horizon. Core Idea: replace an unstructured neighbor set with three semantically anchored sequences, connect both directions through a continuous state stream, and dynamically fuse the resulting representations according to the scene.

Method¶

Overall Architecture¶

The input consists of observed 2D trajectories of an ego agent and its neighbors, while the output concerns only that ego agent rather than jointly generating everyone's future. The social grid first adds empty prediction slots; Cycle Mamba and triplet interactions extract different relationships in parallel; gated fusion and decoding then aggregate across agents and produce multiple candidate trajectories. To avoid the source's reuse of a letter for both agent count and feature dimension, this note denotes the retained agent count by \(N\), the total horizon by \(T=T_{obs}+T_{pred}\), and the embedding dimension by \(d\). The encoded grid has shape \(N\times T\times d\), and all three interaction branches return to this shape before position-wise fusion. Neighbors' prediction slots are latent representation slots, not observed futures, and do not imply access to their true future trajectories. Dashed edges in the diagram indicate training supervision; inference does not access future ground truth.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    Input["Observed 2D trajectories<br/>of ego and neighbors"] --> Grid["Social Grid"]
    subgraph Triplet["Cycle Mamba and Triplet Interactions"]
        direction TB
        Temporal["Temporal scan"]
        Ego["Egocentric scan"]
        Goal["Goal-centric scan"]
    end
    Grid --> Temporal
    Grid --> Ego
    Grid --> Goal
    Temporal --> Fusion["Gated Fusion and Decoding"]
    Ego --> Fusion
    Goal --> Fusion
    Fusion --> Output["K candidate trajectories<br/>for the ego agent"]
    Output -.-> Loss["Training: best-of-K MSE"]
    Truth["Future ground truth<br/>training only"] -.-> Loss

Key Designs¶

1. Social Grid: establish which agents interact and what each token represents

At the last observed timestep, the model selects spatial neighbors around the ego agent; the main text gives 10 m as an example local radius. This is a selection condition at the current timestep, not a neighbor graph continuously updated throughout prediction, nor confirmation of a shared threshold across all datasets. Each retained agent occupies one row, and columns contain the observed timesteps followed by the prediction timesteps. Observed slots contain 2D coordinates, future slots are initialized with zero vectors, and an MLP projects them into features. The resulting grid is an agent-by-time tensor layout, not a discretized spatial map with occupancy labels. This common layout supports row-wise temporal scans and a final global scan along the agent axis. Ego identity enters neighbor sequences explicitly through the current-state and endpoint tokens inserted later.

Figure 3 describes the preparation as a sorted egocentric grid, but the main text does not specify the sorting key or how ties are handled. It is therefore unjustified to assume sorting by distance, bearing, or player role, or to describe the architecture as a proven permutation-invariant set network. What is established is that neighbors are no longer processed without an ego reference: local filtering and explicit ego anchors provide structure. The inputs also delimit its scope: the method mainly uses coordinate trajectories and does not explicitly ingest a semantic map or court boundaries. The initial endpoint token comes from encoding a zero-filled slot, not from an externally supplied navigation intention. Distinguishing an endpoint-slot representation from known endpoint coordinates is essential to understanding goal-centric interaction.

2. Cycle Mamba and Triplet Interactions: read different semantic anchors through a continuous state stream

Cycle Mamba takes a feature sequence, constructs its reverse, appends the original sequence, and processes both continuously with the same Mamba model. The following notation makes the sequence order in Section 4.1, Eq. (4), explicit:

\[ S_{cycle}=[\operatorname{reverse}(S);S]=(s_L,\ldots,s_1,s_1,\ldots,s_L). \]

The crucial operation is not merely repeating the input, but retaining the hidden state at the concatenation boundary. The first token of the forward segment therefore receives a compressed memory of the whole input accumulated by the reverse segment. After scanning, the first output half is flipped back into the original order and aligned with the second half; the main text gives element-wise addition as the merge operation. Conventional bidirectional Mamba branches meet only at their outputs, whereas this design already transfers information through recurrent state. The scan length becomes \(2L\), but there is one set of scan parameters; parameter sharing does not automatically halve computation or latency. Bidirectional context remains context within the input sequence: slots after the observation horizon still contain no future ground truth. The introduction and Figure 2 caption differ from Section 4.1 in their description of directional order; this note follows Eq. (4) and the subsequent explanation that the forward segment inherits the state.

The three interaction branches receive the same initial grid in parallel and each use Cycle Mamba, rather than feeding one branch's output sequentially into the next. The temporal scan reads each agent's full time axis, retaining individual motion patterns and allowing observations to influence prediction slots. The egocentric scan inserts the ego's current-state token at \(T_{obs}\) between each neighbor's observation segment and prediction segment. This location connects a neighbor's past motion with the ego's current state within one scan, rather than concatenating two independently encoded vectors afterward. Following the scan, learnable weighted sums aggregate information from relevant positions and neighbors, and the extra inserted tokens are removed to restore the original temporal length. The goal-centric scan instead appends the ego's endpoint-slot token at \(T\) to each neighbor sequence before analogous scanning and aggregation. It organizes neighbor information around the predicted-endpoint representation, supplementing endpoint-related influences that a current-state anchor alone may miss. At the base model's input, this remains a latent endpoint representation, not a destination first estimated by another network and then supplied as a coordinate condition. Both social branches ultimately match the temporal branch's shape, preserving different contexts without forcing fusion at this stage. The extracted indices and operators in Eqs. (12) and (13) are corrupted, preventing reliable reconstruction of the exact aggregation implementation; this explanation follows the prose on weighted aggregation and shape restoration without inventing the authors' equations.

3. Gated Fusion and Decoding: select interaction sources before integrating across agents

The branches are not added with equal weights: the model concatenates their representations and uses an MLP with Softmax to compute three corresponding weights. It then combines temporal, egocentric, and goal-centric representations through an element-wise weighted sum, producing a fused grid. These weights depend on the input context rather than being three fixed dataset-level hyperparameters. The model can consequently emphasize individual history during independent motion and social branches when interactions are dense. This also explains the parallel arrangement: fusion can compare several interpretations instead of receiving one representation already overwritten by subsequent branches. The authors call this mechanism the social gate; an ablation tests learnable weighting, but the main text does not provide a quantitative scene-wise interpretability evaluation of the weights.

After gating, a global Mamba scan along the agent axis adds information exchange between rows. The three temporal-axis scans and the global agent-axis scan have different roles; the former should not be treated as already equivalent to arbitrary pairwise message passing. Only the ego agent's resulting representation is selected and passed to a decoder comprising bidirectional Mamba and \(K\) MLP projection heads. These heads output \(K\times T_{pred}\times 2\) candidate coordinates, rather than predicting only an endpoint followed by linear interpolation. Global scanning still propagates state sequentially, so linear scan cost does not guarantee equivalence under arbitrary neighbor permutations. Predicting everyone in a scene would require reorganizing computation around different ego selections or designing additional sharing; the main text does not establish complexity for joint whole-scene prediction.

A Worked Example¶

Under the NBA-Full protocol, the model observes 10 frames and predicts the next 20, corresponding to 2.0 s and 4.0 s respectively. This example illustrates information flow without inventing experimental coordinates, prediction probabilities, or gate weights. Suppose the ego agent is a player making a cut, with a nearby defender potentially blocking the intended direction of motion. The social grid creates 30 temporal slots for the retained players and other agents, with zeros in the final 20 slots. The temporal branch reads each agent's motion trend, while the egocentric branch inserts the ego's current token between each neighbor's history and future slots. The goal-centric branch appends the ego's endpoint-slot token to each neighbor sequence so that endpoint-related representations can collect neighbor influences. Cycle Mamba's continuous bidirectional processing operates on these constructed sequences, not on future game footage. After gated fusion, the global scan exchanges information across agents and the decoder outputs 20 candidate futures; this 20 is the default hypothesis count, which happens to equal the NBA prediction-frame count. During training, the candidate closest to the true future determines the sample's loss; at test time, no ground truth helps the model choose a final route.

Loss & Training¶

The model uses best-of-\(K\) mean squared error: it computes the average squared distance from each candidate to the true future and applies the sample loss only to the smallest one. The following restates Eq. (17) from the textual definition in Section 4.3, using a local prediction-horizon index for clarity rather than transcribing the corrupted equation:

\[ \mathcal{L}_e=\min_{k\in\{1,\ldots,K\}}\frac{1}{T_{pred}}\sum_{\tau=1}^{T_{pred}}\left\|\hat{y}_{e,k,\tau}-y_{e,\tau}\right\|_2^2. \]

The batch objective averages sample losses, with \(K=20\) by default. This encourages at least one trajectory to match the true future, but does not separately guarantee that every candidate is feasible, distinct, or assigned a calibrated probability. Multiple outputs therefore do not automatically constitute a complete model of the future distribution, and best-of-\(K\) evaluation does not replace a deployment-time selection policy. The main text states that NBA-Full training starts from scratch without the large-scale external pretraining used by some competitors; efficiency measurements use one NVIDIA A100, training batch size 128, and inference batch size 1. The authors also substitute Social-Mamba for the social encoder in the flow-matching framework MoFlow, but this does not turn the base best-of-\(K\) model into a flow-matching model. Implementation details are deferred to an appendix; the supplied full-text extraction ends with references on page 18 and contains no appendix, so the optimizer, complete training schedule, and MoFlow interface details cannot be verified here.

Key Experimental Results¶

Main Results¶

The default metrics are minADE20 and minFDE20 in meters, with lower values better: the former selects the candidate with the lowest horizon-averaged Euclidean distance, while the latter selects by endpoint distance alone. Their best candidates need not be identical, and the squared-distance training loss differs from both evaluation metrics. The table below excerpts source Table 1 on page 10, using NBA-Full with 10 observed frames and 20 predicted frames. An asterisk denotes pretraining on large-scale external trajectory data, so both architecture and training-data conditions vary across these methods.

Method	ADE ↓ (m)	FDE ↓ (m)	Parameters ↓ (M)	GFLOPs ↓
Social-Transmotion	0.78	1.01	2.0	0.87
Multi-Transmotion*	0.75	0.97	5.7	0.87
OmniTraj*	0.73	0.94	7.5	1.45
Social-Mamba	0.72	0.92	1.9	0.66

The absolute ADE advantage over OmniTraj is 0.01 m, whereas parameter and GFLOP differences are much larger; the accuracy improvement should not be portrayed as an order-of-magnitude change. Elsewhere, Table 3 on page 11 reports 0.25/0.38 on SDD versus NMRF's 0.25/0.39, with 8 observed frames and 12 predicted frames and distances measured in meters rather than pixels. Table 4 on page 11 reports 0.13/0.21 on JRDB at a 4.8 s prediction horizon versus NMRF's 0.15/0.23, using 9 observed frames and 12 predicted frames. The paper's summary of five benchmarks involves multiple NBA settings and should not be read as five entirely independent data sources.

Ablation Study¶

The following interaction ablation excerpts Table 8 on page 14, under the main text's NBA-Full ablation setting, reporting ADE/FDE in meters with the default \(K=20\). Temporal and global scans are always retained, so this tests the gains from the two additional social branches rather than establishing that the retained scans are indispensable.

Config	Temporal scan	Egocentric scan	Goal-centric scan	Global scan	ADE ↓	FDE ↓
Base interactions	Retained	Removed	Removed	Retained	0.735	0.939
Add current-state anchor	Retained	Retained	Removed	Retained	0.729	0.928
Add endpoint anchor	Retained	Removed	Retained	Retained	0.727	0.928
Full model	Retained	Retained	Retained	Retained	0.719	0.919

Each branch helps individually and retaining both performs best, supporting complementarity in this setting; the table provides no confidence intervals, so statistical significance cannot be claimed. Table 11 on page 14 reports 0.744/0.954 for equal addition versus 0.719/0.919 for learnable weighting, showing that fusion is another important source of improvement. Table 9 on page 14 reports 0.741/0.948 with 2.4M parameters for conventional bidirectional Mamba versus 0.719/0.919 with 1.9M for Cycle Mamba; the full-model parameter count is not halved.

Key Findings¶

Parameter count alone does not establish efficiency; the following excerpts Table 7 on page 12, measured on one NVIDIA A100 with inference batch size 1. Model memory follows the source table's terminology and is not total peak runtime GPU memory.

Model	Inference time ↓ (ms)	Model memory ↓ (MB)
Social-Transmotion	1.8	7.6
Multi-Transmotion	7.3	21.8
Social-Mamba	3.4	7.3

Social-Mamba is faster and smaller than Multi-Transmotion but slower than Social-Transmotion; parallel attention implementations retain an advantage for short sequences.
Table 5 on page 11 reports 0.71/0.87 → 0.70/0.85 for the MoFlow encoder replacement on NBA-LED, with parameters changing from 1.3M → 0.5M; the text's claim of a 2.3-fold smaller encoder does not match the ratio of these rounded values, so both are retained without forcing agreement.
NBA-Full Table 1 and the ablations display results at different precision: 0.72/0.92 and 0.719/0.919 should not be mistaken for different performance conclusions.

Highlights & Insights¶

Choosing whose token to insert and where to insert it makes social modeling more structured than pooling all neighbors into one vector. Current-state and endpoint slots provide different relational references.
Cycle Mamba changes how state flows are connected rather than simply adding another direction. It may transfer to encoding tasks requiring full-input context, but cannot be copied unchanged into online causal tasks.
Preserving branch-specific representations until gating leaves room to select information sources by scene. Ablations explain the value of this design more directly than aggregate leaderboard results alone.

Limitations & Future Work¶

The authors acknowledge occasional NBA boundary violations and illustrate one in Figure 4(i); without explicit map or boundary constraints, low average error does not guarantee executable paths.
From an evaluation perspective, minADE/minFDE assess only the best candidates and do not sufficiently establish collision rates, probability calibration, or consistency of joint multi-agent predictions. These are follow-up evaluations proposed by this note.
Fixed local neighbor selection may omit currently distant agents that enter interaction range later; dynamic neighbor selection is a possible extension, not an improvement already validated by the paper.
Neighbor sorting is underspecified, the supplied extraction lacks the appendix, and some equations are corrupted; reproduction should verify sorting, token aggregation, and training configuration in the code rather than treating this explanation as a complete implementation specification.

vs Social-LSTM / Trajectron++: the former uses social pooling and the latter graph-structured dynamic interactions; Social-Mamba expresses relationships through sequence anchors and state scans. It avoids fully connected attention but does not losslessly reproduce arbitrary explicit graph message passing.
vs U2Diff / Sports-Traj: the source characterizes their Mamba usage as primarily temporal; this paper extends Mamba to neighbor interactions with ego-specific semantics rather than merely replacing a temporal encoder.
vs MambaPTP: according to the discussion on page 3, its social interactions primarily occur during decoding with generic neighbor scans; this paper constructs triplet interactions during encoding. This is the authors' positioning, not an independently verified reading of that method's full text.
Relation to MoFlow: flow matching supplies the generative mechanism while Social-Mamba supplies social conditioning, so they can be combined. A useful test would add map conditioning and check whether better encoding reduces both best-candidate error and boundary violations.

Rating¶

Novelty: 4/5. Continuous bidirectional state flow and semantically anchored scans form a coherent design beyond replacing Transformer layers with Mamba.
Experimental Thoroughness: 4/5. Multiple data settings, module ablations, and efficiency comparisons are provided, but ordering robustness, calibration, and safety metrics remain untested.
Writing Quality: 3/5. The overall architecture is clear, but directional descriptions conflict, sorting details are limited, and the available extraction contains damaged equations.
Value: 4/5. The method offers a reusable lightweight social trajectory encoder; deployment still requires checking latency conditions, candidate selection, and scene constraints.