Silhouette-based Gait Foundation Model¶
Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/ShiqiYu/OpenGait
Area: Human Understanding
Keywords: gait recognition, part-aware pretraining, self-supervised learning, foundation models, gait healthcare
TL;DR¶
FoundationGait combines multisource silhouette sequences with independent body-part processing before encoding, preserving complementary local information as gait models grow and achieving 48.0% and 64.5% self-supervised Rank-1 on Gait3D and OU-MVLP, respectively, while supporting linear probing and fine-tuning for healthcare tasks.
Background & Motivation¶
Gait recognition seeks identity cues in body shape and movement during walking rather than relying on clothing texture or backgrounds. Silhouette sequences are therefore a common input: low-resolution binary foregrounds reduce appearance interference but concentrate useful information in contours and subtle motion. Many methods divide the final feature map into horizontal strips and aggregate information from different body regions. This assumes that the strips still contain distinct local evidence when pooling occurs. However, the paper finds that enlarging DeepGaitV2 and GaitSSB does not consistently help, and insufficient training data does not fully explain the problem.
Even after scaling GaitSSB on more than 2M sequences, the authors observe increasingly homogeneous part features and saturated overall recognition gains. If a deep network has already mixed similar whole-body information into different height positions, splitting its output into strips cannot recover the lost local diversity. Figure 3 supports this interpretation through individual-part versus fused recognition performance, while the Grad-CAM results in Figure 2 suggest that larger models may rely on a few cues around the head or chest. This does not prove that every scaling failure has the same cause, but it identifies an actionable bottleneck: restrict cross-part mixing before features are formed. Meanwhile, enlarging a homogeneous dataset is insufficient; many walking videos from one source may still miss important variations in clothing, viewpoint, and atypical gait.
The paper consequently puts identification and healthcare analysis into a shared pretraining problem: learn gait representations from multiple sources, then attach outputs for specific tasks. Identity and disease labels are excluded from pretraining so that representation learning is not explicitly restricted to one classification objective. Here, a foundation model means a scalable pretrained backbone within the gait domain, not a general model that understands arbitrary video or directly produces clinical diagnoses. Healthcare support comes from subsequent labeled linear probing or fine-tuning, rather than unsupervised disease discovery. Core Idea: split body regions before student encoding, encode them independently, and restore the whole-body structure, while multisource data and a full-view teacher constrain the representation so that larger models preserve rather than erase fine-grained gait cues.
Method¶
Overall Architecture¶
FoundationGait uses a deeper DeepGaitV2 backbone and takes silhouette sequences normalized to \(64\times44\) as input. During pretraining, two non-overlapping clips are sampled from the same sequence, independently augmented, and passed to the teacher and student. The teacher sees a complete clip; the student splits its clip along height at a specified granularity, processes the parts independently with a shared encoder, and reassembles their feature maps in spatial order. The reassembled output passes through spatiotemporal aggregation, a projector, and a predictor for contrastive learning against the teacher's full-view projected feature. The resulting representation must be supported by local evidence while remaining consistent with the complete gait sequence; this is not a reconstruction model for recovering deleted pixels.
Self-supervised testing uses teacher prediction features; downstream tasks reuse the pretrained teacher parameters and replace the predictor with a task head. InfoNCE and EMA in the diagram belong to pretraining: deployment does not require simultaneous student and teacher execution with negative-sample comparisons. Part-aware training remains active during fine-tuning, whereas linear probing freezes the representation and learns only the classification head.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
corpus["Multisource Unlabeled Corpus"] --> clips["Two clips from one sequence<br/>Independent augmentation"]
clips --> student["Pre-Encoding Part Splitting<br/>Independent encoding and reassembly"]
clips --> teacher["Full-view teacher<br/>Encoding and projection"]
student --> contrast["Momentum Contrastive Learning<br/>Aggregation, projection, prediction"]
teacher -->|Teacher projection target| contrast
contrast -.->|EMA update| teacher
teacher --> output["Teacher representation<br/>Self-supervised testing or task adaptation"]
Key Designs¶
1. Multisource Unlabeled Corpus: expand gait variation rather than repeat one type of video
WebGait-2M combines the training splits of 12 public datasets, with Table 1 reporting 2,358,547 sequences and 231,675,844 frames. It includes laboratory datasets such as CASIA-B and OU-MVLP alongside in-the-wild sources such as Gait3D, GREW, and GaitLU-1M. Scoliosis1K, D-Gait, and RA-GAR add movement variation associated with scoliosis, depression risk, and gait attributes. Identity and disease annotations are not used in pretraining: each sequence is treated as a separate instance rather than grouping all sequences of one person into an identity class. Thus, clips from the same sequence form a positive pair; instance contrast must not be described as supervised identity contrast. The authors state that WebGait-2M is a collection name rather than a reprocessed or redistributed dataset, so access remains subject to the original datasets' licenses.
Simply concatenating sources would let large datasets dominate batches, so the paper uses softmax-normalized sampling based on log dataset size with a scale of 3.0. GaitLU-1M is additionally downweighted by a factor of 10 because of its limited viewpoint diversity. This gives smaller datasets with different clothing, scenes, and healthcare-related variation an opportunity to shape the representation. Table 7(a) also holds the data scale at approximately 1M and compares GaitLU-1M against WebGait-1M, preventing dataset size alone from explaining every improvement. FoundationGait-0.13B increases Gait3D Rank-1 from 26.6% to 42.5% in this comparison, supporting the importance of source diversity. However, pretraining includes unlabeled training samples from later evaluation datasets, so most self-supervised results are not zero-shot evaluations on entirely unseen domains.
2. Pre-Encoding Part Splitting: prevent cross-region interference before features become homogeneous
The student input is divided into \(p\) horizontal regions along height, using a consistent partition across all frames of the clip. Each region passes independently through the shared student backbone, preventing it from directly borrowing another region's features during encoding. The resulting local feature maps are concatenated along height in their original order before horizontal and temporal aggregation. The key difference from final-layer strip pooling is intervention timing: pooling divides a representation after whole-body mixing, while this approach restricts the mixing itself. It also differs from random cropping or masking because no body's region of silhouette information is deliberately discarded. The reassembled representation still covers the whole body, but its components originate from isolated encoding processes.
Training uses \(P=\{1,2,4,8\}\), dividing a batch into sub-batches assigned different values of \(p\). Each sample does not pass through all 4 granularities, nor does deployment require voting across 4 independent models. \(p=1\) retains complete context, whereas larger values force the model to exploit more localized motion and contour changes. Joint training across granularities lets the shared backbone handle whole bodies without relying exclusively on the most salient local cue. Half of the samples also receive a height-wise shifted-window operation during splitting, changing the local partition positions. The main text specifies the direction and application fraction; this note does not invent an exact displacement or boundary implementation. Inference uses the teacher's full-view path, so the principal change is a training-time information constraint rather than an additional human-parsing network.
3. Momentum Contrastive Learning: align locally encoded evidence with stable whole-gait targets
The teacher contains an encoder, projector, and predictor, with corresponding components in the student. It processes the other complete silhouette clip and produces a contrastive target after spatiotemporal aggregation and projection; the student predicts from its reassembled whole-body feature. InfoNCE brings the student prediction closer to the teacher projection from the same sequence, with other sequence instances providing negative pairs within the batch. Similarity is measured by cosine similarity, and the main text specifies the temperature as \(\tau=16\). The numerator and denominator of Equation (3) are corrupted in the text extraction and cannot be checked reliably symbol by symbol, so this note does not reconstruct it as the authors' exact equation. The verified learning relationship follows Figure 4, Algorithm 1, and the prose: student predictions align with teacher projections, not student projections directly with teacher predictions.
Back-propagation updates the student; the teacher receives no gradient from this branch and instead follows an exponential moving average of student parameters. EMA momentum increases from 0.99 to 1.0 under a cosine schedule, gradually stabilizing the target. The asymmetric predictor and cross-component alignment follow established self-supervised representation learning, while part splitting supplies the paper's gait-specific prior. Although the teacher does not directly receive split parts, parameter averaging transfers the local discriminative ability that the student is forced to learn. Self-supervised evaluation uses features from the teacher predictor; downstream adaptation replaces that predictor while retaining the pretrained encoder and projector. This output choice matters: selecting an arbitrary intermediate layer would not reproduce the reported evaluation setup.
A Worked Example¶
Consider one walking sequence during pretraining: sample two non-overlapping clips, each containing 16 frames with silhouette dimensions \(64\times44\). One augmented clip goes to the teacher to produce a complete-gait projection target. If the other clip belongs to the \(p=4\) sub-batch, it is split into 4 height regions, each with input height 16 and width 44. The student encodes these regions independently and restores their local features into a spatially complete map from top to bottom. Here, complete means restored coverage, not that the regions have exchanged information inside the backbone. After aggregation, projection, and prediction, the student representation should be closer to the same-sequence teacher target than to other sequences' targets. This example explains the input flow; it is not a separately reported experimental sample. After training, self-supervised recognition requires only the teacher path to extract test-clip features, followed by identity retrieval under the dataset protocol.
Loss & Training¶
Pretraining uses SGD with an initial learning rate of 0.05, momentum of 0.9, and weight decay of \(5\times10^{-4}\). The paper reports a batch size of 512, sampling 16 frames per clip. The 0.13B model trains for 80K iterations with a cosine learning-rate schedule and a minimum learning rate of \(1\times10^{-5}\). Other sizes train for 40K iterations, multiplying the learning rate by 0.1 at 20K and 30K. Consequently, better performance from the largest model also involves longer training; cross-size differences cannot be attributed entirely to parameter count. The 0.13B designation in Table 2(a) refers to a 132.3M-parameter backbone; including the 32.6M mapping components gives 164.9M total parameters. The largest model reportedly takes approximately 178 hours to pretrain on 16 V100 GPUs with 32GB each, indicating a substantial training requirement.
Recognition fine-tuning updates only the final two backbone blocks at learning rates of 0.001 and 0.005; the projector and task head use 0.01 and 0.1. Gait3D fine-tuning uses \(P=\{1,2\}\), while CCPG and CCGR-MINI use \(P=\{1,2,4,8\}\). Healthcare adaptation first performs linear probing of the classification head and then initializes further fine-tuning from that result. Linear probing uses \(P=\{1\}\) and healthcare fine-tuning uses \(P=\{1,2,4,8\}\); they should not be treated as equivalent training budgets. The main text delegates additional loss and adaptation details to the supplementary material; the assigned cache does not provide a reliable expansion of those details, so unverified supervised loss combinations are omitted.
Key Experimental Results¶
Main Results¶
The following selection from Table 3 on page 10 reports self-supervised results, all pretrained on WebGait-2M without labeled target-task fine-tuning. Values are percentages; Rank-1 measures correct identity at the top retrieval position, and CCPG CL denotes the full-clothing-change condition.
| Method | OU-MVLP Rank-1 | Gait3D Rank-1 | Gait3D Rank-5 | CCPG CL Rank-1 |
|---|---|---|---|---|
| GaitSSB | 44.4 | 27.4 | 43.3 | 16.3 |
| GaitSSB-0.03B | 39.2 | 33.2 | 51.2 | 16.0 |
| GaitSSB-0.13B | 38.9 | 34.6 | 53.2 | 16.3 |
| FoundationGait-0.03B | 57.0 | 41.1 | 59.0 | 26.8 |
| FoundationGait-0.13B | 64.5 | 48.0 | 66.5 | 27.6 |
Against GaitSSB at the same 0.13B backbone size, FoundationGait improves Gait3D Rank-1 by 13.4 percentage points and OU-MVLP by 25.6 percentage points. Against the small GaitSSB, the Gait3D gain is 20.6 percentage points; these comparisons must not be attributed to the same baseline. Labeled fine-tuning results are separately reported in Table 4(c) on page 11: FoundationGait-0.13B reaches 79.3% Rank-1 and 74.0% mAP on Gait3D. These cannot be ranked as equivalent conditions with the non-fine-tuned results above; mAP measures full retrieval-ranking quality rather than being another name for Rank-1.
Ablation Study¶
The following results come from Table 6(a) on page 14 and compare student processing strategies with the 0.13B pretraining model fixed; all numeric results are recognition percentages. Only the clearly identified CASIA-B conditions and Gait3D Rank-1 are retained; NM, BG, and CL denote normal walking, carrying a bag, and changing clothes.
| Student processing | P | CASIA-B NM | CASIA-B BG | CASIA-B CL | Gait3D Rank-1 |
|---|---|---|---|---|---|
| Crop | Not applicable | 70.8 | 59.3 | 27.9 | 17.2 |
| Mask | Not applicable | 87.6 | 68.1 | 36.2 | 38.1 |
| Part-aware | [1] | 86.6 | 67.0 | 28.7 | 39.6 |
| Part-aware | [1, 4] | 92.2 | 73.8 | 37.7 | 45.3 |
| Part-aware | [1, 2, 4, 8] | 94.0 | 75.4 | 40.2 | 48.0 |
Moving from unsplit [1] to [1, 2, 4, 8] improves Gait3D Rank-1 by 8.4 percentage points and CASIA-B CL by 11.5 percentage points. Crop and Mask both underperform the full part-aware approach, supporting preservation of all regions with restricted encoding interactions over deletion of local information for this objective. The final column of Table 6(a) is labeled mAP, but its [1, 2, 4, 8] value of 66.5 matches Gait3D Rank-5 in Table 3; the meaning of 60.3 following 45.3 also requires verification. The text cache alone cannot establish whether this is an original table-layout issue or an extraction issue, so that column is excluded from quantitative conclusions here.
To examine whether cross-task adaptation benefits similarly, the following selection reproduces Scoliosis1K results from Table 5(a) on page 13, in percentages.
| Mode | Method | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|
| Linear probe | FoundationGait-0.03B | 71.1 | 59.7 | 75.3 | 56.7 |
| Linear probe | FoundationGait-0.13B | 73.5 | 61.0 | 68.5 | 56.9 |
| Fine-tuning | FoundationGait-0.03B | 97.3 | 95.4 | 88.9 | 91.7 |
| Fine-tuning | FoundationGait-0.13B | 96.0 | 80.1 | 97.5 | 85.1 |
The larger fine-tuned model has higher recall but lower precision, accuracy, and F1 than 0.03B on this task, so downstream improvement is not uniformly monotonic with size. The introduction claims 97.0% accuracy for scoliosis screening, whereas Table 5(a) lists 97.3% and 96.0%; this note records the table configurations separately rather than silently reconciling the discrepancy.
Key Findings¶
- Multisource data and part constraints must work together: even with more data, scaling GaitSSB still degrades performance on OU-MVLP.
- The unseen-domain experiment is a closer test of cross-domain transfer: Table 6(c) on page 14 reports 50.4% for the 0.13B model on DroneGait high-pitch NM, versus 36.0% for GaitSSB.
- Cross-modality results require adaptation: the human-parsing inputs in Table 6(d) involve fine-tuning, not direct transfer to another modality without training.
Highlights & Insights¶
- Where the part prior acts matters more than merely having local pooling. Moving the intervention before encoding explains why existing part pooling may still fail to support scaling.
- A full-view teacher and a split-input student create a clear training asymmetry. Local clips must provide evidence without locking inference into a multi-branch local network.
- One unlabeled backbone supports identity retrieval and healthcare attribute learning, demonstrating reusable domain representations. Its success boundaries should be judged by each task's adaptation and metrics rather than the foundation-model label.
Limitations & Future Work¶
- The authors have not demonstrated 1B scaling: the largest backbone is 0.13B. ViT architectures and RGB modeling are also future directions, not implemented capabilities.
- The largest model has a longer pretraining schedule, and the self-supervised comparison relies mainly on reproducible GaitSSB. Compute-matched comparisons would better separate architecture, size, and training-budget effects.
- Healthcare tasks show substantial size and decision-threshold trade-offs; high recall does not guarantee more reliable clinical screening. This note's interpretation is that external validation, calibration, and population-stratified assessment remain necessary; the paper does not establish a replacement for clinical diagnosis.
- Silhouettes discard RGB texture but can still carry identity and health information, so they do not automatically guarantee privacy. Applications require review of consent, data governance, and sensitive-attribute inference risks.
- Equation (3), Table 6(a)'s column meaning, and the introduction's accuracy claim have the verification limitations described above; they are not used to reconstruct unconfirmed equations or derive new experimental claims.
Related Work & Insights¶
- vs GaitSSB: both learn from unlabeled gait data; FoundationGait changes scaling behavior through part-aware training and a multisource corpus rather than introducing gait self-supervision for the first time.
- vs GaitPart / DeepGaitV2: the method inherits local-body modeling ideas and uses a DeepGaitV2 backbone; its key addition is pre-encoding splitting during training, not a wholly new visual backbone.
- vs BigGait / BiggerGait: these methods leverage RGB and existing large vision models, while FoundationGait pretrains directly in the silhouette domain; BiggerGait's CCPG mean of 93.0% in Table 4(a) still exceeds its 87.2%.
- Possible extension (note interpretation): measuring inter-part feature similarity and fusion gains while changing split granularity under equal compute could further test whether preserved diversity directly causes the scaling gains.
Rating¶
- Novelty: 4/5. Moving part isolation before encoding provides a concrete, testable response to gait-model scaling failure.
- Experimental Thoroughness: 4/5. Recognition, healthcare, unseen domains, and modality adaptation are covered, but compute-matched comparisons and reporting consistency remain incomplete.
- Writing Quality: 3/5. The mechanism and algorithm are clear, while some numbers, column labels, and extracted equations complicate verification.
- Value: 4/5. A reusable gait-domain pretraining approach, with further validation required for healthcare and large-scale deployment.