Skip to content

🔄 Self-Supervised Learning

🧠 NeurIPS2026 · 64 paper notes

📌 Same area in other venues: 🎞️ ECCV2026 (24) · 📷 CVPR2026 (98) · 🔬 ICLR2026 (82) · 💬 ACL2026 (1) · 🧪 ICML2026 (28) · 🤖 AAAI2026 (16)

🔥 Top topics: Self-Supervised Learning ×11 · Diffusion Models ×2 · Adversarial Robustness ×2 · Layout & Composition ×2 · Biomolecules ×2

AmbientFM: A Foundation Model for Ambient Sensing

AmbientFM pre-trains a unified foundation backbone on 9.2 million unlabeled in-the-wild WiFi CSI samples collected over 439 days from 20 commercial IoT device types, combining adaptive frequency aggregation, wireless-tailored contrastive learning, masked reconstruction, and physics-informed autocorrelation prediction to transfer to 9 downstream sensing tasks with >0.90 AUROC under lightweight adaptation.

An Open-Source Training Dataset for Foundation Models for Black-box Optimization

Presents BBO-Pile, the first open-source large-scale pre-training dataset for black-box optimization foundation models (557K trajectories across 3,095 tasks and ~2.5B tokens), and empirically establishes compute and token scaling laws using a family of causal Transformers from 2M to 80M parameters.

Approximation in Contrastive Representation Learning

This paper develops an approximation theory for contrastive representation learning (CRL) by interpreting two-tower inner-product scoring as spectral truncation of a compact pointwise mutual information (PMI) operator, establishing explicit neural network approximation rates and proving fast calibration to downstream retrieval under a density-ratio Tsybakov noise condition.

Baba in Wonderland: Online Self-Supervised Dynamics Discovery for Executable World Models

For symbolic environments where rule vocabularies are prior-misaligned with pretrained semantics, this paper introduces Alice, a closed-loop system that transforms preservation conflicts from rejected program updates into hypothesis classes that simultaneously supply compact counterexamples for code repair and drive class-aware frontier exploration.

Benchmarking Attention for Tabular Foundation Models

Focusing on the unique 2D alternating row and column attention patterns of tabular foundation models, this paper systematically benchmarks 7 attention backends across three GPU generations, quantifies the memory contiguity overhead of strided tensor transpositions, and proposes a decoupled mixed-backend dispatch strategy that yields substantial end-to-end inference speedups.

Beyond IID: How General Are Tabular Foundation Models, Really?

This paper introduces BeyondArena, a unified holistic benchmark comprising 142 curated real-world datasets alongside the DataFoundry framework, systematically probing the true generalization limits of tabular foundation models across IID and non-IID regimes and showing that tuned traditional models still dominate challenging scenarios.

Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction

Addressing the structural misalignment and predictive unidentifiability caused by enforcing Euclidean isotropic Gaussian marginals in JEPAs, this paper introduces HamJEPA, shifting structural inductive biases from single-view marginals to cross-view Hamiltonian phase-space coupling via symplectic leapfrog integrators with non-isotropic spectral anti-collapse bounds.

Beyond Missing Rates: Rethinking Incomplete Multi-View Clustering with Protocol Divergence

The paper exposes evaluation-protocol divergence in incomplete multi-view clustering through effective missing rate and complete-sample proportion, and introduces CRAFT, which fuses only each sample's observed views: it wins 12 of 13 information-matched incomplete-training conditions, while a separate evaluation tests cross-protocol deployment of checkpoints trained with all views available.

Beyond ViT Tokens: Masked-Diffusion Pretrained Convolutional Pathology Foundation Model for Cell-Level Dense Prediction

Addressing the disruption of spatial continuity and local morphological detail caused by patch tokenization in ViT-based pathology models, this paper proposes CMD, a pure convolutional foundation model using pixel-space masked-diffusion pretraining and frozen pathology semantic guidance that outperforms ViT baselines and end-to-end segmentors under frozen-backbone settings.

Calibrating Scientific Foundation Models with Inference-Time Stochastic Attention

Addressing the failure of deterministic scientific foundation models to provide calibrated uncertainty, this paper introduces Stochastic Attention—an inference-time mechanism that replaces softmax expectations with normalized multinomial sample averages, enabling well-calibrated, sharp prediction intervals on frozen backbones via a 1D post-hoc tuning problem.

Can 4D Foundation Models Remember?

This paper introduces PersistBench, the first reference-based benchmark that leverages omniscient 360° videos to systematically evaluate visual memory in 4D foundation models across object permanence, motion continuity, and appearance preservation, revealing that existing models suffer catastrophic performance degradation once objects leave the field of view ("seeing is not remembering").

CanViT: Toward Active-Vision Foundation Models

CanViT introduces the first task- and policy-agnostic active-vision foundation model by decoupling instantaneous glimpse processing from persistent scene-wide memory (the canvas) via asymmetric Canvas Attention and scene-relative RoPE, achieving state-of-the-art results on ADE20K segmentation (45.9% mIoU) and ImageNet-1k (84.5% top-1) with 20x fewer FLOPs.

CASL-VAE: Learning Structured Latent Variables from Unpaired Data for Semi-Supervised Representation

Addressing the challenge of entangled representations and heterogeneous target variation in unpaired reference-target populations, CASL-VAE factorizes the latent space into continuous common factors and hierarchical salient factors ("discrete subtypes + continuous within-subtype variation"), enabling unsupervised target subtype discovery and semantic-preserving counterfactual paired-sample generation.

Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production

Catch Your Breath (CYB) frames per-token generation as a sequential-decision process with temporal discounting, enabling language models to autonomously emit <don't know> abstention tokens to request additional <pause> computation steps on demand with zero extra compute or memory overhead.

Contrast Encodes Inductive Bias: Separating Slow Noise from Dynamics in Predictive Representation Learning

This paper demonstrates that contrastive predictive objectives (such as JEPA and DySIB) with across-trajectory negatives inevitably collapse under temporally persistent noise by encoding trajectory-specific nuisances, and proves that replacing them with within-trajectory negatives (WTN) eliminates the predictive shortcut, recovering true physical dynamical states.

Data Density Scaling Laws for Image Self-Distillation

Systematically investigates the trade-off between unique sample size and repetition density in vision self-distillation under fixed compute budgets, uncovering a counter-intuitive regime where repeatedly training on smaller subsets outperforms the full dataset in redundant domains, and proposing a novel density-interaction scaling law.

Data Density Scaling Laws for Image Self-Distillation

Challenging the standard scaling assumption that increasing unique samples always improves representation quality, this paper systematically explores the trade-off between unique sample count and repetition density under fixed compute budgets for image self-distillation, demonstrating that repeatedly training on a small subset can significantly outperform full-pool pretraining in highly redundant domains, and proposing a density-interaction scaling law.

Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Diffusion Meta-Prompting (DMP) trains a conditional 1D diffusion model directly over a repository of pre-learned prompts to synthesize task-specific prompts from natural language descriptions without touching task data or loss functions, coupled with a training-anchored test-time steering mechanism for robust downstream generalization.

Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities

The paper reveals that independently trained models converge not on absolute sample positions in latent space, but on inter-sample directional displacement vectors; using trivial orthogonal Procrustes alignment and consensus decomposition, it resolves the debate surrounding the Platonic Representation Hypothesis and introduces Shadow Casting to transfer model capabilities without fine-tuning.

Diversity Curves for Graph Representation Learning

The paper introduces Diversity Curves, a size-aware unsupervised graph representation framework that tracks metric space spread across multiscale edge contraction coarsening, providing provably enhanced expressivity and superior geometry-aware clustering across varying graph sizes.

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

By linearizing the coupled gradient flow of Joint-Embedding Predictive Architectures (JEPAs) around the trivial fixed point, this paper formulates representation learning as a spectral competition between driving and decay forces, establishing the escape threshold \(\mu = \gamma / \sigma > 1\), bounding representation rank via supercritical modes, and introducing the identity-biased ResidualPred predictor to eliminate early collapse.

Effective Biological Representation Learning by Masking Gene Expression

TxFM is an asymmetric masked autoencoder for transcriptomics that pairs a bottleneck CLS token and lightweight MLP decoder with a novel rectified tanh activation and Poisson reconstruction loss, outperforming 16 foundation models trained on 100x larger corpora using only 1.4M curated samples.

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching

SFM approximately interprets linear gradient matching on frozen pretrained vision backbones as matching relative vectors between class centers, then supervises synthetic images with global statistics computed once, improving classification with one image per class; its approximately 10-fold memory reduction and fourfold speedup compare single-augmentation SFM against ten-augmentation LGM, rather than equal-budget configurations.

Environment-Robust Representation Learning with Empirical Bayes

Addressing multi-environment prediction where latent variables shift across environments while generating mechanisms remain invariant, this paper proposes Empirical-Bayes Environment-Robust (EBER) learning, which models observation-adaptive environment weights to aggregate environment-specific priors and posteriors, naturally inducing an environment-balancing variational objective without ad-hoc penalties.

Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons

To overcome the limitation of ECG foundation models being restricted to 10-second snapshots, this paper introduces a parameter-efficient framework that extends frozen backbones to long and variable-length inputs via hierarchical positional encodings and teacher-guided dual semantic regularizations.

Extracting Governing Equations from Latent Dynamics via Multi-View Contrastive Learning

This paper introduces DYSCO, a multi-view temporal contrastive learning framework that leverages independent noisy observations of the same underlying dynamical process to denoise observations, reconstruct latent trajectories, and identify governing vector fields up to an affine gauge, enabling downstream symbolic equation discovery.

FMMI: Flow Matching Mutual Information Estimation

FMMI reframes mutual information estimation by learning a continuous normalizing flow that transports the product of marginal distributions directly into the joint distribution via simulation-free flow matching, estimating mutual information and high-order O-information across high dimensions and large MI values with 20× fewer training steps than diffusion baselines.

Handwritten Text Recognition Lives in the High-Pixel Variance Subspace

Using pixel-PCA projections and frozen-encoder probes, this paper studies the distribution of handwritten text recognition signals: high-variance pixel directions support transcription better on six Latin-script benchmarks, while real-data-pretrained MAE achieves 5.5% mean probe CER and 4.5% mean CER after adding a language decoder and full fine-tuning.

How Data Augmentation Shapes Neural Representations

By embedding neural network hidden representations into a Riemannian shape metric space invariant to scaling, translation, rotation, and reflection, this work reveals that data augmentation strength steers representations along smooth, structured trajectories, where geodesic angles between trajectory tangents reliably predict gains from model ensembling.

I Have a Stream: Making Self-Supervised Learning Work on Continuous Video

StreamMAE preserves MAE's pixel reconstruction objective while adapting temporally ordered video training through stronger regularization, example-level DataDrop, and motion-biased two-stage cropping, improving classification, segmentation, and depth transfer on WT++12h and approaching same-data offline i.i.d. MAE on most metrics.

Inertia-1: An Open Exploration of Wearable Motion Foundation Models

Inertia-1 presents an open benchmark and systematic exploration of wearable motion foundation models across 18.2M hours of accelerometry, 10 self-supervised objectives, 15 datasets, and over 1,000 controlled models, disentangling how sensing pipelines, learning algorithms, and scale jointly dictate cross-task transferability.

Laplacian Heads Improve Transformers by Smoothing Token Representations

Replacing a subset of standard softmax attention matrices with parameter-free graph Laplacian operators enables direct control over within-sequence representation variance via graph heat diffusion, consistently boosting performance and representational geometry across supervised learning, language modeling, and self-supervised learning.

Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence

Under a unified readout protocol, this study evaluates four frozen video encoders, finds that V-JEPA variants better preserve usable action semantics under degraded inputs and support more reliable planning in limited simulated manipulation experiments, but neither evaluates generation quality nor causally isolates the pretraining objective.

Learnable Spectral Activations

This paper introduces Learnable Spectral Activations (LSA), which replace fixed neuron-level nonlinearities in implicit neural representations with a residual truncated Fourier series to decouple linear feature selection from internal spectral shaping, concentrating significantly more target-signal energy into the leading empirical NTK eigenmodes.

Learning Compositional Latent Structure with Vector Networks

Vector Networks (VN) deconstruct dense weight matrices into reusable libraries of rank-1 weight atoms, dynamically inferring instance-specific sparse combinations via layer-local energy minimization and updating only active atoms through residual-scaled Atomic-Hebb plasticity to make compositional generalization a structural property.

Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

Addressing the prohibitive cost of test-time multimodal sensing when downstream objectives are unknown a priori, this paper introduces echo-k, a task-agnostic reinforcement learning policy that uses latent representations of frozen foundation models as self-supervised proxy targets for long-horizon adaptive feature acquisition.

MONROE: A Molecular Foundation Model for In-Context Probabilistic Inference

MONROE couples stereochemistry-aware graph pretraining across 81 million molecules with TabPFN in-context probabilistic inference, eliminating downstream fine-tuning and establishing new state-of-the-art accuracy on Polaris and MoleculeACE activity cliffs.

MT-CC: Multi-Group Temperature Scaling for Asymmetric Calibration Behavior in Class-Incremental Learning

Addressing the discrepancy between accuracy degradation and confidence changes caused by catastrophic forgetting in class-incremental learning, MT-CC introduces a lightweight grouping network conditioned on task-aware and logit-margin statistics to route samples to group-specific temperatures, accompanied by a groupwise consistency loss that substantially reduces calibration error without altering classification accuracy.

NOCE-Net: Representation Learning for Battery Operational Context via Nested Sequence Modelling

NOCE-Net introduces a bi-level nested sequence model and the re-BatteryData standardization pipeline that segments operational telemetry into full equivalent cycles based on absolute charge throughput, combining multi-patch Transformer encoders with a GRU backbone to predict forward voltage responses from control inputs while learning degradation-indicating representations without capacity labels.

OdysSim: Building Foundation Models for Human Behavior Simulation

The paper presents OdysSim, the largest open systematic investigation of behavioral foundation models, uniting 62 datasets and 23 benchmark tasks under the 5-axis Soul framework, and introducing an end-to-end training recipe (midtraining + task-specific RL + expert distillation) that enables an 8B model to match frontier LLMs on Soul-Index while achieving state-of-the-art reaction alignment with real users in zero-shot user simulation.

One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models

Addressing the irregular sampling and tightly coupled continuous measurements in structured electronic health records, ORA proposes a marked time-to-event pretraining objective that jointly models event timing and numerical values via a discretized composite code-specific likelihood, markedly outperforming next-token prediction across classification, survival, and regression tasks.

OpenMHC: Accelerating the Science of Wearable Foundation Models

Stanford and Imperial College London release OpenMHC, the largest open-access wearable health dataset and benchmark spanning >60 million hours across 19 sensor channels from 11,894 participants, alongside open-source implementations and systematic evaluations of wearable foundation models like Google LSM-2 and Apple WBM.

PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment

Addressing the lack of a rigorous optimization objective and the vulnerability to representation collapse in self-distillation dynamics for non-contrastive self-supervised learning, this paper characterizes the stable fixed points of JEPA dynamics from a nonlinear canonical correlation analysis (CCA) perspective and proposes PEIRA—an unconstrained objective based on trace alignment of optimal regularized linear regressors—theoretically eliminating degenerate solutions while providing an efficient stochastic composite training algorithm.

PRISM: Prior Relational Information for Self-supervised Modeling to Enhance Solubility OOD Generalization

PRISM converts expert-defined physicochemical and substructural priors into relational self-supervised teachers via soft conditional neighborhood distribution matching, training decoupled encoders fused downstream to dramatically enhance solubility OOD generalization across unseen scaffolds, solutes, pure solvents, and mixed-solvent pairs.

Pro-t-VAE: Closed-Form Student's t Mixtures for Robust Prototype Discovery

Addressing the sensitivity of Gaussian-prior clustering to heavy-tailed noise and the computational burden of diffusion-based prototype learning, Pro-t-VAE derives a fully closed-form, unbiased ELBO via Gaussian-Gamma scale-mixture conjugacy, achieving adaptive outlier downweighting and zero prototype collapse across vision benchmarks and 3D neuroimaging scans.

RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain

Addressing the challenge of balancing population-shared and subject-specific representations in speech-elicited fMRI responses, this paper presents RABBiT, a compact audio-to-fMRI foundation encoder that combines brain-tuning, a Temporal Brain Transformer, and Shared-Idiosyncratic Decomposition (SID) to achieve superior zero-shot population prediction and few-shot individual adaptation with only 115K tunable parameters from 5–10 minutes of calibration data.

RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers

Proposes Register Attention Transformers (RATS), which replaces standard self-attention with a per-head partitioned register bottleneck running a three-step compress–communicate–broadcast routing, enabling register tokens to spontaneously emerge as semantic object parts under standard self-supervised DINO objectives without auxiliary losses or part annotations.

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

Addressing the exponential combinatorial bottleneck and multi-encoder structural complexity of multimodal disentangled representation learning beyond two modalities, RePercENT introduces a plug-and-play self-supervised framework based on pairwise decomposition, Semantic Encoding, and Group Slot Attention, achieving linear-complexity disentanglement with a single encoder per modality alongside formal Minimum Necessary Information (MNI) optimality guarantees.

Revisiting Cross-View Completion: Self-Supervised Pre-Training via Reconstruction Error Comparison

Gekko introduces the relative reconstruction error improvement between cross-view completion and masked autoencoding as a self-supervised proxy for co-visibility, training a single shared network across three passes with an error-downweighted loss to unlock dense binocular signals from raw video without any 3D annotations and yielding up to a 6x gain on relative pose estimation over CroCo.

SapiensID 2.0: Aligning Human Recognition Foundation Models with Human Perception

SapiensID 2.0 remedies the "semantic blindness" and lack of temporal dynamics in biometric foundation models by coupling MLLM-derived bipartite semantic distillation (Invariant Trait Alignment and Transient Noise Disentanglement) with a Kinematic Semantic Attention Head, unifying face verification, person re-identification, and gait recognition under a single 63M ViT backbone.

ScreenShot: A Foundation Model for Few-Shot Combination Drug Screening

ScreenShot is a hierarchical transformer foundation model pretrained on 40 pan-cancer drug screening datasets that predicts combination drug response directly from functional drug-dose-viability measurements via few-shot in-context learning without molecular profiling or fine-tuning, and drives embedding-based active learning that achieves equal hit recall with only one-third of the experimental screening budget.

Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloning

Keyframe Mnemonics provides an annotation-free self-supervised memory framework that leverages neural network simplicity bias and masked sampling to extract decision-critical observations, granting behavior cloning policies robust horizon-invariant generalization across horizons orders of magnitude beyond training.

SIEVE: Overcoming Topological Obstruction in Equivariant Self-Supervised Learning

Addressing the topological collapse induced by isotropic masking in SE(3)-equivariant masked autoencoding, this paper introduces SIEVE, an equivariant self-supervised framework that employs spherical projection masking to preserve pose while distilling intrinsic geometry into node-level invariant scalar embeddings.

SpatialBench: Is Your Spatial Foundation Model an All-Round Player

Addressing fragmented evaluation protocols and severe embodied out-of-distribution blind spots in spatial foundation models, this paper introduces SpatialBench—the first benchmark spanning 6 paradigms and 41 model variants across 19 datasets under deterministic multi-density sampling—reveals fundamental accuracy-scalability trade-offs, and presents the 5.5M-frame DA-Next-5M dataset alongside the DA-Next baseline.

SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings

Addressing the issue where Euclidean isotropic Gaussian priors in self-supervised learning introduce local density-skewed biases into nearest-neighbor estimation, this paper proves from a minimax prediction risk perspective that a uniform distribution on the hypersphere is the uniquely optimal representation geometry across linear regression, k-nearest neighbors, and exponential-kernel ridge regression, and introduces SPHERE-JEPA with Sketched Uniform Spherical Regularization (SUSReg) to achieve substantial gains on ImageNet-1K linear probing and nonparametric retrieval.

SS3D: End2End Self-Supervised 3D from Web Videos

Addressing the collapse of SfM self-supervision on unconstrained web videos caused by weak multi-view geometric signals and corpus heterogeneity, SS3D introduces a Multi-View Signal Proxy (MVSP) measuring epipolar vs. homography residual ratios for curriculum sampling, together with unsupervised sub-domain expert clustering and student distillation, delivering the first end-to-end self-supervised 3D foundation model pretrained on web-scale video (YouTube-8M) to jointly predict depth, camera motion, and intrinsics.

Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding

SGMA combines content-adaptive quadtree tokenization with cross-scale structure-conditioned masking for MAE pretraining, preserving scientific microstructure within a fixed sequence budget for standard ViTs; SGMA-SAM reaches 95.68% Dice on SpringXCT and SGMA-SAM 2 reaches 83.21% on PAIP, while the reported maximum 24.8× inference speedup comes from a separate, shorter-sequence configuration.

Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning

Tadpole introduces a self-supervised 3D PDE foundation model pre-trained as an autoencoder via an online GPU pseudo-spectral simulation pipeline that completely bypasses storage and I/O bottlenecks; combined with Tadpole-DFT (re-introduced skip connections, latent transformation, and LoRA), it outperforms massive dynamics-pre-trained foundation models in temporal rollout with a fraction of trainable parameters.

TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders

TRL-Bench establishes a standardized, cross-paradigm representation-level evaluation benchmark for tabular encoders by decoupling downstream task adaptation and probing exported row, column, and table embeddings across 16 tasks and a 47,772-table compositional data lake.

Understanding the Surprising Generalization Properties of Tabular Foundation Models

This paper demonstrates that tabular foundation models (TFMs) can achieve surprisingly robust out-of-domain in-context generalization from self-supervised pre-training on just a single real table, showing that feature-induced task diversity rather than instance count governs transferability and establishing a task-centric, retrieval-and-aggregation theoretical framework.

What DNA Foundation Models Learn Beyond Sequence Composition

This paper evaluates frozen representations from five DNA foundation models against normalized \(k\)-mer baselines across 57 genomic classification benchmarks, showing that simple compositional features remain highly competitive with most models and only NTv3 consistently provides incremental predictive value and sequence-order sensitivity.

Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have

FINO introduces a label-free representation adaptation framework that integrates high-cardinality discrete and continuous metadata into self-supervised learning, consistently outperforming full fine-tuning and specialized domain SOTA using only lightweight frozen probes.

Why DiT Models Underperform as Representation Learners without Long Skip Connections

Through controlled from-scratch training and mutual information probing, this paper reveals that standard isotropic DiTs underperform as representation learners due to the lack of long skip connections, proposing and validating the diffusion noise bypassing hypothesis which shows that long skip connections shortcut trajectory noise and prevent intermediate semantic pollution.

Your Self-Supervised Projection Head Captures Object Co-Occurrence Statistics

Reveals that multi-layer projection heads in visual self-supervised learning implicitly capture natural object co-occurrence statistics while the visual backbone specializes in discriminating object categories, and demonstrating that encoding object co-occurrences significantly accelerates category representation learning.