The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning

Conference: ICML 2025
arXiv: 2406.04328
Code: -
Area: Brain-Computer Interfaces / Self-Supervised Learning / Speech Decoding
Keywords: MEG, Self-Supervised Learning, Pretext Task, Speech Decoding, Cross-Subject Generalization, Brain Signals, Scaling

TL;DR

Developing neuroscience-inspired self-supervised pretext tasks and a heterogeneous brain signal processing architecture, this work scales MEG speech decoding to approximately 400 hours and 900 subjects, outperforming the SOTA by 15-27%. It matches surgical-grade decoding performance using non-invasive data for the first time, demonstrating robust generalization across datasets, subjects, and tasks.

Background & Motivation

Background

Background: Richard Sutton's Bitter Lesson states that general methods leveraging large-scale computation will ultimately outperform model-based approaches. However, this lesson has not yet been fully embraced in the domain of brain signals:

Limitations of Prior Work

Limitations of Prior Work: Existing speech decoding models are rarely trained across subjects or on combined datasets.

Key Challenge

Key Challenge: Individual differences (anatomical structures, variations in scanning hardware) make data aggregation highly challenging.

Mechanism

Mechanism: Data scale is constrained by the collectable volume from individual subjects.

MEG signals are rich (with better spatial resolution than EEG and higher sampling rates than fMRI) but relatively scarce. This work leverages self-supervised learning to overcome label scarcity, designing domain-specific pretext tasks to achieve large-scale pre-training across heterogeneous data.

Method

Architecture Design

Two Stages: Pretext task pre-training \(\rightarrow\) frozen backbone + linear probe fine-tuning.

Dataset-Conditional Linear Layers: Project signals with varying sensor numbers \(S\) to a shared dimension \(d_{shared}\).

Cortex Encoder: A wave-to-wave convolutional encoder based on SEANet (neural audio encoder), which takes \(\mathbb{R}^{S \times t}\) as input and outputs \(\tau\) embeddings of dimension \(d_{backbone}\).

Subject Conditioning: Subject-specific information is injected at the encoder bottleneck using FiLM (Feature-wise Linear Modulation), similar to speaker conditioning in speech recognition.

Three Pretext Tasks

1. Band Prediction

A bandstop filter is applied to the signal to remove a frequency band \(\omega\), and the network predicts which band was removed:

\[\mathcal{L}_{band} = \sum_{x \in B} \mathcal{L}_{CE}(f_{band}(g(x^{\omega'})), \omega)\]

Frequency bands include: \(\delta\) (0.1-4Hz), \(\theta\) (4-8Hz), \(\alpha\) (8-12Hz), \(\beta\) (12-30Hz), \(\gamma\) (30-70Hz), low-high \(\gamma\) (70-100Hz), and high-high \(\gamma\) (100-150Hz).

2. Phase Shift Prediction

A discrete phase shift \(\phi\) is applied to a random proportion of sensors \(\rho \in [0, 0.5]\), and the network predicts the shift amount:

\[\mathcal{L}_{phase} = \sum_{x \in B} \mathcal{L}_{CE}(f_{phase}(g(x^\phi)), \phi)\]

3. Amplitude Scale Prediction

A discrete scale factor \(A \in [-2, 2]\) (discretized into 16 values) is applied to random sensors, and the network predicts the scale factor:

\[\mathcal{L}_{amplitude} = \sum_{x \in B} \mathcal{L}_{CE}(f_{amplitude}(g(x^A)), A)\]

Combined Loss:

\[\mathcal{L}_{SSL} = w_1 \mathcal{L}_{band} + w_2 \mathcal{L}_{phase} + w_3 \mathcal{L}_{amplitude}\]

Design Principles

The three tasks capture frequency-domain, time-domain phase coupling, and spatial amplitude variation information, respectively. They are inherently independent of the number of sensors, allowing seamless handling of heterogeneous data.

Key Experimental Results

Speech Detection (ROC AUC)

Main Results

Method	ROC AUC
Random	.500
Linear	.539
BIOT (SOTA SSL)	.615
BrainBERT (SOTA SSL)	.556
EEGPT (SOTA SSL)	.602
Ours (best)	.705
BrainBERT (Surgical Data)	.71

Scaling Results

• Performance scales log-linearly (or log-log in some tasks) with the amount of unlabeled data without showing saturation.
• Sustained improvements are observed from the smallest dataset up to 160 hours of Cam-CAN.
• Increasing the number of subjects (rather than the data volume per subject) also consistently yields improvements.

Dataset Aggregation

Ablation Study

Pre-training Data	Hours	ROC AUC
CamCAN	159	.630
MOUS	160	.614
CamCAN + MOUS	319	.638

Generalization to New Subjects

This study demonstrates generalization to new subjects (out-of-distribution subjects) in MEG speech decoding for the first time, exhibiting a positive log-linear trend as pre-training data scales.

Key Findings

1. Combining the three pretext tasks outperforms any single task.
2. Pre-training data contains no language-related data yet still improves speech decoding performance.
3. Data recorded from different scanning hardware can be successfully aggregated.
4. Non-invasive MEG matches surgical-grade SSL performance for the first time (.705 vs .71).

Highlights & Insights

1. The Bitter Lesson in Brain Sciences: Clearly demonstrating that "more data + general methods" remains highly effective in the domain of brain signals.
2. Neuroscience-Inspired Pretext Designs: Each pretext task maps directly to known biological properties of brain functional frequency bands.
3. Unified Solution for Cross-Subject, Cross-Dataset, and Cross-Task Generalization: Resolving the long-standing issue of fragmentation in brain signal analysis.
4. Matching Surgical-Grade Performance: Reaching the performance level of invasive methods using non-invasive MEG, which holds significant promise for clinical BCI deployment.

Limitations & Future Work

• Validation is limited to speech detection and speech production classification, without scaling to full speech transcription.
• The set of pretext tasks is not exhaustive; more effective transformations may still exist.
• Spatial geometric relationships of sensors are not explicitly utilized.
• The focus remains on auditory speech, without exploring imagined or attempted speech.
• Computational resources constrain the aggregation of even larger-scale datasets.

Related Work & Insights

• Speech Decoding: Moses et al. (Surgical BCI), Tang et al. (Non-invasive fMRI), Défossez et al. (MEG)
• Brain Signal SSL: BIOT, BrainBERT, EEGPT, MBrain
• Pretext Tasks: Image rotation prediction, jigsaw puzzle solving, colorization

Rating

⭐⭐⭐⭐⭐ (5/5)

Broad in vision, this work systematically introduces the Bitter Lesson philosophy into the field of brain signals. The methodology features both neuroscience insights and engineering practicality. The scaling evidence is compelling, and several "first-of-its-kind" outcomes (cross-dataset aggregation, new subject generalization, and matching surgical-grade performance) mark milestone achievements.

Related Papers

• [ICCV 2025] An OpenMind for 3D Medical Vision Self-supervised Learning
• [NeurIPS 2025] Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation
• [CVPR 2026] SPECTRE: Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers
• [NeurIPS 2025] Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
• [NeurIPS 2025] Self-Supervised Learning via Flow-Guided Neural Operator on Time-Series Data