title: >- [Paper Note] MuSViT: A Foundation Vision Model for Sheet Music Representation description: >- [ECCV 2026][Audio & Speech][Sheet Music Representation] The first foundation vision model for sheet music pre-trained via Masked Autoencoders on 9.7M IMSLP pages with a two-stage curriculum, outperforming general-purpose vision backbones and task-specific SOTA across four downstream benchmarks. tags: - ECCV 2026 - Audio & Speech - Sheet Music Representation - Foundation Vision Model - Masked Autoencoder - Optical Music Recognition date: 2026-09-19 content_hash: 472083362469af38
MuSViT: A Foundation Vision Model for Sheet Music Representation¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://grfia.dlsi.ua.es/musvit
Area: Audio & Speech
Keywords: Sheet Music Representation / Foundation Vision Model / Masked Autoencoder / Optical Music Recognition / Curriculum Learning
TL;DR¶
Addressing the failure of general-purpose vision models to capture structured musical notation, MuSViT introduces the first foundation vision transformer for sheet music, pre-trained on 9.7M IMSLP pages via a two-stage MAE curriculum and decisively outperforming general vision backbones and task-specific SOTA across four downstream tasks.
Background & Motivation¶
Music scores constitute a priceless component of human culture, with centuries of musical heritage preserved in archives and libraries. Thanks to large-scale digitization initiatives, millions of score pages are now digitally accessible through online portals such as the International Music Score Library Project (IMSLP). However, the vast majority of this treasure exists solely as raw scanned bitmaps. Due to the prohibitive cost of manual transcription, these scores remain unindexed and unsearchable, resulting in a stark bottleneck where vast cultural archives remain practically inaccessible for automated computational analysis and retrieval.
Traditional Optical Music Recognition (OMR) systems have historically relied on isolated end-to-end architectures or multi-stage pipelines trained on limited annotated collections. Because of narrow training distributions, these specialized systems are notoriously brittle, suffering severe performance drops when encountering unseen historical engraving conventions, complex polyphonic typesetting, or visual scanning artifacts. Concurrently, while general-purpose self-supervised vision models (e.g., DINOv3) and vision-language models (e.g., PaliGemma 2, Qwen3-VL, Kosmos-2.5) possess enormous parameter scales, their representations are optimized for natural scene semantics or continuous text. They lack the specialized visual inductive biases needed to decode the discrete, fine-grained, two-dimensional spatial grammar linking pitch positions and rhythmic duration on a musical staff.
Core idea: build MuSViT, the first foundation vision model dedicated to sheet music representation, utilizing 16×16 fine-grained patches and a high-ratio (up to 70%) Masked Autoencoder (MAE) framework governed by a two-stage curriculum—progressing from synthetic clean scores to 9.7 million real-world IMSLP pages—to compel the model to internalize the underlying spatial and symbolic grammar of musical notation purely from unannotated images.
Method¶
Overall Architecture¶
MuSViT provides a general, reusable vision backbone capable of zero-shot feature extraction or high-capacity transfer across diverse musical visual tasks spanning micro-level symbol localization, line-level transcription, full-page end-to-end reading, and macro-level score difficulty classification. The pipeline consists of fine-grained patchification, a two-stage curriculum pre-training schedule, and a dual-track downstream evaluation protocol.
During pre-training, input score images are partitioned into non-overlapping fine-grained patches and subjected to heavy random masking (50% to 70%). A standard Vision Transformer (ViT) encoder processes exclusively the visible patches. A lightweight decoder, guided by 2D sinusoidal positional encodings and learnable mask tokens, reconstructs the normalized pixel values of the masked patches. Downstream, the learned representation is assessed via both linear probing with frozen backbones and end-to-end fine-tuning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Music Score Input<br/>512×512 Crops / 1024×1024 Full Page"] --> B["Two-Stage Curriculum Pre-Training<br/>Synthetic warm-up to real IMSLP adaptation"]
B --> C["Fine-Grained Patches & High-Ratio MAE<br/>P=16 patches with 50% to 70% masking"]
C --> D["2D Sinusoidal PE & Asymmetric ViT<br/>Explicit vertical pitch binding on visible tokens"]
D --> E["Dual-Track Downstream Adaptation<br/>Linear probing for features / fine-tuning for SOTA"]
Key Designs¶
1. Two-Stage Curriculum Pre-Training: Stabilizing Transfer from Clean Synthetics to 9.7M Real-World Scores
Direct self-supervised training on real-world scanned scores causes severe optimization instability: heterogeneous historical engraving styles, faded paper textures, ink bleed-through, and dense staves trigger catastrophic dimensional collapse, where the decoder merely outputs the mean background pixel values. To overcome this barrier, MuSViT adopts a two-stage curriculum learning schedule. In Stage 1 (Synthetic Warm-up), the encoder is initialized on the clean, synthetic DeepScoresV2 dataset using 512×512 random crops (\(N = 32 \times 32 = 1,024\) patches) with a moderate 50% masking ratio. This enables the network to easily capture elementary notehead geometries and rigid staff-line configurations. In Stage 2 (Real-World Adaptation), training transitions to the full IMSLP corpus comprising 9.7 million pages across 400,000 compositions. The input expands to full pages resized to 1024×1024 (\(N = 64 \times 64 = 4,096\) patches) and the masking ratio increases to 70%. This progression enables the model to incorporate full-page global context and multi-system layout topologies without suffering representation collapse.
2. Fine-Grained Patches and High-Ratio MAE: Compelling the Acquisition of Symbolic Music Grammar
Music notation is a highly concentrated visual symbolic language: noteheads, stems, flags, accidentals, and rests are extremely small and packed closely together. Standard ViT patch sizes (e.g., 32×32) encompass entire measures or multiple notes, causing distinct musical symbols to fuse into single tokens. MuSViT selects \(P = 16\), ensuring that each patch primarily captures an elementary glyph fragment. Operating under a 70% masking ratio, entire measures and sequential passages are occluded. Unlike natural images where missing patches can be reconstructed via low-frequency texture interpolation, reconstructing occluded music scores requires inferring what symbol must appear (duration and semantics) and precisely where it falls vertically on the staff (pitch) based on surrounding harmonic and melodic context. This pixel-level reconstruction task forces the encoder to develop an intrinsic understanding of musical grammar without any supervisory labels.
3. 2D Sinusoidal Positional Encoding and Asymmetric ViT: Explicit Pitch-Staff Geometry and Efficient Encoding
In sheet music, vertical position strictly dictates pitch frequency, while horizontal position governs temporal execution order. Conventional 1D flattened positional encodings obscure this 2D geometry, forcing the network to waste representational capacity rediscovering the staff coordinate plane. MuSViT applies explicit 2D sinusoidal positional encodings, decomposing spatial coordinates into orthogonal vertical and horizontal frequency components. The primary MuSViT encoder comprises a 12-layer ViT (\(d = 768\), ~85M parameters) that processes only the 30% unmasked patches, maintaining high computational efficiency over long patch sequences. A lightweight Transformer decoder processes the full sequence augmented with learnable mask tokens during pre-training and is discarded downstream. For edge and resource-constrained environments, the authors also introduce MuSViTLight (\(d = 384\), ~25M parameters).
4. Dual-Track Downstream Adaptation: Disentangling Representation Quality from Task Specialization
To rigorously evaluate the general utility of MuSViT representations, the authors introduce complementary evaluation protocols spanning sequence transcription, dense localization, and document-level classification: - Full-Page and Staff-Level Score Recognition: Evaluated on scanned pianoform pages (Mozarteum, Polish Digital Scores) using an autoregressive Transformer decoder, and on individual staff images across 5 historical/modern benchmarks using an RNN decoder. - Dense Music Symbol Detection: Evaluated on DeepScoresV2 (135 symbol classes) by attaching a Faster R-CNN detection head and tuning the backbone with parameter-efficient LoRA. - Score Difficulty Classification: Aggregating page-level embeddings via mean pooling or recurrent GRU networks on FreeScores, Can I Play It?, and PianoStreet to directly predict performance difficulty without intermediate transcription.
Loss & Training¶
The self-supervised training objective is the mean squared error (MSE) computed exclusively over the masked patch indices \(\mathcal{M}\): $\(\mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \| x_i - \hat{x}_i \|_2^2\)$ where \(x_i\) and \(\hat{x}_i\) denote the normalized ground-truth and predicted pixel vectors of patch \(i\). Crucially, unlike contrastive or distillation frameworks (such as DINO) that rely on aggressive data augmentations like Gaussian blur or high-frequency filtering—which obliterate staff lines and erase pitch information—MuSViT employs minimal augmentation consisting solely of random resized cropping, preserving the structural integrity of musical notation.
Key Experimental Results¶
Main Results¶
MuSViT and MuSViTLight are systematically compared against four cutting-edge general vision and multimodal foundation models: PaliGemma 2, Kosmos-2.5, Qwen3-VL, and DINOv3-7B.
Table 1: Linear probing performance across four downstream benchmarks (frozen encoder)
Note: Full-page and staff-level recognition report Symbol Error Rate (SER %, lower is better); symbol detection reports mAP and weighted mAP (w-mAP %, higher is better); difficulty classification reports exact accuracy Acc0 and 1-off accuracy Acc1 (%, higher is better).
| Model Type | Backbone Encoder | Full-Page SER ↓ | Staff-Level SER ↓ | Symbol Det. mAP ↑ | Symbol Det. w-mAP ↑ | Difficulty Acc0 ↑ | Difficulty Acc1 ↑ |
|---|---|---|---|---|---|---|---|
| General VLM | PaliGemma 2 [34] | 48.6 | 23.9 | 31.7 | 39.0 | 46.8 | 83.9 |
| Document VLM | Kosmos-2.5 [22] | 62.4 | 47.5 | 42.7 | 47.4 | 34.3 | 69.6 |
| General VLM | Qwen3-VL [4] | 51.0 | 21.0 | 67.1 | 61.0 | 43.3 | 83.1 |
| General SSL Vision | DINOv3-7B [33] | 56.9 | 32.1 | 70.4 | 62.0 | 45.5 | 83.9 |
| Foundation Model (Ours) | MuSViT | 16.4 | 18.4 | 79.7 | 80.7 | 47.4 | 87.1 |
| Lightweight (Ours) | MuSViTLight | 20.9 | 23.0 | 79.1 | 80.4 | 47.4 | 84.4 |
Table 2: Downstream fine-tuning performance compared with task-specific state of the art
| Downstream Task | Evaluated Datasets | Primary Metric | Previous Task SOTA | MuSViTLight | MuSViT (Ours) | Relative Margin vs SOTA |
|---|---|---|---|---|---|---|
| Full-Page Recognition | Mozarteum + Polish Scores | Avg. SER (%) ↓ | 20.0 [32] | 11.8 | 10.9 | -9.1 percentage points |
| Staff-Level Recognition | Capitan, FMT, etc. (5 sets) | Avg. SER (%) ↓ | 8.0 [24] | 9.8 | 8.6 | Within 0.6 points |
| Music Symbol Detection | DeepScoresV2 (135 classes) | \(\text{mAP}_{50}\) (%) ↑ | 90.5 [21] | 96.6 | 97.0 | +6.5 percentage points |
| Difficulty Classification | FreeScores, PianoStreet, etc. | Acc0 / Acc1 (%) ↑ | 38.4 / 84.3 [29] | 54.0 / 88.9 | 54.2 / 89.3 | Acc0 improved by +15.8% |
Ablation Study¶
Table 3: Embedding-transcription consistency analysis with symbolic music distances
Note: Evaluates Pearson (\(\rho^p\)) and Spearman (\(\rho^s\)) correlations between visual embedding Euclidean distances and ground-truth symbolic transcription distances (Levenshtein Edit Distance and Token Histogram Frequency Distance). Positive values reflect semantic alignment.
| Encoder Backbone | Edit Dist. Pearson \(\rho_{\text{ed}}^p\) | Histogram Pearson \(\rho_{\text{h}}^p\) | Edit Dist. Spearman \(\rho_{\text{ed}}^s\) | Histogram Spearman \(\rho_{\text{h}}^s\) | Alignment Implication |
|---|---|---|---|---|---|
| PaliGemma 2 | -0.110 | -0.127 | -0.120 | -0.132 | Weakly anti-correlated; misled by surface styling |
| Kosmos-2.5 | -0.080 | -0.130 | -0.114 | -0.153 | Misaligned with musical symbolic syntax |
| Qwen3-VL | -0.041 | -0.052 | -0.009 | -0.009 | Near-zero correlation; blind to note relations |
| DINOv3-7B | -0.080 | -0.100 | -0.135 | -0.152 | 7B parameters still yield negative correlations |
| MuSViT | 0.606 | 0.665 | 0.691 | 0.714 | Strong positive alignment with musical grammar |
| MuSViTLight | 0.618 | 0.646 | 0.658 | 0.662 | Compact variant preserves strong symbolic alignment |
Curriculum Pre-Training Ablation Insights: Ablating the synthetic warm-up stage on DeepScoresV2 by training single-stage MAE directly on raw IMSLP leads to severe dimensional collapse. An analysis of the singular value spectrum confirms that without the synthetic curriculum, effective rank plummets and the decoder outputs constant average pixel values, verifying that synthetic initialization is indispensable for stable convergence.
Key Findings¶
- General vision foundation models suffer from severe domain blindness: Under frozen linear probing on full-page transcription, PaliGemma 2 (48.6%), Qwen3-VL (51.0%), and DINOv3-7B (56.9%) fail completely, whereas MuSViT achieves 16.4% SER. The embedding-transcription consistency analysis demonstrates that general vision embeddings are anti-correlated with symbolic music content (-0.04 to -0.15), proving that scale cannot compensate for domain-specific inductive biases.
- Language-aligned pre-training degrades fine-grained spatial localization: In music symbol detection, vision-language backbones like PaliGemma 2 (mAP 31.7%) lag far behind vision-only models like DINOv3-7B (70.4%) and MuSViT (79.7%), indicating that global image-text contrastive objectives wash out localized spatial coordinate sensitivity.
- Full-page transcription benefits disproportionately from domain pre-training: While staff-level recognition shows modest gains (within 0.6% of SOTA due to constrained 1D line syntax), full-page end-to-end recognition experiences a massive 9.1-point SER reduction over task-specific SOTA, highlighting the impact of learned multi-system reading order and 2D spatial layouts.
Highlights & Insights¶
- Curriculum-guided self-supervision overcomes document collapse: The transition from clean synthetic pages to noisy historical IMSLP archives provides a robust blueprint for training foundation models on specialized, high-entropy document domains where standard MAE collapses.
- Mathematical validation via embedding consistency: Rather than relying exclusively on downstream metrics, the authors utilize correlation coefficients between visual embedding distance and transcription edit distance, proving mathematically that MuSViT constructs a semantically meaningful musical representation space.
- Remarkable efficiency against large-scale models: MuSViT (85M) and MuSViTLight (25M) deliver superior task-specific performance while requiring 5× to 82× fewer parameters and 16× to 260× fewer GFLOPs compared to multi-billion parameter backbones like DINOv3-7B and Qwen3-VL.
Limitations & Future Work¶
- Omission of cross-modal auditory pre-training: MuSViT is pre-trained exclusively on visual score sheets without grounding against audio recordings, missing potential cross-modal synergies for tasks like score-audio synchronization.
- Biased coverage toward Western standard notation: Pre-training data from IMSLP predominantly reflects Common Western Music Notation (CWMN) and Renaissance mensural notation, leaving numerical notation (Jianpu), guitar tabs, and non-Western notation systems largely unrepresented.
- Future directions: Integrating MuSViT as the visual front-end of a specialized Music-VLM to support conversational score analysis, automated musicological transcription, and multi-track compositional generation.
Related Work & Insights¶
- vs General-Purpose Vision Backbones (DINOv3, PaliGemma 2): General foundation models excel at natural objects and broad layout analysis but lack the spatial precision required for sub-millimeter staff line coordinate tracking. MuSViT establishes that domain-specific foundation models remain indispensable for specialized symbolic media.
- vs Traditional Multi-Stage OMR Pipelines: Conventional OMR models suffer from compounding errors across layout segmentation and symbol parsing. MuSViT's robust visual representations enable end-to-end full-page autoregressive models to surpass legacy modular systems by nearly 10 SER points.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ The first foundation vision model for sheet music, pioneering a robust two-stage MAE curriculum for symbolic document imagery.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across four distinct tasks, contrasting frozen linear probing, fine-tuning, and geometric representation consistency against top-tier vision backbones.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear pedagogical exposition, elegant mathematical analysis of representation alignment, and self-contained empirical rigor.
- Value: ⭐⭐⭐⭐⭐ Open-sourcing weights, code, and evaluation suites for a 9.7M-page pre-trained model provides an invaluable cornerstone for automated musicology and digital heritage preservation.