Skip to content

Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/RAN-rz/seeing_the_unseen
Area: Multimodal VLM
Keywords: Pixel-based Language Models, Visual Orthographic Similarity, Cross-script Transfer, Continued Pre-training, Low-resource Language Adaptation

TL;DR

Focusing on the adaptation bottlenecks of pixel-based language models on morphologically complex, low-resource scripts, this paper introduces a four-dimensional rendering-level visual orthographic metric and demonstrates through Tibetan case studies that orthographic proximity enhances semantic transfer while revealing an unexpected adaptation capacity asymmetry between monolingual and multilingual pre-training baselines.

Background & Motivation

Modern multilingual natural language processing has long been constrained by subword tokenizers such as BPE and UnigramLM. When dealing with low-resource languages, especially those written in non-Latin scripts, fixed vocabulary sizes lead to severe out-of-vocabulary (OOV) rates, token fragmentation, and disproportionate computational overhead caused by excessive Unicode code-points per word. Pixel-based language models, such as PIXEL, pixel-m4, and PIXAR, circumvent these structural tokenization barriers by rendering text directly as sequences of image patches processed by Vision Transformers. This design inherently unifies the input representation space and makes cross-lingual transfer theoretically feasible purely through visual and orthographic overlaps, liberating models from discrete vocabulary constraints.

However, while pixel-based architectures have exhibited intriguing cross-script generalization on high-resource benchmarks, the exact mechanisms underlying visual transfer have remained an uninspected black box. Prior investigations into cross-script transfer typically reduce script relationships to coarse binary buckets (such as "similar" versus "dissimilar") or rely on broad phylogenetic language families (such as grouping all Brahmic-derived scripts together). In reality, typographic and orthographic variations within a script family are continuous and gradient, manifesting in complex differences in stroke density, glyph aspect ratios, conjunct stacking, and ink distribution. Without fine-grained mathematical metrics formulated at the exact patch level consumed by the model, practitioners lack empirical principles to select optimal transfer languages or allocate scarce pre-training compute when adapting models to unseen scripts.

This methodological void is especially acute for languages with intricate morphological and orthographic profiles. Tibetan represents a prime exemplar: written in the Brahmic-derived Tibetan abugida (Tibt), it features dense syllable clusters centered around root consonants with complex vertical superscripts, subscripts, and attached vowels, delimited by tsheg rather than spaces. Furthermore, its agglutinative morphology and affixation structures impose severe challenges on conventional tokenizers. Core idea: this paper establishes a four-dimensional rendering-level orthographic similarity metric capturing pixel density, grapheme clusters, code-points, and stacking ratios, demonstrating via Tibetan continued pre-training that visual proximity reliably guides semantic cross-lingual transfer while exposing a fundamental adaptation asymmetry between monolingual and multilingual pre-training foundations.

Method

Overall Architecture

The framework establishes an end-to-end investigative pipeline spanning rendering-level orthographic metric quantification, cross-script continued pre-training (CPT), and multi-task evaluation across distinct linguistic hierarchies. The system begins by normalizing multilingual parallel corpora and rendering them into continuous sequences of greyscale image patches using a unified typography engine. On this patch-level representation, four quantitative metrics are extracted to construct language-specific feature vectors and compute pairwise Euclidean distances. Next, using monolingual pixel-mono and multilingual pixel-m4 as dual starting points, controlled CPT experiments are conducted across varying data scales and auxiliary language mixtures. Finally, fine-tuning across three representative Tibetan tasks—Text Classification, Named Entity Recognition, and Part-of-Speech tagging—reveals how low-level visual alignment modulates high-level semantic and syntactic abstractions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multilingual Text Corpora<br/>Tibetan / Dzongkha / Nepali / Burmese / Khmer / Chinese"] --> B["Continuous Sentence-Level Rendering<br/>PangoCairo + Noto Sans to 16x16 Greyscale Patches"]
    B --> C["Rendering-Level Orthographic Metrics<br/>Extract PD, GC, CP, CG for Vectors and Distance Matrix"]
    C --> D["Dual-Baseline Continued Pre-training Protocol<br/>CPT with pixel-mono & pixel-m4 across Scaled Conditions"]
    D --> E["Hierarchical Downstream Task Evaluation<br/>Text Classification TNCC / Sequence Labeling NER / Syntax POS"]

Key Designs

1. Rendering-Level Orthographic Metrics: Addressing the inability of discrete linguistic classifications to quantify continuous visual input features

Traditional cross-lingual transfer relies on lexical overlap, which fails completely in tokenizer-free pixel models. To address this, the authors introduce four complementary geometric and structural metrics calculated directly on the rendered blocks processed by the model. Let a corpus consist of \(N\) blocks, each containing \(T\) patches of size \(H \times W\) (both 16 pixels), where \(x_{i,j,k} \in [0, 255]\) denotes the greyscale value of the \(k\)-th pixel in the \(j\)-th patch of block \(i\): - Pixel Density (PD): Quantifies ink coverage and visual glyph complexity, using threshold \(\tau = 250\) to eliminate anti-aliasing artifacts: $\(\text{PD} = \frac{1}{N \cdot T \cdot H \cdot W} \sum_{i=1}^N \sum_{j=1}^T \sum_{k=1}^{H \cdot W} \mathbb{I}[x_{i,j,k} < \tau]\)$ - Average Grapheme Clusters per Block (GC): Encodes the average number of human-perceived characters \(g_i\) rendered per block: \(\text{GC} = \frac{1}{N} \sum_{i=1}^N g_i\). - Average Code-Points per Block (CP): Captures the raw underlying Unicode encoding length \(c_i\) per input block: \(\text{CP} = \frac{1}{N} \sum_{i=1}^N c_i\). - Code-Points-to-Grapheme Ratio (CG): Evaluates glyph compositionality and vertical stacking depth as \(\text{CG} = \frac{\text{CP}}{\text{GC}}\). For scripts with frequent consonant conjuncts and diacritics like Tibetan, multiple code-points synthesize a single perceived glyph, yielding \(\text{CG} \gg 1.0\), whereas logographic Han and alphabetic Latin scripts remain close to 1.0.

Validated across 41 languages from the Parallel Bible Corpus (PBC) covering six major script families, hierarchical Ward clustering on standardized 4D vectors achieves strong script alignment (Purity=0.683, ARI=0.325, NMI=0.548). The metrics capture fine-grained typographic continuums relative to Tibetan (bo): Dzongkha (dz, distance 0.52) is closest due to sharing the Tibt script, followed by Khmer (km, 1.589/2.02) and Devanagari Nepali (ne, 2.32/3.36), whereas curvilinear Burmese (my, 3.97/7.319) and logographic Chinese (zh, 2.70/4.666) lie furthest along the geometric continuum.

2. Continuous Sentence-Level Rendering: Addressing typographic fragmentation caused by bi-gram patch segmentation in complex scripts

Prior multilingual pixel architectures like pixel-m4 adopted a bi-gram rendering strategy to reduce patch redundancy and boost token co-occurrence frequencies. However, the authors identify that bi-gram splitting is destructive for scripts with intricate vertical conjunct stacking. In Tibetan, where single grapheme clusters integrate up to eight code-points across vertical and horizontal planes, arbitrary bi-gram chunking shears coherent orthographic units and severs perceptual glyph integrity.

The authors instead design a continuous sentence-level rendering pipeline using the PangoCairoTextRenderer with Google Noto Sans fonts. Normalized sentences are concatenated sequentially into blocks spanning 23 to 511 patches of \(16 \times 16\) pixels. This approach preserves intra-syllable diacritic structures and tsheg boundary markers intact. Crucially, experimental data volumes are standardized in terms of actual rendered blocks (~22 sentences per block) rather than deceptive sentence or character counts, providing a rigorous, patch-aligned foundation for controlled data ablation.

3. Dual-Baseline Continued Pre-training Protocol: Resolving adaptation capacity trade-offs between monolingual and multilingual foundations

To dissect how prior script exposure modulates subsequent low-resource adaptation, the authors implement dual CPT trajectories starting from two distinct initializations: English-only pixel-mono (providing a zero-proximity baseline) and pixel-m4 (pre-trained on English, Chinese, Ukrainian, and Hindi, thereby embedding prior Brahmic Devanagari exposure).

Under identical Masked Autoencoding objectives, experiments evaluate Tibetan data scaling (1k to 35k blocks), auxiliary language transfer at a fixed budget (24k blocks each for dz, ne, my, km, and zh), and multi-script mixtures (107k blocks). Structural probing reveals a layer-wise division of labor: lower Vision Transformer layers capture visual-orthographic patterns, whereas higher layers abstract semantic and syntactic structures. This dynamic induces a critical adaptation asymmetry: while pixel-m4 possesses superior zero-shot capabilities due to its Hindi pre-training, its shared multilingual representation space exhibits saturation under CPT. Conversely, pixel-mono retains expansive plasticity, achieving dramatic semantic performance leaps when adapted with multi-script auxiliary data.

Loss & Training

During CPT, models are optimized via span-masked patch reconstruction following the masked autoencoder paradigm. A span masking ratio of 0.25 with a maximum span length of 6 patches is applied. Optimization is performed using AdamW (\(\beta_1=0.9, \beta_2=0.999\), weight decay 0.05) with a base learning rate of \(1.5 \times 10^{-4}\) scaled by effective batch size, guided by a cosine decay schedule over 15 epochs with a 5% warmup ratio. The effective batch size is 32 across NVIDIA H100 GPUs (totaling 170–190 GPU hours).

Downstream fine-tuning configures sequence lengths of 256 patches for Text Classification and POS tagging, and 196 patches for NER. For token-level sequence labeling (NER and POS), text is rendered as word lists to enforce strict word-to-patch alignment for loss computation. Fine-tuning uses AdamW with an initial learning rate of \(3.0 \times 10^{-5}\), linear decay, and early stopping on validation macro-F1 (patience 20 for classification, 5 for tagging).

Key Experimental Results

Main Results

Evaluations span three Tibetan downstream benchmarks: TNCC Text Classification (12 classes, macro-F1), Tibetan-Mongolian Newspaper NER (5 entity types, macro-F1), and TDTreebank v1.1 POS tagging (17 UPOS tags, adjusted macro-F1 excluding 4 sparse classes). Results compare unadapted baselines against various CPT regimes, with Glot500-base (subword, 511 languages, 1.2B sentences) and CANINE-S (character, 104 languages) serving as external reference points.

Experimental Condition CPT Corpus Scale NER F1 (%) POS F1 (%) TextCL F1 (%) Underlying Mechanism & Observations
pixel-mono (no CPT) 0 blocks 59.49±0.11 70.46±0.23 41.73±0.20 Pure English Latin baseline; zero Tibetan script exposure
pixel-m4 (no CPT) 0 blocks 69.08±0.02 70.57±0.85 44.34±0.53 Benefits from Devanagari (Hindi) Brahmic pre-training prior
pixel-m4 + bo 35k blocks 70.19±1.28 71.44±0.11 47.72±0.81 Tibetan-only CPT; noticeable semantic and entity gains
pixel-m4 + bo + dz 35k bo + 24k dz 69.68±0.66 71.76±0.20 47.48±0.14 Adding Dzongkha (same Tibt script) peaks syntactic POS
pixel-m4 + bo + my 35k bo + 24k my 68.26±1.80 71.15±0.54 46.34±2.40 Adding distant Burmese degrades NER and classification
pixel-m4 + bo + ne 35k bo + 24k ne 69.38±0.57 71.26±0.48 47.75±0.13 Moderate orthographic proximity; balanced performance
pixel-m4 + allmixed 107k blocks (4 langs) 70.55±0.02 71.72±0.12 48.61±0.47 Optimal pixel-m4 configuration across semantic tasks
pixel-mono + allmixed 107k blocks (4 langs) 68.26±0.41 71.41±0.09 54.57±0.39 Massive semantic adaptation leap; outstrips pixel-m4 by 5.96 pts
Glot500-base (subword ref) 1.225B sents (inc. bo) 82.59 72.47 62.14 Upper-bound reference trained on massive multilingual text
CANINE-S (character ref) 104 Wikipedia langs 78.72 77.10 52.13 Tokenizer-free baseline matched/exceeded by pixel models on TextCL

Ablation Study

Table 1: Impact of auxiliary language orthographic proximity on cross-lingual transfer (pixel-mono, 24k blocks each)

Models are adapted from pixel-mono using a fixed 24k block budget of auxiliary languages, ordered from highest to lowest orthographic proximity to Tibetan (dz > km > ne > zh > my):

Auxiliary Language Script Family Euclidean Distance to bo TextCL F1 (%) NER F1 (%) POS F1 (%) Behavioral Insights & Transfer Dynamics
Dzongkha (dz) Tibetan (Tibt) 0.52 53.41±0.13 67.59±1.00 71.33±0.80 Same script; high-quality corpus even surpasses bo 24k self-CPT
Khmer (km) Khmer (Khmr) 1.59 / 2.02 51.85±0.73 60.74±1.29 70.40±1.45 Closest geometric PBC neighbor; strong classification but poor NER
Nepali (ne) Devanagari (Deva) 2.32 / 3.36 47.67±0.30 64.14±1.52 70.58±0.08 Moderate proximity; contains continuous shirorekha headline
Chinese (zh) Simplified Han (Hans) 2.70 / 4.67 50.85±1.42 61.63±0.45 69.77±0.86 High pixel density but CG=1.00; lacks multi-codepoint stacking
Burmese (my) Myanmar (Mymr) 3.97 / 7.32 46.15±0.85 63.68±0.01 70.95±0.54 Curvilinear script; lowest classification performance
bo (Tibetan self-CPT) Tibetan (Tibt) 0.00 (Anchor) 50.56±0.81 65.46±0.57 71.53±0.21 Target language self-adaptation baseline

Table 2: Data scale sensitivity ablation during Tibetan CPT (pixel-mono)

CPT Tibetan Data Budget TextCL F1 (%) Gain over Baseline NER F1 (%) Gain over Baseline POS F1 (%) Gain over Baseline Marginal Utility Trend
No CPT (0k) 41.73±0.20 — 59.49±0.11 — 70.46±0.23 — Unadapted monolingual baseline
+ 1k blocks 45.82±0.78 +4.09 63.18±0.33 +3.69 70.82±0.03 +0.36 Steep initial gains via early visual alignment
+ 4k blocks 45.82±0.19 +4.09 65.09±0.33 +5.60 70.99±0.04 +0.53 NER continues rising; TextCL plateaus temporarily
+ 8k blocks 48.41±0.26 +6.68 64.75±1.79 +5.26 71.08±0.12 +0.62 Steady semantic recovery across longer context
+ 24k blocks 50.56±0.81 +8.83 65.46±0.57 +5.97 71.53±0.21 +1.07 Entity recognition nears saturation threshold
+ 35k blocks 52.75±0.10 +11.02 65.59±0.92 +6.10 71.53±0.23 +1.07 TextCL displays persistent upward trajectory

Key Findings

  • Orthographic proximity governs semantic cross-lingual transfer: For pixel-mono adapted on auxiliary languages, text classification accuracy directly tracks orthographic proximity (dz 53.41 > ne 47.67 > my 46.15). Remarkably, Dzongkha transfer (53.41) outstrips Tibetan self-CPT (50.56), propelled by the superior lexical diversity and domain cleanliness of Dzongkha official government publications.
  • Front-loaded adaptation gains under ultra-low-resource budgets: Introducing merely 1k blocks (~22k sentences) yields an immediate surge of +4.09 F1 in classification and +3.69 F1 in NER, capturing 40% to 60% of total gains attainable with 35k blocks. This proves that low-level vision layers in Transformers calibrate to basic stroke and script topologies with exceptional data efficiency.
  • Task abstraction decoupling: Semantic understanding tasks (TextCL) scale continuously with CPT data without reaching saturation at 35k blocks, whereas sequence labeling (NER) plateaus near 24k blocks. Conversely, syntactic POS tagging exhibits near-complete insensitivity across all data scales (+1.07 F1 total), demonstrating that low-level orthographic realignment primarily enriches semantic lexical associations rather than abstract morphosyntactic induction.
  • Pre-training starting point asymmetry: Although pixel-m4 starts with a 7–10 point advantage over pixel-mono on zero-shot NER and TextCL, pixel-mono demonstrates dramatically higher plasticity when subjected to multi-script CPT. On 107k mixed blocks, pixel-mono surges to 54.57 on Text Classification—beating pixel-m4 + allmixed (48.61) by nearly 6 F1 points and outperforming CANINE-S (52.13).

Highlights & Insights

  • Continuous geometric metrics replace discrete typological proxies: By quantifying rendering statistics (PD, GC, CP, CG), this work bridges computer vision and computational linguistics, providing the first continuous, mathematically grounded distance measure tailored for patch-based language models.
  • Debunking the necessity of massive pre-training corpora for new scripts: The discovery of front-loaded gains demonstrates that pixel models do not require millions of sentences to acquire foreign writing systems; modest, high-quality rendered subsets (1k–4k blocks) suffice to bridge cross-script gaps.
  • Challenging the default dominance of multilingual pre-training foundations: Demonstrating that a monolingual base model can out-adapt a multilingual foundation on long-context semantic tasks under identical mixed CPT conditions offers critical architectural insights into representation interference and parameter rigidity.

Limitations & Future Work

  • Linguistic and script scope boundaries: Empirical validations center primarily around Tibetan and selected Brahmic-family scripts alongside Chinese, leaving abjads (Arabic/Hebrew) and other non-Latin scripts for future exploration.
  • Task-specific metric divergence: While the four metrics successfully rank overall script clustering and classification transfer, localized anomalies emerge: Khmer exhibits the closest geometric profile to Tibetan but underperforms on NER (60.74 vs Burmese 63.68), indicating that unsegmented script layouts and word-boundary dynamics introduce orthogonal task complexities.
  • Future directions: The authors plan to extend the metric suite into an automated data-selection engine that dynamically curates multi-script rendering shards to optimize pre-training mixtures for arbitrary low-resource targets.
  • vs. Subword Multilingual LMs (Glot500, mBERT, NLLB): Subword models suffer from extreme token fragmentation and inflated sequence lengths on non-Latin scripts; pixel models completely eliminate tokenizer unfairness, allowing compact, direct visual processing that excels on low-resource semantic classification.
  • vs. Character/Byte-level Encoders (CANINE-S, ByT5): While token-free, character models serialize text along 1D strings, failing to exploit 2D spatial conjunct topologies; pixel-based models match or surpass CANINE-S on document-level classification while leveraging visual proximity.
  • vs. Prior Pixel Models (PIXEL, pixel-m4, PIXAR): Earlier architectures evaluated high-resource languages or relied on bi-gram rendering strategies that disrupt complex scripts; this work pioneers continuous sentence-level rendering and provides the first fine-grained quantitative framework for low-resource pixel adaptation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first rendering-level 4D orthographic metric suite, fundamentally advancing continuous cross-script transfer analysis for pixel models.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 41 languages on the Parallel Bible Corpus, combined with extensive Tibetan multi-scale and multi-script ablation suites.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exemplary structure and clarity, meticulously detailing mathematical definitions, anomalous findings, and representational trade-offs.
  • Value: ⭐⭐⭐⭐⭐ Exceptional practical utility for low-resource language engineering, multimodal tokenization-free architectures, and cross-lingual pre-training strategies.