Skip to content

Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Canonical Color, Concept Decodability, Linear Probing, Vision Encoders, Multimodal VLMs

TL;DR

By adopting "canonical color" as a controlled lens and removing chromatic and luminance cues via grayscale conversion and histogram equalization, this work demonstrates that object-level conceptual knowledge remains linearly decodable in frozen vision encoders and reveals that VLM post-training reformats rather than erases these visual semantic representations downstream.

Background & Motivation

Human conceptual knowledge routinely associates visual objects with diagnostic prototypical attributes, among which canonical color (e.g., bananas are yellow, flamingos are pink) plays a critical role in categorization and semantic reasoning. In pure language models, learning such commonsense physical properties is severely hampered by reporting bias, where standard textual corpora disproportionately document unusual events while omitting obvious world knowledge. Multimodal vision-language models (VLMs) promise to alleviate this bottleneck through grounded perceptual learning. However, modern representation studies face an unresolved question: when a model demonstrates proficiency on downstream tasks, does its vision encoder construct authentic object-level conceptual knowledge, or does it merely rely on shallow, retinal-level chromatic cues present in the input pixels?

Previous representation probing investigations frequently evaluated linear classifiers on raw RGB images, which unavoidably entangles the detection of conceptual knowledge with immediately visible surface pixels. Meanwhile, evaluating full end-to-end VLMs via generated text conflates visual comprehension with language generation priors, dataset hallucination tendencies, and weak cross-modal attribute binding. Even behavioral evaluations on counterfactual stimuli (e.g., recoloring an object to a non-canonical shade like a blue strawberry) can only document macroeconomic model performance conflicts; they remain a black box unable to isolate where along the vision-language backbone conceptual semantic knowledge is encoded, degraded, or restored.

To disentangle surface perception from deep conceptual representations, this work capitalizes on the unique physical property of color: unlike complex visual attributes such as texture, geometry, or material properties, color can be eliminated from an image via grayscale conversion while preserving the underlying object structure intact. This observation motivates the central experimental inquiry: when an encoder is fed a black-and-white depiction of a banana that is completely stripped of chromatic cues and simple brightness disparities, does its internal feature space still render "yellow" linearly accessible? Core idea: by utilizing physically removable yet conceptually bound "canonical color" as a controlled lens under rigorous grayscale and histogram equalization controls, one can systematically isolate concept-level linear decodability in vision encoders and map the representational reformatting across the multi-stage VLM backbone.

Method

Overall Architecture

The methodological pipeline consists of three structured stages: (1) curated canonical color dataset construction with multi-level input de-biasing and counterfactual validation; (2) dual-task orthogonal linear probing across diverse frozen vision encoders and model scales; and (3) end-to-end representational tracking across matched pre-/post-VLM vision towers, learnable visual interfaces, and decoder-side visual token representations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image Corpus<br/>708 Object Classes × 10 Canonical Colors"] --> B["Controlled Data Curation & Color Stripping<br/>Grayscale Conversion + Histogram Equalization + Visual-CounterFact Validation"]
    B --> C["Layer-wise Feature Extraction<br/>Cached Patch-Averaged States from Frozen ViT / Swin"]
    C --> D["Dual-Task Orthogonal Linear Probing<br/>5-Fold Class-Level CV (Color) & Within-Class Holdout (Object Identity)"]
    D --> E["Cross-Stack VLM Reformatting Tracking<br/>Pre-/Post-VLM Vision Towers → Visual Interface (Q-Former) → Decoder Visual Tokens"]

Key Designs

1. Controlled Data Curation & Color Stripping: Eliminating Chromatic Cues and Validating Against Counterfactual Surface Biases

To ensure that probing reflects abstract conceptual knowledge rather than surface pixel correlations, the authors construct a benchmark derived from Wikidata entities possessing an unambiguous has-color (P462) property, mapped into 10 fundamental color categories (white, yellow, pink, green, black, blue, purple, red, brown, and gray). After strict manual curation to remove ambiguous imagery and downsampling Pokémon concepts to 50 to prevent taxonomic distortion, the dataset encompasses 708 distinct object classes with up to 5 verified images per class. On the input side, images undergo luminance conversion replicated across three channels, followed by per-image histogram equalization to eliminate residual brightness or contrast cues that correlate with particular colors. To validate this control, the authors evaluate Visual-CounterFact pairs: when linear probes are trained on recolored images, actual surface color prediction drops to the majority-class random baseline (~14%), whereas canonical color accuracy on both original and counterfactual grayscale images remains robustly elevated and traces nearly identical layer-wise trajectories. This proves that the residual signal reflects intrinsic canonical object semantics rather than residual pixel features.

2. Dual-Task Orthogonal Linear Probing: Disentangling Semantic Attribute Decodability from Object Identification

To determine whether canonical color decodability is simply a trivial consequence of object classification, the authors formulate a rigorous dual-probe framework. For canonical color probing, an L2-regularized multinomial logistic regression probe is trained under a 5-fold object-class-level cross-validation protocol: all images belonging to the same object class are strictly confined to a single fold. This guarantees that test classes are entirely unseen during probe training, forcing the probe to infer color from generalized conceptual representations (e.g., generalizing from black-and-white lemons to black-and-white bananas) rather than memorizing individual object instances. For object classification, a within-class image hold-out scheme evaluates 708-way categorization over patch-averaged layer representations. By analyzing Spearman rank correlations between layer-wise trajectories and measuring "predicted-object color consistency" (the percentage of instances where the color probe matches the canonical color of the object predicted by the object probe), the authors empirically characterize the relationship between categorical identification and semantic property decodability.

3. Cross-Stack VLM Reformatting Tracking: Tracing Semantic Transformations Across Vision Towers, Interfaces, and Language Decoders

To examine how multimodal post-training reshapes the internal geometry of visual representations, the authors bypass black-box generated responses and directly probe intermediate representations across three matched pre- and post-VLM architectural pairs: OpenAI CLIP ViT-L/14@336px vs. Molmo-7B-D, CLIP ViT-L/14@224px vs. mPLUG-Owl, and SigLIP-So400m@224px vs. PaliGemma-3B. Grayscale probing is performed systematically across standalone vision towers, intermediate projection layers (e.g., the 6-layer Q-Former visual abstractor in mPLUG-Owl), and decoder-side visual-token positions (visual-avg states averaged across visual token slots in each language decoder block). A blank-image baseline is incorporated to verify that recovered decoder signals depend strictly on visual input rather than autonomous language model hallucinations. This setup reveals that post-training induces a representational reformatting—where linear accessibility may collapse in the standalone vision tower but is systematically restored through the visual abstractor and decoder layers.

Benchmark & Evaluation Protocol

The empirical investigation evaluates five diverse vision encoders covering language-supervised objectives (CLIP, SigLIP), self-supervised self-distillation (DINOv2), masked autoencoding (ViT-MAE), and hierarchical local-window architectures (Swin-V2), with DINOv2 further tested across four parameter scales (Small 22M, Base 86M, Large 304M, Giant 1.1B). For full VLMs, behavioural prompting queries (Visual Color, Canonical Color, and Object Category) are benchmarked against internal peak decoder visual-token linear probes to contrast superficial behavioural text generation against authentic internal conceptual decodability.

Key Experimental Results

Main Results

Under histogram-equalized grayscale inputs, best-layer canonical color probing accuracy across all vision encoders significantly exceeds the majority-class baseline (13.98%), demonstrating that conceptual color is linearly accessible despite substantial drops from the RGB upper bound:

Vision Encoder Scale / Params Grayscale + HE Acc. (%) 95% Confidence Interval vs. Baseline (13.98%) RGB \(\to\) Gray Drop (pp)
CLIP (ViT-B/32) 86M (Base) 41.5% [38.4, 44.6] \(p < .001\) 34.9 pp (\(p < .001\))
DINOv2 (Base) 86M (Base) 43.8% [37.0, 50.7] \(p < .001\) 32.8 pp (\(p < .001\))
SigLIP (Base) 86M (Base) 42.4% [37.7, 47.2] \(p < .001\) 39.5 pp (\(p < .001\))
ViT-MAE (Base) 86M (Base) 40.3% [35.3, 45.4] \(p < .001\) 35.3 pp (\(p < .001\))
Swin-V2 (Base) 88M (Base) 40.2% [35.8, 44.6] \(p < .001\) 37.1 pp (\(p < .001\))

In matched pre-/post-VLM comparisons, VLM post-training induces model-dependent changes in standalone vision tower decodability:

Model Transition Probing Task Pre-VLM Tower Acc. (%) Post-VLM Tower Acc. (%) Difference (pp) Paired Test Significance
CLIP \(\to\) Molmo Canonical Color 43.9% 36.8% -7.1 pp \(p < .001\)
CLIP \(\to\) Molmo Object Identity 65.0% 26.2% -38.8 pp \(p < .05\)
CLIP \(\to\) mPLUG-Owl Canonical Color 43.0% 30.8% -12.3 pp \(p < .001\)
CLIP \(\to\) mPLUG-Owl Object Identity 64.4% 7.0% -57.4 pp \(p < .001\)
SigLIP \(\to\) PaliGemma Canonical Color 43.3% 42.9% -0.4 pp n.s.
SigLIP \(\to\) PaliGemma Object Identity 73.3% 71.5% -1.9 pp \(p < .001\)

Ablation Study & In-Depth Analysis

1. Model Scaling within the DINOv2 Family: Across identical 5-fold evaluation splits, canonical color decodability scales monotonically with model capacity across all input conditions (Spearman \(\rho = 1.00\)):

Model Scale Parameter Count RGB Input (%) Grayscale Input (%) Grayscale + HE (%)
DINOv2-S 22M 71.1% 46.2% 41.8%
DINOv2-B 86M 77.0% 48.6% 43.8%
DINOv2-L 304M 79.4% 51.0% 46.0%
DINOv2-g 1.1B 80.6% 51.8% 48.0%

2. Prompt-Level Behavioral Answers vs. Internal Decoder Probes on Grayscale Images: Final generation prompts fail to accurately reflect visual perceptual fidelity, whereas internal decoder visual-avg probes reveal substantial semantic recovery:

Evaluation Dimension / Task Molmo-7B-D mPLUG-Owl (7B) PaliGemma-3B Mix
Behavioral Prompt: Visual Color (Expected: gray) 23.4% 14.9% 17.3%
Behavioral Prompt: Canonical Color (Real world) 44.5% 25.6% 4.1%
Behavioral Prompt: Object Category (Identification) 53.0% 57.1% 61.3%
Internal Probe: Dec. color probe (Peak visual-avg) 42.7% 40.6% 43.2%
Internal Probe: Dec. object probe (Peak visual-avg) 59.7% 57.5% 69.9%

Key Findings

  • Conceptual Color is Linearly Decodable Without Visible Color: Even after removing pixel-level color and luminance biases, all five vision encoders achieve 40.2%–43.8% accuracy on unseen object classes, substantially outperforming the 13.98% majority baseline. This establishes that visual encoders encode category-level conceptual knowledge rather than merely reading out chromatic pixels.
  • Conceptual Attributes and Categorical Identity are Strongly Correlated but Distinct: Layer-wise Spearman correlation between color probing and object identity ranges from 0.601 to 0.895 across encoders, and predicted-object color consistency reaches 55%–65% in mid-to-late layers. However, color decodability frequently emerges prior to peak object recognition, indicating that attribute knowledge is not a simple linear byproduct of categorical classification.
  • VLM Post-Training Induces Semantic Reformatting Rather Than Semantic Deletion: In Molmo and mPLUG-Owl, VLM fine-tuning severely degrades standalone vision tower linear decodability (object classification drops by 38.8 pp and 57.4 pp). However, this information is systematically recovered at the visual interface (e.g., Q-Former abstractor) and inside decoder visual tokens, confirming that multimodal alignment reorganizes the geometric subspace of visual tokens for LLM ingestion.

Highlights & Insights

  • Physical Grounding as an Information-Theoretic Control: The authors identify color as a unique visual modality that can be physically stripped from the input without corrupting higher-order object topology, enabling a clean separation between perceptual pixel readouts and abstract conceptual knowledge.
  • Empirical Validation of the Representational Reformatting Hypothesis: By mapping the layer-by-layer semantic trajectory through vision towers, abstractors, and decoder tokens, the paper provides compelling empirical evidence that multimodal training reformulates internal feature representations into non-linear or projected spaces optimized for language processing.
  • De-biasing Behavioral Evaluation in VLMs: The findings highlight how behavioral prompt answering conflates language priors, output surface tokens, and visual perception, demonstrating that internal representation probing offers a far more faithful and granular diagnostic tool for multimodal foundation models.

Limitations & Future Work

  • Narrow Attribute Scope: The strict experimental controls rely on the physical separability of color via grayscale conversion. Other essential visual concepts—such as texture, material, affordance, and physical mass—cannot be removed from images without fundamentally disrupting object identity, requiring alternative causal intervention designs.
  • Boundaries of Linear Probing: The study measures linear accessibility; it cannot definitively prove whether representations that fail linear probing in the post-VLM vision tower still retain the information in a non-linear manifold accessible via multi-layer MLPs or attention heads.
  • Future Directions: Extending the controlled probing framework to intermediate representation editing (activation steering) to evaluate whether manipulating canonical attribute subspaces in decoder visual tokens can directly mitigate counterfactual hallucination in downstream VLM reasoning.
  • vs Oneață et al. (2025): Oneață et al. probed semantic attributes in frozen vision encoders using raw RGB inputs, conflating visible chromatic pixels with conceptual knowledge. This paper introduces grayscale and histogram equalization controls to rigorously isolate conceptual decodability.
  • vs Golovanevsky et al. (2025) (Visual-CounterFact): Golovanevsky et al. analyzed macro-level conflicts between visual inputs and memorized priors. This work adopts their counterfactual images as a representation-level control and extends the inquiry into layer-wise localization across the entire VLM architecture.
  • vs Tang et al. (2023) (Concept Association Bias): Tang et al. studied behavioral biases where VLMs hallucinate missing concepts. This work provides the representational explanation: conceptual properties remain actively encoded and decodable inside decoder visual-token representations even when the standalone vision tower exhibits reduced linear accessibility.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant experimental framing using canonical color as a causal lens to isolate abstract conceptual decodability.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 vision encoders, 4 model scales, counterfactual controls, and 3 full VLM pipelines.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous prose with coherent scientific reasoning, clear hypotheses, and exhaustive statistical validation.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational diagnostic methodology for understanding internal semantic representations and cross-modal alignment dynamics in VLMs.