Skip to content

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/RISys-Lab/ReasonCLIP
Area: Multimodal VLM
Keywords: visually grounded commonsense reasoning, vision-language model (VLM), CLIP, continual pretraining, benchmark

TL;DR

ReasonCLIP-58M presents a non-intrusive two-stage continual pretraining framework powered by 58.9M visually grounded reasoning samples (ReasonLite-42M and ReasonPro-16M), injecting deep commonsense inference capabilities into CLIP without modifying backbone architectures while enhancing both retrieval and compositional reasoning.

Background & Motivation

Contrastive Language-Image Pretraining (CLIP) and its architectural derivatives serve as the quintessential visual foundation for modern multimodal systems and Multimodal Large Language Models (MLLMs). By aligning paired visual and textual representations across expansive corpora through contrastive loss, CLIP establishes effective cross-modal representations. However, mainstream CLIP pretraining remains heavily dominated by web-scraped descriptive image-text pairs, in which text primarily captures superficial semantic correspondence (e.g., "two cats standing on a table with two monitors"). Such supervision trains models to correlate co-occurring object nouns and visible regions, but completely bypasses higher-level reasoning over physical dynamics, geometric constraints, biological intentions, and causal affordances grounded in visual scenes.

As downstream applications increasingly demand rigorous compositional reasoning, multi-step understanding, and embodied decision-making, the mismatch between descriptive pretraining objectives and reasoning-oriented downstream requirements has become acute. Recent advances largely prioritize scaling descriptive data corpora (such as DataComp and MetaCLIP) or engineering architectural variants, leaving the lack of reasoning-aware representations fundamentally unaddressed. Concurrently, naively fine-tuning contrastive encoders on complex multi-step reasoning outputs designed for generative MLLMs frequently distorts the original metric space, inducing catastrophic forgetting of descriptive alignment and degrading general retrieval performance.

To overcome this dilemma, this paper investigates whether a standard CLIP representation space can acquire visually grounded reasoning capabilities through non-intrusive continual pretraining alone. Core idea: curate a 58.9M dataset of visually grounded commonsense reasoning pairs spanning open-form inferences (ReasonLite-42M) and category-structured supervision (ReasonPro-16M), and execute a two-stage continual pretraining framework combining dynamic annealing dual-objective alignment with explicit classification-guided category discrimination to seamlessly endow CLIP encoders with reasoning ability without sacrificing descriptive alignment.

Method

Overall Architecture

ReasonCLIP-58M maintains the standard architecture of visual and textual encoders, incurring zero additional inference latency or parameter overhead. The framework encompasses dataset construction, progressive two-stage continual pretraining, and plug-and-play downstream integration. On the data side, the authors build ReasonLite-42M for open-form verifiable reasoning and ReasonPro-16M across five structured perception categories from CC12M. On the training side, Stage 1 deploys dual-objective alignment with dynamic weight annealing to ensure smooth transition from descriptive to reasoning sensitivity, while Stage 2 leverages dual-sided lightweight classification heads to enforce structured discrimination across distinct reasoning types. Finally, the trained encoder serves as a drop-in vision tower for downstream MLLM architectures such as LLaVA-NeXT.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: CC12M images and refined descriptive captions Tb"] --> B["Two-tier complementary reasoning dataset construction<br/>Evidence-grounded generation for ReasonLite-42M and ReasonPro-16M"]
    B --> C["Dynamic annealing dual-objective reasoning alignment<br/>Joint optimization of Tb and Trl with schedule and L2 regularization"]
    C --> D["Explicit classification-guided category discrimination<br/>Targeting 5 physical reasoning categories via image/text classification heads"]
    D --> E["Zero-overhead drop-in multimodal integration<br/>Replacing vision tower in MLLMs without extra latency or tuning"]

Key Designs

1. Two-tier complementary reasoning dataset construction: from open-form inferences to category-structured supervision Existing multimodal reasoning datasets are primarily tailored for generative models, frequently containing ungrounded speculation or overly lengthy reasoning chains unsuitable for contrastive representation learning. To produce compact, verifiable, and visually grounded supervision, the authors retain 10.4M valid images from CC12M and deploy Qwen2.5-VL-72B to synthesize three factually consistent, high-quality descriptive captions per image (\(T_b\)), yielding the CC12M-Refined corpus (31.2M pairs) to eliminate raw web noise. The dataset is then split to construct two complementary collections. First, ReasonLite-42M focuses on open-form reasoning: the annotation model extracts visually observable evidence sets from each image and \(T_b\), clusters them into three semantically coherent groups, and infers a visually verifiable commonsense statement \(T_{rl}\) for each cluster (e.g., deducing "conveniently carried and eaten" from "neatly arranged in a portable container"), totaling 42M pairs. Second, ReasonPro-16M establishes category-specific supervision over five structured perceptual reasoning domains: Spatial/Geometric (S), Attribute/State (A), Creature/Action (C), Temporal/Phase (T), and Intuitive Physics (P). Using Qwen3-VL-32B, images with at least three valid reasoning categories are identified from the remaining 5.7M images, and three targeted category-specific reasoning captions (\(T_{rp}\)) are generated per image, producing 16.6M structured pairs.

2. Dynamic annealing dual-objective reasoning alignment: preserving descriptive foundations during transition Directly tuning a pretrained CLIP model on pure reasoning text disrupts its established semantic metric space, precipitating catastrophic forgetting. In Stage 1, the model is trained on ReasonLite-42M using both descriptive captions \(T_b\) and open reasoning captions \(T_{rl}\) under a joint dual-objective loss. To orchestrate a smooth progression from strict descriptive anchoring to reasoning sensitivity, the relative weight between descriptive and reasoning objectives is governed by a time-dependent annealing factor \(\lambda(t)\):

\[\mathcal{L}^{(1)} = \lambda(t)\,\mathcal{L}_{\text{align}}(x, T_b) + \bigl(1 - \lambda(t)\bigr)\,\mathcal{L}_{\text{align}}(x, T_{rl}) + \beta\,\|\theta - \theta_0\|_2^2\]

where \(\mathcal{L}_{\text{align}}\) denotes the backbone-specific alignment objective (symmetric InfoNCE for CLIP; sigmoid binary cross-entropy for SigLIP), \(\theta\) and \(\theta_0\) represent the active and initial model parameters respectively, and the \(\ell_2\) regularization term prevents parameter drift. The annealing coefficient \(\lambda(t)\) smoothly decays from \(\lambda_{\max}\) to \(\lambda_{\min}\):

\[\lambda(t) = \lambda_{\min} + (\lambda_{\max} - \lambda_{\min}) \cdot \mathrm{clip}\left(\frac{t_2 - t}{t_2 - t_1}, \; 0, \; 1\right)\]

This scheduling ensures that descriptive alignment dominates early pretraining to maintain grounding, progressively ramping up reasoning supervision to harmonize both modalities within a unified representation space.

3. Explicit classification-guided category discrimination: organizing multi-dimensional reasoning representations While Stage 1 imparts general reasoning awareness, the latent space still lacks explicit disentanglement among distinct physical reasoning patterns. Stage 2 transitions training entirely to ReasonPro-16M, retiring descriptive captions \(T_b\) and focusing exclusively on category-specific reasoning captions \(T_{rp}\). To encourage the representation space to structure and separate varied reasoning patterns, lightweight MLP classification heads \(g_v(\cdot)\) and \(g_t(\cdot)\) are appended to the vision and text encoders during training, formulating a joint discrimination loss:

\[\mathcal{L}^{(2)} = \mathcal{L}_{\text{align}}(x, T_{rp}) + \gamma\,\Bigl(\mathcal{L}_{\text{img-cls}}(x, Y) + \mathcal{L}_{\text{txt-cls}}(T_{rp}, c)\Bigr)\]

Because natural scenes often exhibit multiple simultaneous physical dimensions, the vision branch employs binary cross-entropy over multi-hot ground truth labels \(\mathbf{y} \in \{0, 1\}^5\): \(\mathcal{L}_{\text{img-cls}} = \mathrm{BCE}\bigl(g_v(x), \mathbf{y}\bigr)\). Conversely, since each caption \(T_{rp}\) targets an explicit category \(c \in \{1,\dots,5\}\), the text branch optimizes standard multi-class cross-entropy: \(\mathcal{L}_{\text{txt-cls}} = \mathrm{CE}\bigl(g_t(T_{rp}), c\bigr)\). These lightweight auxiliary heads enforce structured feature clustering during training and are discarded at inference, maintaining zero inference overhead.

4. Zero-overhead drop-in multimodal integration: delivering reasoning-aware vision towers to MLLMs In modern MLLM architectures such as LLaVA-NeXT, reasoning bottlenecks often stem not from the language backbone, but from the visual encoder capturing only superficial object co-occurrences. Stage 3 demonstrates the drop-in integration of ReasonCLIP. The original CLIP visual encoder is directly swapped with the continually pretrained ReasonCLIP encoder \(\phi_{\text{ReasonCLIP}}\) within downstream systems:

\[\mathcal{M}(x, \cdot) = \mathcal{F}\bigl(\phi_{\text{Baseline}}(x), \cdot\bigr) \longrightarrow \mathcal{F}\bigl(\phi_{\text{ReasonCLIP}}(x), \cdot\bigr)\]

Because ReasonCLIP preserves the identical ViT architecture, token dimensions, and positional encodings, downstream multimodal architectures ingest enriched, reasoning-aware visual representations without requiring structural adaptation, architectural modifications, or increased inference compute.

Loss & Training

  • Baseline Setups (Stage 0): To isolate the impact of structured stages, two baselines are established: S0-Des (trained exclusively on descriptive text \(T_b\)) and S0-Rea (naive mixture of all reasoning captions without stage distinction or classification supervision).
  • Optimization Strategy: Training is conducted on NVIDIA A100 64GB GPUs with effective batch sizes of 24,576 or 32,768. Stage 1 and Stage 2 each run for one epoch on ReasonLite-42M and ReasonPro-16M, respectively. Models span CLIP ViT-B/32, ViT-L/14 (224px and 336px), SigLIP/SigLIP2 So400m/14, and SigLIP2 Giant-Opt (1B).

Key Experimental Results

Main Results

The models are evaluated across standard text-image retrieval benchmarks (COCO-5K, Flickr-30K), long-form retrieval (Urban-1K), diagnostic reasoning retrieval (RCLIP-V3-5K), and diverse compositional reasoning benchmarks.

Model & Scale Pretraining Data COCO-5K (I→T / T→I R@1) Flickr-30K (I→T / T→I R@1) Urban-1K (I→T / T→I R@1) RCLIP-V3-5K (I→T / T→I R@1)
ViT-Base (86M)
OpenCLIP [34] 0.4B 52.3 / 34.2 39.7 / 24.0 55.8 / 55.4 54.8 / 27.8
MetaCLIP [97] 0.4B 51.8 / 35.9 39.3 / 25.7 57.2 / 52.6 56.5 / 30.1
DataComp [26] 1.4B 53.4 / 37.2 39.0 / 24.6 64.4 / 59.9 60.8 / 32.2
Vanilla CLIP [69] 0.4B 50.0 / 30.4 40.7 / 21.8 61.0 / 46.8 51.0 / 26.7
+ Stage 1 +42M 56.2 / 37.9 42.5 / 29.1 70.4 / 68.6 56.0 / 30.4
+ Stage 2 (ReasonCLIP) +16M 52.3 / 37.0 41.9 / 27.5 59.2 / 60.4 55.3 / 33.8
ViT-Large (307M, 336px)
ViTamin [10] 1.0B 64.3 / 47.3 56.1 / 39.9 80.6 / 76.0 69.7 / 39.8
EVA-CLIP-02 [79] 2.0B 64.2 / 47.9 59.7 / 43.4 77.1 / 70.7 67.9 / 39.1
SigLIP2 [106] 10.0B 70.9 / 55.5 68.6 / 51.2 66.9 / 61.9 66.5 / 41.8
Vanilla CLIP [69] 0.4B 58.0 / 37.0 51.4 / 31.6 73.0 / 57.0 55.9 / 33.2
+ Stage 1 +42M 65.1 / 46.9 61.0 / 41.9 83.0 / 81.9 67.2 / 38.5
+ Stage 2 (ReasonCLIP) +16M 61.3 / 47.1 57.2 / 40.8 75.5 / 78.2 68.0 / 42.8
ViT-So400m (428M)
Vanilla SigLIP [106] 10.0B 72.6 / 54.3 69.9 / 51.3 74.5 / 73.4 70.5 / 43.4
+ Stage 1 +42M 73.6 / 56.5 69.3 / 52.3 77.5 / 82.3 68.3 / 45.5
+ Stage 2 (ReasonSigLIP) +16M 73.7 / 57.3 73.3 / 53.9 78.6 / 80.6 72.4 / 49.5

On fine-grained compositional benchmarks (WhatsUp, VALSE, CREPE, SugarCREPE, SugarCrepe++), ReasonCLIP achieves consistent gains across backbones: on CLIP-B/32, average compositionality increases from 52.1 to 57.7 (+5.6 points), with WhatsUp jumping by +10.1 points and SugarCrepe++ TOT surging by +14.8 points. On SigLIP-So400m, SugarCrepe++ TOT gains +21.7 points (51.2 to 72.9). When deployed as a frozen visual encoder in LLaVA-NeXT (Qwen3-1.7B), ReasonCLIP improves OKVQA from 34.0 to 35.8 and MMVP from 58.7 to 62.3, confirming positive transfer to downstream multimodal generation without hurting standard benchmarks like SciQA.

Ablation Study

The ablation investigates the impact of training stages and hyperparameters on standard retrieval and fine-grained reasoning diagnostic performance on RCLIP-Bench (V1: Visual Grounding, V2: Evidence Awareness, V3: Visual Reasoning).

Configuration / Variant COCO Retrieval (R@1) Flickr Retrieval (R@1) RCLIP-V1 (Grounding) RCLIP-V2 (Evidence) RCLIP-V3 (Reasoning) Note
Stage 1 Default 55.6 50.5 26.6 19.2 24.3 Dual alignment + dynamic decay + L2 reg
- Descriptive-heavy (Des.-heavy) 55.8 48.6 26.1 18.4 24.1 Higher weight on descriptive branch hurts reasoning
- Reasoning-heavy (Rea.-heavy) 52.5 46.0 23.5 15.3 19.0 Overweighting reasoning induces feature drift
- Exponential schedule (Exp-Sched) 53.2 47.7 25.8 17.6 23.3 Non-linear decay transition is suboptimal
Stage 2 Default 53.2 47.3 28.6 18.3 23.1 Category alignment + dual classification heads
- w/o classification (w/o cls) 53.3 47.3 28.6 18.0 22.6 Removing classification heads hurts reasoning separation
- With L2 reg (w/ L2) 53.1 47.1 28.4 17.3 23.0 Parameter anchoring restricts category feature formation
- Single-label classification 52.8 46.5 28.1 17.5 22.0 Single label cannot capture multi-property visual scenes

Key Findings

  • Stage Specialization and Capacity Synergy: Stage 1 simultaneously bolsters descriptive retrieval and activates reasoning sensitivity via dual supervision, whereas Stage 2 organizes fine-grained feature boundaries across categories. For smaller backbones (CLIP-B), heavy explicit supervision in Stage 2 slightly taxes descriptive metrics, whereas for larger models (SigLIP-So400m), Stage 2 delivers substantial gains across both retrieval and reasoning.
  • Criticality of Grounded Visual Anchor: Neglecting factual grounding (Rea.-heavy) precipitates severe representation collapse, dropping V3 reasoning performance from 24.3 down to 19.0, demonstrating that valid commonsense reasoning cannot emerge in abstraction from precise visual facts.
  • Multilabel Supervision Mirrors Physical Reality: Visual scenes naturally contain overlapping physical dimensions (e.g., motion interacting with gravity and spatial relations). Constraining the image head to single-label classification degrades V3 by 1.1 points, demonstrating the necessity of multilabel formulation.

Highlights & Insights

  • Progressive Representation Evolution: By decoupling pretraining into smooth transition (Stage 1) and explicit categorization (Stage 2), this work proves that CLIP's latent space is not restricted to superficial descriptive alignment, but can accommodate structured reasoning dimensions without architectural modifications.
  • Evidence-Clustered Data Synthesis Methodology: The proposed pipeline of partitioning visual evidence before prompting commonsense inferences prevents LLM hallucination and supplies high-fidelity contrastive supervision.
  • Three-Tier Diagnostic RCLIP-Bench: Disentangling failures into perceptual error (V1), ungrounded inference over faulty facts (V2), and invalid reasoning over correct visual facts (V3) provides an actionable, fine-grained benchmark for evaluating visual reasoning models.

Limitations & Future Work

  • Training Scale Constraints: Pretraining on 58.9M samples is modest compared to billion-scale commercial foundations; scaling reasoning supervision remains a vital next step.
  • Discriminative vs. Formal Generative Logic: Contrastive objectives excel at single-step intuitive commonsense judgment, but show limitations when confronted with multi-step formal symbolic calculations or complex algorithmic puzzles.
  • Future Trajectories: Extending visually grounded reasoning representations to dense spatial-temporal tasks, such as open-world 3D embodied manipulation and generative world models, represents a promising horizon.
  • vs MetaCLIP / DataComp: While large-scale data filtering recipes scale descriptive coverage, they remain anchored to shallow semantic co-occurrence. ReasonCLIP demonstrates that targeted reasoning supervision yields higher compositional gains on equivalent or smaller pretraining budgets.
  • vs READ-CLIP / Triplet-CLIP: Prior compositional methods primarily rely on task-specific hard negative mining on small benchmarks like MS-COCO; ReasonCLIP functions at the foundation pretraining level and can serve as a superior base to stack further compositional tuning gains.
  • vs DANCE: DANCE injects external knowledge graphs via riddles, risking domain bias and complex pipeline construction. ReasonCLIP derives reasoning directly from intrinsic visual evidence, ensuring visual fidelity.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering large-scale, visually grounded commonsense reasoning continual pretraining for CLIP-style encoders.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation spanning Base to 1B parameters across retrieval, compositionality, diagnostic tiers, and downstream MLLMs.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, crisp figures, clean mathematical formulation, and transparent analysis.
  • Value: ⭐⭐⭐⭐⭐ Provides a viable, non-intrusive paradigm for upgrading multimodal vision backbones into reasoning-aware foundational representations.