ECCV 2026 Multimodal VLM Contrastive Language-Image Pretraining Mamba / State Space Model Vision-Language Model Native Variable Resolution Out-of-Distribution Robustness

CLIMP: Contrastive Language-Image Mamba Pretraining¶

Conference: ECCV 2026
Paper: ECCV Page
Code: https://github.com/NimrodShabtay/CLIMP
Area: Multimodal VLM
Keywords: Contrastive Language-Image Pretraining, Mamba / State Space Model, Vision-Language Model, Native Variable Resolution, Out-of-Distribution Robustness

TL;DR¶

CLIMP replaces both towers of CLIP with state space models — VMamba with its SS2D four-direction cross-scan for vision and a Mamba-1/Mamba-2 language model with last-token pooling for text — and trains them on CC12M with CLIP's original symmetric contrastive objective, trading quadratic complexity for linear cost while gaining better zero-shot retrieval, stronger out-of-distribution robustness (surpassing CLIP-ViT-B/16 trained on 167× more data on ImageNet-O), and native support for arbitrary resolutions and arbitrarily long text.

Background & Motivation¶

CLIP learns transferable visual representations by mapping images and text into a shared embedding space under natural-language supervision, which is what enables zero-shot transfer. Nearly every improvement since — OpenCLIP, SigLIP, EVA-CLIP, MetaCLIP, SigLIP 2 — iterates inside the same skeleton: the vision side is always a Vision Transformer (ViT). Global self-attention lets any two patches interact directly, but its cost grows quadratically with sequence length, so high-resolution inputs quickly become unaffordable. Making one ViT handle several resolutions additionally requires positional-encoding interpolation (RoPE-ViT) or a dedicated variable-resolution training scheme (FlexViT, NaFlex). Worse, pairwise token interaction readily latches onto spurious correlations in the training data, and prior work has shown that CLIP's robustness is markedly overestimated on datasets explicitly designed to probe spurious features.

The other line of progress is state space models. Mamba makes the SSM parameters input-dependent through a selection mechanism, keeping complexity linear in sequence length; visual adaptations such as Vision Mamba, SiMBA, and VMamba have already shown this works for classification, and SSM inductive biases differ from attention — they favor smoothness and locality, which helps sample efficiency and out-of-distribution generalization. Yet the existing work, CLIP-Mamba, only mixes Mamba and Transformer modules inside individual towers and never tests what happens when both towers are SSM. Meanwhile CLIP's text tower remains capped at 77 tokens, which directly blocks long captions in dense-captioning retrieval. This leaves a clear gap: if both towers become SSM, can a single model obtain linear complexity, the robustness that comes with spatial inductive bias, and the ability to process text of arbitrary length?

The approach here is to swap out both of CLIP's encoders and change nothing else: VMamba on the vision side, unfolding the 2D image into a sequence via the SS2D four-direction cross-scan; a pretrained Mamba language model on the text side, taking the hidden state of the last non-padding token as the sentence representation; and CLIP's original symmetric contrastive loss as the alignment objective. The payoff is not merely cheaper compute — the scan-based receptive field dilutes scattered spurious cues, and the model no longer depends on explicit positional encodings, yielding tighter cross-modal alignment and lower hubness (the tendency for a few embeddings to dominate nearest-neighbor lists), which in turn improves retrieval and out-of-distribution robustness. Core idea: replace CLIP's vision tower with VMamba's SS2D cross-scan and its text tower with an autoregressive Mamba LLM using last-token pooling, so that a fully SSM dual encoder delivers linear complexity, native variable resolution, arbitrarily long text, and stronger out-of-distribution robustness at once.

Method¶

Overall Architecture¶

CLIMP keeps CLIP's dual-encoder structure: images and text each pass through one encoder into a shared embedding space and are aligned contrastively within the batch. The difference is that both towers are state space models, so memory and compute scale linearly rather than quadratically with sequence length in both modalities. On the vision side, an \(H\times W\) image is split into non-overlapping \(P\times P\) patches and fed through VMamba's VSS blocks; hierarchical downsampling progressively reduces the feature-map resolution, and a learned projection \(W_v\) maps the final features into the shared space. On the text side, a token sequence is processed by a pretrained Mamba language model (Mamba-1 or Mamba-2) whose last non-padding hidden state is projected by \(W_t\) into the same space. The pipeline involves no positional-encoding interpolation and no task-specific training for resolution or text length — the most visible difference from FlexViT, NaFlex, and RoPE-ViT.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image I: trained at 224×224"] --> B["VMamba vision tower<br/>SS2D four-direction cross-scan"]
    B --> C["Native-resolution inference<br/>no positional-encoding interpolation"]
    C --> D["Vision embedding (Wv projection)"]
    E["Text T: arbitrary length"] --> F["Mamba text tower<br/>last non-padding token pooling"]
    F --> G["Text embedding (Wt projection)"]
    D --> H["Shared embedding space"]
    G --> H
    H --> I["In-batch symmetric contrastive loss"]

Key Designs¶

1. VMamba vision tower: SS2D four-direction cross-scan in place of pairwise token attention

ViT suffers from two problems: global attention blows up quadratically at high resolution, and pairwise interaction easily locks onto a single spurious token locally. VMamba's alternative treats the image as a sequence, but must first resolve the tension that images are inherently 2D and have no natural order. Its answer is SS2D (2D selective scan): the patch sequence is traversed along four paths (top-left→bottom-right, bottom-right→top-left, top-right→bottom-left, bottom-left→top-right), each path running Mamba's selective recurrence internally, and the four outputs are then fused. Every patch therefore gathers context from all four spatial directions, effectively establishing a global receptive field while each path still costs a linear amount. The underlying SSM recurrence is

\[h_t = \bar{A}h_{t-1} + \bar{B}x_t,\qquad y_t = Ch_t + Dx_t\]

where the latent state \(h_t\) acts as a memory of the whole input history, \(\bar{A}\) sets the forgetting rate, and \(\bar{B}\) and \(C\) control what is written into and read from that memory; Mamba-1 further makes \(B\), \(C\), and the step size \(\Delta\) functions of the input \(x_t\) for content-aware filtering. (⚠️ This equation is corrupted by OCR in the cached text; it is given here in the standard SSM discretized form — refer to the original paper.)

Why this bias helps contrastive learning: Mamba-2 constrains the transition matrix to \(\bar{A}=\lambda I\), so the influence of a state \(k\) steps in the past decays as \(\lambda^k\) and the model is pushed toward locality and smoothness. Mamba-1's selective mechanism has no such closed form, but the authors confirm empirically that both variants exhibit a strong spatial inductive bias (see the CIFAR-10 shuffle experiment below). In the CLIP setting, attention's failure mode is precisely to latch onto one spurious token locally, whereas scanning accumulates the whole spatial sequence into the latent state, so scattered spurious cues are diluted during accumulation — this is the mechanism behind the large ImageNet-O gain and the lower hubness. A second difference is where positional information lives: ViT relies on explicit positional encodings, VMamba carries it implicitly in the scan order, which leads directly to the third design.

2. Mamba text tower: last non-padding token pooling removes the fixed 77-token ceiling

CLIP's text tower is a 77-token Transformer, so long captions are simply truncated, and Mamba language models are causal (autoregressive) with no off-the-shelf bidirectional encoder available. Rather than reworking the backbone, the authors turn this property into a structural advantage: because of the causal recurrence, the hidden state at step \(t\) has only seen the first \(t\) tokens, so the hidden state of the last non-padding token is the only position that has seen the entire sentence. Taking that state as the text representation and projecting it with \(W_t\) is therefore not a workaround but the natural counterpart of CLIP's use of the [EOS] position. And since sequence position is carried by the recurrence rather than a fixed positional encoding, sequence length is bounded by no preset limit.

Whether this choice actually matters is verified by ablation: with the vision tower fixed, VMamba paired with Mamba-1 beats VMamba with LLaMA by +3.5% Acc@1 and +3.9% TR@5, while RoPE-ViT barely reacts to the text-tower swap (±1%). Matched SSM towers thus learn more compatible cross-modal representations. The authors also test Hydra, a bidirectional SSM variant, plus RoBERTa-L and BERT-L; the bidirectional SSM is worse, indicating that bidirectionality per se is not the point — the representation quality of a mature pretrained LLM is. Notably, no model in the dense-captioning evaluation is restricted to 77 tokens (LLaMA-3.2-1B itself has an 8K native context), so CLIMP's advantage there comes from representation quality rather than from competitors being truncated, a point the paper is careful to state.

3. Native-resolution inference: the scan carries position implicitly, so no interpolation and no specialized training

For a model trained at 224×224 to run directly at 896×896, ViT-based approaches must either interpolate positional encodings (RoPE-ViT) or design a variable-resolution architecture with a matching training scheme (FlexViT, NaFlex). VMamba's spatial position is given implicitly by the scan paths and is independent of the patch-grid size, so the same weights can be trained at 224 and run forward at 896 without architectural changes or retraining. The cost side follows directly: the patch count grows linearly with pixels, and while attention is quadratic in patch count, the SSM is linear, so the gap widens with resolution. At 896×896, CLIMP needs only 10.0 MB of resolution-specific memory (50.4 MB for ViT variants, roughly 1/5) and 259.7 GFLOPs (457.9 GFLOPs for ViT variants, roughly 1/1.8), with Figure 3 reporting a 4–57× memory advantage across resolutions. Notably, at high resolution CLIMP beats not only the interpolation-based RoPE-ViT but also FlexViT and NaFlex, which were purpose-built for variable resolution — suggesting that carrying position in the scan order is a cheaper route than patch-based training schemes.

Loss & Training¶

The training objective deliberately stays unchanged: CLIP's in-batch symmetric InfoNCE, with L2-normalized image and text embeddings, a learnable temperature, and all non-matching samples in a batch serving as negatives. CLIMP only swaps the backbones and leaves the loss alone, so any gain can only be attributed to the architecture rather than to the training recipe.

\[\mathcal{L}=-\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j}\exp(\mathrm{sim}(v_i,t_j)/\tau)}+\log\frac{\exp(\mathrm{sim}(t_i,v_i)/\tau)}{\sum_{j}\exp(\mathrm{sim}(t_i,v_j)/\tau)}\right]\]

(⚠️ The paper does not list this equation; it is given here in the symmetric InfoNCE form from the original CLIP paper — refer to the original paper.)

Training protocol: all models are trained on CC12M (about 12M image-text pairs) for 10 epochs at 224×224 with AdamW and a cosine learning-rate schedule, peak LR \(5\times10^{-5}\), batch size 2048, and projection dimension 768. Every vision tower is base-sized (about 86M parameters) and initialized from ImageNet-1K. CLIMP uses Mamba-1 (1.4B) and Mamba-2 (1.3B) as its two text-tower variants, while all Transformer baselines use LLaMA-3.2-1B. To separate architectural from data-scale effects, all models share the same data and protocol; additionally, a MetaCLIP-ViT-B/16 baseline whose vision tower was pretrained on MetaCLIP-400M (400× larger than ImageNet-1K) serves as a comparison point with vastly more data.

Key Experimental Results¶

Main Results¶

Averages on CLIP-Benchmark (28 classification datasets plus 3 retrieval datasets, 31 in total):

Vision Tower	Text Tower	Acc@1	Acc@5	IR@5	TR@5
FlexViT-B/16	LLaMA-3.2	26.3	57.4	62.4	72.3
NaFlex-B/16	LLaMA-3.2	26.1	56.6	61.5	73.3
ViT-B/16	LLaMA-3.2	27.3	56.4	62.8	72.9
RoPE-ViT-B/16	LLaMA-3.2	27.3	59.0	63.4	72.9
MetaCLIP-ViT-B/16†	LLaMA-3.2	25.7	56.9	66.4	76.2
CLIMP (VMamba-B)	Mamba-2	26.9	59.0	65.2	75.3
CLIMP (VMamba-B)	Mamba-1	29.6	58.5	65.5	77.0

†MetaCLIP-ViT's vision tower was pretrained on MetaCLIP-400M (400× larger than ImageNet-1K); despite that data advantage it still achieves lower classification accuracy than CLIMP.

Out-of-distribution robustness (five ImageNet variants, top-1/top-5):

Vision Tower	Text Tower	IN-V2	IN-R	IN-A	IN-O	Sketch	Avg
FlexViT-B/16	LLaMA-3.2	32.8/62.5	45.4/74.3	13.2/41.2	38.1/68.2	24.6/50.0	30.8/59.2
NaFlex-B/16	LLaMA-3.2	31.4/60.5	44.8/72.5	12.0/36.0	38.5/67.6	23.5/49.1	30.0/57.1
ViT-B/16	LLaMA-3.2	30.0/57.5	42.9/69.7	13.9/37.6	27.7/55.8	21.7/46.1	27.2/53.3
RoPE-ViT-B/16	LLaMA-3.2	34.4/65.4	47.8/76.0	16.3/46.3	40.1/69.9	27.4/53.1	33.2/62.1
CLIMP (VMamba-B)	Mamba-2	37.0/67.6	46.6/74.5	15.6/45.5	49.8/77.0	27.0/54.2	34.8/63.7
CLIMP (VMamba-B)	Mamba-1	37.5/68.4	46.2/74.5	15.5/45.3	48.1/78.8	27.5/54.4	35.2/64.3

Resolution extrapolation and dense-captioning retrieval (all models trained only at 224×224, no fine-tuning at test time):

Setting	Metric	CLIMP-Mamba-1	CLIMP-Mamba-2	Best Transformer Baseline
CLIP-Bench @224	Acc@1 / IR@5 / TR@5	29.6/65.5/77.0	26.9/65.2/75.3	27.3/63.4/72.9 (RoPE-ViT)
CLIP-Bench @384	Acc@1 / IR@5 / TR@5	27.3/65.0/75.2	24.9/65.2/74.6	25.9/63.6/75.0 (FlexViT)
NoCaps @896	IR@5 / TR@5	57.1/53.5	57.1/53.8	51.6/50.8 (NaFlex)
Crossmodal-3600 @896	IR@5 / TR@5	55.9/45.4	54.7/46.0	48.2/42.0 (NaFlex)
Flickr8k-Rephrased (avg 134 tokens, 98.3% over 77)	Image R@1 / Text R@1	67.0/81.3	63.3/75.7	60.7/77.6 (RoPE-ViT)
DOCCI (avg 142 tokens, 94.4% over 77, images at 896)	Image R@1 / Text R@1	37.3/23.8	33.4/24.6	29.1/26.1 (NaFlex)

Ablation Study¶

Swapping only the text tower while fixing the vision tower, to probe tower pairing (CLIP-Benchmark averages):

Vision Tower	Text Tower	Acc@1/Acc@5	IR@5/TR@5
RoPE-ViT	LLaMA-3.2	27.27/58.95	63.35/72.93
RoPE-ViT	Qwen2-1.5B	27.30/58.80	60.40/70.90
RoPE-ViT	Mamba-2	27.42/58.69	62.59/72.99
RoPE-ViT	Mamba-1	28.10/58.54	63.17/74.28
RoPE-ViT	RoBERTa-L	27.47/58.64	64.82/74.72
RoPE-ViT	BERT-L	27.80/58.84	63.93/74.15
VMamba	LLaMA-3.2	26.13/57.30	63.21/73.16
VMamba	Hydra (bidirectional SSM)	28.63/56.98	61.24/71.35
VMamba	Mamba-2	26.94/59.04	65.17/75.25
VMamba	Mamba-1	29.59/58.53	65.54/77.03

Embedding geometry analysis (measured on NoCaps; higher alignment is better, lower hubness is better):

Vision Tower	Text Tower	Alignment	Text Hubness	Image Hubness
FlexViT	LLaMA	1.111	1.23	1.19
NaFlex	LLaMA	1.039	1.24	0.93
RoPE-ViT	LLaMA	1.052	1.25	1.04
CLIMP (VMamba)	Mamba-1	0.982	1.15	1.01
CLIMP (VMamba)	Mamba-2	1.000	1.13	1.01

(⚠️ The arrow notation in the header of Table 8 is not clearly distinguishable in the cached text; this table follows the authors' prose statement that CLIMP attains the best alignment at 0.982 and the lowest text hubness at 1.13.)

Key Findings¶

Mamba-1 consistently beats Mamba-2 — speed has a price. Mamba-2 gains a 2–8× speedup by simplifying the transition matrix from a structured diagonal (HiPPO-initialized) to a scalar identity and by sharing weights across heads, but per-token expressivity drops accordingly; Mamba-1's per-channel selective scan, with \(\Delta\), \(B\), and \(C\) all input-dependent, filters more finely and yields more discriminative representations for contrastive alignment. This is also why VMamba+Mamba-1 is best across the board in the text-tower ablation.
Architectural inductive bias can outweigh data scale. On ImageNet-O, Mamba-2 reaches 49.8 and Mamba-1 48.1, i.e. +9.7 / +8.0 over the best Transformer baseline (RoPE-ViT at 40.1), and both surpass CLIP-ViT-B/16 (42.3) trained on LAION-2B, a dataset 167× larger than CC12M. The same logic repeats on CLIP-Benchmark: MetaCLIP-ViT's vision tower was pretrained on 400× more data yet posts the lowest classification accuracy at 25.7.
The failure modes are complementary, not one-sided. RoPE-ViT wins on ImageNet-R (47.8 vs 46.2) and ImageNet-A (16.3 vs 15.5), where CLIMP trails by 1–2 points; the authors acknowledge that VMamba's spatial features transfer less well under strong style shift than ViT's abstract attention features. CLIMP's gains concentrate on distribution shift (IN-V2, +3.1/+2.6), natural out-of-distribution data (IN-O), and Sketch.
The higher the resolution, the larger the advantage. At 896×896 CLIMP leads RoPE-ViT by 18–19 points on both image and text retrieval (NoCaps 57.1 vs 38.6 and 53.5 vs 34.9; Crossmodal-3600 55.9 vs 36.8 and 45.4 vs 26.9), and it surpasses FlexViT and NaFlex, which were designed specifically for variable resolution. The abstract's "up to 6.6% higher retrieval accuracy at 16× training resolution" corresponds to comparisons against NaFlex, the strongest variable-resolution baseline; against RoPE-ViT the gaps are far larger, so the magnitude of the claim depends heavily on which baseline is chosen.
Dense captioning: image retrieval leads, text retrieval does not necessarily. On the rephrased Flickr8k-test (98.3% of captions exceed 77 tokens) CLIMP-Mamba-1 beats the best baseline on image R@1 by 6.3, and on DOCCI by 8.2. But on DOCCI text retrieval NaFlex is best (26.1 vs 24.6/23.8 for CLIMP), which the authors attribute to its sequence packing design — removing the 77-token ceiling does not automatically win on the text side, and the paper states this honestly.
The spatial inductive bias has direct controlled evidence. On CIFAR-10, a 3-layer VMamba (0.33M) versus a 3-layer ViT (0.35M): with normal patch order VMamba achieves lower training loss (1.028 vs 1.121) and higher accuracy (69.33 vs 67.50); shuffling the patch order degrades VMamba sharply (loss 1.612, accuracy 46.44) while ViT stays relatively robust (1.328 / 59.72). The SSM encoder is thus genuinely exploiting sequential spatial structure rather than winning on parameter count.
Scaling curves are unsaturated. Across model sizes, CLIMP leads at every scale: 38.8 vs 32.5 (FlexViT) and 33.2 (RoPE-ViT) at 22–30M, and 43.5 vs 38.3 and 40.7 at 87M. Growing the data from 1M to 12M (Conceptual Captions) keeps improving with no visible inflection, supporting the claim that the advantages are architectural and should persist at larger scales.
Comparison against convolutional backbones. On the 38 OpenCLIP benchmarks CLIMP outperforms all ResNet-based CLIP variants by more than 4 points on average (details in the supplementary material), indicating the gains are not merely "slightly better than ViT."

Highlights & Insights¶

Aligning the pooling position with causality: in an autoregressive model the last token is the only position with full context, so last-token pooling is not a compromise but the choice best matched to the causal structure — and it happens to remove the 77-token ceiling. The reasoning pattern of "translating an architectural constraint into a design advantage" is worth reusing.
Matched architectures matter: RoPE-ViT barely reacts to the text tower (±1%), whereas VMamba gains +3.5% Acc@1 and +3.9% TR@5 from a Mamba text tower. This is directly useful for cross-modal architecture search: matching the family of the vision and text towers may pay off more than strengthening either tower alone, and the effect only becomes visible through a cross ablation like the 2D grid in the text-tower table.
Replacing positional encoding with scan order: turning "position" from an explicitly interpolable parameter into a structural prior gives resolution extrapolation for free. The idea transfers to any vision task that needs variable resolution (high-resolution detection, remote sensing, pathology slides) as long as position is genuinely carried by structure rather than by encodings.
Hubness as a retrieval diagnostic: retrieval quality depends not only on alignment but also on whether a few "hub" vectors monopolize nearest-neighbor lists. Using alignment / uniformity / hubness as a trio to explain why retrieval improves, and linking low text hubness to a weaker reliance on spuriously correlated features, is a diagnostic package that ports to any retrieval system.
The controlled experimental design is itself a highlight: every model shares the same data and protocol, and a MetaCLIP-ViT with 400× more data is added as a reference point, turning the usual "architecture or data?" argument into something measurable. The authors also devote a full subsection to where Transformers remain better, which is rare in a paper with such a clear thesis.

Limitations & Future Work¶

Modest scale, so the extrapolation is not empirically demonstrated. Training uses only CC12M (about 12M pairs), the vision tower is base-sized at roughly 86M parameters initialized from ImageNet-1K, and the text tower is an off-the-shelf LLM without large-scale joint pretraining. The unsaturated scaling curves argue that the advantages persist at larger scales, but that remains extrapolation rather than evidence.
Averages hide per-dataset differences. The CLIP-Benchmark average masks the losses on ImageNet-R and ImageNet-A, and the abstract's "up to 6.6%" is on a different footing from the +18–19% over RoPE-ViT in the main-text table; readers need to check which baseline a claim refers to.
The scan paths are an unablated hyper-parameter. SS2D uses four traversal paths, yet neither the number of paths nor the choice of directions is studied, and this is precisely the component that governs the implicit positional information.
The robustness causality is only correlational. The paper attributes the OOD improvement to low hubness leading to less reliance on spurious features, but currently only shows a correlation between geometric metrics and accuracy, with no interventional experiment manipulating spurious features.
Efficiency is reported only as FLOPs and memory, not latency. Real GPU speed at linear complexity depends on the quality of the scan kernel implementation (Mamba-2's 2–8× speedup is largely a training-time effect), and actual deployment throughput is not measured.
Improvement directions: scale up the vision tower and train from scratch, verify whether cross-modal family matching still holds under different frozen/unfrozen combinations, ablate the scan directions and path count, and extend to generative vision-language tasks and hybrid architectures (which the authors also list as future work).

vs CLIP / OpenCLIP: this paper keeps the identical dual-encoder contrastive framework and loss, swapping only the backbones. The difference is that it shows a backbone swap alone improves retrieval, robustness, resolution, and long-text handling simultaneously, without touching the loss or the data; the downside is a much smaller scale, so absolute accuracy is not directly comparable.
vs SigLIP / EVA-CLIP / MetaCLIP: those works improve data curation, the loss form (pairwise sigmoid), the training recipe, and the encoders, but keep a ViT vision tower. MetaCLIP here serves as the "data dividend" reference point, and the two conclusions are complementary rather than opposed — data dividends and architectural inductive bias can stack.
vs CLIP-Mamba: it mixes Mamba and Transformer modules inside individual towers and does not cover retrieval, robustness, resolution, or dense captioning evaluation; CLIMP is a fully SSM dual encoder with the full evaluation, which is the basis for its "first" claim.
vs LLM2CLIP / UniViTAR: they replace CLIP's text tower with a Transformer LLM (plus a lightweight adaptor) while keeping a ViT vision tower; CLIMP keeps both modalities linear at the cost of not adapting the text tower with dedicated bidirectional pretraining.
vs Vim / SiMBA / VMamba: the first two validate visual SSMs on classification and VMamba introduces SS2D; CLIMP is the first to systematically place VMamba inside the CLIP framework and additionally uncovers the gain from matching the vision and text families.
vs FlexViT / NaFlex / RoPE-ViT: they accommodate high resolution through positional-encoding interpolation or dedicated variable-resolution training; CLIMP carries position implicitly in the scan order, using neither interpolation nor extra training, and still surpasses these purpose-built schemes at high resolution.

Rating¶

Novelty: ⭐⭐⭐⭐ First fully SSM dual-encoder CLIP model, but mainly a correct combination of existing components (VMamba + Mamba LLM + contrastive loss), with no new loss or training mechanism.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers retrieval, robustness, resolution, dense captioning, geometry analysis, efficiency, model/data scaling, and a controlled spatial-bias experiment, with same-data same-protocol cross ablations and a data-scale reference point.
Writing Quality: ⭐⭐⭐⭐ Claims map cleanly onto evidence and a dedicated subsection on where Transformers remain better shows unusual restraint; the downside is that the contrastive loss and pooling equations are absent from the main text.
Value: ⭐⭐⭐⭐ Gives a positive answer to whether SSMs can replace Transformers for vision-language pretraining and offers two practical paths (variable resolution and long text); the deployment payoff still needs latency measurements to confirm.