On the Reliability of Cue Conflict and Beyond¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: Interpretability
Keywords: shape-texture bias; cue-conflict benchmark; benchmark reliability; ranking-based metric; sensitivity vs. preference

TL;DR¶

This paper systematically questions the reliability of the widely used cue-conflict benchmark on both the stimulus-construction and the measurement side, identifying four problems — stylization entangles the cues, cue informativeness is imbalanced, ratio-based bias hides absolute cue sensitivity, and a restricted label space distorts predictions — and offers REFINED-BIAS as a replacement, built from human-perceptual cue definitions, controlled purified cue generation, and a full-label-space MRR sensitivity metric; on 32 training-strategy models, multiple architecture families, OpenCLIP/DINOv2 large models, and multi-level quantized models it resolves contradictory conclusions that the original benchmark could not explain.

Background & Motivation¶

Understanding which visual cues a neural network actually relies on is one of the most human-readable windows into its decision process. On this line, the cue-conflict benchmark of Geirhos et al. (2018) is close to a de facto standard: it uses image stylization to put the shape of class A and the texture of class B into the same image, defines "shape preference / texture preference" from the model's choice under the conflicting cues, and derives a highly influential conclusion — a human-like strong shape bias is associated with higher in-domain accuracy. A large body of later work builds on it to relate shape bias to generalization and robustness, and it is still cited continuously. Yet the benchmark was originally adopted as a pragmatic compromise: more purified stimuli (such as sketches) introduced a large domain shift, so stylization became an expedient that preserved natural image statistics while still inducing cue conflict — it was designed to be feasible, not to be reliable.

This paper starts from that tacit premise. The authors find that the stylization-based instantiation yields empirically unstable images: in many of them even humans cannot say clearly what the shape and the texture are, and machines struggle just as much. Shape (content features) and texture (Gram-matrix style statistics) are defined by the internal features of the stylization model, so the two are not cleanly separated, and nothing in the pipeline controls the relative informativeness of the two cues. This directly explains the conflicting conclusions in recent literature — using the same cue-conflict benchmark, one line of work reports that stronger texture preference yields better performance, another that stronger shape preference is the key, and yet another that ImageNet-pretrained ResNet-50 does not actually rely on texture very much. The measurement itself is just as problematic: bias is written as a ratio between correct shape and correct texture predictions, a quantity that is entirely blind to absolute sensitivity (models at 8% vs. 2% and at 80% vs. 20% receive the same score), and evaluation further restricts predictions to a handful of preselected candidate classes. The paper gives a vivid illustration: the model's true ranking is rabbit, cat, dog while the ground-truth label is cat — in the full decision space the model is wrong, but once the candidate set is trimmed to {cat, dog}, cat jumps to first place and the model "appears" to have used the cue correctly, systematically overestimating cue usage.

The paper therefore turns reliability into three testable questions: do cue-conflict images really instantiate perceptually valid and separable shape/texture cues? Does the mixed-cue construction guarantee that the two cues are comparable in informativeness and thus fair? Are both cues reliably recognizable by humans and models alike? The authors stress that earlier critiques (Tartaglini et al. filling shape silhouettes with textures, Burgert et al.'s controlled suppression, Wen et al.'s global shape degradation, Doshi et al.'s texture-suppressed shape cues) each addressed only one facet; none revisited how cues are built, how predictions are evaluated, and how cue-specific sensitivity should be measured in a full-label setting within one framework. This paper's angle is a unified re-examination: define and generate cues by human perceptual standards rather than model heuristics, and report both preference and absolute sensitivity through a ranking-based metric over the complete decision space. Core idea: replace "stylization-built conflict images + ratio-based accuracy bias" with "a perceptual-definition-driven purified cue dataset + a full-label-space ranking-based sensitivity metric," so that measured preference is no longer confounded by cue validity, cue balance, and recognizability artifacts.

Method¶

Overall Architecture¶

REFINED-BIAS is an integrated dataset-plus-metric framework whose goal is to repair shape-texture bias diagnosis all the way from "preference" to "preference plus sensitivity." It has three parts: a cue set and balanced dataset defined by human perceptual criteria (fixing cue purity and cue balance), a controlled purified cue generation and human per-image verification pipeline (fixing recognizability), and a ranking-based sensitivity metric computed over the model's complete output space (fixing both the ratio metric that hides sensitivity and the restricted labels that distort predictions). At evaluation time, shape cues and texture cues are fed to the model separately, full logits are retrieved for each, a sensitivity value is computed for each cue, and the two are normalized into a preference — this is what finally separates "how much cue a model can use" from "which cue it leans toward" into two independently comparable quantities.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["20 ImageNet-derived superclasses<br/>10 shape-dominant / 10 texture-dominant"] --> B["Rebuild cues and dataset from human perception<br/>300 source images per class, 6,000 total"]
    B --> C["Controlled purified generation + human recognizability check<br/>separate shape and texture branches"]
    C --> D["MRR sensitivity metric over the full label space<br/>Shape-Sens / Texture-Sens"]
    D --> E["Preference = normalized sensitivity<br/>fair cross-model comparison"]

Key Designs¶

1. Define shape and texture by human perception, and rebuild a balanced cue dataset on that basis

Stylization treats shape as "content features" and texture as "style statistics" — a definition that comes from the internal representation of a particular model rather than from human perception, so the resulting "shape" may retain only local edges while the "texture" drags the object's outline along with it. This paper instead adopts operational human-perceptual definitions: texture is a pattern that repeats consistently within image patches of various sizes (for example honeycomb or dishrag, categories that remain recognizable even when cut into small patches), while shape is a non-repeating geometric structure, explicitly split into global geometry (overall silhouette) and local geometry (distinctive substructures not repeated across the object). The distinction is not a semantic nicety: ipod and comic book share the same rectangular global geometry and can only be separated by local structure, and cue-conflict's content features correlate only with local edges rather than tracking holistic shape.

Building on these definitions, the authors select 20 ImageNet-derived superclasses — 10 shape-dominant (e.g., clock, hourglass) and 10 texture-dominant (e.g., strawberry, brain coral) — chosen on the basis of human perceptual judgments, and additionally require that either shape or texture be the most discriminative feature of the class; this requirement also mitigates the domain shift suffered by CNNs. 300 images are collected per class (10 from ImageNet, the rest from the web), 6,000 in total, roughly five times larger than cue-conflict. This matters: cue-conflict uses only 10 shape and 3 texture source images per class, 1,280 conflict images in total, selected as a subset from 7,680 (160 × 48) candidate pairs with some sources reused more often than others, so image-level variation (such as resolution differences that make a pattern look coarser or finer) directly contaminates preference measurement. Enlarging and balancing the source pool ensures that category differences, not the appearance of a few source images, drive the model's choice.

2. Controlled purified cue generation with human recognizability verification

Definitions alone do not guarantee that generated images actually carry the two kinds of information. The authors build on the cue-separation idea of Mohla et al. but deliberately do not replicate their protocol, because applying it directly leaves data-quality issues unresolved: prior texture cues show local and global shape leakage and reduced resolution, while shape cues are drowned in cluttered backgrounds that damage the object's structural integrity; the stylization approach of Geirhos et al. simply inherits these inherent defects.

The paper consequently redesigns a precise and carefully controlled generation pipeline in which shape and texture cues are rendered cleanly at full resolution (the shape cue presenting the target's structural skeleton, the texture cue presenting a multi-scale consistent repeating pattern), backed by human inspection of every generated cue image. The effect of this pipeline is directly quantifiable: averaged over all ImageNet-1k pretrained CNNs, REFINED-BIAS reaches 46% top-1 for shape and 63% for texture, whereas cue-conflict reaches only 4% (shape) and 21% (texture) under full predictions, and only 20% and 30% even if its habitual restricted-label predictions are counted. In other words, the former lets CNNs actually "see" the cues, while the latter largely measures domain shift. A parallel human study checks recognizability: 88 participants each completed 100 classification tasks, evenly split between shape and texture. Measured by Fleiss' κ for inter-rater agreement, REFINED-BIAS reaches κ = 0.98 for shape (near-perfect) and 0.80 for texture (substantial), whereas cue-conflict reaches only 0.73 for shape and drops to 0.29 for texture — its texture cues are inherently ambiguous to human observers. Using "can humans consistently name this cue" as the quality criterion turns the benchmark itself into a subject of testing, which is what most distinguishes this work from yet another "new benchmark" paper.

3. An MRR sensitivity metric over the full label space that separates preference from absolute sensitivity

The measurement has two sources of trouble. First, conventional bias scores rest directly on the accuracy of cue recognition, so hard or long-tailed classes inflate or deflate the overall number. Second, ratios such as ns/(nt+ns) put accuracy into both the numerator and the denominator, so if one cue's sensitivity collapses overall, the ratio can actually rise — "shape bias grew" may mean only that "texture sensitivity fell further." The two fixes map onto these one-to-one: replace accuracy with a ranking-based metric, and use it only in the denominator.

Concretely, the model outputs full logits for each cue image, these are ranked, and the rank of the correct cue label is used to average its reciprocal, yielding two sensitivities:

\[\text{Shape-Sens}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{r_{\text{shape},i}},\qquad \text{Texture-Sens}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{r_{\text{texture},i}}\]

where \(N\) is the number of samples and \(r_{\text{shape},i}\) and \(r_{\text{texture},i}\) are the ranks of the correct shape/texture label for sample \(i\) in the model's full ranked predictions. Preference is obtained by normalizing the two sensitivities:

\[\text{Shape preference}=\frac{\text{Shape-Sens}}{\text{Shape-Sens}+\text{Texture-Sens}}\]

The ranking-based form (here Mean Reciprocal Rank) is the key: accuracy assigns zero to both the 2nd and the 100th place, whereas MRR gives 1/2 versus 1/100, measuring how close the model came to being right — exactly the granularity cue utilization deserves. Because the rank is taken over the full logits, the distortion described earlier — trimming the candidate classes promotes cat from rank 2 to rank 1 and manufactures a correct cue recognition out of nothing — is eliminated at the root; and with sensitivity appearing only in the denominator, comparing absolute magnitudes across models becomes meaningful for the first time.

A Worked Example¶

A walkthrough using the illustration in the paper: the model produces full rankings for three cue images. On the book image the top three are book, clock, ipod, and the correct label book ranks 1st, contributing \(1/1=1.0\); on the clock image the ranking is book, clock, ipod, tiger, zebra, and the correct label clock ranks 2nd, contributing \(1/2=0.5\); on the ipod image the ranking is book, clock, ipod, tiger, zebra, and the correct label ipod ranks 3rd, contributing \(1/3\approx 0.33\). The MRR over the three images is \((1.0+0.5+0.33)/3\approx 0.61\) ⚠️ the summary value is printed in the original figure in decimal form; the figure above is recomputed from the formula.

Against this, the distortion of the conventional practice is plain: if a model's true ranking is rabbit, cat, dog while the ground-truth label is cat, it is wrong in the full space; but when evaluation keeps only the two candidates {cat, dog}, cat becomes rank 1 and the model is recorded as having correctly recognized the cue. That is precisely why this paper takes ranks over the full label space.

Key Experimental Results¶

The experiments are organized around two goals: first validate whether REFINED-BIAS itself is correct (does it stably reproduce known training-strategy effects), then use it to see how bias varies with architecture, scale, and quantization.

Main Results¶

Table 1: Direct comparison of cue quality (human agreement via Fleiss' κ; recognition is average top-1 over ImageNet-1k pretrained CNNs)

Benchmark	Shape κ	Texture κ	Shape top-1	Texture top-1
cue-conflict	0.7276	0.2937	4% (full labels) / 20% (restricted labels)	21% (full labels) / 30% (restricted labels)
REFINED-BIAS	0.9836	0.7973	46%	63%

Table 2: Effect of training strategies on preference (t-test against the "random cropping only" baseline, α = 0.05)

Strategy family	Expected direction	cue-conflict p	REFINED-BIAS p	Note
Mixed Aug	shape	2.72e-04	0.002	both detect it; ours also reflects mild texture degradation
Texture Dist	shape	0.003	0.010	consistent
Shape Aug	shape	0.181	0.018	cue-conflict not significant
Contrastive	shape	0.684	0.009	cue-conflict fails entirely
Adversarial	neither	5.19e-05	0.489	cue-conflict falsely reports a stronger shape preference than shape-focused methods

The setup uses 32 ImageNet-1k pretrained models, all based on the same ResNet-50 architecture: the baseline uses random cropping only, and each remaining model adds exactly one training strategy on top (3 shape augmentation, 3 contrastive learning, 5 texture distortion, 8 mixed augmentation, 12 adversarial training). Fixing the architecture while varying only the strategy isolates inductive bias as a confounding factor.

Ablation Study¶

Table 3: Correlation of preference / sensitivity with ImageNet-1k top-1 accuracy (Pearson r, all p < 0.05)

Benchmark and metric	Fixed architecture, varying strategy	Fixed strategy, varying architecture
cue-conflict preference	r = −0.927 (higher texture preference, better accuracy)	r = 0.607 (higher shape preference, better accuracy)
REFINED-BIAS preference	r = 0.549	r = 0.897
REFINED-BIAS sensitivity	shape 0.770 / texture 0.551	shape 0.939 / texture −0.473

Table 4: Validation on large-scale and quantized models

Setting	Model	Metric	cue-conflict	REFINED-BIAS
Large-model preference (t-test against the 0.5 no-preference baseline)	OpenCLIP ViT-B/32	p	0.276 (not significant)	0.002
	OpenCLIP ViT-B/16	p	0.037	0.002
	OpenCLIP ViT-L/16	p	0.565 (not significant)	0.001
	OpenCLIP ViT-S/14	p	0.030	3.18e-04
	DINOv2 ViT-B/14	p	0.011	0.001
	another DINOv2 variant ⚠️ this row's model name is jumbled with the previous one in the original typesetting	p	5.26e-21	3.71e-04
Quantization (ResNet18/34/50/101/152, FP32 → FQ2)	five ResNets	Kendall ρ of ImageNet top-1	— (the benchmark itself) −0.696	—
		ρ of preference	0.644 (shape preference rises as quantization strengthens)	−0.562 (shape preference falls)
		ρ of sensitivity	—	shape −0.636 / texture −0.584 (both fall)

Key Findings¶

The most common failure of the original benchmark is "what should be significant is not." Shape augmentation — a strategy explicitly aimed at shape-based recognition — and contrastive learning are both judged ineffective by cue-conflict (p = 0.181 and p = 0.684), while REFINED-BIAS brings them to 0.018 and 0.009. This is not an incidental difference in statistical power: when the model cannot recognize the cue in the first place, the strategy's effect has no way to show up.
Adversarial training is the litmus test for benchmark reliability. cue-conflict reports that adversarial training produces a stronger shape-preference increase than shape-focused methods (p = 5.19e-05), yet adversarial training targets imperceptible perturbations and has nothing to do with shape perception, so the conclusion is implausible on its face; on this paper's dataset the change is not significant (p = 0.489), though the preference value does sit on the texture side. Moreover, the baseline ResNet-50 shows a texture preference of 0.49 here whereas cue-conflict reports 0.77 — contradicting the recent finding that ImageNet-pretrained ResNet-50 does not strongly rely on texture.
The sensitivity metric reveals model differences the preference metric cannot see. Under the preference metric, adversarial training looks like the extreme user of texture or of shape, depending on which dataset is used; the sensitivity curves show that adversarial training actually raises neither cue's utilization, and that what genuinely uses both shape and texture is mixed augmentation. Across models, both sensitivities rise with in-domain accuracy (fixed architecture, varying strategy: shape r = 0.770, texture r = 0.551), consistent with the established conclusion that shape and texture cues are complementary and jointly improve performance.
The existing contradiction is resolved. cue-conflict gives opposite directions in its two settings — fixed architecture with varying strategy favors texture preference (r = −0.927), while fixed strategy with varying architecture favors shape preference (r = 0.607) — leaving "which one actually drives performance" unanswerable; REFINED-BIAS points the same way in both settings (r = 0.549 and 0.897), stably identifying shape preference as the more important performance contributor.
Architecture conclusions also become clearer. On the sensitivity axis, Swin and CMT show higher shape sensitivity than the original ViT (architecture axis r = 0.939), indicating that local-to-global feature aggregation genuinely improves shape perception, whereas the cue-conflict preference metric does not show this advantage (CMT barely changes and Swin's shape preference actually drops). The same holds for large models: OpenCLIP variants fail to stably exhibit shape preference on cue-conflict (ViT-B/32 p = 0.276, ViT-L/16 p = 0.565) but are all significant on REFINED-BIAS; under linear probing both shape and texture sensitivity correlate with accuracy (OpenCLIP r = 0.987 / 0.961, DINOv2 r = 0.879 / 0.991).
Quantization is the setting that backs the conclusion into a corner. As precision drops from FP32 to FQ2, cue-conflict reports shape preference increasing (ρ = 0.644), i.e., "the model relies on shape more" — yet in-domain accuracy falls over the same range (ρ = −0.696), which contradicts its own central claim that a stronger shape bias accompanies better recognition performance. REFINED-BIAS instead shows shape and texture sensitivity falling together (ρ = −0.636 / −0.584) with shape preference also falling (ρ = −0.562), matching the direction of the accuracy drop and indicating that it tracks an overall degradation of visual cue perception.

Highlights & Insights¶

Treating the benchmark itself as a subject. Using Fleiss' κ to measure "can humans consistently say what this cue is" turns "is the cue recognizable" from a qualitative complaint into a testable experimental quantity (texture κ rises from 0.29 to 0.80). The recipe transfers to any benchmark that depends on stimuli conveying the intended information — perturbation strength in robustness benchmarks, prompt legibility in hallucination benchmarks.
"Denominator only" is a general-purpose metric repair. Having spotted that accuracy sits in both numerator and denominator and distorts the ratio, the paper makes the minimal change: switch to a ranking-based quantity and keep it in the denominator alone. Any ratio-style metric (class share, bias ratio, fairness quotient) can be run through the same self-check at almost no cost.
Using MRR rather than accuracy to measure cue utilization. Accuracy treats the 2nd and the 100th place identically, while MRR keeps "nearly right" and "completely blind" apart via 1/2 versus 1/100. This substitution transfers directly to probing analyses and representation diagnostics.
The full label space is the foundation of the whole argument. A restricted candidate set can conjure a correct prediction out of a simple trimming (ground truth cat, true top-1 rabbit), which is a reminder for all bias and short-answer evaluation: whenever the output space is filtered, results over the complete space must be reported as a control.

Limitations & Future Work¶

The authors acknowledge that while the shape cues offer diagnostic clarity, they may not fully capture 3D geometry or viewpoint dependencies; completely isolating texture from residual shape impressions remains an open problem; and expanding the set of cue classes is needed to broaden the scope of bias analysis.
Points one can press on: the 20 superclasses are selected by human perceptual judgment, and the selection criterion ("shape or texture is the most discriminative feature of the class") may itself confine the conclusions to a comparatively easy subset of categories, leaving cue usage on harder categories uncovered; evaluation centers on ImageNet-derived categories and models trained on them, without touching detection, segmentation, or other multi-task settings; the quantization analysis covers only the ResNet family; and because cue images are inspected one by one by humans, the dataset is expensive to scale, with no reported upper bound on that process.
Directions for improvement: generalize the sensitivity metric from two cues to a joint multi-cue sensitivity vector (shape / texture / color / depth); use the same purified cues to probe generative and multimodal models (image-text aligned models after CLIP, conditional encoders of diffusion models); and fold the "human agreement κ" check into the generation pipeline as an automatic filtering stage to cut manual inspection cost and improve scalability.

vs cue-conflict (Geirhos et al., 2018): they build 1,280 conflict images by stylization, measure bias as ns/(nt+ns), and evaluate over just the shape and texture candidate classes; this paper switches to perceptually defined purified cues (6,000 images, ~5×), computes MRR sensitivity over full logits, and separates preference from absolute sensitivity. It keeps the original's core question but rejects its instantiation rather than its insight — the paper itself stresses that it still evaluates cue preference, only more precisely.
vs Mohla et al. (2022): the cue generation borrows their cue-separation idea, but the paper points out that applying it directly leaves shape leakage and reduced resolution in texture cues and background clutter in shape cues, so the generation pipeline and human verification were redesigned.
vs Burgert et al. (2025): they analyze feature reliance through controlled suppression but do not fully handle low-level texture cues and remain bound by ratio-based bias; this paper agrees with their conclusion (ImageNet-pretrained ResNet-50 does not strongly rely on texture) while also repairing the metric.
vs Tartaglini et al. (2022): they fill shape silhouettes with textures, overlooking local shape features, and still use a relative bias metric, which limits fair cross-model comparison.
vs Wen et al. (2023) / Doshi et al. (2025): the former measures sensitivity under global shape degradation, the latter uses texture-suppressed global shape cues; each covers only the single facet of "shape sensitivity," whereas this paper argues for redoing cue construction, prediction evaluation, and sensitivity measurement as a whole.

Rating¶

Novelty: ⭐⭐⭐⭐ [Not a new method but a systematic re-examination of a de facto standard, fixing dataset and metric at once and turning the benchmark itself into a reliability check — a solid but improvement-type contribution]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [32 training-strategy models, multiple architecture families, two large-model families, five quantization levels, plus an 88-participant human study — unusually thorough for a benchmark paper]
Writing Quality: ⭐⭐⭐⭐ [The three-problems-to-three-designs mapping is clear and the argument chain is tight; some figures and tables are cramped in the typeset layout]
Value: ⭐⭐⭐⭐⭐ [Provides a directly usable public dataset and metric, and gives a directionally consistent answer to the long-running "shape bias vs. performance" dispute, with a normative effect on later bias research]