(How) Do MLLMs Report Bistable Images Like Humans?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/rtakatsky/mllm-bistable-images
Area: LLM Reasoning
Keywords: Bistable Images, Multimodal Large Language Models, Mechanistic Interpretability, Modulability and Exclusivity, Machine Cognition
TL;DR¶
Evaluating the canonical duck–rabbit and 50 synthetic Visual Anagrams, this paper demonstrates that Multimodal Large Language Models (MLLMs) report ambiguous images with human-consistent modulability and report exclusivity, revealing through circuit interventions that bottom-up and top-down modulations follow distinct computational pathways and that exclusivity is causally rooted in single-object construal.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) have established remarkable visual-linguistic reasoning capabilities, yet the internal mechanisms governing how they resolve intrinsic perceptual ambiguity remain poorly understood. In cognitive science, bistable images such as the classical duck–rabbit figure represent an elegant testbed for perceptual decision-making: a single static visual input simultaneously supports multiple mutually incompatible high-level interpretations. Human observers typically report only one coherent interpretation at any given instant, while their reported percept can be systematically shifted by sensory modifications (such as image orientation and focal fixation) as well as cognitive expectations (such as contextual primes and verbal prompts). Examining whether MLLMs exhibit analogous report behavior provides essential insights into building predictable and robust multimodal agents, as well as advancing cognitive simulation to understand how underdetermined sensory evidence is transformed into discrete linguistic reports.
However, investigating ambiguous perception in contemporary web-scale MLLMs faces two substantial hurdles. First, widely known optical illusions and bistable figures are ubiquitous across pre-training datasets, making it difficult to disentangle genuine stimulus-grounded visual inference from memorized text associations and superficial template matching. Second, prior comparative studies have largely focused on broad benchmark evaluations that highlight model-human divergences without isolating tractable dimensions of perceptual processing or dissecting the internal representations that mediate them. In human perception, the hallmark of bistability lies not in arbitrary classification bias, but in two operational dimensions: modulability—the degree to which perceptual reports can be systematically steered by bottom-up visual manipulations and top-down semantic priors; and exclusivity—the strict commitment to reporting a single coherent percept rather than concurrently enumerating all potential alternatives.
To resolve these challenges, this paper operationalizes modulability and exclusivity across the LLaVA architecture family as well as representative open-source MLLMs, introducing a pipeline of 50 novel, diffusion-generated Visual Anagrams to eliminate memorization confounds. Core idea: MLLMs reliably replicate human-consistent modulability under both visual and linguistic perturbations while preserving strict reporting exclusivity; mechanistically, this behavior emerges from competing image-token representations resolved in middle-to-late transformer layers, where bottom-up cues directly modulate image-side evidence whereas top-down linguistic cues propagate through text-side residual streams and feed-forward networks rather than cross-modal attention queries, with single-object count encoding serving as a shared causal substrate for exclusive reporting.
Method¶
Overall Architecture¶
The paper formulates an integrated empirical and mechanistic interpretability pipeline to investigate ambiguous visual processing. On the input side, the benchmark incorporates the canonical Harper's duck–rabbit, 50 synthetic Visual Anagrams verified via an external VLM judge, and matched non-ambiguous controls (side-by-side two-animal configurations and isolated single-animal controls). The analytical workflow couples behavioral probing (quantifying rotation, spatial red-circle cueing, linguistic prefix/semantic priors, and beam-search exclusivity) with internal mechanistic tools, including layerwise Direct Logit Attribution (DLA), Logit Lens dominant patch localization, cross-modal query activation patching, and single-token causal embedding interventions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Bistable Stimuli<br/>(Harper Duck-Rabbit / 50 Visual Anagrams)"] --> B["Dual Behavioral Probing Protocol<br/>Bottom-Up/Top-Down Modulability + Beam Exclusivity"]
B --> C["Layerwise Direct Logit Attribution (DLA)<br/>Isolating Mid-to-Late Layer Attention & Token Rivalry"]
C --> D["Top-Down Resampling Ablation & Query Decoupling<br/>Residual Stream & MLP Transfer vs Cross-Modal Attention"]
D --> E["Single-Token Image Embedding Intervention<br/>Causal Link between Object-Count and Exclusivity"]
E --> F["Computational Mechanism Synthesis<br/>Modulable Reporting & Grounded Single-Entity Representation"]
Key Designs¶
1. Dual Behavioral Probing Protocol: Decoupling Bottom-Up Saliency and Top-Down Linguistic Priors
To overcome the ambiguity of unconstrained generation and establish rigorous comparisons with psychophysical data, the authors design precise behavioral probes. Bottom-up modulability is evaluated by rotating stimuli at \(1^\circ\) increments around the center, fitting logistic psychometric functions to identify the \(50\%\) subjective ambiguity boundary angle \(\theta_0\), and by sweeping a 24-pixel radius red circle across an image grid to simulate human visual fixation patterns. Top-down modulability injects orthographic prefix cues (e.g., "It starts with 'r'.") or semantic property cues (e.g., "It's terrestrial.") appended to a standard query ("List every animal in the image, each in one word." with continuation prefix "I see a"). Exclusivity is evaluated via 8-beam search with candidate continuations classified into single-animal versus multi-animal categories by an external evaluator, paired with a direct numerical object-count probe ("How many objects are in the image? Answer with a number only.").
2. Layerwise Direct Logit Attribution (DLA): Isolating Mid-to-Late Layer Attention and Token Rivalry
To reveal how competing semantic hypotheses evolve along the language model backbone without explicit prompting biases, the method applies Direct Logit Attribution (DLA). The unnormalized logit contributions for competing percept tokens (e.g., "_bird" and "_rabb") are decomposed across transformer blocks into attention outputs (attn_out), multi-layer perceptrons (mlp_out), and accumulated residual stream states (resid_post). The decomposition identifies that competition between the two interpretations rises steeply after layer 20 and remains acute across layers 20–26, driven predominantly by attention directed toward image tokens. Applying Logit Lens with an internal confidence threshold (\(\ge 0.1\)) identifies distinct, spatially separated "dominant patches" supporting each interpretation. Zero-ablating these dominant image tokens drives the output probability of the corresponding animal token to nearly zero, proving that image tokens serve as the direct causal substrates carrying conflicting hypotheses.
3. Top-Down Resampling Ablation and Query Decoupling: Demonstrating Textual Routing Over Cross-Modal Rewriting
A conventional intuition suggests that top-down textual instructions steer multimodal models by altering the text query vectors to selectively attend to specific image regions. The authors directly test this hypothesis using activation patching (resampling ablation), replacing activations from corrupted cue runs (e.g., "It starts with 'x'.") into clean cue runs (e.g., "It starts with 'r'."). The causal analysis demonstrates that prompt information transfers from prefix tokens to the final prediction token primarily across layers 15–20, where early-to-mid MLPs play an indispensable role in transforming surface tokens into conceptual entity priors. Crucially, patching the cross-modal query vectors that compute similarities against image keys recovers only \(\sim 3\%\) of the total top-down effect, while bottom-up rotation runs show virtually zero query-mediated transfer. This confirms that top-down modulation operates primarily through the text-side language model residual stream rather than by re-indexing visual attention.
4. Single-Token Image Embedding Intervention: Establishing the Shared Causal Substrate Between Object Count and Exclusivity
To resolve whether report exclusivity reflects an output vocabulary bottleneck or an integrated semantic construal of the visual scene as a single object, the method executes single-token image embedding patching from matched two-animal controls into bistable runs. For each patched token, the intervention measures changes in the explicit object-count contrast (\(\text{logit}("_2") - \text{logit}("_1")\)) alongside the continuation exclusivity contrast (\(\text{logit}("_and") - \text{logit}(".")\)). The results reveal a specific, compact subset of image tokens whose transplantation simultaneously drives substantial shifts toward multi-object counting and non-exclusive conjunction continuations. This establishes a direct causal bridge showing that exclusive reporting is intrinsically rooted in a single-entity structural representation of the ambiguous image region.
Benchmark & Evaluation Protocol¶
The core evaluation covers five models across the LLaVA family (LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-v1.6-Vicuna-7B, LLaVA-v1.6-Mistral-7B, and Llama3-LLaVA-Next-8B), complemented by four non-LLaVA architectures (Qwen2-VL-7B-Instruct, SmolVLM-Instruct, IDEFICS2-8B, and InstructBLIP-Vicuna-7B). The benchmark comprises the classical Harper duck–rabbit image, 50 synthetic Visual Anagrams generated via DeepFloyd IF and Stable Diffusion x4 upscaler passing dual-orientation validity filtering via GPT-5-nano (minimum \(4/5\) consensus at both \(\pm 45^\circ\)), matched two-animal side-by-side controls, single-animal controls, and the Raven–Bear figure–ground ambiguous stimulus as an informative boundary condition.
Key Experimental Results¶
Main Results: Object Count Probing and Beam Continuation Exclusivity Across Models¶
The table below quotes the empirical evaluation from Table 2 of the paper, detailing the first-token object count probability and beam-search exclusivity shares across the five LLaVA models on ambiguous and control stimuli.
| Stimulus Type | Evaluation Metric / Dimension | LLaVA-1.5-7B | LLaVA-1.5-13B | v1.6-Vicuna | v1.6-Mistral | Llama3-NeXT |
|---|---|---|---|---|---|---|
| Harper Duck–Rabbit | Object Count \(P("1") / P("2")\) | 0.88 / 0.10 | 0.97 / 0.03 | 0.99 / 0.00 | 0.99 / 0.01 | 0.99 / 0.01 |
| Beam Mass Share: Single / Multiple | 0.93 / 0.07 | 1.00 / 0.00 | 1.00 / 0.00 | 1.00 / 0.00 | 1.00 / 0.00 | |
| Visual Anagrams (VA) | Object Count \(P("1") / P("2")\) | 0.88 / 0.09 | 0.89 / 0.10 | 0.99 / 0.01 | 0.96 / 0.03 | 0.96 / 0.04 |
| Beam Mass Share: Single / Multiple | 0.88 / 0.12 | 0.97 / 0.03 | 0.99 / 0.00 | 1.00 / 0.00 | 0.93 / 0.00 | |
| Two-Animal Control | Object Count \(P("1") / P("2")\) | 0.01 / 0.87 | 0.01 / 0.97 | 0.00 / 0.98 | 0.03 / 0.95 | 0.00 / 0.99 |
| Beam Mass Share: Single / Multiple | 0.02 / 0.98 | 0.01 / 0.99 | 0.01 / 0.99 | 0.01 / 0.99 | 0.00 / 1.00 | |
| Raven–Bear Stimulus | Object Count \(P("1") / P("2")\) | 0.18 / 0.66 | 0.35 / 0.60 | 0.60 / 0.38 | 0.94 / 0.05 | 0.84 / 0.15 |
| Beam Mass Share: Single / Multiple | 0.49 / 0.51 | 0.57 / 0.43 | 0.55 / 0.45 | 1.00 / 0.00 | 1.00 / 0.00 |
Ablation Study: Causal Interventions and Mechanistic Recovery Effects¶
The table below summarizes key causal patching experiments, ablations, and control baselines across the paper's mechanistic interpretability sections.
| Intervention / Setting | Target Variable / Analysis | Observed Value / Explanatory Share | Comparison Baseline | Mechanistic Insight & Conclusion |
|---|---|---|---|---|
| Query-Side Patching | Top-down linguistic modulation effect | \(\sim 3\%\) effect explained | Total effect (\(100\%\)) | Language priors do not primarily alter cross-modal attention queries |
| Query-Side Patching | Bottom-up rotation modulation effect | \(< 0.5\%\) (negligible) | Total effect (\(100\%\)) | Visual transformations do not trigger indirect downstream query loops |
| Dominant Patch Ablation | Target animal token logit (bird/rabbit) | Drops to \(\approx 0.00\) | Baseline prob (\(0.40 \sim 0.60\)) | Confirms dominant visual patches causally ground the reported percept |
| Single-Animal Cue Control | Cued percept probability on unsupported image | \(0.03 \sim 0.05\) | Ambiguous image (\(0.23 \sim 0.43\)) | Linguistic priors cannot induce hallucinated percepts lacking visual support |
| Explicit "Two Animals" Hint | Multiple-animal beam mass on bistable images | Increases to \(37\% \sim 56\%\) (1.5 models) | Default query (\(\le 12\%\)) | Proves exclusivity is a default perceptual construal rather than hard decoding lock |
Key Findings¶
- Alignment with Human Fixation Dynamics: Red-circle visual cueing systematically shifts token preferences in spatial regions that closely correlate with human gaze density patterns reported in visual psychophysics (e.g., circular cues at the duck's beak elevate duck logits, while placement near the rabbit's mouth shifts logits toward rabbit).
- Linguistic Modulation is Visually Bounded: In the single-animal rabbit control, injecting the duck-oriented prefix cue
"It starts with 'd'."suppresses rabbit-related tokens but redistributes probability mass to other d-initial animals (e.g.,deer,dog,donkey) while keepingduckbelow \(0.05\). This demonstrates that linguistic modulation adjudicates between visually grounded hypotheses rather than inducing ungrounded hallucination. - Figure–Ground Decoupling of Exclusivity: On the Raven–Bear figure–ground stimulus, models favoring a two-entity parsing (LLaVA-1.5-7B/13B, \(P("2") = 0.66 / 0.60\)) allocate roughly half their beam mass to enumerating both animals, whereas models maintaining a single-entity parsing (LLaVA-v1.6-Mistral and Llama3-NeXT, \(P("1") = 0.94 / 0.84\)) remain completely exclusive (\(1.00 / 0.00\)).
Highlights & Insights¶
- Bridging Cognitive Vision and Mechanistic Circuits: The paper bridges classical cognitive paradigms with modern mechanistic interpretability, translating abstract psychological constructs like "perceptual rivalry" into layerwise DLA trajectories, dominant patch masks, and causal activation swaps.
- Debunking Cross-Modal Query Modulation: Challenging the intuitive expectation that top-down prompts act by reshaping visual query vectors, the authors conclusively demonstrate that linguistic guidance flows via internal text residual streams and early-to-mid MLPs, leaving cross-modal attention queries virtually unperturbed.
- De-confounding Memorization via Synthetic Anagrams: Leveraging generative diffusion models to construct 50 verified Visual Anagrams provides a reusable blueprint for probing multimodal illusions without contamination from web pre-training data.
Limitations & Future Work¶
- Absence of Temporal Switching Dynamics: Human bistable perception exhibits spontaneous, oscillatory perceptual switching over extended viewing, driven by neural adaptation, microsaccades, and active gaze shifts. The static feed-forward forward-pass architecture studied here cannot capture dynamic temporal alternations. Future work could examine active-vision systems like DeepEyes that sample saccadic views iteratively.
- Restriction to Object-Category Ambiguity: The primary investigations focus on static object-category ambiguity within a single contiguous region, leaving other multistable modalities—such as 3D Necker cube depth reversals and kinetic motion quartets—for future operationalization.
- Epistemic Boundary Between Report and Phenomenology: The authors explicitly note that model report exclusivity represents computational convergence during language generation and must not be conflated with conscious experience or subjective perceptual awareness.
Related Work & Insights¶
- vs Panagopoulou et al. (2024): While Panagopoulou et al. conducted a broad black-box benchmark across 12 MLLMs emphasizing failure modes and human discrepancies, this work performs a depth-first mechanistic dissection isolating how human-like modulability and exclusivity emerge within neural circuits.
- vs Neo et al. (2025) & Jiang et al. (2025a): Prior works employed Logit Lens and attention knock-outs primarily for hallucination detection; this paper extends internal confidence mapping into causal patching, identifying the shared neural substrate linking numerical object counting to narrative exclusivity.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering mechanistic investigation of intrinsic visual ambiguity and bistable report behavior in MLLMs]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive coverage across 9 models, 50 synthetic anagrams, layerwise DLA, and cross-modal causal patching]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear conceptualization, elegant methodology, and rigorous experimental exposition]
- Value: ⭐⭐⭐⭐⭐ [Provides a definitive milestone for multimodal mechanistic interpretability and cognitive benchmarking]