PercepTax: Benchmarking Cross-Property Reasoning in Vision-Language Models¶
Conference: ECCV 2026
Paper: ECCV Official Link
Area: LLM Reasoning
Keywords: vision-language models, cross-property reasoning, perceptual taxonomy, physical intelligence, benchmark
TL;DR¶
PercepTax introduces a benchmark for hierarchical perceptual taxonomy and cross-property reasoning across material, affordance, function, and physical attributes, revealing that frontier vision-language models suffer substantial performance drops when jointly integrating multiple physical constraints.
Background & Motivation¶
When humans interact with the physical world, they solve tasks by naturally synthesizing environmental spatial structures with object-level physical characteristics. In unfamiliar or unequipped scenarios—such as requiring immediate protection when no shield is available—humans form a hierarchical cognitive chain: they first perceive the available objects in the scene, decompose candidate objects into multi-dimensional physical properties (e.g., surface flatness, rigidity, dimensions, and portability), and ultimately select the most suitable makeshift item (such as using a sturdy notebook or rigid sign to defend against an impact). This perceptual taxonomy process, transitioning from spatial perception to object identification and compositional property reasoning, forms the bedrock of physical intelligence.
However, existing benchmarks for vision-language models evaluate scene understanding in fragmented, isolated dimensions. Some benchmarks focus exclusively on 3D spatial relationship reasoning, while others assess single property families—such as material recognition, affordance prediction, or isolated intuitive physics—under separate schemas. These formulations bypass a critical capability required for real-world interactions: cross-property reasoning, where a model must simultaneously evaluate material durability, shape-driven manipulability, functional compatibility, and 3D spatial constraints to reach a viable physical decision. Prior to this work, no benchmark provided unified annotations across all property families or measured a model's capacity for joint constraint satisfaction.
This paper addresses this gap by introducing PercepTax, a structured benchmark for physical intelligence grounded in perceptual taxonomy. The authors formalize four property families across a fine-grained attribute space, mapping 3,173 object classes to comprehensive property profiles across both high-fidelity simulation and diverse real scenes. Core idea: formalize physical scene cognition into a three-level hierarchy of "scene → objects → properties" (Perceptual Taxonomy) and construct cross-property reasoning tasks requiring simultaneous satisfaction of material, physical attributes, affordances, and functional constraints to pinpoint the fundamental bottlenecks of current VLMs.
Method¶
Overall Architecture¶
PercepTax organizes scene cognition around a three-level hierarchy: scene → objects → properties. At the scene level, it couples high-precision synthetic ground truth and foundation-model detections on real images, providing 2D instance segmentation masks and 3D oriented bounding boxes with camera-frame coordinates and yaw angles. At the object level, instances receive category labels and descriptive texts. At the property level, every object is assigned a comprehensive profile spanning all four property families. Based on this taxonomy, an automated question generation engine leverages 48 parameterized templates to synthesize questions across four graduating difficulty levels, enforcing a strict unique-candidate constraint followed by systematic human verification.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Domain Image Collection<br/>UE5 synthetic scenes + OpenImages real data"] --> B["Scene & Object Annotations<br/>2D instance masks + 3D bounding boxes + text cues"]
B --> C["Property Dictionary & Clustering<br/>Four families with 54 physical attribute labels"]
C --> D["Parameterized QA Engine<br/>48 graduated cognitive reasoning templates"]
D --> E["Unique Candidate Constraint & Validation<br/>Ambiguity filtering + global human verification"]
E --> F["PercepTax Benchmark Release<br/>28,033 graduated visual evaluation questions"]
Key Designs¶
1. Unified Cross-Property Dictionary and Object Profiles: Establishing Complete Physical Priors
Previous benchmarks could not support cross-property reasoning because each operated under an isolated schema, preventing different property constraints from closing in on a single object. PercepTax establishes a unified dictionary of 54 fine-grained physical attributes grouped into four property families: 14 materials (such as metal, wood, paper, defining intrinsic substance and durability), 10 affordances (such as graspable, supportable, containable, defining interaction potentials), 13 functionalities (such as seating furniture, storage container, signage, defining intended purposes), and 17 physical properties (such as rigidity, fragility, elasticity, mass, and stability, governing responses under external forces). For 3,173 annotated object classes, foundation models generate context-aware descriptions, which are clustered via HDBSCAN and K-Means and verified by human annotators, establishing a complete ground-truth profile for every object.
2. Graduated Perceptual Taxonomy Tasks: Decoupling Visual Perception from Multi-Step Reasoning
To systematically diagnose where physical understanding breaks down, PercepTax categorizes questions into four graduated reasoning tasks: Object Description (22.1%), evaluating the baseline alignment between natural-language attribute cues and visual bounding boxes; 3D Spatial Relationship (19.4%), measuring relative spatial arrangements (left/right, above/below, front/behind, closer/farther) computed from 3D camera-frame bounding boxes; Single-Property Understanding (1.75%), probing individual attribute recognition in isolation; and Cross-Property Reasoning (56.7%), the central challenge of the benchmark. Cross-property questions are framed around goal-oriented physical dilemmas (such as repurposing an everyday object as an improvised stepstool or protective shield), requiring models to execute joint filter-and-eliminate operations across multiple interacting physical constraints.
3. Unique Candidate Constraint and Quality Filtering Pipeline: Ensuring Unambiguous Ground Truth
To prevent spurious guessing and ambiguous evaluation, the question generation engine enforces a strict Unique Candidate Constraint. For each synthesized question, the engine scans all bounded objects in the scene against the target property filters; if more than one object satisfies the complete set of required constraints, the question candidate is discarded. The engine incorporates lightweight LLM rephrasing for linguistic variety and applies automatic logical validity checks. Finally, an extensive human verification stage discards 31.7% of flawed bounding boxes, mismatched attributes, or ambiguous candidates, yielding 28,033 verified questions (including 50 expert-crafted real-world stress-test probes) to ensure high-fidelity benchmarking.
Key Experimental Results¶
Main Results¶
Evaluation spans premier closed-source models (Gemini 2.5 Pro, GPT-5, Claude Sonnet 4.5) and leading open-source models (Qwen3-VL, InternVL3.5, and LLaVA variants). All models are evaluated in an open-ended free-form format, judged by an LLM verifier against ground-truth answers.
| Model | Split | Spatial Relation (S.R.) | Object Desc. (Obj. Desc.) | Single-Prop. | Cross-Prop. | All Tasks |
|---|---|---|---|---|---|---|
| Human Upper Bound | Simulation (Sim.) | 94.81% | 99.26% | 90.37% | 94.07% | 94.63% |
| Human Upper Bound | Real Image (Real) | 89.70% | 91.11% | 88.15% | 92.96% | 90.48% |
| Gemini 2.5 Pro | Simulation (Sim.) | 60.63% | 53.81% | 30.77% | 38.20% | 41.87% |
| Gemini 2.5 Pro | Real Image (Real) | 74.30% | 84.70% | 55.29% | 53.00% | 57.38% |
| GPT-5 | Simulation (Sim.) | 54.56% | 54.24% | 25.90% | 29.92% | 34.82% |
| GPT-5 | Real Image (Real) | 76.23% | 87.06% | 47.77% | 39.84% | 48.78% |
| Claude Sonnet 4.5 | Simulation (Sim.) | 46.97% | 29.24% | 16.00% | 21.64% | 26.21% |
| Claude Sonnet 4.5 | Real Image (Real) | 69.38% | 66.67% | 43.59% | 32.95% | 42.48% |
| Qwen3-VL-32B | Simulation (Sim.) | 52.89% | 38.56% | 25.99% | 30.30% | 34.53% |
| Qwen3-VL-32B | Real Image (Real) | 72.16% | 81.18% | 50.37% | 41.66% | 50.08% |
| InternVL3.5-30B-A3B | Simulation (Sim.) | 45.20% | 27.54% | 16.02% | 17.61% | 23.47% |
| InternVL3.5-30B-A3B | Real Image (Real) | 76.23% | 72.16% | 37.06% | 25.57% | 37.32% |
| Qwen3-VL-8B | Real Image (Real) | 73.23% | 77.65% | 45.04% | 39.08% | 46.54% |
| LLaVA-Next-Vicuna-7B | Real Image (Real) | 55.89% | 7.45% | 19.77% | 9.20% | 17.91% |
Ablation Study¶
To disentangle whether failures originate from visual property perception or cross-property integration, the authors evaluate models with ground-truth property labels appended to prompts (Oracle Property Analysis) and dissect residual errors into three root failure modes.
| Model & Split | Single-Prop. Base | Single-Prop. Oracle | Cross-Prop. Base | Cross-Prop. Oracle | Semantic Bias (SB) | Constraint Neglect (CN) | Physical Commonsense Gap (PCG) |
|---|---|---|---|---|---|---|---|
| Gemini 2.5 Pro (Sim.) | 30.77% | 50.88% | 38.20% | 43.33% | 37.08% | 7.69% | 55.24% |
| Gemini 2.5 Pro (Real) | 55.29% | 82.05% | 53.00% | 62.56% | 21.55% | 27.99% | 49.06% |
| Qwen3-VL-32B (Sim.) | 25.99% | 29.41% | 30.30% | 33.93% | 43.10% | 12.72% | 44.18% |
| Qwen3-VL-32B (Real) | 50.37% | 74.44% | 41.66% | 47.89% | 21.74% | 28.56% | 48.44% |
Key Findings¶
- Severe Disconnect Between Grounding and Integration: State-of-the-art models excel at object description (GPT-5 scores 87.06% on real images) but drop drastically on cross-property reasoning to 39.84% (a gap of over 47 points from its perceptual competence and 53.12 points below human upper bound). This confirms that high visual recognition accuracy does not translate into physical constraint satisfaction.
- Physical Commonsense Gap Dominates Residual Failures: Even when models receive perfect oracle property labels, cross-property accuracy remains low (Gemini 2.5 Pro reaches only 62.56% on real images). Between 44% and 55% of residual failures stem from the Physical Commonsense Gap (PCG), where models fail to deduce implicit real-world physical dynamics (e.g., heat resistance or water vulnerability) from explicit material labels.
- Taxonomic In-Context Learning Facilitates Sim-to-Real Transfer: Providing a single structured simulation exemplar in prompt context (PercepTax ICL) improves real-image cross-property reasoning across all tested models (GPT-5 improves from 39.84% to 43.60%, and Gemini 2.5 Pro from 53.00% to 54.86%), showing that hierarchical reasoning scaffolds help align models toward multi-property filtering.
Highlights & Insights¶
- Pioneer of cross-property visual reasoning: moves beyond single-dimension benchmarks by framing grounded visual reasoning as a joint constraint satisfaction problem across material, physics, affordance, and function.
- Principled diagnostic oracle and failure decomposition: distinguishes perceptual recognition errors from cognitive integration bottlenecks, demonstrating quantitatively that compositional physical reasoning is the primary roadblock for current VLMs.
- Dual-domain data construction: combines Unreal Engine 5 photorealistic synthetic scenes (providing exact 3D geometry and unoccluded ground truth) with OpenImages real-world photos, ensuring both metric rigor and visual diversity.
Limitations & Future Work¶
- Static image evaluation: the benchmark relies on static single frames, omitting temporal physical transformations such as fluid dynamics, fracture mechanics, or deformable body interactions under applied force.
- Discrete categorical labels: physical properties are primarily represented as discrete semantic categories rather than continuous physical quantities (e.g., precise load-bearing limits in kilograms or exact thermal conductivity).
- Future modeling avenues: warrants post-training reinforcement learning schemes with verifiable multi-constraint physical rewards and architectures with explicit hierarchical perceptual reasoning priors.
Related Work & Insights¶
- vs PhysBench & PhysGame: Prior physics benchmarks target isolated multiple-choice physics questions or video-based physics violation detection; PercepTax annotates complete multi-property profiles for each object within grounded 3D scenes to evaluate multi-constraint physical filtering.
- vs Hypo3D & RoboAfford: Hypo3D evaluates hypothetical reasoning restricted to material and function, while RoboAfford focuses on contact affordances for robotic manipulation; PercepTax integrates four comprehensive property families with 54 attributes and centers on human-like cross-property problem-solving.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First systematic formulation and benchmark for cross-property reasoning in multimodal physical intelligence.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 10 proprietary and open-source models with oracle property injections, error decomposition, and ICL transfers.
- Writing Quality: ⭐⭐⭐⭐⭐ Clearly structured with rigorous mathematical and cognitive motivation, well-designed tables, and informative diagrams.
- Value: ⭐⭐⭐⭐⭐ Provides an essential diagnostic evaluation suite for embodied AI, visual reasoning, and multimodal foundation models.