Probing Commonsense Reasoning in Vision-Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/xx19/ObSchema
Area: Multimodal VLM
Keywords: Entity Attribute Schema Induction, Observation-to-Schema Framework, Scientific Literature Extraction, Attribute-Condition Decoupling, Value-Aware Deduplication
TL;DR¶
Addressing the challenges of scattered entity properties, heterogeneous nomenclature, and entangled experimental conditions in scientific corpora, ObSchema establishes an observation-to-schema batch-induction framework that decouples attribute-condition observation tuples, accumulates dual-signal corpus evidence for gatekeeper promotion, and performs value-aware refinement to induce compact, high-precision hierarchical schemas.
Background & Motivation¶
Scientific literature documents rich, fine-grained entity property and performance measurements essential for accelerating scientific literature retrieval, hypothesis reasoning, and autonomous discovery. However, this knowledge is rarely structured: it remains deeply scattered across thousands of heterogeneous articles, described with diverse and inconsistent terminology, and heavily scoped by experimental protocols, operating temperatures, or measurement rates. Closed-schema information extraction approaches rely on predefined, static attribute vocabularies. While functional for narrow domains, scientific entities span far too many evolving physical and chemical dimensions to anticipate in advance, causing closed vocabularies to miss salient emerging attributes while accumulating irrelevant, sparse categories.
Open-schema information extraction methods alleviate vocabulary bottlenecks by directly prompting large language models to generate free-form attribute labels from text. Nevertheless, open extractions suffer from catastrophic instability across documents, regularly absorbing measurement conditions into composite labels. In battery science, for instance, a single experimental passage might yield overlapping labels such as "capacity", "specific capacity", "discharge capacity at 0.5C", and "capacity retention after 300 cycles". These properties represent related-yet-distinct physical quantities with entangled condition modifiers, making them unsuitable for coherent ontology construction. Crucially, conventional taxonomy induction algorithms merge concepts based purely on semantic text embeddings, which collapses in scientific settings: "capacity" (e.g., 150 mAh) and "specific capacity" (e.g., 170 mAh/g) appear almost synonymous in embedding space, yet their underlying physical normalization bases differ fundamentally.
This work addresses this challenge by shifting the paradigm from rigid predefined vocabularies or uncurated open-schema chaos to corpus-grounded entity-centric schema induction. Rather than attempting to enumerate all conceivable attributes in a scientific field, the goal is to induce a hierarchical schema capturing the recurring, type-level attributes that are genuinely salient in the target corpus, decoupling experimental conditions and organizing them into coherent scientific dimensions. Core idea: propose ObSchema, an observation-to-schema batch-processing framework that extracts condition-decoupled n-ary observation tuples, gates attribute promotion via the geometric mean of cross-document popularity and category-relative prominence, and constructs compact, non-redundant hierarchical property schemas via value-aware conservative merging and adaptive aspect grouping.
Method¶
Overall Architecture¶
ObSchema operates incrementally over document batches describing a designated entity type (such as battery cells or petrophysical samples), starting with a minimal set of user-defined coarse categories (e.g., cell performance and cell configuration). The pipeline advances across three mutually reinforcing stages: first, observation extraction parses raw paragraphs into structured observation tuples that decouple physical attributes from experimental conditions, guided by the running schema as soft guidance; second, attribute organization maps raw candidates into the evolving hierarchy through category gating and top-down descent, holding them in a pending pool until their dual-signal corpus evidence score satisfies a strict threshold; third, schema refinement periodically applies value-aware duplicate merging and adaptive aspect grouping to induce intermediate scientific dimensions, producing a compact, multi-level tree ontology.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Scientific Document Batches Bt<br/>Schema St-1 as Soft Guidance"] --> B["Decoupled Observation Tuple Extraction<br/>Extract Entity, Attribute, Value, Definition, Conditions"]
B --> C["Dual-Signal Evidence Gate Promotion<br/>Geometric Mean of Cross-Doc Popularity & Category Prominence"]
C -->|Evidence Score s(a) >= 0.15| D["Value-Aware Conservative Merging<br/>Verify Physical Equivalence via Values, Units & Definitions"]
C -->|Insufficient Evidence| P["Pending Candidate Pool<br/>Accumulate Multi-Batch Corpus Support"]
D --> E["Adaptive Hierarchical Aspect Grouping<br/>LLM-Induced Intermediate Dimensions & Re-parenting"]
E --> F["Compact Hierarchical Schema Tree St"]
Key Designs¶
1. Decoupled Observation Tuple Extraction: Separating Intrinsic Properties from Experimental Conditions
Conventional open information extraction conflates experimental measurement conditions (such as temperature, C-rate, or cycle count) with property names, generating over-specified compound labels like capacity_at_0.5C or coulombic_efficiency_at_high_temperature that fragment the schema across papers. ObSchema addresses this by formulating paragraph-level extraction as \(n\)-ary observation tuples \(o = (e, a_{\text{raw}}, \delta, v, c, Q)\). Here, \(e\) denotes the target entity mention, \(a_{\text{raw}}\) is the raw property name, \(\delta\) represents a contextual one-sentence natural language definition generated within the passage, \(v\) captures the extracted scalar or qualitative measurement value, \(c \in \mathcal{C}\) indicates the coarse category assignment, and \(Q = \{(q_i, w_i)\}\) explicitly gathers all condition type-value pairs scoping the measurement. By isolating measurement variables into \(Q\), the extractor preserves the core physical identity of the attribute while maintaining granular contextual provenance for downstream validation.
2. Dual-Signal Evidence Gate Promotion: Balancing Corpus-Wide Generality with Local Prominence To prevent idiosyncratic, passage-specific attributes from infiltrating the active schema while avoiding the suppression of vital niche sub-disciplines, ObSchema decouples semantic placement from formal schema commitment. Newly classified attributes are routed into a pending pool to accumulate corpus-level statistics across batches. For candidate attribute \(a\) assigned to category \(c\), the promotion score \(s(a) \in [0, 1]\) is formulated as the geometric mean of two complementary signals:
The corpus-wide popularity term \(\text{pop}(a) = \frac{\log(1 + \min(d_a, e_a))}{\log(1 + k_{\text{seed}})}\) scales with the minimum of distinct source documents \(d_a\) and distinct entity mentions \(e_a\) (normalized by batch size constant \(k_{\text{seed}}\)), strictly penalizing candidates repeated within a single paper. The local prominence term \(\text{dist}(a) = 1 - \exp(-\kappa \frac{e_a}{M_c})\) measures the attribute's entity support relative to the leading leaf attribute \(M_c\) in category \(c\), with saturation rate \(\kappa = 15.0\). The geometric mean guarantees that an attribute is promoted to the active schema if and only if it exhibits both corpus-wide presence and prominent category-level significance (exceeding threshold \(\theta = 0.15\)), preventing singleton artifacts from corrupting the hierarchy.
3. Value-Aware Conservative Merging: Deduplicating Properties via Physical Grounding Following promotion, superficial lexical variations and cross-batch duplicates remain unresolved, yet standard embedding similarity collapses distinct scientific quantities that happen to share lexical tokens (such as absolute capacity in mAh versus mass-normalized capacity in mAh/g). ObSchema deploys a value-aware merging verification mechanism. Candidate duplicates identified via embedding proximity within each coarse category are presented to the LLM alongside their representative numerical values, physical units, contextual definitions, and supporting text evidence. The LLM verifies whether candidates represent the exact same underlying physical quantity under identical normalization bases. Confirmed equivalent nodes pool their aliases, document counts, and condition distributions, while attributes differing in normalization, directional measurement, or temporal scope remain strictly separate, ensuring maximum structural compactness without sacrificing scientific fidelity.
4. Adaptive Hierarchical Aspect Grouping: Inducing Intermediate Scientific Dimensions Even after merging, coarse categories frequently accumulate broad flat arrays of leaf nodes lacking intrinsic parent-child relationships, rendering the schema unnavigable. Lexical clustering is inadequate because terms like "specific capacity" and "capacity retention" share vocabulary roots while representing orthogonal scientific concepts (storage capacity vs. degradation kinetics). ObSchema introduces intermediate parent nodes via an evidence-grounded three-step procedure: (i) Aspect Induction prompts the LLM to analyze all leaves under a coarse category and propose a concise set of organizing scientific dimensions; (ii) Group Formation matches each leaf node to its best-supporting aspect utilizing its name, definition, representative values, units, and textual evidence; (iii) Consolidation prompts the LLM to verify and instantiate an intermediate parent node only when grouped leaves exhibit unified semantic coherence. If the evidence does not justify grouping, candidate leaves remain under their original parent, avoiding rigid uniform-depth hierarchies.
Key Experimental Results¶
Main Results¶
ObSchema was evaluated on two challenging physical science corpora (Battery Science: 185 articles, 927 paragraphs; Geoscience: 118 articles, 378 paragraphs) spanning publications from 2007 to 2025. Evaluation was conducted using GPT-5-mini as an automated LLM judge across four rigorous metrics: Attribute Validity (AV), Path Granularity (PG), Sibling Coherence (SC), and Uniqueness (Uniq.), alongside their unweighted Average (Avg.). ObSchema was compared against three established baselines—Chain-of-Layer, TaxoAdapt, and Iterative LLM Induction—under two backbone LLMs:
| Domain | Base Model | Method | Attribute Validity (AV) | Path Granularity (PG) | Sibling Coherence (SC) | Uniqueness (Uniq.) | Average (Avg.) |
|---|---|---|---|---|---|---|---|
| Battery Science | Qwen3-30B-A3B | Chain-of-Layer | 0.860 | 0.711 | 0.667 | 0.383 | 0.655 |
| Battery Science | Qwen3-30B-A3B | TaxoAdapt | 0.951 | 0.717 | 0.626 | 0.559 | 0.713 |
| Battery Science | Qwen3-30B-A3B | Iterative LLM Induction | 0.935 | 0.665 | 0.756 | 0.733 | 0.772 |
| Battery Science | Qwen3-30B-A3B | ObSchema (Ours) | 0.964 | 0.807 | 0.816 | 0.830 | 0.854 |
| Geoscience | Qwen3-30B-A3B | Chain-of-Layer | 0.950 | 0.703 | 0.708 | 0.594 | 0.739 |
| Geoscience | Qwen3-30B-A3B | TaxoAdapt | 0.964 | 0.800 | 0.630 | 0.690 | 0.771 |
| Geoscience | Qwen3-30B-A3B | Iterative LLM Induction | 0.891 | 0.589 | 0.556 | 0.724 | 0.690 |
| Geoscience | Qwen3-30B-A3B | ObSchema (Ours) | 0.991 | 0.798 | 0.818 | 0.807 | 0.853 |
| Battery Science | Qwen3-14B | Chain-of-Layer | 0.817 | 0.683 | 0.538 | 0.447 | 0.621 |
| Battery Science | Qwen3-14B | TaxoAdapt | 0.976 | 0.808 | 0.548 | 0.651 | 0.746 |
| Battery Science | Qwen3-14B | Iterative LLM Induction | 0.912 | 0.537 | 0.444 | 0.658 | 0.638 |
| Battery Science | Qwen3-14B | ObSchema (Ours) | 1.000 | 0.935 | 0.667 | 0.704 | 0.827 |
| Geoscience | Qwen3-14B | Chain-of-Layer | 0.920 | 0.802 | 0.606 | 0.564 | 0.723 |
| Geoscience | Qwen3-14B | TaxoAdapt | 0.919 | 0.787 | 0.539 | 0.679 | 0.731 |
| Geoscience | Qwen3-14B | Iterative LLM Induction | 0.754 | 0.521 | 0.333 | 0.771 | 0.595 |
| Geoscience | Qwen3-14B | ObSchema (Ours) | 1.000 | 0.864 | 0.800 | 0.911 | 0.894 |
Ablation Study & Backbone Robustness¶
Ablation experiments on Qwen3-30B-A3B isolate the impact of hierarchical classification and schema refinement. Cross-backbone robustness benchmarks confirm generalizability across diverse model scales and architectures:
| Evaluation Track | Configuration / Model Backbone | Attribute Validity (AV) | Path Granularity (PG) | Sibling Coherence (SC) | Uniqueness (Uniq.) | Overall Average (Avg.) | Description |
|---|---|---|---|---|---|---|---|
| Battery Ablation | Full ObSchema | 0.961 | 0.833 | 0.788 | 0.833 | 0.853 | Complete batch induction pipeline |
| Battery Ablation | w/o Hierarchical Classification | 0.929 | 0.718 | 0.619 | 0.772 | 0.760 | Bypasses category gating & descent |
| Battery Ablation | w/o Schema Refinement | 0.951 | 0.817 | 0.595 | 0.619 | 0.745 | Omits value-aware merge & grouping |
| Geoscience Ablation | Full ObSchema | 0.992 | 0.901 | 0.785 | 0.807 | 0.871 | Complete framework on Geoscience |
| Geoscience Ablation | w/o Hierarchical Classification | 0.968 | 0.748 | 0.559 | 0.778 | 0.763 | Severe degradation across all metrics |
| Geoscience Ablation | w/o Schema Refinement | 0.986 | 0.928 | 0.528 | 0.725 | 0.791 | High leaf redundancy and flat trees |
| Model Generalization | Mistral-Small-24B (Battery) | 0.976 | 0.827 | 0.771 | 0.882 | 0.864 | Top performance on dense backbone |
| Model Generalization | GPT-5-nano (Battery) | 0.885 | 0.789 | 0.750 | 0.628 | 0.763 | Competitive lightweight proprietary model |
| Model Generalization | Mistral-Small-24B (Geoscience) | 1.000 | 0.906 | 0.776 | 0.862 | 0.886 | Flawless validity and superior uniqueness |
| Model Generalization | GPT-5-nano (Geoscience) | 0.972 | 0.814 | 0.759 | 0.783 | 0.832 | Balanced performance on domain entities |
Key Findings¶
- Substantial Gains in Schema Uniqueness: ObSchema achieved massive improvements in Uniqueness over the strongest baseline (+9.7 points on Battery and +11.7 points on Geoscience with Qwen3-30B-A3B). This represents the metric with the highest human-judge agreement (85%), demonstrating that value-aware deduplication and dual-signal gating eliminate redundant attributes far more effectively than lexical similarity matching.
- Zero Hallucination on Smaller Dense Models: Under the smaller Qwen3-14B backbone, ObSchema reached a perfect 1.000 Attribute Validity score across both domains, whereas baselines degraded into generating procedural steps or unmeasurable concepts. The evidence-based promotion threshold acts as a dependable filter, insulating the schema from noisy proposals.
- Precision vs. Coverage Balance on Expert Benchmarks: When evaluated against a 60-attribute battery expert reference list, an audit of 82 promoted nodes revealed a 98.8% precision rate for core and valid niche attributes (84.1% core, 14.6% valid niche, 0.0% invalid, and only 1.2% redundant). The 16.7% reference omission rate stems from deliberate conservatism: rare canonical attributes (e.g., voltage plateau) were correctly extracted into the pending pool but fell just short of the strict corpus-wide evidence threshold.
- Superior Inference Throughput: By structuring document processing into parallelizable batches and executing refinement strictly at batch boundaries, ObSchema required only 14.00 minutes on Battery and 8.67 minutes on Geoscience. In contrast, Iterative LLM Induction required 60.42 minutes on Battery due to sequential passage-by-passage dependency, establishing ObSchema's computational efficiency.
Highlights & Insights¶
- Decoupled Six-Tuple Representation: Structuring extractions as \((e, a_{\text{raw}}, \delta, v, c, Q)\) fundamentally solves condition entanglement at the source, preventing experimental configurations from polluting attribute namespaces.
- Dual-Signal Multiplicative Promotion: The geometric mean between cross-document/mention popularity \(\text{pop}(a)\) and category-relative prominence \(\text{dist}(a)\) mathematically enforces both corpus-wide consensus and local scientific relevance without manual weight tuning.
- Value-Grounded Deduplication: Integrating representative values, measurement units, and scientific definitions into LLM merge decisions establishes a rigorous defense against incorrect semantic collapses (e.g., distinguishing capacity from mass-normalized specific capacity).
Limitations & Future Work¶
- Domain Scope Limited to Physical Sciences: The methodology has been validated on battery science and geoscience, which possess structured, quantitative measurement conventions. Extending ObSchema to qualitative or context-fluid fields like biomedicine, chemistry, or computational social science may necessitate specialized observation extraction prompts and condition parsers.
- Conservative Coverage Trade-Off: The strict promotion threshold (\(\theta = 0.15\)) occasionally withholds canonical domain concepts that are universally understood in textbooks but infrequently reiterated in cutting-edge research articles (e.g., voltage plateau). Incorporating external domain knowledge graph priors or expert whitelist seeding could recover rare canonical attributes without diluting precision.
Related Work & Insights¶
- vs. Closed-Schema Extraction (e.g., MAVE, OpenTag): Traditional closed extraction requires expert-curated property lists and cannot discover emerging scientific metrics; ObSchema dynamically induces schemas directly from unstructured corpora without manual enumeration.
- vs. Open Information Extraction (e.g., ODIE, EDC): Open IE extracts fragmented property-condition compounds that lack cross-document unification; ObSchema explicitly decouples conditions and synthesizes local extractions into a coherent type-level ontology.
- vs. Taxonomy Induction (e.g., Chain-of-Layer, TaxoAdapt): Existing taxonomy builders organize abstract topical concepts using textual similarity, failing when subtle unit or normalization differences dictate concept boundaries; ObSchema introduces value-aware verification and evidence-driven grouping tailored to fine-grained physical entities.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the problem of entity-centric attribute schema induction and introduces an observation-to-schema pipeline with decoupled conditions.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain and multi-backbone evaluations, paired with ablation studies, downstream extraction validation, and expert-curated ground truth auditing.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous mathematical and empirical framing that directly targets the structural pitfalls of scientific information extraction.
- Value: ⭐⭐⭐⭐⭐ Provides an automated, robust blueprint for organizing scientific literature into machine-readable knowledge graphs and scientific AI ontologies.