Skip to content

LLM Agents as Computational Typologists

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/changbingY/typology-autosearch
Area: LLM Agent
Keywords: Computational Typology, LLM Agents, Reference Grammars, Interlinear Glossed Text, Typological Hypothesis Testing

TL;DR

AutoTypologist is the first LLM agent architecture designed for evidence-grounded computational linguistic typology, combining hierarchical grammar pre-retrieval, interlinear glossed text (IGT) quantitative analysis, dynamic evidence graph tracking, and post-hoc adversarial auditing to perform feature coding and cross-linguistic hypothesis testing across 25 diverse languages.

Background & Motivation

Linguistic typology seeks to uncover the foundational principles and structural diversity of human language through cross-linguistic comparison. This discipline underpins major global databases such as WALS, APiCS, and Grambank, which decompose linguistic structure into hundreds of fine-grained grammatical features across thousands of languages. Traditionally, constructing and verifying such databases demands decades of labor-intensive fieldwork and expert curation by trained linguists who meticulously read massive reference grammars, decipher interlinear glossed text (IGT) samples, and reconcile idiosyncratic descriptive conventions. As linguistic fieldwork races against the rapid extinction of endangered languages, manual typological feature coding faces severe scalability bottlenecks and human expert bandwidth constraints.

Although LLM agents have recently demonstrated remarkable capabilities in accelerating scientific discovery across natural sciences and economics, their application to computational linguistics has lagged behind. Meta-linguistic reasoning over reference grammars presents steep technical hurdles: descriptive grammars are book-length treatises spanning hundreds of pages with heterogeneous terminology, while linguistic empirical evidence is primarily encoded in multi-tier IGT annotations (source forms, morphemic segmentations, grammatical tags, and free translations). Existing NLP approaches rely either on task-specific rule induction or brute-force prompting for low-resource translation, both of which struggle with hallucination, term mismatch, and lack of verifiable provenance when handling complex grammatical systems across diverse language families.

Bridging this gap requires formalizing the typologist's methodology into an inspectable, closed-loop agentic workflow that couples top-down prose comprehension with bottom-up corpus verification. Rather than treating typological judgment as a one-shot classification problem, an agent must formulate explicit investigation plans, query specialized linguistic tools over grammar text and glossed corpora, maintain an evolving network of empirical claims, and withstand adversarial audit. Core idea: formalize linguistic typology as an agentic ReAct workflow named AutoTypologist that leverages hierarchical grammar indexing, specialized IGT analysis tools, dynamic evidence graph tracking, and post-hoc adversarial auditing to execute verifiable feature coding and cross-linguistic hypothesis testing.

Method

Overall Architecture

AutoTypologist is structured as a multi-stage LLM agent architecture that transforms raw reference grammars and extracted IGT examples into evidence-grounded typological judgments. The system takes as input a digitized reference grammar partitioned into structured, summarized chunks, an annotated IGT corpus, and a typological query (e.g., assigning a Grambank feature value or testing a Greenbergian universal). It outputs a categorical decision accompanied by a verifiable citation trail and confidence score.

The execution pipeline unfolds across five sequential phases: first, Hierarchical Pre-retrieval & Planning scans high-level table of contents outlines and corpus-wide IGT statistics to produce a targeted investigation plan while fetching full sections and primary gloss distributions; next, the agent enters an Evidence-Graph-Driven ReAct Investigation loop, iteratively executing actions via specialized linguistic tools and updating its evidence state; throughout this process, Contradiction Resolution & Adversarial Auditing monitors conflicting claims and subjects tentative conclusions to a rigorous post-hoc challenge; finally, Formal Feature Coding & Hypothesis Synthesis maps verified claims onto formal coding schemes or synthesizes multi-language evidence profiles to evaluate typological universals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Reference Grammar + IGT Corpus + Typological Query"] --> B["Hierarchical Pre-retrieval & Planning<br/>TOC outline scan and dual-pass retrieval"]
    B --> C["Evidence-Graph-Driven ReAct Investigation<br/>Multi-round tool execution and claim tracking"]
    C --> D["Contradiction Resolution & Adversarial Auditing<br/>Conflict adjudication and post-hoc skepticism"]
    D --> E["Formal Feature Coding & Hypothesis Synthesis<br/>Grambank policy mapping and cross-lingual profiling"]
    E --> F["Output: Evidence-Grounded Judgments with Provenance"]

Key Designs

1. Hierarchical Pre-retrieval & Planning: Scaffolding macro-level book outlines and micro-level corpus statistics before iterative reasoning Standard semantic search over hundreds of dense grammar pages often retrieves fragmented passages that mention a technical term out of context while missing overarching analytical frameworks. To circumvent this, the initial phase constructs a comprehensive search strategy from high-level table of contents metadata and aggregate IGT tag frequencies (e.g., checking overall frequencies of past tense PST or perfective PFV tags). Before the iterative reasoning loop begins, the agent executes a targeted two-pass pre-retrieval: it fetches and reads relevant grammar sections in their entirety rather than in isolated chunks, while concurrently executing planned quantitative corpus checks such as construction frequency analysis. This establishes an empirical grounding that prevents the agent from wandering down irrelevant analytical paths.

2. Evidence-Graph-Driven ReAct Investigation: Constraining exploration through specialized linguistic tools while accumulating verifiable claims Unconstrained natural language generation easily succumbs to confirmation bias and ungrounded speculation. AutoTypologist restricts the agent's action space to three families of domain-specific tools: grammar-reading tools (e.g., read_full_section, follow_cross_references, extract_author_claims), IGT-analysis tools (analyse_tag, analyse_construction, analyse_absence), and specialized IGT-only exploratory tools (analyse_morpheme_position, parse_example_structure). Over a budget of up to 10 ReAct rounds, each tool observation is parsed into atomic claims added to a structured evidence graph. Each node explicitly logs claim text, evidence type, unique grammar chunk IDs or IGT sample IDs, support/refute designations, and confidence scores, providing complete end-to-end auditability back to primary source materials.

3. Contradiction Resolution & Adversarial Auditing: Actively reconciling competing claims and subjecting tentative outputs to a skeptic review Descriptive grammars frequently contain apparent internal contradictions, such as an author discussing a morpheme under nominal derivations while noting its clitic-like syntactic distribution elsewhere. When the evidence graph detects contradictory claims, the agent pauses normal search to invoke a dedicated contradiction resolution module that weighs descriptive authority, contextual restrictions, and dialectal or register variation. Furthermore, once the ReAct loop concludes, a separate adversarial auditor agent is deployed to scrutinize the synthesized conclusion against the accumulated evidence graph. The auditor actively attempts to disprove the findingโ€”evaluating whether observed IGT patterns are merely accidental correlations, whether grammatical descriptions describe marginal phenomena, and whether counterexamples were ignoredโ€”holding the authority to uphold, weaken, or overturn the preliminary verdict.

4. Formal Feature Coding & Hypothesis Synthesis: Aligning with rigorous comparative linguistics frameworks for single-language and universal analysis To convert empirical evidence into scientific knowledge, the final stage interfaces with standardized typological frameworks. For Typological Feature Coding, the agent loads formal Grambank coding criteria (e.g., coding '1' for non-deictic definite articles, '0' for rigorous absence across comprehensive texts, and '?' for under-analyzed grammars) and maps graph evidence directly into categorical codes. For Typological Hypothesis Testing, the agent parses implicational universals (such as Greenbergian rules \(P \implies Q\)) into formal antecedent and consequent checks, concurrently evaluating a balanced cross-linguistic sample of 10 languages from distinct families to classify each language into one of four distinct categories: supporting evidence, counter-evidence, irrelevant evidence (antecedent N/A), or insufficient evidence.

A Worked Example: Coding Japhug Object Nominalization (GB049)

The agent's trajectory on coding Grambank feature GB049 in the Sino-Tibetan language Japhug illustrates how iterative exploration and adversarial auditing prevent false negatives: 1. Initial Search: The agent queries search_text("object nominal patient derivation", top_k=5). The evidence indicates that nominalizing prefixes produce action nominals or abstract nouns, but highlights highly productive Object-Verb nominal compounding patterns. 2. Iterative Refinement: Over multiple rounds calling extract_author_claims("Nominal morphology > Nominal derivations"), the agent notes that dedicated affixal derivation of patient nouns is marginal or non-productive. At iteration 10, the ReAct loop concludes with an initial prediction of label 0 (absence of productive pattern; Confidence: 0.10). 3. Auditor Intervention: The post-hoc auditor reviews the full evidence graph, noting that the coding policy considers productive Object-Verb nominal compounds as an affirmative realization of object nominalization. Finding that the primary agent artificially narrowed the scope to single affixes, the auditor issues an overturned verdict and flags the instance for human expert review, averting an erroneous false-negative coding.

Key Experimental Results

Main Results: Typological Feature Coding

AutoTypologist was evaluated across 25 open-source reference grammars representing 14 language families on all 195 Grambank features. The evaluation benchmarks zero-shot prompting, a corpus-level majority-vote baseline, and three distinct information conditions: IGT only, Grammar only, and Grammar + IGT. Performance is reported in Weighted F1 and Macro F1.

Model / Baseline Information Condition Weighted F1 (%) Macro F1 (%) Key Takeaway
Majority Vote Corpus Prior 62.5 41.5 Highly competitive due to extreme class imbalance
Qwen2.5-7B Zero-shot 42.3 26.7 Severe degradation without grounding context
Qwen2.5-7B IGT 52.7 34.0 Modest induction capability from glosses alone
Qwen2.5-7B Grammar 55.9 41.3 Reaches majority baseline Macro F1 (+14.6% over zero-shot)
Qwen2.5-7B Grammar+IGT 50.6 40.3 Suffers slight degradation from evidence integration noise
Qwen3.5-9B Zero-shot 40.9 31.1 Weak zero-shot generalization on low-resource grammars
Qwen3.5-9B IGT 54.4 38.1 Capable of extracting basic morphological regularities
Qwen3.5-9B Grammar 60.1 47.3 Solid improvement, outperforming majority voting by 5.8% Macro F1
Qwen3.5-9B Grammar+IGT 52.8 41.7 Integration bottleneck reduces overall precision
Qwen3.6-27B Zero-shot 58.1 40.2 Strong meta-linguistic baseline understanding
Qwen3.6-27B IGT 63.8 45.4 Surpasses majority vote on both weighted and macro metrics
Qwen3.6-27B Grammar 70.7 59.2 Best overall performance, exceeding majority Macro F1 by +17.7%
Qwen3.6-27B Grammar+IGT 67.8 53.1 Grammar prose remains primary driver of accuracy
Gemma4-31B Zero-shot 56.8 43.4 Competitive open-weights baseline
Gemma4-31B IGT 64.5 50.5 Achieves impressive 50.5% Macro F1 from pure gloss induction
Gemma4-31B Grammar 69.6 54.6 Strong long-context synthesis across full chapters
Gemma4-31B Grammar+IGT 67.5 52.3 Consistent performance across language families
GPT5.4-mini Zero-shot 52.2 38.3 Highly conservative zero-shot baseline
GPT5.4-mini IGT 58.5 42.8 Reliable pattern extraction across well-attested paradigms
GPT5.4-mini Grammar 69.1 56.4 Strong closed-source performance
GPT5.4-mini Grammar+IGT 65.6 54.4 Strong precision but frequently predicts uncertain (?)

Ablation Study: Post-hoc Auditor Contribution

The contribution of the adversarial auditing stage was ablated under the Grammar-only condition on the Chicham language Aguaruna:

Model W-F1 w/o Audit (%) W-F1 w/ Audit (%) \(\Delta\) W-F1 M-F1 w/o Audit (%) M-F1 w/ Audit (%) \(\Delta\) M-F1 Note
GPT5.4-mini 67.2 67.5 +0.3 68.3 68.5 +0.2 Marginally prunes false-positive claims
Gemma4-31B 67.0 69.5 +2.5 68.5 69.9 +1.4 Rectifies hasty conclusions from isolated passages
Qwen3.6-27B 58.6 61.1 +2.5 69.6 70.9 +1.3 Delivers consistent gains across all metric cuts

Typological Hypothesis Testing: Evidence Profiles (10-Language Sample)

Evaluated across 10 Greenbergian universals and their manually permuted adversarial counterparts (averaged across 3 runs):

Model Condition Standard: # For Standard: # Against Standard: # N/A Standard: # Insuf. Adversarial: # For Adversarial: # Against Adversarial: # N/A Adversarial: # Insuf.
Qwen3.6-27B IGT 4.47 0.13 1.73 3.67 0.63 4.67 1.80 2.90
Qwen3.6-27B Grammar 3.57 0.10 1.83 4.50 0.07 3.80 1.70 4.43
Qwen3.6-27B Grammar+IGT 5.73 0.10 2.13 2.03 0.03 5.27 2.20 2.50
Gemma4-31B IGT 4.77 0.60 2.37 2.27 1.13 4.73 2.10 2.03
Gemma4-31B Grammar 5.53 0.23 3.00 1.23 0.07 5.73 2.63 1.57
Gemma4-31B Grammar+IGT 6.33 0.07 2.70 0.90 0.20 5.73 2.63 1.43
GPT5.4-mini IGT 1.93 0.03 1.07 6.97 0.13 1.40 0.90 7.57
GPT5.4-mini Grammar 1.33 0.00 0.67 8.00 0.03 0.87 0.73 8.37
GPT5.4-mini Grammar+IGT 1.47 0.00 0.67 7.87 0.07 0.83 0.63 8.47

Key Findings

  • Descriptive Grammar Prose is Essential: Providing access to reference grammar text yields massive gains across all architectures, with Qwen3.6-27B leaping from 40.2% Macro F1 (zero-shot) to 59.2% (Grammar). LLMs demonstrate strong aptitude for parsing linguists' analytical prose.
  • The "Grammar + IGT" Integration Tax: Contrary to common intuition, appending IGT examples to full grammar prose does not improve performance; instead, it causes slight performance drops (1.5% to 7% Macro F1), verified via paired bootstrap tests (10,000 resamples). Introducing fine-grained empirical examples introduces retrieval distraction and complicates multi-hop evidence reconciliation.
  • Divergent Agent Personas: Models exhibit pronounced behavioral traits during open-ended hypothesis testing. GPT5.4-mini is excessively risk-averse, classifying roughly 80% of language cases as "insufficient evidence". Conversely, Gemma4-31B and Qwen3.6-27B readily draw decisive conclusions, correctly validating genuine universals while identifying robust counterexamples (falsifying adversarial hypotheses across 5.27โ€“5.73 languages).
  • Uneven Competence Across Linguistic Domains: Performance peaks in Nominal Categories and Word Order, but struggles in Complex Sentences and Verbal Categories (tense-aspect-modality systems), where morpheme boundaries and periphrastic constructions challenge automated parsing.

Highlights & Insights

  • Deconstructing Scholarly Fieldwork into Atomic Agent Tools: The paper avoids naive end-to-end long-context prompting, instead providing explicit tooling for morpheme alignment, absence checking, and cross-reference traversal, successfully transforming an opaque black-box task into an interpretable scientific workflow.
  • The Pitfall of Multi-granularity Evidence Blending: The empirical finding that combining high-level grammatical prose with raw IGT examples degrades accuracy offers a crucial insight for agent system design: raw granular data can act as an informational distracter when higher-level abstractions are already available.
  • Adversarial Falsification Paradigm: Employing inverted Greenbergian universals establishes a powerful evaluation methodology for scientific research agents, demonstrating that AutoTypologist does not merely regurgitate memorized typological consensus but actively seeks out contradictory empirical evidence.

Limitations & Future Work

  • Validation Asymmetry: While AutoTypologist rapidly synthesizes hypotheses and gathers references, validating its decisions requires substantial effort from human linguistic experts, leaving an asymmetric verification burden.
  • Modality and Static Document Constraints: The framework is currently confined to written text and static grammar books, completely ignoring spoken recordings, acoustic prosody, and visual sign language modalities vital to comprehensive language documentation.
  • Persistent Formal Category Confusions: Qualitative audit of 180 error cases reveals that agents frequently mistake bound affixes for free particles, confuse nominal case clitics with verbal argument indexation, and falter when grammar terminology deviates from standardized Grambank definitions.
  • vs. Rule Induction from Glossed Corpora (Howell et al., 2017; Chaudhary et al., 2020): Prior systems used specialized statistical classifiers restricted to narrow morphological sub-problems on small datasets; AutoTypologist is a general-purpose agent operating over unrestricted natural language grammar monographs.
  • vs. Grammar-Assisted Machine Translation (Tanzer et al., 2024; Hus and Anastasopoulos, 2024): Earlier work utilized reference grammars merely as in-context prompts to boost sentence translation; this work establishes typological analysis as an autonomous scientific end in itself.

Rating

  • Novelty: โญโญโญโญโญ First end-to-end evidence-grounded LLM agent for linguistic typology, pioneering AI-assisted scientific discovery in theoretical linguistics.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation covering 25 languages across 14 families, 195 Grambank features, bootstrap significance testing, and in-depth manual adjudication of 180 errors.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear framing, rigorous task formalization, and honest reporting of agent failure modes and limitations.
  • Value: โญโญโญโญโญ Provides an indispensable open-source foundation for low-resource language preservation, typological database expansion, and AI-linguist collaboration.