Skip to content

IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ayushbits/IndicParam
Area: Multilingual & Translation
Keywords: Low-Resource Indic Languages, LLM Benchmark, Language Understanding vs World Knowledge, Cross-Lingual Transfer, UGC-NET

TL;DR

IndicParam presents a human-verified graduate-level benchmark of 13,207 questions across 11 low- and extremely low-resource Indic languages derived from the official UGC-NET national examinations, systematically decoupling pure linguistic proficiency from general knowledge and demonstrating that even the top-performing model (Gemini-2.5) achieves only 58.0% average accuracy.

Background & Motivation

Large language models (LLMs) have achieved unprecedented breakthroughs across standard multilingual benchmarks, particularly in high- and medium-resource scenarios such as machine translation, named entity recognition, and reading comprehension. However, comprehensive and realistic evaluation for low-resource and extremely low-resource languages remains severely underdeveloped. The Indian subcontinent is home to roughly 1.4 billion people and over 120 recognized languages spanning diverse families including Indo-European, Dravidian, Austro-Asiatic, and Sino-Tibetan. While the Indian Constitution formally recognizes 22 scheduled languages, existing Indic benchmarks (such as IndicGLUE and IndicXTREME) disproportionately target the 11 most widely spoken languages, reflecting the skewed availability of massive web-scale corpora. In pretraining datasets such as FineWeb2โ€”which aggregates approximately 3 trillion words across 1,800 non-English languagesโ€”the 11 languages analyzed in this paper account for 4.49 billion words. Crucially, four relatively well-covered languages (Nepali, Marathi, Gujarati, and Odia) comprise 4.45 billion of these tokens, leaving only 36 million words in total for the remaining seven languages (Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, and Konkani), despite representing nearly 280 million speakers.

Furthermore, existing evaluation suites that do touch upon tail Indic languages suffer from critical methodological shortcomings. Prominent efforts such as IndicGenBench heavily rely on machine-translated versions of English datasets (e.g., XQuAD or FLORES) and constrain their question-answering tasks to span extraction from provided contexts. Consequently, they fail to evaluate an LLM's capacity for autonomous factual recall and deep contextual reasoning within authentic regional cultural contexts. Prior benchmarks also fail to isolate an LLM's grammatical and morphosyntactic competence from its factual world knowledge, making it impossible to diagnose whether a failure originates from language decoding errors or missing world knowledge. Moreover, conventional benchmarks focus almost exclusively on simple 4-way multiple-choice questions, overlooking the complex assessment formats commonly required in higher education.

To bridge these gaps, this work constructs an authentic, graduate-level evaluation resource curated directly from official national competitive examinations. Core idea: IndicParam builds a 13,207-item human-curated benchmark from official Indian UGC-NET graduate-level examinations across 11 low- and extremely low-resource Indic languages, explicitly decoupling linguistic proficiency (LU) from factual knowledge (GK) across six complex question formats to expose the true boundaries and transfer bottlenecks of 20 frontier LLMs.

Method

Overall Architecture

The construction of IndicParam establishes an end-to-end framework to transform raw, heterogeneous academic examination papers into an auditable, machine-readable benchmark. The pipeline comprises four principal stages: first, native double-column examination PDFs and official answer keys administered between 2012 and 2018 were retrieved from the National Testing Agency (NTA) portal; second, a script-tailored OCR pipeline coupled with two-tier human verification was executed to convert scanned documents into clean digital text; third, each item was systematically annotated along two orthogonal axes: capability category (Language Understanding vs. General Knowledge) and question format (MCQ, matching, ordering, assertion-reasoning, etc.); fourth, 20 open-weight and proprietary frontier LLMs were evaluated under a standardized zero-shot evaluation protocol.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw UGC-NET Graduate Exam Papers<br/>Covering 11 low/extremely low-resource languages"] --> B["Script-Tailored OCR & Two-Step Verification<br/>LLMWhisperer and Surya OCR + native-speaker audit"]
    B --> C["Dual Capability Decoupling: LU vs. GK<br/>Separating morphosyntax from factual recall"]
    C --> D["Multi-Format Assessment Taxonomy<br/>Structuring 6 question types including A&R and Ordering"]
    D --> E["Standardized Zero-Shot LLM Evaluation<br/>Testing 20 frontier open-weight and closed systems"]

Key Designs

1. Script-Tailored OCR & Two-Step Verification: eliminating extraction artifacts in complex scripts Indic languages feature intricate vowel signs, diacritics, and complex consonant conjuncts that frequently break standard OCR systems, especially when processing double-column historical exam sheets. To maximize data fidelity, the authors implemented a script-differentiated digitization pipeline: for Devanagari-based languages (Sanskrit, Maithili, Marathi, Dogri, Rajasthani, Nepali, Bodo, Konkani), they deployed LLMWhisperer, an engine specifically engineered for conjunct retention. For non-Devanagari scripts (Gujarati, Odia, Santali's Ol Chiki), they applied Surya OCR, followed by comprehensive manual post-editing by native experts. The dataset then underwent a rigorous two-tier review: native speakers rectified character drops, standardized punctuation, and matched options to official answer keys, after which senior reviewers cross-verified items against original exam papers. An audit on a randomized 5% sample (10,427 words and 61,547 characters) confirmed a Word Error Rate (WER) of only 0.68% and a Character Error Rate (CER) of 0.12%, guaranteeing that model performance reflects true linguistic reasoning rather than OCR noise.

2. Dual Capability Decoupling (LU vs. GK): distinguishing linguistic competence from factual recall A core bottleneck in multilingual evaluation is the inability to determine whether an error stems from an inability to decode grammar or an absence of factual knowledge. IndicParam addresses this by classifying every question into one of two functional categories: Language Understanding (LU) and General Knowledge (GK). LU questions explicitly evaluate linguistic mechanicsโ€”including phonology, morphological inflection, case markers, syntactic agreement, and semantic discourse rules (e.g., compound formation and Sandhi in Sanskrit). GK questions test domain-specific factual recall across literature, history, philosophy, and cultural entities. The proportion of LU questions ranges from 0.6% in Gujarati to 27.8% in Rajasthani, 20.7% in Sanskrit, and 20.3% in Odia. This categorization enables researchers to observe how scaling parameters impacts factual knowledge transfer compared to grammatical comprehension in scarce data regimes.

3. Multi-Format Assessment Taxonomy: capturing graduate-level reasoning across six question formats Rather than restricting evaluations to straightforward single-fact MCQs, IndicParam captures the structural diversity of real-world graduate examinations. Each question is categorized into one of six distinct structural formats: Multiple-Choice Questions (MCQ, 73.1%), List Matching (9.0%), Temporal and Logical Ordering (6.5%), Assertion and Reasoning (A&R, 6.1%), Identification of Incorrect Statements (IS, 4.1%), and Blank Filling (1.1%). In several specific languages, non-MCQ formats dominateโ€”accounting for 78.3% of questions in Dogri and 67.0% in Gujarati. Handling these non-standard formats requires multi-hop relational deduction, cross-referencing multiple premises, and verifying causal claims, presenting a rigorous test of an LLM's logical alignment.

Unified Evaluation Setup

All 20 models were evaluated using a standardized zero-shot prompt that specifies the target language and strictly instructs the model to return only the single letter representing the chosen option (A, B, C, or D). Open-weight models were evaluated in bf16 precision using greedy decoding (\(\text{temperature}=0\), \(\text{max\_tokens}=50\), batch size 16), while proprietary closed models were evaluated via their official APIs.

Key Experimental Results

Main Results

The evaluation benchmarked 20 competitive LLMs across four size categories: small (<8B), medium (8Bโ€“27B), large/MoE (โ‰ฅ27B), and frontier proprietary closed models. The table below reports the accuracy (%) across the eight extremely low-resource languages in IndicParam:

Size Model Dogri (doi) Maithili (mai) Rajasthani (raj) Sanskrit (san) Sans-Eng (code-mix) Bodo (brx) Santali (sat) Konkani (gom) Average
Closed Gemini-2.5 54.2 54.0 66.6 72.4 80.3 48.1 52.9 48.4 59.2
Closed GPT-5 45.7 43.4 44.6 54.8 64.6 31.6 40.4 39.0 45.1
Closed Claude-Haiku-4.5 44.8 42.4 44.1 47.8 58.1 28.1 40.4 42.1 43.0
Closed Grok-4-Fast 40.2 38.1 39.1 42.9 54.2 30.6 38.8 37.4 39.7
Large Open DeepSeek-V3.2 43.3 41.4 40.1 51.3 61.4 36.4 41.5 37.6 43.7
Large Open Llama-3.3-70B 37.3 37.5 38.2 41.7 51.6 28.6 40.1 37.3 37.6
MoE Open gpt-oss-120b 39.7 35.6 39.4 40.2 48.1 28.3 35.6 35.4 37.4
Medium Open Gemma-3-27B 37.6 34.4 39.8 38.6 49.1 28.9 36.7 36.5 37.3
Medium Open Mistral-3.1-24B 38.0 37.4 34.8 40.0 44.1 30.1 35.7 32.0 36.2
Medium Open Qwen3-32B 43.4 34.0 33.8 39.2 48.6 31.8 39.7 34.5 33.9
Small Open Qwen3-4B 34.6 31.6 30.0 33.8 33.6 25.1 33.8 29.6 31.2
Small Open Qwen2.5-3B 32.1 28.8 28.6 31.9 34.8 28.3 33.1 31.4 30.9
Small Open Llama-3.2-3B 27.9 21.9 26.3 28.0 27.9 27.8 27.4 28.8 26.9

The table below summarizes performance across the four relatively higher-resource languages (low-resource subset):

Size Model Nepali (nep) Gujarati (guj) Marathi (mar) Odia (ory) Average
Closed Gemini-2.5 52.0 51.7 55.7 64.5 54.9
Closed GPT-5 43.4 40.0 48.4 48.7 44.9
Closed Claude-Haiku-4.5 43.8 37.0 43.9 44.0 42.0
Large Open DeepSeek-V3.2 39.7 39.5 44.6 42.1 41.6
MoE Open Qwen3-A3B 37.2 33.5 43.5 40.4 38.7
MoE Open gpt-oss-120b 37.1 37.0 39.7 39.9 38.3
Large Open Llama-3.3-70B 35.0 35.9 40.8 40.7 37.9
Medium Open Gemma-3-27B 39.3 31.1 39.9 39.9 37.4
Medium Open Mistral-3.1-24B 34.9 32.6 32.9 27.6 32.6
Medium Open Qwen3-32B 33.0 28.4 33.2 28.9 31.2
Small Open Qwen3-4B 28.7 27.4 30.5 34.7 29.8
Small Open Qwen2.5-3B 27.1 29.5 27.9 28.8 28.2
Small Open Llama-3.2-3B 26.2 26.1 28.6 27.0 27.1

Ablation & Sub-Capability Analysis: Scaling Disparities in GK vs. LU

Evaluating general knowledge against linguistic understanding reveals fundamentally divergent scaling dynamics across models:

Capability Dimension Scope & Core Focus Top Performing Models (Accuracy) Bottom Performing Models (Accuracy) Correlation with Model Scale (Pearson \(r\))
General Knowledge (GK) Fact retrieval, literature, history, entities Gemini-2.5 (58.0%), GPT-5 (54.8%), DeepSeek-3.2 (43.3%) Qwen3-4B (31.3%), Llama-3.2-3B (26.7%) \(r = 0.91\) (Strong linear correlation)
Language Understanding (LU) Morphology, syntax, phonology, grammar Gemini-2.5 (69.8%), GPT-5 (52.5%), Claude-4.5 (49.8%) Llama-3.2-3B (28.4%), Qwen2.5-3B (29.2%) \(r = 0.73\) (Moderate correlation)

In format-wise evaluations, closed models demonstrated superior robustness across complex question formats: GPT-5 dominated MCQs (46.1%) and Blanks (47.8%), Claude-4.5 led in Identifying Incorrect Statements (45.7%), while DeepSeek-3.2 achieved an outstanding 47.2% on List Matching among open-source models.

Key Findings

  • Pervasive cross-lingual transfer bottlenecks: Across all 20 evaluated LLMs, only Gemini-2.5 achieved an average accuracy above 50% (58.0% overall). All other models scored below 46%, with models on underrepresented languages like Bodo and Santali performing close to random chance (25%โ€“36%).
  • GK benefits from raw scale, while LU requires specialized adaptation: General knowledge exhibits a strong correlation with parameter scale (\(r = 0.91\)), demonstrating that multilingual representations effectively transfer cross-lingual factual knowledge. Conversely, pure linguistic competence correlates more weakly with scale (\(r = 0.73\)). Medium models with high-quality multilingual pretrainingโ€”such as Gemma-3-27B (40.9% LU) and Mistral-3.1-24B (42.7% LU)โ€”consistently outperform larger dense models, proving that targeted morphological tuning can overcome sheer parameter deficits.
  • Typological proximity and academic presence drive positive transfer: Rajasthani achieved disproportionately strong results (Gemini at 66.6%) despite virtually zero web text in FineWeb2, driven by its high typological and lexical proximity to Hindi. Similarly, Sanskrit and code-mixed Sanskrit-English demonstrated the highest overall scores (Gemini reaching 72.4% and 80.3% respectively), reflecting the extensive linguistic and digitized literature available for classical languages.

Highlights & Insights

  • Authentic competitive exam corpus over synthetic translations: By extracting questions directly from national-level UGC-NET examinations, IndicParam avoids translationese and trivial context-matching, delivering a genuinely discriminative, graduate-level testbed.
  • Explicit decoupling of linguistic vs. factual reasoning: The benchmark provides concrete empirical proof that factual recall scales smoothly with compute while morphological and syntactic parsing in low-resource scripts requires explicit linguistic inductive bias and dedicated vocabulary coverage.
  • Robust script-tailored OCR pipeline: Implementing distinct OCR engines for Devanagari and non-Devanagari scripts followed by double-blind expert review secured an exceptional 0.68% WER across complex scripts.

Limitations & Future Work

  • Domain bias toward humanities and academia: The dataset is drawn entirely from national eligibility tests in linguistics and literature, biasing content toward formal academic discourse rather than colloquial, everyday language use.
  • Restricted zero-shot single-letter protocol: Models were evaluated strictly in a zero-shot, letter-selection setting. Alternative strategies such as chain-of-thought (CoT), few-shot in-context learning, and retrieval augmentation were not investigated.
  • Future expansion: Future iterations should incorporate additional non-scheduled languages (particularly Sino-Tibetan languages of Northeast India) and explore test-time compute scaling on low-resource reasoning tasks.
  • vs IndicGLUE / IndicXTREME: Prior suites focus on classification and translation for major high-resource Indic languages; IndicParam extends coverage to 11 severely neglected languages using graduate-level QA.
  • vs IndicGenBench: IndicGenBench relies on span extraction from translated passages; IndicParam assesses closed-book parametric knowledge and authentic cultural understanding.
  • vs ParamBench / MILU: ParamBench evaluates graduate-level exams exclusively in Hindi; MILU includes several regional languages but focuses on general school-level QA without separating LU from GK.

Rating

  • Novelty: โญโญโญโญโญ First authentic graduate-level benchmark for low-resource Indic languages with explicit LU/GK decoupling.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation of 20 frontier models across 11 languages, 6 question formats, and 2 knowledge types.
  • Writing Quality: โญโญโญโญโญ Detailed error analysis, clear taxonomy, and transparent documentation of data provenance and quality control.
  • Value: โญโญโญโญโญ Sets a new gold standard for evaluating low-resource multilingual generalization and Indic language capabilities.