UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multilingual & Machine Translation
Keywords: Urdu Benchmark, LLM Evaluation, Multitask Language Understanding, Native Educational Assessment, Cross-Lingual Knowledge Transfer
TL;DR¶
UrduMMLU is the first large-scale, natively authored MMLU-style benchmark for Urdu, featuring 26,389 four-option multiple-choice questions across 26 subdomains and 5 broad domains derived from Pakistani public examination board PDFs and native MCQ banks; evaluation of 30 leading LLMs reveals that while top systems achieve over 97% on STEM, they suffer a dramatic 25 to 40 percentage-point collapse on Urdu-centered humanities and cultural subjects.
Background & Motivation¶
In modern NLP, massive multitask language understanding benchmarks such as MMLU and MMLU-PRO have become the de facto gold standard for gauging the cross-disciplinary reasoning and factual competence of Large Language Models. However, these canonical benchmarks are overwhelmingly constructed in English and predominantly mirror Western educational curricula, legal frameworks, and socio-cultural paradigms. To assess LLMs across non-English languages, the prevailing paradigm has largely relied on translating English benchmarks into target languages via machine translation pipelines followed by heuristic filtering or light human post-editing (e.g., MMLU-PROX, GLOBAL-MMLU, INDICMMLU-PRO). This translation-centric methodology suffers from severe conceptual and empirical blind spots: it merely evaluates whether a model can decipher translated representations of globally shared scientific facts, while completely failing to assess whether the model possesses genuine knowledge of the target language's native literature, regional history, civic systems, jurisprudence, and indigenous academic curricula.
This critical evaluation vacuum is especially acute for Urdu, an Indo-Aryan language spoken by more than 230 million people with a centuries-old literary tradition and structured educational infrastructure. Prior Urdu evaluation benchmarks have been narrowly restricted to task-level NLP diagnostics, basic reading comprehension (such as UQuAD+), minimal-pair syntactic probing (UrBLiMP), or translated reasoning suites like UrduBench. None of these resources probe broad, multi-subject educational proficiency grounded in genuine Urdu-medium instructional materials. Consequently, the research community has remained unable to verify whether high-capacity multilingual models actually understand native Urdu cultural and academic knowledge or whether their perceived capability is merely a superficial artifact of memorizing translated English corpora. Building a native benchmark directly from historical examination papers poses steep engineering and logistical challenges, including heterogeneous layout parsing, high OCR noise on Nastaliq typography, bidirectional text rendering failures, and the complete lack of official answer keys for past public examination papers.
To overcome these fundamental barriers, this paper develops a comprehensive end-to-end framework that harvests raw examination past-papers and native online question banks across Pakistani Secondary School Certificate (SSC, Grades 9–10) and Higher Secondary School Certificate (HSSC, Grades 11–12) curricula. Core idea: abandon the superficial paradigm of translating Western benchmarks by directly extracting, cleaning, and adjudicating examination materials natively authored for Urdu-medium secondary education, establishing UrduMMLU—a 26,389-item benchmark spanning 26 subjects across 5 domains backed by a 16-stage pipeline and dual-annotator consensus filtering to expose true multilingual competence and knowledge boundaries in regional contexts.
Method¶
Overall Architecture¶
The UrduMMLU construction framework is engineered to deliver native authenticity, typographic purity, and gold-standard label fidelity. The pipeline sources candidate items from Pakistani public examination board past-paper PDFs and reputable native Urdu MCQ testing portals, channeling them through a 16-stage pipeline encompassing vision-language page filtering, OCR transcription, bidirectional script correction, orthographic and punctuation normalization, dual-annotator independent blind labeling with zero-tolerance consensus filtering, near-duplicate clustering, and curricular quota balancing.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Pakistani Board PDFs / Native MCQ Web Portals"] --> B["Multimodal Candidate Harvesting & Vision-Language OCR Extraction"]
B --> C["Bidirectional Script & Orthographic Text Cleaning Pipeline"]
C --> D["Dual-Annotator Blind Labeling & Zero-Tolerance Consensus Adjudication"]
D --> E["Stratified Deduplication & National Curricular Quota Balancing"]
E --> F["26,389 Standardized 4-Option MCQ UrduMMLU Benchmark"]
Key Designs¶
1. Multimodal Candidate Harvesting & Vision-Language OCR Extraction: Heterogeneous layouts and OCR corruption -> Vision-language filtering and structured prompt transcription -> Preserving authentic question stems and poetic contexts
Past-paper PDFs issued by Pakistani educational boards (such as FBISE and BISE Multan) feature heavily corrupted, multi-column, and non-standardized typographical layouts. Off-the-shelf OCR engines routinely fail on Urdu's cursive Nastaliq script, severing poetic meters, dropping diacritics, and scrambling mathematical notation. The authors implemented a two-tier multimodal ingestion architecture: in the first tier, each PDF page was converted into high-resolution imagery and screened with Claude Sonnet 4.6 to strictly discard English-medium pages, subjective non-MCQ sections, answer keys, and syllabus boilerplate; in the second tier, remaining pages were transcribed using Gemini 3 Flash under a dedicated vision-language OCR prompt specifically calibrated for Urdu examination scripts. This prompt faithfully preserved full question stems, answer options, classical ghazal/nazm couplets, and necessary textual context, while isolating Latin characters and dropping physically degraded, illegible items rather than attempting hallucinated reconstructions. For native websites, structured HTML scraping harvested stems, options, categories, and published human-validated answer keys.
2. Bidirectional Script & Orthographic Text Cleaning Pipeline: BiDi rendering collapse and Arabic-Urdu codepoint collision -> 16-stage directional isolation and Unicode normalization -> Eradicating typographic and parsing ambiguity
Because Urdu is written right-to-left (RTL), embedding left-to-right (LTR) mathematical formulas, chemical equations, Latin technical terms, or numerals frequently triggers catastrophic bidirectional (BiDi) rendering bugs where word order is inverted. Furthermore, web-scraped data routinely conflates visually identical but computationally distinct Arabic codepoints with Urdu-specific characters (e.g., Arabic ك and ي instead of Urdu ک and ی). The cleaning pipeline applied systematic Unicode bidirectional control marks (wrapping mixed Latin and digit fragments in LRI...PDI formatting isolators) to enforce deterministic visual order across any rendering engine. Next, NFKC character normalization remapped all rogue Arabic codepoints to canonical Urdu equivalents, standardized curly quotation marks to straight quotes while repairing missing opening quotes, transformed ASCII punctuation (?, ,, .) into authentic Urdu glyphs (؟, ،, ۔), unified diverse fill-in-the-blank tokens into exactly four underscores (____), and stripped redundant option letter prefixes (e.g., removing embedded A., B., or Urdu alef/bay prefixes) to prevent option-order cueing.
3. Dual-Annotator Blind Labeling & Zero-Tolerance Consensus Adjudication: Unlabeled exam past-papers and subjective divergence -> Two-expert independent double-blind labeling with strict exclusion criteria -> Achieving 0.8663 fixed-chance-adjusted agreement
While web-derived questions possessed published answer keys, the 12,759 examination-derived questions had no available answer keys and required complete manual annotation. Single-annotator labeling in complex humanities and religious domains risks subjective bias and factual errors. The authors recruited 17 qualified annotators (94.1% native Urdu speakers, 88.2% holding a Bachelor's or Master's degree) and built a custom interactive annotation dashboard. Every examination item was assigned to two independent annotators matched to the subject matter. Annotators selected the single best choice among A–D, with permission to consult Wikipedia or reference texts for objective historical dates or terminology within a strict 15–30 second verification budget. The protocol enforced a zero-tolerance inclusion rule: an item was retained in the benchmark if and only if both annotators independently picked the identical option without marking the item as "unsure/skip" and without triggering any of the five flag categories (multiple correct answers, no correct answer, ambiguity, missing image/diagram reference, or out-of-scope sectarian/hyper-local trivia). Across 16,070 answer-eligible items, observed consensus reached \(p_o = 0.8998\), translating to a fixed-chance-adjusted agreement of 0.8663, establishing exceptional empirical ground-truth reliability.
4. Stratified Deduplication & National Curricular Quota Balancing: Heavy source skew and domain imbalance -> Two-pass Jaccard clustering and stratified withholding -> Replicating the authentic secondary school educational structure
The raw cleaned pool aggregated 40,927 candidate items, with severe skew toward STEM and basic social sciences from prep portals like Ustad 360, while secondary-level humanities were underrepresented relative to the actual Pakistani matriculation and intermediate curricula. To prevent synthetic distortion, the team executed a two-pass deduplication and balancing strategy: first, exact text matches were consolidated while preserving multi-source provenance tags; second, fuzzy duplicate detection grouped items into 27,251 clusters using an n-word Jaccard similarity threshold (\(\ge 0.85\)), eliminating 2,356 conflicting or near-identical items. Following deduplication, a deliberate balancing step withheld 8,946 overrepresented STEM and social science items, deliberately aligning the released dataset with the natural subject distribution of Pakistani SSC/HSSC education, where humanities (41.6%) and social sciences (30.2%) constitute the dominant core of Urdu-medium schooling.
Benchmark Construction and Quality Adjudication Protocol¶
The finalized UrduMMLU benchmark comprises 26,389 validated four-option multiple-choice questions: - Provenance Composition: Web-scraped MCQ banks contribute 13,630 items (51.7%) with verified keys; examination board past-papers contribute 12,759 items (48.3%) established through dual human consensus (Table 1). - Domain & Subdomain Taxonomy (5 Domains, 26 Subdomains): - Humanities (10,982 items): Urdu Literature (5,859), Urdu Language (2,507), Islamic Studies (1,485), Urdu Grammar (755), Ethics (228), Fine Arts (148). - Social Sciences (7,959 items): Pakistan Studies (1,886), Economics (907), Education (836), Sociology (778), Civics (742), Health & Physical Education (675), Geography (630), Psychology (460), Commerce (437), Current & International Affairs (423), Psychometrics (185). - STEM (5,113 items): Chemistry (936), Biology (931), General Science (926), Mathematics (876), Computer Science (823), Physics (621). - Profession (973 items): Professional Development (514), Professional Studies (459). - Other (1,362 items): General Knowledge (1,362). - Dataset Properties & Option Balance: Median serialized item length is 76 characters (median question stem is 39 characters). The correct answer distribution across choices A, B, C, and D is virtually uniform (6,673 / 6,486 / 6,549 / 6,681, each representing between 24.6% and 25.3%), eliminating positional bias. - Annotation Verification Metric: Of 17,565 raw exam items entering annotation, exclusions comprised 1,611 answer disagreements, 1,247 flagged defects, 243 unsure selections, and 5 single annotations. For the remaining 16,070 answer-eligible items, observed agreement was \(p_o = 14,459 / 16,070 = 0.8998\). Adjusted for the four-option random chance baseline of 0.25: $\(p_{\text{adjusted}} = \frac{p_o - 0.25}{1 - 0.25} = \frac{0.8998 - 0.25}{0.75} = 0.8663\)$
Key Experimental Results¶
Main Results¶
The authors evaluated 30 open-weight and proprietary language models under zero-shot prompting across English and Urdu instruction templates. All-item exact-match accuracy was strictly enforced: unparsable generations, empty responses, refusal messages, and decoding loops were retained in the denominator and scored as incorrect, alongside tracking the invalid-output rate (Inv%). The table below presents representative zero-shot accuracies (%):
| Model Category | Model Name | Param Count | EN Prompt STEM | EN Prompt Humanities | EN Prompt Overall | EN Prompt Inv% | UR Prompt STEM | UR Prompt Humanities | UR Prompt Overall | UR Prompt Inv% |
|---|---|---|---|---|---|---|---|---|---|---|
| Proprietary APIs | Gemini-3.5-Flash | N/D | 97.56 | 85.16 | 90.23 | 0.1 | 97.75 | 85.49 | 90.45 | <0.1 |
| Gemini-3.1-Flash-Lite | N/D | 96.85 | 74.39 | 84.69 | <0.1 | 97.09 | 74.57 | 84.81 | <0.1 | |
| GPT-5.4 | N/D | 95.13 | 69.47 | 80.94 | 0.0 | 97.40 | 74.29 | 84.32 | 0.4 | |
| Claude-Sonnet-4.6 | N/D | 96.34 | 72.87 | 83.04 | 0.0 | 96.26 | 72.87 | 83.07 | 0.0 | |
| GPT-5.4-mini | N/D | 88.34 | 62.98 | 73.55 | 0.0 | 88.25 | 62.51 | 73.63 | 0.0 | |
| Claude-Haiku-4.5 | N/D | 90.44 | 58.67 | 71.44 | 0.1 | 91.92 | 59.42 | 72.52 | <0.1 | |
| Open-weight (>25B) | DeepSeek-V4-Flash | 284B | 97.38 | 68.79 | 82.45 | 0.1 | 97.36 | 67.30 | 81.33 | 0.3 |
| Gemma-4-31B-IT | 31B | 93.33 | 63.77 | 76.49 | <0.1 | 93.86 | 63.38 | 76.49 | <0.1 | |
| LLaMA-4-Maverick-17B-128E | 402B | 91.96 | 63.43 | 75.59 | <0.1 | 92.24 | 63.41 | 75.93 | <0.1 | |
| Qwen3.6-27B | 27B | 90.46 | 56.09 | 69.33 | 0.0 | 91.12 | 55.86 | 69.81 | 0.0 | |
| LLaMA-3.3-70B | 70B | 81.32 | 56.45 | 68.67 | 0.0 | 78.39 | 56.24 | 67.10 | <0.1 | |
| Open-weight (\(\le\)25B) | Ministral-3-8B | 8B | 67.81 | 45.38 | 54.85 | 0.0 | 71.37 | 45.86 | 57.08 | 0.0 |
| Gemma-2-9B-IT | 9B | 67.08 | 47.33 | 55.37 | 0.0 | 69.02 | 48.21 | 56.89 | 0.0 | |
| Qwen3-8B | 8B | 70.70 | 39.31 | 51.05 | 0.0 | 74.05 | 30.77 | 48.81 | 0.5 | |
| Qwen3-4B-Instruct-2507 | 4B | 68.61 | 42.44 | 50.92 | 0.0 | 68.75 | 43.11 | 51.79 | 0.0 | |
| LLaMA-3.1-8B | 8B | 46.96 | 36.47 | 43.37 | 0.0 | 46.49 | 37.71 | 43.91 | 0.0 | |
| Phi-4-mini | 3B | 37.02 | 28.92 | 33.89 | <0.1 | 37.08 | 28.77 | 34.21 | <0.1 | |
| BLOOMZ-7B | 7B | 24.98 | 23.93 | 26.45 | 11.1 | 22.75 | 17.75 | 20.55 | 34.5 | |
| BLOOMZ-3B | 3B | 27.44 | 26.89 | 28.72 | 6.5 | 9.74 | 3.60 | 7.40 | 74.2 | |
| Urdu-Tuned Models | Qalb-1.0-8B | 8B | 38.18 | 30.07 | 34.82 | 0.0 | 30.59 | 31.39 | 31.54 | 11.3 |
| Alif-1.0-8B | 8B | 40.62 | 25.75 | 34.55 | 0.6 | 29.04 | 27.98 | 28.62 | 12.6 |
Few-Shot Experiments & Cross-Benchmark Comparison¶
The authors conducted few-shot experiments on four open-weight models across 0-, 1-, 3-, and 5-shot configurations on 26,259 items (holding out 130 demonstrations). Additionally, they benchmarked UrduMMLU (U) against the translated MMLU-PROX Urdu (P) across 9 overlapping subjects:
| Evaluation Dimension | Model Name | 0-shot (EN / UR) | 1-shot (EN / UR) | 3-shot (EN / UR) | 5-shot (EN / UR) | Net 5-shot Gain |
|---|---|---|---|---|---|---|
| Few-Shot Progression (Table 5) | LLaMA-3.1-8B | 43.35 / 43.89 | 45.00 / 45.15 | 46.16 / 46.38 | 46.66 / 46.17 | +3.32 / +2.28 |
| Gemma-3-4B-IT | 43.98 / 44.91 | 45.97 / 46.45 | 46.13 / 46.40 | 46.31 / 46.04 | +2.33 / +1.13 | |
| Qwen3-8B | 51.02 / 48.73 | 50.30 / 51.91 | 52.33 / 53.31 | 53.27 / 53.56 | +2.25 / +4.83 | |
| Qwen3-4B-Instruct-2507 | 50.92 / 51.80 | 52.64 / 52.14 | 53.49 / 52.97 | 53.72 / 53.00 | +2.80 / +1.20 | |
| 4-Model Mean | 47.32 / 47.33 | 48.48 / 48.91 | 49.53 / 49.77 | 49.99 / 49.69 | +2.67 / +2.36 |
The table below details cross-benchmark accuracy gaps on 9 shared subjects (\(\Delta = U - P\)):
| Broad Domain | Shared Subject | Items (U / P) | Gemma-4-31B-IT (U / P / \(\Delta\)) | Qwen3.6-27B (U / P / \(\Delta\)) | LLaMA-3.1-8B (U / P / \(\Delta\)) |
|---|---|---|---|---|---|
| STEM | Mathematics | 876 / 242 | 93.6 / 68.3 / +25.3* | 88.9 / 61.2 / +27.8* | 43.4 / 31.4 / +12.0* |
| Physics | 621 / 127 | 95.7 / 80.0 / +15.7* | 94.8 / 79.2 / +15.6* | 43.6 / 36.2 / +7.4 | |
| Chemistry | 936 / 115 | 92.5 / 80.7 / +11.8* | 92.9 / 81.7 / +11.2* | 42.2 / 40.9 / +1.3 | |
| Biology | 931 / 139 | 94.0 / 91.7 / +2.3 | 88.8 / 92.4 / -3.5 | 46.0 / 48.1 / -2.1 | |
| Computer Science | 823 / 62 | 94.4 / 93.9 / +0.5 | 93.1 / 91.3 / +1.8 | 51.8 / 45.8 / +6.0 | |
| Social Sciences | Economics | 907 / 358 | 84.7 / 86.3 / -1.6 | 81.5 / 85.9 / -4.4 | 49.3 / 40.3 / +8.9* |
| Sociology | 778 / 52 | 91.1 / 82.7 / +8.4 | 89.3 / 79.2 / +10.1 | 68.9 / 43.8 / +25.0* | |
| Geography | 630 / 65 | 87.0 / 82.8 / +4.2 | 84.3 / 82.5 / +1.8 | 53.3 / 39.4 / +13.9* | |
| Psychology | 460 / 169 | 90.4 / 85.2 / +5.2 | 85.0 / 83.3 / +1.7 | 44.3 / 41.4 / +2.9 | |
| Macro Average | Macro Average | 6,962 / 1,329 | 91.5 / 83.5 / +8.0 | 88.7 / 81.8 / +6.9 | 49.2 / 40.8 / +8.4 |
Key Findings¶
- The Structural STEM–Humanities Performance Chasm: Across virtually all capable models, accuracy on STEM subjects approaches near-saturation while performance on Urdu-centered Humanities exhibits a massive 20 to 40 point collapse. Under the Urdu prompt, Gemini-3.5-Flash drops from 97.75% on STEM to 85.49% on Humanities (a 12.26-point deficit); DeepSeek-V4-Flash collapses from 97.36% on STEM to 67.30% on Humanities (a 30.06-point plunge); Qwen3-8B plunges by 43.28 points (74.05% vs. 30.77%). This demonstrates that multilingual pretraining successfully transfers universal scientific reasoning expressed in Urdu terminology, but leaves massive blind spots regarding regional literary traditions, prosody, and religious/civic knowledge.
- Prompt Language Decouples Accuracy from Format Robustness: For top proprietary and \(\ge\)25B open models, instruction language has a negligible impact on raw accuracy (e.g., Gemini-3.5-Flash shifts by only +0.22%, DeepSeek-V4-Flash by -1.12%). However, for smaller open models, Urdu prompting triggers catastrophic decoding failures: BLOOMZ-3B's invalid-output rate explodes to 74.2% under Urdu instructions (yielding an overall score of 7.40%), and Urdu-tuned models like Qalb-1.0-8B and Alif-1.0-8B suffer 11.3% and 12.6% invalid rates, demonstrating severe format-following degradation in non-English settings.
- Underperformance of Urdu-Specialized Models: Domain-specific fine-tuned models such as Qalb-1.0-8B (31.54% overall) and Alif-1.0-8B (28.62% overall) fail to outperform general-purpose models of identical scale. Even on the indigenous Urdu Literature subdomain, Ministral-3-8B achieves 39.4% accuracy under both prompts, decisively beating Qalb (31.2%) and Alif (27.2%). This highlights that current low-resource continuous pretraining and synthetic tuning often degrade general reasoning without instilling deep factual mastery.
- The Psychometrics Bottleneck: Psychometrics (PMT) proved to be the most challenging subdomain in the benchmark; not a single evaluated model achieved 60% accuracy under either prompt language (Gemini-3.5-Flash topped at 57.30% under English prompts, and Claude-Sonnet-4.6 led with 52.97% under Urdu prompts). Because secondary-level psychometrics tests analogical reasoning, numeric sequences, and pattern deduction in Urdu, it exposes persistent reasoning bottlenecks that factual memorization cannot resolve.
- Marginal Utility of Few-Shot In-Context Learning: Increasing exemplars from 0-shot to 5-shot yields modest average gains of only +2.67 points in English and +2.36 points in Urdu across four open-weight models. Performance gains saturate by 3-shot, demonstrating that in-context demonstrations reinforce formatting compliance but cannot inject missing regional world knowledge.
Highlights & Insights¶
- Native Curriculum Grounding Over Mechanical Translation: By utilizing genuine past-papers and native online testing banks from Pakistani SSC and HSSC education, UrduMMLU uncovers profound linguistic and cultural knowledge deficits that translated English benchmarks (like MMLU-PROX) systematically obscure.
- Rigorous Multimodal OCR and Dual Blind-Consensus Protocol: The authors set a methodological standard for low-resource non-Latin benchmarks by orchestrating a 16-stage pipeline that combines vision-language OCR extraction, Unicode bidirectional isolation, and dual-annotator blind verification, achieving a high consensus agreement rate of 0.8663.
- Strict All-Item Scoring with Format Audit: By penalizing formatting collapses, repetitions, and conversational refusals as incorrect in the all-item denominator, the benchmark provides an honest, unvarnished audit of model reliability in non-English production environments.
Limitations & Future Work¶
- Geographic and Curricular Boundaries: The benchmark is centered on the Pakistani national curriculum and secondary education boards (SSC/HSSC), omitting Indian Urdu curricula, university-level curricula, non-standard regional dialects, and colloquial Urdu-English code-switching.
- Multiple-Choice Constraint: Restricted to 4-option MCQs, the benchmark cannot evaluate generative capabilities such as creative Urdu poetry composition, essay summarization, or dialogue coherence.
- Future Directions: Extending evaluation to open-ended generation and multi-turn complex reasoning, expanding coverage to pan-South Asian Urdu varieties, and leveraging UrduMMLU's diagnosis to drive native pretraining on high-quality literary and academic Urdu archives.
Related Work & Insights¶
- vs. MMLU-PROX / GLOBAL-MMLU / INDICMMLU-PRO: These benchmarks translate Western questions into Urdu, testing translated universal concepts. UrduMMLU uses native questions authored by Pakistani educators, incorporating authentic literary history, classical poetic structures, and regional civic content.
- vs. URDUBENCH: While UrduBench translates English reasoning benchmarks (e.g., MGSM, MATH-500) into Urdu, UrduMMLU provides an authentic multi-subject evaluation across the complete natural and social sciences curriculum.
- vs. ARABICMMLU / CMMLU / INDOMMLU: UrduMMLU extends the native benchmark paradigm to South Asian Perso-Arabic scripts, offering an advanced technical blueprint for bidirectional layout isolation and dual-annotator quality control.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering native multitask benchmark for Urdu grounded in real national secondary examination curricula.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation of 30 state-of-the-art models across dual prompt languages, few-shot extensions, and cross-benchmark comparisons.
- Writing Quality: ⭐⭐⭐⭐⭐ Outstanding clarity, thorough documentation of annotation metrics, and exhaustive statistical reconciliation.
- Value: ⭐⭐⭐⭐⭐ Foundational evaluation infrastructure for advancing linguistically inclusive and culturally grounded AI for 230M+ Urdu speakers.