Skip to content

Jais 2: A Family of Arabic-Centric Open Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/collections/inception42/jais-2-family
Area: Large Language Models / Arabic-Centric Open Large Language Models
Keywords: Jais 2, Arabic Large Language Models, Maximal Update Parameterization (µP), Cultural Alignment, Direct Preference Optimization (DPO)

TL;DR

Jais 2 introduces open-weight, bilingual Arabic-centric foundation models in 8B and 70B parameter scales (the largest trained from scratch), utilizing an expanded 8× FFN ratio, \(\text{ReLU}^2\) activations, a custom 150K BPE vocabulary, and a comprehensive culturally grounded post-training pipeline to set new state-of-the-art results across OALL2, AraGen, and native cultural evaluations.

Background & Motivation

Despite the swift evolution of generative large language models, global foundation architectures (such as Llama 3, Qwen2.5, and Gemma-3) remain predominantly centered around English and a handful of Latin-script, high-resource languages. Arabic—a language spoken by over 400 million people across diverse regions—presents intricate morphological complexity, pervasive diglossia where Modern Standard Arabic (MSA) coexists with distinct regional colloquial dialects, and rich cultural traditions. Because global web corpora dedicate only a minor fraction of training tokens to Arabic, mainstream foundation models suffer from severe subword over-fragmentation, surface-level linguistic comprehension, and hallucinations when handling dialectal subtleties or cultural nuances.

Early bilingual initiatives, including Jais, AceGPT, ALLaM, and Fanar, demonstrated the viability of balanced Arabic-English pretraining and instruction tuning. Concurrently, regional projects such as Nile-Chat (for Egyptian Arabic) and Atlas-Chat (for Moroccan Darija) highlighted the necessity of modeling vernacular speech directly. However, existing open systems remained fundamentally constrained: they were either limited to smaller parameter envelopes (7B–13B), continually fine-tuned on non-Arabic base architectures with severe catastrophic forgetting, or restricted to narrow domains. Crucially, cultural domains deeply woven into Arabic life—such as classical poetic meter analysis, Islamic jurisprudence (fiqh), and symbolic dream interpretation—lacked rigorously curated, contamination-free benchmarks and systematic alignment strategies.

To bridge these foundational gaps, this work presents Jais 2, a family of open-weight bilingual models trained from scratch, featuring an efficient 8B model alongside a 70B variant—the largest open Arabic-centric model built from the ground up to date. The core idea is to combine architectural co-design (an 8× intermediate FFN ratio, \(\text{ReLU}^2\) activations, and µP hyperparameter transfer) with a tailored 150K vocabulary and a unified training curriculum spanning multi-stage pretraining with domain upsampling, continual pretraining (CPT), culturally grounded SFT, and DPO alignment.

Method

Overall Architecture

Jais 2 adopts a standard decoder-only autoregressive Transformer architecture configured in two parameter sizes: 8B (32 layers, hidden size 3,328) and 70B (68 layers, hidden size 7,168). The end-to-end development pipeline progresses through five systematic stages: first, Stage 1 pretraining on 2.2T multilingual tokens (including 624B purpose-curated Arabic tokens); second, Stage 2 annealing with domain-specific upsampling on the final 5%–10% compute budget; third, Continual Pretraining (CPT) with 50% replay to inject domain depth without catastrophic forgetting; fourth, culturally grounded Instruction Fine-Tuning (IFT) across 20 million pairs; and finally, Direct Preference Optimization (DPO) to enforce human alignment and regional safety.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["2.2T Multilingual Pretraining Corpus<br/>624B Arabic + Web / Math / Code"] --> B["Custom 150K BPE Tokenizer<br/>31.25% Weighted Arabic Sampling"]
    B --> C["High-Efficiency Transformer Backbone<br/>8× FFN Ratio + ReLU² Activations + µP"]
    C --> D["Two-Stage Pretraining & Annealing<br/>90% Base Pretraining + 10% Domain Upsampling"]
    D --> E["Continual Pretraining (CPT)<br/>Targeted Arabic Datasets + 50% Pretraining Replay"]
    E --> F["Culturally Grounded SFT (IFT)<br/>20M Instruction Pairs: Fiqh / Poetry / Dreams / 17 Dialects"]
    F --> G["Direct Preference Optimization (DPO)<br/>Implicit Reward Optimization (β=0.1)"]
    G --> H["Open-Weight Jais 2 Family<br/>8B & 70B Base and Chat Checkpoints"]

Key Designs

1. Custom 150K BPE Tokenizer: Alleviating Subword Fragmentation and Preserving Morphology

Standard English-centric tokenizers split morphologically rich Arabic words (which combine prefixes, root patterns, suffixes, and clitics) into excessively long sequences of rare subwords or individual bytes, severely degrading effective context length and computational throughput. Jais 2 introduces a dedicated Byte-Pair Encoding (BPE) tokenizer with a vocabulary size of 150,272 tokens constructed using Hugging Face tokenizers. Training data sampling enforces a 31.25% allocation to Arabic, 20.83% to English, 20.83% to French, 4.17% each to German, Spanish, Italian, and Hindi, and ~1% each to 10 major programming languages (such as Python, Rust, Swift, and C++). Coupled with regex rules that segment contractions and maintain byte alignment while preserving space-prefixed tokens, the tokenizer achieves substantial token compression across Classical Arabic, MSA, mixed Latin-script Arabizi, and colloquial dialects.

2. High-Efficiency Transformer Backbone: 8× FFN Ratio, \(\text{ReLU}^2\) Activation, and Zero-Shot µP Scaling

To maximize model parameter capacity under finite pretraining compute, Jais 2 widens the Feed-Forward Network (FFN) intermediate dimension to 8 times the hidden dimension (26,624 for 8B and 57,344 for 70B)—more than double the ratio found in Llama 3 and Gemma-3. To mitigate the computational overhead of wider projections, the architecture replaces standard SwiGLU/GELU with squared rectified linear unit activations: $\(f(x) = (\max(0, x))^2\)$ \(\text{ReLU}^2\) induces high natural activation sparsity across the wide FFN layers, creating an optimal trade-off between expressive capacity and computational sparsity. Furthermore, the architecture adopts Maximal Update Parameterization (µP) while keeping the width-to-depth ratio near the empirically optimal value of 100. Using a 100M parameter proxy model trained for 20 Tokens-Per-Parameter (TPP), optimal hyperparameters (base learning rate \(\eta=0.0248\), base initialization standard deviation \(\sigma=0.035\), token embedding scaling 67.78, and output unembedding scaling 0.42) are transferred zero-shot to the 70B scale without running prohibitively costly full-scale sweeps.

3. Two-Stage Pretraining & Annealing: Resolving the Arabic-English-Reasoning Pareto Frontier

The foundation models are trained on 2.2T tokens. Stage 1 utilizes over 90% of the pretraining budget using a carefully ablated data mixture: 20% Arabic, 40% English Web, 20% Math, 10% Code, and 10% Synthetic data (Mix 1, which proved superior to candidate Mix 2 across both 2.7B and 6.7B scaling experiments). In Stage 2 annealing (consuming 5%–10% of FLOPs with learning rate decay), the team evaluated multiple domain upsampling configurations. Pushing Arabic upsampling to an extreme 80% (US Mix 2) raised Arabic average accuracy to 43.00% but caused English to drop to 46.99% and GSM8K math accuracy to plunge to 2.43%. Jais 2 settles on a balanced upsampling schedule (US Mix 4) that lifts Arabic proficiency while safeguarding mathematical and code generation skills.

4. Continual Pretraining (CPT): Specialized Knowledge Infusion with Replay Defense

Following pretraining, Jais 2 enters a post-training CPT stage for two epochs, mixing newly curated domain data with a 50% replay of the original pretraining corpus to guard against catastrophic forgetting. CPT targets two objectives: deepening specialized native knowledge in religion, literature, and 17 regional dialects (totaling over 600M dialectal tokens), and reinforcing reasoning weaknesses identified in preliminary evaluations through synthetic textbook content and explanation-enriched multiple-choice reasoning corpora.

5. Culturally Grounded SFT (IFT): Deep Immersion in Fiqh, Poetic Meters, and Symbolic Hermeneutics

To transition beyond generic chat assistants, Jais 2 trains on over 20 million instruction-completion pairs over three epochs, highlighting three culturally native datasets designed with strict train-test separation: - Islamic Jurisprudence (Islamic QA): Scraped and filtered from Islamweb (151,890 cleaned pairs), eliminating formulaic greetings and external URLs while meticulously retaining Qur'anic verses and Hadith citations. 1,000 complex pairs across 20 core fiqh categories were held out exclusively for benchmarking. - Arabic Poetry IFT: Encompassing 35 subtasks spanning Analysis (predicting poet, era, genre, rhyme, and meter from poem text) and Generation (composing verses following classical meters like Al-Tawil or Al-Rajaz), with 427K training and 6.9K testing samples. - Dream Interpretation: Curated from 5 classical hermeneutic authorities (Ibn Sīrīn, Al-Nābulsi, etc.) into 5,568 pairs, partitioned strictly at the symbolic entity level (4,466 for training, 1,102 held out) to evaluate symbolic associative reasoning without leakage.

6. Direct Preference Optimization (DPO): Reward-Model-Free Cultural and Safety Alignment

In the final post-training stage, Jais 2 applies Direct Preference Optimization (DPO) to align the model with human intent and regional ethical norms. Using the SFT checkpoint as reference policy \(\pi_{\text{ref}}\), DPO optimizes the policy \(\pi_\theta\) directly over 200K paired preference tuples \((q, y_w, y_l)\) spanning general dialogue, Arabic dialectal interaction, mathematics, and red-teaming safety: $\(\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(q, y_w, y_l)\sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|q)}{\pi_{\text{ref}}(y_w|q)} - \beta \log \frac{\pi_\theta(y_l|q)}{\pi_{\text{ref}}(y_l|q)} \right) \right]\)$ Hyperparameter tuning identified optimal performance at learning rate \(4.0\times 10^{-5}\), batch size 160, and \(\beta=0.1\), markedly boosting Arabic IFEval strict instruction following and open-ended conversation quality.

Loss & Training

Pretraining is driven by the AdamW optimizer with \(\beta_1 = 0.9, \beta_2 = 0.95\), weight decay 0.1, and \(\epsilon = 10^{-15}\) (lowered from \(10^{-9}\) to prevent the denominator from dominating when EMA squared gradient values are exceptionally small). The learning rate warms up linearly to 0.0248 and decays linearly to 0. Rotary Position Embeddings (RoPE) are used with a base frequency of 500,000, supporting a maximum training context window of 8,192 tokens. Batch sizes were set to 408 for 8B and scaled from 960 to 1,920 for 70B guided by Gradient Noise Scale (GNS) analysis.

Key Experimental Results

Main Results: Comprehensive Evaluation Across Core Arabic Capabilities

The evaluation benchmarks Jais 2 against open models in multiple-choice reasoning (OALL2), generative evaluation (AraGen 3C3H), instruction adherence (Arabic IFEval), Islamic QA, cultural knowledge, dialect identification, poetry analysis, dream interpretation, and summarization. Table 3 presents the compact comparison against the strongest open-weight peer in each parameter bracket:

Evaluation Dimension / Benchmark Jais 2-8B Strongest Peer (≤13B) Jais 2-70B Strongest Peer (>13B) 70B Advantage
OALL2 (Macro-average %) 72.40 Hala-9B (70.02) 79.36 Llama-3.3-70B (74.23) +5.13
AraGen (3C3H Score %) 58.64 Fanar / ALLaM (53.16) 70.71 Qwen2.5-72B (62.58) +8.13
Arabic IFEval (Strict Instruction-level %) 67.09 C4AI-Arabic-7B (70.57) 74.53 Qwen2.5-72B (74.05) +0.48
Islamic Question Answering (Average %) 63.82 ALLaM-7B (66.38) 75.99 Qwen2.5-72B (76.29) -0.30
Arabic Cultural Benchmarks (%) 57.31 Yehia-7B (60.37) 69.64 Qwen2.5-72B (68.56) +1.08
Arabic Dialect Identification (MADAR/QADI %) 44.24 ALLaM-7B (48.11) 59.78 Gemma-3-27B (50.04) +9.74
Arabic Poetry Analysis (14-subtask avg %) 60.44 Fanar-9B (60.07) 69.02 Qwen2.5-72B (61.48) +7.54
Arabic Dream Interpretation (Native %) 67.60 ALLaM-7B (62.89) 85.39 Falcon-H1-34B (62.34) +23.05
Summarization (ROUGE-LSum) 26.04 SILMA-9B (13.86) 31.18 Llama-3.1-70B (13.88) +17.30

Ablation Study: Data Upsampling Mixes During Annealing (Table 2)

Ablation experiments conducted on a 2.7B proxy model at 20 TPP budget, upsampling on the final 10% of training steps:

Variant Arabic Web Math Code Misc. Avg. Arabic (%) Avg. English (%) GSM8K (8-shot)
Baseline 0.27 0.40 0.14 0.10 0.09 42.52 47.50 3.36
US Mix 1 0.40 0.27 0.15 0.10 0.08 42.74 47.63 2.12
US Mix 2 (Extreme Arabic) 0.80 0.07 0.05 0.05 0.03 43.00 46.99 2.43
US Mix 3 (Extreme Math) 0.05 0.10 0.80 0.05 0.00 41.35 46.64 3.34
US Mix 4 (Balanced) 0.20 0.24 0.40 0.15 0.01 42.66 47.71 2.35

Key Findings

  • Consistent Scaling Dividend: While Jais 2-8B leads on 6 of 10 metrics within its parameter bracket (displaying minor deficits in dialect ID and Islamic QA against ALLaM), Jais 2-70B achieves clear dominance on 9 of 10 metrics, demonstrating that architectural and data alignment choices scale exceptionally well to 70B parameters.
  • The Domain Annealing Trade-off: Extreme single-language upsampling (80% Arabic in US Mix 2) induces immediate degradation on English and mathematical benchmarks; a balanced ratio during annealing successfully anchors target language fluency without sacrificing formal reasoning.
  • Massive Gains in Culturally Grounded Tasks: Jais 2-70B establishes unprecedented leads in non-trivial cultural domains, outperforming Qwen2.5-72B and Llama-3.3-70B by 23.05 points in dream interpretation, 17.30 points in summarization ROUGE-LSum, and 9.74 points in dialect identification.

Highlights & Insights

  • Synergy of 8× FFN and \(\text{ReLU}^2\) Sparsity: Combining an exceptionally wide FFN intermediate layer with squared ReLU activations provides higher parameter capacity for multilingual memorization while generating internal activation sparsity that offsets compute overhead.
  • Symbol-Level Train-Test Partitioning: Rather than relying on simple sentence-level deduplication, partitioning data strictly at the "dream symbol" and "jurisprudential subcategory" levels establishes an authentic barrier against data leakage in cultural benchmarking.
  • Ultra-Fast Hardware Inference: Deployed on Cerebras CS-3 systems, the 70B model achieves inference speeds reaching 2,000 tokens per second, making real-time interactive generation practical for large Arabic models.

Limitations & Future Work

  • Proprietary Pretraining Data Restrictions: Due to licensing covenants with third-party providers, the raw 624B Arabic pretraining corpus cannot be fully released, though all post-training recipes and weights are made public.
  • Variability in Low-Resource Dialects: While 17 dialects are represented, long-tail dialects (e.g., specific rural or Bedouin variants) and informal Latinized Arabizi still exhibit occasional generation inconsistencies.
  • English Factual Recall Trade-Off: Compared to models pretrained exclusively on tens of trillions of English tokens, Jais 2 scores lower on purely English extractive question-answering benchmarks such as RACE and DROP.
  • vs Jais (v1) / Jais-chat: Jais 2 expands vocabulary size from 84K to 150K, extends training context from 2K to 8K, switches from ALiBi to RoPE, and introduces two-stage annealing and cultural fine-tuning.
  • vs Qwen2.5-72B / Llama-3.3-70B: Although larger general models maintain an edge in English math and coding, Jais 2-70B creates a substantial performance gap in dialect identification, classical meter analysis, and Arabic native text generation.
  • vs ALLaM / Fanar: While prior regional models are restricted to 7B–9B scales or continually trained on external backbones, Jais 2 provides the first full-scale open 70B model trained from scratch with commercial permissibility.

Rating

  • Novelty: ⭐⭐⭐⭐☆ [8× FFN with \(\text{ReLU}^2\) and µP scaling combined with end-to-end Arabic cultural curation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Broad evaluation across 10+ core benchmarks, thorough ablations, and extensive baseline comparisons]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural narrative, comprehensive appendices, and transparent data documentation]
  • Value: ⭐⭐⭐⭐⭐ [A foundational open-weight contribution for Arabic NLP and a blueprint for training non-English foundation models from scratch]