ESGPT: A Data-Centric ESG Language Model with Unified Instruction and Benchmark Resources¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/nowzer0/ESGPT
Area: LLM Reasoning
Keywords: ESG language models, data-centric learning, instruction tuning, complex reasoning evaluation, domain benchmark
TL;DR¶
To tackle fragmentation in ESG datasets and non-standardized evaluation criteria, this paper develops ESG-Instruct (a 30K balanced instruction tuning dataset) and ESGU-Bench (a benchmark spanning 6 core understanding tasks), training an 8B model ESGPT that outperforms large models like GPT-4.1 and DeepSeek-V4 in ESG reasoning.
Background & Motivation¶
Environmental, Social, and Governance (ESG) criteria have become indispensable metrics for evaluating corporate sustainability and long-term viability. ESG text analysis spans corporate annual disclosures, sustainability reports, regulatory filings, financial news, and global disclosure standards (such as SASB, TCFD, and IFRS). Interpreting this content demands nuanced domain-specific reasoning that goes far beyond surface-level keyword spotting or standard text classification; it requires identifying ESG relevance, parsing non-explicit corporate action commitments, diagnosing operational risk severity, and executing multi-hop factual verification over complex financial disclosures.
Despite the rapid emergence of general-purpose Large Language Models (LLMs), existing models display substantial variance and frequent failures when evaluated on specialized ESG tasks. The fundamental obstacle lies in the severe fragmentation of public ESG resources. Prior datasets are scattered across different languages, incompatible input/output structures, heterogeneous annotation granularities, and ad-hoc task definitions. Moreover, because creating high-quality ESG corpora historically required restricted source access and costly financial domain expertise, the field has lacked a unified instruction foundation for domain adaptation and a standardized, contamination-free evaluation benchmark.
Relying merely on model scale or general API prompts cannot overcome these structural deficits, nor can it ensure the evidence-grounded conservatism necessary for corporate disclosure. The core idea is to implement a systematic data-centric pipeline that consolidates fragmented public resources into a unified six-category ESG taxonomy, constructing a balanced instruction set (ESG-Instruct) and an expert-reviewed benchmark (ESGU-Bench) to train an evidence-faithful 8B specialist model (ESGPT).
Method¶
Overall Architecture¶
The proposed framework encompasses an end-to-end data-centric pipeline comprising four primary phases: multi-source public dataset collection and multi-stage quality control, unified taxonomy alignment across six core ESG capabilities, balanced instruction tuning dataset synthesis (ESG-Instruct), and contamination-free benchmark construction (ESGU-Bench). The resulting dataset trains ESGPT via Parameter-Efficient Fine-Tuning (PEFT/LoRA), followed by rigorous comparative evaluation against 35 state-of-the-art open-source and proprietary LLMs.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Source Public ESG Datasets<br/>Disclosures/News/Standards/Multilingual Corpora"] --> B["Multi-Stage Quality Control & Translation<br/>Gemini 3.1 Flash-Lite with nuance-preserving prompt"]
B --> C["Unified Six-Category Taxonomy Mapping<br/>Harmonized input structures & valid JSON schemas"]
C --> D["Balanced Instruction Set (ESG-Instruct)<br/>30K instances: 35 tasks + 10% Alpaca general chat"]
D --> E["Data-Centric Specialist Adaptation (ESGPT)<br/>Qwen3-8B base + targeted all-linear LoRA modules"]
C --> F["Expert-Reviewed Benchmark (ESGU-Bench)<br/>9,514 instances / 21 tasks with zero test leakage"]
E --> G["Holistic Evaluation & Disclosure Generation<br/>Category accuracy breakdown + TCFD fidelity audit"]
F --> G
Key Designs¶
1. Multi-Stage Quality Control and Nuance-Preserving Translation: Standardizing Heterogeneous Sources Public ESG corpora are intrinsically multilingual, with substantial high-quality resources available only in Korean, French, or Chinese. To bridge linguistic divides without corrupting sensitive disclosure nuances, the authors instituted a rigorous multi-stage quality pipeline. For 129,745 non-English instances, translation was conducted using Gemini 3.1 Flash-Lite guided by a dedicated prompt engineered for sustainability and financial reporting. The prompt strictly forbade summarization, fact expansion, and arbitrary ambiguity removal, enforcing exact retention of proper nouns, quantitative values, temporal markers, and verification predicates. Post-translation processing removed duplicate records, repaired malformed schema keys, resolved label discrepancies, and excluded unverified entries (accumulating over 160 GPU hours and $187 in translation expense).
2. Unified Six-Category Taxonomy and Structured JSON Prompting: Harmonizing Diverse Task Spaces To integrate 35 disparate subtasks under a cohesive capability spectrum, the authors established a unified six-category ESG taxonomy: 1. ESG Relevance Detection: Determining whether input statements substantively touch environmental, social, or governance themes; 2. ESG Topic Identification: Granular categorization into primary pillars, SASB issue topics, or TCFD recommendations; 3. ESG Commitment Analysis: Distinguishing general rhetorical statements from binding action pledges, net-zero targets, and verification horizons; 4. ESG Impact Analysis: Classifying event sentiment, opportunity vs. risk framing, impact duration, and severity tier; 5. ESG Claim Verification: Assessing whether factual claims are supported, refuted, or unevidenced by provided source excerpts; 6. ESG Question Answering: Grounded extractive and descriptive reasoning across sustainability reports and data tables. Every task is formalized into a chat-prompt template that mandates deterministic, parsable JSON outputs conforming to designated schemas.
3. Balanced Class Sampling and Zero-Leakage Contamination Guard: Building Reliable Resources Naively pooling raw training instances induces severe representation bias toward data-heavy tasks at the expense of difficult reasoning tasks. In synthesizing the 30K ESG-Instruct dataset, instance counts were systematically calibrated across all six categories: Relevance Detection (17.2%, 5,160), Topic Identification (17.2%, 5,170), Impact Analysis (17.2%, 5,170), Question Answering (17.2%, 5,164), Commitment Analysis (15.8%, 4,736), and Claim Verification (5.3%, 1,600), augmented with 10.0% (3,000) general instruction samples from Stanford Alpaca to preserve instruction followability. For ESGU-Bench, 9,514 human-annotated or expert-reviewed test instances across 21 tasks were isolated. Five levels of contamination screening were conducted: exact input match, input-target pair match, intra-category 3-gram and 5-gram Jaccard similarity (threshold \(\ge 0.80\)), and document identifier overlap. All checks returned zero leakage.
4. Evidence Boundary Preservation and Report Generation: Preventing Greenwashing Hallucinations In corporate reporting, unsubstantiated claims constitute greenwashing liability. During TCFD-style corporate disclosure generation, general-purpose LLMs frequently extrapolate broader governance structures (e.g., general anti-corruption programs) into unsupported climate oversight assertions. ESGPT enforces strict evidence grounding. When analyzing disclosures with incomplete climate reporting (such as Petrobras NYSE_PBR_2016), ESGPT accurately reflects the negative space of evidence—explicitly stating that Scope 3 emissions or specific climate metrics are omitted—rather than manufacturing synthetic compliance.
Loss & Training¶
ESGPT adopts Qwen3-8B (with parallel exploration on Llama-3.1-8B-Instruct) as the base architecture, trained with Low-Rank Adaptation (LoRA).
- Target Modules: Adapters are applied to all Transformer linear projection layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.
- LoRA Configuration: Rank \(r = 64\), scaling factor \(\alpha = 128\), and dropout rate of 0.05. Maximum input context length is 4096 tokens.
- Optimization Strategy: Fine-tuned for 3 epochs using the AdamW optimizer in bfloat16 precision. The per-device batch size is 1 with 16 gradient accumulation steps (effective batch size of 16). The initial learning rate is \(1 \times 10^{-4}\) governed by a cosine learning rate scheduler with a 3% warmup ratio and zero weight decay. Training completes on a single NVIDIA A100 GPU in approximately 10 hours and 16 minutes for the 30K dataset.
Key Experimental Results¶
Main Results¶
On ESGU-Bench, evaluated across 35 competitive models spanning 0.5B to 2.2T parameters in a zero-shot setting, ESGPT achieves the top overall accuracy of 66.96% (5-seed mean 66.96% ± 0.10%), significantly outperforming the second-best model DeepSeek-V4-Flash under a two-tailed t-test (\(p < 0.001\)).
| Model Type | Model Name | Size | Relevance Detect | Topic ID | Commitment Anal | Impact Anal | Claim Verif | Domain QA | Overall Acc (%) |
|---|---|---|---|---|---|---|---|---|---|
| Open Source LLM | Qwen2.5-72B | 72B | 76.77 | 37.49 | 62.33 | 68.02 | 55.89 | 61.88 | 60.40 |
| Open Source LLM | Qwen3-8B (Base) | 8B | 73.28 | 43.61 | 65.70 | 59.14 | 53.64 | 60.21 | 59.26 |
| Open Source LLM | Llama-3.1-8B-Instruct | 8B | 75.26 | 37.28 | 67.00 | 56.66 | 53.21 | 62.50 | 58.65 |
| Open Source LLM | Llama-3.3-70B | 70B | 81.87 | 38.16 | 63.62 | 66.42 | 55.50 | 65.58 | 61.86 |
| Open Source LLM | Gemma-3-27B | 27B | 80.36 | 35.23 | 67.10 | 65.56 | 58.93 | 45.95 | 58.85 |
| Distilled Reasoner | DeepSeek-R1-Distill-Qwen-14B | 14B | 47.88 | 12.13 | 32.96 | 22.84 | 9.42 | 62.85 | 31.35 |
| Proprietary API | DeepSeek-V4-Flash | 284B | 82.63 | 48.87 | 66.26 | 71.11 | 58.42 | 67.69 | 65.83 |
| Proprietary API | DeepSeek-V4-Pro | 1600B | 79.70 | 46.82 | 66.26 | 66.79 | 56.32 | 68.05 | 63.99 |
| Proprietary API | GPT-4.1 | 2200B† | 81.30 | 46.82 | 68.67 | 65.06 | 47.79 | 69.19 | 63.14 |
| Proprietary API | GPT-5.1 | 1300B† | 79.13 | 49.92 | 67.66 | 62.59 | 45.52 | 72.89 | 62.95 |
| Domain Baseline | SusGen-GPT-Llama | 8B | 67.61 | 37.70 | 62.00 | 53.70 | 56.66 | 57.66 | 55.89 |
| Domain Baseline | SusGen-GPT-Mistral | 7B | 71.20 | 34.10 | 58.52 | 62.84 | 52.98 | 61.44 | 56.85 |
| Proposed Model | ESGPT (Ours) | 8B | 84.32 | 54.94 | 73.37 | 65.93 | 59.40 | 63.82 | 66.96 |
Ablation Study: Training Data Scale Across Base Models¶
The table below details performance trajectories as instruction training instances scale from 10K to the full untrimmed pool (~130K) across Qwen3-8B and Llama-3.1-8B-Instruct base models:
| Base Model | Scale Variant | Training Time | Relevance Detect | Topic ID | Commitment Anal | Impact Anal | Claim Verif | Domain QA | Overall Acc (%) |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | ESGPT-10K | 03h 05m 49s | 81.68 | 53.72 | 69.28 | 63.58 | 60.34 | 62.15 | 65.13 |
| Qwen3-8B | ESGPT-20K | 06h 41m 06s | 82.53 | 53.93 | 69.62 | 63.95 | 60.39 | 63.12 | 65.59 |
| Qwen3-8B | ESGPT-30K (Ours) | 10h 16m 24s | 84.32 | 54.94 | 73.37 | 65.93 | 59.40 | 63.82 | 66.96 |
| Qwen3-8B | ESGPT-50K | 17h 26m 58s | 84.80 | 55.69 | 71.69 | 66.30 | 59.01 | 64.52 | 67.00 |
| Qwen3-8B | ESGPT-Full | 146h 29m 35s | 84.23 | 54.60 | 71.19 | 64.20 | 50.15 | 41.73 | 61.02 |
| Llama-3.1-8B | ESGPT-10K | 03h 16m 39s | 81.78 | 49.21 | 69.06 | 61.36 | 55.55 | 58.10 | 62.51 |
| Llama-3.1-8B | ESGPT-20K | 07h 20m 15s | 82.25 | 51.72 | 70.35 | 62.35 | 56.45 | 59.68 | 63.80 |
| Llama-3.1-8B | ESGPT-30K | 11h 23m 51s | 82.53 | 52.72 | 71.97 | 63.83 | 53.23 | 61.18 | 64.24 |
| Llama-3.1-8B | ESGPT-50K | 19h 31m 03s | 84.14 | 52.80 | 70.63 | 67.04 | 53.88 | 59.86 | 64.72 |
| Llama-3.1-8B | ESGPT-Full | 164h 02m 13s | 83.38 | 45.69 | 68.33 | 62.84 | 42.61 | 33.01 | 55.98 |
Key Findings¶
- Data Scaling Inverted-U Phenomenon: Simply ingesting all raw instances without category balance (ESGPT-Full) causes acute performance regression (falling to 61.02% on Qwen3 and 55.98% on Llama-3.1), with Question Answering collapsing from 63.82% to 41.73%. Oversaturating training on abundant superficial classification corpora drowns out complex evidence-verification tasks.
- Remarkable 10K Sample Efficiency: Tuning on just 10K balanced instruction pairs for 3 GPU hours yields 65.13% accuracy, surpassing frontier proprietary systems GPT-4.1 (63.14%) and GPT-5.1 (62.95%).
- Domain Specialization vs. General Reasoning Divergence: Giant closed-source models exhibit strong prior knowledge in open-ended question answering (GPT-5.1 leads with 72.89%) and generalized impact identification (DeepSeek-V4-Flash leads with 71.11%). However, on framework-constrained tasks like Topic Identification (ESGPT leads by +5.02% over GPT-5.1) and Commitment Analysis (ESGPT leads by +4.70% over GPT-4.1), specialized instruction tuning provides decisive advantages.
Highlights & Insights¶
- Curated Quality Outweighs Brute-Force Volume: In complex regulatory domains, fine-grained task balancing and denoising are far more decisive than dataset size; 30K balanced examples on a single GPU beat massive uncurated pools requiring 146+ GPU hours.
- Strict Evidence Grounding Prevents Greenwashing: By penalizing unsupported speculative leaps in instruction tuning, ESGPT preserves evidence boundaries in sustainability reporting, avoiding false compliance declarations common in general LLMs.
- Contamination-Proof Benchmark Baseline: ESGU-Bench sets a rigorous benchmark standard across six core capabilities with zero lexical or n-gram overlap with the training set.
Limitations & Future Work¶
- English-Centric Evaluation Scope: While training utilizes non-English data via translation, both benchmark evaluation and target disclosures remain English-dominant; jurisdiction-specific regulatory dialects (e.g., EU CSRD vs. SEC mandates) require future multilingual expansion.
- Hardware-Constrained Parameter Footprint: ESGPT is developed solely on an 8B base with LoRA due to single-A100 compute constraints; scaling to full-parameter tuning on 70B+ architectures remains unexamined.
- Decision-Support Boundary: Because regulatory interpretations carry legal liabilities, ESGPT is designated strictly as an assistive auditor rather than an autonomous substitute for certified human ESG reviewers.
Related Work & Insights¶
- vs SusGen-GPT: SusGen-GPT focuses on sustainability report drafting via financial NLP, but exhibits hallucination tendencies when evidence is weak; ESGPT prioritizes verification fidelity and category-balanced reasoning.
- vs ESGBERT / ClimateBERT: Earlier BERT-era models are restricted to discriminative classification on isolated sentences; ESGPT provides a unified generative interface spanning classification, verification, and grounded QA.
- vs ESGenius / SustainableQA: Prior benchmarks targeted isolated QA or knowledge retrieval; ESGU-Bench integrates these into a comprehensive 6-category evaluation ecosystem.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Establishes a rigorous data-centric pipeline and unified taxonomy for previously disjointed ESG tasks.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 35 frontier models across 6 categories, supported by rigorous scale ablations, multi-seed validation, and TCFD report audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured with clear task definitions, detailed contamination checks, and candid limitation analysis.
- Value: ⭐⭐⭐⭐⭐ Provides invaluable public open-source resources (ESG-Instruct, ESGU-Bench, ESGPT) accelerating AI research in sustainable finance and corporate disclosure.