Skip to content

Evaluating and Improving LLM Self-Modeling

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/safety-research/self-modeling-eval
Area: LLM Evaluation
Keywords: LLM self-modeling, metacognitive evaluation, reinforcement learning, behavioral prediction, prompt perturbation

TL;DR

This paper operationalizes LLM self-modeling as an empirical behavioral prediction capability over verifiable input-output distributions, introduces an unbiased skill benchmark spanning 10 task formats across four domains, and demonstrates that multi-task RL on closed-loop synthetic perturbation data reliably enhances self-modeling skill without requiring privileged internal introspection.

Background & Motivation

Human metacognitive self-modeling exhibits diverse yet systematic imperfections: subjective confidence judgments correlate only moderately with factual task performance, and self-forecasts regarding behavioral adjustments in hypothetical scenarios regularly succumb to self-serving biases. Large language models (LLMs) are now routinely queried on analogous self-reflective dimensions, such as reporting epistemic certainty, predicting whether prompt modifications would alter final conclusions, or diagnosing which components of an input drove their reasoning trajectory. However, even contemporary frontier systemsโ€”including GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.7โ€”display acute failures on elementary counterfactual behavioral queries. In simple mathematical word problems perturbed with minor contradictory premises, models alter their sampled numerical answer in over 90% of trials, yet simultaneously report that the edit would exert zero impact on their final output.

The core tension underlying this deficiency stems from a dual methodological bottleneck in evaluation and post-training. On the evaluation side, existing investigations of model self-knowledge typically conflate behavioral self-prediction with mechanistic introspection or privileged access to hidden internal states, leaving benchmarks without objective empirical ground truth. Furthermore, high class imbalances on discrete tasks permit naive dummy predictors (e.g., constantly predicting "No flip") to achieve high nominal accuracy, disguising a total absence of instance-specific self-modeling ability. On the training side, standard supervised fine-tuning (SFT) over synthetic counterfactual explanations suffers from severe format collapse: auxiliary generator models cannot pinpoint target models' internal decision boundaries without verification, and SFT pulls target models into narrow completion distributions that catastrophically fail to generalize across diverse evaluation formats.

The angle of attack in this paper is to treat self-modeling in a strictly behavioral senseโ€”measuring the empirical concordance between what a model says about its behavior and what its sampled completions actually exhibit across isolated interventions. By grounding ground truth in repeated behavioral sampling, the authors construct an unbiased skill benchmark and couple it with an active generate-verify-revise synthetic data pipeline optimized via reinforcement learning. Core idea: formalize LLM self-modeling as a measurable, verifiable input-output behavioral prediction capability, construct an unbiased multi-format benchmark penalizing trivial baselines, and generalize self-modeling skill across diverse task formats using a closed-loop generate-verify-revise pipeline optimized with format-gated reinforcement learning.

Method

Overall Architecture

The proposed evaluation and alignment framework comprises three sequential components: first, a Multi-Format Verifiable Behavioral Evaluation Suite that computes empirical ground truths from independent model rollouts and quantifies net performance via baseline-adjusted Skill Scores; second, a Closed-Loop Generate-Verify-Revise Synthetic Data Pipeline that decomposes seed prompts and crafts targeted perturbations iteratively validated by the target model's actual sampling; and third, a Format-Gated Decentralized Reinforcement Learning Pipeline that optimizes policy LoRA adapters against precomputed initial ground truths using verifiable format and task rewards with group-mean centered advantages.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Evaluated Target Model & Benchmarks<br/>(GSM8K / HumanEval / BBQ / WildGuardTest)"] --> N1["Multi-Format Verifiable Behavioral Evaluation Suite<br/>Empirical rollout sampling & dummy-predictor baseline calibration"]
    Src["Unseen Source Datasets<br/>(100+ single-turn HF datasets / 20 BLOOM multi-turn tracks)"] --> N2["Closed-Loop Generate-Verify-Revise Synthetic Data Pipeline<br/>Structural decomposition & target model rollout verification"]
    N2 --> N3["Format-Gated Decentralized Reinforcement Learning Pipeline<br/>Format bonus gating & group-relative advantage optimization"]
    N1 -.->|Provides verifiable rewards and evaluation protocol| N3
    N3 --> Out["Optimized LLM with Generalizable Self-Modeling Skill"]

Key Designs

1. Multi-Format Verifiable Behavioral Evaluation Suite: Calibrating Trivial Baselines and Unifying Empirical Ground Truth Evaluating whether a model accurately self-models its behavior is easily compromised when evaluated by raw task accuracy. In binary flip detection (FLIP-DECISION), if a target model alters its decision on only 5% of perturbed prompts, a dummy rule predicting "No" achieves a 95% raw score despite possessing zero input-sensitive metacognition. The evaluation framework resolves this by establishing 10 diverse self-modeling tasks spanning binary choice, multiple-choice, continuous scalar estimation, and open-ended text generation across math, coding, fairness, and safety benchmarks. For each item \(x\) and model \(M\), behavioral ground truth \(g_t(x; M)\) is obtained by independently sampling rollouts under baseline and perturbed prompts, while an isolated meta-prompt \(q_t(x)\) elicits the self-report \(r_t(x; M)\). Raw task performance \(s_t(M; \mathcal{D})\) is mapped against an empirical dummy predictor \(b_t\), yielding the unbiased Skill Score: $\(\text{Skill}_t(M; \mathcal{D}) = s_t(M; \mathcal{D}) - b_t(M; \mathcal{D})\)$ where \(b_t\) represents the marginal empirical mode for categorical tasks, the ground truth mean for continuous scalar tasks (\(1 - \text{Var}(g)\)), and zero for generative tasks. Macro-averaging across groups, seeds, and tasks prevents any single domain or output format from dominating aggregate assessment.

2. Closed-Loop Generate-Verify-Revise Synthetic Data Pipeline: Resolving Boundary Misalignment via Active Target Feedback Constructing effective training data requires identifying prompt edits that genuinely probe a target model's internal decision boundaries without leaking test benchmarks. Because an external auxiliary generator (e.g., Claude Opus 4.5) cannot perfectly foresee another model's fragile beliefs, single-pass prompt generation produces heavily skewed distributionsโ€”severely under-flipping mathematical facts while over-flipping subjective open generation. The synthesis pipeline addresses this through an active feedback loop: a structural decomposer first isolates six elements (content, format, context, assumption, constraint, implicit premise); a generator proposes candidate perturbations intended to be flip-inducing, non-flip, or boundary; the target model executes rollouts under both original and modified prompts; and an automated verifier checks whether the empirical outcome matches the targeted class. Mismatched proposals are iteratively refined for up to five feedback rounds, accompanied by stratified downsampling to balance class proportions across tasks before export.

3. Format-Gated Decentralized Reinforcement Learning Pipeline: Preventing Format Collapse and Driving Cross-Task Generalization Direct supervised fine-tuning (SFT) on self-modeling demonstrations induces severe format collapse: models overfit to specific demonstration syntax and produce unparseable outputs when presented with out-of-distribution (OOD) evaluation queries, dropping strict benchmark scores to zero. To overcome this, the framework implements verifiable reinforcement learning (RLVR) with LoRA parameterization. Ground truth labels \(g_t\) are precomputed once from the frozen base policy \(\pi_{\theta_0}\)โ€”leveraging the finding that LoRA updates maintain high behavioral consistency with the base model, rendering expensive online ground-truth refreshes superfluous. The scalar training reward combines a strict format bonus with the task-specific metric: $\(R(r_t, g_t) = \mathbf{1}[\text{format correct}] \cdot 0.5 + R_{\text{task}}(r_t, g_t)\)$ where \(R_{\text{task}} \in [0, 1]\) directly mirrors the evaluation metric. Updating the policy across \(K=16\) sampled rollouts per prompt with mean-centered group advantages \(A_k = R_k - \bar{R}\) enforces parsing discipline without requiring unstable PPO ratio clipping or KL reference penalties, establishing robust transfer across disparate self-modeling formats.

Loss & Training

The RLVR training objective minimizes an importance-sampled policy gradient loss across task distribution \(\mathcal{D}_{\text{train}}\): $\(\mathcal{L}(\theta) = \mathbb{E}_{q \sim \mathcal{D}_{\text{train}}} \left[ \sum_{k=1}^K \frac{\pi_\theta(r_k \mid q)}{\pi_{\theta_{\text{old}}}(r_k \mid q)} \cdot (R_k - \bar{R}) \right]\)$ where \(\bar{R} = \frac{1}{K}\sum_{k=1}^K R_k\) is the empirical rollout baseline. LoRA rank is set to 32, batch size is 64 prompts per step (1,024 trajectories per gradient step), and optimization utilizes AdamW with a learning rate of \(2 \times 10^{-5}\) under cosine decay and 50 warm-up steps. Continuous probability predictions are clipped to \([0, 1]\) prior to reward assignment. Single-task training runs for approximately 200 steps (~10 epochs), while multi-task learning (MTL) balances samples evenly across task categories within the same overall update budget.

Key Experimental Results

Main Results

The benchmark evaluated 18 frontier proprietary and open-weight models across tasks E1โ€“E9 (reported as macro-averaged Skill Score, where theoretical ceiling is +1.0 and dummy predictor is 0.0), alongside multi-task RL post-trained open-weight models.

Model Model Family Strict Raw Score (\(s_t\)) Dummy Baseline (\(b_t\)) Aggregate Skill Score (\(\text{Skill}\)) Observations
DeepSeek-V3.1 Proprietary / Frontier 0.812 0.665 +0.147 ยฑ 0.019 Highest aggregate skill across leaderboard
gpt-5.4 Proprietary / Frontier 0.826 0.695 +0.131 Strongest commercial proprietary baseline
Kimi-K2.5 Proprietary / Frontier 0.804 0.678 +0.126 Consistent long-context self-prediction
Qwen3-8B (Base) Open-source / 8B 0.702 0.594 +0.108 ยฑ 0.013 Highest untrained open-weight baseline
Qwen3.5-35B-A3B Open-source / 35B MoE 0.741 0.634 +0.107 Competitive open MoE architecture
Claude Opus 4.5 Proprietary / Frontier 0.796 0.695 +0.101 Balanced across confidence & attribution
GPT-4o Proprietary / Commercial 0.763 0.672 +0.091 Standard production deployment model
GPT-OSS-20B (Base) Open-source / 20B 0.693 0.613 +0.080 ยฑ 0.016 Explicit dual-channel reasoning model
Llama-3.3-70B Open-source / 70B 0.724 0.668 +0.056 Parameter scaling shows diminishing self-modeling
Gemini 3.1 Pro Proprietary / Frontier 0.738 0.698 +0.040 High raw accuracy masked by high dummy baseline
Llama-3.1-8B (Base) Open-source / 8B 0.585 0.592 -0.007 ยฑ 0.034 Untrained base underperforms trivial predictor
Llama-3.1-8B (MTL-RL) Ours (Finetuned) 0.677 0.585 +0.092 ยฑ 0.006 Substantial gain (\(\Delta = +0.099\)) over base
Qwen3-8B (MTL-RL) Ours (Finetuned) 0.729 0.609 +0.120 ยฑ 0.007 Surpasses Claude Opus 4.5 and Kimi-K2.5
GPT-OSS-20B (MTL-RL) Ours (Finetuned) 0.740 0.613 +0.127 ยฑ 0.008 Parallels leading frontier proprietary systems

Ablation Study

Ablations conducted on Llama-3.1-8B compare the generalization of single-task post-training on FLIP-RATE (E3) across the multi-task evaluation suite (E1โ€“E9 macro average) under varying optimization and data configurations.

Post-Training Recipe Training Data Specification Aggregate Suite Skill Gain Over Untrained Base (\(\Delta\)) Empirical Dynamics
Untrained Base (Reference) None -0.007 โ€” Low parse compliance on strict JSON
SFT (simulation_raw) 1,275 rows (E3 slice) -0.069 -0.062 Severe format collapse; OOD tasks score 0.0
SFT + Online Label Refresh 1,275 rows (periodic GT refresh) -0.105 -0.098 Online relabeling fails to prevent format drift
SFT + RL Pipeline SFT initialized, followed by RL -0.110 -0.103 RL cannot recover from SFT representation collapse
RL + Online Label Refresh RLVR with dynamic GT refresh +0.039 +0.046 Refresh adds 25-60% runtime without quality gain
RL Full Unbalanced Data (Epoch 1) 16,864 rows (unfiltered corpus) +0.037 +0.044 Peaks at epoch 1 then degrades due to imbalance
RL Balanced Slice (Epoch 8) 1,275 rows (stratified balanced) +0.070 +0.077 Steady monotonic climb across training epochs
Multi-Task RL (MTL) Multi-task balanced mixture +0.092 +0.099 Uniformly lifts all 9 evaluation formats

Key Findings

  • Substantial ceiling gap across all models: The top-performing model across the 18-model sweep, DeepSeek-V3.1, achieves an aggregate skill score of only +0.147 (against a theoretical ceiling of +1.0), demonstrating that current LLMs remain fundamentally unreliable at forecasting their own input-output behaviors.
  • SFT suffers from format collapse whereas RL generalizes: Across both Llama and Qwen models, single-task SFT degrades out-of-distribution self-modeling skills below untrained baselines due to rigid output template memorization. In contrast, format-gated RL maintains general capability on MMLU-Pro while reliably transferring self-modeling gains across held-out task formats.
  • Cross-model transfer disproves privileged internal access: In cross-model evaluation setups where Model A predicts the behavioral labels of Model B, superior explainer models (such as fine-tuned Qwen3-8B) predict Llama's behavioral ground truth more accurately than Llama predicts its own behavior. This indicates that self-modeling improvements stem from general behavioral modeling and inductive causal reasoning rather than privileged introspection into private internal activations.

Highlights & Insights

  • Baseline-subtracted Skill Score formalization: Establishes an essential methodological correction for metacognitive evaluation by demonstrating that raw task accuracy on imbalanced behavioral datasets is severely confounded by trivial majority-class baselines.
  • Closed-loop generate-verify-revise synthesis: Systematically uncovers that auxiliary generator LLMs exhibit severe blind spots regarding target models' decision boundaries, overcoming this limitation through active multi-round empirical sampling verification.
  • Introspection as learned behavioral modeling: Provides strong empirical evidence that self-explanation is an exteriorized behavioral prediction task amenable to standard reinforcement learning rather than an inaccessible mechanistic property requiring internal probe engineering.

Limitations & Future Work

  • Constrained static single-turn scope: The primary evaluation suite focuses on isolated text prompts with immediate scalar or discrete targets, leaving unexamined the long-horizon self-modeling challenges present in multi-turn agentic environments with external memory and complex tool use.
  • Non-adaptive perturbation search: Perturbations in the evaluation suite are uniformly sampled from a curated pool; adaptive adversarial search tuned to individual models' decision boundaries could expose deeper blind spots but would complicate cross-model comparability.
  • Modest downstream utility in practical auditing: Downstream evaluations in Petri red-teaming and prompt engineering demonstrate positive but modest quantitative improvements, indicating that current self-reports remain noisy candidate signals that cannot yet substitute for empirical verification probes.
  • vs. Mechanistic Introspection & SAE Probes (Li et al., 2025; Lindsey, 2026): Mechanistic approaches evaluate whether models can report on sparse autoencoder features or internal activations. This work avoids assumptions of internal access, demonstrating that behavioral self-prediction is fully measurable via external sampling and trainable via RL.
  • vs. Unverified Post-Hoc Self-Explanations (Mayne et al., 2025; Chuang et al., 2026): Prior studies emphasize the unfaithfulness and sycophancy of rationales generated post-hoc. This work frames explanation as forward behavioral prediction verified against rigorous empirical rollouts.
  • vs. Classical Uncertainty Calibration (Fleming, 2024; Shi et al., 2025): While classical calibration assesses confidence on the immediate output, this benchmark expands self-modeling to counterfactual sensitivity, component attribution, and minimal perturbation proposals.

Rating

  • Novelty: โญโญโญโญโญ Formulates an objective, verifiable behavioral foundation for LLM self-modeling, sidestepping metaphysical debates on introspection.
  • Experimental Thoroughness: โญโญโญโญโญ Sweeps 18 frontier and open-weight models, provides comprehensive SFT vs. RL ablations, and validates findings across two downstream applications.
  • Writing Quality: โญโญโญโญโญ Methodological definitions, baseline calibrations, and error-mode taxonomies are articulated with outstanding technical clarity.
  • Value: โญโญโญโญโญ Establishes a foundational benchmark and training recipe for AI alignment, model auditing, and metacognitive evaluation.