RECAP: Regression Evaluation for Continual Adaptation of Prompts¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Continual Prompt Adaptation, Regression Evaluation, Proactive Adaptation, LLM Benchmarking, Catastrophic Forgetting
TL;DR¶
Addressing production agentic setups where models must immediately comply with dynamic constraint specifications without test data or feedback, this paper introduces RECAP—the first constraint-level continual learning benchmark—and reveals that existing prompt adaptation methods fail to beat a simple constraint-appending baseline while inducing context bloat, spurious conflicts, and severe refusal cascades.
Background & Motivation¶
In production environments, agentic systems face operational constraints that evolve continuously across interactions: an external tool response may suddenly restrict allowable output length, an enterprise governance update may mandate a specific compliance disclaimer, or a user personalization change may alter expected tone and structure. In realistic deployments, these requirements accumulate in a decentralized manner from heterogeneous tools, multi-tenant workflows, and disparate runtime sessions, making it infeasible to pause serving, collect the entire global constraint inventory, and rerun joint prompt optimization from scratch. More critically, the moment a new rule or modification is introduced, the system must respect the updated specification from the very next execution without experiencing regression or forgetting on previously established constraints.
However, existing evaluation paradigms and optimization techniques remain fundamentally decoupled from this operational reality. Popular instruction-following benchmarks—such as IFEval, FollowBench, and InFoBench—treat constraint compliance as an entirely static, single-turn evaluation task, lacking any temporal mechanism to evaluate incremental additions, revisions, or removals over time. On the optimization front, automated prompt engineering algorithms (such as DSPy/MIPROv2 and GEPA) and reactive context refinement frameworks (like ACE) invariably rely on the availability of representative evaluation datasets and iterative feedback from execution failures. Meanwhile, classic continual learning literature has focused heavily on weight regularization, episodic replay, or parameterized prompt embeddings, providing little guidance for frozen-weight large language models that must adapt on the fly purely through natural language prompt adjustments under strict latency budgets.
This discrepancy highlights an urgent blind spot: the proactive adaptation setting, where a system receives solely an updated constraint specification and must generalize immediately before observing any real test queries or evaluation feedback. Investigating whether existing prompt optimization algorithms can maintain longitudinal reliability under frozen weights and zero test-time feedback is crucial for production agents. Core idea: convert multi-constraint instruction datasets into a temporal evaluation stream of typed add, edit, and delete operations under a strictly proactive adapt-then-test protocol, introducing RECAP with shadow evaluation and decomposed metrics to expose structural failures in continual prompt adaptation.
Method¶
Overall Architecture¶
The RECAP framework adopts an adapt-then-test continual learning evaluation protocol specifically formulated for text-level prompt evolution. Grounded on the RECAST-30K corpus (derived from Tulu 3 Persona IF), RECAP untangles foundational user instructions from diverse operational constraints across a 21-type taxonomy (comprising 8 deterministic rule-based types and 13 qualitative types assessed by an LLM judge). The temporal evaluation stream is governed by structured schedules executing sequences of atomic operations: ADD introduces a new constraint, EDIT revises an existing constraint's target value, and DELETE removes an active constraint entirely. At each step \(k\), candidate adaptation algorithms receive solely the constraint specification and metadata via the adapt() interface, performing internal adaptation without test data; subsequently, the serving model is evaluated across all active constraints along with shadow constraints representing deleted or modified requirements.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Evolving Constraint Stream<br/>ADD / EDIT / DELETE Specifications"] --> B["Stage 1: Proactive Adaptation Protocol<br/>Specification-only adapt interface via self-play"]
B --> C["Stage 2: Shadow Evaluation Mechanism<br/>Active constraints test + historical shadow tracking"]
C --> D["Stage 3: Decomposed Metric Suite<br/>Multi-dimensional quantification of regression & fidelity"]
D --> E["Output: Longitudinal Diagnostic Profile"]
Key Designs¶
1. Proactive Adaptation Protocol: Zero-Feedback Specification Isolation
Traditional prompt optimization loops presuppose an iterative trial-and-error environment with access to task-level loss signals or test ground truth. In contrast, RECAP enforces an information-isolated proactive interface: M.adapt(model, context) receives only the current step's operation delta (step_ops) and structural metadata (regression_meta), with zero access to incoming test prompts or downstream task performance. While adaptation algorithms are permitted to execute internal synthetic self-play (generating candidate drafts and evaluating them against the specification), they must complete all internal iterations upfront and deliver a serving prompt ready for zero-shot deployment on the live task stream.
2. Shadow Evaluation Mechanism: Longitudinal Tracking of Edits and Deletions Standard evaluation pipelines discard obsolete rules when constraints are edited or deleted, blinding researchers to subtle temporal failures. RECAP introduces a shadow evaluation paradigm: whenever a constraint is updated or deleted, its legacy specification is retained as a background "shadow" evaluator in all subsequent time steps. Model outputs are simultaneously evaluated against active rules and these shadow evaluators, directly tracking whether models suffer from edit reversion (relapsing into obsolete parameter values over time) or unlearning rebound (unintentionally re-satisfying a deleted constraint due to prompt artifact drift).
3. Decomposed Metric Suite: Orthogonal Quantification of Continual Dynamics Relying exclusively on the aggregate mean satisfaction rate (\(r_{\text{sat}}\)) obscures catastrophic local regression. RECAP establishes a comprehensive metric suite covering forward transfer, regression, and unlearning dynamics: - Peak Forgetting (\(f_{\text{peak}}\)): Measures the maximum performance regression from a constraint's historical peak satisfaction: $\(f_{\text{peak}}(t, k) = \max\left(0, \max_{j < k} r_{\text{sat}}(t, j) - r_{\text{sat}}(t, k)\right)\)$ - Collateral Damage (\(\text{coll}\)): Quantifies the average satisfaction drop across non-targeted active constraints \(R_k\) immediately following an operation on constraint \(t^*\): $\(\text{coll}(k) = \frac{1}{|R_k|} \sum_{t \in R_k} \max\left(0, r_{\text{sat}}(t, k-1) - r_{\text{sat}}(t, k)\right)\)$ - Edit Switch (\(\text{switch}\)): Evaluates whether an EDIT operation effectively transitions outputs from the stale constraint to the new specification, defined as \(\text{switch} = \frac{\text{Adapted}}{\text{Both} + \text{Stale} + \text{Adapted}}\). - Unlearning Fidelity (\(\text{UF}\)): Assesses how closely model outputs revert to their unconstrained baseline behavior \(r_{\text{def}}(d)\) after constraint \(d\) is deleted: $\(\text{UF}(d, k) = 1 - \min\left(1, \frac{|r_{\text{post}}(d, k) - r_{\text{def}}(d)|}{\max(\epsilon, |r_{\text{pre}}(d) - r_{\text{def}}(d)|)}\right)\)$
Benchmark & Evaluation Protocol¶
The benchmark evaluates 6 paradigm-spanning prompt adaptation algorithms: the unadapted baseline (Base LLM, which simply appends active constraint strings into user prompts at inference), in-context few-shot prompting (ICL), adaptive memory retrieval (Dynamic Cheatsheet), and three automated optimization frameworks leveraging synthetic self-play (ACE with generate-evaluate-reflect-curate loops, GEPA using genetic prompt mutation, and MIPROv2 proposing instructions guided by historical scores). Three evaluation schedules stress-test these methods across 15–20 steps: Interleaved-20 (11 adds, 5 edits, 4 deletes mixed dynamically), Clustered-20 (phased sequential blocks of ADDs, EDITs, then DELETEs), and Rule-Only-15 (6 deterministic rule-based constraint types eliminating LLM judge variance).
Key Experimental Results¶
Main Results¶
The benchmark encompasses 90 experimental conditions across 5 prominent foundation backbones: Llama-3.1-8B, Llama-3.3-70B, GPT-OSS-20B, GPT-OSS-120B, and Gemma-4-26B-A4B. Claude Sonnet 4.5 acts as the external binary evaluator for qualitative constraints. Table 1 summarizes the core performance indicators across the large open-weight backbones:
| Backbone | Method | sat (↑) | Forg. (↓) | Coll. (↓) | Switch (↑) | \(\text{UF}_{\text{sus}}\) (↑) |
|---|---|---|---|---|---|---|
| Llama-3.3-70B | Base LLM | 0.595 | 0.235 | 0.048 | 0.295 | 0.642 |
| ICL | 0.592 | 0.188 | 0.033 | 0.301 | 0.650 | |
| Dyn. Cheatsheet | 0.600 | 0.204 | 0.048 | 0.300 | 0.693 | |
| ACE | 0.602 | 0.212 | 0.035 | 0.290 | 0.683 | |
| GEPA | 0.603 | 0.198 | 0.048 | 0.291 | 0.643 | |
| MIPROv2 | 0.595 | 0.209 | 0.034 | 0.282 | 0.672 | |
| GPT-OSS-120B | Base LLM | 0.630 | 0.179 | 0.053 | 0.309 | 0.674 |
| ICL | 0.598 | 0.211 | 0.040 | 0.317 | 0.730 | |
| Dyn. Cheatsheet | 0.610 | 0.224 | 0.050 | 0.315 | 0.709 | |
| ACE | 0.454 | 0.330 | 0.050 | 0.289 | 0.610 | |
| GEPA | 0.506 | 0.268 | 0.049 | 0.301 | 0.648 | |
| MIPROv2 | 0.571 | 0.222 | 0.053 | 0.256 | 0.635 | |
| Gemma-4-26B-A4B | Base LLM | 0.677 | 0.163 | 0.044 | 0.325 | 0.702 |
| ICL | 0.627 | 0.167 | 0.049 | 0.321 | 0.701 | |
| Dyn. Cheatsheet | 0.678 | 0.134 | 0.031 | 0.334 | 0.717 | |
| ACE | 0.504 | 0.272 | 0.056 | 0.288 | 0.719 | |
| GEPA | 0.658 | 0.161 | 0.036 | 0.311 | 0.749 | |
| MIPROv2 | 0.660 | 0.172 | 0.054 | 0.312 | 0.767 |
Paired statistical significance tests across all 15 backbone \(\times\) schedule cells confirm that no adaptation method statistically outperforms Base LLM on mean satisfaction (all Holm-corrected one-sided superiority \(p_{\text{sup}} \ge 0.84\)). On the GPT-OSS family, adaptation methods cause severe degradation (e.g., ACE satisfaction plunges by \(-0.176\) on GPT-OSS-120B). On Llama-70B, all adaptive methods converge within narrow statistical noise of the baseline while consuming up to \(1.5\times\) latency and substantially higher token volumes.
Ablation Study¶
To investigate whether adaptation failure stems from compute bottlenecks during self-play or an information gap in the delta-only interface, the authors conducted two targeted ablations on GPT-OSS-120B:
Table 1: Self-Play Budget Scaling for ACE on GPT-OSS-120B | Compute Budget | Calls / Step | sat (↑) | Forg. (↓) | Refusal Rate (%) | Total Tokens | |---|---|---|---|---|---| | Base LLM (0×) | 0 | 0.630 | 0.179 | 0.0% | 0.80M | | 1× Budget (Default) | 4 | 0.454 | 0.330 | 12.4% | 1.62M | | 3× Budget | 12 | 0.421 | 0.326 | 28.6% | 2.39M | | 5× Budget | 20 | 0.340 | 0.390 | 47.8% | 2.82M |
Table 2: Full-Constraints Information Gap Ablation (GPT-OSS-120B, 3-Schedule Mean) | Condition | Interface Context | sat (↑) | Forg. (↓) | BWT (↑) | Switch (↑) | Total Tokens | |---|---|---|---|---|---|---| | Base LLM | Test-time active constraints only | 0.630 | 0.179 | 0.033 | 0.309 | 0.80M | | ACE (Full-Context) | Full active constraints during adapt() | 0.614 | 0.203 | -0.032 | 0.321 | 2.26M | | ACE (Standard Delta)| Single operation delta during adapt() | 0.454 | 0.330 | -0.180 | 0.289 | 1.62M |
Key Findings¶
- More Self-Play Compute Accelerates Failure: Scaling ACE's self-play budget from \(1\times\) to \(5\times\) causes satisfaction to collapse monotonically from 0.454 down to 0.340. The playbook expands from 9.1K to 15.6K characters, triggering phantom contradictions that spike model refusal rates from 12.4% to 47.8%.
- Closing the Information Gap Yields Expensive Neutrality: Providing the full constraint inventory to
adapt()recovers 91% of ACE's deficit (0.454 \(\to\) 0.614), yet still fails to beat Base LLM (0.630) while incurring \(2.8\times\) the token overhead. The multi-stage reflection pipeline introduces indirection and linguistic bloat that degrades pure constraint adherence. - Model Scale Dominates Optimization Methods: Across benchmarks, intrinsic backbone capability dictates performance thresholds (Llama-70B plateauing at \(\sim 0.60\) and Llama-8B at \(\sim 0.49\)). The nominal variances among adaptive algorithms remain secondary to underlying model scale.
Highlights & Insights¶
- Taxonomy of 6 Structural Failure Modes: The study systematically documents six characteristic failure modes in proactive prompt adaptation: Specification Lock (MIPROv2 hardcoding obsolete constraint values), Progressive Refusal Cascade (ACE bloating playbooks into phantom conflicts), Demo Noise (ICL injecting preamble artifacts), Prefix Contamination (GEPA prepending rigid prefixes), Inert Growth (Dynamic Cheatsheet collecting generic advice), and Keyword Anti-Unlearning (semantic persistence resisting deletion).
- "The Constraints Are the Optimal Prompt": For self-contained constraint-satisfaction tasks under zero external feedback, meta-cognitive transformations (reflection, playbook curation, evolutionary mutation) inevitably dilute semantic signals; directly presenting the raw constraint specifications to the model remains the optimal strategy.
- Robustness Against Judge Bias via Rule-Only Validation: Reproducing identical no-improvement trends on the Rule-Only-15 schedule firmly demonstrates that the observed degradation reflects genuine cognitive interference rather than stochastic evaluation noise from the LLM judge.
Limitations & Future Work¶
- Prompt-Level Focus: The investigation is confined to frozen-weight prompt adaptation, leaving parameter-efficient fine-tuning or continual LoRA updates unexplored under proactive constraints.
- Absence of Ground-Truth Oracle Comparisons: The study does not benchmark against reactive systems equipped with real-time test feedback or specialized execution oracles, which would further bound the theoretical cost of the proactive information gap.
- General Instruction Datasets: Evaluation constraints stem from broad instruction-following corpora (RECAST); domain-specific agent streams (e.g., specialized financial auditing or robotic tool-use) may exhibit distinct operational dynamics.
Related Work & Insights¶
- vs. Static Instruction-Following (IFEval, FollowBench, InFoBench): Prior benchmarks evaluate multi-constraint satisfaction in isolated, static prompts; RECAP pioneers longitudinal stream evaluation to capture interference, forgetting, and regression under continuous rule updates.
- vs. Automated Prompt Engineering (DSPy MIPROv2, GEPA): Conventional offline optimizers depend on labeled validation sets to navigate prompt search spaces; under proactive zero-shot constraints, their self-play mechanisms overfit to synthetic artifacts and suffer specification lock.
- vs. Reactive Context Evolution (ACE): Reactive frameworks rely on observing downstream deployment errors to iteratively patch playbooks; without runtime error signals, recursive self-reflection compounds hallucinations and provokes refusal cascades.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First systematic formulation of proactive constraint-level continual learning and shadow evaluation for LLMs.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ 90 experimental conditions across 5 model backbones, rigorous non-parametric statistical testing, and comprehensive compute ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent narrative, lucid problem formulation, and exceptionally insightful failure mode diagnostics.
- Value: ⭐⭐⭐⭐⭐ Provides indispensable negative results that recalibrate prompt engineering assumptions for enterprise agentic architectures.