SkillBrew: Multi-Objective Curation of Skill Banks for LLM Agents¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Applied-Machine-Learning-Lab/EMNLP26_SkillBrew
Area: LLM Agent
Keywords: Agent Skill Bank, Multi-Objective Optimization, Pareto Front, Counterfactual Replay, Skill Curation
TL;DR¶
Formalizes skill bank curation for retrieval-augmented LLM agents as a constrained multi-objective Pareto optimization problem over utility, diversity, and coverage, and introduces a bi-level propose-then-verify framework driven by counterfactual leave-one-out replay to continuously evolve compact, high-performing skill repositories.
Background & Motivation¶
Retrieval-augmented large language model (LLM) agents deployed on long-horizon, open-ended tasks increasingly rely on skill banksโcollections of reusable textual procedural principles that guide decision making across recurring episodes. Standardized by specifications such as Anthropic Agent Skills, these banks serve as external procedural memory for frozen agents. However, existing approaches predominantly construct skill libraries in an append-only fashion (exemplified by Voyager and Trace2Skill), continuously accumulating newly synthesized lessons without ever pruning obsolete, redundant, or harmful rules. Over extended deployment, this monotonic accumulation results in swollen, noisy repositories where irrelevant skills crowd out valuable context during top-k retrieval, severely degrading agent decision quality.
The core tension is that the structural quality of a skill bank is inherently multi-objective and mutually constraining: an ideal repository must simultaneously be useful (boosting task success), diverse (minimizing semantic redundancy among stored principles), and broadly covering (exercising skills across the entire query distribution). Aggressively hoarding skills to maximize task coverage introduces redundancy that collapses internal diversity, whereas naively pruning skills to enforce diversity harms generalizability on long-tail tasks. Furthermore, individual skills are rarely evaluated in isolation; co-retrieval interactions confound credit assignment, and the space of natural language edits is open-ended and non-differentiable. Prior attempts at curation either collapse evaluation into a single scalar heuristic or judge skills in isolation, which inherently fails to untangle these multi-dimensional trade-offs.
To resolve these challenges without costly agent fine-tuning, the authors argue that skill banks must be curated as holistic knowledge systems rather than isolated entries. Under an asymmetric formulation, utility serves as an explicit constraint while diversity and coverage act as structural regularizers, reflecting that a bank must first be functional before its organization matters. Core idea: formalize skill bank curation as a constrained multi-objective Pareto optimization problem over utility, diversity, and coverage, leveraging counterfactual leave-one-out replay for precise credit assignment and a bi-level propose-then-verify loop to iteratively discover non-dominated, compact skill repositories.
Method¶
Overall Architecture¶
SkillBrew is a training-free, iterative framework that executes global bank-level refinement across a fixed budget of rounds. The task distribution is partitioned into a support set \(\mathcal{D}_{\text{support}}\) and a held-out query set \(\mathcal{D}_{\text{query}}\), strictly decoupling edit generation from verification to prevent overfitting. During an initial cold-start stage, an initial bank \(\mathcal{B}^{(0)}\) is seeded from no-retrieval failure trajectories contrasted against successful rollouts. The curation loop then alternates between two levels: an Inner Loop that collects factual and counterfactual rollout evidence on \(\mathcal{D}_{\text{support}}\) to populate three edit pools (ADD, REWRITE, REMOVE) and proposes \(K\) candidate banks, and an Outer Loop that profiles each candidate bank across utility \(J_{\text{util}}\), diversity \(J_{\text{div}}\), and coverage \(J_{\text{cov}}\) on \(\mathcal{D}_{\text{query}}\), selecting the winning bank \(\mathcal{B}^{(i+1)}\) via a front-adaptive Pareto selector with an \(\epsilon\)-utility safeguard.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Environment rollouts<br/>Disjoint support and query splits"] --> B["Cold-start initialization<br/>No-retrieval failure clustering into B(0)"]
subgraph IN["Inner loop: candidate proposal on support set"]
direction TB
C1["Counterfactual leave-one-out replay<br/>Isolates marginal contribution Delta"] --> C2["Tri-pool edit generation<br/>ADD distiller / REWRITE diagnoser / REMOVE pruner"]
C2 --> C3["Global bank edit planning<br/>Protected KEEP slots into K candidate banks"]
end
B --> C1
subgraph OUT["Outer loop: Pareto-aware selection on query set"]
direction TB
D1["Three-dimensional objective profiling<br/>Utility Jutil + Gram determinant Jdiv + Coverage Jcov"] --> D2["Adaptive Pareto selection<br/>Epsilon-utility safeguard and hypervolume tie-break"]
end
C3 --> D1
D2 -->|Iterative multi-round refinement| E["Curated skill bank B*"]
Key Designs¶
1. Counterfactual leave-one-out replay: disentangling credit assignment under co-retrieval
When an agent retrieves multiple skills simultaneously (\(R_t = \text{RET}(\mathcal{B}, t)\)), standard outcome rewards cannot isolate which specific rule aided or impaired execution, leading to noisy additions and erroneous pruning. SkillBrew addresses this bottleneck by executing counterfactual leave-one-out rollouts on the support set \(\mathcal{D}_{\text{support}}\): for each retrieved skill \(s \in R_t\), the episode is replayed under identical initial environment conditions with \(s\) removed from the prompt context, yielding trajectory \(\tilde{\tau}_{t\setminus s} = \pi_{\text{agent}}(t \mid R_t \setminus \{s\})\). The marginal causal contribution is measured as: $\(\Delta(\tau, s) = r(\tau, s) - r(\tau, \emptyset)\)$ By aggregating these factual-counterfactual trajectory pairs across all episodes retrieving \(s\), the Skill Diagnoser categorizes empirical patterns into distinct regimes (such as cases where the skill caused failure that would have succeeded without it) to issue grounded, fine-grained KEEP, REWRITE, or REMOVE verdicts.
2. Tri-pool edit generation: salvaging flawed skills while enabling surgical pruning
Prior curation frameworks either expand monotonically or apply coarse-grained binary deletion, unnecessarily discarding skills whose core reasoning is sound but whose surface instructions contain brittle edge cases. SkillBrew establishes three explicit edit pools. Uncovered task failures from the no-retrieval partition are clustered by the Skill Distiller to synthesize entirely new principles into \(\mathcal{E}_{\text{add}}\). For retrieved skills exhibiting systematic failure modes, the Skill Diagnoser inspects the counterfactual traces: if the underlying premise has merit but specific trigger conditions or sub-steps mislead the agent, a revised when_to_apply condition or updated action sequence is produced and directed to \(\mathcal{E}_{\text{rewrite}}\); if the strategy is fundamentally flawed and irreparable, it is dispatched to \(\mathcal{E}_{\text{remove}}\). Crucially, verified beneficial skills form a protected \(\mathcal{E}_{\text{keep}}\) set that cannot be mutated within the round, ensuring stable retention of working capabilities.
3. Global bank edit planning: composing multi-objective candidate banks under protected slots
Applying isolated per-skill verdicts mechanically can inadvertently damage the collective coverage or diversity of the repository. The Edit Planner operates as a macro-level coordinator, conditioning on the complete current bank \(\mathcal{B}^{(i)}\), all three candidate edit pools, and the protected \(\mathcal{E}_{\text{keep}}\) slots. The planner adopts distinct high-level editing philosophies (termed "lenses," such as prioritizing domain coverage expansion, aggressive redundancy compression, or conservative semantic rewrites) to generate \(K\) mutually diverse, Pareto-relevant candidate banks \(\mathcal{K}^{(i)} = \{\mathcal{B}_1^{(i+1)}, \dots, \mathcal{B}_K^{(i+1)}\}\). This discrete synthesis converts an intractable open-ended editing space into a structured candidate set amenable to multi-objective verification.
4. Adaptive Pareto selection: balancing structural regularizers under an epsilon-utility safeguard
To optimize the repository without arbitrarily collapsing conflicting criteria into an ad-hoc scalar weight, the outer loop evaluates each candidate bank on the query set across three principled objectives: counterfactual-weighted empirical utility \(J_{\text{util}}(\mathcal{B})\), geometric embedding diversity defined via the Gram matrix determinant of normalized embeddings: $\(J_{\text{div}}(\mathcal{B}) = \det(G + \lambda I)^{1 / |\mathcal{B}|}\)$ and query coverage defined as the product of retrieval density and bank usage \(J_{\text{cov}}(\mathcal{B}) = \text{density} \cdot \text{usage}\). To resolve the non-dominated Pareto front \(\mathcal{F}^{(i)}\) (which includes the unchanged null candidate \(\mathcal{B}^{(i)}\)), the selector implements an adaptive threshold: $\(\text{Tied} = \{\mathcal{B} \in \mathcal{F}^{(i)} : J_{\text{util}}(\mathcal{B}) \ge u_{\max} - \epsilon\}\)$ with \(\epsilon = 0.03\). Any ties within this near-optimal utility bracket are broken by maximal hypervolume contribution over diversity \(J_{\text{div}}\) and coverage \(J_{\text{cov}}\). This mechanism enforces a rigorous worst-case utility bound per round (\(J_{\text{util}}(\mathcal{B}^{(i+1)}) \ge J_{\text{util}}(\mathcal{B}^{(i)}) - \epsilon\)), ensuring that structural gains in diversity never compromise agent task performance.
A Worked Example¶
Consider an embodied household task in ALFWorld requiring the agent to heat a mug in a microwave and place it on a side table (Heat & Place):
1. Initial Trajectory & Failure: In Round 1, the agent retrieves the initial skill Systematic room search (bank size = 35). Because Step 1 of the principle unconditionally commands visiting the destination receptacle first, the agent wanders aimlessly searching for the side table while the mug sits on an adjacent countertop, resulting in a timeout failure.
2. Counterfactual Diagnosis: During the inner loop, leave-one-out replay executes the episode without this skill (\(\tilde{\tau}_{t\setminus s}\)). Without the misleading imperative, the agent directly picks up the visible mug and succeeds. The Skill Diagnoser identifies the specific flaw in sentence 1, issuing a REWRITE verdict with revised instructions to prioritize immediately visible targets before executing room-wide sweeps.
3. Pareto Candidate Verification: The Edit Planner incorporates this rewrite into candidate banks along with candidates pruning redundant search rules. On the query split, a bank executing this rewrite alongside selective pruning preserves utility (\(J_{\text{util}} = 0.423\)) while increasing structural diversity (\(J_{\text{div}} = 0.343\)). The bank contracts to 23 skills by Round 10, elevating the task success rate on Heat tasks from 13.2% (ReAct) to 60.9%.
Key Experimental Results¶
Main Results¶
SkillBrew is evaluated across three interactive agent benchmarks (ALFWorld embodied reasoning, WebShop multi-step online shopping, and ScienceWorld complex scientific experiments) using a frozen Qwen2.5-7B-Instruct worker, benchmarked against ten training-free baselines:
| Benchmark / Task | Metric | ReAct (Baseline) | Voyager (Append-only) | Skill-Pro (Prior SOTA) | SkillBrew (Ours) | Absolute Gain (vs SOTA / ReAct) |
|---|---|---|---|---|---|---|
| ALFWorld (Overall) | Success (%) | 31.2 | 47.0 | 49.3 | 59.0 ยฑ 1.2 | +9.7% / +27.8% |
| - Pick & Place | Success (%) | 48.5 | 62.5 | 66.7 | 70.8 ยฑ 2.4 | +4.1% / +22.3% |
| - Examine in Light | Success (%) | 35.4 | 61.1 | 66.7 | 72.2 ยฑ 3.2 | +5.5% / +36.8% |
| - Clean & Place | Success (%) | 34.3 | 51.6 | 45.2 | 58.1 ยฑ 1.9 | +6.5% / +23.8% |
| - Heat & Place | Success (%) | 13.2 | 52.2 | 43.5 | 60.9 ยฑ 2.5 | +8.7% / +47.7% |
| - Cool & Place | Success (%) | 18.2 | 28.6 | 47.6 | 57.1 ยฑ 2.7 | +9.5% / +38.9% |
| - Pick Two Objects | Success (%) | 17.6 | 17.6 | 23.5 | 29.4 ยฑ 3.4 | +5.9% / +11.8% |
| WebShop | Score (%) | 46.2 | 50.0 | 38.7 | 59.3 ยฑ 0.8 | +9.3% / +13.1% |
| WebShop | Success Rate (%) | 19.5 | 26.4 | 23.2 | 38.4 ยฑ 0.9 | +12.0% / +18.9% |
| ScienceWorld | Score (%) | 11.3 | 10.9 | 17.2 | 25.4 ยฑ 0.6 | +5.5% / +14.1% |
| ScienceWorld | Success Rate (%) | 8.13 | 9.83 | 11.4 | 17.2 ยฑ 0.7 | +4.3% / +9.07% |
Across five worker model backbones spanning open-source and proprietary scales (Table 2), SkillBrew consistently outperforms ReAct. On GPT-4o, ALFWorld average success jumps from 46.4% to 88.1%, and WebShop success surges from 27.2% to 65.0%. Remarkably, equipping the lightweight Qwen3-4B-Instruct worker with a curated skill bank achieves 60.4% on ALFWorld and 36.6% success on WebShop, closely rivaling the performance of the unaugmented frontier model Gemini-2.5-Pro (61.8% and 38.1%), demonstrating that refined procedural knowledge can effectively compensate for backbone parameter scale.
Ablation Study¶
Ablations confirm the critical contribution of both the multi-objective formulation (Table 4) and the discrete edit operations (Table 5):
| Ablation Dimension | Configuration | ALFWorld Avg. SR (%) | WebShop Score (%) | WebShop Succ. (%) | Empirical Impact |
|---|---|---|---|---|---|
| Objective Formulation | \(J_{\text{util}}\) alone | 45.8 | 48.2 | 28.4 | Overfits query trajectories without structural regularization |
| (Table 4) | \(J_{\text{util}} + J_{\text{div}}\) | 51.4 | 47.6 | 36.5 | Penalizes semantic redundancy, boosting generalizability |
| \(J_{\text{util}} + J_{\text{cov}}\) | 52.6 | 51.0 | 37.6 | Discourages dormant skills and increases retrieval spread | |
| Full (\(J_{\text{util}} + J_{\text{div}} + J_{\text{cov}}\)) | 59.0 | 59.3 | 38.4 | Combines utility constraint with structural volume trade-off | |
| Edit Operations | ADD only (Append-only baseline) | 47.0 | 50.0 | 26.4 | Monotonic growth leads to bloated, noisy retrieval slots |
| (Table 5) | ADD + REWRITE | 53.5 | 51.0 | 37.6 | Recovers flawed but viable principles (+6.5% gain) |
| ADD + REMOVE | 48.3 | 49.5 | 34.5 | Coarse pruning discards salvageable procedural insights | |
| Full (ADD + REWRITE + REMOVE) | 59.0 | 59.3 | 38.4 | Synergistic combination of refinement, pruning, and expansion |
In cross-worker bank transfer experiments (Table 3), a skill bank curated using Qwen3-4B elevates GPT-4o from 46.4% to 78.4% on ALFWorld (within 9.7% of GPT-4o's natively curated bank of 88.1%), while a GPT-4o curated bank boosts Qwen3-4B from 38.8% to 53.7%. This bilateral transferability confirms that the curated principles represent generalized environmental operational logic rather than model-specific prompting artifacts.
Key Findings¶
- REWRITE is the primary driver of skill quality: While ADD+REMOVE achieves only a 1.3% gain over ADD-only, adding REWRITE yields a massive 6.5% improvement on ALFWorld. In complex agent environments, most failed skills capture correct task concepts but falter due to minor instructional ambiguities, making rewrites substantially more effective than outright deletion.
- Natural contraction and stabilization of bank size: Unlike append-only architectures that expand indefinitely, SkillBrew's curation loop contracts the skill bank over rounds (from 35 skills at cold start to a stable 23 on ALFWorld), eliminating ~34% of redundant entries while monotonically improving downstream success rates.
- High efficiency via content-addressed caching: To mitigate the rollout overhead of leave-one-out counterfactuals, a SHA-256 content-addressed cache achieves a hit rate exceeding 60% from round 2 onward. On ALFWorld, this cuts total wall-clock time from 1,818.3 minutes to 473.8 minutes (a 3.8x speedup) with a total API cost of only $11.20, making the pipeline practically deployable on a single multi-GPU node.
Highlights & Insights¶
- Shift from local filtering to global Pareto curation: Formalizes skill bank optimization at the global system level under a multi-objective formulation, avoiding the collapse of multi-dimensional trade-offs into single scalar scores.
- Causal counterfactual grounding: Employs environment-level leave-one-out rollouts to establish rigorous empirical attribution for multi-skill retrieval contexts, circumventing reliance on subjective LLM scoring.
- Parameter-efficient capability enhancement: Demonstrates that principled refinement of external procedural text allows a 4B parameter agent to approach the decision quality of an uncurated frontier model, providing a viable blueprint for self-improving edge agents.
Limitations & Future Work¶
- Frozen worker policy: The agent's internal weights remain static while the skill bank evolves, leaving co-adaptation between policy fine-tuning (e.g., via RL) and skill bank curation as an open direction.
- Assumption of stationary task distributions: Evaluated within defined benchmark splits; extending the framework to continuously shifting, open-world task streams requires adaptive drift detection mechanisms.
- Rollout cost in stochastic environments: Counterfactual leave-one-out replay relies on environment determinism or repeatability; adapting the attribution pipeline to irreversible or highly non-deterministic real-world scenarios may require learned world models.
Related Work & Insights¶
- vs Voyager / Trace2Skill: Early approaches expand skill banks in an append-only manner, inevitably accumulating redundant and outdated entries; SkillBrew introduces global multi-objective curation and discrete prune/rewrite operations that cause the bank size to converge stably.
- vs ExpeL / Skill-Pro: Prior curation methods evaluate individual skills against a single scalar metric; SkillBrew treats the bank as a collective asset and resolves utility-diversity-coverage trade-offs using Pareto-aware selection with an epsilon safeguard.
Rating¶
- Novelty: โญโญโญโญโญ First formulation of agent skill bank curation as a constrained multi-objective Pareto optimization problem with counterfactual attribution.
- Experimental Thoroughness: โญโญโญโญโญ Thorough evaluation across 3 benchmarks, 5 worker backbones, transferability analysis, cost profiling, and systematic ablations.
- Writing Quality: โญโญโญโญโญ Clear mathematical formalization, cohesive narrative progression, and well-structured algorithmic workflows.
- Value: โญโญโญโญโญ Highly actionable and cost-effective framework for building self-improving, production-grade LLM agent systems.