SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/MemTensor/skills-vote
Area: LLM Agent
Keywords: Agent Skill Governance, Skill Lifecycle, Agentic Search Recommendation, Subtask Attribution, Evidence-Gated Evolution
TL;DR¶
SkillsVote establishes a full-lifecycle governance framework for agent skills across open-source profiling, pre-task agentic library search recommendation, post-task subtask attribution, and evidence-gated evolution, resolving context distraction and library pollution to achieve robust test-time online self-evolution and offline experience transfer on Terminal-Bench 2.0 and SWE-Bench Pro.
Background & Motivation¶
Long-horizon LLM agents tackling software engineering, system administration, and terminal operations generate intermediate decision and execution traces that contain reusable problem-solving knowledge. However, raw trajectories are lengthy, noisy, tightly coupled to ephemeral local runtime states, and fragile across environment shifts. Packaging procedural guidelines, executable scripts, and dependency boundaries into structured directory artifacts—namely Agent Skills centered on SKILL.md—has emerged as a standard format for durable experience reuse. Yet, as open-source skill marketplaces like SkillsMP and skills.sh scale beyond a million packages, the ecosystem exhibits extreme redundancy, uneven quality, and fragile runtime assumptions. Directly exposing uncurated or full skill libraries to agents enlarges the search space, introduces weak or conflicting guidance, and induces severe negative transfer.
The failure modes deepen when agents attempt self-evolution from execution feedback. Existing trajectory-level distillation methods operate on sparse task-level success signals, making fine-grained credit assignment difficult and causing library pollution: incidental or spurious steps during lucky completions are canonized into faulty rules, while external network interruptions or environment crashes trigger misguided edits to valid skills. Conversely, isolating single-step tool calls yields fragmented actions that lack standalone procedural semantics.
The core tension is that autonomous agents require modular, reusable experience to conquer long horizons, yet unconstrained skill retrieval and indiscriminate trajectory distillation inevitably cause context distraction and cumulative library pollution. This work tackles the challenge by treating Agent Skills as lifecycle-governed software artifacts rather than static prompt snippets. Core idea: couple pre-task agentic library search recommendation with post-task subtask-level causal attribution and evidence-gated evolution into an auditable closed loop, filtering irrelevant context before execution and admitting only verified, reusable exploration into persistent library updates afterward.
Method¶
Overall Architecture¶
SkillsVote structures the lifecycle governance of agent skills into four sequential, coupled stages: open-source skill profiling, pre-task recommendation, post-task attribution, and controlled evolution. During offline curation, millions of open-source skill packages from GitHub undergo format validation, deduplication, runtime profiling, quality auditing, and verifiability analysis. Prior to task execution, a recommendation agent conducts interactive inspection over the local read-only skill directory using filesystem tools, supplying the solver agent with a compact skill subset along with concise usage guidance. Following task completion, an attribution module resumes the solver's native session, segments the full trajectory into coherent subtasks, and assigns causal responsibility. Finally, an evolution module groups and filters admissible subtasks to apply targeted edits to existing skills or create new standalone skills.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Million-Scale Open Skill Corpus<br/>1.68M raw GitHub skill packages"] --> B["Open Skill Profiling & Verifiable Task Synthesis<br/>Runtime / quality / verifiability 3D filtering"]
B --> C["Governed Ready Skill Library<br/>Structured folders: SKILL.md/scripts/assets"]
C --> D["Pre-task Agentic Search & Recommendation<br/>Read-only filesystem inspection + usage guide"]
D --> E["Solver Agent Task Execution<br/>Executes long-horizon task with recommended skills"]
E --> F["Subtask Segmentation & Causal Attribution<br/>Breakpoint slicing + responsibility assignment"]
F --> G["Evidence-Gated Controlled Evolution<br/>Admissibility gate + batch aggregation + Edit/Create"]
G -->|Updates and creations| C
Key Designs¶
1. Open Skill Profiling & Verifiable Task Synthesis: Curating High-Assurance Ecosystem Artifacts
To transform heterogeneous open-source repositories into trustworthy agent assets, SkillsVote establishes a rigorous curation pipeline across 1.68 million candidates. After format checking via official validation scripts and content deduplication based on the earliest Git commit timestamps, a dedicated profiling agent inspects candidate directories using native tools (Grep, Glob, Read) to generate three structured schemas:
- Runtime Requirement Profile (SkillEnvironmentRubric): Formally documents required operating systems (Linux/macOS/Windows), write scopes (read/workspace/system), privilege assumptions, external connectivity (offline/online/secured), environment variables, required CLI binaries, and MCP servers.
- Quality Profile (SkillQualityRubric): Audits packages for internal goal consistency across text and scripts, operational completeness of prerequisites and command sequences, and reusable task orientation.
- Verifiability Profile (SkillVerifiabilityRubric): Assesses whether success conditions can be determined with low ambiguity, whether the runtime environment can be reproduced inside a container sandbox, and the cost of task construction.
For skills passing the verifiability rubric, SkillsVote automatically synthesizes executable benchmark tasks in Harbor container format, complete with instructions and objective verifier scripts, converting static instructional text into empirically measurable agent benchmarks.
2. Pre-task Agentic Search & Recommendation: Filesystem-Native Context Isolation
Standard skill retrieval methods rely on embedding-based top-k matches or superficial name/description routing, which degrade when navigating tens of thousands of overlapping candidates. SkillsVote models pre-task selection as an agentic search task over a local read-only filesystem:
A dedicated recommender agent operates within an isolated sandbox rooted at the candidate skill directory. Treating the downstream task instruction strictly as a capability requirement rather than an execution command, the agent navigates directories, inspects SKILL.md files, scripts, and references, and outputs a deduplicated list of selected skills (skill_names) alongside a synthesized usage guide (optimized_context). This guide specifies which task phase each skill covers, how to compose multiple tools, and explicit applicability boundaries. Only the selected skill directories are copied into the solver agent's environment, actively pruning distractors and preventing negative transfer.
3. Subtask Segmentation & Causal Attribution: Bridging Trajectories and Evolving Knowledge
To resolve the granularity mismatch between coarse task-level outcomes and uninformative single-step tool calls, SkillsVote introduces the subtask as the fundamental evolvable unit. A subtask is defined as the smallest semantically complete segment featuring a standalone goal, a primary evaluation signal (environment, human, or unknown), and at most one associated skill context. Trajectories are segmented strictly when one of these three boundaries shifts.
By resuming the solver agent's original session to retain native chain-of-thought and intermediate tool outputs, the attribution module compresses execution evidence along three axes:
- Outcome Evidence: Categorizes the verification source, distinguishing verifiable sandbox execution feedback from ungrounded claims.
- Responsibility Assignment: Maps outcomes to 11 causal categories, identifying whether success stems from skill guidance, independent exploration, or exploration despite irrelevant skills, and whether failures originate from flawed skill instructions, model reasoning limits, client-side container faults, or external service outages.
- Reusable Delta: Traces the exact line numbers in referenced skill documents (skill_refs) that shaped execution, extracting only portable invariant procedures, prerequisite checks, or recovery maneuvers while strictly discarding task-specific literals, gold answers, and temporary retries.
4. Evidence-Gated Controlled Evolution: Guarding Against Library Degradation and Spurious Edits
To prevent skill corruption over continuous interaction streams, SkillsVote enforces strict evidence gating and routing rules:
- Strict Admissibility: Only subtasks classified under the three successful exploration categories with non-empty reusable exploration payloads are admitted into evolution. Failed, ambiguous, or ungrounded subtasks are routed exclusively to diagnostic logs without updating persistent skills.
- Batch Aggregation: Admissible subtasks originating from the same batch that support identical procedural enhancements, preconditions, or workarounds are merged into a single consolidated proposal.
- Action Routing: Dispatches updates into five operational categories:
* error_fix: Corrects verified errors in existing skills (Edit).
* knowledge_addition: Inserts missing operational steps, fallback routines, or commands into an existing skill (Edit).
* prerequisite_addition: Adds dependency prerequisites, environment constraints, or warnings (Edit).
* create_skill: Generates a new standalone skill package when successful exploration spans beyond existing skill boundaries (Create).
* skip: Aborts modification when evidence is redundant or weakly supported.
A Worked Example¶
During offline evolution on Terminal-Bench Pro transferring to Terminal-Bench 2.0 Hard, a source task required configuring an Apache web server with rate limiting and comprehensive logging. The agent discovered that standard modules lacked precise concurrency throttling, successfully installed libapache2-mod-qos, configured QS_SrvMaxConnPerIP, and established persistent configurations under ports.conf and sites-available. The attribution module assigned this unit to success_no_skill_seen, and the evolution stage created a new skill ubuntu-apache-vhost.
In a subsequent offline task requiring an analytics virtual host, the agent followed ubuntu-apache-vhost to start the service, discovered systemctl was missing in the minimal container, fell back to service apache2 start, and validated bindings via apache2ctl -S. Attributed as success_skill_used_with_extra_exploration, this fallback rule was edited directly into ubuntu-apache-vhost.
When evaluated on the unseen Terminal-Bench 2.0 task configure-git-webserver (requiring a Git push to trigger Web deployment), the recommendation module retrieved ubuntu-apache-vhost. The solver agent reused the persistent service setup and end-to-end curl validation patterns, successfully solving the task on the first attempt, whereas the baseline agent without skills relied on an ad-hoc Node.js script that lacked persistence and runtime validation, resulting in test failure.
Harness Integration & Lifecycle Hook¶
SkillsVote is built atop the Harbor evaluation framework, using prebuilt container images with Node.js 22, nvm 0.40.4, and Codex CLI 0.125.0 to eliminate runtime installation latency. The solver agent executes with default system skills and plugins disabled to prevent extraneous prompt contamination. Attribution and evolution hook into Harbor's trial-end hook on the host side after container termination. If Harbor schedules a retry for an environmental failure, attribution is paused until the final attempt finishes, guaranteeing that only terminal execution evidence updates the library.
Key Experimental Results¶
Main Results¶
On Terminal-Bench 2.0 (89 terminal tasks, reported as avg@5 Accuracy over 5 independent runs) and SWE-Bench Pro (731 realistic software engineering tasks, reported as avg@1 Resolve Rate), the core evaluation results are:
| Benchmark & Metric | Backbone Model | Baseline w/o Skills | Online SkillsVote | Online ReasoningBank | Online skill-creator | Offline TB-Pro Transfer | Offline Curated Library |
|---|---|---|---|---|---|---|---|
| Terminal-Bench 2.0 (avg@5) | GPT-5.2 medium | 51.1 | 53.7 (+2.6) | 52.1 (+1.0) | 53.7 (+2.6) | 58.9 (+7.8) | 54.8 (+3.7) |
| Terminal-Bench 2.0 (avg@5) | GPT-5.4 mini | 51.7 | 52.8 (+1.1) | 50.6 (-1.1) | 47.6 (-4.1) | 57.5 (+5.8) | 55.7 (+4.0) |
| Terminal-Bench 2.0 (avg@5) | GPT-5.5 xhigh | 79.8 | 80.7 (+0.9) | 77.7 (-2.1) | 78.8 (-1.0) | 81.2 (+1.4) | 78.1 (-1.7) |
| SWE-Bench Pro (avg@1) | GPT-5.2 medium | 47.6 | 50.3 (+2.7) | 47.7 (+0.1) | 48.6 (+1.0) | – | 44.2 (-3.4) |
| SWE-Bench Pro (avg@1) | GPT-5.4 mini | 46.9 | 49.5 (+2.6) | 49.0 (+2.1) | 49.2 (+2.3) | – | 48.4 (+1.5) |
| SWE-Bench Pro (avg@1) | GPT-5.5 xhigh | 58.4 | 59.6 (+1.2) | 56.1 (-2.3) | 55.8 (-2.6) | – | 56.1 (-2.3) |
Across all six benchmark and backbone pairings, online SkillsVote consistently outperforms the baseline without skills and achieves the top or tied-for-top performance among online methods. Under the high-reasoning GPT-5.5 backbone, baselines suffer noticeable regressions (e.g., ReasoningBank and skill-creator drop by 2.3 and 2.6 percentage points on SWE-Bench Pro), whereas SkillsVote preserves steady positive gains.
Extended Routing and Recommendation Ablation¶
Evaluated on SkillRouter's public benchmark with 75 expert-verified queries across a distractor-heavy library scaling up to 79,141 candidate skills (FC@10 measures the percentage of queries where all ground-truth skills are retrieved in the top 10):
| Candidate Pool Size | Routing Method & Model | Hit@1 (%) | Recall@10 (%) | FC@10 (%) | Average Query Cost ($) |
|---|---|---|---|---|---|
| Full (79,141) | SkillRouter (0.6B Dual-Encoder) | 65.3 | 67.2 | 46.7 | – |
| Full (79,141) | SkillsVote (GPT-5.4 mini xhigh) | 52.0 | 48.0 | 34.7 | 0.076 |
| Full (79,141) | SkillsVote (GPT-5.4 xhigh) | 65.3 | 66.1 | 54.7 | 0.090 |
| Full (79,141) | SkillsVote (GPT-5.5 xhigh) | 70.7 | 74.2 | 62.7 | 0.158 |
Task-level delta ablation of the pre-task recommendation module on Terminal-Bench 2.0 Hard reveals its role in eliminating negative transfer: - Online without Recommendation (Direct Exposure): Mean gain/loss delta is +3.3% / -6.7% (net negative transfer). - Online with Recommendation: Mean delta shifts to a balanced +6.0% / -6.0% (suppressing noise from under-specified skills). - Offline without Recommendation: Mean gain/loss delta is +11.3% / -3.3%. - Offline with Recommendation: Mean delta improves to +15.3% / -2.0% (maximizing positive transfer while curbing degradation).
Key Findings¶
- Superiority of Trajectory-Derived Procedural Transfer: Frozen libraries synthesized from only 48 historical Terminal-Bench Pro tasks transfer substantially better to unseen Terminal-Bench 2.0 tasks (+7.8% on GPT-5.2, +5.8% on GPT-5.4 mini) than the 10k curated open-source library, proving that attributed execution invariants capture transferable procedural workflows across distribution shifts.
- Complementarity of Recommendation and Evolution: Evolution acts as a generator of reusable skills, while recommendation functions as a task-conditioned gatekeeper. Uncontrolled exposure of the evolving library degrades performance due to attention dilution and distractor confusion.
- GitHub Stars Do Not Indicate Execution Quality: Profiling over 1.68 million skills reveals that more than 79% reside in repositories with fewer than 100 stars; progressive quality filtering does not shift this distribution, showing that popularity metrics cannot substitute for explicit runtime and verifiability governance.
Highlights & Insights¶
- Agent Skills as Version-Controlled Software Artifacts: Managing skills as directory structures with
SKILL.md, scripts, and resources enables Git-like differential updates (edits vs. creations) and auditable governance rather than ephemeral prompt concatenation. - Subtask Causal Attribution Decouples Environment from Skill Quality: Slicing trajectories by objective evaluation signals isolates external environment failures from true skill deficiencies, preventing destructive updates triggered by transient sandbox glitches.
- Zero-Gradient Test-Time Lifelong Learning: Delivers cumulative performance improvements for frozen commercial models purely through the structured accumulation and refinement of external skill libraries without model fine-tuning.
Limitations & Future Work¶
- Domain Scope Centered on Verifier-Backed Tasks: Evaluated primarily on terminal interaction and software repository debugging benchmarks with programmatic test suites, leaving open-ended creative writing, multimodal physical embodiment, and human-preference-centric tasks for future exploration.
- Dependence on Trajectory and Verifier Fidelity: The attribution module relies on transparent container execution logs and informative test summaries; incomplete verifiers or opaque runtime logs may cause conservative policies to discard valid exploratory insights.
- Harness and Cross-Model Generalization: Validated within Codex-driven Harbor setups; cross-harness portability across varied agent runtime abstractions requires broader cross-platform evaluation.
Related Work & Insights¶
- vs ReasoningBank / Reflexion / ExpeL: While early memory methods record unstructured textual heuristics or raw trajectory exemplars, SkillsVote packages executable procedural assets and performs fine-grained subtask attribution to eliminate credit assignment ambiguity.
- vs SkillRouter / DCI: SkillRouter employs dense bi-encoders over skill text that degrade under 80k confusable pools; SkillsVote leverages filesystem-native agentic search, boosting complete coverage (FC@10) from 46.7% to 62.7% with sub-linear cost scaling.
- vs skill-creator / Trace2Skill: Unlike unconstrained trajectory-to-skill generators that encode incidental state, SkillsVote introduces explicit admissibility gating, responsibility attribution, and batch aggregation to guarantee defensive, pollution-free evolution.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first unified full-lifecycle governance paradigm spanning open-source profiling, interactive pre-task recommendation, subtask causal attribution, and evidence-gated evolution.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across a million-scale corpus audit, 80k-scale stress-test routing, and extensive multi-model benchmarks on Terminal-Bench 2.0 and SWE-Bench Pro.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate taxonomy, detailed schema formalisms, and transparent case walkthroughs illustrating exact failure modes and transfer mechanisms.
- Value: ⭐⭐⭐⭐⭐ Delivers an indispensable software-engineering blueprint for scalable, contamination-free lifelong experience accumulation in autonomous AI agents.