Skip to content

Evaluating Coding Agents on Kernel Exploit Generation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://kex-bench.github.io
Area: Reasoning
Keywords: coding agents, kernel exploitation, exploit primitives, interactive debugging, deterministic verifier

TL;DR

The authors introduce KEX-bench, the first benchmark evaluating coding agents on deterministic kernel exploit primitive generation across 40 real-world Linux and Windows CVEs, revealing that while frontier models frequently trigger kernel faults, turning raw crashes into controlled exploit primitives remains a fundamental bottleneck.

Background & Motivation

Autonomous coding agents powered by frontier large language models have demonstrated rapid progress in real-world vulnerability discovery. Recent achievements include Google Big Sleep autonomously uncovering an exploitable SQLite stack-buffer underflow missed by 150 CPU-hours of coverage-guided fuzzing, and systems like Claude Mythos discovering zero-day vulnerabilities across production web browsers and operating system kernels. However, in security engineering and capability assessment, bug discovery does not equate to exploitability. A simple fuzzing crash or stack trace merely indicates that an untrusted path was reached or an unhandled exception occurred. It fails to measure whether an agent can bypass modern kernel mitigations such as Kernel Address Space Layout Randomization (KASLR), shape intricate kernel allocator state, or achieve precise control-flow redirection under strict hardware privilege boundaries.

Existing cybersecurity benchmarks fail to measure this low-level capability boundary. Popular Capture-The-Flag (CTF) suites like Cybench or NYU CTF Bench rely on stylized, pre-packaged challenges that lack the real-world scale and complex asynchronous state machines of production operating systems. Conversely, real-world benchmarks such as CVE-Bench focus predominantly on web applications and high-level software architectures, avoiding the nuances of kernel-level memory corruption. In studies evaluating agents against known one-day vulnerabilities, evaluations frequently rely on unconstrained end-to-end local privilege escalation (LPE) or treat unhandled kernel panics as successful outcomes, omitting fine-grained, deterministic verification of underlying kernel memory states. Consequently, current research cannot discern whether an agent merely destabilizes a system by accident or genuinely comprehends the underlying memory layout to craft reliable, controlled exploit primitives.

A rigorous assessment framework requires placing coding agents within a sandboxed, iterative development loop: compiling C payloads, inspecting KASAN/WinDbg diagnostics, and refining attack strategies against an isolated, resettable execution environment. More importantly, the benchmark must replace subjective textual claims or unhandled crashes with tamper-proof, deterministic verification anchored in live kernel state. Core idea: introduce KEX-bench, a benchmark of 45 task instances across 40 real-world Linux and Windows CVEs evaluating coding agents on five distinct exploit primitives via isolated virtual machines and deterministic, unforgeable verifiers, uncovering a stark "trigger-to-primitive" capability gap where agents frequently induce crashes but fail to shape them into verified primitives.

Method

Overall Architecture

KEX-bench abstracts kernel exploitation across Linux and Windows into a standardized agent–proxy–MCP–VM–verifier pipeline. The agent interacts exclusively through the Model Context Protocol (MCP), where platform-specific tools provide controlled execution, compilation, interactive debugging, and binary analysis under a strict global tool-call budget enforced by a stateless proxy. Exploits execute in isolated QEMU/KVM virtual machines with snapshot rollback capabilities, ensuring complete state resets across attempts. Success is determined strictly by an external, deterministic verifier that directly inspects kernel memory structures rather than relying on agent self-reporting or system crashes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Description Input<br/>Platform + Target CVE Patch + Requested Primitive"] --> B["Controlled MCP Protocol Layer<br/>Static Reverse Engineering + Build/Execution + Live Debugging"]
    B --> C["Dual-Platform Isolated Execution Sandboxes<br/>Linux QEMU/virtme-ng & Windows Paired Debugger/Debuggee VMs"]
    C --> D["Deterministic Kernel Primitive Verifiers<br/>Verifier Modules & WinDbg State Readback of Live Kernel States"]
    D --> E["Dual-Setting Closed-Loop Evaluation<br/>w/o PoC Autonomous Construction vs w/ PoC Adaptation"]

Key Designs

1. Controlled MCP Protocol Layer: Standardizing Tools and Enforcing Strict Call Budgets

To simulate a realistic exploit development workflow without granting arbitrary host privileges, KEX-bench standardizes all agent interactions through platform-specific Model Context Protocol (MCP) servers. On Linux, the interface exposes the Verifier MCP (compilation and execution within the guest VM, primitive verification), Debugger MCP (interactive GDB scripting, runtime symbol resolution, pahole struct layout inspection, and KASAN diagnostic reports), and Filesystem MCP (read-only ripgrep access to kernel source trees). On Windows, it provides the Kernel MCP (guest PowerShell execution, snapshot rollbacks, crash validation), PyGhidra MCP (headless decompilation, cross-reference analysis, and call-graph generation for closed-source drivers), and WinDbg-ext MCP (validated live kernel debugging commands and pool inspection). A proxy intercepts all traffic to strictly enforce ceilings of 200 counted calls on Linux and 300 on Windows, bounding long-horizon exploration.

2. Dual-Platform Isolated Execution Sandboxes: Guaranteeing Idempotence and Host Safety

Because kernel exploits frequently result in unrecoverable system hangs, corrupt slab caches, or trigger catastrophic kernel panics, KEX-bench implements isolated virtualization topologies for both target platforms. On Linux, each CVE runs inside a dedicated QEMU/KVM instance managed via virtme-ng with an OverlayFS root filesystem, enabling sub-second rollbacks to a pristine kernel state where agent payloads execute under unprivileged user credentials. On Windows, KEX-bench introduces a novel "paired-VM topology": a Debuggee VM executes the unpatched target driver, while a separate Debugger VM connects to it over the network via KDNET (Windows kernel debugging protocol), sharing a synchronized workspace over VirtIO-FS. Even when a payload triggers a BugCheck (BSOD) or freezes the Debuggee, the Debugger VM remains fully operational, enabling automated crash dump inspection and post-crash register analysis before rolling back the Debuggee from a per-CVE base disk snapshot.

3. Deterministic Kernel Primitive Verifiers: Eliminating False Positives and Crash Spoofing

KEX-bench strictly rejects system crashes, error return codes, or agent-generated textual claims, requiring tangible proof of kernel-state manipulation across five foundational primitives. On Linux, four primitives are verified: LEAK requires calculating and reporting the true randomized kernel text base (_text), verified by reading /proc/kallsyms; RIP demands redirecting kernel control flow to cause a General Protection Fault at a fresh, random 64-bit target address, verified via kernel ring buffer inspection; HEAPR and HEAPW utilize a custom in-kernel heap_verifier module that allocates monitored victim chunks across 9 kmalloc size classes (16 to 2048 bytes) populated with boot-time dynamic random 8-byte magic values, requiring the agent to read out or overwrite the monitored chunk. On Windows, Arbitrary Address Write (AAW) selects a dynamic memory location in ntoskrnl.exe and a randomized 64-bit goal value before each attempt, accepting success only when the Debugger VM reads back that exact value at that exact kernel address.

4. Dual-Setting Closed-Loop Evaluation: Disentangling Vulnerability Reachability from Exploitation

To diagnose where agents fail along the multi-stage exploitation chain, KEX-bench evaluates each task under two distinct regimes. In the without-PoC (w/o PoC) setting, the agent receives only the high-level CVE description, the vendor security patch diff, and the target primitive specification, forcing it to independently deduce the vulnerable path, identify reachability preconditions, trigger the fault, and construct the exploit primitive. In the with-PoC (w/ PoC) setting, the agent is additionally supplied with a public or reference Proof-of-Concept that reliably triggers the vulnerability, eliminating the trigger discovery bottleneck and directly measuring the model's capacity to adapt an existing bug trigger into a verifier-accepted exploit primitive.

Key Experimental Results

Main Results

The authors evaluated 11 agent-model configurations spanning 3 agent scaffolds (CLAUDECODE, CODEX, OPENCODE) and 7 frontier or open-weight foundation models across all 45 tasks. Each configuration was evaluated using one bounded trajectory per task (pass@1).

The following table reports the solved tasks and pass rates across Linux (25 tasks) and Windows (20 tasks) under both settings (derived from Table 2 in the paper):

Agent Scaffold Model w/o PoC: Linux (25) w/o PoC: Win AAW (20) w/o PoC: Total (45) w/ PoC: Linux (25) w/ PoC: Win AAW (20) w/ PoC: Total (45)
CLAUDECODE Opus 4.6 14/25 (56.0%) 1/20 (5.0%) 15/45 (33.3%) 23/25 (92.0%) 8/20 (40.0%) 31/45 (68.9%)
CLAUDECODE Sonnet 4.6 7/25 (28.0%) 0/20 (0.0%) 7/45 (15.6%) 19/25 (76.0%) 5/20 (25.0%) 24/45 (53.3%)
CLAUDECODE Haiku 4.5 0/25 (0.0%) 0/20 (0.0%) 0/45 (0.0%) 0/25 (0.0%) 0/20 (0.0%) 0/45 (0.0%)
CODEX GPT-5.4 5/25 (20.0%) 1/20 (5.0%) 6/45 (13.3%) 12/25 (48.0%) 7/20 (35.0%) 19/45 (42.2%)
CODEX GPT-5.4 Mini 1/25 (4.0%) 0/20 (0.0%) 1/45 (2.2%) 3/25 (12.0%) 2/20 (10.0%) 5/45 (11.1%)
CODEX GPT-5.3-Codex 3/25 (12.0%) 1/20 (5.0%) 4/45 (8.9%) 6/25 (24.0%) 5/20 (25.0%) 11/45 (24.4%)
OPENCODE Sonnet 4.6 4/25 (16.0%) 0/20 (0.0%) 4/45 (8.9%) 10/25 (40.0%) 6/20 (30.0%) 16/45 (35.6%)
OPENCODE Haiku 4.5 0/25 (0.0%) 0/20 (0.0%) 0/45 (0.0%) 2/25 (8.0%) 0/20 (0.0%) 2/45 (4.4%)
OPENCODE GPT-5.4 Mini 0/25 (0.0%) 0/20 (0.0%) 0/45 (0.0%) 2/25 (8.0%) 0/20 (0.0%) 2/45 (4.4%)
OPENCODE Kimi K2.5 0/25 (0.0%) 0/20 (0.0%) 0/45 (0.0%) 2/25 (8.0%) 0/20 (0.0%) 2/45 (4.4%)
OPENCODE Gemma 4 31B 0/25 (0.0%) 0/20 (0.0%) 0/45 (0.0%) 1/25 (4.0%) 0/20 (0.0%) 1/45 (2.2%)

Primitive-Specific and Failure Mode Analysis

To examine why agents struggle during autonomous exploit synthesis, the evaluation recorded confirmed bug triggers (kernel panics or KASAN reports) versus verifier successes. The following table contrasts trigger reachability against verified primitive success under the w/o PoC setting (derived from Table 3 in the paper), highlighting the trigger-to-primitive conversion rate:

Configuration Linux Triggers / KASAN (Total 25) Linux Verified Success (Total 25) Linux Conversion Rate (Success / Trigger) Windows Triggers (Total 20) Windows Verified Success (Total 20) Windows Conversion Rate (Success / Trigger)
CLAUDECODE / Opus 4.6 21 / 25 14 / 25 66.7% 5 / 20 1 / 20 20.0%
CLAUDECODE / Sonnet 4.6 15 / 25 7 / 25 46.7% 1 / 20 0 / 20 0.0%
CLAUDECODE / Haiku 4.5 3 / 25 0 / 25 0.0% 0 / 20 0 / 20 0.0%
CODEX / GPT-5.4 17 / 25 5 / 25 29.4% 5 / 20 1 / 20 20.0%
CODEX / GPT-5.4 Mini 6 / 25 1 / 25 16.7% 1 / 20 0 / 20 0.0%
CODEX / GPT-5.3-Codex 17 / 25 3 / 25 17.6% 5 / 20 1 / 20 20.0%
OPENCODE / Sonnet 4.6 12 / 25 4 / 25 33.3% 1 / 20 0 / 20 0.0%
OPENCODE / GPT-5.4 Mini 5 / 25 0 / 25 0.0% 0 / 20 0 / 20 0.0%
OPENCODE / Kimi K2.5 4 / 25 0 / 25 0.0% 0 / 20 0 / 20 0.0%
OPENCODE / Gemma 4 31B 1 / 25 0 / 25 0.0% 0 / 20 0 / 20 0.0%

Looking across individual primitives (Figure 4), success rates vary substantially. In the w/o PoC setting, LEAK achieves 9/99 (9.1%), RIP achieves 13/66 (19.7%), HEAPR achieves 6/66 (9.1%), HEAPW achieves 6/44 (13.6%), and Windows AAW drops to 3/220 (1.4%). With a reference PoC provided, all rates increase significantly: LEAK to 26/66 (39.4%), RIP to 26/66 (39.4%), HEAPR to 17/66 (25.8%), HEAPW to 18/44 (40.9%), and AAW to 33/220 (15.0%).

Key Findings

  • The Trigger-to-Primitive Gap: Reaching a vulnerable code path and provoking a kernel fault is largely tractable for leading models (CLAUDECODE/Opus 4.6 triggers faults in 84% of Linux tasks and CODEX/GPT-5.4 in 68%). However, converting that unstable fault into a verified primitive is where agents predominantly fail. For CLAUDECODE/Opus 4.6 on Linux w/o PoC, 7 of its 11 failed tasks successfully provoked KASAN reports or kernel crashes before failing to establish the target primitive.
  • Reference PoCs Provide Massive Leverage: Supplying a reference PoC roughly triples the overall benchmark success rate, rising from 7.5% (37/495) to 22.8% (113/495) across all configurations. The PoC eliminates the need to reverse-engineer driver entry points, craft complex netlink message sequences, or resolve complex reachability constraints, freeing the budget for primitive shaping.
  • Scaffold Coordination Dictates Success: In a strictly controlled comparison of 70 tasks using the identical Sonnet 4.6 model, prompts, and tool sets (Table 5), CLAUDECODE solved 31 tasks compared to OPENCODE's 20 (\(p = 0.035\), McNemar test). Trajectory analysis indicates that OPENCODE made substantially more filesystem calls and compilation attempts but frequently exhausted its budget before converting execution feedback into targeted verifier submissions.

Highlights & Insights

  • Shifting the Benchmark Paradigm from Crash to Primitive: KEX-bench formally establishes that evaluating cybersecurity agents on crashes alone provides a misleading measure of offensive capability. By defining five concrete kernel memory-state primitives, it isolates true exploit-shaping logic from accidental instability.
  • Zero False-Positive Dynamic In-Kernel Verifiers: The benchmark's verification suite underwent an extensive validation campaign across 108 test cases and 324 fresh VM executions (Table 4), demonstrating zero false acceptances across output spoofing, crash-only submissions, stale replay artifacts, and wrong-value writes.
  • Paired-VM Architecture for Real-World Debugging: Separating the victim execution environment from the debugger environment mirrors professional vulnerability research methodologies, allowing automated agents to leverage WinDbg KDNET kernel hooks even across catastrophic operating system crashes.

Limitations & Future Work

  • Platform and Primitive Confounding: In the current dataset, the four primitives LEAK, RIP, HEAPR, and HEAPW exist only on Linux, whereas AAW exists only on Windows. Furthermore, Linux offers full source code while Windows provides compiled binary drivers, preventing an orthogonal separation between operating system complexity and primitive difficulty.
  • Single-Trajectory Pass@1 Measurement: Primary results represent a single bounded trajectory per cell. While a repeated 5-run study on CODEX/GPT-5.4 established stable mean performance (40.8% on Linux, 24.0% on Windows), stochastic agent exploration across complex search spaces leaves noticeable run-to-run variance on individual tasks.
  • Lack of Multi-Primitive Exploit Chaining: KEX-bench stops at individual primitive generation rather than evaluating the complete weaponization pipeline (e.g., combining a LEAK with an AAW to overwrite PreviousMode and achieve system privilege escalation).
  • vs Cybench & NYU CTF Bench (Zhang et al., 2025; Shao et al., 2024): Traditional cybersecurity benchmarks assess synthetic user-space CTF puzzles; KEX-bench targets production-grade Linux and Windows kernels affected by real-world CVEs under active mitigation schemes.
  • vs CVE-Bench & ExploitGym (Zhu et al., 2025b; Wang et al., 2026): CVE-Bench evaluates web applications, while ExploitGym emphasizes user-space binaries and full exploit scripts. KEX-bench uniquely formalizes deterministic verification of kernel-space state transitions.
  • vs Classical AEG Systems (Avgerinos et al., 2011; Cha et al., 2012): Classical Automatic Exploit Generation relies on symbolic execution and SMT solvers, which scale poorly on large OS kernels due to path explosion. KEX-bench evaluates LLM agents navigating these complex state spaces through natural language reasoning and iterative tool feedback.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first benchmark for verifier-confirmed kernel exploit primitive generation, introducing the "trigger-to-primitive" capability paradigm.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage of 40 real CVEs, 45 tasks, 11 agent-model configurations, 324 verifier validation tests, and deep qualitative trajectory analysis.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear exposition, precise technical definitions of kernel exploitation mechanics, and insightful failure taxonomy.
  • Value: ⭐⭐⭐⭐⭐ Serves as a landmark benchmark for evaluating the autonomous reasoning boundaries of coding agents in safety-critical, low-level systems.