ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/C-lister/ChainSWE
Area: Code Intelligence
Keywords: Coding Agents, Continuous Software Maintenance, Dependency Evolution, Accumulated State Errors, SWE Benchmark
TL;DR¶
Introduces ChainSWE, the first benchmark evaluating coding agents on sequential multi-bug software maintenance across shared codebases, demonstrating that accumulated repository state drift causes resolution rates to drop by up to 70% along the repair chain.
Background & Motivation¶
Language model (LM) agents have demonstrated substantial automated capabilities across various repository-level software engineering (SWE) tasks, spanning autonomous code repair, automated build environment setup, and full-repository synthesis. Prevailing evaluation benchmarksโincluding SWE-bench and its recent extensions such as SWE-bench Live, SWE-rebench, SWE-bench Pro, and SWE-Gymโall follow an isolated, single-issue evaluation paradigm: for every task, a clean sandbox container is spun up at a pre-specified base commit, the agent receives an isolated problem statement, edits code locally, and is graded against a hard-coded test suite. This protocol effectively collapses a continuous software maintenance lifecycle into disjoint, memoryless sessions, ignoring the cumulative multi-file dependencies and long-horizon evolution inherent to real-world engineering.
In real-world software maintenance, human developers do not operate in a vacuum; they must resolve continuous streams of related defects on top of evolving repository states that they themselves have just modified. An isolated testing protocol fails to examine whether an agent avoids extraneous edits that pollute the baseline, whether it satisfies latent multi-file preconditions, or whether it can maintain system consistency across successive repairs. When an agent is deployed to fix multiple bugs in sequence within the same codebase, minor regressions or omitted sub-edits from earlier patches compound over time, producing severe downstream degradation that isolated benchmarks cannot capture.
To bridge this crucial evaluation gap, the field requires a realistic benchmark featuring chronological defect chains with verifiable code dependencies and containerized replay capabilities. Core idea: construct ChainSWE by mining chronological bug chains with AST-verified code overlap across real Python repositories, evaluating coding agents on their self-generated, evolving repository states to decouple single-shot bug-fixing capability from accumulated state management and expose chain errors driven by overshoot and undershoot.
Method¶
Overall Architecture¶
ChainSWE establishes an end-to-end pipeline that pools, filters, validates, and benchmarks continuous bug-fixing sequences harvested from six existing SWE-bench-family datasets. Rather than simply measuring whether an agent can produce an isolated fix, the framework rigorously assesses whether earlier patches leave behind a sound and viable substrate for subsequent tasks. The entire pipeline comprises chronological chain mining, code-overlap and containerized test replay filtering, difficulty alignment, and a controlled three-mode evaluation protocol.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Defect Pool & Chronological Clustering<br/>6 SWE benchmarks / repo commit sliding window"] --> B["Code-Overlap & Containerized Replay Filter<br/>AST symbol overlap + Oracle test replay"]
B --> C["Chain Selection & Tractability Filtering<br/>Drop len-2 redundancy / Qwen3.7-Max tractability"]
C --> D["Three-Mode Evaluation Protocol<br/>ORACLE vs. SEQ vs. SEQ+MEM"]
D --> E["Chain Error Diagnosis & Failure Fingerprinting<br/>Overshoot / Undershoot & regression attribution"]
Key Designs¶
1. Code-Overlap and Containerized Replay Filter: Guaranteeing Strong Dependencies and Buildability
To ensure candidate chains represent genuinely interdependent bug streams rather than disconnected tasks within the same repository, the pipeline applies a multi-stage filtering protocol. For every repository in the pooled source, issues are sorted chronologically by commit date. A sliding window links consecutive instances if the latter's base commit is the immediate successor of the former, or if both gold and test patches apply cleanly without merge conflicts onto the resolved state. An AST parser then inspects the git diff hunks to extract modified files, functions, and classes, immediately pruning any candidate window with zero pairwise overlap. Finally, a containerized filter executes in Docker from the first bug's base commit, sequentially applying test and gold patches; a candidate chain is accepted only if every single bug turns its FAIL_TO_PASS tests green without regressing any PASS_TO_PASS tests.
2. Difficulty-Aware Selection and Non-Redundant Chain Filtering: Isolating Cross-Task State Overhead If a chain fails simply because a single isolated issue is intrinsically untestable or overly complex, it provides minimal insight into how well an agent carries state across successive fixes. ChainSWE therefore incorporates a two-phase data selection mechanism. First, all candidate chains of length 2 are pruned: they lack sufficient depth to stress cumulative state compounding, and virtually all length-2 sequences exist as contiguous prefixes inside length-3 or longer chains. Second, a difficulty-based filter evaluates each candidate chain under ORACLE mode using Qwen3.7-Max, retaining only chains where more than half of the constituent bugs (at least \(\lfloor k/2 \rfloor + 1\) bugs for a chain of length \(k\)) are solved in isolation. This eliminates tasks dominated by individual bug hardness and guarantees that observed degradations stem from accumulated repository state drift.
3. Three-Mode Evaluation Protocol: Decoupling Repository State from Dialogue Context To systematically isolate and quantify different sources of failure across continuous rollouts, ChainSWE employs a standardized SWE-EDIT agent harness across three orthogonal modes: - ORACLE Mode: Before solving each bug, the container resets to the initial commit and automatically applies gold fix and test patches for all prior bugs in the chain. The agent operates exclusively on an oracle-quality codebase, establishing an upper-bound baseline of isolated task difficulty. - SEQ (Sequential) Mode: The agent works through bugs 1 to \(n\) in sequence within a single persistent container. Each generated patch is committed and forms the starting substrate for the next bug. The agent harness and conversation history are reset between bugs, isolating the performance penalty caused purely by accumulated physical repository drift. - SEQ+MEM (Sequential + Memory) Mode: Identical to SEQ in persistent repository accumulation, but additionally carries the agent's full multi-turn conversational transcript across tasks. This isolates the contribution of dialogue-level memory and investigates whether retaining past interaction history can mitigate code-level state corruption.
4. Chain Error Diagnosis and Failure Fingerprinting: Exposing Mechanisms of Hidden Breakage
A failure where an issue passes under ORACLE but fails under SEQ is formally classified as a Chain Error. These errors originate primarily from two complementary failure modes: Overshoot, where an agent touches unrelated files beyond the gold patch scope, corrupting regions that downstream bugs later target; and Undershoot, where an agent addresses only the files explicitly mentioned in the prompt, omitting necessary refactors and leaving unmet preconditions. When executed in the evaluation harness, chain errors manifest as three distinct behavioral fingerprints: breaking previously functional code (Broke working code, 40%), failing to establish preconditions to resolve the current bug (Left bug unfixed, 25%), or producing malformed code that prevents patch application or crashes test collection (Produced no usable patch, 34%).
A Worked Example¶
Consider a representative chain from amaranth-lang/amaranth. Bug 1 (amaranth-1265) requires a gold refactoring spanning four interdependent files: _ast.py, _ir.py, _xfrm.py, and sim/_pyrtl.py. During evaluation, the agent modifies only _ir.py and _xfrm.py (along with top-level exports), completely omitting _ast.py and the simulation backend, thereby exhibiting an Undershoot error. When the subsequent bug (amaranth-1268) begins in SEQ mode, the agent produces an accurate modification to its designated target lib/wiring.py. However, running the test suite causes test_is_compliant to fail because the underlying AST precondition from Bug 1 was never established. The exact same Bug 2 passes cleanly under ORACLE mode, proving that the failure is entirely an artifact of inherited state incompleteness.
Key Experimental Results¶
Main Results¶
Evaluating seven state-of-the-art language models on ChainSWE across 100 complete chains (304 individual bugs) using the SWE-EDIT scaffold under three context configurations (BASELINE raw history, SUMMARIZE periodic compaction, SUB-AGENT delegation) highlights a uniform degradation from ORACLE to sequential execution.
| Model | Context Window | BASELINE Per-Bug (%) | BASELINE Chain (%) | SUMMARIZE Per-Bug (%) | SUB-AGENT Per-Bug (%) |
|---|---|---|---|---|---|
| GPT-5.5 (ORACLE) | 1M | 69.1 | 26.0 | 68.1 | 62.5 |
| GPT-5.5 (SEQ) | 1M | 49.0 (-29%) | 23.0 (-12%) | 42.4 (-38%) | 35.2 (-44%) |
| GPT-5.5 (SEQ+MEM) | 1M | 47.4 (-31%) | 19.0 (-27%) | 49.7 (-27%) | 43.4 (-31%) |
| Claude-Opus-4.7 (ORACLE) | 1M | 64.5 | 22.0 | 65.5 | 63.5 |
| Claude-Opus-4.7 (SEQ) | 1M | 40.5 (-37%) | 18.0 (-18%) | 39.5 (-40%) | 38.0 (-40%) |
| Claude-Opus-4.7 (SEQ+MEM) | 1M | 39.5 (-39%) | 17.0 (-23%) | 41.5 (-37%) | 40.0 (-37%) |
| DeepSeek-V4-Pro (ORACLE) | 1M | 42.8 | 10.0 | 50.3 | 48.4 |
| DeepSeek-V4-Pro (SEQ) | 1M | 29.6 (-31%) | 5.0 (-50%) | 27.3 (-46%) | 25.0 (-48%) |
| DeepSeek-V4-Pro (SEQ+MEM) | 1M | 28.6 (-33%) | 9.0 (-10%) | 26.0 (-48%) | 28.0 (-42%) |
| Gemini-3.1-Pro (ORACLE) | 1M | 61.8 | 20.0 | 61.5 | 61.5 |
| Gemini-3.1-Pro (SEQ) | 1M | 36.5 (-41%) | 18.0 (-10%) | 39.5 (-36%) | 37.2 (-40%) |
| Gemini-3.1-Pro (SEQ+MEM) | 1M | 37.5 (-39%) | 18.0 (-10%) | 39.8 (-35%) | 37.2 (-40%) |
Ablation Study: Position Depth and Context Management Strategy¶
Across 97 representative chains of length 3, evaluating per-position resolution reveals how performance degrades as tasks deepen along the chain (averaged over all seven models). Under ORACLE resets, tasks at deeper positions show slightly higher resolution rates (rising from 58.6% to 67.0%), whereas sequential execution experiences catastrophic collapse.
| Context Config | Eval Mode | Pos. 1 (%) | Pos. 2 (%) | Pos. 3 (%) | Relative Drop at Pos. 3 vs. ORACLE |
|---|---|---|---|---|---|
| BASELINE | ORACLE | 58.6 | 64.3 | 67.0 | - |
| BASELINE | SEQ | 57.1 | 39.3 | 27.7 | -59% |
| BASELINE | SEQ+MEM | 56.5 | 38.0 | 28.2 | -58% |
| SUMMARIZE | ORACLE | 58.0 | 66.0 | 68.4 | - |
| SUMMARIZE | SEQ | 60.0 | 35.4 | 20.6 | -70% |
| SUMMARIZE | SEQ+MEM | 59.3 | 36.6 | 22.0 | -68% |
| SUB-AGENT | ORACLE | 49.5 | 58.1 | 59.2 | - |
| SUB-AGENT | SEQ | 50.3 | 29.9 | 17.5 | -70% |
| SUB-AGENT | SEQ+MEM | 50.4 | 31.3 | 19.2 | -67% |
Key Findings¶
- Compounding Penalty at Deeper Positions: At Position 1, SEQ matches ORACLE performance across all models (e.g., 62.5% vs. 61.2% for GPT-5.5). However, by Position 3, resolution rates plummet (e.g., GPT-5.5 drops to 28.2% under SEQ, a 63% decline; GPT-5.4-nano falls from 59.1% to 21.6%). Across rewriting configurations, degradation at Position 3 reaches 70%.
- Marginal Benefit from Transcript Memory: Preserving conversational transcripts across bugs (SEQ+MEM, averaging 36.9% across models) yields negligible improvement over resetting history (SEQ, averaging 36.5%). Except for frontier reasoning models like GPT-5.5 which show modest gains (+4.6 points averaged over configs), dialogue history cannot reverse physical code damage already committed to disk.
- Raw History Outperforms Active Context Rewriting: Raw interaction retention (BASELINE) consistently demonstrates higher robustness than active summarization (SUMMARIZE, -40% drop) or sub-agent editing delegation (SUB-AGENT, -41% drop). Summarization and delegation introduce information loss and hallucinated abstractions that exacerbate patch misalignment.
- Undershoot Dominates Chain Errors: Among chain errors traceable to a single shared file, prior omissions (undershoot) outnumber extraneous edits (overshoot) by roughly 9 to 1, showing that agents are far more prone to shipping partial, myopic fixes than making rampant unwanted modifications.
Highlights & Insights¶
- Exposing the Long-Horizon Deployment Gap: Moving beyond isolated single-shot leaderboards, ChainSWE establishes the first controlled environment that quantifies how self-generated code drift degrades continuous agentic maintenance.
- Decoupling Physical Repository State from Cognitive History: By comparing ORACLE, SEQ, and SEQ+MEM, the paper provides rigorous empirical proof that conversation memory alone cannot compensate for underlying repository corruption.
- Counter-Intuitive Context Management Dynamics: Popular long-context techniques such as conversation summarization and sub-agent offloading, while effective for token budgeting in single-turn benchmarks, act as error amplifiers in continuous multi-task rollouts.
Limitations & Future Work¶
- Language Coverage Restricted to Python: Due to the ecosystem constraints of existing SWE-bench datasets, ChainSWE currently evaluates only Python codebases, omitting statically typed environments (e.g., Java, Rust, Go) where compiler diagnostics could catch interface regressions earlier.
- Constrained Chain Length: Verified, overlapping multi-bug chains in public repositories are scarce, limiting current chain lengths to 3โ5 instances; longer industrial maintenance horizons remain to be explored.
- Need for Repository-Level Tracking and Rollback: Future agent harnesses must incorporate cross-task dependency graphing, automated regression testing, and explicit checkpointing/rollback mechanisms rather than relying purely on LLM in-context reasoning.
Related Work & Insights¶
- vs SWE-bench / SWE-bench Live / SWE-bench Pro: Standard benchmarks evaluate models on isolated issues with complete sandbox resets; ChainSWE introduces sequential dependency, showing that high single-shot accuracy does not guarantee multi-step codebase stewardship.
- vs Long-term Conversational Memory Benchmarks (e.g., LongMemEval): Traditional memory benchmarks focus on dialogue-level recall across multi-session interactions with a passive environment; ChainSWE requires reasoning over active, persistent code changes with concrete execution semantics.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering continuous, multi-bug SWE benchmarking that exposes self-generated state drift.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 7 frontier LMs, 3 context strategies, and 100 verified bug chains, backed by comprehensive failure mode attribution.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, clean conceptual definitions, and rich empirical evidence.
- Value: โญโญโญโญโญ Provides critical insights and essential diagnostic tools for the next generation of production-ready software engineering agents.