Skip to content

The Limits of Long-Context Reasoning in Automated Bug Fixing

Conference: ICLR 2026 arXiv: 2602.16069 Code: None Area: Agent / Code Keywords: long-context reasoning, automated bug fixing, SWE-bench, agentic workflow, context window

TL;DR

This paper systematically evaluates the limits of current LLMs in long-context code debugging. It finds that the success of agentic workflows stems from task decomposition rather than long-context reasoning (successful trajectories consume only 20–30K tokens), while performance degrades sharply under 64K single-pass patch generation (GPT-5-nano achieves 0%), revealing a significant gap between nominal context length and actual usable context capacity.

Background & Motivation

Background: LLMs have made notable progress in code repair, with resolve rates on benchmarks such as SWE-bench steadily improving, primarily through agentic workflows (e.g., SWE-agent).

Limitations of Prior Work: The success of agentic approaches is commonly attributed to the long-context reasoning capabilities of LLMs, yet this assumption has never been rigorously validated. A substantial gap may exist between the nominal context window (e.g., 128K) and the range over which reliable reasoning is actually achievable.

Key Challenge: Whether the success of agentic frameworks derives from "long-context reasoning" or from "task decomposition that reduces the problem to short contexts" remains unresolved.

Goal: Through controlled experiments, this work disentangles the contributions of agentic decomposition and long-context reasoning, quantifying the true capability of LLMs in long-context code repair.

Key Insight: Comparing the performance of the same model under agentic mode (incremental exploration) versus 64K single-pass mode (complete context provided at once).

Core Idea: The actual long-context reasoning capability of current LLMs falls far short of what their nominal context lengths would suggest.

Method

Overall Architecture

A two-stage experimental design: ① Agentic evaluation — mini-SWE-agent (bash-only command-line workflow) is evaluated on SWE-bench Verified, with token consumption distributions analyzed across successful and failed trajectories; ② 64K single-pass patch generation — BM25 retrieval combined with gold patch file injection constructs a 64K-token context, requiring the model to generate a patch in a single pass.

Key Designs

  1. Token Consumption Distribution Analysis:

    • Function: Measures the token distribution of successful and failed agentic trajectories.
    • Core Finding: Successful trajectories typically consume only 20K–30K tokens, far below the nominal context window; failed samples consume more tokens, becoming "lost" in the context.
    • Design Motivation: If agentic success relied on long-context reasoning, successful trajectories should consume more tokens — the opposite is observed.

  2. 64K Single-Pass Pipeline Design:

    • Function: Constructs a complete context containing sufficient information to test the model's single-pass reasoning capability.
    • Core Design: BM25 retrieval of code blocks combined with injection of files involved in the gold patch, ensuring 100% recall. The input is a 64K-token full context with edit instructions; the output is a unified diff patch.
    • Design Motivation: Eliminates the decomposition contribution of the agentic framework, directly testing whether a model can "reason to the answer given all the information."

  3. Failure Mode Classification:

    • Hallucinated diff: Chunk header line numbers far exceed the actual file length.
    • Incorrect file references: Patch targets point to non-existent file paths.
    • Format errors: Unparseable diff headers.
    • These failures indicate that models lose basic understanding of code structure in long contexts.

Key Experimental Results

Main Results — Agentic vs. 64K Single-Pass

Model Agentic Resolve 64K Single-Pass Resolve
GPT-5-nano 31% 0%
DeepSeek-R1-0528 30.3% N/A
Qwen3-32B 15.2% N/A
Qwen3-Coder-30B-A3B N/A 7%

Agentic 31% vs. 64K 0% — the same model, an enormous gap.

Token Distribution Analysis

Category Avg. Token Consumption Characteristics
Agentic Success ~20–30K Efficient, focused
Agentic Failure >30K Scattered, divergent

Key Findings

  • Agentic success ≠ long-context capability: Successful trajectories consume far fewer tokens than the context window limit.
  • GPT-5-nano completely fails under 64K single-pass mode (0%) yet achieves 31% in agentic mode — indicating that task decomposition is the operative factor.
  • Qwen3-Coder achieves only 7% at 64K — even specialized code models cannot effectively leverage long contexts.
  • Failure modes are predominantly "hallucinations": models lose basic understanding of code structure in long contexts.
  • Nominal context lengths (128K+) represent "on-paper capability"; the range of reliable reasoning in practice may be only 20–30K tokens.

Highlights & Insights

  • The core insight is striking: the success of agentic approaches has been misattributed to "long-context reasoning," when it actually stems from "task decomposition reducing the problem to short contexts" — a finding that should recalibrate the understanding of the broader LLM agent community.
  • Elegant experimental design: BM25 combined with gold file injection ensures 100% recall, eliminating the confound of insufficient information and directly testing reasoning capacity.
  • Failure mode analysis is valuable: patterns such as hallucinated diffs demonstrate that models in long contexts do not merely "fail to locate information" but "lose basic reasoning ability."
  • Implication: The core value of agent frameworks lies in "keeping each step's context within a reliable range," not in enabling models to process long contexts directly.

Limitations & Future Work

  • Only 100 SWE-bench Verified samples are used, limiting statistical power.
  • The 64K experiment covers only GPT-5-nano and Qwen3-Coder, leaving many models untested.
  • No distinction is drawn between whether long-context failures stem from "confusion due to information overload" or "harder problems that inherently require more context."
  • mini-SWE-agent is a simplified framework; full-featured SWE-agent may exhibit different token distributions.
  • Longer contexts (e.g., 256K, 1M) are not tested.
  • vs. SWE-agent: The success of SWE-agent should not be interpreted as "LLMs can handle long-context code," but rather as "the agent framework effectively decomposes the problem."
  • vs. Needle-in-Haystack: NIAH tests retrieval; this paper tests reasoning — the gap between the two indicates that "can find" ≠ "can reason."
  • vs. RAG: RAG is itself a strategy for avoiding long-context reasoning, consistent with the findings of this paper.
  • Implication for agent design: Efforts should focus on optimizing task decomposition strategies rather than pursuing ever-longer context windows.

Supplementary Technical Details

Why Does 64K Fail?

In a 64K-token code context, the model must simultaneously understand code structure, localize the bug, and generate a correctly formatted diff. In agentic mode, these three steps are completed incrementally (each step handling only a few thousand tokens), whereas in single-pass mode all reasoning must occur within a single forward pass. The effective receptive field of the model's attention mechanism over long sequences is far smaller than the theoretical window size.

Design of mini-SWE-agent

mini-SWE-agent uses a linear history — after each bash command is executed, the output is appended to the message stream without compression or summarization. This makes token consumption analysis more accurate, but also means the history may contain substantial irrelevant information.

Rating

  • Novelty: ⭐⭐⭐⭐ Core insight is valuable; experimental design is clever.
  • Experimental Thoroughness: ⭐⭐⭐ Small sample size; limited model coverage.
  • Writing Quality: ⭐⭐⭐⭐ Arguments are clear; data presentation is intuitive.
  • Value: ⭐⭐⭐⭐⭐ Makes an important corrective contribution to the understanding of LLM long-context capabilities.