Skip to content

A^2Agent: Action-Aware Reinforcement Learning for Repository-Level Code Localization Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/donian00/A2Agent
Area: LLM Reasoning / Code Agents
Keywords: Code Localization, Action-Aware Reinforcement Learning, Exploration History Grouping, Discovery-Commit Ratio, Software Engineering Agents

TL;DR

Addressing the issues where trajectory-level sparse rewards cannot attribute per-turn action credit and agents frequently explore gold code regions without committing them, A2Agent couples a dense per-turn discovery sequence with a terminal Discovery-Commit reward and estimates action-level advantages via exploration history grouping, allowing a 4B model to outperform baselines up to 8× larger across SWE-Bench Verified (+1.58% F1) and SWE-Bench Pro (+8.55% F1).

Background & Motivation

Repository-level code localization is an indispensable prerequisite in automated software engineering, aiming to pinpoint the exact files, classes/modules, and functions requiring modification to resolve a real GitHub issue. Because contemporary industrial repositories are massive and inter-file dependency structures are intricate, single-turn LLM inference cannot encompass the entire repository context. Consequently, agentic paradigms that iteratively invoke specialized tools (e.g., code snippet search, symbol tracing, and dependency graph traversal) have become the established standard. However, existing post-training methodologies primarily rely on supervised fine-tuning (SFT) or reinforcement learning with verifiable rewards (RLVR) based on terminal trajectory-level outcomes, leaving individual intermediate decisions unguided.

This reliance on sparse trajectory-level supervision incurs two fundamental structural bottlenecks. First is the credit assignment failure in multi-turn exploration: during sequential navigation, an agent performs both effective targeted inspections and unhelpful or misleading tool invocations. Coarse group-relative methods like GRPO or GSPO assign a single uniform scalar advantage to the entire trajectory, failing to reinforce effective exploration actions or discourage counterproductive detours. Second is the pervasive Discovery-Commit Gap: agents often successfully encounter the gold code regions during exploration but subsequently fail to recognize or commit them to the final prediction, choosing instead to invoke extra tools and submit erroneous locations. Under standard terminal evaluation, these explored-but-uncommitted trajectories are penalized identically to trajectories that never reached the correct elements, discarding vital signals of partial success.

To address these limitations, this paper proposes A2Agent, an action-aware reinforcement learning framework for repository-level code localization. The core insight is that since code repositories remain static and deterministic during localization, an agent's accumulated interaction history serves as an effective state proxy; grouping rollouts that share identical exploration contexts establishes localized counterfactual baselines without requiring a learned value function. Core idea: build a dense reward sequence combining per-turn discovery with terminal Discovery-Commit alignment, and perform action-level advantage estimation via exploration-history grouping, enabling granular action credit assignment and high-fidelity commitment on fixed software repositories.

Method

Overall Architecture

A2Agent models repository-level code localization as a multi-turn decision-making process over an action space of six deterministic repository tools. At each turn, given an issue and past interaction history, the policy outputs a tool call and receives code slices or structural graphs until triggering a completion action to return localized files, modules, and functions. The training pipeline consists of two stages: first, a Teacher SFT Warm-up fine-tunes the base model on successful trajectories filtered from a strong teacher to instill tool syntax and navigation basics; second, Action-Aware Reinforcement Learning samples multiple rollouts per issue, computes per-turn discounted returns, groups turns sharing historical action contexts to estimate action-level advantages, and optimizes the policy via an advantage-weighted regression objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Issue q & Repository Codebase"] --> B["Teacher SFT Warm-up<br/>Distill tool proficiency into pi_sft"]
    B --> C["Rollout Collection<br/>Sample N=8 trajectories per issue"]
    C --> D["Two-Level Reward Computation<br/>Per-turn discovery delta_t + Terminal F1 & DC reward"]
    D --> E["State Proxy & Action-History Grouping<br/>Cluster turns sharing preceding K-action sequence"]
    E --> F["Action-Level Advantage Estimation<br/>Context baseline normalization + Multi-depth anchor"]
    F --> G["Advantage-Weighted Policy Optimization<br/>Clipped exponential token-level ML objective"]
    G -->|Iterative policy updates over rollouts| C
    G --> H["Final Output Prediction y_hat<br/>Files F_hat / Modules M_hat / Functions U_hat"]

Key Designs

1. Two-Level Reward Sequence: Resolving Uncommitted Discovery and Sparse Signal Bottlenecks

To eliminate gradient starvation on partially correct rollouts and incentivize agents to retain observed targets, A2Agent constructs a per-turn discounted return sequence \(G_t = \sum_{k=t}^T \gamma^{k-t} r_k\) spanning intermediate turns and the final decision. At the terminal turn \(T\), the reward blends multi-granularity \(F_1\) accuracy with a novel Discovery-Commit (DC) reward \(r_{\text{DC}}\): $\(r_T = r_{F1} + r_{\text{DC}} = (F1_F + F1_M + F1_U) + \frac{1}{|I|}\sum_{g\in I}\frac{|E_g \cap Y_g \cap O_g|}{|E_g \cap Y_g|}\)$ where \(E_g\), \(Y_g\), and \(O_g\) denote the sets of explored, gold, and predicted locations at granularity \(g \in \{F, M, U\}\), and \(I\) indexes granularities where at least one gold location was explored. This penalizes the failure mode where the agent navigates across the gold entity but omits it from the final answer. For intermediate turns \(t < T\), the environment immediately dispenses a discovery reward whenever an action uncovers a new gold location: \(r_t = \delta_t^f + \delta_t^m + \delta_t^u\). This preserves credit for critical early breakthroughs and provides non-zero learning signals for trajectories that fail at the final turn.

2. State Proxy and Action-History Grouping: Value-Free Action-Level Counterfactual Baselines

In repository localization, the codebase is static and read-only, meaning an agent's internal state is fully defined by its accumulated interaction history. Because 79.2% of turns follow distinct preceding actions, treating an action in isolation without context introduces massive bias. A2Agent defines the state proxy as the sequence of \(K\) preceding actions \(\phi_K(t) = \langle a_{t-K}, \dots, a_{t-1} \rangle\). Within rollouts sampled for the same issue, turns sharing the identical \(\phi_K(t)\) are collected into a context group \(\mathcal{G}_t^{(K)}\). Because repository tools are deterministic, matching action sequences yield identical observation inputs. Excluding the current action \(a_t\) from the grouping key creates an empirical counterfactual baseline representing the expected outcome from that state, allowing the action-level advantage to isolate the marginal contribution of choosing \(a_t\): $\(\hat{A}_t^{(K)} = \frac{G_t - \text{mean}_{j \in \mathcal{G}_t^{(K)}} G_{tj}}{\sigma_{\mathcal{G}_t^{(K)}} + \epsilon}\)$

3. Multi-Depth Context Combination and Trajectory Anchoring: Robust Variance Control

Relying on a single history depth \(K\) faces a strict bias-variance trade-off: large \(K\) provides exact contextual matching but causes group sizes to shrink drastically, whereas small \(K\) pools disparate turns and injects noise. To stabilize estimation, A2Agent combines advantages across depths \(K=1, \dots, H\), setting \(\hat{A}_t^{(K)}=0\) whenever a group contains only one turn. To guarantee a valid baseline for turns whose history cannot be matched at any depth, the framework incorporates the trajectory-level group-relative advantage \(A(\tau_i)\) as an anchor: $\(A_t = \omega_0 A(\tau_i) + \sum_{K=1}^H \omega_K \hat{A}_t^{(K)}\)$ with weights constrained to \(\sum_{K=0}^H \omega_K = 1\) and scaled proportionally to \(K+1\). Setting \(H=2\) ensures that whenever sufficient rollouts share common exploration prefixes, fine-grained action advantages guide the update, while gracefully defaulting to safe trajectory-level signals when exploration diverges.

Loss & Training

Rather than applying standard PPO or GRPO policy gradients which can exhibit training instability over long multi-turn token sequences, A2Agent adopts an Advantage-Weighted Regression (AWR) framework. The composite advantage \(A_t\) is transformed into a positive token weight \(w_t\) through double clipping: $\(\tilde{A}_t = \text{clip}(A_t, -\kappa, \kappa), \quad w_t = \text{clip}\left(\exp(\beta \tilde{A}_t), w_{\min}, w_{\max}\right)\)$ The policy parameters \(\theta\) are updated on the set of assistant-generated tokens \(\mathcal{V}(\tau)\) by minimizing the weighted negative log-likelihood: $\(\mathcal{L}(\theta) = \mathbb{E}_{\tau \sim \mathcal{D}} \left[ \frac{1}{|\mathcal{V}(\tau)|} \sum_{i \in \mathcal{V}(\tau)} w_{t_i} \cdot \left(-\log \pi_\theta(y_i \mid y_{<i}, \tau_{<t_i})\right) \right]\)$ In practice, the policy is initialized with LoRA (rank 16) on 1,410 teacher-filtered trajectories using Qwen3-30B-A3B-Instruct. The RL phase executes 2 epochs with batch size 16 across 3 rollout-training iterations (\(N=8\) rollouts per issue, max 20 tool steps, context window 40K, discount factor \(\gamma=0.9\), temperature \(\beta=1.5\), clip bounds \(\kappa=5.0\), \(w_{\min}=0.1, w_{\max}=3.0\), and AdamW with learning rate \(5\times 10^{-5}\)).

Key Experimental Results

Main Results

Evaluation is performed on SWE-Bench Verified (500 human-verified instances) and SWE-Bench Pro (266 long-horizon instances from 41 repositories) across file-, module-, and function-level F1, Precision, and Recall.

Benchmark / Metric Method Backbone LLM File F1 (%) Module F1 (%) Function F1 (%)
SWE-Bench Verified RepoSearcher Qwen2.5-32B 32.25 – 28.03
SWE-Bench Verified RepoNavigator Qwen2.5-32B 67.75 – 34.09
SWE-Bench Verified CodeScout-14B Dedicated Fine-tuning 68.57 50.88 40.32
SWE-Bench Verified A2Agent (Ours) Qwen3-4B-Instruct 72.64 46.94 38.59
SWE-Bench Verified A2Agent (Ours) Qwen3-8B 73.31 47.98 41.01
SWE-Bench Pro RepoSearcher Qwen2.5-32B 3.81 – 2.31
SWE-Bench Pro RepoNavigator Qwen2.5-32B 57.57 – 20.72
SWE-Bench Pro CodeScout-14B Dedicated Fine-tuning 53.63 37.13 28.74
SWE-Bench Pro A2Agent (Ours) Qwen3-4B-Instruct 58.19 38.38 29.23
SWE-Bench Pro A2Agent (Ours) Qwen3-8B 60.05 39.07 30.60

In downstream issue resolution experiments using the Agentless repair pipeline (with Qwen3-30B-A3B-Instruct generating patches), localization predictions from RepoSearcher, CoSIL, Agentless, LocAgent, and A2Agent achieved resolved rates of 26.82%, 27.64%, 29.07%, 31.78%, and 38.07% respectively, closing in toward the Oracle localization upper bound of 45.93%.

Ablation Study

Ablation experiments conducted on SWE-Bench Verified under the Qwen3-8B backbone isolate the contributions of History Grouping (\(H=2\)) and the Discovery-Commit reward (\(r_{\text{DC}}\)):

Model Configuration History Grouping \(r_{\text{DC}}\) Reward File F1 (%) Module F1 (%) Function F1 (%)
A2Agent (Full) ✓ ✓ 73.31 47.98 41.01
w/o History Grouping ✗ ✓ 71.73 (-1.58) 46.42 (-1.56) 39.55 (-1.46)
w/o \(r_{\text{DC}}\) Reward ✓ ✗ 72.19 (-1.12) 47.10 (-0.88) 40.19 (-0.82)
w/o Both Components ✗ ✗ 71.01 (-2.30) 45.63 (-2.35) 38.72 (-2.29)
Pure Teacher SFT – – 66.20 (-7.11) 41.15 (-6.83) 35.41 (-5.60)
Standard GRPO Baseline – – 67.47 (-5.84) 42.33 (-5.65) 34.28 (-6.73)

Key Findings

  • Complementary Core Components: Removing History Grouping degrades file-level F1 by 1.58 percentage points, while removing \(r_{\text{DC}}\) causes a 1.12 pp decline. Eliminating both causes a 2.30 pp drop. Even when stripped of both mechanisms, the per-turn discovery reward alone exceeds trajectory-level GRPO by 3.54 pp, demonstrating that dense turn feedback fundamentally outperforms sparse terminal outcomes.
  • Shrinking the Discovery-Commit Gap: Across baselines (LocAgent, RepoSearcher, CoSIL), uncommitted discovery accounts for more than 40% of all failure cases. Incorporating \(r_{\text{DC}}\) dramatically curbs this failure rate while simultaneously elevating precision, proving that the model learns to commit its findings rather than blindly trading precision for recall.
  • Optimal History Depth at \(H=2\): Analyzing depths \(H \in \{1, 2, 3, 4\}\) shows that while larger depths match turns under finer context, group coverage plunges below 20% beyond \(H=2\), introducing estimation noise that degrades F1. \(H=2\) strikes the ideal balance between context specificity and sample support.
  • Inference Acceleration via Concise Observations: On SWE-Bench Verified, A2Agent completes inference in an average of 18.08 seconds per instance (compared to 86.23s for LocAgent and 77.57s for CoSIL) despite invoking a similar number of tool calls (11.72 vs. 6.20–17.55). This efficiency gain stems from learning focused searches over redundant queries, trimming average observation tokens by ~40% and drastically reducing multi-turn KV prefill overhead.

Highlights & Insights

  • Formulating action history as a state proxy: By recognizing that code repositories are static and read-only, the authors convert what would normally require an expensive state-branching simulator into an offline grouping operation over existing rollout batches.
  • Addressing the Discovery-Commit failure mode: Identifying and mathematically targeting the gap between encountering relevant code and formally submitting it transforms a common, neglected agent pathology into an actionable reinforcement objective.
  • Superior parameter efficiency: Showing that a 4B parameter model optimized with fine-grained credit assignment outperforms 14B–32B models provides compelling evidence that credit assignment quality matters more than raw parameter scale in agentic code reasoning.

Limitations & Future Work

  • Reliance on pre-indexed read-only code environments: The method assumes a fixed repository snapshot and requires indexing dependency graphs and BM25 tables (taking 1–2 hours per repository), making it unsuitable as-is for dynamic execution environments involving runtime code editing or interactive test execution.
  • Group sparsity in divergent rollout regimes: In highly complex tasks where tool call trajectories diverge early, matching identical length-2 action histories across a small batch (\(N=8\)) can fail, causing the estimator to fall back to trajectory-level signals.
  • Disconnection from downstream patch execution: The localization reward optimizes symbol overlap rather than downstream unit test verification; joint reinforcement learning coupling localization policies with code repair engines remains an open direction.
  • vs RepoNavigator / CodeScout: These systems rely on group-relative trajectory reinforcement learning (GRPO / GSPO) with uniform advantages across all turns. A2Agent demonstrates that decomposing rewards into per-turn discovery and history-grouped baselines provides much cleaner credit attribution.
  • vs LocAgent / CoSIL / RepoSearcher: Prior works focus heavily on tool expansion (e.g., custom call graphs or terminal commands). A2Agent reuses the LocAgent tool suite verbatim, proving that algorithmic advances in RL credit assignment yield larger gains than expanding the tool set.
  • vs Step-level RL (VinePPO / Step-level Monte Carlo): Traditional step-level credit assignment requires re-executing rollouts from intermediate checkpoints, which is computationally prohibitive in multi-turn LLMs. A2Agent achieves comparable fine-grained attribution by exploiting the natural deterministic overlap across independent rollouts.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Leveraging action histories as state proxies for value-free advantage estimation in static environments is elegant and practical.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage spanning SWE-Bench Verified, SWE-Bench Pro, ablation depths, token/time efficiency, and downstream repair pipelines.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous task formulation, crystal-clear motivation, and detailed empirical analyses.
  • Value: ⭐⭐⭐⭐⭐ Offers a high-impact, parameter-efficient training recipe for complex agentic reasoning tasks.