Skip to content

SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidance

Conference: NeurIPS 2026
arXiv: 2609.30192
Code: https://github.com/Susan571/SAGE-NeurIPS2026
Area: LLM Reasoning
Keywords: long-horizon reasoning, reinforcement learning, topological guidance, hyperbolic geometry, algebraic sparsification

TL;DR

Addressing exploration bias and compounding bias in sparse-reward long-horizon reasoning, SAGE incorporates algebraic sparsification and hyperbolic structural guidance during post-training, delivering up to an 8-fold improvement on complex symbolic reasoning with zero inference overhead.

Background & Motivation

Post-training optimization via reinforcement learning has emerged as a cornerstone for enhancing large language model reasoning, driving remarkable breakthroughs in mathematical problem-solving and code generation through outcome-supervised objectives. However, as reasoning horizons stretch across dozens of sequential steps and feedback is restricted to terminal outcome correctness, outcome-based post-training exhibits severe brittleness. Without intermediate credit assignment, the optimization trajectory struggles to maintain long-term feasibility, frequently leading the model to hallucinate locally coherent and syntactically valid reasoning steps that ultimately derail the global solution.

This vulnerability stems from two coupled biases inherent to complex reasoning spaces. The first is exploration bias, induced by the exponential growth of structural complexity: as search trees deepen, valid solution paths inhabit an exponentially vanishing fraction of reachable trajectories, while vast volumes of locally plausible but globally dead-end branches overwhelm the rollout space. Consequently, outcome-based objectives bias sampling toward branches that are simply easy to sample under reference priors rather than paths capable of successful extension. The second is compounding bias, triggered by sparse-reward regimes: early subtle deviations receive no immediate penalty and progressively accumulate across steps, dragging the policy irreversibly away from feasibility. Under standard KL regularization, weak terminal rewards further degrade the policy gradient into an anchor that mimics reference behavior, perpetuating local drift. Existing remedies rely heavily on expensive step-wise process reward models (PRMs) or heuristic syntactic/entropy rewards, neither of which directly characterizes the intrinsic geometry of feasible reasoning manifolds.

To fundamentally diagnose and address these failure modes under structural complexity and sparse feedback, the authors introduce Symbolic Closure Analysis (SCA) as a theoretical lens, establishing prefix-closure properties and variance decompositions of feasible reasoning manifolds. Core idea: by projecting locally admissible candidates onto operator-indexed algebraic subspaces to curb spurious branching, and mapping reasoning prefixes into negative-curvature hyperbolic space for continuous depth-aware topological guidance, SAGE alleviates both exploration and compounding biases during RL post-training without requiring step-level annotations or imposing any inference-time overhead.

Method

Overall Architecture

SAGE (Structural Admissibility-Guided Exploration) internalizes structural priors of the reasoning space into policy parameters during post-training reinforcement learning. The training pipeline consists of three interconnected phases: first, extracting reasoning prefixes and predicting unresolved symbolic/semantic residuals using frozen representations; second, computing algebraic sparsification scores to evaluate operator compatibility alongside hyperbolic geodesic distances to measure progress toward target anchors in the Poincaré ball; third, combining these dual structural potentials into guided candidate sampling and advantage estimation, steering policy updates toward globally viable trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Problem Input / Current Reasoning State s_t"] --> B["Candidate Step Generation & Residual Estimation"]
    B --> C["Algebraic Sparsification: Projection to Subspace"]
    B --> D["Hyperbolic Structural Guidance: Poincaré Geodesic"]
    C --> E["Structural Prior Fusion & Guided Rollout Sampling"]
    D --> E
    E --> F["Topology-Augmented Advantage Estimation & Policy Update"]
    F --> G["Inference: Standard Direct Autoregressive Policy"]

Key Designs

1. Algebraic Sparsification: Suppressing Spurious Branching on Residual Subspaces To counter exploration bias—where countless locally admissible operators branch into unproductive regions—this design leverages subspace projection energy to evaluate whether a candidate operator actually resolves active task constraints. Given a frozen prefix representation, a regularized linear probe estimates the unresolved residual vector \(r_t \in \mathbb{R}^d\). For a candidate operator \(L_j\) associated with principal subspace \(S_j\) and projection matrix \(P_{S_j} = U_j U_j^\top\), the algebraic sparsification potential is computed as: $\(\Psi_{\mathcal{P}}(r_t, S_j) = \frac{\|P_{S_j} r_t\|_2^2}{\|r_t\|_2^2 + \varepsilon}\)$ This potential quantifies the proportion of residual energy explained by the candidate operator. Candidates that fail to tackle the core unresolved constraints exhibit near-zero projection energy, effectively dampening spurious exploratory rollouts without requiring rigid discrete pruning during inference.

2. Hyperbolic Structural Guidance: Providing Continuous Depth-Wise Topological Potentials To mitigate compounding bias caused by sparse outcome rewards, the model requires intermediate structural signals indicating whether an evolving reasoning trajectory approaches the target solution manifold. Since reasoning trees exhibit hierarchical, tree-like topologies that distort heavily when embedded in Euclidean space, SAGE maps states into a negatively curved Poincaré ball \(\mathbb{D}_c\). Intermediate state representations are mapped via fixed linear projections and radial scaling into hyperbolic embeddings \(\mathcal{E}(s_t \circ a_t)\), and evaluated against target anchor \(\mathcal{E}(g)\) using Poincaré distance \(d_{\mathbb{D}_c}\): $\(\Psi_{\mathcal{H}}(s_t, a_t, g) = \exp\left(-\frac{d_{\mathbb{D}_c}(\mathcal{E}(s_t \circ a_t), \mathcal{E}(g))}{\kappa}\right)\)$ This geometric potential provides a smooth, dense signal that penalizes early deviation along the tree depth. Even in rollout batches where terminal rewards are entirely absent (\(R_{\mathrm{term}}=0\)), non-vanishing structural variance supplies informative gradient directions, preventing compounding drift into unrecoverable states.

3. Structure-Guided Sampling and Advantage Estimation: Joint Dual-Potential Policy Optimization Algebraic sparsification and hyperbolic guidance are combined into a joint structural potential \(\Psi_{\mathrm{SAGE}}(s_t, a_t) = \alpha \Psi_{\mathcal{P}}(s_t, a_t) + \gamma \Psi_{\mathcal{H}}(s_t, a_t)\). During rollout generation, SAGE modulates the proposal distribution over candidate steps \(\mathcal{C}_t\) with length-normalized log-probabilities: $\(P_{\mathrm{sample}}(a_t^{(k)} \mid s_t, \mathcal{C}_t) \propto \exp\left(\bar{\ell}_{\theta_{\mathrm{old}}}(a_t^{(k)} \mid s_t) + \lambda \Psi_{\mathrm{SAGE}}(s_t, a_t^{(k)})\right)\)$ Trajectory-level rewards are augmented with the average structural potential: \(\widetilde{R}(\tau_i) = R_{\mathrm{term}}(\tau_i) + \eta \cdot \frac{1}{T_i} \sum_{t=1}^{T_i} \Psi_{\mathrm{SAGE}}(s_t^i, a_t^i)\), from which group-relative advantages \(A_i\) are computed. This formulation guarantees a stable, non-vanishing policy update signal even under severe reward sparsity.

Loss & Training

SAGE is optimized using a step-level KL-regularized group-relative policy optimization objective: $\(\mathcal{L}_{\mathrm{SAGE}}(\theta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{T_i} \sum_{t=1}^{T_i} \left[ \min\left(\rho_{i,t}(\theta) A_i, \text{clip}(\rho_{i,t}(\theta), 1-\epsilon, 1+\epsilon) A_i\right) - \delta \log \frac{\pi_\theta(a_{i,t} \mid s_{i,t})}{\pi_{\mathrm{ref}}(a_{i,t} \mid s_{i,t})} \right]\)$ where \(\rho_{i,t}(\theta) = \pi_\theta(a_{i,t} \mid s_{i,t}) / \pi_{\theta_{\mathrm{old}}}(a_{i,t} \mid s_{i,t})\). Training utilizes AdamW on a fixed entropy-filtered prompt subset. Crucially, all structural probes, Poincaré projections, and potential evaluations are restricted to the training phase; inference uses the standard autoregressive policy \(\pi_\theta\) directly, incurring zero runtime computational or latency penalties.

Key Experimental Results

Main Results

SAGE was thoroughly evaluated across 7 model families and 12 benchmarks spanning closed-form mathematics (MATH, Minerva, OlympiadBench, AIME24, AMC23, GSM8K, Putnam), free-form natural reasoning (MMLU-Pro, GPQA, BBH-H, ARC-C), and open-ended long-horizon symbolic deduction (Andrews-Curtis problem).

The table below presents accuracy (%) on mathematical reasoning benchmarks across various model scales:

Model & Method MATH OlympiadBench AIME24 AMC23 GSM8K Putnam Average
Llama-3.3-70B-Instruct (Flagship) 66.08 32.94 17.31 29.02 80.47 9.91 38.48
Qwen3.5-2B (Base) 50.64 24.58 9.41 43.36 47.38 3.66 27.24
Qwen3.5-2B w/ SFT 60.41 27.96 3.74 37.88 55.06 4.63 30.89
Qwen3.5-2B w/ GRPO 73.39 33.86 16.05 50.94 63.12 6.79 39.63
Qwen3.5-2B w/ EMPO 71.94 37.04 12.88 54.71 66.54 7.48 40.28
Qwen3.5-2B w/ SAGE (Ours) 74.61 38.25 15.72 55.81 67.06 9.31 42.11
Qwen3.5-9B (Base) 65.11 27.31 6.38 39.73 45.59 4.61 29.01
Qwen3.5-9B w/ GRPO 75.96 38.82 19.51 56.96 62.77 7.66 43.16
Qwen3.5-9B w/ EMPO 78.24 37.03 20.88 64.88 65.55 8.01 44.84
Qwen3.5-9B w/ SAGE (Ours) 79.97 41.59 25.58 63.91 71.72 11.47 47.95
Qwen3.5-35B (Base) 70.28 48.81 35.05 42.33 76.28 11.36 45.21
Qwen3.5-35B w/ GRPO 82.18 65.31 57.94 68.92 91.51 19.48 61.92
Qwen3.5-35B w/ EMPO 81.84 64.41 62.31 67.38 92.29 19.97 62.37
Qwen3.5-35B w/ SAGE (Ours) 84.72 68.26 62.04 70.87 94.16 22.62 64.86

On the open long-horizon Andrews-Curtis task, SAGE achieved an 8-fold increase in Lean-verified proof rate over base models.

Ablation Study

To verify the distinct contributions of algebraic sparsification \(\Psi_{\mathcal{P}}\) and hyperbolic structural guidance \(\Psi_{\mathcal{H}}\), ablations were conducted on Qwen3.5-9B across OlympiadBench and BBH-H:

Variant Olympiad Acc (%) Olympiad Rew./1k BBH-H Acc (%) BBH-H Rew./1k
GRPO Baseline 38.82 16.08 41.69 20.06
EMPO Baseline 37.03 15.03 44.04 19.02
SAGE w/o \(\Psi_{\mathcal{H}}\) (No Hyperbolic) 39.84 18.47 39.06 22.36
SAGE w/o \(\Psi_{\mathcal{P}}\) (No Sparsification) 39.12 17.68 38.85 21.18
SAGE w/ Euclidean Distance 38.01 16.92 38.02 20.89
SAGE w/ Shuffled Target \(g\) 37.99 17.01 37.83 19.91
Full SAGE 41.59 21.38 45.31 25.76

In ablation on the Andrews-Curtis task with Qwen3.5-35B, removing \(\Psi_{\mathcal{H}}\) dropped Lean-verified proof success from 23.69% to 18.82%, while removing \(\Psi_{\mathcal{P}}\) lowered it to 18.07%, validating that both geometric signals are critical for long-horizon feasibility.

Key Findings

  • Complementarity of Dual Guidance: Removing either \(\Psi_{\mathcal{P}}\) or \(\Psi_{\mathcal{H}}\) causes a sharp performance drop, confirming that pruning spurious branch exploration and curbing depth-wise error accumulation are both essential.
  • Superiority of Hyperbolic Geometry: Replacing the Poincaré geodesic metric with Euclidean distance degrades performance by over 3.5%, proving that hyperbolic negative curvature uniquely matches the tree-structured hierarchy of reasoning.
  • Cross-Scale Parameter Efficiency: Qwen3.5-35B w/ SAGE reaches 64.86% average accuracy across math competitions, significantly outperforming the 70B parameter flagship Llama-3.3-70B-Instruct (38.48%).

Highlights & Insights

  • Training-Only Guidance, Zero-Cost Inference: Structural potentials modulate rollout distributions and advantage functions strictly during post-training, leaving the final policy weights purely autoregressive without test-time inference overhead.
  • From Theoretical Diagnosis to Executable Priors: The SCA framework pinpoints reference-anchoring failures under sparse rewards and translates the diagnosis into computable subspace projection and hyperbolic distance metrics.
  • Breakthrough on Formal Theorem Proving: Achieving up to an 8-fold gain on Lean-verified Andrews-Curtis problems demonstrates a viable new paradigm for tackling open mathematical conjectures using LLMs.

Limitations & Future Work

  • Reliance on Pseudo-Label Heuristics in Natural Reasoning: In informal tasks, residual definitions and coarse operator clustering rely on empirical approximations rather than strict symbolic group axioms.
  • Static Target Anchors: Target embeddings remain fixed during reasoning, which may limit performance on tasks with dynamic or branching multi-goal specifications.
  • Future Directions: Exploring learnable dynamic curvature parameters to adapt to varying problem complexities, and extending SAGE to multi-agent planning, code synthesis, and formal software verification.
  • vs. Process Reward Models (GRPO-PRM / Lightman et al.): PRMs require costly step-level manual labels or fragile LLM judges; SAGE derives continuous step-wise guidance from intrinsic geometric structures without intermediate annotations.
  • vs. Intrinsic Exploration Heuristics (EMPO / Count-based RL): Prior exploratory methods rely on token entropy or visit counts without modeling reasoning feasibility manifolds; SAGE directs exploration specifically toward algebraically admissible and hierarchically convergent paths.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant integration of algebraic subspace projection and Poincaré hyperbolic geometry to resolve fundamental long-horizon reasoning biases.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across 7 model families, 12 formal and informal benchmarks, rigorous ablation studies, and Lean-compiler verification.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, rigorous theoretical proofs in SCA, and a natural bridge between theoretical insight and algorithm design.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical post-training methodology for long-horizon reasoning and mathematical discovery with zero inference overhead.