SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidance¶

Conference: NeurIPS 2026
arXiv: 2609.30192
Code: https://github.com/Susan571/SAGE-NeurIPS2026
Area: LLM Reasoning
Keywords: long-horizon reasoning, structural priors, algebraic sparsification, hyperbolic geometry, group-relative policy optimization

TL;DR¶

SAGE uses two soft potentials—whether an operation addresses unresolved constraints and whether its resulting state approaches a target structure—to guide both candidate sampling and rewards during post-training, improving long-horizon reasoning while retaining ordinary inference-time decoding; Qwen3.5-35B mathematical average accuracy rises from 61.92% with GRPO to 64.86%.

Background & Motivation¶

Long-horizon reasoning is difficult not only because individual steps can be inaccurate, but also because each step admits many plausible continuations while successful paths are scarce. Reinforcement learning that rewards only the final answer must first sample an entire successful trajectory before increasing its probability. As reasoning deepens, expanding branches consume the sampling budget on locally valid operations that no longer help solve the problem. The paper calls this exploration bias. Andrews–Curtis (AC) trivialization of group presentations provides a concrete example: inversion, relator multiplication, and conjugation may all be legal, but legality does not imply sustained simplification of the presentation.

A second failure develops within an unfolding trajectory: small early deviations receive no timely feedback, and later steps continue along them until failure emerges at the endpoint. This is compounding bias. Process reward models and formal verifiers can supply intermediate feedback, but the former depend on the quality of constructed process signals and the latter on domain coverage. Meanwhile, when terminal rewards are extremely rare, KL regularization continuously pulls the policy toward the reference model. Optimization may primarily learn not to change rather than how to complete a long chain. Theorem 3.5 characterizes this reference anchoring under bounded terminal rewards and fixed KL strength; it is not an unconditional failure theorem for all reinforcement learning systems.

The authors introduce Symbolic Closure Analysis (SCA), defining a prefix-closed trajectory set through local admissibility and translating the analysis into training signals. Two questions must remain distinct: whether every transition obeys the rules and whether the trajectory ultimately succeeds; successful trajectories form only a subset of locally admissible ones. Core Idea: use residual–operator compatibility to reduce unproductive exploration and hyperbolic target distance to provide structural feedback before the endpoint, absorbing both preferences into the policy rather than retaining structural scorers at inference time.

Method¶

Overall Architecture¶

SAGE is a post-training framework, not an inference-time search algorithm. Its inputs are the task prompt, current reasoning prefix, and candidate operations or reasoning steps proposed by the old policy; its output is an updated language-model policy. Training prepares task-specific structural priors, computes algebraic sparsification and hyperbolic structural guidance, and uses the potentials for candidate reweighting and trajectory rewards. Testing invokes only the trained model.

Symbolic tasks expose explicit states, operations, and targets; mathematical and free-form tasks instead require approximate structural representations. A frozen reference model encodes prefixes, a fixed probe predicts unresolved structural constraints, an operator classifier assigns candidates to coarse operation types, and training-rollout-derived subspaces and target anchors support scoring. These modules are neither new inference-time verifiers nor final-answer correctness classifiers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training prompts and reference rollouts"] --> B["Structural Prior Construction"]
    B --> C["Algebraic Sparsification"]
    B --> D["Hyperbolic Structural Guidance"]
    P["Old-policy candidates and prefixes"] --> C
    P --> D
    C --> E["Guided Sampling and Advantage"]
    D --> E
    R["Training terminal reward"] -.->|Reward and advantage only| E
    E --> F["Policy update"]
    F --> G["Test prompt → direct decoding"]

Structural priors, both scorers, and the guidance procedure operate during training. The dashed edge denotes terminal supervision, not test-time data flow. The scoring branches run in parallel; the designs below follow the order of structural priors, algebraic scoring, hyperbolic scoring, and joint optimization.

Key Designs¶

1. Structural Prior Construction: make unresolved requirements computable

SCA first defines local admissibility: if a prefix already contains an invalid transition, every continuation preserving that prefix remains outside the locally admissible set. Prefix closure does not mean the model can identify a successful continuation. Appendix A explicitly notes that the admissible set can contain many unsuccessful trajectories, so improved local validity cannot be equated with complete problem-solving ability.

For AC, residuals describe generator counts, relator lengths, and unresolved relator components in the current presentation; the target is the task-specified trivial presentation. Mathematical residuals represent unresolved variables, equations, operation categories, and answer-schema constraints. Free-form tasks use prompt requirements and semantic structure. Target anchors are constructed from prompts and training-rollout representations, without using test labels, gold reasoning traces, or gold answers to build these structural modules.

For non-symbolic tasks, a residual probe is fitted by regularized regression on structural pseudo-labels from training rollouts. These labels include operation type, constraint coverage, answer-schema status, and format consistency. Residuals are grouped by operator type, and principal directions define each operator subspace; probes and projections are fixed before guided training. Although these signals do not require human-written step-correctness annotations, they still depend on operator classification, pseudo-labels, and task interfaces. They are not devoid of supervision or priors. Moreover, the joint training reward retains terminal feedback: structural modules avoiding terminal-correctness labels does not mean the entire training framework avoids terminal rewards.

2. Algebraic Sparsification: favor operations that explain the current residual

When a problem remains blocked on particular constraints, generating unrelated steps can produce fluent text without progress. SAGE associates each operator type with a subspace, projects the current residual onto the candidate's operator subspace, and scores the fraction of residual energy it explains:

\[ \Psi_{P}(r_t,S_j)=\frac{\|P_{S_j}r_t\|_2^2}{\|r_t\|_2^2+\varepsilon}. \]

Here \(r_t\) represents unresolved structure, \(P_{S_j}\) projects onto the operator subspace, and \(\varepsilon\) prevents a degenerate denominator. A high score indicates stronger compatibility between the operation type and residual, not correctness of the particular step or guaranteed residual reduction. Mathematical types include simplification, substitution, numerical evaluation, equation formation, formula invocation, case splitting, constraint checking, and final-answer extraction. Free-form types include factual retrieval, comparison, elimination, aggregation, inference, format normalization, and final-answer commitment.

This sparsification primarily concentrates exploration through soft reweighting, rather than deleting every low-scoring candidate. Appendix C invokes classical Orthogonal Matching Pursuit recovery conditions to explain why operator-relevant directions may be identified. That result requires a noiseless sparse representation and dictionary coherence assumptions; it does not establish global problem-solving guarantees for language models. Subspace compatibility in non-symbolic tasks is especially a learned proxy, not a hard admissibility certificate.

3. Hyperbolic Structural Guidance: evaluate target distance before terminal feedback

Current residual compatibility alone can overlook where a trajectory is heading, so the second branch evaluates the state after a candidate step. A frozen reference model generates state features, a fixed linear projection reduces structural dimensionality, and radial mapping places the representation in the Poincaré ball. The target anchor uses the same representation pipeline. Negative curvature is motivated by the rapidly expanding hierarchy of reasoning trees, whose structure can be represented more naturally in hyperbolic than Euclidean space.

\[ \Psi_{H}(s_t,a_t,g)=\exp\!\left(-\frac{d_{\mathbb{D}_c}(E(s_t\circ a_t),E(g))}{\kappa}\right). \]

\(g\) is the target structure, \(s_t\circ a_t\) is the state after the candidate operation, \(d_{\mathbb{D}_c}\) is Poincaré distance with curvature parameter \(c\), and \(\kappa>0\) controls scoring sharpness. Candidates closer to the anchor receive higher scores. This provides feedback before the endpoint, but formally it is a target-distance potential, not explicit depth-label supervision or the improvement in distance between adjacent states.

For AC, the anchor has the explicit meaning of a trivial presentation. For natural language, whether a semantic anchor represents a successful direction requires empirical validation. Shuffled-anchor and Euclidean-distance controls examine structural alignment and geometry; they do not prove that every high-scoring prefix can succeed.

4. Guided Sampling and Advantage: change exploration and enable updates in zero-terminal-reward groups

The potentials combine with nonnegative weights as \(\Psi_{\mathrm{SAGE}}=\alpha\Psi_P+\gamma\Psi_H\). In non-symbolic tasks, an action is not one token but a reasoning segment generated by the old policy until a delimiter, sentence boundary, answer marker, end-of-sequence token, or maximum step length. At each state, a finite candidate set is sampled first, then reweighted using length-normalized old-policy log-probability and structural potentials:

\[ P_{\mathrm{sample}}(a_t^{(k)}\mid s_t,\mathcal C_t) =\frac{\exp(\bar\ell_{\theta_{\mathrm{old}}}(a_t^{(k)}\mid s_t)+\lambda\Psi_{\mathrm{SAGE}}(s_t,a_t^{(k)}))} {\sum_{a\in\mathcal C_t}\exp(\bar\ell_{\theta_{\mathrm{old}}}(a\mid s_t)+\lambda\Psi_{\mathrm{SAGE}}(s_t,a))}. \]

\(\bar\ell\) averages token log-probabilities inside a candidate, preventing a preference for shorter steps arising solely from probability products; \(\lambda\) controls sampling guidance. Normalization covers only the current candidate set \(\mathcal C_t\), not the full vocabulary or all textual continuations. An operation absent from the proposed candidates cannot be selected through reweighting, so exploration remains limited by old-policy candidate coverage.

The average structural score of each trajectory is then added to its terminal reward, followed by standardization within the rollout group:

\[ \widetilde R(\tau_i)=R_{\mathrm{term}}(\tau_i) +\eta\frac{1}{T_i}\sum_{t=1}^{T_i}\Psi_{\mathrm{SAGE}}(s_t^i,a_t^i), \qquad A_i=\frac{\widetilde R(\tau_i)-\operatorname{mean}_j\widetilde R(\tau_j)}{\operatorname{std}_j\widetilde R(\tau_j)+\varepsilon}. \]

\(\eta\) controls reward augmentation and \(T_i\) is the number of trajectory steps. Averaging rather than summing prevents longer trajectories from accumulating more structural reward merely by taking additional steps. If every terminal reward in a group is zero, variation in structural scores can still supply an update signal. If structural scores are also identical, the mechanism cannot manufacture discriminative information.

A Worked Example¶

Consider a mathematical prefix that has introduced variables but has not yet established the necessary equation constraints. The old policy might propose further paraphrasing, equation formation, or an immediate answer. The operator classifier assigns their types, while the residual probe represents unresolved equation and answer-schema requirements. Algebraic sparsification favors operation types aligned with these requirements.

Next, each candidate's resulting state is evaluated by hyperbolic distance to a prompt-conditioned target anchor. A candidate with a suitable operator type but an off-target resulting state need not receive the highest combined score. Sampling still incorporates old-policy probabilities and remains stochastic. Across complete rollouts, augmented rewards can generate updates through structural differences even when none produces a correct final answer.

This is a mechanism illustration, not a numerical step-by-step example supplied by the paper, and it does not assert that equation formation must always outperform other operations. At test time, the model does not generate a candidate batch for these probes to score; it directly continues using the updated policy.

Loss & Training¶

The paper uses a clipped group-relative update with a reference-policy KL penalty, applying the trajectory advantage to individual steps and averaging over trajectory length. Its policy ratio is \(\rho_{i,t}(\theta)=\pi_\theta(a_{i,t}\mid s_{i,t})/\pi_{\theta_{\mathrm{old}}}(a_{i,t}\mid s_{i,t})\), whose denominator is not the structurally reweighted \(P_{\mathrm{sample}}\). Since finite-candidate guidance changes the rollout distribution, and neither the main text nor Appendix E gives a complete behavior-distribution correction derivation, this ratio should not be described as a proven unbiased importance weight for guided sampling.

Controlled experiments first compute semantic entropy from the same reference policy's rollouts and retain training prompts between lower and upper entropy thresholds once. Structured answers use canonicalized matching, while free-form answers use semantic equivalence judgments. GRPO, EMPO, PRM-GRPO, and SAGE share the retained subset, budgets, and optimization schedule. Evaluation uses complete test sets without entropy filtering, structural scoring, semantic clustering, or local checkers.

Appendix E mentions AdamW where applicable, but the cached text does not provide all numerical configurations needed for direct reproduction, such as candidate count, maximum step length, and probe dimensions. Training-subset selection and probe preparation are part of the cost. No additional inference-time cost does not mean free training-time guidance.

Key Experimental Results¶

Main Results¶

The table selects results illustrating differences across scales and tasks. Mathematical average accuracy is the reported aggregate over seven tasks in the source table; other rows are individual metrics and should not be merged into one overall average. Values come from Tables 1 and 2. Gains are percentage points (pp), not relative percentages.

Model / task	GRPO (%)	EMPO (%)	SAGE (%)	Gain over the better of the two RL baselines
Qwen3.5-2B / mathematical average	39.63	40.28	42.11	+1.83 pp
Qwen3.5-9B / mathematical average	43.16	44.84	47.95	+3.11 pp
Qwen3.5-35B / mathematical average	61.92	62.37	64.86	+2.49 pp
Qwen3.5-9B / MMLU-Pro average	39.33	37.91	39.99	+0.66 pp
Qwen3.5-9B / GPQA	18.21	21.11	20.86	-0.25 pp
Qwen3.6-27B / BBH-H	55.74	57.19	60.25	+3.06 pp
Qwen3.6-27B / ARC-C	51.78	53.26	57.11	+3.85 pp
Qwen3.5-35B / Putnam	19.48	19.97	22.62	+2.65 pp

For 9B mathematics, SFT averages 44.93%, explaining the main text's +3.02 pp over its strongest baseline; the table's +3.11 pp is explicitly restricted to GRPO and EMPO. SAGE is not best on every individual metric: 2B AIME24 reaches 15.72%, below GRPO's 16.05%, and 27B GPQA reaches 32.51%, below GRPO's 34.29%.

Ablation Study¶

Table 3 fixes the training subset, rollout budget, decoding constraints, optimization steps, and KL coefficient on Qwen3.5-9B. Only clearly defined accuracy values are retained below. The original table also reports Rew./1k, but the cache does not sufficiently define reward events or its denominator, so it is not expanded as a fully specified custom metric.

Config	Olympiad accuracy (%)	BBH-H accuracy (%)	Note
Full SAGE	41.59	45.31	Both structural potentials
Without hyperbolic structural guidance	39.84	39.06	1.75 / 6.25 pp below the full model
Without algebraic sparsification	39.12	38.85	2.47 / 6.46 pp below the full model
Euclidean instead of hyperbolic distance	38.01	38.02	3.58 / 7.29 pp below the full model
Shuffled target anchors	37.99	37.83	3.60 / 7.48 pp below the full model

The AC dataset contains 1190 group presentations constructed with \(n\leq7\) and \(|w|\leq7\). AC Validity measures the proportion of syntactically and logically valid steps; Lean-Verified measures the success rate of compiler-checked proofs. Table 4's AC Path column describes path-solving performance, but the cache does not fully elaborate its independent decision protocol; it is not equated with Lean-verified success below.

Qwen3.5-35B config	AC Validity (%)	AC Path (%)	Lean-Verified (%)
GRPO	54.28	23.05	14.64
EMPO	53.18	21.52	13.78
Without hyperbolic structural guidance	57.02	27.14	18.82
Without algebraic sparsification	56.31	25.98	18.07
Full SAGE	59.83	31.76	23.69

Key Findings¶

Complementarity appears on both BBH-H and AC. Removing hyperbolic guidance reduces AC Lean success by 4.87 pp, while removing algebraic sparsification reduces it by 5.62 pp. Local validity and end-to-end success should be evaluated separately.
In Table 4's PRM comparison, 35B Lean success is 14.64% for GRPO, 17.36% for GRPO-PRM, and 23.69% for SAGE. However, PRM Olympiad/BBH-H accuracy is 63.27%/59.12%, below GRPO's 65.31%/63.29%; the source's claim that PRM improves GRPO does not generalize across all columns.
The nearly 8-fold claim comes from Figure 3's comparison between Qwen3 and its base model, not from the GRPO comparison in this table or a general proof resolving the open AC problem. The cache lacks complete bar values, so no values are reverse-engineered from that claim.
Appendix H reports five-seed means and standard deviations for free-form tasks: for example, 9B BBH-H is \(45.31\pm0.71\) for SAGE and \(44.04\pm0.82\) for EMPO. Standard deviations are not confidence intervals and do not directly establish statistical significance. Appendix F reports approximately 87% versus approximately 84% for GRPO at Pass@64; these are approximate figure-based results under a multiple-sampling budget.
Source counts conflict: the abstract and experimental overview report 12 benchmarks and 7 models/families, whereas the contribution list and checklist report 13 benchmarks and 8 models. The setup lists multiple series and sizes that cannot simply be interpreted as seven individual backbones. The overview says 3 baselines, but the setup lists four: SFT, GRPO, EMPO, and GRPO-PRM.
Numerical discrepancies also remain: Section 5.2 says 27B ARC-C is 58.11%, while Tables 2 and 5 report 57.11%. Base 35B ARC-C is 45.37% in Table 2 but \(45.7\pm0.39\) in Table 5. These differences are preserved rather than silently reconciled; the main table above follows Table 2.

Highlights & Insights¶

Structural scoring changes both rewards and which training trajectories are proposed and selected. Reward augmentation supplies learning signals, while sampling guidance changes the paths encountered under the same scarcity of terminal success.
Residual compatibility and target distance answer different questions: which unresolved requirement a step addresses and where the resulting state is heading. This differs from merely increasing sampling entropy and explains why shuffled anchors degrade performance.
Restricting expensive modules to training amortizes structural preferences into policy parameters. This is attractive under tight inference budgets, but training overhead must also be reported to assess total cost.

Limitations & Future Work¶

Non-symbolic residuals, operator types, and target anchors are approximate proxies. Cross-domain transfer, erroneous anchors, and probe mismatch should be tested to avoid mistaking high structural scores for factual correctness.
The concentration bound requires a positive potential margin separating admissible and inadmissible trajectories. Hyperbolic geometry alone does not establish this strong assumption. The admissible set also contains unsuccessful trajectories, so concentration there is not concentration on success.
Finite candidates limit exploration coverage, and the relationship between guided behavior and the stated policy ratio lacks a complete correction account. Candidate-count sweeps, direct old-policy sampling, and explicit behavior-distribution correction would be useful comparisons without assuming unbiased updates in advance.
Appendix I reports 4 NVIDIA A100 and 2 NVIDIA H200 GPUs but does not quantify component runtimes or GPU hours. Training costs for algebraic scoring, hyperbolic distances, probes, and additional candidates remain to be broken down.
Model/benchmark counts, best-performance claims, and table–text discrepancies reduce reporting clarity. The code link is taken from the paper; accessibility and reproducibility were not verified during this offline writing task.

vs GRPO: both standardize rewards within rollout groups. SAGE additionally uses structural priors to change rollout sampling and trajectory scoring; it is not simply a renamed GRPO objective.
vs EMPO: the paper uses EMPO as an exploration-oriented post-training comparison. SAGE explicitly represents residual–operator relations and target geometry. Sharing the filtered training subset helps separate algorithm effects from prompt-retention effects.
vs GRPO-PRM: PRM assigns terminal labels to prefixes to train process rewards, adding local validity targets for AC. SAGE's structural probe avoids those correctness labels, but its full RL framework still uses terminal rewards; the supervision sources should not be conflated.
vs inference-time tree search: SAGE reweights training-rollout candidates without running search or filtering at test time. A transferable research direction is to determine which structural priors remain internalized by a policy, rather than automatically increasing test-time search budgets.

Rating¶

Novelty: 4/5. Residual projection and hyperbolic target potentials jointly shape sampling and group-relative rewards.
Experimental Thoroughness: 3/5. Cross-task results and structural controls are substantial, but cost, sampling correction, and some metric definitions remain incomplete.
Writing Quality: 3/5. The theoretical and methodological narrative is clear, but count discrepancies, best-performance wording, and numerical inconsistencies weaken rigor.
Value: 4/5. The framework offers reusable ideas for structural post-training under sparse terminal rewards, contingent on proxy quality and reproducibility.