Skip to content

Reactivating Test-Time Scaling for Plane Geometry Problem Solving

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Jason8Kang/ReTTS-PGPS
Area: LLM Reasoning
Keywords: Plane Geometry Problem Solving, Test-Time Scaling, Multi-Trace Synthesis, Perception-Augmented Training, Consensus-Guided Ensemble

TL;DR

This paper diagnoses the breakdown of test-time scaling in plane geometry solving caused by rigid symbolic representations and perceptual errors, and re-activates scaling via Multi-Trace Synthesis (MTS), Perception-Augmented (PA) training, and Consensus-Guided Multi-Trace Ensemble (CG-MTE), matching high-budget self-consistency accuracy while slashing sampling overhead by up to 8x.

Background & Motivation

Plane geometry problem (PGP) solving represents a foundational benchmark for multimodal reasoning, demanding both precise visual perception of diagrammatic entities and rigorous multi-step symbolic deduction over geometric theorems. In general mathematical reasoning and code generation, test-time scaling (TTS)—predominantly driven by self-consistency (SC) via sampling multiple trajectories and executing majority voting—has established itself as a potent paradigm to enhance model capabilities without retraining. However, when standard self-consistency is directly deployed on plane geometry problems under the prevailing symbolic program prediction paradigm, its scaling benefits stagnate severely, frequently failing to produce an effective consensus or surpass greedy decoding baselines.

Investigating the root causes of this empirical failure reveals two profound bottlenecks. First, existing geometry solvers and datasets (such as PGPS9K and Geometry3K) rely almost exclusively on domain-specific formal programs. These highly compressed, rigid symbolic representations lack the natural-language deliberation and exploratory freedom inherent in standard chain-of-thought (CoT) prompting. Because each valid formal derivation is typically unique, repeated stochastic sampling either replicates the exact same correct sequence or yields divergent, invalid expressions, resulting in a collapsed search space with negligible diversity. Second, a severe perceptual gap decouples diagram parsing from deductive theorem application. Subtle diagram misinterpretations—such as mistaking a \(101^\circ\) angle for \(104^\circ\) or misidentifying parallel segments—directly corrupt downstream symbolic execution, rendering subsequent program paths futile.

Simply increasing sampling temperatures or inflating candidate budgets cannot overcome these intrinsic structural constraints. The solution requires systematically expanding the reasoning space across complementary execution paradigms while grounding formal deduction in explicit diagrammatic semantics prior to reasoning. Core idea: systematically transform rigid symbolic programs into heterogeneous reasoning traces comprising executable Python code and rationale-enriched chains of thought (MTS), ground deduction on explicitly parsed diagrammatic semantic clauses (PA), and orchestrate inference with a depth-adaptive consensus-guided ensemble (CG-MTE) to reactivate test-time compute scaling at drastically reduced sampling costs.

Method

Overall Architecture

The proposed framework comprises three unified components spanning data curation, model training, and adaptive inference. First, Multi-Trace Synthesis (MTS) unpacks seed symbolic programs into four semantically aligned yet structurally diverse reasoning trace formats. Second, Perception-Augmented (PA) Training implements a two-stage sequential formulation that forces the model to articulate structured geometric clauses before executing theorem deduction. Third, Consensus-Guided Multi-Trace Ensemble (CG-MTE) drives adaptive inference by expanding decoding depth across heterogeneous trace types only until a unique consensus emerges.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Geometry Problem<br/>Diagram D + Question Q"] --> B["Perception-Augmented Training<br/>Parse Explicit Semantic Clauses S"]
    B --> C["Multi-Trace Synthesis<br/>Co-generate 4 Heterogeneous Traces"]
    C --> D["Consensus-Guided Multi-Trace Ensemble<br/>Progressive Depth Expansion with Early Exit"]
    D --> E["Final Numerical Answer A"]

Key Designs

1. Multi-Trace Synthesis: Expanding Rigid Programs into Heterogeneous Executable and Rationale Traces

Addressing the narrow reasoning space and lack of natural-language deliberation inherent in rigid domain-specific programs, Multi-Trace Synthesis (MTS) expands each instantiated formal program into four distinct formats without generating synthetic hallucinated problems: standard Programs, executable Program-Aided Language (PAL) scripts, Chain-of-Thought Programs (CoT-Program), and Chain-of-Thought PAL scripts (CoT-PAL). The synthesis starts by converting abstract programs into self-contained instantiated programs by binding variable slots to actual problem constants (e.g., mapping Geo_Mean N0 N1 N4 to Geo_Mean 3\sqrt{2} z \sqrt{2}). A rule-based translation compiler translates these into standardized SymPy Python scripts featuring algebraic equation setup, symbolic solving, and geometric constraint filtering (discarding non-positive side lengths or degenerate intersection angles). Generated PAL scripts undergo strict execution-based sandbox verification against ground-truth numeric answers with a relative tolerance of \(\epsilon = 10^{-3}\), excluding corrupted samples. Next, multimodal language models enrich these verified paths with natural language rationales: CoT-PAL interleaves explicit geometric justifications prior to each algebraic equation formulation under a generate-and-repair feedback loop, while CoT-Program pairs each symbolic operator with theorem-grounded step-by-step prose. This multi-trace mixture combines the computational precision of programmatic solvers with the deliberate heuristic exploration of natural-language rationales.

2. Perception-Augmented Training: Bridging the Visual-to-Symbolic Gap via Explicit Semantic Clauses

To prevent diagram parsing errors and perceptual hallucinations from propagating unconstrained into symbolic theorem application, Perception-Augmented (PA) Training structures the problem-solving workflow into an explicit perception-then-reasoning pipeline. Rather than directly predicting solution traces from raw visual tokens, the model is trained to first decode the diagram \(D\) and textual question \(Q\) into structured geometric semantic clauses \(S = (S_1, \dots, S_{|S|})\) (e.g., explicit statements of perpendicularity \(AC \perp DB\), known dimensions \(AB = 3\sqrt{2}\), and unknown variable assignments \(BC = z\)). Only after outputting this structured grounding does the model condition on \(S\) to autoregressively synthesize the reasoning trajectory \(T = (T_1, \dots, T_{|T|})\). The joint learning objective maximizes the combined log-likelihood: $\(\mathcal{L} = -\log P_\theta(S, T \mid D, Q) = -\log P_\theta(S \mid D, Q) - \log P_\theta(T \mid S, D, Q) = \mathcal{L}_{\text{perc}} + \mathcal{L}_{\text{reason}}\)$ where the perception loss \(\mathcal{L}_{\text{perc}}\) enforces faithful visual attribute parsing and the reasoning loss \(\mathcal{L}_{\text{reason}}\) disciplines theorem deduction based upon verified visual facts. This explicit intermediate grounding eliminates hallucinated geometric relations and provides a solid foundation for symbolic derivation.

3. Consensus-Guided Multi-Trace Ensemble: Adaptive Test-Time Compute Allocation via Cross-Trace Mode Agreement

Addressing the extreme compute redundancy of standard multi-trace ensembling and high-budget self-consistency—which blindly evaluate maximal sample budgets (e.g., \(V \times D = 4 \times 10 = 40\)) across both trivial and difficult problems—Consensus-Guided Multi-Trace Ensemble (CG-MTE) introduces a threshold-free progressive stopping rule. For any test instance \(x = (D, Q)\), the fine-tuned model is prompted across \(V=4\) trace-specific instructions to generate ranked candidates via beam search. CG-MTE examines candidates incrementally across depth levels \(d \in \{1, \dots, D\}\). At depth \(d\), it gathers the executed answers from the top-\(d\) candidates across all \(V\) traces into an aggregated pool \(S_d(x) = \bigcup_{j=1}^V \{a_{j,1}(x), \dots, a_{j,d}(x)\}\) and computes the modal answer \(v^*_d(x) = \text{mode}(S_d(x))\). If a unique mode exists, the decoding immediately halts, returning \(v^*_d(x)\) as the final prediction. If a tie or no clear consensus occurs, the search depth increments to \(d+1\), dynamically allocating more compute only to ambiguous, uncertain queries. If no unique mode emerges even at maximal depth \(D\), the system cleanly falls back to the top-1 prediction of the most reliable single format (CoT-PAL). Because over 80% of standard geometry problems reach unanimous agreement at \(d=1\), CG-MTE compresses the average sampling number from 40 down to under 5 while preserving high-budget accuracy.

Formal Definition and Evaluation Protocol

A plane geometry problem instance is formally represented as a five-tuple \(I = (D, Q, S, P, A)\), where \(D\) is the diagram image, \(Q\) is the textual problem description, \(S\) denotes structured semantic clauses, \(P = \langle s_1, \dots, s_T \rangle\) is a sequence of formal operators \(opt \in \mathcal{O}\) drawn from 34 Euclidean theorems and axioms, and \(A\) is the ground-truth numerical answer. Performance is benchmarked using top-1 numerical answer accuracy (%). A predicted numeric value \(\hat{a}\) is counted as correct if and only if it matches the ground truth \(a\) within a relative error tolerance: $\(\frac{|\hat{a} - a|}{\max(|a|, 10^{-4})} \le 10^{-3}\)$ Computational efficiency across test-time scaling regimes is benchmarked using the Average Sampling Number (ASN), quantifying the mean candidate generation budget expended per instance to characterize the empirical Pareto frontier.

Key Experimental Results

Main Results

The framework is evaluated across three benchmark datasets (PGPS9K, Geometry3K, and GeoQA) against premier general-purpose MLLMs, specialized neural geometry solvers, and neural-symbolic systems under greedy decoding:

System Category Representative Model / Method PGPS9K (%) Geometry3K (%) GeoQA (%)
General-purpose MLLM GPT-4V 33.3 34.8 43.4
General-purpose MLLM Claude 3.5 Sonnet 27.6 32.0 49.2
General-purpose MLLM Qwen2.5-VL-72B 53.3 50.5 55.5
Neural Geometry Solver PGPSNet 60.3 65.0 –
Neural Geometry Solver LANS 66.7 72.1 –
Neural-symbolic Solver InterGPS (Diagram GT) 59.8 64.2 –
Neural-symbolic Solver GeoDRL 55.6 70.6 –
Fine-tuned Baseline (Direct Program) Qwen2-VL-2B 43.4 41.4 47.1
Fine-tuned Baseline (Direct Program) Qwen2.5-VL-3B 46.7 53.1 53.4
Fine-tuned Baseline (Direct Program) Qwen3-VL-8B 58.6 65.4 60.9
Ours: PA Training on MTS-All Qwen2-VL-2B 50.8 (+7.4) 52.0 (+10.6) 53.5 (+6.4)
Ours: PA Training on MTS-All Qwen2.5-VL-3B 58.3 (+11.6) 64.7 (+11.6) 58.7 (+5.3)
Ours: PA Training on MTS-All Qwen3-VL-8B 71.2 (+12.6) 74.5 (+9.1) 67.2 (+6.3)

Ablation Studies & Test-Time Scaling Comparisons

Ablations on PGPS9K isolate the impact of diagrammatic grounding, trace diversity, and the Pareto efficiency of adaptive decoding mechanisms:

Setting / Decoding Method Training Size / Budget 2B Acc. (%) 3B Acc. (%) 8B Acc. (%) Average Sample No. (ASN)
PA Training on PGPS9K-All (Full) 32K 50.8 58.3 71.2 1.00
w/o PA Training (Direct Program) 32K 45.5 (-5.3) 51.8 (-6.5) 65.4 (-5.8) 1.00
w/o PGPS9K-MTS (Repeated Program) 32K 46.7 (-4.1) 54.4 (-3.9) 67.6 (-3.6) 1.00
Single Trace: Program 8K 46.2 54.2 67.5 1.00
Single Trace: PAL 8K 46.7 54.4 67.4 1.00
Single Trace: CoT-Program 8K 46.0 54.5 67.9 1.00
Single Trace: CoT-PAL 8K 47.2 55.4 68.3 1.00
Unified Model SC@40 (Beam Search) K=40 – 62.0 76.7 40.00
Adaptive Beam SC: Rank-Split Max 40 – 61.1 75.2 4.68
Adaptive Beam SC: Pattern-Split Max 40 – 62.6 75.8 12.72
CG-MTE (Ours) Max 40 – 62.1 76.0 4.89

Key Findings

  • Explicit perception is essential for grounding formal deductions: Removing the semantic clause parsing step (w/o PA Training) causes acute performance drops of 5.3%, 6.5%, and 5.8% across 2B, 3B, and 8B backbones on PGPS9K, verifying that grounding intermediate deductions on parsed visual facts is critical.
  • Trace heterogeneity trumps simple data duplication: Controlling for token and instance budgets, models trained on diverse reasoning formats (MTS-All) outperform those trained on repeated homogeneous programs by roughly 4.0%. Moreover, CoT-PAL achieves 98.5% human-verified reasoning correctness (outperforming CoT-Program's 93.0%) while reducing annotation prompt overhead from 5.1K to 0.9K tokens.
  • CG-MTE achieves optimal Pareto scaling efficiency: On Qwen3-VL-8B, CG-MTE reaches 76.0% accuracy with an ASN of just 4.89, virtually matching high-budget SC@40 (76.7% at ASN = 40.0) while reducing decoding costs by up to 8x. Over 80% of test instances achieve decisive cross-trace consensus at \(d=1\).

Highlights & Insights

  • Reactivating Test-Time Compute via Heterogeneous Rationale Expansion: Solves the failure of self-consistency in geometry problem solving by introducing code-executable and rationale-enriched formats that provide the exploration diversity necessary for majority voting.
  • Synergistic Neuro-Symbolic Verification Loop: Combines rule-based compilation with execution-driven sandbox verification and automated MLLM error-traceback correction, ensuring training data fidelity and preventing hallucinated geometric theorems.
  • Dynamic Allocation of Test-Time Inference Compute: Implements an intuitive, parameter-free stopping rule based on multi-trace modal emergence, directing heavy test-time compute toward ambiguous questions while fast-tracking easy instances.

Limitations & Future Work

  • Dependency on Structured Ground-Truth Annotations: The framework relies on seed formal programs for MTS compilation and requires ground-truth semantic clause annotations during PA training, limiting direct zero-shot application to unannotated geometry datasets.
  • Domain Confinement to 2D Euclidean Geometry: Validation is strictly restricted to plane geometry benchmarks; generalizability to 3D solid geometry, kinematic diagrams, or complex physical reasoning tasks remains unexplored.
  • Potential Latent Semantic Drift in Natural Language Rationale: While code execution ensures final numerical validity, execution sandboxes cannot exhaustively verify whether every intermediate natural language explanation in CoT-PAL strictly reflects the underlying geometric theorem without superficial hallucinations.
  • vs LANS / PGPSNet: Unlike specialized neural geometry solvers that generate single-mode programs end-to-end, this work reframes geometry deduction under modern MLLMs and reactivates test-time search scaling via multi-format execution.
  • vs AlphaGeometry: While AlphaGeometry couples language models with complex external deduction engines to discover auxiliary constructions in Olympiad proofs, this method focuses on practical, self-contained multi-trace synthesis and adaptive inference for standard multimodal question-answering benchmarks.
  • vs GeoThought / TR-CoT: Rather than using teacher MLLMs to generate entirely new geometry problems that risk physical or geometric inconsistency, this method strictly derives diverse reasoning representations from verified formal programs, safeguarding mathematical soundness.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Identifies the exact failure mechanism of test-time scaling in geometry problems and proposes an elegant multi-trace solution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thorough evaluation across three benchmarks, multiple model scales, token-matched ablations, and extensive ASN efficiency curves.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp formal definitions, intuitive figures, and clear empirical insights.
  • Value: ⭐⭐⭐⭐⭐ Highly actionable insights for expanding test-time scaling and hybrid reasoning across code, symbols, and natural language.