Skip to content

Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Pluto-R/DATG
Area: LLM Reasoning
Keywords: Multilingual Mathematical Reasoning, Chain-of-Thought Execution, Directed Acyclic Trace Graphs, Process-Level Diagnosis, Dynamic Intermediate Thought Supervision

TL;DR

This paper demonstrates that the severe multilingual mathematical reasoning gap in large reasoning models stems not just from non-English input misunderstanding but from trace-side execution breakdown; it introduces the Directed Acyclic Trace Graph (DATG) diagnostic framework alongside lightweight dynamic test-time thought supervision (Loop-Retry and Formula-Retry) that substantially recovers target-language reasoning performance.

Background & Motivation

Large reasoning models (LRMs), such as the Qwen3 series and DeepSeek-R1 distilled models, have demonstrated impressive multi-step mathematical problem solving when prompted to generate chain-of-thought (CoT) reasoning in English. However, these gains degrade drastically when models are evaluated on medium- and low-resource languages such as Thai, Bengali, Swahili, and Telugu. For years, the prevailing consensus across academic and industrial research attributed this cross-lingual disparity predominantly to input-side understanding failures: the assumption was that models fail simply because they misread or misunderstand non-English problem statements, derailing the solution before it even begins. Under this assumption, prior efforts have concentrated almost exclusively on question translation, cross-lingual instruction tuning, or multilingual question-alignment pretraining.

Yet, this input-centric explanation is fundamentally incomplete. By carefully decoupling the problem input language from the requested reasoning-trace language on the PolyMath benchmark, the authors uncover a striking phenomenon: even when mathematical problem statements are kept strictly in English (\(en \to x\) setting), forcing the model to generate its thinking trajectory in a non-English language triggers a catastrophic performance drop. For instance, on medium-difficulty problems with Qwen3-4B, English reasoning (\(en \to en\)) achieves 53.6% final-answer accuracy, high-resource target languages average 44.5%, but low-resource languages (LRLs) collapse to an average of 2.0% (with Swahili hitting 0.0%); conversely, keeping the question in a low-resource language while reasoning in English (\(x \to en\)) preserves 29.2% accuracy. This stark contrast confirms that the target reasoning language exerts a profound, independent impact on trace-side reasoning execution itself—how intermediate mathematical entities are derived, how causal dependencies are maintained, and how computations propagate toward the final answer.

Because scalar final-answer accuracy cannot reveal whether a model dropped key intermediate quantities, violated prerequisite ordering, or spiraled into endless loops, there is an urgent need for fine-grained, language-independent process diagnosis. Core idea: isolate multilingual reasoning breakdown to trace-side execution by mapping reasoning traces onto reference-conditioned Directed Acyclic Trace Graphs (DATG) with language-agnostic mathematical anchors and causal dependencies, and deploy dynamic intermediate thought supervision (Loop-Retry and Formula-Retry) to stabilize reasoning trajectories during test-time decoding.

Method

Overall Architecture

The DATG diagnostic and dynamic supervision framework is engineered to audit and rectify the structural decay of non-English reasoning trajectories. The pipeline unfolds across three systematic stages: first, verified reference solutions in English are extracted into language-independent Directed Acyclic Graphs (Reference DAGs) capturing minimal-sufficient mathematical states and causal dependency edges; second, target-language reasoning traces generated under controlled language prefixes are aligned to candidate reference DAGs under a closed-set alignment protocol to compute structural metrics (CAR, PMF, HAR); third, lightweight test-time supervision mechanisms are triggered upon detecting early trace degeneration or missing prerequisites, dynamically steering decoding back onto a sound reasoning path.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multilingual Reasoning Trace Generation<br/>(en→en / en→x / x→x settings)"] --> N1["Language-Agnostic DAG Construction<br/>5 reference derivations mapped to 4-tuple states"]
    N1 --> N2["Closed-Set Trace Alignment & Diagnostics<br/>CAR anchor recall / PMF dependency fidelity / HAR harmful rate"]
    N2 -->|Exposes early loops & missing anchors| N3["Dynamic Test-Time Thought Supervision<br/>Loop-Retry early resampling + Formula-Retry symbolic scaffold"]
    N3 --> Out["Robust Multilingual Mathematical Derivations & Correct Final Answers"]

Key Designs

1. Language-Agnostic DAG Construction: Reference Bias and Natural Language Confounding -> Extract Minimal-Sufficient Symbolic 4-Tuples from Multi-Path References -> Establish Language-Independent Ground-Truth Derivations
Evaluating free-form non-English reasoning against a single rigid English derivation unfairly penalizes alternative valid solution strategies and suffers from translation artifacts. To establish a robust, unbiased diagnostic foundation, DATG generates five diverse English reference derivations per problem using Qwen3.6-Plus and converts each derivation into a directed acyclic graph using GPT-5.4. Each graph node represents a checkable mathematical logic anchor formatted as a clean 4-tuple: $\(\text{node} = (\text{id}, \text{anchor}, \text{description}, \text{parents})\)$ The \(\text{anchor}\) field is strictly confined to language-independent symbolic equations, arithmetic evaluations, algebraic relations, or controlled set/predicate constraints (e.g., \(\text{eggs\_day} = 16\), \(\text{remaining} = 16 - 3 - 4 = 9\)), forbidding conversational prose; \(\text{description}\) provides a brief English readability gloss; and \(\text{parents}\) catalogs prerequisite node IDs required to justify that node. Edges are strictly causal and prerequisite-based rather than chronological, ensuring that independent parallel facts do not artificially constrain downstream reasoning order.

2. Closed-Set Trace Alignment and Three-Dimensional Metrics: Non-English Semantic Ambiguity -> Map Trace Spans to Closed Inventory with Three Orthogonal Metrics -> Disentangle Content Coverage, Topological Fidelity, and Harmful Moves
To diagnose varied non-English formulations without noisy open-ended grading, DATG utilizes GPT-5.4 as an aligner under a closed-set protocol: evidence spans in the target trace are mapped exclusively to existing nodes in the reference DAG and assigned one of four definitive statuses: COMMIT (correctly established), ATTEMPT (partial derivation), ERROR (incompatible/incorrect state), or MISSING (unsupported). From this alignment, DATG derives three complementary structural metrics: - Committed Anchor Recall (CAR): Computes the fraction of reference anchors successfully established by the target trace: \(\text{CAR}(T, G_i) = \frac{1}{|V_i|} \sum_{v \in V_i} \mathbf{1}\{z_i(v) = \text{COMMIT}\}\). - Path Monotonic Fidelity (PMF): Measures whether committed anchors respect the causal dependency structure of reference edge set \(E_i\). An edge \((u, v) \in E_i\) is preserved only if both \(u\) and \(v\) are committed and the textual evidence for \(u\) appears no later than \(v\) in the trace: \(\text{PMF}(T, G_i) = \frac{|M(T, G_i)|}{|E_i|}\). - Harmful Action Rate (HAR): Quantifies the proportion of judgeable trace actions that introduce fatal contradictions, erroneous dependency substitutions, or arithmetic hallucinations: \(\text{HAR}(T, G_i) = \frac{|A_{\text{harm}}(T, G_i)|}{\max(1, |A_{\text{judge}}(T, G_i)|)}\). Together, they identify whether a failure stems from omitted steps (low CAR), disordered reasoning (low PMF), or active mathematical corruption (high HAR).

3. Dynamic Test-Time Thought Supervision: Low-Resource Trajectory Stalling and Entity Hallucination -> Checkpoint Repetition Resampling and Two-Phase Scaffold Guiding -> Stabilize Trace Execution at Negligible Token Overhead
Informed by the DATG diagnostic finding that low-resource traces predominantly fail by dropping transitional anchors and spiraling into repetitive token degeneration, the authors introduce two lightweight test-time interventions. The first is Loop-Retry: an early checkpoint is placed at exactly one quarter of the total generation budget (\(C = B/4\), corresponding to 1024, 2048, and 4096 tokens for low, medium, and high difficulty). The detector monitors the trailing 256 tokens for \(n\)-gram repetition rates (\(\text{rep}_{16}\), \(\text{rep}_{32}\)), type-token ratios (\(\text{ttr}_{256}\)), and consecutive suffix motifs. Paired with a mathematical progress guard checking whether the trailing 1600 characters introduce genuinely new mathematical expressions, any non-progressive loop triggers immediate continuation abortion and resampling under the original prompt. The second is Formula-Retry: a two-phase protocol where Phase 1 generates a compact English/symbolic scaffold of knowns, relationships, and uncomputed formula templates masked with ? inside an <execution_state> block, followed by Phase 2 where target-language derivation proceeds from the preserved scaffold.

Diagnostic Protocol and Dynamic Supervision Mechanism

During full DATG evaluation, the 5-DAG Best-Alignment Protocol guarantees fairness: each target-language reasoning trace \(T\) is aligned across all five candidate reference DAGs \(G_1, \dots, G_5\), and the best match is selected by sequentially maximizing CAR, then maximizing PMF, and finally minimizing HAR. The exact character offsets (\(\text{start\_char}\), \(\text{end\_char}\)) and semantic block identifiers (\(\text{block\_id}\)) are localized in the raw trace to verify that prerequisite anchors chronologically precede dependent anchors.

For dynamic supervision, Loop-Retry is governed by strict mathematical thresholds: a surface loop is flagged if \(\text{rep}_{16} \ge 0.65\), \(\text{rep}_{32} \ge 0.30\), a 4–32 token motif repeats at least three times, or a joint softer condition \(\text{rep}_{16} \ge 0.50 \land \text{ttr}_{256} \le 0.16\) is met. Resampling occurs only when \(\text{math\_progress} = 0\), up to a maximum of five trials with repetition penalty raised slightly from 1.08 to 1.13. This mechanism recovers compromised execution paths while keeping total decoded token cost nearly identical to unconstrained greedy decoding.

Key Experimental Results

Main Results

The diagnostic framework was evaluated across 12 languages on the PolyMath benchmark across three difficulty tiers using the Qwen3 family (1.7B, 4B, 8B) as well as DeepSeek-R1 distilled models. Table 1 summarizes the core DATG structural metrics under the controlled \(en \to x\) trace-language probe across language resource tiers (quoted directly from Table 2 in the paper):

Resource Tier Target Lang. Difficulty CAR ↑ (1.7B / 4B / 8B) PMF ↑ (1.7B / 4B / 8B) HAR ↓ (1.7B / 4B / 8B)
HRL (Ref.) en (\(en \to en\)) Low .94 / .98 / .96 .90 / .96 / .91 .03 / .01 / .01
HRL (Ref.) en (\(en \to en\)) Medium .67 / .78 / .78 .51 / .68 / .67 .09 / .04 / .04
HRL (Ref.) en (\(en \to en\)) High .47 / .61 / .59 .31 / .46 / .45 .14 / .08 / .07
HRL fr (French) Low .90 / .96 / .97 .83 / .93 / .94 .05 / .02 / .02
HRL fr (French) Medium .53 / .74 / .72 .38 / .61 / .60 .17 / .07 / .05
HRL fr (French) High .33 / .52 / .49 .17 / .39 / .35 .24 / .12 / .16
MRL th (Thai) Medium .46 / .67 / .67 .29 / .55 / .54 .27 / .13 / .10
MRL bn (Bengali) Medium .26 / .48 / .53 .11 / .34 / .37 .45 / .25 / .19
LRL te (Telugu) Low .28 / .75 / .86 .15 / .67 / .77 .60 / .18 / .07
LRL te (Telugu) Medium .08 / .19 / .32 .02 / .06 / .18 .71 / .47 / .35
LRL te (Telugu) High .05 / .10 / .18 .01 / .03 / .06 .75 / .56 / .43
LRL sw (Swahili) Low .38 / .59 / .69 .27 / .50 / .62 .50 / .34 / .26
LRL sw (Swahili) Medium .09 / .12 / .28 .05 / .04 / .19 .77 / .69 / .50
LRL sw (Swahili) High .08 / .06 / .11 .04 / .01 / .04 .73 / .68 / .62

Test-Time Control Experiments

On medium-difficulty PolyMath problems, the effectiveness of the proposed dynamic controls was evaluated on low-resource languages (Swahili and Telugu) using Qwen3-4B against standard decoding and retry baselines (quoted directly from Table 4 in the paper):

Control Method sw Acc. (%) te Acc. (%) Avg Acc. (%) Decoded Tokens (M) Retry (%) Avg Trials CAR ↑ PMF ↑ HAR ↓
Loop-Retry (Ours) 35.2 8.8 22.0 1.809 50.4 1.90 .30 .18 .51
Formula-Retry (Ours) 18.4 9.6 14.0 1.937 46.8 2.00 .30 .13 .38
Repetition Penalty 9.6 12.8 11.2 1.623 0.0 1.00 .19 .09 .60
Fixed Retry 14.4 8.8 11.6 1.821 100.0 2.00 .21 .14 .63
Best-of-2 (Oracle Upper) 8.0 12.8 10.4 3.543 100.0 2.00 .18 .08 .62
Random Retry 11.2 8.0 9.6 1.548 52.8 1.53 .22 .13 .61
Base (No Intervention) 4.8 8.0 6.4 1.772 0.0 1.00 .18 .07 .62

Key Findings

  1. Structural Metrics Faithfully Reflect Mathematical Correctness: Traces that produce the correct final answer exhibit consistently high anchor coverage (overall CAR of .91 vs. .36 for incorrect traces) and causal fidelity (PMF of .85 vs. .22), while maintaining a minimal harmful action rate (HAR of .04 vs. .32). Even on high-difficulty problems, correct solutions retain CAR .79 and PMF .67, proving that DATG captures deep causal soundness rather than superficial length or vocabulary overlap.
  2. Low-Resource Languages Suffer Catastrophic Trace-Level Collapse: In medium- and high-difficulty regimes, low-resource reasoning execution deteriorates precipitously. For Qwen3-4B in Swahili, medium-difficulty CAR drops to .12 and PMF drops to .04, while HAR spikes to .69. Models omit indispensable intermediate variables, apply subsequent mathematical operations to mismatched values, and eventually degenerate into non-terminating repetitive babble.
  3. Targeted Intervention Beats Brute-Force Decoding: Best-of-2 expends 3.543M decoded tokens but achieves only 10.4% average accuracy. In contrast, Loop-Retry consumes only 1.809M tokens (a tiny ~2% increase over Base's 1.772M) yet boosts low-resource accuracy from 6.4% to 22.0% (surging from 4.8% to 35.2% on Swahili), simultaneously improving CAR to .30, PMF to .18, and lowering HAR to .51.

Highlights & Insights

  • Decoupling Input Comprehension from Reasoning Execution: By establishing the \(en \to x\) experimental probe, this work overturns the long-standing assumption that multilingual reasoning failures originate solely from misreading non-English questions, pinpointing the target thinking language itself as an autonomous source of logical derailment.
  • Language-Agnostic Process Diagnostics: Representing multi-step derivations as directed acyclic trace graphs over symbolic tuples bridges the gap between heterogeneous natural languages and invariant mathematical causality. Human evaluation confirms 97.5% node validity, 96.5% edge validity, and 90.1% HAR precision.
  • Actionable, Low-Cost Test-Time Stabilization: Demonstrates that severe reasoning failures in underserved languages can be significantly alleviated without costly retraining or pre-translation, simply by monitoring repetition at the first quarter of decoding and guiding execution via uncomputed symbolic scaffolds.

Limitations & Future Work

  • Reliance on Commercial LLMs for Diagnostic Extraction: DATG relies on Qwen3.6-Plus for reference generation and GPT-5.4 for DAG extraction and closed-set alignment; its application to frontier mathematical olympiad domains without established reference solutions remains bounded by the ceiling of current frontier models.
  • Persistent Language Drift in Complex Regimes: While direct-first prefixes enforce target-language thinking, models occasionally experience code-switching back to English when encountering complex algebraic notation in low-resource settings.
  • Automated Induction of Formula Scaffolds: Formula-Retry relies on the model itself generating sound skeletons during Phase 1; on high-difficulty problems, models frequently struggle to produce correct scaffolds without external knowledge retrieval.
  • vs. Multilingual CoT Benchmarks (mCoT, MMATH, PolyMath): Traditional benchmarks evaluate accuracy as a binary scalar or report token-level cross-lingual consistency; DATG provides the first graph-structured, causally audit-able process framework.
  • vs. Process Supervision & PRMs (Math-Shepherd, ProcessBench, ROSCOE): Existing process-reward models operate primarily on English sentences and reward surface lexical flow; DATG operates over language-independent mathematical states, making process evaluation truly cross-lingual.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Decouples trace execution from input understanding and pioneers reference-conditioned trace DAG extraction.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 12 languages, 3 model families, 3 difficulty splits, and extensive human calibration.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent, mathematically disciplined formulation of metrics, and lucid empirical narratives.
  • Value: ⭐⭐⭐⭐⭐ Delivers an indispensable diagnostic toolkit and actionable test-time controls for the multilingual reasoning community.