GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition¶

Conference: ECCV 2026
arXiv: 2602.03370
Code: None
Area: OCR/Document Understanding
Keywords: Handwritten Mathematical Expression Recognition, Masked Diffusion Models, Symbol-Aware Tokenization, Mutual Learning, Iterative Refinement

TL;DR¶

GryphOne redefines handwritten mathematical expression recognition (HMER) from autoregressive sequence generation to an iterative symbol refinement process via discrete masked diffusion. It maintains syntactic consistency under local editing using Symbol-Aware Tokenization (SAT) and enhances refinement stability through Random-Mask Mutual Learning (RMML). This approach comprehensively outperforms re-implemented autoregressive baselines and commercial HMER systems on MathWriting, achieving a 5.51% CER and a 59.9% ExpRate.

Background & Motivation¶

The core difficulty of handwritten mathematical expression recognition (HMER) lies in dual ambiguities: first, symbol ambiguity, where handwriting styles make characters like "z" and "2" hard to distinguish; second, syntactic ambiguity, where spatial layouts (such as superscripts and fractions) are easily misread (e.g., \(\frac{a+b}{c}\) recognized as \(a+\frac{b}{c}\)). Prevailing methods employ an autoregressive (AR) encoder-decoder framework to generate LaTeX sequences token-by-token. However, this design inherently suffers from exposure bias—early prediction errors propagate downstream along the autoregressive chain. Consequently, in complex formulas, the attention mechanism struggles to simultaneously capture local symbolic cues and global structural constraints, where a single minor error can lead to a cascading failure.

Syntax-aware models (e.g., SAN, TAMER) constrain output consistency by encoding syntax trees or graphs, yet they lack syntactic flexibility and exhibit limited generalization when encountering nested structures unseen during training. Non-autoregressive (NAR) models (e.g., NAMER) mitigate exposure bias via parallel prediction but sacrifice iterative correction capabilities—one-shot prediction prevents backtracking to correct already outputted symbols or structures once an error is discovered. The Key Challenge is: AR models possess the capability to build step-by-step but are constrained by the unidirectionality of causal dependence, whereas NAR models eliminate unidirectional dependence but lose the flexibility of progressive refinement. Neither can simultaneously achieve "globally consistent structural reasoning" and "locally flexible symbol refinement".

The Goal of this work is to unify structural consistency and symbolic robustness. The Key Insight is to shift the paradigm of HMER from "generation" to "refinement": instead of writing out the LaTeX sequence from left to right in one go, the process starts from a fully masked sequence and progressively converges to the correct expression through multiple rounds of random unmasking and re-masking. The Core Idea is to implement iterative symbol refinement using a discrete Masked Diffusion Model (MDM). At each step, it predicts the symbols at all masked positions in parallel, and then randomly re-masks a portion according to a diffusion schedule. This allows the model to progressively approach the ground truth in a loop of "guess \(\to\) verify \(\to\) correct", naturally eliminating exposure bias while retaining the freedom of repeated refinement.

Method¶

Overall Architecture¶

The fundamental problem GryphOne aims to solve is how to enable the model to repeatedly refine its predictions until the structure is consistent and symbols are correct, without relying on autoregressive causal dependencies. The overall framework adopts an "encoder-diffusion decoder" architecture: a ViT encoder extracts visual features from the rasterized image of the handwritten formula, and a diffusion decoder performs iterative denoising on a fixed-length masked symbol sequence, eventually outputting the LaTeX sequence. During training, the model learns to recover the complete sequence in a single step from partial observations at arbitrary mask ratios. During inference, starting from a fully masked sequence, it alternates between parallel prediction and random re-masking, arriving at the final result after \(T\) refinement steps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Handwritten Formula Image"] --> B["ViT Encoder<br/>Visual Feature Extraction"]
    B --> D["Masked Diffusion Modeling<br/>Parallel Denoising + Iterative Refinement"]
    C["Symbol-Aware Tokenization SAT<br/>Symbol-Modifier Aligned Embedding"] --> D
    D -->|During Training| E["Random-Mask Mutual Learning RMML<br/>Dual-View KL Consistency Regularization"]
    D -->|During Inference| G["LaTeX Sequence Output"]

Key Designs¶

1. Masked Diffusion Modeling: Redefining HMER as Iterative Symbol Refinement

The root cause of exposure bias in AR decoding is that each step prediction is conditioned on the previous prediction (rather than the ground truth)—the model observes ground truth prefixes during training but must continue generation using its own predictions during inference, causing a discrepancy in conditional distributions between training and inference. GryphOne bypasses this issue entirely using discrete masked diffusion. In the forward process, each position \(i\) of the ground-truth sequence \(x_0\) is independently replaced with a MASK token with probability \(\beta_t = t/T\), yielding \(x_t\). The model learns only a single reverse step—predicting \(p(x_0|x_t)\) in one step from \(x_t\) at any arbitrary mask ratio, computing cross-entropy loss across all positions (both masked and unmasked). Since predictions are parallel and free of causal masking, the input distributions for training and inference are perfectly aligned (both being partially masked sequences), fundamentally eliminating exposure bias.

During inference, the full reverse diffusion process is executed: initializing \(x_T = [\text{MASK}, \dots, \text{MASK}]\), and from \(t=T\) down to \(1\), each step first predicts \(p(x_0|x_t)\) in parallel to obtain the argmax prediction \(\hat{x}_0\). It then randomly re-masks a portion of \(\hat{x}_0\) with probability \((t-1)/T\) based on the forward diffusion schedule to obtain \(x_{t-1}\). This cycle of "predict \(\to\) re-mask \(\to\) re-predict" is fully dual to the denoising behavior during training, and the random re-masking injects controlled perturbations to prevent the model from prematurely converging to structures that are locally plausible but globally inconsistent. The core difference from mask-predict decoding is that mask-predict uses a confidence-based heuristic to select mask locations without an explicit random corruption model, leading to an irregular refinement process. In contrast, GryphOne's diffusion schedule \(\beta_t\) mathematically defines the correspondence between forward corruption and reverse denoising, rendering the refinement trajectory analytical.

2. Symbol-Aware Tokenization (SAT): Localized Editing for Syntactic Consistency

Applying diffusion directly onto raw LaTeX token sequences faces a fundamental obstacle: a local structural edit in LaTeX (e.g., removing a superscript) propagates across multiple tokens. For instance, modifying x_{1}^{y_{2}} to x_{1} y_{2} requires deleting five tokens: ^, {, y, 2, and }. Thus, a local syntactic adjustment manifests as large-scale, non-local changes in the token space. Consequently, the locality assumption of diffusion models (where each position is masked/predicted independently) is heavily corrupted: masking a single ^ collapses the entire downstream superscript syntax, making recovery extremely difficult for the model.

The solution of SAT is to decompose the LaTeX sequence into two aligned sequences: visible symbols (digits, letters, operators) and invisible modifiers (superscript ^, subscript _, curly braces). Modifiers are aligned to their target symbol positions according to specific rules: ^, _, and { are aligned to the nearest visible symbol to their right, while } is aligned to the nearest visible symbol to its left. Multiple modifiers for the same symbol are merged into a single modifier token, and empty tokens are inserted at positions without modifiers. The final embedding for each position is the element-wise sum of the symbol embedding and the modifier embedding:

\[\mathbf{E}_{\text{SAT}}(x_{t,i}) = \mathbf{E}_{\text{sym}}(x_{t,i}) + \mathbf{E}_{\text{mod}}(x_{t,i})\]

Thus, each diffusion unit (a single position) precisely maps to a handwritten glyph and its syntactic modification. The sequence length remains invariant under local structural edits—deleting a superscript only modifies the modifier embedding at a single position without altering the overall sequence length. This preserves the locality assumption of the diffusion process. Moreover, the alignment information between symbols and modifiers in early iterations accelerates structural convergence (experiments show that SAT reduces the SER from 3.71% to 2.21% even at \(T=2\)).

3. Random-Mask Mutual Learning (RMML): Dual-View Consistency Regularization for Refinement Stability

Both the training and inference of masked diffusion carry inherent stochasticity: the mask pattern varies with each sampling, and the same input under different mask configurations can yield inconsistent predictions, which worsens the oscillation of trajectories during inference. The core intuition of RMML is straightforward: a well-performing model should produce similar prediction distributions even when exposed to different masked versions of the same ground truth.

Specifically, in each training iteration, two independent masked versions \(x'_t\) and \(x''_t\) are sampled for the same ground truth \(x_0\). The parameter-sharing decoder processes both to yield prediction distributions \(\hat{x}'_t\) and \(\hat{x}''_t\). The total loss is computed as the sum of four components:

\[\mathcal{L} = \text{CE}(\hat{x}'_t, x_0) + \text{CE}(\hat{x}''_t, x_0) + D_{\text{KL}}(\hat{x}'_t \parallel \hat{x}''_t) + D_{\text{KL}}(\hat{x}''_t \parallel \hat{x}'_t)\]

The first two terms represent standard denoising cross-entropy, while the latter two are symmetric KL divergences that enforce identical prediction distributions across the two masked views. This approach introduces zero additional parameters, incurring only a single extra forward pass during training (sharing the same backbone) and zero overhead during inference. Its effect is particularly pronounced in deep diffusion (\(T=50\)): integrating RMML further reduces the SER from 0.98% to 0.84% (Table 3) while significantly lowering output diversity (Fig. 5), indicating a much more stable refinement process.

Loss & Training¶

The training objective is formulated as \(\mathcal{L}\): the dual-view consistency loss measured by symmetric KL divergence combined with the cross-entropy denoising losses for each view. The default diffusion step is \(T=50\). During training, \(t \sim \mathcal{U}(0, T)\) is randomly sampled for each sample to expose the model to all mask ratios. The optimizer is AdamW with a learning rate of \(10^{-4}\) and weight decay of \(10^{-3}\). The batch size is 32, and the model is trained for 60 epochs. The encoder utilizes DINO ViT (patch size 8, latent dimension 384, input resolution 224x224), while the decoder consists of a 5-layer Transformer with 8 attention heads, 30% dropout, and a maximum sequence length of 150. No data augmentation is applied. During inference, all results are averaged over 10 runs to eliminate stochasticity.

Key Experimental Results¶

Main Results¶

On the MathWriting test set, GryphOne-50 comprehensively outperforms all re-implemented baselines with a 5.51% CER and a 59.9% ExpRate (although ICAL performs best among AR models with a Test CER of 6.03% and EM of 57.7%, GryphOne-50 surpasses it in EM by 2.2 percentage points). Notably, even with only 10 diffusion steps (GryphOne-10), it achieves an FPS of 73.7, which is faster than all AR baselines (where ICAL is only 7.95 FPS) while maintaining over 98% of GryphOne-50's recognition performance. Although MP (mask-predict decoding) also outperforms the AR baselines, it consistently lags behind the diffusion counterpart with the same configuration, proving that the random re-masking under a diffusion schedule is more beneficial for refinement than confidence-guided masking.

Method	FPS↑	Valid CER↓	Valid EM↑	Valid ≤1↑	Test CER↓	Test EM↑	Test ≤1↑
BTTR	8.56	6.45	60.4	72.0	6.85	53.4	65.5
CoMER	8.55	5.96	61.2	72.4	6.50	54.5	65.6
ICAL	7.95	5.43	63.4	74.5	6.03	57.7	68.2
TAMER	6.70	5.79	61.9	73.0	6.35	55.6	66.7
PosFormer	5.04	5.69	62.5	73.6	6.30	56.2	67.1
GryphOne-MP	73.5	5.34	68.2	83.0	6.29	55.8	75.9
GryphOne-10	73.7	4.70	70.6	84.8	5.55	59.3	78.0
GryphOne-50	21.2	4.63	71.0	85.0	5.51	59.9	78.1

In the CROHME 2014-2023 cross-dataset generalization test, GryphOne-50 achieves the highest ExpRate across all four year-versions (2014: 65.2%, 2016: 61.4%, 2019: 61.8%, 2023: 61.2%), outperforming the strongest baseline ICAL by 2.8, 2.4, 0.8, and 2.9 percentage points, respectively (with NAMER literature values as reference; non-re-implemented models are not directly comparable). This validates the generalization capability of the iterative symbol refinement strategy across different data distributions.

Ablation Study¶

The ablation results of SAT and RMML under different diffusion depths \(T\) on the MathWriting validation set demonstrate that the two designs are mutually complementary: SAT yields substantial gains even in shallow diffusion (\(T=2\)), reducing the CER from 5.42 to 5.10 and the SER from 3.71 to 2.21, indicating that symbol-modifier alignment accelerates structural convergence in early iterations. RMML provides more pronounced benefits in deeper diffusion (with the SER dropping from 0.98 to 0.84 at \(T=50\)), showing that consistency regularization is highly effective at stabilizing long-chain refinement. Combining both designs yields the lowest CER and competitive SER across all values of \(T\).

SAT	RMML	T=2 CER↓	T=5 CER↓	T=10 CER↓	T=50 CER↓	T=2 SER↓	T=5 SER↓	T=10 SER↓	T=50 SER↓
-	-	5.42	4.97	4.84	4.75	3.71	1.84	1.24	0.81
-	Y	5.20	4.86	4.75	4.67	2.88	1.42	0.96	0.59
Y	-	5.10	4.84	4.76	4.68	2.21	1.53	1.31	0.98
Y	Y	5.01	4.78	4.70	4.63	1.89	1.33	1.08	0.84

Key Findings¶

SAT and RMML are complementary rather than overlapping: SAT resolves the locality alignment problem between diffusion units and syntactic editing (ensuring that modifications at a single position do not disrupt other positions), whereas RMML addresses the prediction consistency under varying mask configurations. Operating along distinct dimensions, their combination yields optimal results.
Diffusion depth \(T\) serves as a highly flexible control knob for the accuracy-latency trade-off: At \(T=10\), the model reaches an FPS of 73.7 (more than 9\(\times\) faster than AR baselines) and an EM of 70.6%; at \(T=50\), it delivers an FPS of 21.2 and an EM of 71.0%. The performance gain tends to saturate beyond \(T \approx 30\). For practical deployment, a truncated schedule can substantially reduce inference costs with almost no precision loss. This differs fundamentally from AR methods whose latency scales linearly with the expression length; in contrast, GryphOne’s diffusion steps \(T\) are independent of sequence length.
Diffusion refinement effectively corrects structural errors: The step-by-step CER/SER curves (Fig. 3) illustrate that the error rate decreases with \(t\) and converges around step 40. Furthermore, SAT suppresses both CER and SER from the earliest iterations, demonstrating that structural alignment information is effectively utilized during early refinement phases. Error recovery experiments (Fig. 4) reveal that the model struggles most with partially damaged sequences (around \(t \approx 20\)), where the remaining masks are numerous enough to cause incomplete context yet insufficient for complete reconstruction, representing the most challenging phase of refinement.
Low output diversity points to stable convergence: Analysis of output diversity across 10 independent decodings (Fig. 5) shows that the majority of expressions yield only a single unique output. Both SAT and RMML further shrink this diversity, indicating that any remaining stochasticity primarily stems from genuine visual ambiguity (e.g., highly cursive numerators like 14 vs 17 vs 4) rather than model instability.

Highlights & Insights¶

Paradigm shift from "generation" to "refinement": Previous HMER enhancements primarily patched either AR frameworks or one-shot NAR predictions. GryphOne is the first to identify that "the essence of exposure bias lies in the distribution mismatch between training and inference," resolving it fundamentally by exploiting the duality of the forward and reverse processes in diffusion models. This perspective can be extended to any recognition task featuring structured output and long-range dependencies (e.g., formula OCR, sheet music recognition, chemical structure recognition).
The "one-symbol-per-position" alignment concept of SAT is simple yet profound: LaTeX syntactic information is typically scattered across multiple tokens. SAT compresses symbol identity and syntactic role into a single positional vector through element-wise embedding summation, preserving syntactic information while maintaining the locality assumption of diffusion. This technique is essentially a form of "lossless dimensionality reduction"—compressing a syntax tree of \(N\) tokens into an embedding sum over \(N\) independent positions—and can be generalized to any scenario with non-local syntactic dependencies in token sequences (e.g., code completion, structured data generation).
RMML offers a lightweight consistency regularization paradigm: It requires only one additional forward pass and a symmetric KL divergence during training, yielding zero inference overhead while markedly enhancing the stability of diffusion refinement. Its core insight—that a robust denoising model should be mask-invariant—holds broad applicability and can be integrated into any mask-based self-supervised or generative pre-training tasks.

Limitations & Future Work¶

Diffusion inference latency remains a bottleneck for practical deployment: At \(T=50\), it requires 50 decoder forward passes (despite parallel processing at each step), resulting in just 21.2 FPS, significantly lower than the NAR baselines' 73+. While the authors suggest a truncated schedule as a compromise, they do not explore more efficient diffusion sampling strategies (such as ODE acceleration like DPM-Solver, distillation, or adaptive step sizes), which represents the most straightforward path for future improvement.
High demand for training data: Diffusion models are inherently more data-hungry than their AR counterparts. Training on small-scale datasets like CROHME in isolation is insufficient; they rely heavily on MathWriting's 230k real and 400k synthetic samples to demonstrate their advantages. For languages or symbol systems lacking large-scale annotated data, exploring synthetic data augmentation or pre-training followed by fine-tuning will be crucial.
Fixed-resolution ViT encoders provide insufficient support for extreme aspect ratios or ultra-long formulas: A fixed input of 224x224 may lose fine details when handling multi-line formulas, long fraction chains, or matrices. Integrating adaptive-resolution encoding or dynamic patching strategies could be a potential solution.
The alignment rules of SAT may encounter edge cases: Whether the "align to the nearest visible symbol" rule for modifiers correctly handles extreme scenarios, such as nested brackets or complex multi-level superscripts/subscripts, is not thoroughly analyzed in the paper. Alignment errors in highly nested syntactic structures could assign modifier information to incorrect symbol positions, which warrants further verification.

vs NAMER (NAR HMER): NAMER uses a graph decoder to predict all tokens and relationships in a single pass, eliminating exposure bias but failing to support iterative correction—the output cannot be changed once determined. GryphOne's diffusion paradigm naturally supports multi-round refinement, offering greater flexibility in correcting structural errors at the expense of slower inference. These two designs represent opposite ends of the NAR accuracy-speed trade-off spectrum.
vs Mask-Predict (Iterative Masked Modeling): Mask-predict also employs iterative prediction and masking, but its mask selection relies on a confidence-based heuristic and lacks a rigorous forward corruption model. GryphOne unifies training corruption and inference re-masking via a diffusion schedule \(\beta_t = t/T\), making the refinement dynamics theoretically analytical (effectively acting as the reverse process of a discrete absorbing diffusion).
vs TAMER / PosFormer (Syntax-Aware AR): These methods constrain outputs through syntax trees or positional tasks within the AR framework. While they reduce syntactic errors, they still suffer from exposure bias. GryphOne is orthogonal to them; a promising direction would be to integrate syntax-aware concepts into the diffusion framework (e.g., incorporating syntactic validation on top of SAT as a rejection mechanism during inference).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First to apply discrete masked diffusion to HMER. The paradigm shift to "iterative refinement" is pioneering, and both the SAT and RMML designs are elegant and effective.
Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensive comparison against AR/NAR baselines on both MathWriting and CROHME benchmarks, with ablations covering three dimensions (SAT \(\times\) RMML \(\times\) T). The step-wise analysis and error recovery experiments are thorough. However, it lacks a direct comparison with large VLMs (e.g., Uni-MuMER) and does not explore different encoder backbones.
Writing Quality: ⭐⭐⭐⭐⭐ Highly structured. It naturally derives the necessity of diffusion from the respective limitations of AR and NAR models. The pseudo-code and math equations are well-integrated, and the experimental analysis relies on causal reasoning rather than merely listing numbers.
Value: ⭐⭐⭐⭐ Opens up a promising new direction using the diffusion paradigm for HMER. The concept of "refinement over generation" is easily transferable to other structured recognition tasks. Although inference latency remains a challenge for practical deployment, the truncated schedule provides a viable compromise.