Hierarchical Style Aggregation for Versatile Chinese Handwriting Generation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/IIP-Lab-XDU/HiSAC
Area: Image Generation
Keywords: Chinese handwriting generation, few-shot learning, hierarchical style aggregation, frequency-aware encoding, versatile framework

TL;DR

HiSAC explicitly decomposes Chinese characters into radical, component, and stroke structural hierarchies, pairing them with a multi-band frequency-aware style encoder and cross-scale aggregation to unify both online trajectory generation and offline diffusion-based handwriting synthesis.

Background & Motivation

Few-shot Chinese handwriting generation aims to render target characters in a specific writer's handwriting style given only a few sample reference glyphs (e.g., 5 reference images). This capability is crucial for long-tailed and open-set document recognition, digital humanities, and personalized handwriting synthesis. However, Chinese characters feature an immense vocabulary and a deeply compositional, multi-level orthographic structure. Prior methods primarily follow a character-classification paradigm where content encoders are pre-trained on optical character recognition tasks. Consequently, these models learn "what the character is" in an abstract latent space rather than "how the character is constructed" in physical space and temporal trajectory, while simultaneously compressing handwriting style into an undifferentiated, global image-level embedding.

Because handwriting style manifests across multiple distinct granularities—spanning holistic layout aspect ratio, sub-character component spacing, and micro-scale stroke curvature—existing methods suffer from an inherent mismatch between their flat feature representations and the compositional nature of Chinese characters. In few-shot scenarios, this disconnect results in stroke omission, stroke redundancy, or severe structural collapse when synthesizing complex, unseen glyphs. Furthermore, the community has remained fractured between online trajectory-based models (using RNNs or Transformers) and offline pixel-based synthesis (using GANs or Diffusion models), with no unified stylistic representation bridging the two paradigms.

This paper tackles the challenge by aligning the generative process with the intrinsic compositional mechanisms of Chinese characters, while disentangling style representations in both spatial and frequency domains. The core idea is to explicitly decompose characters into a three-level structural hierarchy of radicals, components, and strokes, pair it with a multi-band frequency-aware style encoder, and employ cross-scale hierarchical style aggregation to provide a unified conditioning representation for both online trajectory and offline diffusion generators.

Method

Overall Architecture

The HiSAC framework comprises four interconnected components: First, target and reference characters are decomposed into multi-level structural sequences (radicals, components, and strokes) using standard ideographic description sequences. Second, a multi-band Frequency-Aware Style Encoder (FASE) splits the reference image features into coarse, medium, and fine spectral bands. Third, the Hierarchical Style Aggregation (HiSA) module fuses structural and stylistic representations using cross-attention for radical/component levels and cosine similarity matching for stroke levels. Finally, the aggregated representation \(Z\) serves as a versatile condition to drive either an online Transformer trajectory decoder or an offline conditional diffusion generator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Target Character & Reference Glyphs"] --> B["Hierarchical Char Representation<br/>Radicals / Components / Strokes"]
    A --> C["Frequency-Aware Style Encoder FASE<br/>Coarse / Medium / Fine Bands"]
    B --> D["Hierarchical Style Aggregation HiSA<br/>Cross-Attention & Stroke Cosine Matching"]
    C --> D
    D --> E["Versatile Generation Decoder<br/>Online GMM Trajectory / Offline Diffusion"]
    E --> F["Output: Stylized Handwritten Glyphs"]

Key Designs

1. Hierarchical Char Representation: Grounding Generation in Explicit Compositional Structure To overcome the limitations of recognition-based black-box content encoders, HiSAC leverages an ideographic description sequence lookup table and recursive decomposition algorithm with near-zero computational overhead. The target character \(c_{\text{tgt}}\) is decomposed into three granularities: a radical-level sequence \(R_{\text{tgt}} = \{r_j\}_{j=1}^{n_r}\) capturing global structure, a component-level sequence \(C_{\text{tgt}} = \{c_j\}_{j=1}^{n_c}\) encoding spatial layout, and a stroke-level sequence \(S_{\text{tgt}} = \{s_j\}_{j=1}^{n_s}\) detailing fine-grained stroke sequences. Reference characters are likewise decomposed into stroke sequences \(S_{\text{ref}}\). This formulation replaces opaque semantic labels with explicit structural scaffolding.

2. Frequency-Aware Multi-band Style Encoding (FASE): Tri-Path Spectral Disentanglement Handwriting style intrinsically spans diverse spectral ranges: low frequencies dictate global slant and aspect ratio, while high frequencies dictate stroke sharp turns, hook dynamics, and ink curvature. FASE partitions the patch embeddings \(X \in \mathbb{R}^{n \times d}\) along the channel dimension into coarse \(X_c\), medium \(X_m\), and fine \(X_f\) groups: - Coarse (Narrowband) Branch: Leverages the natural low-pass filtering property of self-attention in Vision Transformers to extract holistic attributes such as glyph slant and character aspect ratio. - Medium (Broadband) Branch: Applies a \(1 \times 1\) pointwise convolution followed by depthwise convolution and GELU activation to capture mid-frequency stroke continuity and curvature. - Fine (Extended-band) Branch: Employs stride-1 max pooling followed by a pointwise convolution to capture high-frequency stroke tips, sharp corners, and micro-dynamics. The outputs are integrated via channel-wise concatenation, LayerNorm, and a residual MLP to obtain style embedding \(Y\). To promote style discrimination across writers, a triplet contrastive loss with temperature \(\tau = 0.07\) is optimized on the extracted style representations: $$ \mathcal{L}{\text{ctl}} = - \log \frac{\exp(Y_a \cdot Y_p / \tau)}{\exp(Y_a \cdot Y_p / \tau) + \sum $$ where }^k \exp(Y_a \cdot Y_{n_i} / \tau)\(Y_a\), \(Y_p\), and \(Y_{n_i}\) denote anchor (generated), positive (same writer), and negative (different writer) samples, all constrained to share the same character category to disentangle style from semantics.

3. Hierarchical Style Aggregation (HiSA): Multi-Granularity Coarse-to-Fine Alignment To transfer reference style features \(Y = [Y_1, \dots, Y_k]\) into the target character structure, HiSA adopts scale-specific matching operations. At the radical and component levels, continuous structural embeddings \(E_r\) and \(E_c\) query the reference style features via cross-attention: $$ Z_{r/c} = \text{softmax}\left( \frac{E_{r/c} Y^\top}{\sqrt{d}} \right) Y $$ At the stroke level, fine-grained details depend on local stroke correspondences. The average cosine similarity \(\alpha_i\) between reference stroke embeddings \(e_{i,j}\) and target stroke embeddings \(e_n\) is computed: $$ \alpha_i = \frac{1}{m_i n_s} \sum_{j=1}^{m_i} \sum_{n=1}^{n_s} \frac{\mathbf{e}{i,j}^\top \mathbf{e}_n}{|\mathbf{e} $$ Normalized weights }| |\mathbf{e}_n|\(\xi_i = \exp(\alpha_i) / \sum_j \exp(\alpha_j)\) aggregate the stroke-level style representation \(Z_s = \sum_i \xi_i Y_i\). The final unified conditioning vector is concatenated as \(Z = \text{Concat}(Z_r, Z_c, Z_s)\).

4. Versatile Generation Decoder: Generator-Agnostic Conditioning for Trajectories and Pixels The aggregated vector \(Z\) is generator-agnostic and plugs directly into either online or offline pipelines: - Online Trajectory Generation: Uses a Transformer decoder where component representation \(E_c\) serves as the initial token and cross-attention attends to \(Z\). The decoder outputs bivariate Gaussian Mixture Model (GMM) parameters for pen movement offsets along with pen-state probabilities (down, up, end), trained with negative log-likelihood \(\mathcal{L}_{\text{GMM}}\) and classification loss. - Offline Image Synthesis: Implements a conditional DDPM where \(Z\) is injected into the U-Net via cross-attention to guide noise prediction \(\epsilon_\theta(x_t, t, Z)\), combined with multi-layer perceptual loss \(\mathcal{L}_{\text{VGG}}\) from a pre-trained VGG16 network.

Loss & Training

• Online Training Objective: $$ \mathcal{L}{\text{online}} = \lambda}} \mathcal{L{\text{ctl}} + \lambda $$ where pen-state cross-entropy loss is weighted by }} \mathcal{L}_{\text{GMM}\(\lambda_{\text{state}} = 2\), and default coefficients are \(\lambda_{\text{ctl}} = 1, \lambda_{\text{GMM}} = 1\).
• Offline Training Objective: $$ \mathcal{L}{\text{offline}} = \lambda}} \mathcal{L{\text{ctl}} + \lambda}} \mathcal{L{\text{MSE}} + \lambda $$ where perceptual loss weight is }} \mathcal{L}_{\text{VGG}\(\lambda_{\text{VGG}} = 0.01\), and others are set to 1.

Key Experimental Results

Main Results

The framework is trained on the CASIA-OLHWDB and CASIA-HWDB benchmarks (approx. 3.7M character samples from 1,020 writers) and tested on the ICDAR 2013 competition dataset (60 writers, 3,755 common Chinese characters). Evaluated with \(k=5\) reference glyphs.

1. Online Chinese Handwriting Generation (Original Paper Table 1)

Metrics evaluate temporal trajectory alignment (DTW, lower is better), stylistic similarity via a style classifier (Style Score, higher is better), and character recognition accuracy (Content Score, higher is better):

Method	Venue	DTW ↓	Style ↑	Content ↑
DeepImitator	PR 2020	1.0476	62.56	92.07
WriteLikeYou	CGF 2021	0.9244	84.78	94.26
SDT	CVPR 2023	0.8780	94.56	96.47
OLHWG	ICLR 2025	0.9173	91.28	90.11
DNA	WACV 2026	0.9041	93.43	96.23
HiSAC (Ours)	ECCV 2026	0.8224	97.01	96.68

HiSAC reduces the DTW trajectory alignment error by 0.0556 compared to the runner-up (SDT 0.8780) and improves Style Score by 2.45 points, setting a new benchmark for trajectory fidelity and stylistic authenticity.

2. Offline Chinese Handwriting Generation (Original Paper Table 2)

Comparisons span both GAN-based and diffusion-based paradigms, evaluating image synthesis quality (FID, LPIPS, RMSE) alongside style and content scores:

Method	Venue	FID ↓	LPIPS ↓	RMSE ↓	Style ↑	Content ↑
HWT	ICCV 2021	162.43	0.3112	0.3245	3.78	5.23
VATr	CVPR 2023	136.78	0.2733	0.3105	38.49	43.25
WordStylist	ICDAR 2023	39.27	0.1725	0.1661	57.74	82.79
DiffusionPen	ECCV 2024	35.14	0.1678	0.1608	61.78	86.18
One-DM	ECCV 2024	35.45	0.1652	0.1520	63.47	86.27
DiffBrush	ICCV 2025	36.41	0.1704	0.1573	66.18	87.11
\(\text{HiSAC}_{\text{Diffusion}}\) (Ours)	ECCV 2026	34.88	0.1627	0.1531	68.21	89.48
\(\text{HiSAC}_{\text{VAR}}\) (Ours)	ECCV 2026	34.17	0.1638	0.1485	68.04	89.22

Ablation Study

Ablation on FASE frequency branches and HiSA structural levels under the online setting (Original Paper Table 3):

FASE-Coarse	FASE-Medium	FASE-Fine	HiSA-Stroke	HiSA-Comp.	HiSA-Radical	DTW ↓	Style ↑	Content ↑	Note
✓	✗	✗	✓	✓	✓	0.8318	93.37	96.49	Preserves only global posture; lacks stroke details
✗	✓	✗	✓	✓	✓	0.8286	93.75	96.58	Preserves mid-frequency continuity
✗	✗	✓	✓	✓	✓	0.8254	94.50	96.75	Preserves high-frequency sharp turns
✓	✓	✓	✗	✗	✗	1.2146	83.22	67.35	Removing HiSA leads to severe structural collapse
✓	✓	✓	✗	✓	✓	0.8373	96.88	95.62	Without stroke alignment, fine dynamics degrade
✓	✓	✓	✓	✗	✓	0.8321	96.62	96.18	Without component alignment, layout stability drops
✓	✓	✓	✓	✓	✗	0.8410	96.91	95.80	Without radical alignment, global skeletal posture degrades
✓	✓	✓	✓	✓	✓	0.8224	97.01	96.68	Full HiSAC model achieves optimal performance

Key Findings

1. Explicit structural priors are essential to prevent glyph collapse: Completely removing the HiSA module causes Content Score to plummet from 96.68 to 67.35, while DTW error surges to 1.2146. This shows that without multi-scale structural guidance, handwriting models generate structurally invalid pseudoglyphs.
2. Frequency bands provide complementary stylistic cues: When using any single FASE branch, Style Score remains between 93.37 and 94.50. Combining all three spectral bands propels the score to 97.01. Spectrum visualizations confirm that low frequencies capture global slant and aspect ratios, whereas high frequencies govern corner sharpness and hook dynamics.
3. Robust generalization across scripts and tasks: In zero-shot Japanese Kanji synthesis (evaluated without Japanese training data), HiSAC leverages shared ideographic compositional logic to render complete, accurate glyphs, avoiding the stroke omission issues seen in SDT and WriteLikeYou. It also transfers effectively to Chinese artistic font generation (Table 4, 16.23 FID, 0.4684 SSIM) and online English word generation.

Highlights & Insights

• Decoupling recognition from generation: Replacing pre-trained classification models with explicit compositional decompositions (radicals, components, and strokes) allows the network to learn how characters are constructed rather than merely what class they belong to.
• Harmonious spatial-spectral duality: Coarse-to-fine spatial structures (radical \(\rightarrow\) component \(\rightarrow\) stroke) mirror low-to-high frequency style dynamics (global posture \(\rightarrow\) mid-stroke continuity \(\rightarrow\) stroke tip curvature), leading to an elegant and unified representation.
• Generator-agnostic design: Condensing multi-scale style into condition vector \(Z\) allows it to drive both autoregressive trajectory prediction (Transformers with GMM) and continuous pixel diffusion (DDPM with U-Net cross-attention) without requiring bespoke architecture modifications.

Limitations & Future Work

• Handling cursive and running script distortions: The current architecture relies on standard ideographic description sequences (IDS). When confronted with extreme cursive handwriting (Cursive/Cao Shu) where strokes are severely merged, omitted, or deformed, rigid hierarchical matching can lead to alignment mismatches.
• Diminishing returns with reference count: Experimental results show performance plateaus when reference count \(k \ge 5\), leaving room to enhance robustness under extreme 1-shot constraints.
• Future Directions: Exploring differentiable soft stroke decomposition mechanisms to handle cursive script synthesis, and transferring the hierarchical formulation to ancient document restoration and degraded manuscript recognition.

Related Work & Insights

• vs SDT (CVPR 2023): SDT disentangles writer and character styles but relies on CNN-Transformer content classifiers. HiSAC reduces online DTW from 0.8780 to 0.8224 and raises Style Score from 94.56 to 97.01, demonstrating that explicit topological hierarchies provide far greater structural stability for unseen characters.
• vs DNA (WACV 2026): DNA introduces component-level information only as an auxiliary signal for content representation. HiSAC explicitly aligns style across radicals, components, and strokes, outperforming DNA across all online trajectory and fidelity metrics.
• vs DiffusionPen / One-DM (ECCV 2024): DiffusionPen and One-DM rely on global style vectors. HiSAC's multi-band FASE and local structural alignment achieve 34.88 FID and 68.21 Style Score in offline synthesis, outperforming DiffusionPen (35.14 FID, 61.78 Style).

Rating

• Novelty: ⭐⭐⭐⭐☆ [Novel dual spatial-frequency decomposition that bridges online trajectory modeling and offline diffusion synthesis]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive comparisons across online/offline benchmarks, detailed ablations, spectral visualizations, and cross-lingual zero-shot tests]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural taxonomy, precise mathematical formulations, and insightful qualitative analyses]
• Value: ⭐⭐⭐⭐☆ [Strong practical value for few-shot handwriting generation, font design, and OCR data augmentation]

Related Papers

• [ECCV 2026] Bridging Online and Offline Handwriting via Differentiable Physical Rendering
• [ICLR 2026] DiffInk: Glyph- and Style-Aware Latent Diffusion Transformer for Text to Online Handwriting Generation
• [ECCV 2026] UltraGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
• [ECCV 2026] Accelerated Likelihood Maximization for Diffusion-based Versatile Content Generation
• [ECCV 2026] Enlightening Photographic Style Transfer with a Self-Supervised Photographic Embedding