Bridging Online and Offline Handwriting via Differentiable Physical Rendering¶

Conference: ECCV 2026
Paper: ECCV Official
Project: https://seonmip.github.io/onoff/
Area: Image Generation
Keywords: handwriting generation / differentiable rendering / physical brush modeling / online-offline handwriting unification / robotic calligraphy

TL;DR¶

This paper introduces a compact six-parameter physical brush model and an analytical differentiable renderer that bridge online stroke kinematics and offline pixel appearance, enabling joint dual-modal handwriting generation and zero-shot robotic calligraphy execution.

Background & Motivation¶

Realistic handwritten text generation is foundational across practical applications including personalized font creation, synthetic training data generation for optical character recognition (OCR), assistive writing systems for individuals with motor impairments, and robotic calligraphy in embodied AI. Historically, research in this space has been divided into two disconnected paradigms: online handwriting generation and offline handwriting generation. Online methods predict sequential trajectories composed of coordinate streams and pen-up/down states over time; while they inherently capture temporal kinematics and provide directly executable motion paths, they omit fine-grained appearance details such as stroke width variations, ink bleeding, and tactile surface textures. In contrast, offline methods directly synthesize static pixel-level images, accurately reproducing rich stylistic textures while discarding temporal stroke order and dynamic writing processes, rendering them inapplicable to physical robotic execution.

Jointly modeling online trajectories and offline images offers strong theoretical and practical advantages, but it has long been hindered by two fundamental bottlenecks. First, there is an absence of an explicit physical formulation connecting stroke kinematics to pixel-level brush marks, leaving the cross-modal mapping severely under-constrained. Second, real-world datasets providing accurately aligned high-resolution offline scans paired with sequential ground-truth stroke trajectories simply do not exist. While classical computer graphics systems simulate 3D bristle dynamics and ink diffusion with high fidelity, their non-differentiable formulations and heavy computational costs prevent gradient backpropagation across kinematic and visual domains.

To resolve the discrepancy between writing dynamics and visual texture, this paper argues that rather than deploying complex full-scale physical fluid simulations, one can distill the most salient geometric and optical factors into a low-dimensional, fully differentiable analytical surrogate. Core idea: propose a compact six-parameter physical brush model and a differentiable renderer that maps online stroke dynamics into offline images, paired with a style parameter observer and a zero-shot diffusion enhancer to establish a unified online–offline handwriting generation pipeline.

Method¶

Overall Architecture¶

Conditioned on style reference images and target text strings, the framework simultaneously outputs physically executable sequential stroke trajectories, analytical rendered handwriting images, and texture-enhanced offline scans. The system consists of four coordinated modules: the brush parameter observer $O$ infers six physical brush attributes from style exemplars; the online stroke generator $G$ predicts multi-character word-level trajectories; the differentiable brush renderer $\mathcal{R}$ rasterizes stroke sequences into stroke-width-aware images; and the zero-shot image enhancer $E$ utilizes a frozen pretrained handwriting diffusion model to introduce realistic paper textures and sensor noise.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: Target Text String + Style Reference Images"] --> O["1. Style-Conditioned Brush Parameter Observer<br/>DINOv3 backbone predicts 6-parameter physical brush model"]
    In --> G["2. Word-Level Online Stroke Generator<br/>Multi-character embeddings & cross-attention for trajectory prediction"]
    O --> R["3. Compact Differentiable Brush Renderer<br/>Velocity-derived pressure proxy + footprint falloff + max composition"]
    G --> R
    G --> OutTraj["Output: Online Executable Trajectory<br/>(Cartesian path for robotic manipulator)"]
    R --> OutRend["Output: Differentiable Rendered Handwriting Image"]
    OutRend --> E["4. Prerender-Guided Zero-Shot Diffusion Enhancer<br/>Intermediate latent prior injection adds natural paper textures"]
    E --> OutOff["Output: High-Fidelity Offline Handwriting Image"]

Key Designs¶

1. Compact Differentiable Brush Renderer: Establishing an Analytical Bridge between Kinematics and Appearance

To overcome the non-differentiability and high latency of classical graphics simulations, the physical interaction between brush bristles and canvas is parameterized into six core factors: $\theta = \{w_{\mathrm{base}}, k_{\mathrm{spread}}, \rho_{\mathrm{ink}}, \sigma_{\mathrm{sharp}}, p_{\mathrm{min}}, p_{\mathrm{max}}\}$. Because online datasets lack explicit pen pressure measurements, the framework estimates an instantaneous pressure proxy by leveraging the inverse relationship between speed and pressure: $v_t = \|\mathbf{u}_{t+1} - \mathbf{u}_t\|_2$, followed by range clipping via tool-specific bounds $[p_{\mathrm{min}}, p_{\mathrm{max}}]$ and temporal exponential smoothing. The effective stroke width dynamically scales with pressure: $$w_t = w_{\mathrm{base}} k_{\mathrm{spread}} p_t$$ For each stroke segment, normalized distance to the centerline combines with sharpness falloff $\gamma$ to define the footprint kernel $K_t(q)$, while transferred opacity obeys the exponential law $O_t = 1 - \exp(-\rho_{\mathrm{ink}} p_t)$. To avoid unnatural darkening at self-overlapping intersections, the renderer implements a differentiable maximum composition over segments: $\mathcal{A}(q) = \max_t \alpha_t(q)$, yielding a continuous grayscale image $I_{\mathrm{rend}} = 1 - \mathcal{A}$.

2. Word-Level Online Stroke Generator: Trajectory Modeling Beyond Single Characters

Previous online stroke models primarily operate on isolated characters, failing to handle continuous cursive ligatures and inter-character spacing across complete words. This method extends transformer-based stroke generation to the word level by replacing isolated character tokens with character embedding sequences extracted from the target text string. Cross-attention layers are incorporated into the autoregressive decoder, allowing the sequential trajectory representations to attend across the entire word sequence dynamically. At each timestep, the generator predicts a 20-component 2D Gaussian mixture for pen coordinate increments alongside discrete pen-state probabilities, capturing structural cursive dynamics end-to-end.

3. Style-Conditioned Brush Parameter Observer: Unsupervised Appearance-to-Physics Inversion

Extracting physical parameters from unconstrained style scans requires effective disentanglement of writer-specific ink deposition. The proposed pipeline creates a synthetic paired dataset $\mathcal{D}_{\mathrm{syn}} = \{(T, S, I_{\mathrm{syn}}, \theta)\}$ containing 155,840 word samples by rendering online trajectories (IAM-OnDB, CASIA) under randomly sampled brush parameters and compositing them onto authentic background patches (cardboard, wood, notebooks). Supervised by this synthetic ground truth, the observer $O$ deploys a DINOv3 backbone to tokenize style images, Prepends a learnable style token, and integrates intra-image self-attention with inter-image cross-attention to output normalized parameter predictions $\theta$ via an MLP head.

4. Prerender-Guided Zero-Shot Diffusion Enhancer: Preserving Geometry while Restoring Photorealism

Although analytically rendered images exhibit accurate geometric outlines, they lack scanner noise, local ink saturation anomalies, and paper grain. Instead of fine-tuning heavyweight diffusion networks, the pipeline utilizes an off-the-shelf handwriting diffusion model in a zero-shot manner. The rendered image $I_{\mathrm{rend}}$ is encoded into latent space via a VAE encoder: $z_{\mathrm{rend}} = \mathcal{E}(I_{\mathrm{rend}})$. At a designated intermediate timestep $t_0$, Gaussian noise is injected into $z_{\mathrm{rend}}$: $$z_{t_0} = \sqrt{\bar{\alpha}_{t_0}} z_{\mathrm{rend}} + \sqrt{1 - \bar{\alpha}_{t_0}} \epsilon, \quad \epsilon \sim \mathcal{N}(0, \mathbf{I})$$ Sampling starts from $z_{t_0}$ rather than standard pure Gaussian noise, which anchors the reverse diffusion trajectory to the strict structural geometry of the rendered stroke while letting diffusion priors synthesize micro-textures and background paper characteristics.

Loss & Training¶

The stroke generator $G$ is trained autoregressively using negative log-likelihood on continuous pen coordinates paired with cross-entropy on pen states: $$\mathcal{L}_G = -\sum_{t} \log p(\mathbf{u}_t) + \lambda_{\mathrm{pen}} \sum_{t} \mathrm{CE}(\hat{\mathbf{m}}_t, \mathbf{m}_t)$$ The brush parameter observer $O$ optimizes parameter regression MSE alongside pixel reconstruction error backpropagated through $\mathcal{R}$: $$\mathcal{L} = \mathrm{MSE}(\theta, \hat{\theta}) + \mathrm{MSE}(I_{\mathrm{rend}}, \hat{I}_{\mathrm{rend}})$$ During diffusion enhancement, noise-injection start steps $t_0$ are tuned to balance structural faithfulness and texture synthesis (e.g., $t_0=50$ for DiffPen+Ours; $t_0=200/900$ for One-DM+Ours).

Key Experimental Results¶

Main Results¶

Online stroke quality is evaluated on IAM-OnDB and CASIA using Dynamic Time Warping (DTW) to quantify temporal alignment error. Offline image synthesis is benchmarked on IAM Words and CVL Words across FID, Background-FID (BFID), Handwriting Distance (HWD), Character Error Rate (CER), and LPIPS.

Task / Dataset	Metric	Ours / One-DM+Ours	Prev. SOTA	Relative Gain
Online Trajectory (Multi-letter)	DTW ↓	0.2936	0.8155 (SDT)	Alignment error reduced by 64.0%
Online Trajectory (Single-letter)	DTW ↓	0.3462	0.6056 (SDT)	Alignment error reduced by 42.8%
Offline IAM Words	FID ↓	25.89	29.00 (One-DM)	Enhanced image fidelity
Offline IAM Words	BFID ↓	8.15	16.35 (One-DM)	Background artifact reduction by 50.2%
Offline IAM Words	HWD ↓	1.6442	1.8306 (One-DM)	Improved style preservation
Offline CVL Words	FID ↓	14.45	19.45 (One-DM)	Cross-dataset generalization boost
Offline CVL Words	BFID ↓	10.47	15.18 (One-DM)	Background fidelity improved
Offline CVL Words	HWD ↓	1.5264	2.1500 (One-DM)	Best style consistency score

Ablation Study¶

The ablation evaluates baseline offline generators against the synthetic paired benchmark to inspect their grasp of physical stroke dynamics, alongside standalone differentiable renderer performance:

Configuration / Method	FID ↓	BFID ↓	HWD ↓	CER ↓	LPIPS ↓	Note
VATr++ (Pure offline baseline)	74.28	33.44	2.5990	0.329	0.622	Ignores motion dynamics; produces artifacts
DiffPen (Diffusion baseline)	89.06	118.04	2.0030	0.496	0.602	Lacks explicit structural skeleton
One-DM (Competitive diffusion)	52.26	19.88	2.3650	0.229	0.539	High recognition but detached from stroke trajectory
Emuru (Autoregressive baseline)	89.97	60.88	2.6220	6.126	0.583	Severe sequence divergence
Our Renderer (Direct analytical output)	11.81	11.03	1.0833	0.361	0.001	Exceptional fidelity without diffusion refinement

Key Findings¶

Explicitly grounding online trajectories with differentiable brush geometry provides immense structural stability: even without zero-shot diffusion enhancement, the analytical renderer achieves 11.81 FID and 1.0833 HWD on the paired evaluation set, outperforming black-box offline baselines.
Prerender guidance reliably boosts existing generative models: initializing DiffPen and One-DM with rendered intermediates consistently drops FID and BFID, raising average user preference ranking from 3.406 to 2.840 for One-DM.
Physical deployment on a 7-DoF robotic manipulator (xArm7) verifies that 2D trajectories directly translate to planar Cartesian tool paths, while predicted pressure proxies map linearly to vertical $z$-axis contact depth, producing authentic calligraphy across pencils, markers, and pens without post-hoc engineering.

Highlights & Insights¶

Analytical Approximation over Heavy Simulation: A low-dimensional six-parameter formulation successfully replicates visual stroke physics with sub-millisecond differentiable execution, avoiding intractable fluid-dynamics compute.
Synthetic Cross-Modal Bootstrapping: Combining analytical rendering with random background textures generates over 150k aligned pairs from trajectory-only data, breaking the dependency on unavailable real-world paired datasets.
Prerender-Guided Latent Initialization: Inserting deterministic geometric priors into intermediate diffusion steps eliminates character hallucinations and spelling distortions without costly model retraining.

Limitations & Future Work¶

Uncalibrated Physical Units: Predicted brush attributes serve as visual surrogate parameters rather than standardized SI physical metrics (e.g., Newtons of normal force), requiring empirical scaling for robotic hardware.
Word-Level Architectural Boundary: The present framework is optimized for word-level generation; scaling to full-page layouts with margin control, paragraph slant, and long-range line spacing warrants further exploration.

vs SDT [CVPR 2023]: SDT disentangles character and writer style for online handwriting but is constrained to isolated single characters and lacks raster rendering; this work supports variable-length word trajectories and full offline image generation.
vs One-DM [ECCV 2024]: One-DM synthesizes high-quality offline text images via diffusion but discards temporal stroke dynamics entirely, forbidding physical actuation; this method provides verified robotic trajectories and further improves One-DM's visual realism when paired.
vs Classical Graphics Models (e.g., Hairy Brushes [1986], Chu & Tai [2002]): Graphics formulations capture bristle mechanics but cannot backpropagate gradients; this paper distills them into a differentiable pipeline that integrates seamlessly with modern neural networks.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates the first differentiable physical brush bridge connecting online trajectories and offline handwriting.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning online DTW metrics, offline image baselines, ablation studies, and physical robotic arm deployment.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, crisp mathematical derivations, and transparent narrative flow.
Value: ⭐⭐⭐⭐⭐ Unlocks new possibilities for digital document restoration, customized typography, and physical robotic handwriting.