TiCRL: Textual Image Classification with Reinforcement Learning-Based Curriculum Learning¶
Conference: ECCV 2026
Paper: ECCV 2026
Area: Reinforcement Learning
Keywords: textual image classification, curriculum learning, deep reinforcement learning, DDPG, document understanding
TL;DR¶
TiCRL introduces an OCR-free visual-textual hybrid difficulty measurer and a continuous-action DDPG scheduler that dynamically adjusts sample difficulty thresholds using real-time training feedback, achieving 83.67% accuracy on CNN with 82% data and 93.02% accuracy on DiT with only 58% data.
Background & Motivation¶
As digital transformation accelerates across global enterprises, vast volumes of paper documents are systematically converted into digital formats. Automated document image classification serves as a crucial cornerstone for downstream information retrieval, text recognition, and automated document analysis. However, real-world document images exhibit extreme structural diversity, spanning from highly rigid, tabular contracts to unstructured, free-form promotional flyers, brochures, and casual correspondence. Traditional convolutional neural networks (CNNs) perform reasonably well on standardized formats, but frequently suffer from severe overfitting or catastrophic generalization collapse on irregular, visually complex layouts. In response, modern solutions increasingly turn to massive multimodal foundation models (such as LayoutLMv3 and UDOP) or heavy OCR pre-processing pipelines, which incur significant inference latency and high computational overhead.
Addressing structural layout diversity without modifying backbone architectures or demanding expensive multi-modal inputs presents an urgent technical challenge. Curriculum learning (CL) mimics human educational paradigms by progressively scheduling training samples from easy to hard, naturally aligning with document datasets characterized by varying visual formality and text densities. Nonetheless, existing CL paradigms face two critical bottlenecks in document analysis: first, conventional difficulty metrics either rely on static heuristic priors or tie themselves to model-dependent loss/gradient signals, necessitating costly recomputation across different architectures; second, standard training schedulers adhere to rigid mathematical pacing functions (such as linear, root, or step schedules) or self-paced learning (SPL), completely ignoring the learner's evolving convergence state and generalization shifts during training.
This paper tackles the challenge by decoupling difficulty estimation from downstream classifiers and reformulating curriculum scheduling as a continuous-action reinforcement learning problem. Core idea: combine an OCR-free hybrid difficulty measurer integrating hand-crafted visual statistics with frozen representation loss, and deploy a learner-aware DDPG scheduler to continuously modulate sample truncation thresholds based on real-time feedback, simultaneously optimizing classification accuracy and data efficiency.
Method¶
Overall Architecture¶
The TiCRL framework decouples curriculum learning into two standalone, coordinated modules: an architecture-agnostic Difficulty Measurer and a learner-aware RL-based Training Scheduler. During offline preprocessing, the difficulty measurer computes an OCR-free, normalized difficulty score \(\in [0, 1]\) for every training document, combining hand-crafted statistical features with a fixed representation loss from an auxiliary network; this score is computed only once and shared across all target backbones. Training then proceeds through two distinct phases: in the Agent Training Stage, a DDPG agent interacts across cycles with a re-initializable classifier, observing an 8-dimensional state vector of training/validation metrics, issuing continuous adjustments \(a_t\) to shift the difficulty threshold \(d_t\), and updating its actor-critic networks using relative loss improvements as reward; in the Curriculum Application Stage, the optimal frozen DDPG policy schedules sample exposure dynamically for the target classifier (either a lightweight CNN trained from scratch or a pretrained Transformer during fine-tuning).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Document Image Dataset"] --> B["Stage 1: OCR-Free Hybrid Difficulty Measurement<br/>5 visual-textual features + frozen ResNet loss"]
B --> C["Stage 2: MDP Formulation & State Perception<br/>8D classifier state st mapped to continuous action at"]
C --> D["Stage 3: DDPG Dynamic Threshold Scheduling<br/>Continuous threshold modulation dt ∈ [0.001, 1.0]"]
D --> E["Stage 4: Adaptive Data Filtering & Policy Transfer<br/>Filter subset with difficulty ≤ dt+1 for classifier"]
E --> F["Output High-Accuracy & Data-Efficient Classifier"]
Key Designs¶
1. OCR-Free Visual-Textual Hybrid Difficulty Measurer: Decoupled and Cross-Architecture Reusable Metric
Traditional text-aware document understanding relies on computationally demanding OCR pipelines, whereas conventional curriculum metrics depend on dynamic classifier loss, requiring complete recalculation whenever backbones or hyperparameters change. TiCRL resolves this dilemma via a four-step hybrid scoring pipeline. Step 1 extracts five complementary structural and statistical descriptors without OCR: Canny edge pixel count captures text density and structural line segments; Histogram of Oriented Gradients (HOG) average gradient magnitude reflects text orientation and layout layout patterns; Gray-Level Co-occurrence Matrix (GLCM) contrast measures local intensity variations; GLCM homogeneity measures texture uniformity; and the cosine similarity between ResNet-50 image embeddings and class representative mean vectors quantifies typicality within each class. Step 2 normalizes these five features and weights them using a weight vector sampled from a symmetric Dirichlet distribution \(\text{Dir}([1, 1, 1, 1, 1])\), providing an unbiased prior across the feature space before min-max scaling to \([0, 1]\). Step 3 introduces model-based prediction difficulty using an ImageNet-pretrained ResNet-18 with a randomly initialized 16-class classification head, executing a single forward pass without backpropagation to obtain cross-entropy loss. Step 4 converts both the visual feature score and the cross-entropy loss into percentile ranks and computes their arithmetic mean, yielding a robust sample difficulty score. This metric is computed once and reused across all architectures (CNN, DiT, UDOP, and LayoutLMv3).
2. MDP Formulation with Multi-Dimensional State Feedback: Sensing Real-Time Learner Dynamics
Predefined pacing functions (such as linear or root schedules) are blind to the learner's actual grasp of the material, forcefully introducing complex samples even when the model experiences convergence stalls. TiCRL formulates curriculum scheduling as a Markov Decision Process (MDP) \(\mathcal{M} = (S, \mathcal{A}, \mathcal{R}, \mathcal{T}, \gamma)\). The state vector \(s_t \in \mathbb{R}^8\) captures comprehensive classifier status: current difficulty threshold \(d_t\), threshold change \(\Delta d_t\), episode index \(e_t\), training loss \(L_{\text{train}}\), validation loss \(L_{\text{val}}\), training accuracy \(\text{Acc}_{\text{train}}\), validation accuracy \(\text{Acc}_{\text{val}}\), and a clipping flag indicating boundary saturation. The action space is formulated as a 1D continuous scalar \(a_t \in [-0.01, 0.01]\), which smoothly shifts the upper bound of sample difficulty for the subsequent training episode: $$ d_{t+1} = \text{clip}(d_t + a_t, 0.001, 1.0) $$ A positive action (\(a_t > 0\)) raises the threshold to introduce harder, less structured documents, while a negative action (\(a_t < 0\)) tightens the bound, steering the classifier back to structured, standard layouts to consolidate representation.
3. Progress-Adaptive Dual-Objective Reward: Balancing Early Convergence and Late Generalization
To ensure the agent prioritizes fast convergence initially while safeguarding against overfitting later, the reward function \(r_t\) dynamically blends training and validation loss improvements: $$ r_t = (1 - \lambda_t) \cdot \max\left(0, \frac{L_{\text{train}}^{\text{prev}} - L_{\text{train}}}{L_{\text{train}}^{\text{prev}} + \epsilon}\right) + \lambda_t \cdot \max\left(0, \frac{L_{\text{val}}^{\text{prev}} - L_{\text{val}}}{L_{\text{val}}^{\text{prev}} + \epsilon}\right) $$ where \(\epsilon = 10^{-6}\) prevents zero-division and \(\lambda_t \in [0, 1]\) increases linearly with training progress \(p_t\) (the cumulative fraction of training data exposed up to the current episode). During early training (\(p_t\) small, \(\lambda_t \to 0\)), the reward is driven primarily by training loss reductions on easy samples, establishing robust feature extractors; as training advances (\(p_t\) large, \(\lambda_t \to 1\)), validation loss improvement becomes the dominant reward source, compelling the agent to strategically expose complex layouts to optimize out-of-distribution generalization. Crucially, validation loss informs only the agent's reward and never directly updates classifier weights.
4. Two-Stage Decoupled DDPG Scheduling with Zero-Shot Cross-Domain Transfer
TiCRL strictly decouples policy acquisition from policy deployment. In the Agent Training Stage, a continuous DDPG actor-critic framework incorporates Ornstein-Uhlenbeck (OU) correlated exploration noise (\(\theta=0.2, \sigma=0.8\)) and target action clipping noise (adapted from TD3) to stabilize critic updates, with target networks maintained via soft updates (\(\tau=0.01\)). Across multiple training cycles, the classifier weights are re-initialized so the scheduler learns robust navigation policies across varying learning conditions. In the Curriculum Application Stage, the optimal DDPG policy is frozen and deployed as an inference scheduler. Because the 8D state vector relies solely on model-agnostic loss and accuracy statistics, the learned scheduling policy transfers zero-shot to completely unseen document datasets and diverse architectures without retraining.
Loss & Training¶
The classifier is trained on the selected subset \(\mathcal{D}_t = \{x_i \in \mathcal{D}_{\text{train}} \mid \text{Score}(x_i) \le d_{t+1}\}\) using standard cross-entropy loss for \(u\) epochs per episode. The DDPG Critic network optimizes the Bellman error: $$ \mathcal{L}{\text{critic}} = \mathbb{E}')\right)\right)^2\right] $$ with target action smoothing noise }\left[\left(Q(s, a) - \left(r + \gamma Q_{\text{target}}(s', \tilde{a\(\tilde{a}' = \mu_{\text{target}}(s') + \text{clip}(\mathcal{N}(0, 0.2), -0.5, 0.5)\). Actor and Critic networks are optimized using Adam with learning rate \(3 \times 10^{-4}\). Early stopping is enforced within each cycle when validation loss fails to improve for 5 consecutive evaluations. By tracking policy-level signals—specifically actor loss minimization and Q-value stabilization—the generalization-optimal agent (Episode 379 in Cycle 6) is selected for all downstream inference.
Key Experimental Results¶
Main Results¶
Experiments were conducted on the RVL-CDIP benchmark (a balanced 100,000-image subset across 16 document classes split 64:16:20 into train/val/test), resized to \(224 \times 224\). The evaluation compares lightweight CNN baselines (~1.35M parameters, trained from scratch) and pretrained Transformer models (DiT, UDOP, LayoutLMv3 fine-tuning).
Table 1: Performance comparison of curriculum strategies on CNN (Mean ± Std over 5 independent runs)
| Method Type | Method | Difficulty Source | Strategy | Accuracy (%) | Data Used (Count / Ratio) |
|---|---|---|---|---|---|
| w/o Curriculum (100% data) | CNN (1.35M) | – | None | 80.43 ± 0.33 | 64,000 (100.0%) |
| w/o Curriculum (100% data) | ResNet-34 | – | None | 78.48 ± 0.39 | 64,000 (100.0%) |
| w/o Curriculum (80% data) | CNN (1.35M) | – | None | 78.69 ± 0.46 | 51,200 (80.0%) |
| w/o Curriculum (80% data) | ResNet-34 | – | None | 77.68 ± 0.24 | 51,200 (80.0%) |
| Pre-defined CL | Linear | Score | Linear fn. | 74.16 ± 1.43 | 48,106 (75.17%) |
| Pre-defined CL | Step | Score | Step fn. | 74.22 ± 0.98 | 48,528 (75.83%) |
| Pre-defined CL | Root | Score | Root fn. | 76.96 ± 0.75 | 53,435 (83.49%) |
| Automated CL | SPL | Loss | Self-paced | 69.89 ± 1.82 | 48,212 (75.33%) |
| Automated CL | ACL | Model+Loss | Pacing fn. | 75.84 ± 0.48 | 46,416 (72.53%) |
| Proposed Method | TiCRL (Ours) | Hybrid Score | DDPG RL | 83.67 ± 1.61 | 52,429 (81.92%) |
Table 2: Fine-tuning performance comparison on document Transformers
| Model | Fine-tuning Epochs | Schedule | Accuracy (%) | Data Used (%) |
|---|---|---|---|---|
| DiT | 150 | Standard (80% data) | 91.07 ± 0.35 | 80.00% |
| UDOP | 15 | Standard (80% data) | 74.00 ± 0.95 | 80.00% |
| LayoutLMv3 | 10 | Standard (80% data) | 91.53 ± 0.26 | 80.00% |
| DiT + TiCRL | 150 | TiCRL Dynamic | 93.02 ± 2.05 | 57.66% |
Ablation Study¶
A component-wise leave-one-out ablation on the hybrid difficulty measurer was conducted under identical CNN configurations on a representative seed.
Table 3: Ablation of the hybrid difficulty measurer components
| Component Removed | Accuracy (%) | Drop (%p) | Note |
|---|---|---|---|
| None (Full TiCRL) | 85.05 | – | Complete 5-feature hybrid scoring + ResNet-18 loss |
| w/o Canny Edge | 81.40 | -3.65 | Removes text density and line edge statistics |
| w/o HOG | 81.63 | -3.42 | Removes gradient orientation and layout directionality |
| w/o GLCM Contrast | 81.07 | -3.98 | Largest degradation; confirms critical role of local contrast |
| w/o GLCM Homogeneity | 82.35 | -2.70 | Removes local texture uniformity cues |
| w/o Cosine Similarity | 81.23 | -3.82 | Removes class-center prototype proximity prior |
Additional ablations on scheduler design: 1. Action Space Dimensionality: A 2D-action agent controlling both initial threshold and data sampling ratio achieved only \(72.30 \pm 10.18\%\) with excessive variance, validating the stability of 1D continuous threshold modulation. 2. Scheduler Heuristics: An EMA-based adaptive heuristic baseline achieved 81.49% accuracy (surpassing 80.43% no-CL baseline), yet fell noticeably short of TiCRL's 83.67%, proving the superiority of RL multi-signal feedback. 3. Agent Selection Criterion: Comparing the generalization-optimal Best Agent against the Final Agent on a 10K small test split showed 80.48% vs. 61.99% (an 18.49%p gap), confirming that policy-level early stopping prevents policy overfitting.
Key Findings¶
- Non-Monotonic Scheduling Dynamic: Rather than strictly increasing the threshold, the DDPG agent frequently contracts the difficulty boundary upon encountering plateaued validation metrics, temporarily shielding the classifier from noise before expanding the difficulty window again.
- Substantial Gains on Hard Categories: On the top-3 hardest classes (advertisements 84.6%, news articles 83.7%, scientific publications 88.6%), TiCRL achieved an average accuracy of 85.6%, far exceeding predefined CL (69.1%) and automated CL (77.1%), while surpassing full-data ResNet-34 using only 82% of the training data.
- Seamless Zero-Shot Cross-Dataset Transfer: When transferred directly to Tobacco-3482 (3,482 images, 10 classes) without retraining, the RVL-CDIP-trained agent attained 86.88% accuracy vs. 85.30% for standard training, highlighting domain-invariant policy generalization.
Highlights & Insights¶
- OCR-Free Hybrid Difficulty Estimation: Elegant integration of classical visual statistics (Canny, HOG, GLCM) with frozen deep representations provides a rich, text-independent structural complexity score that is computed once and shared across all architectures.
- Progress-Adaptive Dual-Reward Objective: Formulating reward annealing to balance early training convergence against late-stage generalization provides an effective, generalizable reward design for continuous curriculum control.
- Diagnostic Identification of Policy Overfitting: Uncovering that RL schedulers suffer from policy overfitting in later cycles and demonstrating that tracking actor loss minima and Q-value stability enables optimal agent selection provides critical practical guidelines for RL-based training automation.
Limitations & Future Work¶
- Limitations Acknowledged by Authors: The framework has been primarily evaluated on single-page 2D document classification; extensions to dense layout analysis (detection/segmentation) or multi-page documents remain unverified.
- Identified Limitations: Using a randomly initialized classification head on ResNet-18 introduces slight stochastic variance in the initial loss scores; resizing documents to \(224 \times 224\) inevitably blurs fine font-level distinctions, biasing the difficulty score toward coarse layout structures.
- Future Directions: Exploring policy-driven continuous sampling probability distributions instead of hard truncation thresholds, and extending TiCRL to data mixture scheduling for large vision-language model instruction tuning.
Related Work & Insights¶
- vs. Pre-defined CL (Linear, Step, Root): Static schedules rigidly follow fixed pacing functions and remain oblivious to the learner's real-time convergence state; TiCRL introduces dynamic closed-loop RL control to adaptively raise or lower difficulty thresholds.
- vs. Self-Paced Learning (SPL) & Adaptive CL (ACL): SPL and ACL tie difficulty assessment to the active learner's loss or a heavy teacher model, requiring costly recomputations whenever architectures change; TiCRL completely decouples difficulty scoring from the classifier.
- vs. Static Coreset Selection (CRAIG, EL2N): Coreset techniques select a fixed data subset prior to training, leaving sample ordering unmanaged; TiCRL optimizes the temporal sequence and exposure timing of data, maximizing sample efficiency.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Decouples OCR-free structural difficulty scoring from continuous DDPG curriculum scheduling.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across CNN from-scratch training, three Transformer models, feature ablations, and zero-shot cross-dataset transfer.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous MDP formulation, clear training dynamics analysis, and insightful policy early-stopping criteria.
- Value: ⭐⭐⭐⭐☆ Highly practical and data-efficient training acceleration framework for document intelligence systems in resource-constrained environments.