TextDS: Parameter-Efficient Representation Alignment for Scene Text Detection under Distribution Shifts

Conference: ECCV2026
arXiv: 2606.28077
Code: https://github.com/ZChenDang/TextDS
Area: Scene Text Detection
Keywords: Scene Text Detection, Distribution Shift, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Representation Alignment

TL;DR

TextDS adopts a dual-encoder architecture featuring SAM2 and DINOv3, achieving robust scene text detection without large-scale scene text pre-training through Step-Wise Low-Rank Adaptation (SWLoRA) and Common Subspace Fusion (CSF). With only 4.9M parameters, it achieves leading performance under domain shifts and image degradations such as rain, fog, exposure, and low resolution.

Background & Motivation

In practical deployment scenarios, scene text detectors inevitably suffer from distribution shifts, including cross-domain variations (different capturing scenarios, font languages, font styles) and imaging degradations (rain/fog, over/underexposure, low resolution). Most existing methods rely heavily on large-scale scene text pre-training datasets such as SynthText-800k/150k to boost in-domain performance, while the evaluation and optimization of robustness under distribution shifts remain understudied. When deployed in real-world environments, rain and fog reduce contrast, underexposure compresses dynamic range, overexposure erases stroke details, and low resolution loses fine textures. These degradations simultaneously undermine pixel-wise separability and geometric cues, making both thresholding/region-growing methods based on pixel clustering and boundary localization based on explicit structural modeling unstable.

The core insight of this paper is that compared to large-scale pre-training on scene text data, vision foundation models (such as SAM2 and DINOv3) already possess powerful and generic visual priors. By leveraging their complementary advantages—SAM2 providing structured multi-scale features and DINOv3 providing domain-robust semantic representations—and converting them into discriminative capabilities required by text detection through efficient adaptation and fusion mechanisms, one can bypass the heavy reliance on scene text pre-training while enhancing robustness under cross-domain generalization and degraded conditions. Core Idea: Building a SAM2-DINOv3 dual-encoder architecture that performs progressive fine-tuning at the SAM2 block level through Step-Wise Low-Rank Adaptation (SWLoRA), and fuses the two branches within a shared subspace through Common Subspace Fusion (CSF) while preserving the orthogonal complementary domain information of DINOv3, achieving robust scene text detection under distribution shifts with only 4.9M trainable parameters.

Method

Overall Architecture

The overall architecture of TextDS is a serial pipeline consisting of a dual-encoder, a fusion module, and a lightweight decoder. The input image is simultaneously fed into two encoding branches: SAM2-Hiera-L and DINOv3 ViT-L/16. The SAM2 branch inserts SWLoRA modules before each Hiera block stage for parameter-efficient fine-tuning, outputting a 4-scale feature pyramid (\(S_1 \sim S_4\)). The DINOv3 branch extracts a single-scale semantic feature \(D\) at a fixed \(448 \times 448\) resolution, which is then aligned to the four scales of SAM2 via four \(1 \times 1\) convolutions and bilinear interpolation. The aligned feature pairs \((S_k, D_k)\) at each scale are fused via the CSF module to output \(F_k\). The fused features of the four scales are fed into a lightweight decoder to output the text probability map.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B["SAM2-Hiera-L<br/>Encoder (with SWLoRA)"]
    A --> C["DINOv3 ViT-L/16<br/>Encoder"]

    B --> D["4-scale Features<br/>S₁ / S₂ / S₃ / S₄"]
    C --> E["Single-scale Feature D"]
    E --> F["1×1 Conv + Interpolation<br/>Aligned to 4 Scales"]

    D --> G["CSF Common Subspace Fusion<br/>×4 Scales"]
    F --> G

    G --> H["Lightweight Decoder"]
    H --> I["Text Probability Map"]

Key Designs

1. SAM2-DINOv3 Dual-Encoder: Replacing Scene Text Pre-training with Complementary Vision Foundation Models

Existing scene text detection methods invariably rely on large-scale synthetic datasets such as SynthText for pre-training to acquire text-aware representations. The proposed approach diverges from this paradigm: the SAM2-Hiera-L encoder retains only the image encoding backbone (excluding the prompt encoder and mask decoder), which naturally outputs a 4-scale feature pyramid (with sequentially decreasing resolutions and channel dimensions of \(144 \to 288 \to 576 \to 1152\)), providing multi-layer structural information ranging from high-resolution stroke/boundary details to low-resolution global layout priors. DINOv3 ViT-L/16 extracts a \(28 \times 28\) single-scale semantic feature from a fixed \(448 \times 448\) input, learning representations robust to domain shifts via self-distillation. To match the multi-scale structure of SAM2, the DINOv3 features are aligned to each SAM2 scale by four independent \(1 \times 1\) convolutions (compressing the 1024-dimensional feature to the respective channel size of each scale) followed by bilinear interpolation. The original weights of both encoders are frozen throughout, with only a small number of parameters in SWLoRA and CSF participating in training — which is key to achieving a mere 4.9M trainable parameters.

2. SWLoRA: Step-Wise Low-Rank Adaptation and Dynamic Early Stopping via Cosine Similarity

Scene text detection has diverse adaptation requirements across different scales: high-resolution layers require fine adjustment of boundaries and strokes, middle layers regulate instance connectivity and deformation consistency, and low-resolution layers primarily contribute to global layout priors and background suppression. The core idea of SWLoRA is to perform an iterative low-rank refinement of up to \(T=5\) steps on each frozen Hiera block, superimposing a LoRA-style residual at each step:

\[x^{(t+1)} = x^{(t)} + \gamma_{t}\,\frac{\alpha}{r}\,W_{up}\big(\text{Dropout}(\text{LN}(W_{down}(x^{(t)})))\big)\]

where \(\gamma_t\) is a learnable step-size scaling factor, \(r\) is the LoRA rank (default to 8), and \(\alpha\) is a scaling constant. Each step is equivalent to applying a low-rank directional correction on the current representation. After refining to the \(t\)-th step, the cosine similarity between the representations of two consecutive steps is calculated:

\[\cos^{(t)} = \frac{\langle \text{vec}(x^{(t+1)}), \text{vec}(x^{(t)}) \rangle}{\|\text{vec}(x^{(t+1)})\|_2 \|\text{vec}(x^{(t)})\|_2 + \varepsilon}\]

If \(\cos^{(t)} > \tau\) (early stopping threshold) and at least 1 step of minimum update has been performed, the refinement terminates, and the current state is fed into the frozen Hiera block. This dynamic early stopping mechanism allows hard samples and blocks requiring heavier adjustments to undergo more refinement steps, while simple samples exit quickly. The average step counts on CTW-1500, Total-Text, and MLT are 3.65, 2.20, and 3.48 respectively (out of a max of 5), corresponding to savings of 27%, 56%, and 30% in fine-tuning computation, striking a balance between accuracy and efficiency.

3. CSF: Common Subspace Fusion and Orthogonal Complement Domain Preservation

The key bottleneck in fusing dual-encoder features is that the two branches contain both shared information (co-attended text regions) and complementary information (structural details from SAM2 vs. domain-robust semantics from DINOv3). Naive concatenation or weighted summation mixes noises, making it difficult to distinguish which information is shared and which is unique to each branch. CSF addresses this challenge at the level of second-order statistics.

First, the feature pair \((S_k, D_k)\) at each scale is projected to a common channel dimension \(C\) via \(1 \times 1\) convolutions. Then, the concatenated global average pooling features are passed through an MLP + Sigmoid to derive adaptive gating weights \(\alpha\), generating the intermediate mixed feature \(M = \alpha \odot \hat{S} + (1-\alpha) \odot \hat{D}\).

Second, \(\hat{S}\) and \(\hat{D}\) are flattened and centered to compute their respective covariance matrices, yielding \(A = \Sigma_S \Sigma_D\). Eigenvalue decomposition is performed on \(A\) to obtain the top \(r=16\) eigenvectors, forming the orthogonal basis \(Q\) of the common subspace. This subspace captures the statistically shared variation patterns between the two branches.

Third (and most elegantly), the cross-branch interaction is restricted within this common subspace, while explicitly preserving the orthogonal complementary domain information of DINOv3:

\[F_k = \mathcal{P}_k(M_k) + (\hat{D}_k - \mathcal{P}_k(\hat{D}_k))\]

where \(\mathcal{P}_k(X) = Q_k(Q_k^\top X)\) denotes the projection operation onto the common subspace. The first term projects and fuses the mixed feature \(M\) within the common subspace, ensuring that the two branches interact only along statistically shared dimensions. The second term preserves the components in \(\hat{D}\) that do not belong to the common subspace intact, preventing the fusion process from accidentally discarding the domain-robust information unique to DINOv3. This "fuse but preserve" design is the core difference between CSF and simple feature concatenation or attention-based fusion.

Loss & Training

The training loss is the sum of binary cross-entropy loss and IoU loss with equal weights, aiming to optimize both pixel-level classification accuracy and boundary overlap quality. It employs the AdamW optimizer with an initial learning rate of 0.001, weight decay of \(5\times10^{-4}\) and a cosine annealing scheduler down to \(1\times10^{-7}\). The model is trained for 50 epochs with a batch size of 4 on a single RTX 4090.

Key Experimental Results

Main Results

TextDS is evaluated against representative recent methods on three standard scene text detection datasets. While leading comprehensively in in-domain testing, it requires only 4.9M trainable parameters (1/5 to 1/8 of existing methods), eliminates the need for SynthText pre-training, and achieves an inference speed of 44.1 FPS.

Dataset	Metric	DB-Net	TextBPN	LRANet	S3INet	TextDS
CTW-1500	F-measure	83.4	85.0	87.4	86.0	90.6
Total-Text	F-measure	84.7	87.9	87.7	88.7	89.1
MLT	F-measure	85.6	87.4	88.3	89.7	92.2
-	Params	28.0M	38.7M	34.3M	38.0M	4.9M
-	FPS	35.0	22.6	38.1	37.3	44.1

On the degraded variant datasets constructed in this paper, TextDS demonstrates exceptional robustness to degradation, with F-measure dropping by only 0.4~1.7 percentage points under various conditions:

Condition	CTW-1500	Total-Text	MLT
Normal	90.6	89.1	92.2
Rain	89.2	87.7	91.6
Fog	89.8	88.3	91.9
Underexposed	90.1	87.8	92.1
Overexposed	89.9	87.9	91.8
Low-Res 256	89.8	88.3	90.9
Low-Res 128	88.9	88.0	88.3

Ablation Study

Configuration	CTW-1500	Total-Text	MLT	Description
Baseline (Single Encoder)	82.2	85.1	86.4	No dual encoders, no adaptation, no fusion
+ Dual Encoder	85.3	86.0	88.1	Introduces DINOv3 branch
+ SWLoRA	87.9	88.3	90.3	Step-wise low-rank adaptation
+ CSF (w/o SWLoRA)	88.2	87.8	91.0	Common subspace fusion
Full Model	90.6	89.1	92.2	All modules

Key Findings

• CSF Contributes Most on MLT: In the MLT multi-lingual cross-domain scenario, adding CSF alone yields a 4.6 percentage point improvement, indicating that common subspace fusion is highly critical for cross-domain semantic alignment.
• Complementarity of SWLoRA and CSF: Applying them individually brings gains of 1.9~2.6 and 1.9~6.0 points respectively (varying across datasets). Combining both yields further improvements, proving that structural adaptation and feature fusion address issues at different levels.
• High Efficiency of Early Stopping: The average execution steps of SWLoRA are significantly lower than the maximum 5 steps, effectively saving fine-tuning computations without degrading accuracy.
• Balanced Degradation Robustness: The performance degradation across different degradation types is closely clustered (1~2 percentage points), demonstrating that the dual-encoder + CSF design is defensive against all types of degradations instead of being specifically optimized for a single one.

Highlights & Insights

• De-pretraining Paradigm: The most elegant aspect of this work is bypassing SynthText pre-training by deploying a complementary combination of generic vision foundation models directly for scene text detection. This is vastly more efficient than training from scratch on synthetic text data and delivers superior generalization. This paradigm can be generalized to other fine-grained vision tasks requiring domain robustness, such as document analysis and remote sensing object detection.
• CSF's "Fuse-but-Preserve" Design: The common subspace projection guarantees that the two branches interact only along statistically shared dimensions, while the orthogonal complementary domain preservation ensures that DINOv3's unique domain-robust representations are not washed away by the fusion operation. This "partial fusion + partial preservation" paradigm can be transferred to any feature fusion scenario invoking heterogeneous encoders.
• Sample-Adaptive Depth of SWLoRA: Utilizing cosine similarity for early stopping is naturally motivated—since not every block or sample requires the same level of fine-tuning—but the multi-step iterative LoRA design is quite novel among LoRA variants, and is organically coupled with the scale-dependent nature of scene text detection.

Limitations & Future Work

• The degraded datasets constructed in the paper are generated via synthetic degradation processes. Real-world degradations under complex conditions (such as lens distortion, motion blur, and compression artifacts) are not yet covered. Semantic generalization from synthetic to real degradations warrants further validation.
• Although the dual-encoder design is parameter-efficient, both encoders require a forward pass during inference, leading to higher actual computational costs compared to end-to-end single-encoder alternatives. While the paper reports 44.1 FPS, the detailed breakdown of inference time between the two encoders is not analyzed.
• SWLoRA is applied solely to the Hiera blocks of the SAM2 encoder and is absent on the DINOv3 side. Whether DINOv3 features can be further refined with similar adapters for better dual-branch alignment remains an open question worth exploring.
• The rank of the common subspace is fixed to \(r=16\) as a default setting across all experiments, and the paper does not discuss whether different datasets require different optimal ranks.

Related Work & Insights

• vs. Traditional Text Detection Methods (DB-Net, TextBPN, LRANet, S3INet, etc.): These methods consistently rely on large-scale SynthText pre-training and are evaluated primarily under ideal conditions. TextDS charts a different path by substituting domain-specific pre-training with generic foundation models and extending the evaluation to distribution shift scenarios. Leading comprehensively on three standard datasets validates the feasibility of this "de-pretraining" approach in the text detection field.
• vs. LLM-Based OCR Solutions (Qwen-OCR, DeepSeek-OCR, PP-OCR, etc.): LLM-based approaches excel in semantic understanding but lack precise pixel-level geometric localization. TextDS, acting as a front-end detector, provides high-quality text regions, thereby complementing LLM-OCR. The paper also demonstrates a visual comparison highlighting this complementarity.

Rating

• Novelty: ⭐⭐⭐⭐⭐ The concept of replacing scene text pre-training with a combination of generic foundation models is highly novel in the text detection domain. The design motivations and implementations of SWLoRA and CSF are distinctive.
• Experimental Thoroughness: ⭐⭐⭐⭐ 3 standard datasets + (5 degradation conditions \(\times\) 3 datasets) = 15 groups of systematic experiments, with a clear ablation study. However, the degradation datasets are synthetically generated rather than real-world datasets.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivations, detailed methodology descriptions, well-organized equations and diagrams, and high information-density comparison tables.
• Value: ⭐⭐⭐⭐⭐ Strong practicality with no scene text pre-training required, only 4.9M parameters, and robustness to degradation, offering insightful contributions to the text detection field.

Related Papers

• [ICCV 2025] Gradient Short-Circuit: Efficient Out-of-Distribution Detection via Feature Intervention
• [ACL 2026] SRA: Span Representation Alignment for Large Language Model Distillation
• [AAAI 2026] From Parameter to Representation: A Closed-Form Approach for Controllable Model Merging
• [ACL 2025] Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis
• [CVPR 2026] Frequency Switching Mechanism for Parameter-Efficient Multi-Task Learning