Back-Tracking from Clarity: Self-Learning to See Text from Afar¶

Conference: ECCV 2026
Paper: Official ECCV page
Full Text: Official PDF
Code: https://github.com/trid2912/BackTrackingText
Area: Object Detection (Scene Text Detection and Recognition)
Keywords: Distant text, back-tracking, carrier tracking, core polygon, pseudo-label self-learning

TL;DR¶

The method propagates reliable detections of clear text backward into earlier, degraded video frames, combining point tracking with carrier geometry to train an unchanged single-frame student; its SceneText50 tracking AUC reaches 44.71 versus 38.65 for CoTracker3.

Background & Motivation¶

Scene text detection locates text, while recognition determines what it says; text spotting combines the two. As a camera approaches a sign, its lettering usually progresses from a small, blurred patch to a readable instance. A system that succeeds only at the latter stage can still provide information too late for navigation. The missing capability is not necessarily knowledge of the characters, but reliable supervision for their distant appearances. Shrinking clear images and adding blur only approximates the perspective, background interference, and motion encountered in real footage.

Video offers a different source of supervision: a confidently detected word was often already present in earlier frames. However, ordinary tracking becomes unreliable precisely where the additional labels are needed, because tiny characters lose distinctive texture. Detection-based association cannot recover instances that the detector never finds. This paper separates the source of the target identity from the source of stable geometry: readable text supplies the former, while its signboard or other carrier supplies the latter. The task is visual text detection and recognition, not translation; multilingual footage does not make the method a multilingual machine-translation contribution.

Using later frames is legitimate for offline training-label generation, but not for claiming causal early recognition at deployment. The authors therefore keep back-tracking and geometric estimation outside the deployed student. Core idea: use the geometric continuity of the object carrying readable text to transfer reliable supervision into earlier, unclear views, then teach a single-frame model to detect and read those difficult instances sooner.

Method¶

Overall Architecture¶

Training consumes unlabeled videos with substantial text-scale changes and an existing reliable text detector, producing pseudo-labels and a fine-tuned student. The pipeline proceeds through Clear Detections and Point Back-Tracking, Carrier Geometry Back-Tracking, and Pseudo-Label Fusion and Self-Learning. DeepSolo supplies detections, CoTracker3 propagates text boundary points, and SAM-2 supports carrier tracking; core polygons turn carrier masks into geometric correspondences. At inference, the student receives only the current image and predicts text locations, plus recognized text when the chosen architecture supports spotting.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unlabeled video"] --> B["Clear Detections and<br/>Point Back-Tracking"]
    B --> C["Carrier Geometry<br/>Back-Tracking"]
    C --> D["Pseudo-Label Fusion<br/>and Self-Learning"]
    D --> E["Single-frame student<br/>Text locations and recognition"]

Key Designs¶

1. Clear Detections and Point Back-Tracking: supervise difficult views from reliable later observations

Instead of forcing the teacher to guess on degraded text, the pipeline begins with large, clear instances that it already handles well. DeepSolo supplies these text regions, and the system propagates their boundary points backward through the video. Section 3.2 specifies CoTracker3 for point tracking and retains results only when their confidence exceeds 0.8. This constructs supervision across different appearances of the same real instance, rather than treating a resized crop as a complete model of distance degradation. The resulting early-frame examples can include actual perspective changes, background clutter, and motion blur.

Point tracking remains useful while local features are sufficiently distinctive, but character boundaries can drift into the background when text becomes tiny. Confidence filtering rejects some unreliable trajectories without solving the resulting shortage of difficult examples. The carrier branch provides an additional source of geometric information rather than replacing point tracking wholesale. The label-generation description refers to further carrier back-tracking for instances reaching the final frame, and the ablation discussion describes the carrier as a fallback when point tracking fails. The main text does not fully specify a frame-by-frame switching state machine, so precise retry counts or failure thresholds cannot be reconstructed from it.

2. Carrier Geometry Back-Tracking: recover the text box from a more stable supporting object

A carrier is the object on which text is printed, such as a signboard or vehicle panel, not simply an enlarged text crop. The detected text center becomes a single-point prompt for SAM-2, which produces corresponding carrier masks in neighboring frames. The larger object can retain recognizable boundaries after individual characters lose useful texture. However, SAM-2 does not directly supply correspondences between the corners of the text quadrilaterals. Knowing that two masks belong to the same object does not yet determine how the lettering rotates, scales, or changes perspective within that object.

Core polygons convert these free-form masks into a more consistent geometric representation. A raw convex hull may change its number of vertices between frames, making correspondence unstable under small boundary perturbations. The proposed construction uses predefined edge orientations, a vertex-count constraint, and a circumscribing fit to obtain comparable boundaries. Its intuitive goal is stable topology with a tight approximation; the formal definition permits no more than the specified number of vertices and constrains angles except at vertices coinciding with the mask's convex hull. It is not equivalent to arbitrarily sampling a fixed number of contour pixels, nor does it recover dense correspondence throughout the carrier.

The method estimates a homography between consecutive core polygons using RANSAC, then projects the known text quadrilateral into the earlier frame. It transfers the text region's geometric relationship to its carrier rather than labeling the whole carrier as text. RANSAC reduces sensitivity to unreliable matches, whereas the core polygon improves the representation from which matches are established. The paper measures approximation looseness as the circumscribing polygon's perimeter divided by the convex polygon's perimeter, and discusses convergence of that ratio toward 1 as the vertex count increases. This concerns the mask's convex hull, not exact recovery of an arbitrary concave mask or a three-dimensional surface. Theorem 1's bound is corrupted in the local text extraction, so this note does not reconstruct the equation; the mechanism above is independently supported by Section 3.1's prose.

3. Pseudo-Label Fusion and Self-Learning: preserve direct observations and supplement missed instances

Independent detections and propagated regions can overlap within a frame, producing conflicting supervision if all are retained indiscriminately. The paper prioritizes detection masks and adds a back-tracked mask only when it does not significantly overlap an existing detection. Tracking consequently serves mainly to fill gaps rather than overwrite a reliable current-frame observation. The main text does not give a numerical threshold for significant overlap, so no fixed IoU constant is assumed here. Label quality comes from confidence filtering, geometric propagation, and this priority rule together, not simply from generating more boxes.

The student freezes its image encoder, updates decoder modules, and uses an exponential moving average (EMA) during training. The contribution is therefore not a heavier online video network but a different source of supervision for an existing detector. Future frames, CoTracker3, SAM-2, core polygons, and RANSAC are used only in offline label generation. The trained student operates independently on each frame without those modules. The absence of additional inference overhead is relative to the base student architecture and excludes video collection, offline tracking, and fine-tuning costs. Likewise, annotation-free adaptation does not mean the system contains no prior human supervision: the teacher and general-purpose models are already pretrained.

A Worked Example¶

Consider approaching a shop sign that is readable late in a clip but appears as a small blurred region near the beginning; this is an illustrative walkthrough, not an additional measured example. A confident late-frame detection initializes backward point tracking, with unreliable results removed by the confidence condition. When character points lack stable texture, the carrier branch uses the text-center prompt to track the signboard. Core polygons parameterize its masks, and the estimated homography maps the known text box into an earlier image. If a reliable direct detection already covers that region, detection takes priority; otherwise an eligible propagated label can fill the gap. After training on such examples, the student predicts directly from new early frames without waiting for the text to approach the camera. Thus, future-assisted training and causal deployment are compatible, provided evaluation images are not used to generate training supervision.

Loss & Training¶

Section 3.2 reports AdamW, weight decay \(10^{-4}\), EMA decay 0.999, batch size 8, and one NVIDIA RTX A6000 with 48GB memory. The main paper does not fully enumerate detection and recognition loss combinations, learning rates, training duration, or filtering of transcription pseudo-labels, so no unified loss is invented here. SceneText50 comprises 841 clips sampled from 50 long videos, with 67,371 annotated frames and 14,263 text trajectories according to Section 4. Clips from each source video are divided chronologically: approximately the first two thirds are used for training and the final third for testing. Human annotations support evaluation rather than the self-learning training targets. This models adaptation within a broad environment, not generalization under a source-video-, city-, or country-disjoint split.

Key Experimental Results¶

Main Results¶

The following selection from Table 2 reports SceneText50 backward text tracking, not end-to-end recognition accuracy. Trackers receive the ground-truth box in the final frame and proceed backward; all displayed metrics are higher-is-better. AUC integrates the success curve over IoU thresholds from 0 to 1, [email protected] measures the fraction of frames above 0.5 IoU, and P@20 measures the fraction with center error within 20 pixels.

Method	AUC ↑	[email protected] ↑	P@20 ↑
MixFormer	37.77	38.39	66.48
SAM-2	26.43	22.38	56.09
CoTracker3	38.65	44.01	57.02
Proposed full method	44.71	49.23	74.62

The method improves AUC over CoTracker3 by 6.06 percentage points and P@20 over MixFormer, the strongest displayed baseline on that metric, by 8.14 points. Ground-truth initialization is cleaner than the teacher predictions used for actual label generation, so these numbers are not direct measurements of overall pseudo-label accuracy. Table 2 does not explicitly state whether tracking evaluation is restricted to Section 4's test clips; the exact evaluation partition requires checking the implementation.

Ablation Study¶

Table 4 compares point-only tracking, carrier-only tracking, and their combination under the same metric definitions. The table does not separately identify its data partition; its full-method row matches the SceneText50 row in Table 2, which does not establish an additional source-video-disjoint test.

Tracking configuration	AUC ↑	[email protected] ↑	P@20 ↑
Point-based only	38.62	44.23	57.44
Carrier-based only	34.68	36.00	57.66
Full combination	44.71	49.23	74.62

Carrier-only AUC is lower than point-only AUC, supporting complementarity rather than universal replacement of text-point tracking. The full combination gains 6.09 AUC points and 17.18 P@20 points over the point-only configuration. Table 4's point-tracking row differs slightly from Table 2's CoTracker3 row; both are preserved as reported rather than silently merged into one baseline.

Key Findings¶

In Table 3, DeepSolo recall is 40.27 with tracked labels versus 37.75 with synthetic augmentation, while precision decreases slightly from 52.25 to 51.88. Better recall does not imply improvement on every metric.
The augmentation baseline samples ArTVideo and DSText frames, halves their size, and applies Gaussian blur, producing 1,000 images. The main text does not explicitly guarantee matched data volume and complete training budgets across strategies.
Section 5.3 reports a plateau beyond 16 core-polygon vertices in the carrier-only setting. This does not prove a uniquely optimal vertex count for the full pipeline.
Figure 7 reports a better false-alarm-rate versus average-delay trade-off, but the local text lacks reliably readable curve coordinates. No exact frame or second reduction is claimed here.
Detection delay counts frames from an instance's first appearance to its first correct detection, penalizing missed instances with maximum delay; FAR is the fraction of declarations that are incorrect. Both are lower-is-better, and spotting additionally requires correct recognition; neither is per-frame computational latency.

Highlights & Insights¶

Supervision moves from an easy moment to a difficult moment, rather than only from a clean image to an artificially degraded one. Real temporal trajectories become a source of small-text training examples.
Text establishes target identity while carrier structure stabilizes location. Separating the semantic target from the object supplying motion cues is more distinctive than simply attaching a generic tracker.
Core polygons address unstable cross-frame correspondence, not merely mask compression. Their combination with robust geometric estimation targets a bottleneck that stronger segmentation alone does not resolve.
Additional complexity stays on the training side. This is useful when deployment requires independent single-frame predictions without an online tracking stack.

Limitations & Future Work¶

The authors explicitly restrict current collection to Latin-script text despite covering multiple countries and languages. Non-Latin generalization is an extension direction, not an established experimental result.
The paper discusses point drift and degraded appearance but has no complete standalone limitations section. The following scope considerations are reader assessments.
The method requires text to become sufficiently clear later and its carrier to remain trackable. Persistent blur, substantial occlusion, or incorrect carrier segmentation can still prevent reliable supervision.
Homography transfer assumes sufficiently consistent local geometry. Nonplanar carriers, deformation, or text moving relative to the carrier may violate that assumption, and the paper does not quantify these failure categories.
Chronological splitting within source videos suits environmental adaptation but does not establish equal gains in wholly unfamiliar environments. Source-video-disjoint and cross-script tests would better delimit generalization.
A corrupted theorem equation, incomplete training settings, and partially specified label-filtering details limit reproduction from the main text alone. The code URL is stated in the paper but was not checked online.

vs DeepSolo: DeepSolo supplies reliable clear-text predictions and a trainable spotting architecture. This work primarily changes supervision instead of replacing the full inference architecture.
vs CoTracker3: Point tracking can accurately propagate local positions but struggles when character texture vanishes. Carrier context supplements these difficult cases rather than making point tracking obsolete.
vs SAM-2: SAM-2 provides carrier segmentation and temporal association; core polygons and RANSAC bridge that object-level association to text-box geometry. The direct SAM-2 baseline does not contain this bridge.
vs synthetic small-text augmentation: Resizing and blurring are cheaper, while back-tracking captures real degradation trajectories. The existing comparison does not fully disentangle data source, scale, and label-generation strategy.

Rating¶

Novelty: 4/5. Clear-to-degraded supervision and carrier geometry form a coherent contribution, although foundation components and pseudo-label self-training have precedents.
Experimental Thoroughness: 3/5. Multiple trackers, detector architectures, and genuine ablations are included, but budget matching, cross-domain isolation, and reproducibility details remain limited.
Writing Quality: 3/5. The main mechanism is understandable, while some algorithmic interfaces are underspecified and the locally extracted theorem cannot be verified directly.
Value: 4/5. The approach addresses a practical distant-text supervision problem without adding a tracker to student inference.