ClearText-Video: A Large-Scale Text-Centric Video Dataset Bridging Video Restoration and Scene-Text Enhancement¶
Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/jinlong17/CTVid-Bench
Area: Video Understanding
Keywords: Scene text, video restoration, paired quality evaluation, spatiotemporal QA, confidence calibration
TL;DR¶
ClearText-Video (CTVid) connects low-level restoration with text-grounded video QA through high-quality, degraded, and restored versions of the same content, using 4,639 videos to show that sharper-looking inputs need not be read more accurately and that in-dataset fine-tuning can improve accuracy while reducing an uncertainty-aware metric.
Background & Motivation¶
Text in real-world video is both image texture and semantic evidence that must not be rewritten. Adding sharper natural textures can improve the appearance of a super-resolved image, but inventing a character stroke can change a road sign, business name, or instruction. Conventional restoration leaderboards emphasize pixel error, perceptual similarity, and naturalness. These measurements do not establish whether restored text retains its meaning, much less whether a downstream model can answer questions correctly.
Meanwhile, EgoTextVQA, NewsVideoQA, and ViTXT-GQA have advanced scene-text evaluation, video reasoning, and spatiotemporal grounding, but generally do not systematically provide multiple quality versions of identical content. Testing clean and blurry inputs on different videos confounds quality with scene composition, question difficulty, and text content. CTVid therefore contributes a controlled data resource built from high-quality capture, human text annotation, reproducible degradation, and a shared QA protocol, rather than a new restoration network.
Core idea: keep video content, text ground truth, and question-answer pairs fixed while changing input quality, then jointly measure visual appearance, textual fidelity, and spatiotemporal QA to test whether restoration preserves the evidence needed for reasoning.
Method¶
Overall Architecture¶
CTVid first constructs high-quality videos with frame-level text boxes, transcripts, and captions. It generates low-resolution or blurry versions of the same content and applies existing restoration models to obtain restored inputs. A parallel annotation branch builds spatial and temporal questions from the shared text ground truth, allowing the same QA set to be evaluated on HQ (high-quality), DQ (degraded-quality), and RQ (restored-quality) videos.
The diagram describes data construction and evaluation, not a neural architecture proposed by the paper. Human-verified annotations support both question construction and reference-based restoration evaluation: a restoration model cannot hide a changed character by changing the expected answer.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Text-rich videos"] --> B["Human-verified<br/>text ground truth"]
B --> C["Content-matched<br/>quality variants"]
B --> D["Spatial and<br/>temporal QA"]
C --> E["Joint fidelity and<br/>reliability evaluation"]
D --> E
E --> F["Restoration metrics<br/>and QA performance"]
Key Designs¶
1. Human-verified text ground truth: separate OCR proposals from evaluation targets
The authors primarily capture videos with a Sony A7R V and a 28–70 mm f/3.5–5.6 FE lens, adding 15% online text-centric footage for diversity. Each 10-second recording is temporally center-cropped to a 4-second clip, with 120 frames extracted for annotation. Starting from more than 6.4K candidate videos, 16 annotators perform two rounds of annotation and verification. The final resource contains 4,639 HQ videos, 550K+ frames, 1.6M text instances, and 220K+ question-answer pairs, split into 4,327 training and 312 testing videos.
PaddleOCR supplies coarse detection candidates only; annotators refine boxes in Labelme and verify transcripts. They subsequently review the alignment among clips, boxes, text, and QA pairs, remove ambiguous questions, correct answers, and screen privacy-sensitive content. Thus, the later OCR evaluation does not simply use that same recognizer's outputs as ground truth. The test set contains 74% offline-captured and 26% online clips, with balanced English/Chinese coverage within each source type and an easy/medium/hard question distribution of 46%/37%/17%. These test-set proportions should not be confused with the online-source proportion of the full collection.
2. Content-matched quality variants: make quality a separately comparable variable
HQ videos provide the high-quality reference, listed at 1920×1080 resolution in Table 1. Super-resolution inputs are generated by 4-fold bicubic downsampling. Image SR restores individual frames, whereas video SR can exploit neighboring frames. Frame-level captions are available; methods requiring a video-level semantic condition can use the middle frame's caption. This is an available conditioning option, not evidence that every evaluated baseline actually uses captions.
Deblurring inputs are not synthesized by merely applying a static blur kernel. RIFE first performs 16-fold interpolation, increasing 30 fps clips to 480 fps, after which every 65 frames are temporally fused to emulate camera or object motion during exposure. The cached paper does not provide complete fusion weights, window strides, or boundary-handling equations, so this should not be rewritten as a specific convolution implementation. Spatial QA uses six inputs: HQ, DQ-Low_res, DQ-Blur, and RQ outputs from DOVE, MIMO, and S3Diff. DOVE and S3Diff restore low-resolution inputs; MIMO restores blur. Restoration gains must therefore be compared with the corresponding DQ source, not calculated across unrelated degradation types.
3. Spatial and temporal QA: read the text and recover its relationship to the video
Spatial questions start from metadata describing text content, position, scene, and attributes. A rule-based partitioner analyzes text layout and chooses cuts that isolate meaningful text groups. GPT-4o helps propose question types and natural-language formulations, while a rule-based pipeline derives answers from recorded text and inserts the relevant locations. A validation module checks spatial correctness, structure, and answerability. Human reviewers then ensure that answering requires spatial relationships rather than simply matching strings in a transcript list. Question formats include multiple choice, true/false, and fill-in-the-blank.
Temporal questions do not rely on a language model to guess trajectories. Each text instance is tracked across the 120 frames through its position, area, speed, and acceleration. Deterministic rules generate questions in five categories: visibility, spatial localization over time, motion and trajectory shape, size and scale changes, and boundary interactions. Humans retain meaningful questions and refine their wording. The spatial and temporal branches share text annotations but test different abilities: the former requires associating text with regions, whereas the latter requires connecting the same text instance across frames. Producing an OCR transcript alone is consequently not equivalent to solving the task.
4. Joint fidelity and reliability evaluation: distinguish restoration, recognition, and reasoning success
Restoration is evaluated with reference-based metrics including PSNR, SSIM, LPIPS, DISTS, and FID, alongside no-reference metrics such as NIQE, MANIQA, MUSIQ, and CLIPIQA. A fixed PaddleOCR pipeline probes textual quality. Detection metrics comprise precision, recall, and their harmonic mean F1; recognition metrics comprise accuracy, Accrec, and normalized edit distance, NEDrec. The original table explicitly marks NEDrec as lower-is-better, but the cache does not specify its normalization denominator, which is not reconstructed here. Fixing the OCR probe supports repeatable method comparisons without eliminating the recognizer's own language and font biases.
Spatial QA reports accuracy (Acc), uncertainty-aware accuracy (UAcc), over-confidence ratio (OC), and abstention rate (Abs). Abs is the fraction of queries on which the model withholds an answer. UAcc and OC combine correctness with confidence; the paper references prior definitions but does not expand their thresholds or equations in the cache. UAcc should therefore not be casually redefined as accuracy with incorrect answers subtracted. The temporal table reports only Acc across five quality conditions, with no separate DQ-Low_res column. All temporal conditions are uniformly downsampled by a factor of 2 to fit memory constraints, while frame sampling is minimized where possible to preserve continuity. Spatial and temporal scores are not directly subtractable measurements under an identical protocol.
Loss & Training¶
The paper introduces no new restoration loss. Its supervised fine-tuning experiment uses Qwen2.5-VL-7B, the CTVid training split, and the official implementation's default configuration. System prompts are loaded dynamically according to question type. The vision encoder and multimodal projector are frozen to preserve pretrained visual representations, and training runs for 10 epochs on 8 NVIDIA A100 GPUs.
The cache does not list the learning rate, batch size, optimizer details, or an explicit objective equation; a guessed configuration should not be substituted for “official defaults.” This experiment tests whether in-dataset instruction tuning improves QA, not whether the visual encoder learns a new restoration capability, nor whether the improvement generalizes to other datasets.
Key Experimental Results¶
Main Results¶
The paper reports 18 restoration methods and 16 spatial QA model configurations; the restoration tables additionally contain versions trained on different datasets. The following selection from original Table 2 compares super-resolution methods only. PSNR is in dB, and detection/recognition columns are percentages. Deblurring results are not mixed into this comparison.
| SR method | Type | PSNR ↑ | F1det ↑ | Accrec ↑ | NEDrec ↓ |
|---|---|---|---|---|---|
| SwinIR | Image | 28.18 | 70.50 | 36.76 | 54.96 |
| S3Diff | Image | 26.05 | 77.17 | 38.42 | 58.03 |
| RealBasicVSR | Video | 28.26 | 69.23 | 41.97 | 59.30 |
| DOVE | Video | 30.24 | 74.53 | 44.72 | 63.14 |
S3Diff improves detection F1 and recognition accuracy over SwinIR without improving PSNR or lowering NEDrec. DOVE leads the displayed methods in PSNR and Accrec, but that does not make it the leader on every text metric. Some metric names in the paper's narrative are inconsistent with the original table. This note follows the table headers and row values rather than repeating broad claims that one method wins every textual metric.
The next table selects rows from original Table 4. All entries are spatial QA Acc (%), and the RQ columns retain their restoration source so they can be compared with the appropriate DQ input.
| Model | HQ | DQ-Low_res | DQ-Blur | RQ-DOVE | RQ-MIMO | RQ-S3DIFF |
|---|---|---|---|---|---|---|
| Gemini-2.5-pro | 71.67 | 65.00 | 60.00 | 65.00 | 63.33 | 70.00 |
| Claude-Sonnet-4.6 | 70.00 | 63.33 | 60.00 | 65.00 | 70.00 | 61.67 |
| Qwen3-VL-8B | 50.56 | 45.43 | 44.46 | 47.70 | 46.91 | 46.64 |
| Qwen2.5-VL-7B | 49.99 | 40.61 | 42.16 | 45.37 | 44.85 | 45.00 |
| Qwen2.5-VL-7B-SFT | 58.43 | 50.72 | 49.79 | 53.21 | 52.89 | 51.31 |
For Claude, restoring low-resolution inputs with S3Diff reduces accuracy from 63.33% to 61.67%. Yet restoring blurry inputs with MIMO increases the same model's accuracy from 60.00% to 70.00%. These source-matched comparisons show more directly than absolute restored-input scores that the benefit depends on the degradation, restoration method, and downstream model together.
Ablation Study¶
This benchmark paper has no newly proposed network modules to remove. The most direct training control compares the same Qwen2.5-VL-7B backbone with and without CTVid SFT. The following values are the HQ condition from original Table 4, all in percent.
| Configuration | Acc ↑ | OC ↓ | UAcc ↑ | Abs |
|---|---|---|---|---|
| Qwen2.5-VL-7B, without CTVid SFT | 49.99 | 21.36 | 60.66 | 0.17 |
| Qwen2.5-VL-7B, with CTVid SFT | 58.43 | 1.73 | 43.71 | 0.08 |
Fine-tuning increases HQ Acc by 8.44 percentage points and reduces OC by 19.63 points, while UAcc decreases by 16.95 points. Under low resolution, Acc rises from 40.61% to 50.72%, a gain of 10.11 points. The comparison supports effective in-dataset adaptation, not a blanket improvement in reliability. Without the metric equations and confidence distributions, these aggregate values do not establish the specific mechanism behind the calibration trade-off.
Key Findings¶
- Averaged over the 16 spatial QA configurations, the paper reports a 3.14-point accuracy reduction from HQ under low resolution and a 6.05-point reduction under blur. This is an average, not a universal per-model relationship: Qwen2.5-VL-7B performs worse under low resolution than under blur.
- In original Table 5, Claude-Sonnet-4.6 reaches 60.02% temporal QA accuracy on HQ and 60.37% on RQ-DOVE; Kimi-VL-16B obtains 53.95% and 54.18%, respectively. Readable individual frames do not make trajectory and visibility reasoning straightforward.
- The authors report that OCR+LLM is substantially worse than direct multimodal reasoning, but the cached main text provides no numerical table for this control. The finding remains qualitative here, without an invented gap or a claim that every possible OCR architecture has been ruled out.
Highlights & Insights¶
- Make semantic damage observable. Content-matched inputs reveal whether restoration changes characters instead of confusing scene difficulty with restoration benefit. The same evaluation principle is relevant to receipts and instrument readings, where semantic rewriting is unacceptable.
- Textual fidelity and visual quality are different targets. The super-resolution table gives different method rankings across metrics. Sharpness, natural appearance, and character correctness should be measured separately rather than substituting a perceptual score for reading success.
- Validate adaptation and reliability separately. SFT delivers a real accuracy gain while UAcc decreases. A deployed assistant should be assessed on incorrect answers, confidence, and abstention as well as its headline accuracy.
Limitations & Future Work¶
- The authors explicitly acknowledge limited degradation coverage. Bicubic downsampling and synthetic motion blur do not systematically cover exposure changes, rolling shutter, sensor noise, ISP effects, or complex compression. The results cannot represent every real user video.
- Temporal and language coverage remain limited. Four-second clips and English/Chinese coverage control annotation cost but do not establish performance on long occlusions, cross-shot text association, or broader writing systems. These are inferred scope limits of the data design.
- Statistical and implementation reporting needs expansion. The cache does not show confidence intervals, detailed proprietary-model evaluation sample counts, or full UAcc/OC implementations, and some narrative metric references are inconsistent. Small leaderboard differences should not be overinterpreted.
- An OCR probe is not a human-reading upper bound. Multiple recognizers and human character checks could distinguish restoration failure from recognition failure. The OCR+LLM control would also benefit from published input formats, layout information, and per-question outputs.
- Release promises and current availability are distinct. The paper says splits, annotations, scripts, prompts, and baseline outputs will be released. Self-captured clips use a research license; restricted online clips are redistributed only where permitted, otherwise with source and segment information. This note does not verify the repository's current release status online.
Related Work & Insights¶
- Compared with EgoTextVQA, NewsVideoQA, and ViTXT-GQA: These resources emphasize egocentric scene text, news-video reading, or spatiotemporal grounding. CTVid adds controlled quality variation for identical text-grounded questions rather than replacing their entire task coverage.
- Compared with Real-CE: Real-CE focuses on Chinese/English scene-text image restoration. CTVid adds temporal relationships and downstream QA, making it possible to test whether character recovery translates into semantic benefits.
- Compared with DOVE, S3Diff, and MIMO-UNet+: These are evaluated restoration methods or RQ generators, not modules introduced by CTVid. The benchmark could support text-aware restoration objectives, but jointly training fidelity and QA models is a future direction, not an algorithm demonstrated in this paper.
Rating¶
- Novelty: 4/5. The contribution is the content-matched multi-quality protocol and cross-level evaluation, not a new model architecture.
- Experimental Thoroughness: 4/5. Restoration, spatial QA, temporal QA, and fine-tuning controls are covered, but statistical reporting and numerical diagnostics remain limited.
- Writing Quality: 3/5. The research question is clear, while some narrative metric references conflict with tables and reliability definitions are not self-contained.
- Value: 4/5. Useful for testing restoration benefits in text-sensitive applications, with realistic degradations and release reproducibility still requiring further validation.