LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

Conference: ECCV 2026
arXiv: 2606.23539
Code: https://github.com/bokufa/LightSTAR
Area: Information Retrieval/RAG
Keywords: Visual Document Retrieval, Multimodal RAG, Candidate Selection, Visual Reranking, Efficient Retrieval

TL;DR

To address the issue where MLLMs are too slow to encode all pages in visual document retrieval, LightSTAR proposes first filtering out candidates with high recall using an LLM-free lightweight visual selection module, and then performing vision-adaptive semantic refinement only on those candidates. This achieves 89.1 NDCG@5 on ViDoRe while reducing the end-to-end latency for 5,000 pages to 123.9s.

Background & Motivation

Background: Enterprise search, technical document Q&A, research assistants, and multimodal RAG increasingly need to locate relevant pages in long PDFs, scanned files, or tabular reports. Visual document retrieval directly treats pages as images, preserving layout, charts, tables, and font information that are easily lost by OCR. Methods like ColPali, ColQwen, and VisRAG-Ret have demonstrated that MLLMs with late interaction can significantly improve retrieval quality.

Limitations of Prior Work: The cost of strong retrieval quality is expensive all-page MLLM encoding. Long documents or large repositories may contain thousands to tens of thousands of pages, while only a very small number of pages are truly relevant to a query. If every page passes through heavy vision-language models, indexing, online retrieval, and frequent updates will be bottlenecked by latency and GPU memory.

Key Challenge: Visual document retrieval simultaneously requires "fast enough coarse selection" and "visually layout-aware fine-grained reranking". Using only lightweight models like OCR/BM25 or CLIP is fast but misses layout, tables, and visual semantics. Full-scale MLLM fine-ranking yields high quality but wastes computation on a large number of clearly irrelevant pages.

Goal: Find a practical trade-off: enable lightweight models to perform high-recall candidate selection on the entire database to restrict expensive MLLM computations to top candidates, while maintaining a final ranking quality close to full-scale refinement.

Key Insight: The authors observe that real document retrieval queries are often keyword-anchored, containing entities, terminology, and attributes that directly appear in the visible text of relevant pages. This vocabulary is sufficient to eliminate a vast majority of irrelevant pages using cheap visual-textual signals.

Core Idea: LightSTAR splits retrieval into two stages: LLM-free Visual Selection, which scales down the candidate pool with high recall using content keywords and lightweight visual patch embeddings; and Vision-adaptive Semantic Refinement, which performs fine-grained semantic matching on candidate pages using MLLMs, enhanced by layout-adaptive feature fusion and hard negative training to improve reranking discriminative power.

Method

Overall Architecture

Given a query \(q\) and a document page image collection \(D=\{d_1,...,d_N\}\), LightSTAR does not run MLLM directly on all pages. Instead, it first scores the entire corpus using a lightweight text-aware visual encoder to obtain a candidate pool \(D_c\), where \(|D_c|\ll N\). Then, it calls the vision-language refinement model only on \(D_c\). The complexity is reduced from approximately \(O(N·C_{MLLM})\) to \(O(N·C_{VE}+|D_c|·C_{MLLM})\), where the lightweight visual encoding cost \(C_{VE}\) is much smaller than the MLLM cost.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text query + Document image collection"] --> B["Content-grounded query embedding<br/>Preserving visually visible content words"]
    B --> C["LLM-free Visual Selection<br/>Lightweight ViT patch embedding"]
    C --> D["Scale-adaptive late interaction<br/>Top-k patch aggregation scoring"]
    D --> E["High-recall candidate set D_c"]
    E --> F["Vision-adaptive Semantic Refinement<br/>Text-sensitive + Layout-sensitive feature fusion"]
    F --> G["Hardness-aware contrastive training<br/>Emphasizing similar candidate negative samples"]
    G --> H["Top-5 visual document retrieval results"]

Key Designs

1. Content-grounded query embedding: Translating queries into visually visible retrieval cues

On visual document pages, entities, terminology, numbers, actions, and key attributes are much easier to match quickly than functional words like "the", "of", and "and". LightSTAR uses a POS tagger to filter out conjunctions, prepositions, articles, auxiliary verbs, and punctuation, retaining only semantic content words as \(q'\), which are then mapped to token embeddings aligned with the visual encoder. This step is simple but solves the noise source in late interaction: function words can match similar patches on almost any page, diluting the contribution of real content words.

The goal is not language understanding but visual retrievability constraint. For instance, in the query "What is the amortization schedule of the loan?", the terms that really help locate the page in the document images are "amortization", "schedule", "loan". After discarding functional words, the lightweight visual selection module can more easily focus scores on relevant text regions rather than being distracted by frequent layout elements and background noise. Statistics in the appendix support this hypothesis: the vast majority of queries in various ViDoRe subsets have at least one keyword appearing in the ground-truth page.

2. Scale-adaptive late interaction: Substituting single-patch maximum with top-k average to secure coarse recall

The lightweight visual selection module segments each page image into patches. These patches are processed via InternViT-300M and a projection layer to obtain document patch embeddings \(D=[d_1,...,d_{n_d}]\), and query content words obtain \(Q=[q_1,...,q_{n_q}]\). Standard late interaction usually takes the maximum similarity of each query token across all document tokens. However, document images contain table lines, headers, similar numbers, and noisy patches, meaning a single accidentally high-scoring patch can easily push an irrelevant page to the top.

LightSTAR introduces scale-adaptive top-k aggregation: instead of the single maximum, each query token takes the average similarity of the \(k\) most similar patches, where \(k=max(1, floor(γ·n_d))\) scales dynamically with the page's patch count. Intuitively, complex, high-resolution pages require collective evidence from multiple local patches; simple pages maintain a smaller \(k\) to avoid over-smoothing. For training, an in-batch contrastive objective is utilized to ensure positive pages score higher than the hardest negative page in the batch. LoRA fine-tuning is applied only on the last ViT layer and projector, keeping the coarse stage LLM-free.

3. Vision-adaptive feature fusion: Sectional merging of text-sensitive and layout-sensitive features during candidate refinement

Once candidates are selected, the task becomes harder: candidate pages often share highly similar topics and visible keywords. The true differences might lie in table structures, coordinates, or layout relationships. LightSTAR's refinement is based on InternVL3, but it reuses the visual encoder from the selection stage to avoid introducing a separate visual front-end entirely. It employs a two-branch design on the last ViT layer: one branch inherits text-aware features well-suited for dense text regions; the other branch retains layout-aware features representing background structure, tables, charts, and whitespace.

The fusion strategy is highly pragmatic: the authors estimate which patches look like background or whitespaces using the similarity between text-aware features and a predefined whitespace token, obtaining a mask \(m\) after normalization and thresholding. The final features are formulated as:

\[F_{fused}=(1-m)·F_{text}+m·F_{layout}.\]

In other words, text regions retain the word-alignment capabilities learned during coarse screening, while background and layout areas are injected with the visual structural capabilities of the parent MLLM. This design avoids the extremes of ignoring layout (by using only text-aware) and losing content word alignment (by using only layout-aware).

4. Hardness-aware objective: Teaching the refinement model to distinguish similar candidates

After the LLM-free Visual Selection, the candidate set is no longer made of random negatives but a bunch of highly similar hard negatives. A standard InfoNCE objective weights all negatives similarly, wasting training budget on easily distinguishable pages. LightSTAR assigns a hardness weight to negative samples: the higher the current similarity between a negative sample and the query, the larger its weight. A stop-gradient is applied to prevent the model from taking advantage of the weight itself. Consequently, refinement training concentrates on local boundaries of confusing page pairs, learning finer-grained discriminative boundaries.

A Complete Example

Suppose a user is searching the term "renewable energy revenue table" among massive annual report pages. The first stage retains content words "renewable", "energy", "revenue", "table", performs lightweight patch matching over all pages, filters out those clearly lacking these words or visual regions, and retrieves the top-100 candidates. Among the candidates, multiple pages might contain "energy" and "revenue", but only one page has a table, while others might be body text paragraphs or charts. The second-stage refinement then examines both text semantics and layouts within these candidates: table regions are processed via layout-aware features, and cell texts via text-aware features. Using the boundaries learned from hard negative training, it successfully ranks the actual table page into the top 5.

Loss & Training

Visual Selection uses InternViT-300M with a batch size of 32, trained for 10 epochs at a learning rate of \(5e^{-4}\), applying LoRA (rank 32, alpha 32) on the last ViT layer and projector. Semantic Refinement is based on InternVL3 with a batch size of 80, also trained for 10 epochs. The shared ViT layers are frozen, and LoRA is updated only on the new layout branch, the MLP connector, and the LLM/projection layers. Experiments are conducted using 4 A800 80GB GPUs.

Key Experimental Results

Main Results

The main results report NDCG@5 on the ViDoRe benchmark and end-to-end retrieval latency on a corpus of 5,000 pages.

Method	Parameters	Avg. NDCG@5	Latency (5,000 pages)	Description
BM25	-	69.3	-	Traditional OCR/text baseline
SigLIP	883M	59.8	-	Vision-language contrastive model
VisRAG-Ret	3B	78.0	1219.9s	MLLM retrieval, high cost
ColPali	3B	81.9	257.3s	Strong multi-vector baseline
ColQwen2.5	2B	88.8	466.6s	Previous strongest MLLM baseline
LightSTAR	2B	89.1	123.9s	Highest average accuracy and significantly faster

Ablation Study

Both the two-stage structure and modular components are thoroughly ablated.

Configuration	ViDoRe Metric	Latency	Description
Selection only	80.8 NDCG@5	86.8s	Fast but insufficient refinement
Refinement only	89.3 NDCG@5	465.1s	Best performance but full MLLM is too slow
Selection + Refinement	89.1 NDCG@5	123.9s	Almost no accuracy drop, latency drops heavily
w/o CG-Embed	97.0 Recall@100	-	Selection recall drops below 97.8
w/o SA-LateInt	96.8 Recall@100	-	Single-point max is more easily distracted by noisy patches
w/o Feat-Fusion	88.3 NDCG@5	-	Insufficient layout/background modeling
w/o HA-Obj	87.9 NDCG@5	-	Discriminative ability on hard negatives decreases

Key Findings

• The average Recall@100 of the LLM-free Visual Selection reaches 97.8, which is higher than other lightweight solutions like BM25, BGE-M3, and CLIP/SigLIP, proving that the coarse stage successfully preserves a vast majority of relevant pages.
• Scaling to 7,000 pages, LightSTAR's latency increases from 31.1s (at 500 pages) to 161.1s. The growth rate is significantly flatter than the MLLM baselines; it is 10.4×, 4.1×, and 2.3× faster than VisRAG-Ret, ColQwen2.5, and ColPali, respectively.
• The complete pipeline is only 0.2 NDCG@5 lower than the refinement-only counterpart, but decreases the 5,000-page latency from 465.1s to 123.9s, which is the most critical accuracy-efficiency evidence of this paper.

Highlights & Insights

• Priors closely aligned with real retrieval scenarios: Many queries are indeed keyword-anchored. Thus, weeding out irrelevant pages using cheap visual text cues is a high-ROI system design.
• Not a simple cascade: The selection encoder is reused by refinement, and the adaptive feature fusion encourages synergy between representations at both stages, rather than simply chaining two independent models together.
• Correct choice of target for the coarse stage: The first stage aims for Recall@100 instead of final NDCG; as long as the relevant pages are not filtered out, the second stage can recover the semantic details.
• Hard negative design matches candidate distribution: Since the retrieved candidate pool naturally consists of highly similar pages, the hardness-aware objective aligns much better with data difficulty at the refinement stage than conventional in-batch negatives.

Limitations & Future Work

• The assumption of content-grounded queries depends on the premise that "visually visible text on relevant pages contains the query content words". This is weaker when dealing with pure visual cues, cross-lingual synonym expressions, or OCR-invisible chart semantics.
• Although the latency is far lower than MLLM baselines, 123.9s for 5,000 pages is still not millisecond-level online search. Real-world deployment still requires pre-indexing, batching, caching, and smaller candidate sizes.
• Refinement still relies on an MLLM; the candidate fine-ranking cost can still be a bottleneck for large, frequently updated enterprise databases.
• POS-based filtering works well for English, but multilingual documents require language-specific functional word filtering or learnable query token selection.
• Evaluation was primarily verified on the ViDoRe subset; end-to-end evaluation is still needed for cross-page evidence, multi-page composite answers, and downstream generation quality in real-world RAG.

Related Work & Insights

• vs ColPali / ColQwen: These methods rely on MLLM multi-vector representations to achieve strong accuracy; LightSTAR preserves refinement capabilities but restricts heavy computations to the candidate pool.
• vs VisRAG-Ret: VisRAG-Ret relies heavily on heavy visual language encoding, whereas the system value of LightSTAR lies in significantly lowering the cost of full-corpus traversal.
• vs OCR + BM25/BGE: Traditional text retrieval is fast but discards layouts and images; LightSTAR's first stage still directly processes page images, yielding a more robust performance than pure OCR pipelines.
• Inspiration: Visual RAG systems can be hierarchical, utilizing a "low-cost high-recall selector + high-cost refinement/reader". In the future, the selector could even be trained as a controller that dynamically determines candidate sizes.

Rating

• Novelty: ★★★★☆ The idea of two-stage retrieval is not entirely new, but the integration of query content words, lightweight visual patch selectors, and area-adaptive MLLM refinement is very solid.
• Experimental Thoroughness: ★★★★☆ Main results, latency curves, recall, and ablation studies are complete. End-to-end validation on real RAG tasks is still missing.
• Writing Quality: ★★★★☆ The motivation is clear, and the tables are direct. The method's math equations are somewhat dense, but the system logic is easy to follow.
• Value: ★★★★★ High deployment value for visual document retrieval, particularly suitable for large-scale document databases and multimodal RAG.

Related Papers

• [ACL 2025] Hierarchical Document Refinement for Long-context Retrieval-augmented Generation
• [ICLR 2026] SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAG
• [ICLR 2026] AdaCache: Adaptive Caching and Context Augmentation for Efficient LLM Serving
• [ICLR 2026] ELViS: Efficient Visual Similarity from Local Descriptors that Generalizes Across Domains
• [ACL 2026] A Picture is Worth a Thousand Words? An Empirical Study of Aggregation Strategies for Visual Financial Document Retrieval