AnInter: Anti-Interference Training for Long Context Retrieval¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Information Retrieval & RAG
Keywords: Long Context Retrieval, Dense Retriever, Anti-Interference Training, Retrieval-Augmented Generation, Multi-Positive InfoNCE
TL;DR¶
Targeting the fundamental failure mode where local evidence in long documents is overwhelmed by distracting noise, AnInter introduces an efficient short-text anti-interference training framework that constructs noisy multi-positives via circular sliding concatenation and trains models under Multi-Positive InfoNCE, delivering a +16.4 point average boost over backbones on the challenging LongEmbed benchmark.
Background & Motivation¶
Dense retrievers in retrieval-augmented generation (RAG) systems are responsible for locating external evidence from massive knowledge stores to ground downstream language model generations. As RAG pipelines increasingly expand into complex domains such as legal contracts, enterprise documentation, scientific literature, technical manuals, and meeting transcripts, retrievers are required to handle lengthy multi-page documents rather than isolated hundred-word passages. To support this demand, recent efforts have focused on expanding context windows (such as Qwen3-Embedding extending to 32K tokens), designing long-context encoder architectures (such as M2-BERT), contextualizing chunk embeddings via Late Chunking, or employing fine-grained multi-vector interactions (such as ColBERT and ColPali). Nevertheless, empirical evaluations indicate that simply scaling context windows or continuing contrastive pretraining on long texts fails to solve long-context retrieval degradationโfor instance, Qwen3-Embedding-8B with a 32K context window performs significantly worse on the LongEmbed benchmark than E5-Mistral which possesses only a 4K window.
The core tension behind this breakdown is that the true bottleneck in long-document retrieval is not token capacity, but the acute conflict between local evidence sparsity and expanding contextual noise. In typical long retrieval scenarios, the target gold evidence occupies only a compact local span, while the surrounding paragraphs act as distracting interference. Standard contrastive learning formulations rely strictly on isolated one-to-one query-document pairs, teaching the retriever to pull an isolated document close while repelling all other documents. This setting fails to reflect realistic document segmentation, where target evidence is inherently blended with extraneous topics into a single document unit. When the gold evidence is diluted within distracting text, the model's vector similarity collapses. Furthermore, directly conducting contrastive learning over full-length sequences is computationally prohibitive: the quadratic scaling of Transformer self-attention drastically constrains batch sizes, thereby impairing discriminative in-batch negative learning.
This paper tackles the challenge from a distinct angle: rather than relying on expensive long-sequence pretraining, long-text retrieval can be formulated as an anti-interference capability that can be taught effectively on short texts. By exposing the retriever to synthesized noisy mixed documents while preserving the original evidence, the model learns to maintain invariant vector similarity as distracting noise escalates. Core idea: construct an efficient short-text anti-interference training path AnInter that synthesizes multi-level noisy positive documents via circular sliding windows, retains clean origin pairs as semantic anchors, and optimizes dense retrievers with Multi-Positive InfoNCE to achieve robust generalization on long-context retrieval tasks.
Method¶
Overall Architecture¶
The AnInter framework equips dense retrievers with noise-resilient representations without requiring heavy long-sequence pretraining. The pipeline consists of three cohesive stages: first, candidate query-document pairs are gathered from MS MARCO and SQuAD and filtered under strict dual-stage discriminability criteria to establish a reliable origin corpus; second, a circular adjacent sliding window concatenates clean gold documents with neighboring irrelevant passages across controlled noise levels \(k\) and randomizes segment order to produce diverse noisy positive documents; third, the original clean document is explicitly retained alongside all synthesized noisy positives to construct a comprehensive positive set \(P_i\), and the retriever is optimized under Multi-Positive InfoNCE loss across a global cross-GPU candidate negative pool.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Origin Query-Document Pairs<br/>Filtered from MS MARCO & SQuAD"] --> B["Noisy Positive Synthesis<br/>Circular adjacent window concatenation & random shuffling"]
B --> C["Origin Semantic Anchoring<br/>Forming multi-positive set Pi with clean origin document"]
C --> D["Global Cross-GPU Negative Gathering<br/>Sampling in-batch negatives & gathering across all devices"]
D --> E["Multi-Positive InfoNCE Optimization<br/>Back-propagating multi-positive contrastive loss"]
Key Designs¶
1. Noisy Positive Synthesis: simulating long-document distraction via controlled circular sliding concatenation
Conventional contrastive objectives train models on isolated query-passage pairs, leaving dense retrievers vulnerable when target evidence is embedded among irrelevant text in long documents. To simulate this interference pattern under short-sequence computational budgets, the authors formulate a synthetic data generator parameterized by a noise level \(k\), representing the number of distracting passages concatenated with the gold document. Given a deduplicated corpus of origin documents, the documents are randomly shuffled into a sequence \(\pi\). A circular adjacent window of size \(k+1\) iterates through the shuffled documents, concatenating \(k+1\) consecutive items while randomizing their internal ordering to eliminate positional bias. This circular indexing guarantees that each query receives an equal number of synthesized documents at each noise level without generating duplicates. By selecting \(k \in \{1, 2, 3, 4, 5\}\) during training, each origin document yields \(2 + 3 + 4 + 5 + 6 = 20\) noisy positive documents of varying lengths and noise intensities.
2. Origin Semantic Anchoring: preserving clean matching signals to prevent semantic drift
When transitioning from one-to-one contrastive learning to multi-positive training on noisy documents, there is an inherent danger of diluting the retriever's original semantic matching accuracy. If the model is exposed exclusively to mixed documents, the learned embeddings can drift toward broad contextual representations, losing fine-grained specificity. To prevent this degradation, AnInter implements origin semantic anchoring: for every query \(q_i\), the positive candidate set \(P_i\) explicitly retains the clean origin document \(d_i\) alongside all generated noisy positives: $\(P_i = \{d_i\} \cup P_{i}^{\text{noise}}\)$ Ablation studies reveal that removing the origin document from training causes the overall anti-interference metric \(TanIn\) to drop from 44.3 to 32.5, while origin retrieval accuracy (\(Ori.Hit@1\)) falls from 95.2 to 91.9. The original clean pair acts as a firm semantic anchor, ensuring that the model retains its baseline alignment while learning noise invariance.
3. Multi-Positive InfoNCE: multi-evidence collaborative alignment against global negatives
Standard InfoNCE loss is constrained to single positive-negative pairs. AnInter extends this to a Multi-Positive InfoNCE objective. For query \(q_i\) and each individual document \(p \in P_i\) (encompassing both clean and noisy positives), the positive-level contrastive loss against the negative candidate set \(N_i\) is computed as: $\(\ell(q_i, p) = -\log \frac{\exp(s(q_i, p) / \tau)}{\exp(s(q_i, p) / \tau) + \sum_{n \in N_i} \exp(s(q_i, n) / \tau)}\)$ where \(s(q, d)\) denotes the cosine similarity between \(\ell_2\)-normalized dense embeddings, and \(\tau\) is the temperature parameter. The global loss \(\mathcal{L}_{\text{MPI}}\) averages across all queries and their corresponding positive candidates: $\(\mathcal{L}_{\text{MPI}} = \frac{1}{|B|} \sum_{i=1}^{|B|} \frac{1}{|P_i|} \sum_{p \in P_i} \ell(q_i, p)\)$ In distributed training across 8 RTX PRO 6000 GPUs, each GPU handles 1 query along with its 21 positive documents. To maintain memory stability while retaining strong negative signals, 32, 8, and 4 extra negative documents are sampled per GPU for the 0.6B, 4B, and 8B models, respectively. Document embeddings are gathered across all GPUs via an All-Gather operation to form a shared global negative candidate pool prior to computing loss.
Loss & Training¶
Fine-tuning is conducted on the Qwen3-Embedding series (0.6B, 4B, 8B) utilizing LoRA adaptation applied to all attention projections (\(q, k, v, o\)) and MLP layers (\(gate, up, down\)). LoRA ranks are configured to 128 (0.6B), 256 (4B), and 512 (8B), with \(\alpha = 128\) and dropout 0.05. Models are trained in bf16 mixed precision with gradient checkpointing, a learning rate of \(2 \times 10^{-5}\), a linear warmup ratio of 0.1, weight decay of 0.01, temperature \(\tau = 0.02\), and a maximum sequence length of 10,000 tokens for 20 epochs.
Key Experimental Results¶
Main Results¶
To evaluate transferability from short-text anti-interference fine-tuning to genuine long-context retrieval, AnInter is assessed on the LongEmbed benchmark, which includes synthetic retrieval tasks (Needle, Passkey; Acc@1) and real-world long-context tasks (QMSum, SummScreenFD, 2WikiMultihopQA, NarrativeQA; nDCG@10).
| Model | Context Window | Needle (Acc@1) | Passkey (Acc@1) | QMSum (nDCG@10) | SFD (nDCG@10) | WQA (nDCG@10) | NQA (nDCG@10) | Avg. |
|---|---|---|---|---|---|---|---|---|
| E5Base | 512 | 28.5 | 38.0 | 23.8 | 74.7 | 55.8 | 25.3 | 41.0 |
| Contriever | 512 | 29.0 | 38.5 | 25.5 | 73.5 | 47.3 | 26.7 | 40.1 |
| BGE-M3 | 8192 | 40.5 | 59.3 | 35.5 | 94.0 | 78.0 | 45.8 | 58.9 |
| E5-Mistral | 4096 | 48.3 | 71.0 | 43.6 | 96.8 | 82.0 | 44.6 | 64.4 |
| E5-Mistral + NTK | 32K | 66.8 | 93.8 | 49.2 | 97.1 | 95.2 | 49.8 | 75.3 |
| Qwen3-Embedding-0.6B | 32K | 38.5 | 92.8 | 40.1 | 92.2 | 81.2 | 24.2 | 61.5 |
| Qwen3-Embedding-4B | 32K | 59.0 | 98.5 | 40.8 | 94.9 | 84.9 | 26.8 | 67.5 |
| Qwen3-Embedding-8B | 32K | 48.5 | 93.3 | 39.8 | 82.7 | 57.6 | 19.1 | 56.8 |
| AnInter-0.6B | 10K | 74.8 | 98.5 | 44.6 | 95.4 | 90.3 | 46.3 | 75.0 |
| AnInter-8B | 10K | 86.0 | 99.8 | 55.5 | 94.6 | 79.4 | 54.0 | 78.2 |
| AnInter-4B | 10K | 91.8 | 98.0 | 49.6 | 98.1 | 91.1 | 61.3 | 81.7 |
Furthermore, on the controlled Anti-Interference Capability Benchmark (AICB) spanning 13 noise levels (\(k=1 \sim 60\)), AnInter shows massive resilience gains:
| Model | Ori.Hit@1 | anIn@1 | anIn@5 | anIn@10 | anIn@20 | anIn@40 | anIn@60 | TanIn |
|---|---|---|---|---|---|---|---|---|
| LLaMA-Embed-Nemotron-8B | 83.6 | 50.3 | 8.2 | 4.1 | 1.8 | 0.8 | 0.4 | 5.6 |
| NV-Embed-v2 | 95.4 | 68.8 | 14.1 | 4.7 | 1.7 | 0.3 | 0.9 | 7.4 |
| E5-Mistral-7B-instruct | 71.0 | 33.5 | 10.9 | 5.9 | 3.1 | 1.5 | 1.0 | 5.4 |
| Instructor-XL | 99.0 | 77.7 | 29.6 | 16.1 | 8.4 | 4.3 | 2.9 | 13.7 |
| Qwen3-Embedding-4B | 98.9 | 77.7 | 21.2 | 11.2 | 5.2 | 2.8 | 1.3 | 11.0 |
| Qwen3-Embedding-8B | 99.3 | 79.1 | 24.6 | 8.0 | 3.2 | 1.7 | 1.4 | 10.3 |
| AnInter-0.6B | 90.0 | 78.6 | 53.6 | 44.3 | 28.0 | 17.9 | 13.3 | 29.3 |
| AnInter-4B | 95.2 | 90.2 | 74.8 | 64.9 | 46.6 | 30.4 | 23.7 | 44.3 |
| AnInter-8B | 96.8 | 87.7 | 80.4 | 70.6 | 47.7 | 33.4 | 27.1 | 46.9 |
Ablation Study¶
Ablations on Qwen3-Embedding-4B investigate the role of training noise distributions, semantic anchoring, and progressive curriculum training:
| Configuration | Ori.Hit@1 | anIn@1 | anIn@5 | anIn@10 | anIn@20 | anIn@40 | anIn@60 | TanIn |
|---|---|---|---|---|---|---|---|---|
| Base (no anti-interference) | 98.9 | 77.7 | 21.2 | 11.2 | 5.2 | 2.8 | 1.3 | 11.0 |
| Weak noise \(k=\{1\}\) | 96.2 | 83.4 | 49.2 | 34.7 | 19.8 | 8.7 | 4.9 | 21.9 |
| Moderate noise \(k=\{1, 2, 3\}\) | 95.3 | 85.8 | 58.6 | 44.1 | 28.7 | 14.9 | 9.2 | 28.6 |
| Full baseline \(k=\{1, 2, 3, 4, 5\}\) | 95.2 | 90.2 | 74.8 | 64.9 | 46.6 | 30.4 | 23.7 | 44.3 |
| No origin data | 91.9 | 82.1 | 62.8 | 50.3 | 35.2 | 18.7 | 12.2 | 32.5 |
| Broad noise \(k=\{1, \dots, 10\}\) | 89.3 | 85.9 | 61.8 | 48.7 | 35.1 | 23.2 | 17.2 | 35.2 |
| Progressive \(\{1\sim5\} \to \{6\sim10\}\) | 95.6 | 91.4 | 77.2 | 67.8 | 50.1 | 33.7 | 26.9 | 47.4 |
Key Findings¶
- The primary bottleneck of long-context retrieval is interference vulnerability rather than sequence capacity: While native Qwen3-Embedding backbones achieve over 99% accuracy on isolated pairs, their robustness collapses under even minor noise (\(anIn@5\) falls to 21.2%). By teaching noise invariance, AnInter yields a +16.4 point average gain across LongEmbed tasks without altering positional encodings.
- Moderate noise fine-tuning exhibits extraordinary out-of-distribution extrapolation: Although models are fine-tuned solely with \(k \le 5\), AnInter-8B sustains an \(anIn@60\) score of 27.1 when subjected to 60 concatenated distracting documentsโoutperforming the base model (1.4) by nearly twenty-fold.
- Semantic anchoring is essential for representation stability: Omitting clean origin pairs leads to severe degradation across both clean matching (\(Ori.Hit@1\) drops to 91.9) and composite robustness (\(TanIn\) falls to 32.5), validating that clean pairs act as an anchor preventing semantic drift.
- Backbone quality sets the ceiling for anti-interference gains: Training on subsets of varying base retrieval capabilities shows that as backbone \(Ori.Hit@1\) scales from 37.6 to 98.9, the fine-tuned \(TanIn\) rises monotonically from 8.4 to 44.3.
Highlights & Insights¶
- Efficient short-text recipe for long-context challenges: Instead of consuming massive compute on full long-sequence pretraining, AnInter solves the core failure mode of long-document retrieval by training on short, synthetically concatenated noisy windows, dramatically reducing compute requirements.
- Rigorous evaluation benchmark (AICB): AICB introduces a controlled evaluation protocol that filters candidate pairs to isolate pure interference effects from baseline matching failures, offering a nuanced assessment of noise tolerance.
- Generalizable training paradigm: The combination of Multi-Positive InfoNCE and clean semantic anchoring can be directly adapted to other dual-encoder models, cross-encoders, and rerankers deployed in noisy RAG environments.
Limitations & Future Work¶
- Domain discrepancy in synthetic noise: AICB and the training data construct noisy positives by concatenating disparate QA passages. Real-world legal and financial documents often contain distracting passages with near-identical terminology and themes, representing a higher degree of semantic confusion.
- Minor performance trade-off on standard retrieval: On 10 standard MTEB retrieval tasks, AnInter shows modest drops of 1.5 to 6.9 points compared to its backbones, reflecting a slight trade-off when reallocating representation capacity toward noisy multi-positive alignment.
- Architectural scope and end-to-end evaluation: Experiments were concentrated on the Qwen3-Embedding family. Extending this recipe to other architectures (such as BGE or multi-vector ColPali) and conducting downstream generation evaluations will further validate its real-world impact.
Related Work & Insights¶
- vs LongEmbed / Context Window Scaling: Prior work focused primarily on extending context windows through RoPE interpolation and long-document pretraining; AnInter proves that the critical missing factor in long-context retrieval is anti-interference discriminability.
- vs Late Chunking / ColBERT: Late Chunking requires full-document encoder passes before chunking, and ColBERT incurs significant storage overhead from multi-vector representations; AnInter maintains the low latency and lightweight storage of single-vector dense retrievers while bolstering robustness during training.
Rating¶
- Novelty: โญโญโญโญ [Framing long-context retrieval as an anti-interference challenge with short-text training]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across AICB, LongEmbed, MTEB, and multi-faceted ablation suites]
- Writing Quality: โญโญโญโญโญ [Clear motivation, well-formulated methodology, and transparent experimental findings]
- Value: โญโญโญโญโญ [Provides a practical, high-efficiency blueprint for production RAG systems facing noisy long contexts]