Skip to content

DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Data Contamination Detection, Variant Contamination, Generation Distribution Variance, Synthetic Difficulty, Memory Adherence and Perturbation Drift

TL;DR

Addressing the failure of traditional contamination detectors against semantically equivalent paraphrased variants, DVD introduces a reference-free framework that captures the bimodal variance of low-probability tail-token synthetic difficulty under temperature sampling, achieving up to 0.239 AUC gain over competitive baselines.

Background & Motivation

The recent explosive progress in Large Language Models (LLMs) has been predominantly quantified through remarkable benchmark performances across advanced mathematical reasoning and graduate-level scientific problem solving. However, this empirical paradigm relies heavily on massive, web-scale corpora and automated synthetic data pipelines, bringing the long-standing crisis of data contamination to the forefront of AI evaluation. Unintended overlap between pretraining datasets and evaluation sets inflates benchmark scores through rote memorization rather than genuine multi-step reasoning, creating an illusion of general intelligence that misleads research and risks catastrophic failure in real-world deployments. Traditional decontamination has primarily targeted verbatim memorization using string overlap, n-gram matching, perplexity thresholds, or minimum-likelihood scoring.

With the widespread integration of advanced generative rewrites (such as GPT-4o-driven augmentation), a far more insidious and evasive failure mode has emerged: variant contamination. Variant contamination occurs when training corpora incorporate samples that are logically equivalent to benchmark questions but undergo controlled lexical, syntactic, or structural transformations. Although these variants entirely preserve the underlying solution trajectory and knowledge dependencies, their restructured surface syntax systematically bypasses shallow string and perplexity matching. Crucially, empirical evaluations show that models fine-tuned on such variants still achieve heavily inflated benchmark accuracy, while existing detectors (e.g., Loss, Perplexity, Zlib, CDD, and Min-K%++) suffer severe performance degradation. On SuperGPQA, traditional perplexity and loss metrics collapse to AUCs between 0.33 and 0.42—becoming systematically inverted relative to true labels—while edit-distance-based CDD degrades to random guessing (~0.50).

This widespread failure highlights the core tension: surface-level probabilistic and lexical metrics are fragile to minor phrasing perturbations, which completely obscure shallow memorization patterns. However, from the perspective of underlying generative dynamics, a model exposed to a contaminated problem undergoes a fundamental conflict between a rigid memorized reasoning template and stochastic temperature exploration. Rather than relying on static single-response probabilities or fragile surface tokens, this paper shifts the paradigm toward dynamic uncertainty behavior across repeated decoding runs. Core idea: exploit temperature perturbations to induce a bimodal alternation between memory-adherence and perturbation-drift states, using the inflated variance of synthetic difficulty across low-probability tail tokens as a principled, reference-free indicator of variant contamination.

Method

Overall Architecture

DVD (Detection via Variance of generation Distribution) is a training-free, reference-free contamination detection pipeline that requires only the test instance and access to the target model's output probabilities. Given a test prompt \(p\), the framework performs temperature sampling (\(\tau = 0.8\)) across \(N\) independent stochastic runs, generating a pool of response candidates \(\{a_i^{(1)}, \dots, a_i^{(N)}\}\). For each concatenated sequence \(s_i^{(n)} = (p, a_i^{(n)})\), the model calculates the "synthetic difficulty" \(D_i^{(n)}\) by averaging the log-likelihoods of the \(K\) least probable tokens. Finally, the sample variance \(\text{DVD}_i\) across the \(N\) generations is computed; a large variance flags contamination-induced bimodal state switching, whereas a small variance indicates the natural unimodal noise of uncontaminated generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Test instance x_i + Evaluation prompt p"] --> B["Controlled Variant Benchmark Construction<br/>Entity sub · Scenario conv · Numerical rewrite · Narrative struct"]
    B --> C["Bimodal Generation State Sampling<br/>Repeated decoding with temperature τ for N candidate responses"]
    C --> D["Synthetic Difficulty Modeling<br/>Average log-likelihood of K least-probable tokens D_i^(n)"]
    D --> E["Generation Distribution Variance Estimation<br/>Sample variance Var(D_i) across N runs yielding DVD score"]
    E --> F["Variant Contamination Decision<br/>High variance = Bimodal alternation (Contaminated) / Low variance = Unimodal drift (Clean)"]

Key Designs

1. Controlled Variant Benchmark Construction: Multi-dimensional Semantic Transformations with Quality Filtering

To rigorously formalize and evaluate variant contamination, the authors construct the first dedicated variant contamination benchmark across two challenging domains: Olympiad-level mathematical reasoning (Omni-MATH) and graduate-level multidisciplinary reasoning (SuperGPQA). Given an original question-answer pair \((x_i, y_i)\), a transformation set \(\mathcal{T} = \{T_{ent}, T_{scn}, T_{num}, T_{nar}\}\) is formulated, spanning entity substitution, background scenario conversion, parameter numerical rewriting under constraint solvability, and step-by-step narrative restructuring. Using GPT-4o, candidate variants \(v_i^{(j)}\) are generated and processed through a two-stage rejection sampling pipeline: stage one guarantees that the fundamental problem trajectory and core semantics remain invariant (\(f(v) = f(x)\)), while stage two enforces rigorous solution validity. High-quality filtered variants are then injected into fine-tuning corpora for Qwen2.5 (1.5B to 32B) and Llama3.1 (8B) models under both full fine-tuning and LoRA regimes, providing a reproducible and controlled benchmark for detector evaluation.

2. Bimodal Generation State Sampling: Decoupling Memory Adherence and Perturbation Drift

Unlike conventional detection protocols that inspect a single greedy output, DVD exploits the dynamic response distribution exposed through stochastic temperature sampling (\(\tau = 0.8\)). The theoretical foundation rests on the discovery that contaminated samples evoke two competing latent generative regimes: - Memory-Adherence State (\(Z = M\)): When the stochastic decoding trajectory hits internalized templates acquired during variant training, the model operates with abnormally elevated local confidence on pivotal steps, allocating high conditional probabilities to key reasoning tokens; - Perturbation Drift State (\(Z = U\)): When temperature noise drives the initial generation off the memorized track, the model falls back to open exploratory reasoning, exhibiting low confidence and uniform randomness across tail tokens.

For clean, uncontaminated queries, generation consistently resides in the perturbation-drift state with a single smooth mode; for contaminated queries, the model switches between memory-guided reproduction and stochastic wandering, resulting in pronounced bi-modal distribution shifts across repeated generations.

3. Synthetic Difficulty Modeling: Isolating Memorization Signals via Tail-Token Dynamics

Global sequence perplexity averages over predictable functional words and syntactic glue, severely dampening the localized signal of memorization. To counter this dilution, DVD models uncertainty specifically in the tail distribution via the concept of synthetic difficulty. For each generated response \(a_i^{(n)}\) appended to prompt \(p\), the framework identifies the \(K\) least probable tokens (which empirically correspond to critical algebraic steps, numerical intermediate results, or pivotal deduction milestones) and calculates their mean log-likelihood:

\[D_i^{(n)} = \frac{1}{K} \sum_{j=1}^K \log P_\theta \left(t_j \mid s_i^{(n)}\right)\]

Dividing strictly by \(K\) rather than full response length \(T_i^{(n)}\) maintains an exact per-token average over the critical tail region. In memory-adherence mode, familiarity with the logical core lifts the likelihood of these bottleneck tokens (\(\mu_M > \mu_U\)), while in drift mode they exhibit natural, unconstrained low probabilities.

4. Generation Distribution Variance Estimation: Closed-Form Law of Total Variance Detector

By framing the synthetic difficulty of contaminated samples as a mixture of two latent distributions with expectations \(\mu_M\) and \(\mu_U\) and weights \(\pi_M, \pi_U\), the overall variance decomposes via the Law of Total Variance into within-state variance and between-state variance:

\[\text{Var}(X) = \pi_M \left(\sigma_M^2 + (\mu_M - \mu)^2\right) + \pi_U \left(\sigma_U^2 + (\mu_U - \mu)^2\right)\]

where \(\mu = \pi_M \mu_M + \pi_U \mu_U\). Because \(\mu_M \neq \mu_U\), the between-state discrepancy term \((\mu_M - \mu)^2\) drastically inflates total variance \(\text{Var}(X)\). Conversely, for uncontaminated samples, a single dominant drift state keeps total variance low and bounded. Consequently, the sample variance of synthetic difficulty across \(N\) runs constitutes a parameter-free, threshold-agnostic test statistic:

\[\text{DVD}_i = \frac{1}{N} \sum_{n=1}^N \left(D_i^{(n)} - \bar{D}_i\right)^2, \quad \text{where } \bar{D}_i = \frac{1}{N} \sum_{n=1}^N D_i^{(n)}\]

This scalar directly quantifies the degree of bimodal oscillation and serves as the final contamination score.

Benchmark & Evaluation Protocol

The empirical framework evaluates models under two training regimes: (1) Full Fine-Tuning (1 epoch) on two NVIDIA A800 GPUs using Adam (\(lr = 1 \times 10^{-5}\), cosine decay with 0.1 warmup, bfloat16); (2) LoRA Fine-Tuning (rank 8, 10 epochs) on a single A800 GPU to emulate resource-constrained, high-memorization settings. Evaluated backbones include Qwen2.5-Instruct (1.5B, 3B, 7B, 32B) and Llama3.1-8B-Instruct. The evaluation metric is ROC-AUC, with default hyperparameters fixed at tail-token count \(K = 20\), sampling count \(N = 50\), and temperature \(\tau = 0.8\).

Key Experimental Results

Main Results

The table below reports detection ROC-AUC across Omni-MATH and SuperGPQA under the Full Fine-Tuning (1 epoch) setup:

Method Omni-MATH (1.5B) Omni-MATH (3B) Omni-MATH (7B) Omni-MATH (32B) SuperGPQA (1.5B) SuperGPQA (3B) SuperGPQA (7B) SuperGPQA (32B)
Min-K%++ 0.694 0.693 0.680 0.681 0.422 0.463 0.435 0.436
CDD (Edit Dist) 0.494 0.495 0.512 0.507 0.496 0.504 0.518 0.503
Min-K 0.538 0.560 0.572 0.578 0.501 0.492 0.511 0.539
Perplexity 0.544 0.549 0.557 0.556 0.517 0.520 0.513 0.517
Loss 0.626 0.637 0.650 0.635 0.404 0.406 0.408 0.409
Zlib Entropy 0.573 0.576 0.583 0.581 0.425 0.430 0.428 0.427
Embedding Sim 0.521 0.506 0.533 0.505 0.531 0.529 0.524 0.521
DVD (Ours) 0.744 0.747 0.734 0.667 0.770 0.708 0.740 0.743

Under the 10-epoch LoRA fine-tuning setting, DVD achieves even wider margins: on Omni-MATH, it attains 0.771 (1.5B), 0.745 (3B), 0.731 (7B), and 0.715 (32B), outperforming Min-K%++ (0.608–0.648) by up to 0.125 AUC; on SuperGPQA, DVD reaches 0.674–0.751, while Loss and Min-K%++ collapse into systematic failure (0.333–0.456).

Ablation Study & Robustness Analysis

Comprehensive analyses examine detection sensitivity to sampling budgets and realistic corpus dilution:

1. Sampling Budget Sensitivity (\(N \in \{5, 10, 25, 50\}\), Full Fine-Tuning AUC):

Configuration \(N=5\) \(N=10\) \(N=25\) \(N=50\) (Default) Trend & Practical Implication
Qwen-7B + Omni-MATH 0.526 0.615 0.702 0.734 Sharp gain from \(N=5\) to 25; plateaus at \(N=50\)
Qwen-7B + SuperGPQA 0.513 0.630 0.716 0.740 Practical tradeoff: \(N=25\) delivers 96% peak power
Llama-8B + Omni-MATH 0.541 0.622 0.698 0.728 Robust architecture transfer across Llama backbones
Llama-8B + SuperGPQA 0.520 0.641 0.705 0.735 Confirms moderate sampling budget sufficiency

2. Pretraining-Scale Dilution (\(100\times\) Irrelevant OpenOrca Data Mixture, Full Fine-Tuning):

Model Backbone DVD (Ours) Min-K%++ Perplexity Embedding Sim Robustness Finding
Qwen-1.5B (Full FT) 0.702 0.598 0.472 0.505 Resists 100x dilution with minimal degradation
Qwen-3B (Full FT) 0.710 0.595 0.508 0.512 Baselines collapse to near-random ~0.50
Qwen-7B (Full FT) 0.696 0.581 0.483 0.518 High AUC preserved under pretraining-scale noise
Qwen-32B (Full FT) 0.709 0.599 0.501 0.581 Validates real-world auditing practicality

Key Findings

  • Systematic Breakdown of Perplexity and Loss Baselines: Traditional likelihood-based metrics display severe label inversion on SuperGPQA (AUC down to 0.333). Harder graduate-level questions inherently incur high loss, and rewritten variants lack verbatim token matches, leading the model to assign higher overall loss to contaminated variants than to trivial clean samples.
  • Robust High-Performance Band for Tail Tokens \(K\): Sensitivity sweeps across \(K \in [5, 40]\) show a smooth, unimodal curve peaking at \(K=22\) (\(\text{AUC} = 0.751\)) and maintaining \(>0.74\) throughout \(K \in [18, 26]\). \(K < 15\) introduces finite-sample noise, while \(K > 30\) dilutes the signal with high-frequency functional vocabulary.
  • Orthogonal Synergy with Verbatim Memorization Detectors: When evaluated on exact verbatim contamination datasets (Table 8), Min-K% and CDD lead slightly at 0.82–0.84 AUC, while DVD remains competitive at 0.79–0.81. This proves DVD is complementary to verbatim detectors, enabling hybrid auditing systems.

Highlights & Insights

  • From Static Surface Matching to Dynamic Generation Variance: Rather than inspecting static text similarity or deterministic forward-pass losses, DVD uncovers the latent state transitions triggered by temperature sampling, establishing generation distribution variance as a robust behavioral signature of memorization.
  • Principled Application of the Law of Total Variance: Converting a complex bimodal state alternation problem into a simple, closed-form empirical variance statistic \(\text{Var}(D_i)\) circumvents unstable EM parameter estimation while remaining mathematically grounded.
  • Broad Transferability Beyond Evaluation Auditing: The variance-of-difficulty principle provides a generic probe into an LLM's uncertainty landscape, with immediate applications in black-box membership inference, hallucination scoring, and watermarking detection.

Limitations & Future Work

  • Open-Ended Reasoning Variance Baseline: Tasks with highly diverse solution paths or open-ended narrative goals naturally induce high generation diversity, elevating baseline variance for clean questions and requiring category-level calibration.
  • Inference Budget Overhead: Running \(N = 25 \sim 50\) stochastic samplings per test query imposes non-trivial compute and latency overhead compared to single-pass perplexity scoring.
  • Gap Between Controlled Augmentation and In-the-Wild Leakage: While GPT-4o-generated variants provide controlled and reproducible evaluation, real-world contamination from cross-lingual translation, RAG snippets, and web forum scrapes requires broader empirical validation.
  • vs CDD (Cross-Decoding Divergence): CDD calculates edit distance between greedy decoding and stochastic outputs; its reliance on surface tokens makes it fail (AUC ~0.50) when variants alter phrasing. DVD directly models token likelihood variance, bypassing textual divergence.
  • vs Min-K% / Min-K%++: Min-K% variants score the absolute mean likelihood of tail tokens, which is vulnerable to lexical distribution shift in paraphrased prompts. DVD evaluates the variance of tail tokens across multiple generations, achieving invariance to surface phrasing shifts.
  • vs DCQ (Data Contamination Quiz): DCQ constructs explicit multiple-choice perturbation quizzes to evaluate model preference. DVD requires no query mutation or external prompt engineering, operating strictly on the test item itself.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the variant contamination problem and innovates a dynamic generation-variance framework exploiting bimodal state oscillations.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across two distinct reasoning benchmarks, model scales from 1.5B to 32B, multiple LLM architectures, 100x dilution, and sampling ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity, mathematically rigorous derivation of variance decomposition, and intuitive case-study illustrations.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable, reference-free auditing tool to safeguard benchmark integrity and prevent deceptive evaluation progress in LLMs.