TRACE: A Generalizable Drift Detector for Streaming Data-Driven Optimization

Conference: AAAI 2026 arXiv: 2512.07082 Code: https://github.com/YTALIEN/TRACE Area: LLM Evaluation Keywords: Concept Drift Detection, Streaming Data Optimization, Attention Mechanism, Transfer Learning, Plug-and-Play

TL;DR

This paper proposes TRACE, a transferable concept drift detector based on attention-based sequence learning. By tokenizing statistical features and employing a dual-attention encoder, TRACE learns drift patterns that generalize across tasks, enabling deployment on unseen datasets and integration as a plug-and-play module into streaming data-driven optimization algorithms.

Background & Motivation

Challenges in Streaming Data-Driven Optimization (SDDO)

Many real-world optimization tasks rely on continuously arriving data streams. Examples include: - Traffic optimization in smart cities relying on real-time sensor and surveillance data - Load scheduling in energy systems relying on continuous demand data

In streaming environments, the underlying data distribution may shift unpredictably due to external factors (traffic accidents, weather fluctuations, etc.)—a phenomenon known as Concept Drift.

Overly Strong Assumptions in Existing SDDEA Methods

Streaming data-driven evolutionary algorithms (SDDEAs) have made progress in surrogate model construction and knowledge transfer, but rely on unrealistic assumptions:

Fixed and known drift intervals: In practice, drift is unpredictable.

Immediate availability of complete data per environment: In practice, data arrives incrementally.

Lack of reliable drift detection mechanisms: This can lead to overfitting to outdated distributions or missing recurring patterns.

Limitations of Existing Drift Detection Methods

Drift detection methods from the stream data mining literature (DDM, ADWIN, etc.) are primarily designed for classification tasks:

Assume discrete labels or bounded outputs: Inapplicable to the unbounded real-valued domains in SDDO.

Focus on abrupt changes in prediction error rates: Gradual performance degradation in optimization landscapes is overlooked.

Threshold-based methods lack generalizability: Hand-crafted rules struggle to adapt across different scenarios.

Core need: A flexible, generalizable, and adaptive drift detection method specifically designed for SDDO.

Method

Overall Architecture

TRACE comprises three levels:

1. Stream Tokenization: Converts the data stream into a sequence of statistical features.
2. TRACE Detector: Learns transferable drift patterns via a dual-attention encoder.
3. TRACE-EA: Embeds TRACE into a streaming optimizer via a detection–adaptation loop.

Key Designs

1. Stream Tokenization: Converts the raw data stream into sequences suitable for drift modeling.

Mechanism: Surrogate model prediction errors are used as indirect indicators of distribution shift.

Prediction Error Computation: For each sample \((\mathbf{x}_i, y_i)\) in the stream, the surrogate model prediction error is computed as:

\[e_i = \begin{cases} |(y_i - \hat{y}_i)/y_i| & y_i \neq 0 \\ |y_i - \hat{y}_i| & \text{otherwise} \end{cases}\]

In a stationary environment, errors fluctuate around a stable mean; upon drift, a significant shift in error is observed.

Sliding Window Statistical Features: For a sliding window of length \(n\), a 7-dimensional statistical feature vector is extracted:

\[fv_t = (\mu_t, \sigma_t, \min_t, \max_t, Q1_t, Q2_t, Q3_t)\]

comprising the mean, standard deviation, minimum, maximum, and three quartiles.

Token Sequence Construction: - \(T\) consecutive feature vectors are selected. - An environment context token \(fv_0\) (computed at the start of the current environment) is prepended. - Each training sample is a sequence of \((T+1)\) tokens \(X \in \mathbb{R}^{(T+1) \times d_f}\). - Drift label \(dl\): 0 for no drift; otherwise, the position index \(l\) where drift occurs.

2. Dual-Attention Encoder: Captures both global and local drift patterns.

The encoder consists of two complementary self-attention mechanisms:

Global Multi-Head Self-Attention (G-MSA): - Standard Transformer encoder. - Each token attends to all other tokens. - Captures long-range temporal and structural patterns (drift signals). - Incorporates positional encoding, making it sensitive to the relative order of tokens.

\[\text{G-MSA}(\mathbf{H}^{pos}) = \text{Concat}(h_1^G, \cdots, h_m^G) \mathbf{W}_G\]

Context-Guided Multi-Head Self-Attention (C-MSA):

Core Idea: Drift detection fundamentally requires measuring the deviation of recent data relative to the current environment.

• Only the context token (first token) is used as the query.
• All other tokens serve as keys and values.
• Directly models the relationship between the current environment context and recent data tokens.
• Particularly effective at capturing incremental drift patterns.

Joint Representation: Outputs of G-MSA and C-MSA are concatenated, encoding both global temporal dependencies and local contextual deviations.

3. Drift Classification Head: Pointer-style Localization

A pointer network-inspired classification head predicts the position of drift within the sequence or indicates no drift:

\[\mathbf{y} = \text{Softmax}(\mathbf{W_2} \cdot \text{LayerNorm}(\phi(\mathbf{W_1} \cdot \mathbf{z} + \mathbf{b_1})) + \mathbf{b_2})\]

Output \(\mathbf{y} \in \mathbb{R}^{T+1}\): \(T\) window positions + 1 no-drift class.

4. TRACE-EA: Detection–Adaptation Loop

TRACE is embedded as a plug-and-play module within the DASE framework:

1. Detection: At each time step, a new data batch is received, the sliding window is updated, and a token sequence is constructed as input to TRACE.
2. Adaptation: Upon drift detection, a new environment is instantiated and similar historical environments are retrieved from the archive.
3. Knowledge Transfer: Relevant surrogate models and population knowledge are reused to warm-start optimization in the new environment.
4. Archive Update: Knowledge from the new environment is stored for future use.

Loss & Training

Training Data: Synthetic streaming data generated using SDDObench. - Each instance consists of 60 environments, each with 600–900 samples. - Sliding window size \(n=30\); maximum sequence length 20. - Radial Basis Function Networks (RBFN) are used as surrogate models.

Training Objective: Cross-entropy loss to optimize prediction of the drift index.

Augmentation Strategy: Training samples are randomly truncated after the true drift index, exposing TRACE to diverse temporal patterns.

Training Configuration: Batch size 32, learning rate \(5 \times 10^{-4}\), 50 epochs.

Key Experimental Results

Main Results

Drift Detection Performance (Figure 3, RQ1):

TRACE consistently outperforms all baseline detectors (DDM, ADWIN, HDDM_A/W, KSWIN, RADAR, MCD_DD, HCDD) on both in-distribution (ID, SDDObench) and out-of-distribution (OOD, DBG and GMPB) benchmarks, achieving the highest precision on most tasks.

SDDO Optimization Performance (Table 1, RQ2):

Instance	TRACE-EA	SAEF-1GP	DSEMFS	DASE	Prev. SOTA
F4D1	9.9e-02	4.9e+01	6.4e+01	2.7e-01	2.7e-01
F4D2	2.3e+00	5.8e+01	4.4e+01	2.4e+01	2.4e+01
F1D1	5.2e+01	6.3e+01	5.9e+01	5.9e+01	5.9e+01
F1D6	3.7e+01	5.4e+01	5.3e+01	5.0e+01	4.7e+01

TRACE-EA achieves the lowest \(E_{DT}\) values (closer to the optimum is better) on nearly all benchmarks.

Ablation Study

Contribution of Each Component (Table 2, DBG_F1 D1–D4):

Configuration	D1 Prec/F1	D2 Prec/F1	D3 Prec/F1	D4 Prec/F1
TRACE (Full)	0.77/0.73	0.75/0.71	0.73/0.70	0.69/0.65
w/o G-MSA	0.59/0.25	0.55/0.20	0.51/0.20	0.45/0.17
w/o C-MSA	0.50/0.20	0.48/0.10	0.52/0.15	0.40/0.20
w/o PE	0.65/0.45	0.64/0.46	0.65/0.45	0.63/0.50
Vanilla class	0.60/0.51	0.65/0.55	0.66/0.50	0.65/0.60

Key Observations: - Removing G-MSA causes the largest performance drop (F1: 0.73 → 0.25), indicating that capturing long-range patterns is critical. - Removing C-MSA is equally detrimental (F1 → 0.20), confirming the indispensability of context-aware encoding. - The standard fully connected classification head underperforms the pointer-style head.

Key Findings

1. Strong cross-distribution generalization: Superior performance on OOD datasets (DBG, GMPB) demonstrates the transferability of the learned drift patterns.
2. C-MSA learns meaningful attention distributions: Attention concentrates around drift time points and the context token.
3. G-MSA learns a structured embedding space: PCA visualization reveals clear separation between drift and normal tokens.
4. Effective on real-world streaming clustering tasks: On real datasets such as Electricity and Kddcup99, TRACE-EA achieves lower DBI values with smaller variance.
5. Fast detection response: Low detection latency and minimal computational overhead.

Highlights & Insights

1. Elegant tokenization design: Converting raw data streams into standard sequences via prediction errors and statistical features makes learning-based detection tractable.
2. Complementarity of dual attention: G-MSA captures global temporal patterns while C-MSA anchors to the environment context; both are indispensable.
3. Pointer-style classification head: Beyond binary drift detection, it precisely localizes the drift position, providing richer information.
4. Synthetic training, real-world transfer: Training on SDDObench synthetic data generalizes effectively to real streaming clustering tasks, demonstrating true transferability of the learned drift patterns.
5. Plug-and-play design: TRACE can be embedded into different SDDEA frameworks without modifying optimization logic.

Limitations & Future Work

1. Detection latency from fixed sliding windows: The authors acknowledge this limitation; adaptive window sizing is a natural direction for improvement.
2. Loose coupling between detector and optimizer: The current plug-and-play design may be suboptimal; tighter integration could yield better performance.
3. Dependence on surrogate model quality: The quality of RBFN affects the error signal, which in turn affects detection performance.
4. Sufficiency of 7-dimensional statistical features: Richer features (e.g., autocorrelation, higher-order moments) may provide additional information.
5. Training data diversity: Synthetic data may not cover all real-world drift patterns.

Related Work & Insights

• DDM (Gama et al. 2004): A classical method that monitors changes in the mean of prediction errors.
• ADWIN (Bifet et al. 2007): A two-window method leveraging Hoeffding bounds.
• RADAR (Alsaedi et al. 2023): Detects latent space drift using recurrent variational embeddings.
• MCD-DD (Wan et al. 2024): Estimates maximum concept discrepancy via contrastive learning.
• DASE (Zhong et al. 2025): A streaming optimization method integrating statistical drift detection; serves as the base framework for TRACE-EA.

Insights: 1. Framing temporal drift detection as a sequence classification problem is an effective and principled abstraction. 2. The context token design provides an "anchor" for the sequence, which is the key to enabling C-MSA. 3. Learning in a transferable feature space is more flexible and robust than relying on hand-crafted thresholds.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (The combination of tokenization, dual-attention encoder, and pointer-style classification head is highly novel.)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (ID/OOD generalization, ablation, attention visualization, embedding visualization, real-world application, multi-benchmark comparison.)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear problem formulation, detailed method description, and in-depth visual analysis.)
• Value: ⭐⭐⭐⭐ (Significant contribution to the SDDO field; transferable drift detection has broad applicability across domains.)

Related Papers

• [AAAI 2026] Streaming Generated Gaussian Process Experts for Online Learning and Control: Extended Version
• [ICLR 2026] Soft Quality-Diversity Optimization
• [NeurIPS 2025] Learning Generalizable Shape Completion with SIM(3) Equivariance
• [AAAI 2026] Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
• [AAAI 2026] GDBA Revisited: Unleashing the Power of Guided Local Search for Distributed Constraint Optimization