CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Code: https://github.com/WenDongyp/CL-Anomaly
Area: Model Compression
Keywords: continual learning, anomaly detection, task-private LoRA, mixture of experts, layer-adaptive knowledge transfer

TL;DR

CL-Anomaly adds parallel task-private LoRA and shared experts to a frozen multimodal large language model, then merges old and new shared parameters according to layer-wise task relevance, achieving 85.61% final average accuracy and −0.96 percentage points of backward transfer on an eight-task anomaly detection sequence; its primary contribution is reducing forgetting rather than introducing a new anomaly localizer.

Background & Motivation

Anomaly detection systems continually encounter new products and imaging conditions, sometimes extending from industrial images to medical images and 3D data. Multimodal large language models can answer anomaly-related questions and provide explanations, but full retraining for every new setting is impractical. Retaining and replaying historical data can also face privacy and storage constraints. This paper studies continual instruction tuning on anomaly detection visual question answering data, which should not be conflated with unsupervised anomaly detection trained only on normal examples.

Existing LoRA-based continual learning methods face two contrasting problems. Shared low-rank parameters in MoELoRA and CL-MoE allow related product categories to benefit from one another, but industrial and medical tasks may compete for the same representations. Task isolation prevents old parameters from being overwritten but blocks knowledge exchange. HiDe-LLaVA already combines task-specific expansion with partial fusion. This paper further observes that identically indexed experts at different layers need not serve the same function, and expert activity within a layer changes across tasks, making uniform treatment across layers inadequate.

The issue is therefore not merely retaining old adapters, but deciding which knowledge must remain separate and which task should receive priority in shared updates. Core Idea: preserve knowledge that should not be mixed in task-private LoRA, accumulate transferable representations in shared experts, and use layer-wise routing statistics to set the merging weights for old and new expert parameters.

Method

Overall Architecture

The inputs are visual content and an anomaly-related question; the output is an anomaly judgment or a natural-language explanation. CL-Anomaly freezes the pretrained parameters of LLaVA-OV-7B and adds two low-rank update paths to its language-model linear layers: task-private experts retain separate task parameters, while shared experts use a learned router to capture common knowledge. After training the current task, layer-adaptive knowledge transfer merges shared parameters according to task relevance. This is an update between training tasks, not an extra module executed for every generated answer.

Each adapted layer retains its original linear output and adds the increments from both paths. The paper uses \(h=W_0x+\lambda f_p(x)+(1-\lambda)f_s(x)\), where \(f_p\) and \(f_s\) are the private and shared outputs, respectively, and \(\lambda=0.5\). The combination operates on adaptation increments for the same hidden representation; it does not ask two full models to answer independently and then vote. The paths are parallel rather than sequential.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Visual input and question"] --> B["Frozen multimodal backbone"]
    B --> C["Task-private experts"]
    B --> D["Shared expert routing"]
    D -->|After task training| E["Layer-adaptive knowledge transfer"]
    C --> F["Dual-path increment fusion"]
    D --> F
    E -.->|Update parameters for subsequent use| D
    F --> G["Anomaly judgment and explanation"]

Key Designs

1. Task-private experts: preserve task knowledge in separate parameters

Each task receives a PrivLoRA expert. Training a new task updates only its private low-rank matrices, while previous tasks' private matrices remain frozen. This isolation operates on the parameters of each adapted layer, rather than imposing a soft requirement against forgetting through the training loss. It prevents new-task gradients from directly overwriting old task-specific parameters, but does not guarantee that the whole model never forgets: the shared path and inference-time expert mixing still affect the final answer.

A test image does not require a manually supplied task identifier. During training, the model averages each image's visual patch features and then averages across the task's training images to obtain a task visual anchor. At test time, the same patch-averaging procedure produces an instance anchor. Its cosine similarities to all stored task anchors are passed through softmax to weight the private experts' outputs. Although the paper describes this as expert selection, its computation is a weighted sum over learned experts, not simply choosing the single most similar expert. A test instance can therefore draw on several tasks' private adaptations without this inference-time mixture overwriting their separately trained parameters.

2. Shared expert routing: accumulate knowledge from related tasks in a common representation space

Complete isolation sacrifices positive transfer. For example, different product categories captured under similar imaging conditions can share texture, shape, and defect-recognition patterns. A second path therefore maintains a fixed number of shared LoRA experts, with a trainable gate computing weights from the hidden input and combining expert outputs. Unlike the private path's visual-anchor routing, this router learns assignments directly from the current hidden representation. Shared parameters continue adapting on later tasks, allowing new knowledge to improve representations useful to previous tasks.

The implementation uses \(n=4\) shared experts. Forward routing weights must be distinguished from the key-expert selection described next: the method equations combine shared experts through a weighted sum, whereas subsequent top-\(K\) selection identifies experts deserving special treatment during parameter merging. Consequently, \(K=1\) should not be presented as definitive evidence that only one expert is computed in each forward pass. The private path prevents conflicts in task-specific parameters, while the shared path enables knowledge exchange. It can also complement imperfect visual-anchor routing, but does not eliminate routing errors.

3. Layer-adaptive knowledge transfer: favor old or new parameters according to each layer's task relevance

Ordinary shared experts are repeatedly modified by the current task, so the method first measures importance using gate outputs already produced during training. An expert's task relevance score at a layer is the average routing probability assigned to it across that task's samples. The \(K\) highest-scoring experts in each layer become the task's key experts. These statistics reuse existing forward computations and require no additional forward or backward pass solely to estimate importance. The claimed zero extra cost should be understood in that sense, not as an absence of statistical bookkeeping, storage, or merging overhead.

The method then compares the layer-wise key-expert sets for the current and previous tasks. Experts important only to the current task favor newly trained parameters; experts important only to the previous task favor old parameters. Experts important to both tasks, or neither, use equal merging weights. Let \(\theta_{t-1}^{(l,e)}\) denote the previously accumulated shared-expert parameters and \(\varphi_t^{(l,e)}\) the parameters obtained by fine-tuning that initialization on the current task. The update is:

\[ \theta_t^{(l,e)}=(1-\beta_{l,e})\theta_{t-1}^{(l,e)}+\beta_{l,e}\varphi_t^{(l,e)}. \]

The default base coefficient is \(\beta=0.75\). Current-task-only, previous-task-only, and remaining experts use \(\beta_{l,e}=0.75\), \(\beta_{l,e}=0.25\), and \(\beta_{l,e}=0.5\), respectively. Rather than freezing every important expert, this controls how strongly new parameters can overwrite old ones. Although old parameters accumulate historical tasks, the paper explicitly compares the current and previous tasks' key-expert sets; this should not be expanded into a claim that separate protection coefficients are maintained for every historical task.

A Worked Example

Consider the transition from an industrial-image task to the medical BMAD task. The model adds and trains the medical task's PrivLoRA while freezing existing industrial private experts; its shared path continues training from previous shared parameters. If a shared expert at one layer is important only to the medical task, merging retains 75% of its new parameters and 25% of its old parameters. If it is important only to the previous task, the weights are reversed. This illustrates the paper's default rule rather than reporting an additional experiment.

When an industrial image is encountered again, visual-anchor similarity assigns weights to existing private experts, while the shared path uses the merged parameters. Both increments contribute to the answer. Thus, the old task retains private parameters untouched by medical training while also accessing subsequently updated shared representations. Whether it benefits depends on appropriate routing and whether the shared updates contain transferable information.

Loss & Training

The paper performs continual instruction tuning on anomaly visual question answering, accessing visual inputs, questions, and answers task by task. Only the current private expert and shared path are trained, with the pretrained backbone frozen. The main paper does not specify a new standalone loss term, so distillation, replay, or expert-balancing losses should not be invented. Momentum merging after a task is a parameter operation, not a regularization term.

Experiments use LoRA rank \(r=36\), scaling factor \(\alpha=72\), and \(K=1\) key expert per layer. Each task is trained for 1 epoch with AdamW, a learning rate of \(1\times10^{-5}\), a warm-up ratio of 0.03, and 8 RTX A6000 GPUs. The hyperparameter discussion's reference to a default of 0.5 is potentially confusing in its description of neutral per-expert merging. This note follows the implementation's base \(\beta=0.75\) and the method's case-dependent rule, rather than treating the two symbols as sharing one default.

Key Experimental Results

Main Results

The data come from Anomaly-Instruct-125K. The eight-task main sequence covers three WebAD subsets with 35 classes each, MVTec-AD, BMAD, MVTec-3D, Real3D-AD, and Anomaly-ShapeNet, representing class-incremental, cross-domain, and cross-modal changes. Continual learning baselines use the same LLaVA-OV-7B backbone. Joint training sees all tasks simultaneously without the same ordering constraint and is therefore not an equivalent continual learning baseline.

Avg is the authors' accuracy aggregate across stages; Last is the average task accuracy after all training tasks are complete. Both are percentages, with higher being better. BWT measures how learning later tasks affects earlier-task performance: negative values indicate forgetting, and higher is better, in percentage points. The paper places its exact averaging conventions in Appendix C.1, but the local cache contains only the main paper and references, so an unverified weighting formula for Avg is not supplied here. F1 is the harmonic mean of anomaly precision and recall; the table reports the authors' final-stage aggregates.

Method	Avg ↑ (%)	Last ↑ (%)	BWT ↑ (percentage points)	Final F1 ↑ (%)
LoRA	82.58	77.98	−7.89	74.74
CL-MoE	80.63	78.08	−5.90	73.47
HiDe-LLaVA	77.91	73.32	−6.36	68.61
CL-Anomaly	86.34	85.61	−0.96	82.51

Results come from Tables 1 and 2, with CL-MoE's Last cross-checked against the forward-order result in Table 7. These tables imply improvements over CL-MoE of 5.71, 7.53, and 4.94 percentage points in Avg, Last, and BWT. The first two gains stated in the paper's prose do not fully agree with the tables, so this note uses table-derived differences rather than repeating the stated percentage improvements. Joint training reaches 88.38% final average accuracy, 2.77 percentage points above CL-Anomaly, under a different training protocol.

Ablation Study

Config	Avg ↑ (%)	Last ↑ (%)	BWT ↑ (percentage points)
Full model	86.34	85.61	−0.96
Without PrivLoRA	83.58	81.60	−3.70
Without shared path	84.97	82.63	−3.29
Without layer-adaptive knowledge transfer	85.09	84.06	−1.38

In Table 3, removing PrivLoRA lowers final accuracy by 4.01 percentage points, removing the shared path by 2.98, and removing layer-adaptive knowledge transfer by 1.55. This supports the mechanisms' complementarity, not a claim that their contributions add linearly.

Key Findings

• The private path has the largest effect in these ablations, indicating that shared adaptation alone is insufficient to preserve knowledge across domains and modalities. The benefit from the shared path also shows that complete isolation is insufficient.
• Improvements are not universal across datasets: CL-Anomaly's Last on Anomaly-ShapeNet is 73.64%, below LoRA's 76.36%. Better overall retention should not be presented as superiority on every task.
• Under the random task order, CL-Anomaly achieves 84.84% Last versus 77.99% for CL-MoE. This is evidence from one ordering perturbation, not proof of stability across every permutation or random seed.

Highlights & Insights

• Importance is measured at the layer-and-expert level, rather than assigning one task label to an entire adapter. Identically indexed experts can thus receive different protection at different depths, matching the observed layer-dependent task preferences.
• Isolation and sharing address different error sources rather than replacing one another. Visual anchors determine how private knowledge is combined, hidden-state routing determines how shared knowledge is accessed, and parameter merging controls how shared knowledge changes.

Limitations & Future Work

• The number of private experts grows with the task count. Parameter efficiency does not imply constant storage, and the main paper does not establish bounded storage or inference costs for long task streams.
• A single task visual anchor is an average feature and may inadequately distinguish multimodal within-task distributions. The paper's ablation discussion also acknowledges imperfect anchor routing, so parameter isolation should not be equated with perfect task identification.
• Main results use one instruction dataset and one backbone. They do not directly establish performance under normal-only training, pixel-level localization, or clinical deployment. Expert activation plots for other models are not equivalent to complete cross-backbone performance replications.
• The cache omits the appendices containing exact metric definitions, full data details, and supplementary experiments; equation extraction also contains layout corruption. This note limits its account to mechanisms and results checkable against the main text and tables. Full reproduction still requires consulting the appendices and code.

Related Work & Insights

• vs MoELoRA / CL-MoE: These methods primarily adapt within a shared expert space. CL-Anomaly additionally retains task-private parameters across adapted layers and merges shared parameters according to layer-wise relevance. The main advantage is retention across heterogeneous tasks, at the cost of more task-specific state.
• vs HiDe-LLaVA: HiDe-LLaVA keeps top-layer experts independent while merging non-top-layer experts. CL-Anomaly retains separate private experts at each adapted layer and delegates transfer to a distinct shared path. The difference concerns where isolation occurs and the granularity of shared updates, not simply whether isolation exists.
• vs traditional anomaly detection: Reconstruction-error and feature-matching methods typically output anomaly scores. This paper uses an instruction-based multimodal model to produce judgments and explanations, focusing on continual knowledge updates under different evaluation targets and supervision.

Rating

• Novelty: 4/5. Private/shared dual paths have precedents, while layer-wise key-expert-driven merging makes the combination more targeted.
• Experimental Thoroughness: 4/5. Heterogeneous task sequences, component ablations, and order analysis are included, but the main paper alone cannot verify every reproduction detail.
• Writing Quality: 3/5. The central mechanism is clear, but table/prose gain discrepancies and hyperparameter wording require care.
• Value: 4/5. Useful evidence for parameter-efficient continual learning in anomaly visual question answering, without establishing a replacement for specialized anomaly detection systems.