Safe Generalization: Mitigating Catastrophic Forgetting in Single-Source Multi-Organ Segmentation via Collaborative Causal Learning

Conference: ECCV 2026
Paper: ECCV Official
Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-3794.txt
Area: Medical Imaging
Keywords: Single-Source Domain Generalization, Multi-Organ Segmentation, Causal Intervention, Catastrophic Forgetting, Collaborative Learning

TL;DR

To tackle catastrophic forgetting in single-source domain generalization (CF-SDG) where cross-domain training destroys source domain accuracy, this paper proposes CCL-Net, which severs spurious appearance shortcuts via dual-path causal intervention (front-door orthogonal disentanglement and back-door anatomy-guided local intervention) and preserves source knowledge via dynamic local-global collaborative constraints.

Background & Motivation

In clinical practice, deep learning segmentation models frequently experience severe performance collapse when deployed across different medical centers, scanners, or imaging protocols. Constrained by patient privacy laws and expensive manual annotations, clinical practitioners typically only have access to labeled data from a single center during training. This makes Single-Source Domain Generalization (SDG) a highly practical yet formidable task. Existing SDG approaches largely rely on aggressive data augmentation, style transfer, or feature alignment to suppress distribution discrepancies. However, these techniques typically treat SDG as an unconstrained feature-space deformation. In forcing alignment across unseen distributions, they inadvertently corrupt the sharp, fine-grained discriminative features native to the source domain. Consequently, while cross-domain generalization might improve, performance on the original source domain deteriorates substantially. This neglected phenomenon—termed Catastrophic Forgetting in Single-Source Domain Generalization (CF-SDG)—critically impairs model dependability in source clinical environments and introduces substantial safety risks.

From a causal perspective, the fundamental culprit behind CF-SDG is the failure of existing models to learn genuine causal anatomical invariants, instead falling prey to confounding coupling in multi-organ imaging. During image acquisition, underlying anatomical semantics (\(D\)) dictate both organ geometry (\(S\)) and local tissue intensity/appearance (\(A\)). This induces an intrinsic statistical dependency \(S \leftarrow D \rightarrow A\) within the source domain. Conventional empirical risk minimization (ERM) models easily exploit this shortcut by utilizing appearance factor \(A\) to predict organ labels \(Y\). When imaging parameters alter at unseen test sites, this statistical correlation shatters. Furthermore, applying unconditional style transformations causes excessive representation drift, inducing catastrophic forgetting analogous to task-switching in continual learning.

To overcome this dilemma, models must simultaneously sever spurious shortcuts and firmly anchor source domain knowledge. The core idea is to establish a Collaborative Causal Learning Network (CCL-Net) with dual-path causal interventions and collaborative constraints: employing orthogonal structure disentanglement to build a pure structural mediator pathway (front-door), executing mask-guided local style counterfactual interventions to physically sever confounding backdoor paths, and introducing an auxiliary reconstruction branch with dynamic local-global bias constraints to preserve source domain memory.

Method

Overall Architecture

CCL-Net adopts a dual-task collaborative architecture comprising a primary segmentation branch \(\mathcal{S}_{seg}\) and an auxiliary source image reconstruction branch \(\mathcal{S}_{rec}\). Taking a single-source medical image as input, the network first disentangles spatial geometry from global style using Orthogonal Structure Disentanglement (OSD). In the latent space, Anatomy-Guided Local Causal Intervention (AG-LCI) synthesizes counterfactual representations via organ-mask-guided affine transformations and enforces consistency regularization. Meanwhile, a Collaborative Learning Constraint mechanism dynamically coordinates inter-layer features and task gradients between the segmentation and reconstruction branches, guaranteeing both domain-invariant feature extraction and source memory retention.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Medical Image X"] --> Sh["Shared Shallow Encoder E_shared"]
    Sh --> OSD["1. Orthogonal Structure Disentanglement<br/>Dual-branch encoding & soft orthogonality loss"]
    OSD -->|Purified structure representation| LCI["2. Anatomy-Guided Local Causal Intervention<br/>Mask-guided affine style transfer & consistency loss"]
    LCI --> SegHead["Segmentation Branch S_seg<br/>Multi-organ mask prediction"]
    Sh --> RecHead["Reconstruction Branch S_rec<br/>Source image reconstruction"]
    SegHead <-->|Dynamic feature distance monitoring| CL["3. Collaborative Learning Constraint<br/>Local layer factor + Global task direction gain"]
    CL <--> RecHead
    SegHead --> Out["Robust Generalized Segmentation Output"]

Key Designs

1. Orthogonal Structure Disentanglement: Blocking Non-Causal Style via Front-Door Mediation

Standard single-stream encoders inevitably let scanner-specific style and contrast infiltrate deep representations. Inspired by front-door adjustment, CCL-Net treats anatomical structure as a mediator along the causal path from input image \(X\) to label \(Y\), extracting domain-invariant representations. Above a shared shallow feature extractor \(E_{shared}\), the model branches into a structure encoder \(E_s\) producing spatial tensor \(s \in \mathbb{R}^{C \times H \times W}\) and a style encoder \(E_a\) yielding global vector \(a \in \mathbb{R}^L\). To ensure statistical decorrelation in the latent space, structural features undergo global average pooling (GAP) and project through two-layer MLPs alongside style representations into a shared space \(\tilde{s}, \tilde{a} \in \mathbb{R}^d\). The model minimizes a soft orthogonality loss:

\[ \mathcal{L}_{orth} = \frac{1}{B} \sum_{i=1}^{B} \left( \frac{\tilde{s}_i^\top \tilde{a}_i}{\|\tilde{s}_i\|_2 \|\tilde{a}_i\|_2} \right)^2 \]

Minimizing the squared cosine similarity enforces orthogonality between structure and style subspaces, preventing style leakage into the structural pathway and supplying uncontaminated geometrical information for downstream segmentation.

2. Anatomy-Guided Local Causal Intervention: Severing Backdoor Shortcuts via Counterfactual Synthesis

Even with global feature disentanglement, organ-specific intensity characteristics establish local spurious correlations (e.g., a specific organ consistently matching a distinct intensity range). To sever this backdoor dependency and approximate the causal interventional distribution \(P(Y | do(A))\), CCL-Net samples reference images \(x_j\) from the current batch to represent the marginal style distribution. Rather than executing indiscriminate global mixing, it performs anatomy-guided affine transformation strictly using reference organ ground-truth masks \(M_j\). For foreground organ regions belonging to class \(k\) in source feature \(F_i\), channel-wise mean \(\mu_k^{(j)}\) and standard deviation \(\sigma_k^{(j)}\) from reference feature \(F_j\) replace source statistics:

\[ Z_{mix} = \sum_{k=1}^K \mathbf{1}_{\{M_i=k\}} \odot \left( \sigma_k^{(j)} \frac{F_i - \mu_k^{(i)}}{\sigma_k^{(i)}} + \mu_k^{(j)} \right) + (1 - M_{fg}) \odot F_i \]

where \(M_{fg} = \sum_{k=1}^K M_k\) denotes the union of organ masks. Feeding the counterfactual feature \(Z_{mix}\) to the segmentation head and minimizing the consistency loss \(\mathcal{L}_{con} = \|\mathcal{F}_{seg}(F_i) - \mathcal{F}_{seg}(Z_{mix})\|_2^2\) forces the network to output invariant anatomical boundaries regardless of appearance perturbations.

3. Collaborative Learning Constraint: Mitigating CF-SDG via Dynamic Local-Global Regularization

Relying exclusively on causal style perturbation triggers severe representation drift, eroding source domain discriminative capability. CCL-Net incorporates an auxiliary source image reconstruction branch \(\mathcal{S}_{rec}\) as a memory anchor. To prevent destructive interference between reconstruction and generalization gradients, a two-level collaborative constraint mechanism is implemented: - Local Bias Constraint: Different network depths display uneven sensitivity to domain shift: shallow layers carry low-level style vulnerable to perturbation, whereas deep layers encode invariant semantics. The module tracks Euclidean feature distance transitions across \(k\) update steps to determine whether each layer is stabilizing or diverging, generating a binary factor \(f_{layer}^{(l)} \in \{0.0, 1.0\}\) to adaptively toggle the interaction channel between segmentation and reconstruction; a stochastic perturbation with probability \(\delta=0.05\) prevents the state from freezing in local optima. - Global Bias Constraint: When the auxiliary memory task excessively dictates gradient directions, it stifles causal invariance exploration. The global constraint evaluates optimization direction alignment: \(f_{task}^{(l)} = \text{Sigmoid}(-2 \cdot dir_{seg} + dir_{rec})\). When both branches converge congruently (favoring memory consolidation), a positive gain is allocated; when segmentation explores generalization paths, the memory objective is dynamically down-weighted.

Loss & Training

The collaborative feature constraint \(\mathcal{L}_{Co}\) aggregates mean squared errors across intermediate layers weighted by local and global guidance factors:

\[ \mathcal{L}_{Co} = \frac{1}{N} \sum_{l=1}^N f_{layer}^{(l)} f_{task}^{(l)} \frac{1}{M} \sum_{m=1}^M \left( F_{seg, m}^{(t,l)} - F_{rec, m}^{(t,l)} \right)^2 \]

The overall optimization objective combines segmentation Dice loss, reconstruction Structural Similarity (SSIM) loss, soft orthogonality loss \(\mathcal{L}_{orth}\), and intervention consistency loss \(\mathcal{L}_{con}\):

\[ \mathcal{L}_{total} = \mathcal{L}_{seg} + \mathcal{L}_{rec} + \gamma \mathcal{L}_{orth} + \lambda \mathcal{L}_{con} \]

where \(\mathcal{L}_{seg} = \alpha \mathcal{L}_{Dice} + \beta \mathcal{L}_{Co}\) and \(\mathcal{L}_{rec} = \alpha \mathcal{L}_{SSIM} + \beta \mathcal{L}_{Co}\). Hyperparameters are set via grid search to \(\alpha=0.25, \beta=0.25, \gamma=0.6, \lambda=0.4\).

Key Experimental Results

Main Results

CCL-Net was evaluated on cross-modality abdominal segmentation (CT \(\leftrightarrow\) T2-SPIR MRI) and cross-sequence cardiac segmentation (bSSFP \(\leftrightarrow\) LGE). All baseline methods were reproduced using their official repositories.

Migration Scenario	Metric (Dice %)	Ours (CCL-Net)	Second Best (ICMSeg)	Style Baseline (MixStyle)	Gain vs. Competitors
Abdominal CT \(\to\) MRI	Average Dice	86.23 ± 0.59	85.63 ± 1.24	77.41 ± 2.39	+0.60% vs. ICMSeg / +8.82% vs. MixStyle
Abdominal MRI \(\to\) CT	Average Dice	90.05 ± 0.54	88.98 ± 1.18	62.44 ± 4.03	+1.07% vs. ICMSeg / +27.61% vs. MixStyle
Cardiac bSSFP \(\to\) LGE	Average Dice	86.41 ± 0.23	85.00 ± 1.25	74.55 ± 2.41	+1.41% vs. ICMSeg / +11.86% vs. MixStyle
Cardiac LGE \(\to\) bSSFP	Average Dice	88.42 ± 0.22	87.63 ± 1.27	85.33 ± 2.32	+0.79% vs. ICMSeg / +3.09% vs. MixStyle

On fine-grained structures, CCL-Net shows pronounced advantages: in abdominal MRI \(\to\) CT, it attains 90.89% on liver, 86.97% on right kidney, 91.03% on left kidney, and 91.32% on spleen. In cardiac bSSFP \(\to\) LGE, for the delicate myocardium (MYO) which is highly vulnerable to fragmentation, CCL-Net achieves 81.32% Dice, outperforming ICMSeg (80.02%) and MixStyle (63.75%).

Source Forgetting Metric & Ablation Study

The paper evaluates the Source Forgetting Metric (\(SFM = \text{Dice}_{ERM} - \text{Dice}_{Gen}\), where lower is better), quantifying accuracy degradation on the source domain compared to an ERM model trained exclusively on the source.

Source Domain	ERM Baseline Dice (%)	Cutout SFM (%)	MixStyle SFM (%)	ICMSeg SFM (%)	Ours CCL-Net SFM (%)
Abdominal MRI	93.24	14.90 (Dice: 78.34)	14.03 (Dice: 79.21)	5.92 (Dice: 87.32)	1.07 (Dice: 92.17)
Abdominal CT	92.54	9.76 (Dice: 82.78)	11.41 (Dice: 81.13)	6.22 (Dice: 86.32)	2.22 (Dice: 90.32)
Cardiac bSSFP	92.31	11.99 (Dice: 80.32)	11.53 (Dice: 80.78)	9.99 (Dice: 82.32)	3.29 (Dice: 89.02)
Cardiac LGE	89.55	5.73 (Dice: 83.82)	7.22 (Dice: 82.33)	4.13 (Dice: 85.42)	1.26 (Dice: 88.29)

Component ablation progression (Table 4 & Table 5):

Variant	OSD	AG-LCI	Collaborative Constraint (CL)	CT \(\to\) MRI (Dice %)	MRI \(\to\) CT (Dice %)	bSSFP \(\to\) LGE (Dice %)	Avg. SFM (%)
VA1 (Baseline)	\(\times\)	\(\times\)	\(\times\)	55.98	62.15	53.32	-
VA2	\(\checkmark\)	\(\times\)	\(\times\)	59.33	67.34	59.46	-
VA3	\(\times\)	\(\checkmark\)	\(\times\)	60.76	67.98	63.49	-
VA4	\(\times\)	\(\times\)	\(\checkmark\)	70.59	75.34	72.12	-
VA5 (Causal w/o CL)	\(\checkmark\)	\(\checkmark\)	\(\times\)	76.45	78.30	76.45	8.87
VA6	\(\times\)	\(\checkmark\)	\(\checkmark\)	79.12	81.05	80.56	-
VA7	\(\checkmark\)	\(\times\)	\(\checkmark\)	85.34	86.88	85.43	-
VA8 (Ours Full)	\(\checkmark\)	\(\checkmark\)	\(\checkmark\)	86.23	90.05	86.41	1.96

Key Findings

• Causal Intervention Alone Suffers from Source Forgetting: Variant VA5 proves that while front-door and back-door interventions boost generalization to 76.45%~78.30%, they incur an alarming source forgetting rate of 8.87%, confirming that unconstrained destylization distorts source representations.
• Collaborative Constraints Provide Dual Mutual Gains: Adding CL suppresses average SFM from 8.87% down to 1.96% while delivering a massive generalization jump (from 76.45% in VA5 to 86.23% in VA8), demonstrating that retaining source domain anchors actually stabilizes cross-domain feature learning.
• Superior Causal Robustness: CCL-Net exhibits minimal standard deviation across 5 independent runs (0.2%~0.6%), far lower than conventional style augmentation methods (2.0%~4.0%), confirming that eliminating spurious correlations ensures stable optimization.

Highlights & Insights

• Defining Safe Domain Generalization (CF-SDG): Identifies a major blind spot in domain generalization research—the silent sacrifice of source domain accuracy. In high-stakes medical diagnosis, retaining source reliability is vital for clinical safety.
• Mask-Guided Local Counterfactual Intervention: Unlike naive global style augmentation (such as MixStyle) that degrades organ geometries, AG-LCI performs affine statistics transfer strictly within anatomical boundaries, decoupling tissue intensity from structural morphology without blurring edges.
• Dynamic Layer-Wise Collaborative Regulation: Replaces brittle static multi-task loss weighting with an adaptive feature tracking mechanism (\(f_{layer}\) and \(f_{task}\)), dynamically balancing memory retention and generalization exploration.

Limitations & Future Work

• Requirement of Dense Multi-Organ Annotations: AG-LCI relies on ground-truth segmentation masks \(M_i\) to isolate local organ statistics during training. It cannot be applied directly in weakly supervised or partially labeled settings.
• Computational Overhead during Training: Maintaining an auxiliary reconstruction network \(\mathcal{S}_{rec}\) and computing multi-layer Euclidean feature discrepancies increases GPU memory footprint and training latency (though inference remains lightweight since only \(\mathcal{S}_{seg}\) is retained).
• Future Directions: Exploring unsupervised or prototype-driven regional clustering to relax dense mask dependency, and extending the collaborative causal framework to multi-center federated continual learning settings.

Related Work & Insights

• vs. MixStyle / RandConv: Conventional global style perturbation methods alter full-image statistics, severely degrading anatomical boundaries and causing severe source forgetting (SFM up to 11%~15%). CCL-Net limits interventions to local anatomy via masks and blocks style leakage using orthogonal projection, dropping SFM to 1%~3%.
• vs. CSDG / ICMSeg: Earlier causal medical segmentation methods primarily tackle external acquisition confounders or use contrastive alignment, neglecting internal anatomical backdoor shortcuts and generalization-induced forgetting. CCL-Net addresses both front-door and back-door paths while explicitly mitigating CF-SDG via collaborative reconstruction.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates the critical CF-SDG problem and introduces a principled dual-path causal intervention combined with dynamic local-global collaborative constraints.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across multi-modality abdominal and multi-sequence cardiac datasets, thoroughly assessing both cross-domain Dice and source domain forgetting metrics.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous causal formulations, comprehensive structural causal models, and lucid motivation narrative.
• Value: ⭐⭐⭐⭐⭐ Provides an indispensable and safe framework for deploying reliable medical AI models under real-world single-source constraints.

Related Papers

• [CVPR 2025] Human Knowledge Integrated Multi-modal Learning for Single Source Domain Generalization
• [ECCV 2026] A Dual-space Patch-driven Complementary Learning Framework for Semi-supervised Multi-organ Segmentation
• [ICML 2025] LangDAug: Langevin Data Augmentation for Multi-Source Domain Generalization in Medical Image Segmentation
• [CVPR 2026] Multimodal Causality-Driven Representation Learning for Generalizable Medical Image Segmentation
• [ECCV 2026] Unified Multi-Layer Subspace Modeling for Cross-Domain OOD Detection