Deconfounded Lifelong Learning for Autonomous Driving via Dynamic Knowledge Spaces¶

Conference: ECCV2026
arXiv: 2603.14354
Code: To be confirmed
Area: Autonomous Driving
Keywords: Lifelong Learning, Causal Inference, Front-Door Adjustment, DPMM, End-to-End Autonomous Driving

TL;DR¶

DeLL introduces lifelong learning to closed-loop end-to-end autonomous driving for the first time. It utilizes a Dirichlet Process Mixture Model (DPMM) to construct dual dynamic knowledge spaces (explicit and implicit) and uses dynamically growing knowledge anchors as mediator variables to achieve deconfounding via front-door adjustment, achieving superior lifelong learning performance over PackNet and ER on Bench2Drive with zero extra storage overhead.

Background & Motivation¶

End-to-end autonomous driving has made significant progress in closed-loop CARLA simulations, with methods like Transfuser++, UniAD, and VAD continuously setting new driving score records using cross-modal attention, auxiliary tasks, and Transformer architectures. However, when models must be continuously deployed in non-stationary open-world environments, two fundamental issues arise. The first is catastrophic forgetting: behavior cloning architectures are essentially correlation engines; as they learn new driving capabilities, old ones rapidly degrade as weights are overwritten by new data. Existing lifelong learning methods either require additional storage for historical data (Experience Replay, ER) or require network pruning and retraining (PackNet requires $2\times$ training time). More critically, they remain at the level of "passively preventing parameter overwriting"—knowledge is stacked statically rather than growing dynamically. The second is causal confusion: driving is a partially observable Markov decision process (POMDP) where unobserved confounding variables like sensor noise and environmental changes simultaneously affect perception and decision-making, generating spurious correlations (e.g., "stopping completely upon seeing a stationary vehicle" instead of dynamic car-following). Existing methods have yet to simultaneously address these two intertwined issues.

The core insight of this paper is that knowledge organization and causal deconfounding can be achieved through the same mechanism: using the online clustering results of new scenarios both as a means of knowledge preservation and as a mediator variable for causal inference. Core Idea: This paper proposes the DeLL framework, which utilizes a non-parametric Bayesian DPMM to construct an implicit feature knowledge space and an explicit trajectory knowledge space simultaneously, allowing knowledge clusters to grow autonomously with new scenarios rather than relying on a pre-defined number. The knowledge anchors produced by the DPMM serve as mediator variables to block the backdoor paths of unobserved confounders via front-door adjustment, and an evolutionary trajectory decoder is designed to implement non-autoregressive parallel planning compatible with the dynamic knowledge spaces.

Method¶

Overall Architecture¶

The architecture of DeLL processes multimodal inputs to the final trajectory outputs through four core phases: multimodal perception backbone extracting scenario features $\rightarrow$ DPMM constructing dual dynamic knowledge spaces and producing knowledge anchors $\rightarrow$ Causal Feature Enhancement Module deconfounding through front-door adjustment $\rightarrow$ Evolutionary Trajectory Decoder generating the final path. The key feature is that, upon encountering a new scenario, the DPMM autonomously decides whether to create a new cluster. The knowledge anchors continuously accumulate and are injected back into the forward pass, enabling efficient lifelong knowledge transfer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs"] --> B["Transfuser++ Backbone<br/>→ Fused Scenario Representation"]
    B --> C["DPMM Dual Dynamic Knowledge Space<br/>Feature Space + Trajectory Space Clustering"]
    C -->|Knowledge Anchors as Mediator Variables| D["Causal Feature Enhancement Module<br/>FFEM + TFEM Front-Door Adjustment"]
    D --> E["Evolutionary Trajectory Decoder<br/>Dual-Branch Non-Autoregressive Prediction"]
    E --> F["Final Trajectory"]

Key Designs¶

1. DPMM Dual Dynamic Knowledge Spaces: Letting Knowledge Clusters Grow Autonomously with Data

Traditional lifelong learning methods require predefined task boundaries or knowledge cluster capacities. However, in the open-world scenarios of autonomous driving, when new driving patterns emerge and how many categories exist are unknown. The key innovation of DeLL is replacing fixed-capacity clustering with a Dirichlet Process Mixture Model (DPMM). DPMM is a non-parametric Bayesian model whose core Dirichlet process prior allows data to originate from an infinite number of latent clusters. If a new data point does not match any existing cluster, a new one is automatically created, perfectly adapted to the requirement in lifelong learning where the number of clusters is unknown and continuously growing. The inference of DPMM employs memoVB (memoized variational Bayes) for online coordinate ascent updates, incorporating birth and merge heuristics to dynamically adjust cluster counts and escape local optima.

DeLL instantiates DPMM knowledge spaces at two levels. The Feature Knowledge Space (FKS) is an implicit space that performs online clustering on the fused scenario representations $F_{fused} \in \mathbb{R}^{11 \times 256}$ extracted by the backbone network. The center of each cluster serves as a feature knowledge anchor $A_{feat} \in \mathbb{R}^{K_f \times 2816}$. These anchors implicitly encode latent causal topologies in the environment, such as intersection types, traffic density patterns, and weather conditions. The Trajectory Knowledge Space (TKS) is an explicit kinematic space that applies DPMM clustering directly to expert trajectories in historical training data, constructing a physical prior action library. The cluster centers serve as trajectory knowledge anchors $A_{traj} \in \mathbb{R}^{K_t \times 20}$, covering specific driving maneuver patterns like lane keeping, lane changing, overtaking, and sharp turns. The cluster numbers $K_f$ and $K_t$ of both spaces grow automatically with the learning process. During training, DPMM and neural networks are updated alternately: the current batch of data is first used to update the DPMM cluster structure, and then the updated cluster centers assist in network training.

2. Causal Feature Enhancement Module: Front-Door Adjustment Blocking Spurious Correlations

In driving decision-making, there are unobserved confounding variables $U$ (such as sensor noise and lighting changes) that simultaneously affect the perception input $X$ and decision $Y$, opening the backdoor path $X \leftarrow U \rightarrow Y$. Standard backdoor adjustment requires observing all values of $U$, which is impossible in autonomous driving. DeLL cleverly utilizes the knowledge anchors produced by DPMM as a mediator variable $M$ to block the confounding path via front-door adjustment. The core formula of front-door adjustment is: $$P(Y=y|do(X=x)) = \sum_m P(m|x) \sum_{x'} P(y|x',m) P(x')$$ Its condition holds true if there exists a front-door path $X \rightarrow M \rightarrow Y$ and there is no unblocked backdoor path from $M$ to $Y$.

This causal intervention is decomposed into two cascaded sub-modules using a unified dual-attention and gated-fusion architecture. The Fused Feature Enhancement Module (FFEM) receives the raw fused features $F_{fused}$ and feature knowledge anchors $A_{feat}$. It first projects $A_{feat}$ into a latent space and then computes cross-attention with the input features as queries (Q) and the anchors as keys/values (K/V). This implements the $\sum_m P(m|x)$ term in the formula, identifying the historical causal templates that best match the current scenario. Finally, a sigmoid gating network adaptively calculates fusion weights to smoothly integrate the raw and causally-enhanced features. The Trajectory Feature Enhancement Module (TFEM) receives the subset of the FFEM output responsible for trajectory prediction along with the trajectory anchors $A_{traj}$. After positional encoding and temporal expansion, it repeats the same cross-attention and gating mechanism to output trajectory features embedded with causal kinematic constraints.

3. Evolutionary Trajectory Decoder: Non-Autoregressive Parallel Planning with Knowledge Adaptation

Traditional planning decoder output dimensions are fixed before training, failing to address the contradiction of continuously growing trajectory patterns in lifelong learning. The evolutionary decoder solves this via two designs. First, utilizing the sequence flexibility of Transformers, it maps the dynamically growing trajectory anchors $A_{traj}$ into a dynamically expandable planning token pool via a temporal embedding network. As DPMM naturally increases $K_t$, the token pool expands accordingly, allowing unbounded knowledge acquisition. These tokens act as queries in cross-attention with the scenario context, evaluating the matching degree between historical driving patterns and the current scenario in parallel. Second, the decoder employs a dual-branch decoupled prediction head: a coarse-grained branch computes the selection score of each anchor $Y_{logits} \in \mathbb{R}^{K_t}$, converted to a probability distribution using a temperature Softmax; a fine-grained branch predicts geometric offsets $Y_{offsets} \in \mathbb{R}^{K_t \times 20}$ to correct anchor coordinates. The final execution trajectory is selected via Top-K routing: $$Y^{traj} = Y^{trajs}_{TopK(Y^{probs}, k)}$$ where $Y^{trajs}$ represents the set of candidate trajectories after applying offsets to the anchors. This anchor-selection-based approach is naturally compatible with lifelong learning—when a new scenario generates a new trajectory anchor, it only requires inserting an additional candidate vector into the pool without modifying any existing parameters.

Loss & Training¶

The total loss is formulated as $L = L_{sem} + L_{det} + L_{traj} + L_{speed}$, where the BEV semantic segmentation and speed branches use cross-entropy, and BEV detection follows the CenterNet paradigm. The trajectory loss comprises three components: $L_{prob}$ utilizes KL divergence to align the predicted anchor selection probabilities with soft labels defined by ground-truth (GT) distance; $L_{best}$ measures the deviation of the best-predicted trajectory from the GT using Smooth L1 loss; and $L_{weighted}$ calculates the sum of Smooth L1 losses of all candidate trajectories weighted by GT probabilities. Training is conducted in two phases: first training only the backbone, then joint training of the entire model (30 epochs each). During lifelong learning, the first task undergoes both phases, while subsequent tasks freeze the backbone, reset the optimizer, and train in a single phase for 30 epochs.

Key Experimental Results¶

Main Results¶

Metric	Baseline (TF++)	DeLL (Ours)	Gain
Avg Driving Score ↑	70.55	74.69	+4.14
Avg Success Rate ↑	44.54	50.73	+6.19
Avg Multi-Ab SR ↑	45.23	52.08	+6.85
Forgetting Ratio ↓	44.50	33.97	-10.53
Process FR ↓	40.25	29.80	-10.45
Backward Transfer ↑	52.83	79.63	+26.80
Forward Transfer ↑	41.11	42.88	+1.77

On Bench2Drive, DeLL significantly outperforms the baseline across all sequential tasks. After the final task, it maintains a 68.97% driving score and a 42.73% success rate on the data-sparse GiveWay task, whereas the baseline degrades to 60.89% / 30%. Compared to alternative methods, ER requires a $1.35\times$ data buffer size and PackNet requires twice the training time, while DeLL achieves superior comprehensive performance with zero extra storage and comparable training cost. Under the multi-task (all-data) learning setting, DeLL outperforms all competing methods with an 86.86% DS and a 68.9% average multi-ability success rate, confirming that innovations in lifelong learning architectures also yield gains in static training settings.

Ablation Study¶

Configuration	Avg DS	Avg SR	FR ↓	BT ↑
Baseline (TF++)	70.55	44.54	44.50	52.83
w/o Evolutionary Decoder	72.94	49.82	33.12	72.21
w/o TFEM	73.00	48.36	36.43	77.14
w/o FFEM	73.10	49.33	38.33	77.32
Full Model	74.69	50.73	33.97	79.63

Key Findings¶

Feature deconfounding (FFEM) is critical for forward transfer: Removing FFEM drops FT drastically from 42.88% to 36.59%, proving that deconfounding is indispensable for generalizing to new tasks.
Trajectory knowledge space drives backward transfer: Excluding the evolutionary decoder drops BT from 79.63% to 72.21%, indicating that the dynamically growing trajectory anchor pool is the primary engine for knowledge preservation.
Causal kinematic constraints mitigate process forgetting: Without TFEM, PFR rises from 29.8% to 32.86%, showing that front-door adjustment constraints at the trajectory level effectively suppress disruptive oscillations.
Qualitative Visualization: After sequential learning, the baseline mistakenly learns the correlation "zero speed = stationary obstacle ahead", whereas DeLL maintains correct speed control, demonstrating that causal interventions successfully eliminate spurious causal links.

Highlights & Insights¶

DPMM serves both lifelong learning and causal inference: The clustering output acts as both a means of knowledge organization (dynamically expanding clusters) and naturally as a mediator variable for front-door adjustment—elegantly sharing a single mechanism for two objectives.
Discretization of knowledge anchors clears engineering obstacles for front-door adjustment: Materializing $M$ into a discrete anchor space allows $\sum_m P(m|x)$ to be computed efficiently via cross-attention.
Topological scalability of the evolutionary decoder: Accommodating new trajectory patterns only requires expanding the anchor pool with an additional candidate vector without modifying parameters, which naturally aligns with DPMM.
First closed-loop E2E-AD lifelong learning benchmark: Establishes a three-dimensional evaluation framework consisting of vertical (FR/PFR), horizontal (FT/BT), and comprehensive metrics, offering significant value in filling this research gap.
Positive synergy with multi-task learning: Both ablation and full-data experiments confirm that the architecture tailored for lifelong learning also achieves optimal performance in static training, indicating that a robust lifelong learning method inherently serves as an effective representation learning method.

Limitations & Future Work¶

Alternating DPMM updates increase training overhead: Future research could explore end-to-end joint optimization or more efficient variational inference techniques.
Simulation-to-real domain gap: Discrepancies in scenario distributions and sensor noise still exist between CARLA and the real world.
High dimensionality of feature anchors: $A_{feat} \in \mathbb{R}^{K_f \times 2816}$ may pose a storage bottleneck as $K_f$ grows.
Ego-vehicle centric limitation: Lifelong learning in multi-vehicle cooperative or vehicle-infrastructure cooperative scenarios remains unexplored.

vs PackNet / ER: PackNet requires $2\times$ training time and ER requires $1.35\times$ data storage, while DeLL achieves superior forgetting mitigation and forward transfer with zero extra overhead.
vs Causal AD (GOAT, etc.): Causal driving works mainly focus on deconfounding within a single scenario. DeLL is the first to couple causal intervention with lifelong learning, achieving time-evolving causal representations.
vs Traditional Lifelong Learning: Replacing fixed-cluster clustering methods with non-parametric Bayesian DPMM is inherently more suitable for open-world deployments.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Coupling DPMM with front-door adjustment for autonomous driving lifelong learning is proposed for the first time, with an elegant dual knowledge space architecture.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Double settings (lifelong/all-data), complete ablation studies, and a new evaluation protocol cover qualitative and quantitative analysis comprehensively.
Writing Quality: ⭐⭐⭐⭐ The methodology is clearly described, with rich visualizations (clustering projections, qualitative comparison charts); the DPMM background is slightly lengthy but reader-friendly.
Value: ⭐⭐⭐⭐⭐ Fills the gap in closed-loop E2E-AD lifelong learning, offering valuable insights into the intersection of causal inference and lifelong learning.