Skip to content

Optimization-Guided Diffusion for Interactive Scene Generation

Conference: ECCV 2026
Paper: ECCV 2026 / Project page
Area: Autonomous Driving
Keywords: Interactive Scene Generation, Guided Diffusion Sampling, Constrained Optimization, Adversarial Scenario Generation, Zero-Shot Generalization

TL;DR

Without retraining or modifying any diffusion-based scene generator, OMEGA re-anchors the model-predicted clean sample inside every reverse denoising step through a constrained optimization within a KL-bounded trust region, and schedules structural versus interaction guidance via a two-phase (Warm-up / Rolling-Zero) denoising scheme — its adversarial variant ΩAdv further casts ego–attacker interaction as a distributional game approximated by sensitivity-enhanced iterative best response. It lifts the scene validity rate from 32.35% to 72.27% on nuPlan free exploration, from 61.71% to 81.95% on zero-shot Waymo, and from 11% to 80% under goal-conditioned controllability, while generating roughly 5× more near-collision frames with TTC below 3 s without sacrificing realism.

Background & Motivation

Safety evaluation of autonomous driving systems depends on multi-agent driving scenes that cover both everyday traffic and the rare long-tail interactions that actually decide robustness. Rule-based or physics-based simulators (CARLA, SUMO and the like) reproduce simple traffic patterns but script the interactions, producing unnatural behavior, while real datasets (nuScenes, nuPlan, Waymo) simply do not contain safety-critical events at the frequency needed for on-demand sampling. Attention has therefore shifted to data-driven scene generation, which learns agent behavior distributions directly from large-scale driving logs; among these, diffusion models stand out for modeling high-dimensional, multi-modal joint distributions, as exemplified by SceneDiffuser, Nexus, and VBD.

Two shortcomings of existing diffusion-based generators are hard to avoid. First, physical and social constraints are only implicitly encoded in the denoising network and are not guaranteed at inference time: small per-step deviations accumulate along the Markov chain, and the sampler ends up emitting trajectories that are kinematically infeasible, off-road, or mutually colliding — the Nexus backbone reaches only a 32.35% scene validity rate on nuPlan free exploration, leaving two thirds of its samples unusable for evaluation. Second, long-tail adversarial interactions live in low-density regions of naturalistic data, so training barely pushes the model toward them and it gravitates to dominant motion patterns; existing guidance either requires training an auxiliary collision-reward classifier (AdvDiffuser), hand-designs a shaping objective per intent (DiffScene, CTG), or relies on pre-specified goal conditions (Nexus, MotionDiffuser), all of which stay tied to case-by-case domain expertise and scale poorly across scenarios. Inference-time guidance is not clean either: DPS-style methods back-propagate gradients through the denoiser into clean space and depend on a local linearization of the noise–state mapping, yielding inaccurate update directions, whereas GHC-style projection/clipping enforces feasibility at the cost of discontinuities that damage smoothness and distributional consistency.

The angle taken here is that since the posterior of each reverse step is fully determined by the model-predicted clean sample, that quantity should not be treated as a result to accept but as an anchor to optimize. Core idea: rewrite each denoising step as conditional sampling anchored on the clean-sample estimate, solve a structurally constrained optimization inside a KL trust region around that anchor, and feed the solution back into the mean of the reverse transition — while a two-phase denoising schedule staggers "form the motion first, then open up interaction objectives," and a game-theoretic formulation pulls adversarial generation into the same framework.

Method

Overall Architecture

Two preliminaries frame the problem. A scene is a spatiotemporal tensor \(x\in\mathbb{R}^{A\times\mathcal{T}\times D}\) (\(A\) agents, \(\mathcal{T}\) physical timesteps, \(D\)-dimensional state covering position, heading, velocity and physical size), together with a binary validity mask marking which agents exist at which timesteps. Generation follows the multi-agent inpainting formulation: given an inpainting mask \(\tilde m\) and its context values \(\bar x=\tilde m\odot x\) (all agents' past trajectories, or designated future goals for specific agents), the model must complete the valid but unobserved elements, i.e. the future states of existing agents plus newly entering ones.

OMEGA is training-free: it treats any pretrained diffusion scene generator as a black box and only takes over the inference-time sampling procedure. The input is history (plus optional partial future conditions) and the output is the completed multi-agent future; the flow in between is "denoiser predicts a clean sample → constrained optimization inside a KL trust region yields a new anchor → the reverse transition is resampled from that anchor → guidance objectives switch with the phase," looping step by step. The whole thing decomposes into three designs: how to re-anchor, when to apply which guidance, and how to extend the framework into an adversarial generator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["history + future to complete"] --> B["diffusion model predicts<br/>clean sample"]
    B --> C["Optimization-Guided Re-Anchoring<br/>constrained opt. in KL trust region"]
    C -.->|resample and feed back each step| B
    C --> D
    subgraph P["Phase-Aligned Guidance Schedule"]
        direction TB
        D["Warm-up: full-horizon denoising<br/>structural objectives only"]
        E["Rolling-Zero: per-frame denoising<br/>interaction objectives activated"]
        D --> E
    end
    E --> G["realistic interactive scene"]
    E --> H["Sensitivity-Enhanced Adversarial Generation<br/>SE-IBR approximating Nash"]
    H --> I["safety-critical adversarial scene"]

Key Designs

1. Optimization-guided re-anchoring: turning the model-predicted clean sample from an accepted result into the optimum of a constrained problem

In standard sampling, the network's predicted noise \(\epsilon_\theta(x_t,t)\) is converted into a clean-sample estimate \(\tilde x_0\), which is then plugged into the reverse kernel to obtain \(x_{t-1}\). The paper first rewrites this step as an interpretable three-term decomposition, \(x_{t-1}=A_t\tilde x_0+C_t x_t+\sigma_t z\): a regression term that pulls the sample toward the model-predicted clean state, an inertia term that preserves continuity with the current noisy state \(x_t\), and a noise term that maintains sample diversity, with \(A_t,C_t\) fixed by the variance schedule. The crucial observation is that at step \(t\), \(x_t\), \(A_t\) and \(C_t\) are all known constants, so the entire reverse transition distribution \(Q_t(\tilde x_0)\) is characterized by the single quantity \(\tilde x_0\) — it is the anchor that steers the whole Markov chain.

But \(\tilde x_0\) is purely the product of data-driven regression, so it may violate kinematic or social priors, and letting it drift unchecked across steps moves the sample off the manifold of physically and behaviorally plausible trajectories. OMEGA therefore does not let sampling consume \(\tilde x_0\) directly; it solves a constrained optimization with a KL budget in clean space and uses the optimum \(\hat x_0\) to define a new reverse distribution \(P_t(\hat x_0)=\mathcal{N}(A_t\hat x_0+C_t x_t,\sigma_t^2 I)\), bounded in divergence from the original reverse kernel \(Q_t(\tilde x_0)\) by a threshold \(\kappa_t\):

\[ \hat{x}_0=\arg\max_{x}\ \lambda_t\, r(x)-\frac{A_t}{2\sigma_t^{2}}\lVert x-\tilde{x}_0\rVert_2^{2},\quad \text{s.t.}\ \lVert x-\tilde{x}_0\rVert_2\le\sqrt{2\kappa_t}\,\frac{\sigma_t}{\lvert A_t\rvert},\ h(x)=0,\ g(x)\le 0 \]

Substituting the Gaussian forms into the KL constraint turns it into an adaptive Euclidean trust region whose radius scales with the current diffusion variance \(\sigma_t\). This is the crux of the design: early steps have large variance and let the optimization explore a broad feasible set, whereas late steps at low variance confine updates near the learned data manifold, which stabilizes convergence. Inside the objective, \(r(x)\) carries trajectory-level soft preferences (smoothness, comfort), the equality constraints \(h(x)=0\) encode hard relations such as kinematic feasibility, and the inequality constraints \(g(x)\le 0\) encode collision avoidance and road-boundary adherence. A two-dimensional toy experiment (Fig. 3) shows why both halves are needed: with the guidance objective alone, samples are dragged excessively toward the guidance point and the local sampling geometry is distorted; combining the objective with the KL trust region adapts toward the guidance point while suppressing distributional drift. Compared with DPS-style guidance that differentiates through the denoiser, this solves a constrained problem directly in clean space and therefore gets a more reliable update direction; compared with GHC-style projection or clipping, the soft constraints plus trust region introduce no discontinuity.

2. Phase-aligned guidance schedule: shape the motion over the full horizon first, then release interaction constraints frame by frame

Multi-agent scene generation demands both globally plausible motion evolution and fine-grained local reactivity, two heterogeneous and tightly coupled objectives. Enforcing them uniformly throughout denoising amounts to turning the reverse process into a high-dimensional, tightly coupled optimization over all agents and timesteps; worse, before a basic feasible motion structure exists, the relative configurations between agents are themselves unreliable, so interaction objectives are badly conditioned, conflict with motion-feasibility objectives, and burn optimization budget for nothing. The paper therefore splits denoising into a Warm-up phase and a Rolling-Zero phase, and activates guidance objectives in alignment with the phases.

Warm-up starts from the fully corrupted scene and denoises the entire future horizon as a single sequence under a gradually decreasing noise schedule. This global optimization establishes macro-level spatial organization and dynamic plausibility in one shot and, because the horizon is denoised as a whole, it suppresses autoregressive drift and keeps long-horizon trends stable; the phase halts in a low-noise regime, retaining controlled residual uncertainty as a well-posed initialization for refinement. Rolling-Zero then operates in that low-noise regime and continues removing the residual noise under a time-indexed, per-frame schedule: each step refines one frame conditioned on the history already denoised up to that point and on the still partially masked future. Individual frames can thus respond promptly to evolving interactions without destroying the global structure established during Warm-up.

The guidance objectives switch on the same rhythm: during Warm-up only agent-wise structural objectives are active (motion feasibility, road compliance, state consistency, smoothness), which establish a globally plausible motion layout and coarse behavioral intent; only in Rolling-Zero is inter-agent interaction guidance additionally activated, evaluated per timestep on the latest denoised scene state so that it responds in time to evolving multi-agent motion. The payoff is twofold: the joint optimization burden shrinks, and per-agent parallel computation becomes natural.

3. Sensitivity-enhanced adversarial generation: making the attacker another optimizer in a game rather than a hand-specified trajectory

Long-tail safety-critical interactions sit in low-density regions of the data, so neither retrained classifiers nor hand-crafted shaping functions scale; yet maximizing collision outright produces a flood of meaningless passive crashes (rear-ending a stationary vehicle, for instance) that are worthless for training or evaluation. OMEGAAdv casts ego \(e\) and attacker \(a\) as a distributional game: both solve a constrained optimization isomorphic to the re-anchoring problem — the ego maximizes its own objective \(J^e\) under the trust region and structural constraints to stay realistic and feasible, while the attacker maximizes \(J^a-\alpha J^e\) subject to the same constraints (\(\alpha>0\) trades aggressiveness against realism). The two are coupled through a pair of pairwise safety constraints \(\gamma_e(x^e,x^a)\le 0\) and \(\gamma_a(x^a,x^e)\le 0\) that activate only inside their respective responsibility regions — precisely the line between "meaningful adversarial interaction" and "just crashing into something."

Solving the joint optimum of this noncooperative differential game is computationally prohibitive, so the authors approximate the Nash equilibrium with sensitivity-enhanced iterative best response (SE-IBR): starting from the model-predicted anchors \(\tilde x_0^e,\tilde x_0^a\) as initial strategies, the ego and the attacker alternately update their clean anchors as best responses to each other until convergence. The key is how the attacker anticipates the ego's reaction: the Karush–Kuhn–Tucker optimality conditions of the ego's subproblem yield an analytic approximation of how the ego's optimal value responds to the attacker's decision,

\[ \left.\frac{\mathrm{d}J^{e*}}{\mathrm{d}x^{a}}\right|_{x^{a(l-1)}}\approx-\,\mu^{e(l)}\left.\frac{\partial\gamma_e}{\partial x^{a}}\right|_{(x^{a(l-1)},x^{e(l)})} \]

where \(\mu^{e(l)}\ge 0\) is the KKT multiplier of the ego's collision-avoidance constraint. Once that constraint becomes active (\(\mu>0\)), the attacker acquires a directional incentive proportional to \(\partial\gamma_e/\partial x^a\) to shrink the signed distance in the ego–attacker interaction space; the attacker's update thus becomes a joint maximization of its own objective plus this sensitivity term, which in behavior terms means shrinking the ego's feasible maneuvering set to provoke a genuine avoidance response. To make the active regime reachable in practice, attacker routes are biased during Warm-up toward spatial corridors intersecting the ego's responsibility region. Nothing is retrained and no attacker trajectory or target point is prescribed; moreover, by injecting the ego's planned trajectory into the game as a weak-noise condition, the same framework performs planner-specific adversarial generation.

Loss & Training

There is no training step at all: the pretrained backbone diffusion generator (Nexus in the experiments) is kept frozen and OMEGA only rewrites the inference-time sampling operator, so it can be plugged into any diffusion scene generator (the paper also applies it to VBD). The hyper-parameters of each optimization are the guidance weight \(\lambda_t\) and the KL budget \(\kappa_t\) — the latter, after substituting the Gaussian forms, amounts to a trust-region radius proportional to \(\sigma_t\), so in practice one only controls the allowed magnitude of the per-step anchor displacement. Constraint sets are assembled per phase (structural terms during Warm-up, interaction terms added in Rolling-Zero). The adversarial variant adds an aggressiveness weight \(\alpha\) and the responsibility-region definition that decides when the coupling constraints activate. Cost-wise, optimizing an 8-second scene takes about 5 seconds (4.96±3.92 s per sample for the full configuration); the authors note that per-agent parallelization can push the latency down further and enable integration into closed-loop simulation.

Key Experimental Results

Nexus — a diffusion scene generator pretrained on nuPlan — serves as the frozen backbone, yielding Nexus-Ω; baselines include Diffusion Policy, SceneDiffuser and Nexus reproduced on the same dataset, plus variants that swap the guidance for GHC / CTG / DPS. Evaluation uses nuPlan and zero-shot transfer to Waymo (never seen during training), with the adversarial evaluation conducted on nuPlan.

The three capabilities are measured as follows. Realism / physical plausibility uses scene-level statistics — collision rate, off-road rate, kinematic feasibility, and their conjunction, the scene validity rate (no collision, no off-road and kinematic feasibility all hold) — plus distribution-level JSDs against the ground-truth distribution (nearest-agent distance N-Dist, lateral deviation L-Dev, angular deviation A-Dev, speed Spd). Interactivity uses the ego's risk-exposure distribution — mean per-frame TTC, the fraction of frames with TTC below 1/2/3 s, ego acceleration and jerk, and the ego non-responsible collision rate (Ego NC) — together with the Kinematics / Interaction / Map-Based terms and the Meta score of the Waymo Open Sim Agents protocol. Controllability is probed by specifying a goal endpoint for a target agent and adjusting the inpainting mask accordingly, then checking whether the model satisfies that condition while keeping the scene realistic and feasible.

Main Results

Scene-level metrics for the three settings (P-Sc = per-scene; ↓ lower is better, ↑ higher is better):

Setting Method Collision (%) ↓ Off-road (%) ↓ Kinematic Feas. (%) ↑ Scene Valid (%) ↑
nuPlan free exploration Oracle (GT logs) 13.62 0.00 81.22 71.67
nuPlan free exploration Diffusion Policy 36.58 39.47 87.18 34.49
nuPlan free exploration SceneDiffuser 37.80 41.68 86.96 33.27
nuPlan free exploration Nexus (backbone) 43.12 40.13 88.63 32.35
nuPlan free exploration Nexus-Ω (Ours) 17.36 10.39 91.91 72.27
Waymo zero-shot Oracle 7.47 0.00 96.33 89.33
Waymo zero-shot Diffusion Policy 19.23 11.35 86.74 64.55
Waymo zero-shot SceneDiffuser 22.94 12.18 86.57 61.01
Waymo zero-shot Nexus (backbone) 22.65 12.30 86.08 61.71
Waymo zero-shot Nexus-Ω (Ours) 11.43 4.32 92.65 81.95
nuPlan goal-conditioned Diffusion Policy 40.00 46.00 58.00 20.00
nuPlan goal-conditioned SceneDiffuser 46.00 40.00 59.00 17.00
nuPlan goal-conditioned Nexus (backbone) 51.00 51.00 55.00 11.00
nuPlan goal-conditioned Nexus-Ω (Ours) 12.00 7.00 91.00 80.00

On distribution-level metrics for nuPlan free exploration, N-Dist / L-Dev / A-Dev / Spd (JSD, \(10^{-3}\), lower is better) drop from 1.162 / 0.041 / 0.461 / 1.018 for Nexus to 0.203 / 0.025 / 0.081 / 0.333, i.e. closer to the ground-truth distribution than the physical metrics alone suggest. Under the Waymo Open Sim Agents protocol, the same guidance improves both generators: Nexus's Meta rises from 0.609 to 0.700 (the largest gains come from the Interaction collision term, 0.763→0.884, and the Map-Based off-road term, 0.683→0.919, at the cost of a small drop in linearity-type kinematics terms such as Lin. Acc. 0.377→0.321), while VBD moves from 0.723 to 0.730 — VBD already starts high so its headroom is small, whereas Nexus gains a lot, showing that the guidance mainly repairs interaction and map-based realism under zero-shot shift.

Ablation Study

Guidance-method comparison and component ablation under free exploration (scene-level %):

Config Collision ↓ Off-road ↓ Kinematic Feas. ↑ Scene Valid ↑ Note
Nexus (no guidance) 43.12 40.13 88.63 32.35 backbone
Nexus-GHC 49.05 11.57 50.29 26.06 projection/clipping geometry, discontinuous
Nexus-CTG 26.02 9.97 81.72 50.41 conditional score approximation
Nexus-DPS 27.25 11.34 83.55 47.19 clean-space gradient guidance
w/o Rolling-Zero 34.35 6.64 96.72 60.49 smoother trajectories but weak reactivity; collisions rise
w/o Warm-up 76.90 98.95 0.19 0.00 autoregressive error accumulation, total collapse
w/o Guid. Obj. 41.38 39.72 85.69 33.47 two-phase schedule only, barely above the backbone
w/o Phase Align. 31.54 13.03 75.73 52.05 only half the gain, at 7.85× inference time
w/o KL cons. 63.40 46.38 51.95 18.21 distributional realism and stability both collapse
Nexus-Ω (full) 17.36 10.39 91.91 72.27 —

Main adversarial generation results on nuPlan (against the ground-truth Oracle):

Method Mean TTC (s) ↓ TTC<1s (%) ↑ TTC<3s (%) ↑ Mean Acc. ↑ Mean Jerk ↑ Ego NC (%) ↓ Off-road (%) ↓ Kinematic Feas. (%) ↑
Oracle (GT) 4.904 0.328 2.212 0.360 0.307 0.97 0.00 81.22
Nexus-FT (finetuned on adversarial scenes) 4.695 3.239 7.214 0.355 0.477 9.91 39.77 87.08
Nexus-GC (goal-attacking condition) 4.631 3.462 8.677 0.631 1.384 10.98 45.14 45.16
Nexus-CTGAdv 4.575 3.541 8.747 0.794 2.145 7.12 26.77 85.64
Nexus-ΩAdv (Ours) 4.516 3.450 11.599 0.859 1.972 5.42 15.82 88.98

Note: the acceleration/jerk units are corrupted by OCR in the cached text (rendered as m2/s and m3/s), ⚠️ refer to the original paper; only relative magnitudes are compared here. Ego NC is the ego non-responsible collision rate — lower means the generated collisions are more often ego-responsible interactions rather than passive rear-endings.

Key Findings

  • The guidance objective, not the noise schedule, is what matters. Keeping the two-phase Warm-up/Rolling-Zero schedule but dropping the guidance objectives yields a 33.47% validity rate, essentially unchanged from the backbone's 32.35% — phase-wise denoising by itself buys no physical plausibility; the jump to 72% comes from solving a constrained optimization at every step inside the trust region.
  • The KL constraint is the stability linchpin. Removing it sends the collision rate from 17.36% to 63.40% and the validity rate down to 18.21%, which is worse than using no guidance at all — confirming that maximizing the guidance objective in clean space distorts the sampling geometry (the same phenomenon as in the toy experiment) and must be wrapped in a per-step displacement bound that scales with \(\sigma_t\).
  • Both phases are necessary, and asymmetrically so. Dropping Rolling-Zero produces smoother trajectories with the lowest off-road rate (6.64%) but pushes collisions back up to 34.35%, showing that the per-frame phase is exactly what buys timely reaction to interactions; dropping Warm-up is the worst configuration by far (98.95% off-road, 0.00% valid), because per-frame autoregression with no global structure to anchor it accumulates error until it collapses.
  • Phase alignment also saves compute. Without it a good part of the gain survives (52.05% valid) but inference time per 8-second scene grows from 4.96±3.92 s to 38.94±39.79 s (7.85×) with far larger variance — staggering interaction and structural objectives genuinely alleviates the ill-conditioning of the joint optimization.
  • Adversarial strength and realism improve together. ΩAdv reaches an 11.599% share of frames with TTC below 3 s (versus 2.212% for GT, about 5×) while keeping the off-road rate at 15.82% — far below Nexus-GC's 45.14% — and the highest kinematic feasibility at 88.98%; Ego NC drops from 9.91% for Nexus-FT to 5.42%, indicating that the generated collisions are ego-responsible interactions rather than passive impacts. Putting an ego planner (Replay / CV / IDM) in the loop is consistent: scene minimum TTC falls from 4.97/4.79/4.89 s to 4.64/4.61/4.64 s, ego-at-fault collision rates rise from 0.02%/8.00%/1.34% to 19.50%/17.25%/15.62%, and non-at-fault rates stay low.
  • Zero-shot transfer is the most valuable property here. The backbone is trained only on nuPlan, yet on Waymo the validity rate rises from 61.71% to 81.95% (+20.24 points) with distributional JSDs falling in tandem; guidance acts precisely on interaction and map-compliance, the dimensions most fragile under dataset shift, so the transfer gain is larger than the in-domain one.

Highlights & Insights

  • Turning "the per-step posterior is fully determined by the clean-sample estimate" into an exploitable degree of freedom. This is not an extra loss term but a redefinition of where the mean of the reverse transition comes from: the model supplies the direction, the optimization supplies the constraints, and the KL budget sets how much say each side gets. Any generative task where constraints learned at training time are not guaranteed at inference (human motion, robot trajectories, molecular conformations) can reuse this "re-anchor inside a trust region" recipe.
  • The trust-region radius scaling with \(\sigma_t\) is a neat trick. It produces the usual "explore early, hug the manifold late" behavior — normally obtained by tuning warm-up steps or learning rates — for free out of the diffusion variance, with almost no extra hyper-parameters.
  • Phase alignment treats "when to apply which constraint" as an optimization variable. The underlying observation, that interaction objectives are ill-conditioned before motion structure has formed, matters more than the two-phase split itself: it explains why naively applying everything at every step is both 7.85× slower and roughly half as good.
  • SE-IBR converts "how will the opponent react" into a differentiable gradient via KKT multipliers. The attacker does not blindly maximize collision but pushes in the direction that shrinks the ego's feasible maneuvering set; the activated constraint multiplier \(\mu\) doubles as a natural switch for "is this interaction worth attacking," and it incidentally avoids meaningless passive collisions. This "replace opponent modeling with the sensitivity of optimality conditions" idea transfers to any generative task where one party must provoke a reaction from another (curriculum-style data augmentation, robustness test-case generation).
  • The whole method is a training-free plug-in. The same guidance improves both Nexus (zero-shot) and VBD, showing that it acts on the sampling operator and is decoupled from the backbone's representation — a much cheaper deployment path than retraining or fine-tuning.

Limitations & Future Work

  • The limitation the authors acknowledge is runtime: optimizing one 8-second scene takes about 5 seconds, which rules out direct use in closed-loop simulation. Their proposed direction is per-agent parallelization, left as future work. For large-scale evaluation-set generation this cost remains high (4.96±3.92 s per sample in the ablation, with non-trivial variance).
  • The method's hyper-parameters and constraints are hand-assembled per scenario: the smoothness/comfort preferences in \(r(x)\), the kinematic relations in \(h(x)\), the collision and boundary terms in \(g(x)\), plus the responsibility region and the attacker's initial corridor, all require task-specific setup. The paper gives no general form or automatic calibration for these terms, which sits somewhat in tension with its own critique that prior guidance methods rely on case-by-case domain expertise.
  • The adversarial metrics are largely bespoke (TTC distribution, acceleration/jerk, Ego NC, off-road rate) and are not tied to a public collision-responsibility protocol; the "more meaningful collisions" claim rests on the single Ego NC indicator, so robustness of that conclusion under a different responsibility criterion is worth verifying.
  • The experiments cover two large-scale benchmarks (nuPlan, Waymo) and one backbone family (Nexus, VBD) but report no variance across random seeds or repeated sampling except for inference time; the percentage-based small-sample setting of the controllability table in particular could fluctuate at the ±1 point level.
  • Concrete improvements: make the trust-region budget \(\kappa_t\) and the guidance weight \(\lambda_t\) adaptive schedules rather than fixed values; parallelize the per-frame Rolling-Zero phase in blocks to compress those 5 seconds; extend the two-player game to multiple attackers and multiple egos, and check whether the responsibility-region definition still holds at complex intersections.
  • vs Nexus (the backbone here): Nexus takes the "corner-case finetuning + manually specified goal-conditioned inpainting" route, relying on finetuning to learn long-tail behavior and on human-specified conditions to steer a scene toward a given outcome; this paper leaves weights untouched, performs constrained optimization at inference time, and lets a game generate attacker behavior automatically. Advantages: training-free, reusable across backbones, much more precise controllability (80% vs 11% validity); drawback: it solves an optimization online, so a single generation is slower.
  • vs SceneDiffuser / GHC-style geometric constraints: they project or clip trajectories back into the feasible set, which is simple and direct but introduces discontinuities and hurts distributional consistency (26.06% validity under free exploration, below the unguided backbone). This paper writes constraint satisfaction and distributional consistency into a single optimization with a KL trust region — soft but bounded.
  • vs CTG / DPS-style one-step guidance: they correct the reverse step with gradients through the denoiser (or an approximated conditional score), depending on local linearization of the noise–state mapping and often getting the update direction wrong; this paper solves directly in clean space and quantifies the deviation from the original reverse kernel as an explicit KL budget (validity 50.41% / 47.19% vs 72.27%).
  • vs AdvDiffuser: it injects gradients from an extra collision-reward model into denoising, requiring an auxiliary network to be trained; ΩAdv needs no auxiliary model and instead writes the ego–attacker interdependence as a game, obtaining the attack direction analytically from KKT sensitivity.
  • vs KING / AdvSim: they perturb trajectories via gradients or re-simulate sensor data to induce critical states, balancing perturbation magnitude against realism through engineering; ΩAdv's perturbation is explicitly confined to the learned distribution by the KL trust region, so what it generates is "in-distribution yet dangerous."
  • vs CAT-style adversarial training: those methods train surrounding agents to attack the ego at the cost of retraining a policy; this one is training-free and plug-and-play, and the same framework can tailor attacks to a specified ego planner (Replay / CV / IDM).

Rating

  • Novelty: ⭐⭐⭐⭐ Casting "clean-sample anchor + KL trust-region constrained optimization" as a general diffusion sampling operator, and using KKT sensitivity for the adversarial game, is clean and uncommon; but re-anchoring, trust regions and SE-IBR each have their own lineage, making this a compositional contribution.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Broad coverage — nuPlan, zero-shot Waymo, the Sim Agents protocol, two backbones and a full ablation; marks deducted for missing multi-seed variance and a public responsibility protocol, with the adversarial metrics largely bespoke.
  • Writing Quality: ⭐⭐⭐⭐ The three-term update decomposition, the link between the trust region and \(\sigma_t\), and the necessity of phase alignment are all explained clearly, and the toy experiment makes the KL constraint intuitive; formulas are heavily corrupted in the cached version (⚠️ refer to the original paper).
  • Value: ⭐⭐⭐⭐⭐ Training-free and pluggable into any diffusion generator, covering validity, controllability and adversarial generation in one shot — directly usable by teams working on closed-loop simulation and safety evaluation.