Agent-OBJ: Prompt-Driven 3D Adversaries for Multi-Modal Perception¶

Conference: ECCV2026
Paper: Official paper page / PDF
Area: Autonomous Driving / Multimodal Perception Security
Keywords: Camera-LiDAR fusion, 3D adversarial objects, prompt-driven agents, multi-view consistency, black-box optimization

TL;DR¶

Agent-OBJ searches language prompts controlling the shape and appearance of a 3D object using only detection boxes and confidence scores, combining naturalness and cross-view stability constraints to achieve 64.0% joint disappearance from camera and fusion detectors in simulated nuScenes insertion experiments.

Background & Motivation¶

Cameras capture texture and color, while LiDAR supplies geometry and range, so fusion is expected to correct errors from either sensor. Redundancy does not, however, guarantee independent errors: both sensors observe the same object, whose appearance and geometry may simultaneously differ from the pedestrian patterns learned by their detectors. This paper studies such object-level common failures, rather than adding unrelated noise to images and point clouds.

Image patches primarily affect visual perception, point-cloud attacks primarily manipulate geometry, and several 3D approaches require white-box gradients or differentiable rendering. Searching a large number of 3D parameters through final detection outputs alone is expensive. A shape optimized for one view may become detectable after rotation, while arbitrary geometric distortion can increase disappearance without producing a plausible pedestrian, weakening claims about physical relevance.

The paper treats pretrained generators as a constrained search space in which geometry and appearance changes have interpretable semantics. Detector scores, multiple views, and naturalness feedback enter the same decision loop. Core idea: replace high-dimensional 3D parameter search with semantic and style prompt edits, selecting objects through joint confidence feedback, pedestrian priors, and cross-view stability in a unified camera-LiDAR observation.

Method¶

Overall Architecture¶

Inputs are a scene, a fixed pedestrian base prompt, and editable semantic/style prompts; the output is the 3DGS object with the lowest joint objective within the query budget. A Qwen2.5 agent proposes prompt edits, multi-view generation and personalization produce consistent appearance, and LGM reconstructs the object. Calibration-consistent camera and LiDAR insertion then supplies observations to YOLOv10x and MambaFusion. Scores and regularizers return to the agent to guide another edit.

The threat model concerns a single object at a fixed position and pedestrian disappearance, without access to detector gradients, logits, or intermediate features. This black-box restriction applies to the victim detectors. The naturalness constraint still uses a separate PointNet encoder and nuScenes pedestrian instances; the method is not free of all data priors.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Scene and pedestrian base prompt"] --> B["Decoupled Prompts and<br/>Multi-view Construction"]
    B --> C["Calibration-consistent<br/>Dual-modal Evaluation"]
    C --> D["Joint Objective and<br/>Naturalness Constraints"]
    D --> E["Budgeted Agent Search"]
    E -->|Edit semantic or style prompt| B
    E -->|Budget exhausted| F["Return best-so-far object"]

Key Designs¶

1. Decoupled Prompts and Multi-view Construction: make shape and appearance interpretable search directions

The base prompt preserves pedestrian identity, the semantic prompt specifies structural attributes such as carried items, and the style prompt controls clothing, color, and material. A text-to-multi-view generator first produces geometrically consistent images from the base and semantic prompts. Pretrained MV-Adapter then personalizes their appearance using the style prompt, and LGM reconstructs a 3DGS object. The paper does not clearly identify the particular text-to-multi-view generator, so the entire pipeline should not be attributed to LGM.

This separation does not imply strictly independent shape and appearance factors inside the generator. It gives the agent more interpretable directions to explore. The 3DGS representation consists of anisotropic Gaussians with centers, covariances, opacities, and view-dependent radiance; low-opacity Gaussians are filtered out. Geometry is first made coherent, then appearance is edited while conditioning on the view set, reducing textures that work in only one image. This generation-level consistency is distinct from the later regularizer on cross-view detector behavior.

2. Calibration-consistent Dual-modal Evaluation: make both detectors observe the same object

The camera path renders the object with known intrinsics and extrinsics, then alpha-composites it onto the background while preserving depth ordering. The LiDAR path places it in the ego frame, simulates returns through 32-beam ray casting with range-dependent noise, and merges these returns into the original scene cloud. The pedestrian in the image and the geometry in the point cloud therefore share an insertion transform instead of being independently optimized, physically incompatible inputs.

Feedback must also be associated with the inserted object. Its 2D support is the tight image rectangle enclosing the projected 3D box, whereas the fusion path uses the inserted 3D box. Both use an IoU gate of 0.5, with 2D and 3D IoU respectively. The target score at each view is the highest pedestrian confidence among matching detections, or 0 if no box matches. Joint disappearance requires both detectors to fall below their decision thresholds in the same trial; a failure in only one branch is insufficient.

3. Joint Objective and Naturalness Constraints: control disappearance, object priors, and view variation together

Equation (5) uses a threshold-aware penalty: a branch already below its decision threshold contributes no positive penalty, while a still-detected branch remains penalized. For detector \(m\), score \(s\), and threshold \(\theta_m\),

\[ \phi_m(s)=\max(s-\theta_m,0). \]

The per-view attack loss is the weighted sum of these penalties across the two branches, and averaging over views gives \(\mathcal{L}_{\mathrm{atk}}\). The cached layout of equation (6) is displaced, so the aggregation is explained from its accompanying text rather than reconstructing the damaged equation. Equation (7) clearly specifies the full objective:

\[ \mathcal{L}(O)=\mathcal{L}_{\mathrm{atk}}(O)+\alpha\mathcal{L}_{\mathrm{nat}}(X)+\beta\mathcal{L}_{\mathrm{mv}}(O). \]

Here \(X\) denotes point samples from the object. Naturalness combines the L1 deviation of its box width, height, and length from mean benign pedestrian dimensions with the minimum cosine distance between its PointNet embedding and a pedestrian feature bank. The two terms and their weight are identifiable in equation (8):

\[ \mathcal{L}_{\mathrm{nat}}(X)=\|g(X)-g_{\mathrm{ref}}\|_1+\eta\,\mathrm{Dist}(\Phi(X),\mathcal{M}_{\mathrm{ped}}). \]

The bank contains 71,073 nuScenes pedestrian instances processed by the same preprocessing and encoder. This measures similarity in box dimensions and feature space, not human indistinguishability. Cross-view stability instead penalizes the standard deviation of the per-view attack loss, as equation (9) states:

\[ \mathcal{L}_{\mathrm{mv}}(O)=\operatorname{Std}_{v\in\mathcal{V}}[\mathcal{L}_{\mathrm{atk}}(O;v)]. \]

The mean loss encourages overall effectiveness, the standard-deviation term discourages success at only a few orientations, and naturalness restricts implausible geometry. These objectives are not always aligned: removing naturalness actually increases joint disappearance, an important trade-off retained in the ablation results.

4. Budgeted Agent Search: remember unsuccessful edits but accept only lower total loss

The agent reads recent prompts, both detector scores at each view, and regularizer values, then proposes adding, replacing, or removing one attribute phrase from a semantic or style pool. Each pool contains approximately 30 candidate phrases, while the base prompt remains fixed. A candidate undergoes full construction and evaluation. It is accepted only if its total objective improves on the best-so-far object; otherwise prompts are rolled back and the attempt remains in the history. This is neither gradient descent nor detector retraining.

When naturalness deteriorates, the search prioritizes restoring structure. When a detector remains confident, feedback guides edits to shape or appearance. The paper describes this scheduling at a conceptual level without full system prompts or iteration traces, so it should not be treated as a fully disclosed deterministic policy. Geometry initialization can be reused when only the style prompt changes. This caching reduces generation overhead, not the detector calls required for each evaluated candidate.

Loss & Training¶

Evaluation uses object yaw angles of 0, 90, 180, and 270 degrees, requiring 8 calls per candidate across the two detectors. The paper reports 100 iterations and 800 total queries per keyframe, returning the best object within budget. Algorithm 1, however, explicitly accounts for an initial evaluation by subtracting 1 from the number of candidates the budget permits. Under a strict 800-call cap, there can be at most 100 candidate evaluations including initialization, not 100 updates plus initialization. The source is ambiguous across these descriptions.

Both detector weights default to 1, with \(\alpha=\beta=0.1\) and \(\eta=1\). The LiDAR noise standard deviation follows a linear range model with constant 0.02 and coefficient \(10^{-4}\), using range in meters. The method uses pretrained modules and discrete prompt search rather than a reported end-to-end training procedure. Query counts alone do not establish total generation, personalization, and reconstruction time.

Key Experimental Results¶

Main Results¶

The protocol selects 10 nuScenes validation scenes with 5 keyframes each, giving 50 keyframes. An object is inserted 5-15 meters ahead, optimized separately for each keyframe, and evaluated at four orientations. ASR measures the proportion of trials in which the inserted target is missed; ASRjoint requires both detectors to miss it in the same trial. FeatureDist measures distance to the pedestrian feature bank, with lower values preferred. The following reproduces selected columns from source Table 1. ASR values are percentages, and the plus/minus values denote standard deviations across keyframes; MV-Std measures cross-view variation instead.

Configuration	ASR2D	ASRF	ASRjoint	FeatureDist	MV-Std
Benign Prompt-to-3D object	5.0 ± 2.1	20.0 ± 1.6	1.0 ± 0.7	0.08	2.7
RandEdit, query-matched	14.1 ± 4.3	35.0 ± 8.7	13.9 ± 1.2	0.35	22.5
2D-only objective	78.0 ± 8.1	23.0 ± 3.0	20.3 ± 5.9	0.17	18.3
Fusion-only objective	15.0 ± 6.2	87.0 ± 3.5	13.7 ± 4.8	0.26	14.5
Agent-OBJ	68.0 ± 8.3	96.0 ± 3.7	64.0 ± 6.2	0.12	5.3

Joint success improves over RandEdit by 50.1 percentage points, but camera success is below the 2D-only objective's 78.0%. The contribution is common failure, not independent optimality on every branch. Agent-OBJ has less cross-view variation than the optimization baselines but more than the benign object's 2.7, so it is not the lowest-variation entry overall. Confidence drops relative to the benign object are 0.62 and 0.41 for the two branches.

Ablation Study¶

The following is source Table 3. The removed-joint-objective variant is not explicitly mapped to either single-modality configuration in Table 1; its 40.8 should not be replaced with 20.3 or 13.7.

Configuration	ASRjoint (%)	FeatureDist	MV-Std
Full method	64.0	0.12	5.3
Without joint objective	40.8	0.11	9.4
Without personalization	38.4	0.15	8.2
Without naturalness constraint	80.4	0.36	16.8
Without multi-view stability constraint	30.5	0.14	8.3

Removing multi-view stability reduces joint success by 33.5 percentage points, the largest negative change in this table; removing personalization costs 25.6 percentage points. Naturalness imposes a measurable constraint: removing it gains 16.4 percentage points, but FeatureDist rises from 0.12 to 0.36 and cross-view variation increases substantially.

Key Findings¶

Without re-optimization, joint success is 53.7% when replacing the camera detector with DINO, 66.2% when replacing the fusion detector with BEVFusion, and 58.9% in cross-dataset transfer to KITTI (Table 4). These results support some transferability but do not alone establish that internal fusion coupling causes the failures.
At 200, 400, and 800 queries, Agent-OBJ achieves 37.0%, 45.9%, and 64.0% joint success, versus Greedy's 31.5%, 37.0%, and 61.2% (Table 5). The gap narrows to 2.8 percentage points at the largest budget, with a more pronounced agent advantage at lower budgets.
Queries@80% is 584, compared with 648 for Greedy and 712 for RandEdit. It means reaching 80% of each method's final success rate at 800 queries, not reaching an absolute success rate of 80%.

Highlights & Insights¶

Prompt search supplies a lower-dimensional action space constrained by generative priors. Its value lies in reducing unproductive candidates, not simply substituting language-model calls for traditional search.
Multi-view generation consistency and detector-result stability address complementary problems. The former makes appearance coherent across views, while the latter constrains detector behavior at those views; neither substitutes for the other.
Joint disappearance better tests sensor redundancy than a single-branch success rate. Defensive evaluations should report common failures, individual-branch failures, and naturalness together rather than summarize risk through one attack score.

Limitations & Future Work¶

Physical effectiveness is not established. The cached paper reports simulated rendering and point-cloud insertion, not fabrication, real sensor capture, or closed-loop driving tests. A renderable 3DGS object is not necessarily a manufacturable object, and the paper provides no separate systematic limitations discussion.
Data and view coverage are limited. The 50 keyframes and four canonical orientations do not represent continuous viewpoints, long range, weather, dynamic occlusion, or ego-motion. Generation and evaluation share the same four orientations, without a dense unseen-view test.
Naturalness metrics are narrow. Box dimensions and nearest-neighbor PointNet distance do not replace human judgments or validate real material-dependent LiDAR returns. The benign generated object's 20.0% fusion disappearance also suggests that insertion-domain differences can affect absolute results.
Reporting consistency needs clarification. Table 2 gives ScAR an ASRjoint of 16.3% but ASR2D of 12.5%; Invisible Attack has 20.9% and 18.6%, respectively. A joint event cannot exceed a constituent event under the same trials. Consequently, this note does not use that table to claim strict superiority over every prior method, especially given the different threat models.
Reproduction details remain incomplete. Specific decision thresholds, full prompts and edit lists, generation costs, and initialization accounting require clarification. No actual code URL is available in the source for this note to verify. Unified-protocol reevaluation, unseen views, physical validation, and defensive comparisons are important next steps.

Image patches and FCA/PGA: These predominantly modify camera-visible appearance. Agent-OBJ includes both semantic shape and appearance in object generation while jointly evaluating the LiDAR fusion branch. Different access assumptions and target objects prevent a simple unified numerical ranking.
AdvPC and Frustum Attack: The former follows point-cloud perturbation, while the latter studies black-box camera-LiDAR fusion security. Agent-OBJ searches a text-conditioned 3D object space; this does not by itself identify the specific internal failure mechanism of fusion.
LGM and MV-Adapter: These provide reconstruction and multi-view appearance capabilities rather than newly trained detectors. A reusable research direction is to build interpretable stress tests with existing generative priors, using naturalness and cross-view metrics to constrain evaluation quality.

Rating¶

Novelty: 4/5. Combining prompt factorization, generative 3D representations, and dual-detector black-box feedback is meaningful, while the core generation and representation modules are existing work.
Experimental Thoroughness: 3/5. Ablations, transfer, and query-budget comparisons are present, but samples are limited, physical tests are absent, and some comparison numbers need clarification.
Writing Quality: 3/5. The main workflow and regularizers are clear, but joint metrics, initialization accounting, and reproduction details contain inconsistencies or omissions.
Value: 4/5. The work offers a useful stress-testing perspective on common sensor failures, with conclusions bounded by the current simulation protocol.