AiSCREAM: Absolute Target Localization with Language-Conditioned Cross-View Alignment for Autonomous Vehicles¶
Conference: ECCV 2026
Paper: Official paper page · PDF
Project: AiSCREAM
Area: Autonomous Driving
Keywords: Language-conditioned target localization, generated aerial views, cross-view semantic alignment, multitask learning, prediction-conditioned refinement
TL;DR¶
AiSCREAM uses an aerial image generated from a single front view as an auxiliary spatial cue, aligns the two views through language-conditioned semantic segmentation, and combines multitask localization with MLLM refinement to predict a metric target and acceptable region, reducing DRAMATiST RMSE from the strongest model baseline's 8.56 m to 5.00 m.
Background & Motivation¶
When a passenger asks a vehicle to stop beside a garbage can, recognizing the can is only the beginning. The system must decide which side is navigable and turn an image location into metric coordinates relative to the vehicle. Referring expression detection usually returns an object box, referring segmentation returns a pixel mask, and navigable-region methods such as GENNAV identify where the vehicle should stop. None of these outputs alone guarantees a spatial goal that a downstream planner can use. A single front image makes the conversion especially difficult because perspective compresses distance, landmark relationships, and road boundaries.
Conventional bird's-eye-view representations organize space but may discard appearance details needed by the instruction. The paper emphasizes cues such as color and pedestrian clothing: after reduction to a few semantic categories or a line-drawing-like layout, a pedestrian wearing brown pants may become just another pedestrian. Conversely, asking an MLLM to report a distance directly from an image uses semantic priors but provides little numerical grounding. The proposed intervention is to generate an appearance-preserving aerial image and use it alongside the original observation, without adding a real sensor.
This generated view is neither a measured overhead scene nor an exact geometric projection. Simply feeding it into a localizer is therefore insufficient. The important step is to connect structures that are semantically recognizable in both views so that the original image constrains how the generated prior is used. Core idea: treat a generated aerial view as an auxiliary spatial representation, align it with the front view using consistent semantic masks and bidirectional attention, and ask an MLLM to correct an existing numerical prediction rather than estimate the target distance from scratch.
Method¶
Overall Architecture¶
The task is Referring Navigable Position Localization, or RNPL. Given a navigation instruction and one front-camera image, the model predicts a target center in ego-centric coordinates and a circular acceptable region. Here, absolute means a metric position rather than global map coordinates. The target is assumed to be visible in the image; trajectory prediction and vehicle control are explicitly outside the task.
AiSCREAM has two main modules. AiRAM generates an Aerial Semantic (AiS) view and combines cross-view consistent semantic segmentation (CVSS) with bidirectional cross-attention to construct a fused representation. MTAPL combines that representation with the instruction, predicts the initial position and region, and passes the prediction to an MLLM Reshape Localizer for correction. The HC loss supervises multitask localization during training; it is not a separate inference stage.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Front image + instruction"] --> AiS["Appearance-preserving<br/>aerial generation"]
AiS --> Align["Semantically anchored<br/>cross-view alignment"]
Input -->|Original image and category prompts| Align
Align --> Locate["Multitask localization<br/>and obstacle constraints"]
Input -->|Instruction embedding| Locate
Locate --> Refine["Prediction-conditioned<br/>refinement"]
Input -->|Original image and instruction| Refine
Refine --> Output["Metric target center<br/>and acceptable radius"]
Key Designs¶
1. Appearance-preserving aerial generation: an auxiliary viewpoint for spatial relationships
The AiS generator takes the front image and text conditioning and synthesizes an aerial image in the image domain. The main experiments use Nano Banana. Qwen Image Edit appears as an alternative in the method description, but should not be treated as an equally validated main-table configuration. Instead of projecting features onto a ground plane as in a conventional BEV pipeline, this approach aims to retain colors and scene appearance. The localizer can then use both what a landmark looks like and an overhead arrangement that may make the adjacent target space easier to interpret.
The generated image is not an additional real observation. Its role is to express learned spatial priors in a form that an image encoder can process. Those priors may distort geometry, so the original front view remains in the subsequent computation. Indeed, the aerial-only ablation performs worse than the front-only model. The evidence supports complementary alignment, not the stronger claim that generation reliably reconstructs the true scene geometry.
2. Semantically anchored cross-view alignment: fuse appearance with structure before aligning views
An ordinary segmentation model trained on real camera images can exhibit viewpoint-dependent errors on generated aerial images. CVSS instead uses a language-conditioned segmenter such as SAM3, with text prompts specifying scene categories, to obtain masks for both the front and AiS images. These category prompts establish a shared semantic reference and should not be conflated with the passenger's navigation instruction. Shared categories also do not imply exact pixel correspondences between the two masks.
The front image, aerial image, and their segmentation outputs pass through a shared image encoder and MLP projection, producing four feature streams. The main experiments use DINOv3. Within each view, bidirectional cross-attention (BiCA) first lets image features query segmentation features and then reverses the query/key-value roles; the two attention results are concatenated. Another BiCA operation exchanges information between the two within-view fused representations, followed by self-attention. Semantic masks thus serve as structural anchors for learned correspondence rather than imposing a hard geometric warp. Equations (1) and (2) have damaged equality signs, brackets, and characters in the cache, so this note preserves the computation order supported by the surrounding prose without reconstructing the broken equations.
3. Multitask localization and obstacle constraints: supervise both the target center and acceptable region
Qwen3-Embedding-0.6B encodes the navigation instruction. MTAPL combines this embedding with the cross-view representation using an elementwise Hadamard product and processes the result with a Transformer. Consequently, which landmark is intended and which side should be selected affect numerical localization, rather than appearing only in an earlier perception stage. The shared representation feeds a front-region head, an AiS-region head, and an absolute-target head. The front-region and absolute-target heads are primary outputs, while the AiS-region head supplies auxiliary structural supervision.
The paper calls the first two heads segmentation heads, but writes all three outputs as three-dimensional vectors and ultimately describes a center-and-radius region representation. It would therefore be unjustified to expand their names into a dense-mask decoder that the main text does not specify. The Height-based Cross-View (HC) loss combines absolute-position supervision, auxiliary AiS-region supervision, and an obstacle-overlap penalty; front-region L1 supervision also participates in the overall objective. The obstacle term addresses a specific failure: a predicted center may be close to the landmark while its acceptable region covers an obstacle. Auxiliary region supervision serves a different purpose by retaining target structure across viewpoints. Center error alone cannot capture both effects.
4. Prediction-conditioned refinement: give the MLLM a numerical reference to inspect
After initial localization, the system overlays the absolute-target prediction on the original front image and provides an explicit numerical cue together with the instruction. GPT-5.2 checks whether the predicted position and region are physically and contextually plausible and produces a correction when necessary. Unlike direct image-to-coordinate prompting, this setup gives the model a candidate whose visual location is associated with a metric value. The task becomes assessing and correcting that reference. Exact prompt details are delegated to supplementary material that is not present in the current cache, so no prompt template is invented here.
The final output is \(\tilde{y}_{\mathrm{abs}}=(\tilde{x},\tilde{y},\tilde{r})\), with a target center and the radius of a circular acceptable region. Although the paper calls this a distribution, it is not demonstrated to be a calibrated probability density or a confidence interval. One sentence in the initial-stage description uses an AiS subscript, whereas the overlay and final output explicitly use the absolute-target head. This note follows the latter data flow and flags the notation inconsistency as a reproduction caveat.
A Worked Example¶
Consider the paper's Figure 3 instruction to pull over beside orange traffic cones on the right side of the road. The model must resolve both the intended cones and the space beside them that the vehicle can actually access. AiS generation provides another scene arrangement, CVSS supplies shared-category structural cues, and cross-view attention connects those cues to the original appearance. MTAPL then predicts the instructed target, while obstacle supervision discourages an acceptable region from extending into vehicles or other obstructions.
The MLLM subsequently inspects the overlaid prediction and its consistency with the navigable side of the cones. The paper reports that GENNAV selected the left side of the road and a region overlapping a truck, while direct GPT-5.2 selected a location near the cones but on the inaccessible side. AiSCREAM selected the adjacent drivable region. This is a reported qualitative result, not an instrumented trace of every intermediate module, so intermediate coordinates should not be fabricated. Separately, the task definition gives the illustrative output $ (x,y,r)=(-3.0,10.0,2.0)\,\mathrm{m} $ to explain center coordinates and acceptable radius; these are not measured coordinates for the cones example.
Loss & Training¶
The cached HC-loss equation (3) and overall-loss equation (4) are severely damaged. The weights, exact obstacle penalty, and precise implementation of the height-based mechanism cannot be recovered reliably. The surrounding text supports front-region L1 supervision, absolute-position supervision, auxiliary AiS-region supervision, and an obstacle-overlap constraint. The loss ablations below follow Table 3 without inventing a complete mathematical expression, hyperparameters, or a height-map construction procedure.
Each benchmark has its own training, validation, and test sets, and images are resized to 224×224. The GRiN-Drive split is 8,349/1,163/508; the DRAMATiST split is 4,362/949/930. The trainable model has approximately 41.5M parameters and 662M multiply-add operations, and training on GRiN-Drive takes about 3 hours on an RTX 4090. These counts do not describe the entire foundation-model stack. Likewise, the network-only inference time of about 9 milliseconds does not represent end-to-end latency including generation and refinement.
Key Experimental Results¶
Main Results¶
GRiN-Drive uses its single-target subset. The new DRAMATiST benchmark contains 6,241 samples constructed by inserting DRAMA landmark noun phrases into navigation templates and collecting human target annotations. Metric targets are not directly measured sensor ground truth: the authors extract the annotated binary mask's centroid and largest-inscribed-circle radius, then use a monocular depth model to convert the center into ego-centric coordinates. The other benchmark and segmentation baselines use the same conversion procedure.
RMSE is the primary metric. AED is the average Euclidean distance between predicted and reference target positions. Both are measured in meters and are lower-is-better; mIoU measures mean overlap between predicted and reference regions and is higher-is-better. The precise metric implementations are in unavailable supplementary material, so this note does not assume a particular component normalization for RMSE. The following selection from Table 1 reports means ± standard deviations over five trials.
| Method | GRiN RMSE ↓ | GRiN AED ↓ | GRiN mIoU (%) ↑ | DRAMATiST RMSE ↓ | DRAMATiST AED ↓ | DRAMATiST mIoU (%) ↑ |
|---|---|---|---|---|---|---|
| GENNAV | 10.51 ± 0.52 | 6.35 ± 0.35 | 7.25 ± 0.31 | 8.56 ± 0.52 | 6.45 ± 0.60 | 2.83 ± 0.49 |
| GPT-5.2, direct zero-shot | 12.44 ± 0.15 | 9.07 ± 0.13 | 6.26 ± 0.16 | 11.29 ± 0.24 | 7.91 ± 0.08 | 3.22 ± 0.17 |
| AiSCREAM | 4.72 ± 0.14 | 3.95 ± 0.09 | 11.89 ± 0.69 | 5.00 ± 0.05 | 4.03 ± 0.02 | 6.16 ± 0.37 |
| Human, restricted single-image protocol | 7.45 ± 0.50 | 4.00 ± 0.20 | 4.82 ± 0.56 | 7.00 ± 1.62 | 4.69 ± 1.20 | 6.50 ± 0.63 |
Against GENNAV, AiSCREAM reduces RMSE by 5.79 m and 3.56 m on the two datasets. The human comparison should not be generalized to every metric: human DRAMATiST mIoU is 6.50%, above the model's 6.16%. The human experiment samples 250 instances per dataset, totaling 500 instances and 1,500 predictions from 10 participants, who only see one front image. That setting is not equivalent to real-world judgment with full environmental context.
Baseline training conditions also differ. Segmentation models are fine-tuned, whereas VLA and direct MLLM baselines are evaluated zero-shot; VLA systems additionally receive only one front image without temporal history. The results establish a target-localization advantage under this RNPL protocol, not comprehensive superiority on the baselines' original driving tasks.
Ablation Study¶
The following selection comes from Table 2. RL means MLLM Reshape Localizer, not reinforcement learning. AiS and CVSS are removed relative to the RL-disabled configuration, so their effects must not all be interpreted as independent removals from the complete system.
| Configuration | GRiN RMSE (m) ↓ | GRiN AED (m) ↓ | DRAMATiST RMSE (m) ↓ | DRAMATiST mIoU (%) ↑ |
|---|---|---|---|---|
| Full model | 4.72 ± 0.14 | 3.95 ± 0.09 | 5.00 ± 0.05 | 6.16 ± 0.37 |
| RL disabled | 4.95 ± 0.08 | 4.18 ± 0.06 | 5.19 ± 0.07 | 5.90 ± 0.58 |
| RL disabled, AiS removed | 6.60 ± 0.10 | 5.61 ± 0.10 | 5.57 ± 0.13 | 3.35 ± 0.35 |
| RL disabled, CVSS removed | 5.80 ± 0.10 | 4.83 ± 0.06 | 5.34 ± 0.06 | 3.43 ± 0.23 |
The full-model GRiN mIoU standard deviation is 0.69 in Table 1 and 0.59 in Table 2, with the same mean of 11.89. The main-results table above follows Table 1 rather than silently reconciling the discrepancy. That column is not repeated in the module ablation table.
Table 3 disables RL throughout so that refinement does not obscure the effect of training losses:
| Initial-localizer training configuration | GRiN RMSE (m) ↓ | DRAMATiST RMSE (m) ↓ | DRAMATiST mIoU (%) ↑ |
|---|---|---|---|
| AiS auxiliary and obstacle terms retained | 4.95 ± 0.08 | 5.19 ± 0.07 | 5.90 ± 0.58 |
| AiS auxiliary term removed | 5.17 ± 0.06 | 5.24 ± 0.02 | 5.30 ± 0.29 |
| Obstacle term removed | 5.27 ± 0.16 | 5.24 ± 0.06 | 4.06 ± 0.69 |
Key Findings¶
- On GRiN-Drive with RL disabled, removing AiS adds 1.65 m RMSE and removing CVSS adds 0.85 m, whereas RL itself improves RMSE by only 0.23 m. The main evidence favors the cross-view representation rather than attributing most gains to GPT-5.2.
- Removing the obstacle term on DRAMATiST adds only 0.05 m RMSE but reduces mIoU from 5.90% to 4.06%. Center accuracy and acceptable-region quality address different aspects of the task.
- In Table 4, aerial-only AiSCREAM obtains 7.70 m GRiN RMSE, front-only obtains 6.60 m, and the initial dual-view model obtains 4.95 m. The generated view should complement the original observation, not replace it.
- Table 5 replaces AiS with GaussianLSS and BEVFormer v2 representations, yielding GRiN RMSEs of 6.48 and 6.74 m versus 4.72 m for AiS. This supports the representation choice for the evaluated task, not a blanket claim against all BEV methods or multisensor configurations.
Highlights & Insights¶
- Generated images can organize priors without masquerading as new sensors. Retaining the real observation and constraining the generated view through semantic structure are important parts of the contribution, as the aerial-only ablation demonstrates.
- Correcting a candidate gives an MLLM a stronger reference than reporting coordinates independently. The overlay and numerical cue make refinement concrete, although its relatively modest gain deserves a separate deployment cost-benefit analysis.
- Evaluating both a center and a region better reflects navigation needs. The obstacle term's regional effect is much larger than its center-error improvement suggests, supporting complementary metrics rather than optimization around one average distance.
Limitations & Future Work¶
- The full pipeline is not real-time. The network takes about 9 milliseconds, but end-to-end inference takes about 10.3 seconds, including approximately 8 seconds for AiS generation. The authors suggest distillation, low-step generation, or removing explicit generation at inference; network-only latency must not be presented as vehicle response time.
- Metric reference targets depend on monocular depth. Converting human 2D annotations through a depth estimator can introduce systematic scale errors. Agreement with those references is not independent evidence of true ranging accuracy; sensor-calibrated targets or explicit error propagation would strengthen evaluation.
- Generated geometry lacks direct ground-truth validation. The authors acknowledge the absence of AiS ground truth and test sensitivity through injected Gaussian pixel noise. As an additional critique, pixel noise does not cover structural generation errors such as missing landmarks or incorrect road topology.
- Navigability and safety are not established in closed loop. The setting assumes a visible single target, and a circular region cannot fully represent complex road boundaries or multiple acceptable destinations. Meter-scale errors and low mIoU do not guarantee safe stopping; trajectory execution, collision rates, and dynamic interaction are not evaluated.
- Object-centered reference frames remain difficult. In Figure 4, an instruction to go around a stopped bus and continue forward should place the goal ahead of the bus, but several models predict behind it. Explicit object-orientation reasoning or reference-frame modeling are plausible extensions, not results established by this paper.
Related Work & Insights¶
- Versus GENNAV and referring navigable-region methods: These methods emphasize instruction-conditioned regions, whereas AiSCREAM adds metric target regression and generated cross-view alignment. Baseline regions are converted through monocular depth before comparison, so the observed gap involves both representation and output mechanisms.
- Versus GaussianLSS and BEVFormer v2: AiSCREAM emphasizes appearance retained by image-domain generation, while conventional approaches emphasize geometric organization. Combining generated appearance with independently verifiable geometry is a potential direction, but the replacement experiments do not settle performance in real multisensor systems.
- Versus ORION, SimLingo, and Alpamayo-R1: Their principal objectives concern trajectories or actions, while this evaluation imposes single-frame zero-shot target localization. RNPL could provide an upstream language-goal interface, but joint downstream evaluation is needed before claiming a full driving benefit.
- Resource scope: The official page, PDF, and project page are linked above. This note uses the local main-paper cache and does not present unread supplementary prompts, training hyperparameters, or code details as verified facts.
Rating¶
- Novelty: 4/5. Generated aerial views, shared semantic anchors, and prediction-conditioned refinement form a targeted combination, although the main components come from existing foundation models.
- Experimental Thoroughness: 4/5. Two benchmarks, module and loss ablations, view replacements, and a human comparison offer broad evidence, but independently measured metric targets and closed-loop safety tests are missing.
- Writing Quality: 3/5. The task and module motivations are clear, but notation and table standard deviations are inconsistent, and damaged cached equations limit exact reproduction.
- Value: 4/5. The method offers a useful route from language instructions to spatial goals, with generation cost and acceptable-region precision remaining deployment obstacles.