Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectification¶
Conference: ECCV 2026
Paper: ECCV Official
Project: https://jeffry-wen.github.io/Rule-VLN/
Area: Multimodal VLM / Vision-and-Language Navigation (VLN)
Keywords: Vision-and-Language Navigation, Rule Compliance, Macro-Micro Visual Prompting, Epistemic Mental Map, Inpainting Synthesis
TL;DR¶
Addressing the "Goal-driven trap" where embodied navigation agents prioritize physical geometry while ignoring traffic rules, this paper introduces Rule-VLN, the first large-scale urban benchmark enforcing regulatory compliance, alongside the training-free Semantic Navigation Rectification Module (SNRM) that slashes constraint violations by 19.26% and improves task completion by 5.97%.
Background & Motivation¶
Vision-and-Language Navigation (VLN) requires embodied agents to ground natural language instructions into long-horizon physical actions within complex physical spaces. Empowered by the rapid evolution of Multimodal Large Models (MLLMs), modern agents have exhibited impressive capabilities in goal-oriented trajectory planning and exploration efficiency. However, as embodied navigation transitions from sanitized simulated testbeds to real-world urban streetscapes, the fundamental definition of navigation success must shift from mere physical reachability ("can I go?") to social and regulatory compliance ("may I go?"). In urban driving and walking environments, a path may be completely navigable in terms of three-dimensional geometry, yet strictly forbidden by regulatory constraints such as "No Entry" signs or one-way street rules; disregarding these semantic directives inevitably leads to catastrophic safety hazards and legal violations.
Despite these critical implications, contemporary VLN agents consistently suffer from a severe "Goal-driven trap". State-of-the-art architectures optimize primarily for shortest-path travel distance or coverage, relying heavily on salient geometric connectivity priors while remaining fundamentally oblivious to small-scale yet safety-critical semantic cues. Even when confronted with a prominent red-and-white prohibitory sign, agents routinely drive directly into restricted zones. The root of this failure stems from the absence of safety-critical regulatory annotations in conventional benchmarks like Touchdown, combined with the fact that full end-to-end retraining of foundation models is computationally prohibitive and prone to degrading general navigation capabilities. Furthermore, modeling traffic rules as soft heuristic optimization rewards fails to enforce strict compliance.
To bridge this critical perception-compliance gap, this work tackles the challenge from both benchmark formulation and plug-and-play inference. The authors construct Rule-VLN, a massive 29k-node urban benchmark grounded on the Touchdown topology, accompanied by CityNav-Rules-73K containing 73,937 rule instances across 177 categories. To restore rule compliance without retraining the navigation backbones, they introduce the Semantic Navigation Rectification Module (SNRM). The core idea is to decouple urban regulatory cues into hard semantic graph validity masks via mask-prioritized diffusion injection, and deploy a training-free rectification pipeline combining macro-micro visual prompting with 2D virtual mental map re-planning to enforce compliance and compute detours.
Method¶
Overall Architecture¶
The proposed framework encompasses two complementary pillars: Mask-Prioritized Semantic Injection (MPSI) for benchmark synthesis, and the Semantic Navigation Rectification Module (SNRM) for model-agnostic inference correction. MPSI embeds photorealistic, perspective-consistent traffic signs into real-world panoramas using road segmentation priors, dual-mask conditioning, and distribution-based quality pruning. During online navigation, frozen policy backbones propose tentative actions. SNRM acts as an external semantic reflector: it scans for small-scale regulatory cues via a coarse-to-fine perception pipeline, grounds visual symbols into discrete permissible action subspaces, and dynamically constructs a virtual 2D mental map to execute compliant detour planning whenever an illegal action is detected.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Panoramic Observation + Language Instruction"] --> B["Base Navigation Policy Predicts Action"]
B --> C["Macro-Micro Visual Prompting<br/>DINO Scanning + Bounding Box Crop"]
C --> D["Knowledge-Driven Rule Grounding<br/>SigLIP Knowledge Retrieval + Qwen-VL CoT"]
D -->|Action Safe| E["Execute Base Policy Action"]
D -->|Constraint Conflict| F["Epistemic Mental Map Rectification<br/>2D Virtual Coordinates + Detour Planning"]
F --> G["Loop-Trap Detection & Compliant Waypoint Execution"]
Key Designs¶
1. Mask-Prioritized Semantic Injection (MPSI): Decoupled and Controllable Panoptic Editing Directly synthesizing traffic signs in high-resolution panoramas using generic inpainting models causes severe perspective distortions, boundary artifacts, and illegible text. MPSI resolves this by explicitly decoupling regulatory constraints into geometric shapes and semantic rules. It leverages a road segmentation prior to identify precise physical anchor regions on the street, preventing spatial displacement. A Diffusion Transformer (DiT) conditioned on dual masks (shape boundary mask and rule constraint mask) and CLIP reference visual embeddings synthesizes the sign while confining generative latents strictly within prescribed geometric bounds. Finally, a bimodal Gaussian Mixture Model (GMM) fits the CLIP cosine alignment distribution between generated crops and target rule semantics, automatically pruning low-quality hallucinations before seamlessly stitching the compliant signs into the panoramic background.
2. Dual-Stage Coarse-to-Fine Perception (MMVP & KDRG): Resolving Small-Scale Semantic Signs Standard VLMs struggle to ground small regulatory symbols embedded within expansive, complex panoramic viewpoints due to limited input resolutions. SNRM introduces Macro-Micro Visual Prompting (MMVP), which first scans the panoramic slice with a lightweight object detector (DINO) using generic prompts. Detected regions are transformed into a dual visual prompt: a global macro view highlighting the context bounding box to preserve environmental connectivity, and a magnified micro crop resolving fine symbol details. Following this, Knowledge-Driven Rule Grounding (KDRG) utilizes SigLIP to compute zero-shot similarity between the micro crop and a curated knowledge bank of 177 normalized rule categories, retrieving the highest-confidence rule prior. The macro view, micro crop, text prior, and intended base action are then supplied to a multimodal model (e.g., Qwen3-VL), which executes structured Chain-of-Thought (CoT) reasoning to verify compliance and emit a safety gating token.
3. Epistemic Mental Map & Dynamic Detour Planning: Virtual Topological Navigation When an intended action violates regulatory constraints, the agent cannot simply halt or make arbitrary random turns; it must find an alternate legal trajectory that eventually reconnects with the primary instruction. SNRM instantiates an epistemic mental map by establishing a local 2D virtual coordinate frame centered at the conflict origin \(P_{start} = (0, 0)\), anchoring the expected target coordinate along the planned heading \(\Delta\theta_{exp}\) as \(P_{target\_v} = (\sin(\Delta\theta_{exp}), \cos(\Delta\theta_{exp}))\). As the agent executes compliant turns to reach a deviation coordinate \(P_{dev}\), it selects the next waypoint from candidate neighbors \(\mathcal{C}\) using a greedy distance heuristic equipped with a severe backtracking penalty: $\(C_{best} = \arg\min_{C_i \in \mathcal{C}} \left( \|P(C_i) - P_{target\_v}\| + \lambda \mathbb{I}_{backtrack}(P(C_i)) \right)\)$ where \(\mathbb{I}_{backtrack}\) penalizes candidate locations falling within a critical radius of previously visited nodes. Furthermore, a visual memory buffer computes the feature cosine similarity between the current observation and the conflict origin; exceeding a preset threshold (\(\cos(f_{curr}, f_{start}) > \tau_{sim}\)) indicates a closed-loop trap and triggers immediate forced detour divergence.
A Worked Example¶
Consider an agent receiving the instruction "Face the store with the red sign. Go forward once to the intersection and turn left." At step \(t = 4\), the base model selects Forward based on conventional geometric shortest-path traversal. 1. Perception Check: DINO detects a small circular sign ahead. MMVP extracts a high-resolution crop \(V_{micro}\) and a bounding-box annotated global view \(V_{macro}\). 2. Rule Grounding: SigLIP matches \(V_{micro}\) against the knowledge bank, retrieving "No Entry" as the Top-1 prior. Qwen3-VL conducts CoT reasoning: "A No Entry sign is positioned at the entrance of the upcoming street; moving forward violates this regulation. The proposed action is strictly prohibited." 3. Mental Map Detour: The forward edge is pruned in the semantic mask. The agent anchors its virtual target vector, executes a compliant U-turn to avoid entering the forbidden segment, and proceeds along an alternative open street. 4. Trajectory Re-alignment: Guided by the penalized distance heuristic on the 2D virtual map while avoiding visited nodes, the agent detours around the block and realigns with the global directive at \(t = 33\), safely completing the task.
Loss & Training¶
SNRM operates as a strictly training-free, zero-shot rectification module during deployment; no parameters of the base navigation models (e.g., FLAME, Loc4plan) or the foundation perception backbones (Qwen3-VL, SigLIP) are modified. For the benchmark construction stage, the Diffusion Transformer inside MPSI is fine-tuned using standard diffusion denoising score-matching loss \(\mathcal{L}_{diff} = \mathbb{E}_{t, z_0, \epsilon}[\|\epsilon - \epsilon_\theta(z_t, t, M_S, M_R, V_S', \mathcal{E}_{ref})\|^2]\) to enforce spatial alignment and high-fidelity text-sign synthesis under mask guidance.
Key Experimental Results¶
Main Results¶
Evaluation is performed across the four curriculum difficulty levels of Rule-VLN (Level-1 to Level-4, where the proportion of constrained instructions escalates from 31.44% to 91.13%). Models are evaluated using Task Completion (TC), Success weighted by Path Length (SPL), Shortest-Path Distance (SPD), and Constraint Violation Rate (CVR).
| Model Configuration | Difficulty Level | Task Completion TC(%) ↑ | Path Success SPL(%) ↑ | Shortest Distance SPD ↓ | Violation Rate CVR(%) ↓ |
|---|---|---|---|---|---|
| Loc4plan [MM2024] | Level-1 | 13.01 | 7.72 | 21.81 | 39.45 |
| Loc4plan + SNRM (Ours) | Level-1 | 14.07 (+1.06) | 7.76 (+0.04) | 21.31 (-0.50) | 36.48 (-2.97) |
| FLAME [AAAI2025] | Level-1 | 24.36 | 18.83 | 17.12 | 21.17 |
| FLAME + SNRM (Ours) | Level-1 | 24.36 (+0.00) | 17.72 (-1.11) | 16.58 (-0.54) | 12.47 (-8.70) |
| Loc4plan [MM2024] | Level-2 | 10.95 | 6.78 | 23.54 | 38.67 |
| Loc4plan + SNRM (Ours) | Level-2 | 12.48 (+1.53) | 6.98 (+0.20) | 23.03 (-0.51) | 29.97 (-8.70) |
| FLAME [AAAI2025] | Level-2 | 17.90 | 13.94 | 20.04 | 33.68 |
| FLAME + SNRM (Ours) | Level-2 | 21.45 (+3.55) | 13.91 (-0.03) | 18.30 (-1.74) | 16.35 (-17.33) |
| Loc4plan [MM2024] | Level-3 | 10.42 | 6.43 | 23.39 | 40.80 |
| Loc4plan + SNRM (Ours) | Level-3 | 12.61 (+2.19) | 7.16 (+0.73) | 22.23 (-1.16) | 33.64 (-7.16) |
| FLAME [AAAI2025] | Level-3 | 11.43 | 9.03 | 23.44 | 41.79 |
| FLAME + SNRM (Ours) | Level-3 | 17.40 (+5.97) | 10.51 (+1.48) | 19.96 (-3.48) | 22.53 (-19.26) |
| Loc4plan [MM2024] | Level-4 | 7.30 | 4.43 | 24.50 | 33.42 |
| Loc4plan + SNRM (Ours) | Level-4 | 7.96 (+0.66) | 4.54 (+0.11) | 23.63 (-0.87) | 27.00 (-6.42) |
| FLAME [AAAI2025] | Level-4 | 8.95 | 6.24 | 26.29 | 33.89 |
| FLAME + SNRM (Ours) | Level-4 | 9.94 (+0.99) | 6.39 (+0.15) | 25.61 (-0.68) | 23.68 (-10.21) |
Ablation Study¶
Ablation results on the most challenging Level-4 benchmark evaluating the contribution of each SNRM component on top of the FLAME backbone:
| Setup | MMVP Prompting | KDRG Grounding | Mental Map | Task Completion TC(%) ↑ | Shortest Distance SPD ↓ | Violation Rate CVR(%) ↓ | Note |
|---|---|---|---|---|---|---|---|
| #1 | ✗ | ✗ | ✗ | 8.95 | 26.29 | 33.89 | Baseline FLAME without SNRM |
| #2 | ✓ | ✗ | ✓ | 9.02 | 25.65 | 25.94 | Hallucinates on long-tail signs w/o text priors |
| #3 | ✗ | ✓ | ✓ | 8.88 | 25.69 | 26.86 | Misses small signs without micro-crops |
| #4 | ✓ | ✓ | ✗ | 6.53 | 26.91 | 21.28 | Stops at constraints without detour ability |
| #5 | ✓ | ✓ | ✓ | 9.94 | 25.61 | 23.68 | Full SNRM achieves balanced optimal safety & TC |
Key Findings¶
- Severe Compliance Collapse of Standard SOTA: Baseline models exhibit catastrophic vulnerability when traffic rules are introduced. For instance, unaugmented FLAME experiences a surge in CVR to 41.79% in Level-3, demonstrating that goal-driven agents completely overlook regulatory constraints.
- Zero-Shot Rectification Efficacy: SNRM universally boosts compliance across both small specialized planners (Loc4plan) and multimodal models (FLAME). On Level-3, SNRM reduces FLAME's CVR by 19.26% while simultaneously lifting TC by 5.97%, proving that detour planning actively recovers lost navigation success.
- Mental Map is Indispensable for Active Detours: Ablation setup #4 reveals that perceiving violations without an active mental map causes TC to drop to 6.53% (lower than the unaugmented baseline). Agents merely freeze or fail; dynamic 2D topological re-planning is essential for real-world navigation recovery.
- Latency vs. Triggered Efficiency: SNRM operates via event-driven triggering. For SNRM-8B, the detector only activates deep VLM reasoning on ~26.25% of steps. The lightweight SNRM-4B non-CoT variant adds only 0.224 seconds per step while achieving a 10.55% reduction in CVR.
Highlights & Insights¶
- Paradigm Shift from Geometry to Compliance: Formalizes edge traversability as a dynamic semantic validity mask \(\mathcal{M}(e)\), shifting the VLN benchmark evaluation from pure physical reachability to social and regulatory compliance.
- High-Fidelity MPSI Synthesis: Bypasses traditional diffusion hallucinations through dual-mask conditioning on road segmentations and bimodal GMM filtering, delivering realistic street signage without manual 3D modeling.
- Model-Agnostic, Zero-Shot Safety Layer: Acts as an external semantic reflection wrapper that can be readily equipped onto any proprietary or open-source navigation policy without expensive retraining or catastrophic forgetting.
Limitations & Future Work¶
- Discrete Node Viewpoint Discrepancies: Because Rule-VLN builds upon discrete panoramic nodes from Touchdown, viewing the same sign from different adjacent graph nodes may occasionally exhibit minor lighting or angle inconsistencies across synthesized cuts.
- Static vs. Dynamic Regulations: Current rules focus on static traffic and prohibition signs; extending the paradigm to temporal signals such as dynamic traffic lights and moving pedestrian zones represents a key future direction.
- Real-Time On-Device Constraints: While the event-triggered 4B variant drastically lowers latency, executing multi-turn VLM reasoning still incurs computational overhead that may limit deployment on ultra-high-frequency (e.g., 50Hz) robotics hardware.
Related Work & Insights¶
- vs. Safe-VLN [IEEE LRA 2024]: Safe-VLN focuses on collision avoidance in indoor continuous environments (identifying untraversable physical obstacles), whereas Rule-VLN addresses the challenge where paths are physically open but legally prohibited.
- vs. VLM-Social-Nav [IEEE RAL 2024] & GC-VLN [2025]: Prior compliance works incorporate soft penalty scores into cost functions, which are often overridden by strong goal-seeking gradients; SNRM enforces hard topological constraints coupled with active detour planning.
- vs. FLAME [AAAI 2025] & Loc4plan [ACM MM 2024]: While state-of-the-art baselines achieve strong shortest-path performance in unconstrained settings, they exhibit high violation rates (>30-40%) under regulatory rules; SNRM restores both compliance and completion rates without architectural modifications.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first rule-compliant urban VLN benchmark and presents an elegant training-free dual-stage rectification architecture.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 4-level curriculum evaluation, rigorous ablations, synthesis quality benchmarks, and latency profiling.
- Writing Quality: ⭐⭐⭐⭐⭐ Clearly articulated motivation, rigorous mathematical formulation, and well-structured empirical analysis.
- Value: ⭐⭐⭐⭐⭐ Crucial bridge for transitioning embodied AI agents from simulated sandboxes to safety-compliant real-world urban deployment.