Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/lhy-zjut/SACA
Area: Robotics & Embodied AI
Keywords: continuous vision-language navigation, step-level supervision, failure reuse, contrastive alignment, reinforcement fine-tuning

TL;DR¶

SACA uses frozen vision and language models to identify useful prefixes and divergence points in navigation trajectories, then repairs or rescues failures depending on whether the sampled group contains a success, reaching 60.3% / 60.3% success on R2R-CE / RxR-CE unseen validation without extra data or inference-time auditor overhead.

Background & Motivation¶

Vision-Language Navigation in Continuous Environments (VLN-CE) requires agents to follow natural-language instructions using egocentric visual streams and low-level actions, rather than merely selecting nodes in a predefined navigation graph. An instruction involving passing a table, entering a kitchen, and stopping near a refrigerator spans multiple time steps, each conditioned on current observations and earlier progress. Supervised fine-tuning (SFT) of a Video-LLM can establish action correspondences on expert trajectories without teaching recovery from deviations caused by the policy itself. Once a small early error changes subsequent viewpoints, the model encounters states that are uncommon in demonstrations, allowing errors to compound.

Reinforcement fine-tuning permits environmental exploration, but binary navigation outcomes usually become available only at STOP. Standard GRPO estimates advantages from relative rewards among trajectories for the same instruction; when all fail, identical rewards eliminate the outcome-based relative learning signal. This also conflates immediate failure with a trajectory that completes most of an instruction before taking a wrong turn at the final junction. The authors report that approximately 73% of failed episodes complete initial sub-instructions, suggesting that terminal failure does not make every action useless; the supporting analysis is assigned to Appendix A, which is absent from the supplied cache, so its statistical details could not be checked.

Instead of training a dedicated process reward model, the paper uses existing foundation models to check whether instruction landmarks have visual support. The resulting evidence ranks trajectories and identifies a reusable valid prefix and a divergence point requiring intervention. Because visual matching can be noisy and the best failure is still not a success, the method also restricts update strength and the scope of step-level supervision. Core Idea: separate failed trajectories into credible local experience and identifiable local mistakes, then choose a learning strategy according to the presence of a real success instead of assigning every failure an undifferentiated zero.

Method¶

Overall Architecture¶

Inputs are a navigation instruction, continuous visual observations, and a group of trajectories sampled by the current policy for that instruction. The policy starts from LLaVA-Video-7B, receives SFT on navigation demonstrations, and then undergoes reinforcement fine-tuning with SACA. The training loop passes through Perception-Grounded Step-Aware Auditing, Scenario-Conditioned Group Construction, and Robust Step-Level Optimization to produce an updated navigation policy. The auditor is a training component, not an additional deployed planner; at inference time the policy still produces low-level actions from observations and instructions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Instruction, observations<br/>and sampled trajectory group"] --> Audit["Perception-Grounded<br/>Step-Aware Auditing"]
    Audit --> Group["Scenario-Conditioned<br/>Group Construction"]
    Outcome["Environment outcome rewards"] --> Group
    Group -->|"Success present: repair resampling"| Optimize["Robust Step-Level<br/>Optimization"]
    Group -->|"All fail: all-failure rescue"| Optimize
    Expert["Simulator shortest-path action<br/>training-only local correction"] -.-> Optimize
    Optimize --> Policy["Update navigation policy"]
    Policy --> Inference["Inference: observations and instruction<br/>directly produce low-level actions"]

The two group-construction branches represent mutually exclusive training scenarios, not successive operations applied to every trajectory. Outcome rewards and visual process scores also have different sources and roles: the former establish success, while the latter assess progress within failures. Shortest-path actions supply positives only for local contrastive correction, so the overall training procedure should not be described as entirely free of expert or simulator supervision.

Key Designs¶

1. Perception-Grounded Step-Aware Auditing: assess progress confidence and locate corrective intervention

The Perception-Grounded Step-Aware (PGSA) auditor first uses a frozen small language model, such as Qwen3-0.6B, to parse a long instruction into ordered intermediate landmarks. For the currently active landmark, CLIP supplies global image-text similarity, retaining contextual evidence when the target is still distant. GroundingDINO then predicts a bounding box and detection confidence; once confidence passes the detection threshold, SAM3 produces a more precise object mask and an associated IoU score. Local CLIP matching is subsequently computed over the masked object region to reduce interference from background pixels. This cascade combines global context, object detection, and local object semantics into a step-level Soft Score, rather than merely judging whether an entire frame resembles a kitchen. Operators in the extracted Eq. (1) are damaged, so this note does not present a guessed weighted combination as the authors' exact formula.

The trajectory-level process score follows Eq. (2), subtracting a confidence threshold before averaging positive values:

\[ R_{proc}(\mathcal{T})=\frac{1}{T}\sum_{t=1}^{T}\operatorname{ReLU}(S_t-\tau_s). \]

Here \(T\) is trajectory length, \(S_t\) is the step-level Soft Score, and \(\tau_s\) filters low-confidence matches. ReLU prevents observations below the threshold from contributing reward, while averaging by length avoids the inherent preference for longer trajectories under simple accumulation. This score supports relative ranking; it is neither a calibrated probability of task completion nor proof that an action has the correct direction.

Meanwhile, a stricter hard threshold yields a structural mask, as described on pages 5–6:

\[ M_t=\mathbb{I}(S_t>\tau_h). \]

The active landmark advances only after the hard-match condition holds for \(c\) consecutive steps, preventing a single incidental sighting from counting as sub-instruction completion. The paper uses this structure to partition a trajectory into a valid prefix before the divergence point \(t_{div}\) and a divergent phase starting there. Continuous ranking and discrete correction boundaries therefore serve separate purposes instead of overloading one aggregate reward. However, the piecewise conditions in Eq. (3) are also damaged, and the missing appendix cannot clarify landmark-transition boundary handling; executable divergence-detection pseudocode would therefore be speculative.

2. Scenario-Conditioned Group Construction: repair near-misses when success exists and rescue relative supervision otherwise

By default, the policy samples \(K=8\) trajectories for each instruction and checks whether at least one receives a genuine success reward. When success exists, binary outcomes remain the primary supervision source, and process scores do not redefine success. The method additionally identifies failed trajectories with a large valid-prefix ratio, expressed in the paper as \(t_{div}/T>\eta\). For these near-misses, Repair Resampling (RR) preserves the prefix and replans a suffix from the divergence location, allowing at most \(N_{rep}=3\) attempts. A successful repair becomes an auxiliary demonstration; if every attempt fails, the original failed trajectory is retained. Repair is neither unlimited nor recursive: a successfully stitched trajectory is not repeatedly routed through another repair cycle, explicitly bounding this training budget. The subsequent auxiliary imitation term applies only to successfully repaired suffixes, rather than repeatedly treating their entire prefixes as additional expert demonstrations.

If the whole group fails, SACA switches to All-Failure Rescue (AFR). It selects the failure with the highest process score as the Pseudo-Anchor, without declaring that trajectory a pseudo-success. Hard negatives are mined from the remaining failures using action-level Longest Common Subsequence for prefix similarity together with the process-score gap relative to the anchor. Both quantities enter negative ranking, but the exact combination in Eq. (7) is damaged, so this account preserves the mechanism established by readable prose without guessing its weighted arithmetic. The selected negatives and anchor form a Reflection Subgroup containing at least 2 trajectories, and process advantages are normalized only within that subgroup. The comparison no longer asks which trajectory succeeded, but which related failure contains more useful local experience. The RR near-miss set belongs exclusively to the success-containing branch, while the AFR subgroup belongs exclusively to the all-failure branch; they are not a common pool of pseudo-positive and negative samples.

3. Robust Step-Level Optimization: trust valid prefixes without promoting an entire failure

The all-failure branch first constructs relative advantages using the mean and standard deviation of process scores within the Reflection Subgroup, then applies conservative scaling. Margin-Based Rescue checks how far the anchor's score exceeds the average score of the hard negatives. When this margin falls below \(\delta\), it shrinks advantages by \(\kappa<1\), avoiding strong policy updates based on weak visual evidence. Negative-Only Scaling subsequently multiplies only negative advantages by \(s\in(0,1]\), retaining positive preference while reducing punishment of other potentially plausible routes. These mechanisms address different risks: one weakens the overall update under uncertain evidence, while the other limits excessive rejection caused by noisy ranking.

Beyond trajectory-level updates, step-level constraints apply exclusively to the Pseudo-Anchor, not indiscriminately to every low-quality failure. Consistency Alignment behavior-clones the valid prefix, increasing the probability of actions judged correct by the structural mask. Contrastive Correction acts only at the divergence point, making the policy hidden state there more compatible with a positive action than with the erroneous action. The positive \(a^+\) is the simulator shortest-path action, while the negative \(a^-\) is the action the policy actually selected at that divergence point. This is not expert imitation over an entire failed trajectory: it preserves useful earlier behavior while explicitly discouraging the local decision that produced the deviation. Equations (12)–(16) have missing symbols in the extraction; this note explains optimization directions from readable prose without reconstructing loss signs or the complete contrastive normalization.

A Worked Example¶

Consider the setting in the paper's Figure 2: pass the glass doors, move toward the island, enter the kitchen, and stop in front of the microwave. The frozen language model produces ordered landmarks such as glass doors, island, kitchen, and microwave, and visual evidence advances the auditor's active landmark. Suppose a failed trajectory completes the earlier movements but turns incorrectly at the kitchen entrance; the following is a mechanism illustration, not an additional experimental observation. If another trajectory in the group succeeds, RR can preserve experience before the local deviation and resample a suffix from the divergence location into the kitchen. If every trajectory fails, that near-miss is not automatically labeled successful; it is considered for the anchor role according to its process score. Once selected as the anchor, its valid prefix receives consistency supervision, its incorrect entrance action is contrasted with the shortest-path positive, and the remaining subgroup trajectories provide relative process supervision. This explains why SACA neither discards an entire failure nor unconditionally imitates one that appears close to completing the task.

Loss & Training¶

The success-containing branch uses outcome-advantage GRPO on the original group and an auxiliary imitation term for successfully repaired suffixes. The all-failure branch uses conservative process-advantage GRPO on the Reflection Subgroup, together with anchor-specific consistency alignment and contrastive correction. Page 10 reports approximately 36 hours of SFT on 8 NVIDIA A6000 GPUs, with learning rate \(1\times10^{-5}\), a cosine schedule, and 10% warmup. Reinforcement fine-tuning takes approximately 24 hours, with learning rate \(1\times10^{-6}\), weight decay 0.01, and KL penalty coefficient \(\beta=0.04\). All PGSA models are frozen, so avoiding dedicated reward-model training does not imply that training incurs no additional model-call cost. The paper assigns default auditor thresholds and related settings to Appendix C, but the supplied cache ends with references; unavailable thresholds, loss weights, and finer implementation parameters cannot be reported.

Key Experimental Results¶

Main Results¶

Table 1 on page 9 compares R2R-CE and RxR-CE Val-Unseen, using Matterport3D environments simulated in Habitat. The selection below retains direct single-RGB-policy comparisons; † marks additional training data, with SACA† using ScaleVLN, and must not be conflated with the no-extra-data setting. SR is success rate and SPL is success weighted by path length, both higher-is-better; NE is navigation error in meters, lower-is-better. nDTW measures spatial fidelity between predicted and reference trajectories, higher-is-better; SR, SPL, and nDTW retain the paper's percentage scale.

Method and data setting	R2R NE ↓	R2R SR ↑	R2R SPL ↑	RxR NE ↓	RxR SR ↑	RxR SPL ↑	RxR nDTW ↑
StreamVLN, no extra data	5.43	52.8	47.2	6.72	48.6	42.5	60.2
SACA, no extra data	4.57	60.3	55.1	4.90	60.3	49.8	62.1
StreamVLN†	4.98	56.9	51.9	6.22	52.9	46.0	61.9
SACA†	4.19	64.7	56.9	4.75	62.1	51.7	66.0

Without extra data, SACA exceeds StreamVLN by 7.5 percentage points in R2R SR and 7.9 percentage points in SPL. RxR SR improves by 11.7 percentage points and SPL by 7.3 percentage points, indicating that benefits are not confined to shorter instructions. These are absolute percentage-point differences, not relative percentage increases; cross-paper pretraining, implementations, and total compute are not thereby fully controlled.

Ablation Study¶

Table 2 on page 11 adds components progressively; the selection below retains primary metrics from both Val-Unseen splits without extra data. SS denotes the continuous Soft Score, AFR All-Failure Rescue, and RR Repair Resampling; this is cumulative ablation rather than independent causal attribution to each component.

Config	R2R SR ↑	R2R SPL ↑	RxR SR ↑	RxR SPL ↑	RxR nDTW ↑
SFT baseline	52.8	47.2	48.6	42.5	60.2
Standard GRPO	54.1	48.8	49.9	43.0	60.3
GRPO + SS	55.4	49.6	51.8	44.3	60.7
GRPO + SS + AFR	58.2	52.4	56.4	47.6	61.3
Full SACA: add RR	60.3	55.1	60.3	49.8	62.1

Table 3 on page 13 removes individual objective constraints or robustness mechanisms from the full model under the same evaluation conditions.

Config	R2R SR ↑	R2R SPL ↑	RxR SR ↑	RxR SPL ↑
Full SACA	60.3	55.1	60.3	49.8
Without consistency alignment	58.0	51.6	57.1	46.2
Without contrastive correction	57.3	52.8	56.4	46.5
Without margin-based rescue	58.5	53.0	57.8	47.9
Without negative-only scaling	59.1	54.0	58.6	48.5

Key Findings¶

In Table 2, adding AFR raises RxR SR from 51.8 to 56.4, a 4.6-percentage-point gain; adding RR subsequently reaches 60.3, another 3.9 percentage points.
In Table 3, removing consistency alignment reduces R2R SPL from 55.1 to 51.6, a 3.5-percentage-point drop; removing contrastive correction reduces RxR SR by 3.9 percentage points.
In Table 4 on page 13, RxR SR is 55.2 with \(K=4\) and \(N_{rep}=3\), 60.3 with the default \(K=8\), and 59.9 with \(K=16\); larger groups do not yield monotonic improvement.
In Table 5 on page 14, R2R SR is 57.6 with global CLIP alone, 59.2 after adding boxes, and 60.3 after adding masks, supporting object-grounded auditing without separately establishing divergence-localization accuracy.

Highlights & Insights¶

Separate ranking from boundaries. Continuous process scores compare trajectories, while hard masks determine which steps are reusable, avoiding the assumption that an apparently good trajectory is reliable at every step.
Condition training on available evidence. Genuine outcomes remain primary when success exists, and process signals restore relative supervision when all trajectories fail; this division is more restrained than replacing rewards uniformly.
Correct locally instead of promoting a complete failure. The best failure remains a failure; its reusable content is the prefix and identifiable correction point, not its entire action sequence.

Limitations & Future Work¶

Correct perception does not establish correct action. Visibility, occlusion, repeated objects, and spatial relations can separate visual scores from actual navigation progress; this is a reader analysis of proxy-supervision risks, not a measured error rate reported by the paper.
Training depends on simulator capabilities. Suffix resampling requires local-state restoration, and shortest-path positives require environmental supervision, limiting direct transfer to online training of non-resettable physical robots.
Training costs need finer accounting. The paper reports SFT / RFT duration but provides no main-text breakdown of auditor costs or compute-matched baselines; no extra inference cost does not imply no extra training cost.
Generalization evidence is concentrated in simulation. The experiments cover two MP3D / Habitat navigation benchmarks; the inspected main text contains no physical-robot or dynamic-scene validation.
Reproducibility has a source-access boundary. The cache contains the main paper and references, not the cited Appendices A–C, and multiple equations are damaged; the 73% statistic, exact thresholds, and affected formulas require original materials rather than reconstruction from conventional implementations.

Compared with StreamVLN: StreamVLN emphasizes video-context modeling, whereas SACA focuses on reusing failures during reinforcement fine-tuning; Table 2 starts from an SFT baseline attributed to StreamVLN, so the main contribution is not a new inference-time memory architecture.
Compared with VLN-R1 / GRPO: SACA retains group-relative optimization but adds process-score comparisons for all-failure groups and separate constraints on credible prefixes and local mistakes, directly addressing missing information under sparse rewards.
Compared with dedicated process reward models: Frozen perception models reduce the need for task-specific reward-model training but shift reliability concerns to detection, semantic matching, and threshold selection; zero-shot auditing is not noise-free supervision.
Possible extensions: Auditor uncertainty could determine which prefixes deserve cloning, and alternative correction supervision could avoid shortest-path actions; these are research suggestions, not experiments completed in this paper.

Rating¶

Novelty: 4/5. Connects failure structure, conditional resampling, and conservative process advantages into an explicit training mechanism.
Experimental Thoroughness: 4/5. Two unseen validation benchmarks and layered ablations support the method, while deployment, auditor accuracy, and compute-matched comparisons remain limited.
Writing Quality: 4/5. The main argument and branch scopes are clear; damaged cached equations limit exact mathematical checking without establishing any defect in the original PDF typesetting.
Value: 4/5. Offers a reusable approach to learning from failures in sparse-reward embodied tasks, with practical benefits still dependent on auditor reliability and training-environment capabilities.