HUGE-Bench: A Benchmark for High-Level UAV Vision-Language-Action Tasks

Conference: ECCV 2026
Paper: ECCV paper page
Area: Robotics & Embodied AI
Keywords: UAV vision-language-action, high-level instructions, digital twins, process evaluation, collision safety

TL;DR

HUGE-Bench uses aligned 3DGS-Mesh digital twins reconstructed from real scenes to evaluate short-instruction, multi-stage UAV tasks jointly for process completion and collision safety; across 4 scenes, 8 tasks, and 2.56 million meters of trajectories, the strongest baseline, π0.5, still achieves only 0.581 average trajectory coverage.

Background & Motivation

UAV inspection involves more than locating a destination. An instruction such as inspecting the building on the left implicitly requires identifying the target, approaching it, descending to an inspection altitude, following its boundary, and returning to the initial state. Conventional aerial vision-language navigation often supplies detailed route descriptions and emphasizes endpoint success and path efficiency. This tests navigation, but may not distinguish a completed inspection from a shortcut back to the endpoint: identical endpoints can conceal very different levels of task completion.

Evaluating these high-level behaviors also requires an appropriate environment representation. Meshes support collision checking but may lack realistic visual appearance; 3D Gaussian Splatting (3DGS) renders realistic views but does not itself provide dependable collision geometry. When the perceived world and the collision world disagree, failures become difficult to attribute to the policy, visual domain differences, or safety modeling. Real-world capture, aligned dual-representation reconstruction, and scalable simulation make a benchmark spanning semantics, process, and safety feasible.

The paper therefore contributes task definitions, data, and an evaluation protocol rather than a new general-purpose control network. Short instructions preserve the way operators express intent, stage annotations expose execution progress, and digital twins provide a common basis for perception and collision checking. Core Idea: evaluate whether a short instruction produces a complete and safe flight procedure, using aligned 3DGS-Mesh environments to test semantic grounding, stage progression, and collision safety within one benchmark.

Method

Overall Architecture

HUGE-Bench conditions a policy on a natural-language task and UAV observations, with the policy predicting a next-step action target such as velocity or the next state. The benchmark first builds environments with aligned appearance and geometry, then generates target annotations, spatial referring expressions, and rule-based trajectories. Simulation evaluates how much of the required process the policy covers, whether its ordering is correct, how far it progresses through task stages, and whether it collides.

Three roles must remain distinct: landmark and stage annotations support data construction and evaluation; rule-based paths and planning support demonstration generation; evaluated policies predict actions from observations and language. The task decomposition used to generate demonstrations and the RRT planner should not be presented as inference-time components available to every baseline. Likewise, providing stage annotations does not mean that a policy has learned stage memory.

Because the contribution is a benchmark and data pipeline, the discussion follows task definitions, environment construction, demonstration collection, and evaluation without presenting them as a new neural architecture.

Key Designs

1. High-level tasks and stage annotations: a short instruction implies a flight procedure

The task suite does not merely substitute different target nouns into the same navigation objective. Landing requires orienting toward a target, reaching its overhead position, descending to a specified altitude, and hovering; it does not imply physical touchdown. Inspection-R requires reaching a road, descending, inspecting along the specified direction, and returning to the initial altitude. Inspection-B requires approaching a building boundary, descending, choosing an orbit direction, maintaining boundary-adaptive clearance, and returning to the initial pose.

The remaining tasks add geometric and memory demands. Mapping requires covering a designated region and restoring the initial altitude; Orbit-H controls orbit altitude, while Orbit-R controls an instruction-conditioned radius. Spiral Down requires multiple descending loops, and Traversal requires crossing a designated area while avoiding obstacles. Annotated stages make it possible to identify partial completion, such as reaching a building without completing the orbit, instead of assigning only an endpoint label. This also explains the relevance of an initial-frame reference, progress memory, and altitude perception.

2. Aligned digital twins and spatial references: visible targets acquire queryable 3D locations

The authors capture four outdoor scene types with a DJI M400 carrying a Zenmuse L2: office buildings, dense urban blocks, swamp and farmland, and construction roads. The total area is approximately 6.45 square kilometers. Main capture flights occur at roughly 85 meters, with 5280 × 3956 RGB images and recorded GNSS, UAV poses, and camera poses; supplementary low-altitude views improve reconstruction near the ground. Each scene has both a 3DGS model for appearance and a triangle mesh for collision queries and ray-cast depth.

Landmark annotation extends beyond marking an image location. A grid of top-down cameras at approximately 60 meters produces 3DGS map views and aligned mesh depth. An LLM predicts normalized image coordinates for landmarks, which undergo manual review before back-projection with camera intrinsics and depth. Extended regions such as buildings, roads, and fields additionally receive human masks. Before trajectory collection, the system samples initial views near the chosen target and prompts an LLM with its mask to generate expressions such as the building in the upper-left. This retains concise instructions while reducing ambiguity among similar objects. The language remains machine-generated with supporting human annotations, rather than a complete sample of spontaneous operator requests.

3. Rule-based demonstrations and generalization splits: vary starting views, targets, and wording

The aligned digital twin is imported into Isaac Sim, where task-specific rules generate 3D waypoints. Execution records synchronized RGB, depth, poses, flight states, and collision signals, with subtask-stage annotations also provided. Frames are sampled approximately every 1 meter of travel, with additional samples every 5° of heading change during turns to capture viewpoint transitions that distance-only sampling would miss. Traversal demonstrations use an RRT-based planner. Mesh floater removal and decimation reduce the cost of collision queries against dense geometry. These are demonstration-generation choices, not evidence that the learned policies perform explicit search.

The splits contain 5,330 training trajectories, 593 test-seen trajectories, and 294 test-unseen trajectories. Test-seen targets occur during training, but the initial altitude, horizontal distance, and relative direction are randomized. Test-unseen draws from held-out landmarks and meaning-preserving instruction paraphrases that replace action words or target descriptions. Consequently, unseen does not mean a completely held-out city or scene. Mapping accounts for 20.4% of trajectories and Traversal for 8.1%. Task lengths and geometric complexity also differ, so an aggregate task average cannot replace task-specific diagnosis.

4. Separate process and safety measurements: endpoint success must not hide omissions or collisions

Trajectory Coverage Rate (TCR) treats the predicted trajectory as a polyline. For each sampled ground-truth point, it computes the shortest Euclidean distance to that polyline and measures the fraction of points closer than a threshold. The paper reports TCR@1 m, TCR@2 m, and TCR@5 m, with Avg. TCR summarizing coverage across tasks. Coverage captures which places were visited, but traversing them in reverse can yield similar coverage. Normalized Dynamic Time Warping (nDTW) complements it through sequence alignment sensitive to trajectory order and direction. The main table does not further specify how Avg. TCR combines distance thresholds, so it should not be assigned an assumed single threshold.

Normalized Task Progress (NTP) assigns progress between 0 and 1 based on completion of annotated stages, allowing partial credit instead of treating every unfinished trajectory as zero. The main text does not provide detailed stage weights or stage-passing rules. Safety metrics apply to Traversal: Collision Rate (CR) is the fraction of episodes with at least one collision, while Collision-aware SPL (CSPL) combines endpoint success, collision-free execution, and path efficiency. An episode receives nonzero CSPL only if it reaches the goal without any collision; its efficiency term divides reference path length by the larger of reference and predicted path lengths. Any collision sets that episode's score to zero, even if the goal is eventually reached.

These quantities must be interpreted jointly. A nearly stationary policy can have low CR without covering the trajectory or advancing the task. A shortcut can reach the endpoint while omitting inspection. Process metrics and Traversal safety metrics also do not measure identical task populations, so high CSPL should not be interpreted as reliable completion of all eight high-level task types.

Loss & Training

The paper compares OpenVLA, MemoryVLA, FastVLM, π0, π0.5, and a depth-aware π0.5 without proposing a shared new loss function. MemoryVLA is an OpenVLA-based baseline with implicit memory. FastVLM is not evaluated merely as its original text-output VLM: it encodes visual-language hidden states and attaches an action expert initialized from π0. The depth-aware variant encodes depth maps as additional visual observations.

Evaluation uses observations from two time steps, the initial frame and the current frame, to help policies estimate relative spatial state and task progress. The authors state that baselines share the input format and protocol, but the depth-aware variant adds a sensing modality; it is therefore a controlled modality extension rather than an identical-information comparison. Two frames are also not equivalent to complete historical memory, leaving a meaningful role for the extra memory mechanism in MemoryVLA.

The supplied full-text cache contains the main paper and references but not the cited appendix or supplementary material. Optimizers, training steps, specific hyperparameters, detailed action implementations, and numerical expert-pilot results are unavailable in the readable material and are not inferred here.

Key Experimental Results

Main Results

The following test-set summary reproduces the numerical results in the original Table 2. Values retain the original decimal format; lower CR is better, while higher values are better for the other metrics.

Method	Avg. TCR	nDTW	NTP	CR	CSPL
OpenVLA	0.112	0.011	0.122	0.003	0.032
MemoryVLA	0.234	0.077	0.549	0.072	0.226
FastVLM + π0 action expert	0.285	0.136	0.633	0.060	0.391
π0	0.558	0.443	0.653	0.011	0.802
Depth-aware π0.5	0.577	0.459	0.617	0.013	0.792
π0.5	0.581	0.467	0.618	0.018	0.805

π0.5 leads Avg. TCR, nDTW, and CSPL, while π0 achieves higher NTP. The prose calls π0 the lowest-CR model, although OpenVLA's tabulated 0.003 is numerically lower. The table caption warns that low-performance behaviors such as spinning in place have limited comparative significance. Thus, π0's safety advantage should be qualified as a comparison among meaningfully executing policies: neither erase the OpenVLA exception nor declare OpenVLA the safest policy.

Ablation Study

The main paper has no separate ablation table. The following analysis extracts the memory and depth variant comparisons from Table 2. Differences are absolute changes computed by subtraction for this note, not additional experiments or relative percentage improvements reported by the authors.

Variant comparison	Avg. TCR change	nDTW change	NTP change	CR change	CSPL change
MemoryVLA − OpenVLA	+0.122	+0.066	+0.427	+0.069	+0.194
Depth-aware π0.5 − π0.5	−0.004	−0.008	−0.001	−0.005	−0.013

Implicit memory accompanies substantially stronger stage progression without a corresponding decrease in collision rate. Differences in the baselines' ability to move also prevent interpreting the CR change as an isolated safety effect of memory. Adding depth reduces π0.5's CR from 0.018 to 0.013 but lowers CSPL from 0.805 to 0.792: fewer collisions do not automatically improve the joint outcome of success and path efficiency.

Key Findings

• The best coverage and best stage progression belong to different models: π0.5 reaches Avg. TCR of 0.581, while π0 reaches NTP of 0.653. This supports multidimensional diagnosis instead of single-metric ranking.
• The discussion of Figure 6 identifies structured, landmark-centered landing and orbiting as easier to transfer than inspection, mapping, spiral-down, and traversal. The cache does not expose reliably extractable numerical values from the per-task plot, so no per-task numeric table is invented.
• Memory and depth address different weaknesses, but these results do not establish that any single modality or pretraining scale solves long-horizon safe execution.

Highlights & Insights

• High-level instruction difficulty lies in the implied procedure, not just vocabulary recognition. Explicit return-altitude, inspection-coverage, and orbiting requirements expose capabilities that endpoint metrics overlook.
• Appearance and collision geometry share an aligned environment, providing a clearer reference for failure analysis. This principle transfers to other embodied evaluations requiring both visual realism and contact constraints.
• Low collision rates must be interpreted alongside actual execution capability. OpenVLA illustrates why safety evaluation should not reward inaction as reliable execution.

Limitations & Future Work

• The authors explicitly acknowledge predominantly static environments without dynamic obstacles or changing weather and illumination. Simulation performance does not establish real-world UAV deployment reliability.
• There are only 4 scenes, and unseen evaluation primarily uses landmark holdouts and language paraphrases. Cross-city, cross-sensor, and cross-scene generalization require further splits and validation.
• Rule-based demonstrations and LLM-generated instructions scale efficiently but may constrain behavioral and linguistic diversity. Real operator requests and multiple valid procedures could reduce dependence on a single reference path as the only acceptable behavior.
• TCR is insensitive to ordering and must be paired with nDTW and stage progress. Stage-passing rules for NTP, threshold aggregation, and training details require supplementary material or implementation checks; the main text also provides no confidence intervals.
• Explicit stage memory and geometric safety constraints warrant investigation, followed by hardware-in-the-loop transfer tests. These are directions suggested by the results, not improvements validated in this paper.

Related Work & Insights

• vs AerialVLN / CityNav / OpenFly: These works primarily study language-guided navigation, whereas HUGE-Bench emphasizes multi-stage procedures implicit in short commands. Table 1 reports 32.07 m/word for HUGE-Bench versus 1.68 m/word for OpenFly. This reflects dataset construction, not a cross-benchmark performance gain.
• vs TravelUAV / UAV-Flow: The former emphasizes more realistic aerial navigation and assisted search, while the latter extends language-conditioned action control. HUGE-Bench distinguishes itself through the combination of stage annotations, process evaluation, and collision evaluation, not by being the first language-controlled UAV system.
• vs SAGE-3D-style hybrid representations: Pairing 3DGS with collision-ready meshes is not unique to this paper. The contribution applies that principle to real-scene high-level UAV task generation and diagnostic evaluation. The project page is provided in the paper; the supplied material does not identify a confirmed standalone code repository.

Rating

• Novelty: 4/5. Targeted task abstraction and metric composition, with the main contribution in benchmarking rather than a new control architecture.
• Experimental Thoroughness: 3/5. Multiple models and memory/depth variants are compared, but scene coverage is limited and training and expert-pilot details require supplementary verification.
• Writing Quality: 4/5. Clear motivation and complementary metrics, with qualifications needed for the lowest-CR claim and some aggregation details.
• Value: 4/5. Useful for diagnosing process and safety gaps in high-level UAV tasks, while remaining clearly short of real-world deployment certification.