LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models¶

Conference: ECCV 2026
arXiv: 2606.23686
Paper: Project Page
Code: https://github.com/LIBERO-SAFETY/LIBERO-Safety
Area: robotics
Keywords: VLA safety, robot benchmarking, collision avoidance, semantic safety, data generation pipeline

TL;DR¶

This paper proposes LIBERO-Safety, a comprehensive benchmark for systematically evaluating the physical and semantic safety of Vision-Language-Action (VLA) models. It features a programmatic environment definition language (UBDDL) to generate safety-critical scenarios, an efficient keypose-driven trajectory synthesis pipeline yielding massive, collision-free demonstration data (19,664 trajectories), and a five-dimensional safety task taxonomy (75 tasks). The study reveals a fundamental tension in current VLA models between generalization capability and safe execution—while high-diversity training improves safety trajectory quality, task success rates remain bottlenecked by sub-optimal trajectory synthesis and semantic misalignment.

Background & Motivation¶

Background: Vision-Language-Action (VLA) models have emerged as the mainstream paradigm for generalist robot intelligence, showing significant progress in task execution, generalization capability, and cross-embodiment transfer. Existing benchmarks, such as the LIBERO suite, primarily focus on task-level success rates and evaluate models in static, deterministic environments, neglecting physical risks in real-world deployment.

Limitations of Prior Work: Current safety evaluations face two critical bottlenecks. First, existing benchmarks (including those introducing dynamic elements like VLA-Arena) heavily depend on human teleoperation for data collection, which is time-consuming and difficult to scale, failing to support the training demands of robust foundation models. Second, existing safety evaluations are mostly limited to simple tabletop static obstacles, ignoring multi-dimensional risks in real deployment—including semantic reasoning to refuse malicious instructions, safe interaction in human-robot collaboration, and 3D collision avoidance around complex hand-object configurations.

Key Challenge: While the performance of VLA models on standard task benchmarks is approaching saturation, their physical and semantic safety boundaries have never been systematically evaluated within a unified framework. These two dimensions of safety are rigid prerequisites for real-world deployment but suffer from a long-standing lack of quantifiable, comparable evaluation methods.

Goal: To build a unified evaluation framework covering physical safety (collision avoidance, human-robot interaction safety, spatial avoidance) and semantic safety (risk reasoning, malicious prompt refusal), and to provide a scalable and reproducible data generation scheme.

Key Insight: It is observed that the scalability bottleneck of safety evaluation stems from two levels: rigid scenario definitions (relying on hand-designed fixed templates) and inefficient data collection (1:1 output-to-effort ratio in human teleoperation). By parameterizing scenario definitions and decoupling data collection into high-level semantic intent and low-level motion planning, both diversity and scalability issues can be resolved simultaneously.

Core Idea: Programmatically generate diverse safety critical scenarios using a parameterized Unified Behavior Definition Description Language (UBDDL), and scale up the synthesis of collision-free demonstrations by replacing pure human teleoperation with a keypose-driven and motion-planning-optimized pipeline, thereby constructing a systematic evaluation benchmark covering both physical and semantic safety.

Method¶

Overall Architecture¶

The construction of LIBERO-Safety is divided into four stages: First, parameterizing safety-critical scenarios (including visual perturbations, dynamic entities, and safety constraints) using UBDDL. Second, human annotators label sparse object-centric keyposes (annotating only key operational pose phases). Third, the CuRobo motion planner uses these keyposes as seeds to synthesize fully collision-free trajectories under environmental obstacle constraints. Finally, after manual filtering, a large-scale safety demonstration dataset is formed. Based on this dataset, 8 VLA models are fine-tuned and evaluated for physical safety, and 2 embodied foundation models are evaluated for zero-shot semantic safety.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["UBDDL Task Definition<br/>w/ goals, safety constraints, randomization parameters"] --> B["Parameterized Scenario Generation<br/>7,603 scenario instances"]
    B --> C["Keypose Annotation<br/>Sparse object-frame keyposes"]
    C --> D["CuRobo Trajectory Planning<br/>Synthesize trajectories under collision constraints"]
    D --> E["Manual Filtering + Domain Randomization<br/>19,664 collision-free demonstrations"]
    E --> F["SFT Training (BC)<br/>8 VLA models"]
    E --> G["Zero-Shot Evaluation<br/>2 semantic foundation models"]
    F --> H["Physical Safety Evaluation<br/>SR / CR / LDLJ / Time"]
    G --> I["Semantic Safety Evaluation<br/>Refusal Rate / F1"]

Key Designs¶

1. UBDDL Parameterized Environment Definition: Upgrading Scenarios from Rigid Templates to Programmatic Generation Engines

Existing embodied benchmarks (e.g., LIBERO) rely on rigid scenario templates with hand-crafted, static environmental configurations for each task, failing to cover uncertainties such as sensor noise, viewpoint shifts, and lighting variations in real deployment. UBDDL extends standard BDDL in three core dimensions, formalizing each safety task as a tuple $\mathcal{T} = \langle \mathcal{G}, \mathcal{C}_{\text{safety}}, \mathcal{S}_{\text{init}}, \mathcal{P}_{\text{env}} \rangle$.

Specifically, $\mathcal{G}$ is the symbolic goal condition, $\mathcal{C}_{\text{safety}}$ is the boolean safety predicate enforced at each timestep (e.g., maintaining strict collision-free margins between the robot and obstacles), $\mathcal{S}_{\text{init}}$ specifies initial pose distributions of objects/robots and motion trajectories of dynamic entities, and $\mathcal{P}_{\text{env}} = \{\mathcal{C}_{\text{ext}}, \mathcal{V}_{\text{rand}}, \mathcal{N}_{\text{noise}}\}$ encompasses camera extrinsic parameters, visual variations (programmatic randomization of textures/materials/lighting), and sensor noise (five injection modes: motion blur, Gaussian defocus, fog, glass distortion, etc.).

UBDDL also explicitly introduces a (:dynamics) block to define autonomous moving entities (such as simulated human hands) independent of robot interaction, supporting both linear oscillation and circular rotation kinematics. At the safety constraint level, UBDDL enforces two types of checks: kinematic spatial constraints (physics-engine contact manifold detection, violating safety if there is any contact between the robot and protected entities) and dynamic force constraints (violating safety if end-effector contact force exceeds the threshold $F_{\text{max}}$). Through this parameterization mechanism, UBDDL automatically generates 7,603 unique scenarios with 953 diverse objects and 462 hand-object pairs.

2. Five-Dimensional Safety Task Taxonomy: Decoupling Safety into Independently Evaluable Physical and Semantic Dimensions

The authors operationalize the four foundational safety challenges into five distinct evaluation suites, each progressing through three difficulty tiers: L0 (Basic) to L2 (Out-of-Distribution, OOD generalization), totaling 75 individual tasks.

Affordance-Aware Grasping (AAG): Evaluates the model's ability to understand object affordance and identify safe interaction regions. L0 requires basic grasping under standard poses, L1 introduces random physical translations to test translation invariance, and L2 requires dynamic end-effector orientation adjustment for out-of-distribution rotations.
Human-Robot Interaction (HRI): Evaluates the collaboration safety of robots and human agents sharing a workspace. L0 covers interaction tasks under nominal conditions, L1 introduces spatial perturbations of human agents requiring dynamic trajectory modulation, and L2 injects diverse natural language rephrasings to test semantic robustness.
Tabletop Spatial Avoidance (TSA): Evaluates the ability to synthesize collision-free trajectories in unstructured, cluttered tabletop environments. L0 involves familiar static obstacles, L1 introduces dynamic moving entities requiring real-time reactive avoidance, and L2 tests zero-shot visual generalization on out-of-distribution geometric designs.
Free-Space Hand-Object Avoidance (FSHOA): Evaluates 3D geometric-aware avoidance between the end-effector and complex human hand-object configurations. Integrating the MANO parametric hand model with GrabNet synthesizes diverse grasping poses. L0 covers static hand-object configurations, L1 covers dynamic moving configurations, and L2 covers out-of-distribution human poses and unseen objects.
Semantic Safety Reasoning (SSR): Evaluates risk assessment and physical commonsense reasoning. Qwen3-8B programmatically generates adversarial instructions. L0 requires direct refusal of explicit malicious commands, L1 requires identifying implicit physical commonsense violations, and L2 requires evading subtle contextual traps.

Notably, the training set intentionally excludes the entire SSR suite and all physical suite L2 tasks: excluding SSR ensures the zero-shot purity of semantic safety evaluation, while excluding L2 strictly tests robustness under out-of-distribution configurations.

3. Keypose-Driven Data Generation Pipeline: Decoupling Human Teleoperation's 1:1 Output into 1:M Scalable Synthesis

This is the core design for resolving the data collection bottleneck. Human teleoperation on average takes 7.4 minutes per task with a 1:1 output-to-effort ratio, which is insufficient for generating large-scale diverse data required by safety benchmarks. The keypose-driven pipeline decomposes this process into two phases: humans only need to define a sparse sequence of keyposes represented in the target object's local frame $\mathcal{T}_{\text{keypose}} = \{\mathbf{T}_k\}_{k=1}^{K}$ (e.g., pre-grasp, grasp, pre-place, place), compressing human workload to 1.8 minutes per task. Then, the CuRobo motion planner uses these keyposes as sub-goals to synthesize complete trajectories $\mathbf{x}(t) \in SE(3)$ satisfying strict collision constraints under obstacle layout:

\[\mathcal{A}(\mathbf{x}(t)) \cap \mathcal{O} = \emptyset, \quad \forall t \in [0, T]\]

where $\mathcal{A}(\mathbf{x}(t)) \in $ is the physical volume occupied by the robot at pose $\mathbf{x}(t)$, and $\mathcal{O}$ represents the obstacle set.

The ingenuity of this design lies in defining keyposes in the object's coordinate frame rather than the world frame—this allows the same manipulation primitive to generalize to arbitrary spatial layouts, naturally enabling a 1:M data synthesis ratio. To boost diversity, each keypose is augmented into N=5 spatial variants, and one variant is randomly sampled at each phase of the sequence to construct a combinatorially explosive demonstration distribution. Finally, all trajectories undergo strict manual inspection to filter 19,664 high-quality collision-free demonstrations, further augmented with extensive domain randomization across both visual (textures, lighting, camera extrinsics) and physical (scene layouts, robot initial poses) dimensions.

4. Cross-Paradigm Dual-Track Evaluation Protocol: Decoupling Physical and Semantic Safety with Distinct Metrics

The Embodied Physical Safety Track covers the first 4 suites, and 8 VLA models are evaluated across three architectural paradigms: Standard VLA (OpenVLA, OpenVLA-OFT, $\pi_0$, $\pi_{0.5}$), World Model-based VLA (UniVLA, VLA-JEPA), and Dual-System VLA (GR00T N1.5, GR00T N1.6). All models undergo Supervised Fine-Tuning (SFT) through Behavior Cloning (BC) on the training set, with performance averaged over 10 independent trials under 3 random seeds. If a safety constraint violation (collision or force limit exceeded) occurs, the episode terminates immediately and is marked as a failure. The primary metric is SR (Success Rate, requiring both goal completion and zero safety violations), while auxiliary metrics include CR (Collision Rate, statistically counting terminations caused by collisions), LDLJ (dimensionless log-dynamic-jerk, measuring trajectory smoothness), and execution time. LDLJ is defined as:

\[\text{LDLJ} = -\ln\left(\frac{T^{3}}{v_{\text{peak}}^{2}}\int_{0}^{T}\left\|\frac{\text{d}^{3}\mathbf{x}(t)}{\text{d}t^{3}}\right\|^{2}\text{d}t\right)\]

The Semantic Safety Reasoning Track only evaluates two embodied foundation models, RoboBrain2.0 (7B, Chain-of-Thought reasoning) and RynnBrain-CoP (8B, Chain-of-Point physical reasoning). Given static environment observations and natural language instructions, the models determine whether executing the manipulation is safe. The primary metric is Refusal Rate (RR).

Loss & Training¶

All VLA models in the physical safety track adopt the Supervised Fine-Tuning (SFT) paradigm, using Behavior Cloning (BC) as the objective. Models are initialized with their respective official weights and trained on 8 NVIDIA A800 GPUs. The OpenVLA series uses LoRA (rank=32), a learning rate of $5 \times 10^{-4}$, and is trained for 150k-200k steps; $\pi_0$ and $\pi_{0.5}$ utilize the Flow-Matching framework with a learning rate of $2.5-5 \times 10^{-5}$, a global batch size of 32-256, and EMA weight smoothing (decay rate of 0.99-0.999); UniVLA and VLA-JEPA adopt a differential learning rate strategy (lower learning rate for the backbone, higher for the action head) to protect pre-trained knowledge; the GR00T series freezes the vision-language backbone, fine-tuning only the multimodal projector and diffusion action decoder, with GR00T N1.6 additionally using state dropout ($p=0.8$) to prevent proprioceptive overfitting. The two models in the semantic safety track are evaluated zero-shot without additional fine-tuning.

Key Experimental Results¶

Main Results¶

The table below extracts the success rates (SR%) of representative models across different paradigms on four suites and three difficulty levels in the physical safety track. $\pi_{0.5}$ performs best overall across all suites and difficulties, while the base OpenVLA model fails almost completely, and all models exhibit significant performance degradation at the L2 level.

Model	Paradigm	AAG L0/L1/L2	HRI L0/L1/L2	TSA L0/L1/L2	FSHOA L0/L1/L2
OpenVLA	Standard VLA	6.0 / 8.0 / 4.0	4.0 / 22.0 / 0.7	13.3 / 3.3 / 12.7	0.0 / 20.7 / 0.0
OpenVLA-OFT	Standard VLA	50.0 / 79.3 / 1.3	68.0 / 80.0 / 68.7	65.3 / 41.3 / 40.0	61.3 / 50.7 / 42.7
$\pi_{0.5}$	Standard VLA	78.7 / 59.3 / 35.3	84.7 / 88.7 / 83.3	58.0 / 62.7 / 56.7	55.3 / 58.7 / 51.3
VLA-JEPA	World Model	60.7 / 44.0 / 16.7	81.3 / 83.3 / 59.3	52.7 / 48.0 / 49.3	44.7 / 53.3 / 46.7
GR00T N1.6	Dual-System	64.7 / 51.3 / 19.3	74.7 / 85.3 / 73.3	55.3 / 48.7 / 52.7	50.7 / 54.7 / 50.0

Results from the Semantic Safety Reasoning Track show that RoboBrain2.0 reaches an RR of 80% at L0, but drops to 40% at L1 and only 20% at L2; RynnBrain-CoP scores only 56% at L0, but reaches 72% at L1 and 36% at L2, indicating that architectures embedded with physical reasoning chains are more robust against high-difficulty semantic traps.

Ablation Study¶

Effect of data scale on safety (FSHOA L2):

Configuration	SR(%)	LDLJ	Time(s)	CR(%)
CuRobo (Privileged Planner Upper Bound)	87.0	-14.47	261	0.0
$\pi_{0.5}$ (50 demos/task)	48.9	-17.78	362.5	12.7
$\pi_{0.5}$ (500 demos/task)	51.3	-17.47	355.0	10.0

Scaling demonstration data from 50 to 500 trajectories improves the SR by 2.4%, reduces the CR from 12.7% to 10.0%, and improves both LDLJ and execution time, verifying the positive effect of state space coverage on safe execution.

Robustness analysis under environmental perturbations (TSA L0, fully trained $\pi_{0.5}$ model):

Perturbation Type	SR(%)	LDLJ	Time(s)	CR(%)
Original Baseline	58.0	-17.64	361.7	2.7
Sensor Noise	58.0	-17.82	365.9	3.3
Robot Initial State	60.3	-17.45	342.8	4.7
Viewpoint Shift	60.7	-17.72	362.6	4.7
Scene Variation	60.0	-17.72	357.6	2.0
Object Layout Perturbation	56.3	-17.78	369.2	6.3
Unseen Object	56.7	-17.67	366.0	3.3

The results demonstrate that VLA models have good zero-shot robustness against visual local perturbations (noise, texture, single viewpoint shift), with SR stabilizing between 56% and 61%. However, under object layout perturbation, CR rises to 6.3% and SR drops to 56.3%, exposing current VLA models' fundamental bottleneck in spatial understanding.

Key Findings¶

The joint training strategy of $\pi_{0.5}$ (internet-scale pre-training + sub-task planning) achieves the best overall performance across all suites and difficulties, but the standard deviation across independent trials remains significant, indicating that reactive safety is still sensitive to environment initialization.
Under L2 out-of-distribution conditions, OpenVLA-OFT drops precipitously from 79.3% in L1 (AAG) to 1.3%, demonstrating that parameter-efficient fine-tuning degrades during severe distribution shift and is insufficient to replace explicit safety alignment.
High-diversity training data encourages emergent spatial reasoning: in zero-shot testing where all obstacles are removed, the model no longer blindly reproduces non-linear detour trajectories from the training data but directly synthesizes kinematically optimal straight-line paths.
In semantic safety reasoning, RoboBrain2.0 performs well on simple, explicit refusals but degrades rapidly under complex traps, whereas RynnBrain-CoP's Chain-of-Point physical reasoning exhibits higher resilience under L1 and L2, indicating that explicit reasoning frameworks are indispensable for semantic safety alignment.

Highlights & Insights¶

UBDDL Parameterized Design: Upgrades environment definitions from rigid templates to programmatic generation languages, serving a role similar to a "scene compiler"—allowing the generation of massive, diverse scenarios by simply editing parameters. This concept can be directly transferred to other embodied evaluations requiring large-scale scene diversity.
Object-Frame Keypose Definition: This is the most ingenious trick for scaling up data. Traditional teleoperation operates in the world frame and must be collected from scratch for new layouts. Switching to the object's local frame allows the same keypose sequence to apply to any location the object appears, naturally enabling 1:N data augmentation without any additional data augmentation strategies.
L2 as an Out-of-Distribution Probe: Excluding L2 completely from the training set is a clever design—it simultaneously tests physical generalization (new object geometry/new poses) and linguistic generalization (instruction rephrasings), enabling the benchmark to evaluate both "what has been learned" and "what can be transferred".
Chunk-Level CBF Post-Processing: The proposed upgrade of the step-level Control Barrier Function (CBF) filtering to chunk-level joint optimization in the appendix has high practical value—traditional step-level CBF corrections introduce high-frequency jitter, while chunk-level optimization explicitly incorporates a temporal smoothness regularization term $\lambda\sum\|u_{t+k} - u_{t+k-1}\|_2^2$ into the objective function to solve the constraint optimization problem for the entire action chunk at once, maintaining trajectory smoothness while ensuring safety. This design can be directly integrated into any VLA policy that outputs action chunks as a deployed safety post-processing module.

Limitations & Future Work¶

Sim-to-Real Gap: The authors candidly acknowledge that current physics engines cannot fully replicate complex contact dynamics, soft-body deformation, and fine friction models in the real world. The virtual hands generated by MANO also lack real human cognitive unpredictability and spontaneous behavioral variations. This is an inherent limitation of simulation benchmarks and the Achilles' heel of evaluation frameworks.
Silence on Negative Demonstrations: The current training set contains only collision-free safe trajectories and does not include "dangerous yet learnable" negative demonstrations (such as state-action pairs at the moment of collision). This forces the model to learn "what is safe" purely through imitation, preventing it from understanding "unsafe boundaries" through contrastive learning, which may limit the model's judgment in boundary cases. Future work could explore incorporating hard-negative unsafe trajectories during training.
Decoupling of Physical Safety and Task Success: The paper reveals a profound phenomenon—models often complete tasks but violate safety, or keep collision-free but fail the task (e.g., kinematic deadlock or timeout due to overly conservative avoidance), showing that current BC paradigms struggle to balance safety constraints and task goals. Future work may need to introduce constrained reinforcement learning or safety-aware planning to explicitly model this trade-off.
Human Involvement in Keypose Pipeline: Although keypose annotation compresses workload to 1.8 min/task, the pipeline is still not fully automated. Future work can explore using VLMs to automatically generate keypose annotations for end-to-end automated data generation.
Limited Scope of Semantic Safety Evaluation: The SSR suite currently only evaluates safety judgments at the instruction level, without tracking the full pipeline from semantic safety understanding to physical action execution; plus, it only covers two embodied foundation models, lacking semantic safety evaluations for the 8 VLA models in the physical safety track.

vs. LIBERO Suite: LIBERO/LIBERO-Plus/LIBERO-X focus on task execution and perceptual robustness, but evaluate entirely in static environments without safety constraints. LIBERO-Safety superimposes physical collision constraints and semantic safety judgments on top of this, serving as a "safety dimension completion" to the existing LIBERO ecosystem rather than a replacement.
vs. SafeVLA / Latent Safety Filter: These works introduce safety mechanisms at the policy optimization level (safety objective functions, latent safety constraints), representing "how to make models safer" solutions; LIBERO-Safety represents the "how to evaluate model safety" benchmark route. The two are complementary—methods like SafeVLA can be benchmarked using LIBERO-Safety.
vs. VLA-Arena: VLA-Arena introduces dynamic elements and basic safety constraints, but data collection still relies on human teleoperation and safety evaluations are confined to tabletop static obstacles. LIBERO-Safety significantly expands in terms of scene diversity (5 dimensions × 3 levels), data scalability (1:M output via keypose pipeline), and safety coverage (adding hand-object avoidance and semantic safety).
vs. RoboCasa / CALVIN: These benchmarks emphasize task diversity and long-horizon planning, but do not distinguish between task success and safe execution. A core contribution of LIBERO-Safety is decoupling these two—using SR for "can it succeed" and CR / LDLJ for "is it safe / smooth", which is crucial for evaluating real-world VLA deployment.

Rating¶

Novelty: ⭐⭐⭐⭐ Systematizes safety evaluation from scattered tabletop obstacle avoidance into a five-dimensional taxonomy of 75 tasks. The parameterized scene definition via UBDDL and the keypose pipeline are pragmatic technical innovations, though the core ideas (safety evaluation + programmatic scene generation) themselves are not entirely brand-new concepts.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 10 models across 4 architectural paradigms with full-matrix results of 75 tasks × 3 difficulty levels. Includes data scale ablations, environmental perturbation ablations, zero-shot baselines, obstacle-free SFT baselines, and real-world validations. The experimental design and analysis depth are highly robust.
Writing Quality: ⭐⭐⭐⭐ The layout of the paper is clear, and the logical progression of the 5 suites and 8 key findings makes the vast experimental results easy to digest. The appendix is detailed, containing the complete UBDDL definitions, task configurations, and hyperparameter tables. A minor downside is the high-density information pacing in some long sentences.
Value: ⭐⭐⭐⭐⭐ Provides the most systematic and reproducible benchmark framework and data pipeline for VLA safety evaluation to date. The revealed generalization-safety tension (highlighting sub-optimal trajectory synthesis and semantic misalignment as key failure modes) points out clear research directions, and the dataset and evaluation protocols are poised to become standard testing platforms in the field.