HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task¶

Conference: ECCV 2026
Paper: ECCV paper page
Area: Robotics
Keywords: embodied safeguarding, contextual risk, visual grounding, process rewards, counterfactual safe pairs

TL;DR¶

HomeGuard trains an independent safeguard with context-guided reasoning that localizes interaction targets and environmental constraints before judging risks, raising the 8B model's risk match rate on HomeSafe-Bench from 33.20% to 74.90% while reducing false alarms on safe scenarios from 32.62% to 13.14%.

Background & Motivation¶

Household hazards do not always originate from malicious instructions. Heating food in a microwave is ordinarily benign, but metal cutlery left on the plate changes its safety; placing a bottle in a cabinet can also become dangerous when the bottle contains toxic chemicals and the cabinet stores food. Such contextual risks depend jointly on the instruction, the target's state, and its surroundings. Screening linguistic intent alone cannot resolve them.

Rule-based safeguards can translate a scene into object descriptions and logical constraints, but object relationships in cluttered homes are difficult to enumerate, and perceptual errors propagate through the rules. Asking a vision-language model (VLM) for a holistic safety judgment is more flexible, yet it can miss small objects or infer nonexistent hazards merely from the presence of appliances. The paper therefore advances a testable hypothesis: the bottleneck is not only safety knowledge but also task-relevant visual attention and grounding. Supplying the base model with ground-truth target and constraint boxes substantially improves its performance.

HomeGuard consequently uses a safeguard interface independent of the downstream robot architecture, requiring the model to explain both the hazard and its location. Core Idea: make target and constraint regions mandatory intermediate evidence for risk reasoning, then train their localization with process rewards so that increased refusal is not mistaken for improved safety.

Method¶

Overall Architecture¶

The inputs are a household scene image and a natural-language instruction. Outputs include a safety decision, a hazard explanation, violated safety principles, and two-dimensional bounding boxes for interaction targets and environmental constraints. A target is an object or location involved in the requested interaction; a constraint is a background object that can affect execution safety without necessarily being manipulated.

Context-Guided Chain-of-Thought (CG-CoT) proceeds through intent screening, interaction target inspection, environmental constraint analysis, and integrated risk assessment. The first three are not separate detector calls: they are stages of a structured output protocol learned by the same VLM. Likewise, "active perception" here does not mean that a robot moves its camera to acquire additional views.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Scene image + instruction"] --> Intent["Intent Screening"]
    Intent -->|No explicit malice| Target["Interaction Target<br/>Inspection"]
    Target --> Context["Environmental<br/>Constraint Analysis"]
    Context --> Judge["Integrated Risk<br/>Assessment"]
    Judge --> Output["Decision + explanation<br/>+ visual anchors"]
    Intent -->|Explicit malice; terminate early| Output
    Output --> Planner["Downstream planner"]

The diagram depicts inference data flow. During training, annotated HomeSafe data first supports supervised fine-tuning, followed by GRPO optimization of final judgments and intermediate grounding. Ground-truth boxes enter training rewards or a dedicated oracle baseline, not normal deployment inputs. A downstream planner can use predicted anchors to revise its plan, but HomeGuard is not a complete motion controller.

Key Designs¶

1. Intent Screening: separate malicious commands from contextual hazards

The model first checks whether the instruction explicitly requests destructive or otherwise malicious behavior and terminates reasoning early when such intent is detected. An ordinary household instruction cannot be approved merely because its wording is harmless: its execution conditions must still be checked visually. This separates commands that can already be judged from language from contextual hazards that require visual examination.

2. Interaction Target Inspection: bind the instruction to specific objects and states

The model emits [target_area][description][bbox][state], where bbox contains the two-dimensional top-left and bottom-right coordinates, and state describes safety-relevant properties such as material, temperature, or spatial position. Targets include not only objects to grasp but also destination regions, such as an empty shelf location. Placing a bottle in a cabinet thus becomes an operation between a particular bottle and a particular destination rather than a relation between two abstract nouns.

Explicit binding helps prevent nearby, irrelevant objects from being mistaken for the requested target. However, producing a state description does not establish that a sensor measured temperature or electrical status. The method still infers attributes from visual and semantic cues, so reliability depends on the available evidence.

3. Environmental Constraint Analysis: inspect spatial neighborhoods and functional connections

After locating the targets, the model identifies objects that could interfere with safe execution and expresses them as [constraint_area][description][bbox][state]. Spatial neighborhoods include flammables near heat sources, metal cutlery inside a container, and electronics near a pouring location. Functional connections can extend to an appliance's wiring, so this is neither a fixed-radius crop nor a nearest-object heuristic.

Separating targets from constraints matters because a model can localize the intended teacup correctly while entirely missing the nearby laptop. Constraint analysis asks how material, placement, fragility, or active state causes a hazard instead of simply listing objects from stereotypically dangerous categories. Training rewards the two box types separately with IoU, preventing aggregate grounding scores from concealing missed background constraints.

4. Integrated Risk Assessment: ground both refusal and approval in visual evidence

The model combines intent, target states, and environmental constraints to determine whether execution violates a safety principle. Safe cases return [safety_hazard]["no safety hazard"]; unsafe cases produce [safety_hazard][description] and [safety_principle][description]. The reasoning trace is enclosed in <think>...</think>, followed by a separate final judgment.

The objective goes beyond correct binary classification. A model that says "unsafe" for the wrong reason may still fail to guide a planner toward a suitable correction. Training therefore also supervises hazard explanations and principle categories. At deployment, boxes provide spatial cues for actions such as removing a hazard before continuing, but they are not three-dimensional collision-avoidance certificates and do not guarantee that a planner uses them correctly.

A Worked Example¶

In Figure 2, the instruction asks the agent to put the bottle left of the toaster into an open cabinet. Intent screening treats this as routine storage. Target inspection, however, recognizes toxicity from warning labels on the bottle and identifies an empty cabinet location as the destination.

Constraint analysis then notices spices and food beside that location rather than treating available space as sufficient evidence of safety. Integrated assessment rejects direct placement because of contamination and accidental ingestion risks, invoking the principle that food and toxic chemicals should be stored separately. The instruction has not become more malicious; visible object states and neighborhood relationships make an ordinary command hazardous. This also explains why adding safety principles to a prompt alone is insufficient.

Loss & Training¶

HomeSafe uses real indoor images from CA-1M and SUNRGBD and contains 10,257 unsafe scenarios and 5,710 safe scenarios, covering 7 hazard categories and 33 safety principles. Its pipeline first plans hazardous and safe edits for an instruction, inserts or replaces objects while preserving the real background, applies fidelity and hazard-consistency filters, and annotates target boxes, constraint boxes, states, and reasoning traces.

Counterfactual safe pairs are particularly important: the instruction remains unchanged while the visual hazard factor changes, for example by replacing a metal container with a ceramic one. This prevents classification based solely on instruction keywords. The overall data counts do not indicate strict one-to-one pairing, so the two class totals must not be described as equal numbers of paired images. The paper states that the separately human-verified test set does not overlap with training; this alone does not establish independence at every source-scene level.

The first stage uses Low-Rank Adaptation (LoRA) for supervised fine-tuning (SFT), mixing grounded four-step reasoning examples with general instruction-following data to teach safety principles and output structure while retaining general capabilities. The second stage performs reinforcement fine-tuning (RFT) with Group Relative Policy Optimization (GRPO), optimizing both final outcomes and intermediate visual evidence.

Outcome rewards cover format compliance, binary safety accuracy, semantic consistency of hazard explanations, and safety-principle ID agreement. Semantic consistency uses cosine similarity between all-MiniLM-L6-v2 sentence embeddings. The principle-ID check additionally constrains hazard categories, addressing cases where similar embeddings conceal an incorrect category.

Process rewards calculate IoU between predicted and ground-truth target boxes and, separately, constraint boxes. An intermediate box that cannot be parsed in the prescribed format receives zero reward, removing the need for a separate intermediate-format reward. Intermediate state descriptions are not rewarded by exact textual matching, avoiding penalties for legitimate alternative expressions. After combining the reward signals, outputs sampled for the same input are normalized within their group:

\[ A_i = \frac{r_i - \operatorname{mean}(\{r_j\})}{\operatorname{std}(\{r_j\})}. \]

Here \(r_i\) is the total reward for candidate output \(i\), and \(A_i\) is its GRPO relative advantage. Equations (1) and (2) are incompletely extracted in the supplied text cache, so the precise weighted total-reward formula is not reconstructed. The cache also omits the referenced appendices, preventing confirmation of reward weights, sampling group size, and complete training hyperparameters from the available material.

Key Experimental Results¶

Main Results¶

HomeSafe-Bench contains 512 unsafe images and 272 safe images. RIR is the fraction of unsafe scenarios correctly identified as risky. RMR measures semantic agreement between generated hazard explanations and ground-truth descriptions, judged by Qwen3-VL-235B-A22B-Thinking. OR is the fraction of safe scenarios incorrectly flagged as unsafe. T-IoU and C-IoU measure grounding overlap for target and constraint regions, respectively.

The following entries come from Table 1. RIR, RMR, and OR are percentages; IoU values are not percentages. High RIR must be considered alongside RMR and OR.

Model	RIR ↑	RMR ↑	T-IoU ↑	C-IoU ↑	OR ↓
Qwen3-VL-4B-Thinking	66.67	33.14	0.5902	0.2212	29.31
Qwen3-VL-8B-Thinking	67.58	33.20	0.5732	0.2694	32.62
Gemini-3-pro	87.45	60.98	0.6745	0.5217	25.23
AgentSpec	66.67	36.67	/	/	28.41
HomeGuard-4B	90.00	63.72	0.6709	0.4873	21.96
HomeGuard-8B	90.98	74.90	0.7206	0.5562	13.14

Compared with their respective base models, the 4B and 8B variants improve RMR by 30.58 and 41.70 percentage points and reduce OR by 7.35 and 19.48 percentage points. These are not relative percentage improvements or overall classification accuracy gains. Against Gemini-3-pro, the 8B model improves RMR by 13.92 percentage points and reduces OR by 12.09 percentage points.

Ablation Study¶

Table 3 uses Qwen3-VL-4B-Thinking. direct GT bbox supplies ground-truth target and constraint boxes to the model without fine-tuning; it is an oracle prompting comparison, not an equivalent alternative to automatic grounding at deployment.

Config	RIR ↑	RMR ↑	T-IoU ↑	C-IoU ↑	OR ↓
Without constraint IoU reward	85.49	50.20	0.6764	0.2604	22.43
Without target IoU reward	90.00	56.27	0.5642	0.4239	37.03
Without all IoU rewards	90.98	50.98	0.5956	0.4052	42.35
direct GT bbox	84.71	51.96	0.7936	0.6242	22.79
direct CoT	70.07	43.50	0.6488	0.3302	23.53
direct principle	71.26	47.83	0.4308	0.2743	35.56
HomeGuard-4B	90.00	63.72	0.6709	0.4873	21.96

Key Findings¶

Constraint supervision directly affects whether an explanation identifies the correct hazard: removing it reduces RMR from 63.72% to 50.20% and C-IoU from 0.4873 to 0.2604. Correctly recognizing the interaction target alone is insufficient.
Removing all IoU rewards raises RIR from 90.00% to 90.98%, but OR rises from 21.96% to 42.35%. This supports the interpretation that grounding supervision reduces unsupported refusals; detection rate alone is a poor basis for ranking safeguards.
In Table 2, the 8B model reaches 94.73%/72.12% RIR/RMR on EARBench. On MSSBench, however, its RIR/OR of 77.63%/25.00% trails Gemini-3-Pro's 85.53%/10.53%. Effective transfer does not imply leadership on every dataset.
In the IS-Bench planning experiment in Section 4.5, safety tips raise Qwen3-VL-8B-Thinking's task success rate from 61.36% to 73.91% and safe success rate from 29.54% to 45.65%, gains of 12.55 and 16.11 percentage points. This does not replace closed-loop safety validation on physical robots.

Highlights & Insights¶

Safety knowledge needs localizable evidence. The key is not merely longer explanations but explanations tied to specific regions. Separate target and constraint supervision distinguishes selecting the wrong object from missing a surrounding hazard.
Counterfactual safe pairs discourage category shortcuts. The same benign instruction can be safe or unsafe under different visible states, encouraging attention to materials and relationships. This addresses household risks more directly than simply adding more malicious instructions.
Visual anchors connect screening to plan repair. A refusal label stops a task, whereas a hazard location can help a planner remove a conditional danger first. The teacup, teapot, and laptop example illustrates this interface's potential without establishing geometric safety guarantees.

Limitations & Future Work¶

The authors acknowledge that the current method uses only two-dimensional images; precise clearance, occlusion, and three-dimensional reachability need depth or point-cloud information. Two-dimensional boxes cannot directly constrain an entire robot trajectory.
Static risk assessment does not cover moving people or objects. Video-based temporal modeling and long-horizon closed-loop deployment on physical robots are future work, not evidence that this system already certifies household robot safety.
Both training and the internal benchmark rely on edit-based synthesis, which may retain generator artifacts or hazard-category biases. External benchmarks help, but more natural, unedited household evaluation remains necessary.
RMR relies on a model judge, and semantic similarity in the reward is not an accident probability. The supplied cache lacks appendix prompts and implementation details, limiting independent inspection of the evaluation criteria and exact reproduction of training.

vs ThinkSafe / Poex: The paper treats these as prompt-based runtime safeguards, whereas HomeGuard learns visually grounded judgments through training. Its gains use specialized data and supervision and should not be interpreted as an equal-training-budget architectural comparison.
vs AgentSpec: AgentSpec represents and enforces constraints with a domain-specific language, while HomeGuard learns visual evidence for more complex contexts. Verifiable logic and open-ended visual understanding provide different capabilities; learned grounding does not replace formal guarantees.
vs embodied safety alignment: Training a planner or action model for safety couples the solution to its behavioral interface. HomeGuard offers an independent checker that can be attached to different planners. A concrete extension is to test whether grounding confidence can support graded decisions to approve, request further observation, or refuse rather than retaining only binary decisions.

Rating¶

Novelty: 4/5. Combines target/constraint grounding, counterfactual safe examples, and process rewards around a concrete embodied safeguarding problem.
Experimental Thoroughness: 4/5. Includes an internal benchmark, four external benchmarks, grounding-reward ablations, and planning applications, but lacks closed-loop physical robot evaluation.
Writing Quality: 4/5. The method and ablations support each other; percentage-point gains need careful interpretation, and the available text extraction contains damaged equations.
Value: 4/5. Provides a transferable mechanism for reducing missed hazards and unsupported refusals, but static two-dimensional judgments are not deployment safety guarantees.