Falcon: Functional Assembly and Language for Compositional Reasoning in X-ray¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: Multimodal VLM
Keywords: X-ray baggage screening, compositional threat reasoning, functional assembly, structured safety state, referring functional grounding

TL;DR¶

The paper redefines "threat" in X-ray baggage screening from "an object of some category was detected" to "spatially separated components can, by functional compatibility, be assembled into a usable device"; it contributes Falcon-X (a benchmark of ~7,000 real scans plus counterfactual extensions, with dense masks and structured supervision over component presence, functional links, and scene risk) and Falcon, which compresses a variable number of component instances into a fixed structured safety state via segmentation-aware region encoding and three component queries, then injects that state as tokens into an LLM, surpassing the next best method by 30.38 points in cIoU on referring functional grounding (RFG) with 50.45 cIoU.

Background & Motivation¶

X-ray baggage screening is a scene dominated by transparent materials, severe overlap, and heavy clutter. Over the past decade, deep learning has pushed prohibited-item detection quite far on large-scale benchmarks such as SIXray, OPIXray, HiXray, and PIDray, and recent work (e.g., the STCray dataset and the Sting-Bee model) has brought multimodal models into the domain so that a machine can caption a scan and answer questions about it. Yet all of these systems, old and new, are object-centric at heart: threat is attributed to a single category — a gun is a gun, a knife is a knife. In real screening the risk often does not arise that way. Improvised explosive devices (IEDs) are frequently transported dismantled, with battery, detonator, and main charge separated inside the same bag. Individually all three are benign — one battery is no different from another — and only their functional compatibility makes them a threat. The question that actually matters is therefore not object recognition but relational safety inference: under severe superposition and transparency, do several spatially separated components collectively constitute a functional threat?

Prior work cannot answer this, for two reasons. The surface one is the evaluation protocol: existing X-ray benchmarks are almost all closed-set and object-level, annotating intact prohibited items (a whole gun, a whole pair of scissors) and measuring "was it detected." No task asks whether several parts can be assembled into something. The deeper reason is a double barrier of data and supervision. Multimodal large language models (MLLMs) can encode regions as tokens and can refer and segment, but their inference about relations between regions rests entirely on implicit attention; nothing explicitly supervises how much the battery-to-detonator compatibility should weigh in a risk judgment. At the same time, training "missing component" reasoning requires samples spanning the whole spectrum — battery only, battery plus detonator, all three present — and such partially assembled real X-ray images are essentially unobtainable from a screening line: one cannot plausibly lay out a dismantled IED inside real luggage and scan it. Relationally consistent safety inference in this imagery has therefore remained largely unexplored.

This paper's angle is that since risk is by nature a property of the relations between components, it should not be left hidden in attention weights but written down as explicit intermediate quantities. The authors define the safety state as a triple — which components are present (a presence vector), how compatible they are at the type level (a functional-link matrix), and the resulting scene risk (a continuous score) — and anchor "component" to pixels through segmentation masks. On the data side, mask-guided inpainting synthesizes controllable missing-component variants from the same real image, filling in the supervision that was lacking; on the model side, a structured adapter is inserted between region-level perception and language decoding, compressing a variable number of regions into a fixed functional state and turning that state into tokens the LLM can read. Core idea: recast compositional threat from implicit attention-based inference into an explicit structured safety state — presence vector plus type-level functional compatibility matrix plus scene risk — predicted from mask-aware region features by component-query attention and injected as seven safety tokens into a frozen LLM, so that functional-assembly reasoning becomes a supervised, ablatable computation on the perception side.

Method¶

Overall Architecture¶

Falcon addresses the information break along the chain "region-level perception → functional state → language." Given an X-ray image (optionally with a query or a referring expression), the model first performs segmentation-aware instance localization, then aggregates instance features into a structured safety state organized by component type, and finally hands that state, together with image, region, and text tokens, to an LLM that decodes localization results, missing-component judgments, and risk scores. The paper writes this chain explicitly as \(I \rightarrow R \rightarrow \{h_c\} \rightarrow (\hat{\mathbf{p}}, \hat{L}, \hat{r}) \rightarrow Y\), where \(R\) denotes instance regions, \(\{h_c\}\) component-level slot embeddings, \((\hat{\mathbf{p}}, \hat{L}, \hat{r})\) the predicted presence, functional links, and scene risk, and \(Y\) the generated response. This chain is what the paper calls a semantic bottleneck: the only new information the language side receives is these few structured scalars, so relational reasoning is forced to happen on the perception and adapter side.

Concretely, it is assembled from four pieces. A DINOv2-L/14 backbone encodes the 448×448 input into dense patch tokens, which are then linearly aggregated over non-overlapping 2×2 windows, cutting the sequence length to a quarter while preserving spatial locality, yielding a feature map \(F\). A class-agnostic RF-DETR head produces instance proposals (boxes, masks, confidences) on \(F\); after NMS and score filtering the top-100 are kept, and each proposal's box features and mask features are fused into a "language-aligned" region token. The Structured Safety Adapter (SSA) takes this variable-sized set of region tokens, aggregates it through three learned component queries into three component slots, and predicts component presence, pairwise functional links, and scene risk in parallel from the slot embeddings. The seven scalars are then tokenized one by one, concatenated with image, region, and text tokens, and fed to Vicuna-7B to generate the answer. Training has three stages: the detector first, then the SSA alone with the detector and LLM frozen, and finally LoRA instruction tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["X-ray image + query"] --> B["DINOv2 encoding<br/>2×2 token aggregation"]
    B --> C["Mask-aware region encoding<br/>ROIAlign + mask pooling"]
    C --> D["Structured safety state<br/>SSA component slots + three heads"]
    D --> E["Safety token injection<br/>7 structured tokens"]
    E --> F["Vicuna-7B decoding"]
    F --> G["grounding / missing part / risk"]
    H["Counterfactual data construction<br/>mask-guided inpainting"] -->|training supervision| D

Key Designs¶

1. Mask-aware region encoding: making visual tokens hug component outlines under superposition

In an X-ray image objects interpenetrate and overlap, so a bounding box routinely swallows a large amount of background clutter, diluting whatever mean feature is pooled inside it. Falcon therefore does not take the box alone: for each proposal it extracts box features with ROIAlign from the feature map \(F\) on one path, and on the other path performs mask pooling, i.e. the weighted average of backbone features inside the predicted mask,

\[f_i^{\text{mask}} = \frac{\sum_x m_i(x)\,F(x)}{\sum_x m_i(x)}\]

The two paths are concatenated and passed through a learnable projection to give the region embedding \(z_i = W[f_i^{\text{roi}} \,\|\, f_i^{\text{mask}}]\), a language-aligned region token for the downstream modules. The value of this step is stressed throughout the paper: precisely because components are occluded and fragmented under superposition, only the exact spatial support a mask provides can separate "the partially visible battery" from "the pile of cables around it" at the feature level; and since referring functional grounding must output pixel-level results downstream, it depends directly on this mask pathway. In other words, the first step of compositional reasoning has to yield clean regions, or every later relational judgment is made on noise.

2. Structured safety state: compressing a variable region set into fixed functional slots with three component queries

The number of component instances in an image is arbitrary — there may be one battery, or two batteries plus a detonator — but the very notion of a functional assembly is defined between component types, and with a variable count there is no fixed-dimensional target to supervise. The SSA's answer is to maintain three learned component queries \(q_c\) (for battery, detonator, and main charge respectively), let each query attend over the whole region set with scaled dot-product attention, and aggregate the region embeddings into one component slot by those weights,

\[A_{c,i} = \mathrm{softmax}_i\!\left(\frac{q_c^{\top} z_i}{\sqrt{d}}\right), \qquad h_c = \sum_{i=1}^{N} A_{c,i}\, z_i\]

Multiple instances of the same component type contribute proportionally to a single slot, and slot specialization emerges from component-level supervision during training (the paper labels this query aggregation CAA without expanding the acronym). On top of the three slot embeddings the SSA emits three families of structured signals in parallel. Each slot passes through a linear projection and a sigmoid to give that component's presence probability \(\hat{p}_c\). For the three predefined component pairs (battery–detonator, battery–main charge, detonator–main charge), the two slot embeddings are concatenated and fed to a light MLP that outputs a pairwise functional-link probability \(\hat{\ell}_{uv} = \sigma(f_\ell([h_u; h_v]))\); the paper clarifies that "link" here means functional compatibility, not physical connectivity. Scene risk is regressed from the three slot embeddings together with the three link probabilities, \(\hat{r} = \sigma(f_r([h_1, h_2, h_3, \hat{\ell}_{bd}, \hat{\ell}_{be}, \hat{\ell}_{de}]))\), rather than directly from raw visual tokens. Hanging risk off the link probabilities instead of reading it off the image is the pivotal choice here: it makes risk genuinely a function of "presence × compatibility" in the computation graph, so that "all three present but incompatible" and "one missing but highly compatible" become structurally distinguishable, instead of being left for the LLM to guess.

3. Safety token injection: turning structured predictions into a decoding precondition for the LLM

If the structured predictions served only as an auxiliary training loss, the LLM would never see the state at decoding time and the relational bias would be wasted. Falcon instead turns the seven scalars \(v = [\hat{r}, \hat{p}_1, \hat{p}_2, \hat{p}_3, \hat{\ell}_{bd}, \hat{\ell}_{be}, \hat{\ell}_{de}]\) one by one into tokens: \(t_k = e_k^{\text{type}} + \phi(v_k)\), where the type embedding \(e_k^{\text{type}}\) tells the model which variable this is and the projection \(\phi\) moves the value into the LLM's embedding space. The seven safety tokens are concatenated with the image, region, and text tokens into \([T_{\text{image}} \| T_{\text{region}} \| T_{\text{SSA}} \| T_{\text{text}}]\) and handed to Vicuna-7B. The reason this works is the strong informational constraint it creates during the second training stage: with both the detector and the LLM frozen, the only pathway that still carries gradients is the SSA and the projection layers, so to improve the generated text the model has no choice but to put the relational judgment into these seven numbers. The subsequent ablation, where switching on the link head yields the largest gain, is exactly this constraint paying off — it confirms the relational quantities really carry most of the information, rather than the LLM being able to supply it by looking at the image.

4. Counterfactual data construction: manufacturing controllable missing-component states with mask-guided inpainting

Missing-component identification and functional completeness require samples spanning single components, partial assemblies, and complete configurations, yet images of "half-dismantled" assemblies are essentially never collected on a real screening line. The obvious shortcut is to erase the pixels of a selected component (mask zero-filling), but that leaves visible artificial traces and the model may well learn "report a missing component whenever I see a clean rectangular hole," substituting a shortcut for reasoning. Falcon instead uses mask-guided inpainting: a background mask taken from the same image is used to cover the selected component instance, so what fills the hole is genuine X-ray clutter and occlusion structure and the result stays semantically coherent; enumerating feasible subsets of the component set \(C\) then yields a balanced corpus covering the various combinations, roughly 50,000 images in total including the real ones. A companion decision concerns the risk labels: risk is provided by expert annotators and deliberately retains perceptual uncertainty — a scene missing one component may still receive a high score because the part could be occluded or concealed. This seemingly minor setting matters, because it severs the trivial derivation of \(r\) from the presence vector \(\mathbf{y}\) and forces uncertainty-aware judgment.

A Worked Example¶

Take the qualitative example in the paper — a bag containing a detonator and a battery but no main charge — and walk it through. The image is resized to 448×448 and fed to DINOv2; patch tokens are aggregated over 2×2 windows into the feature map. RF-DETR generates up to 300 class-agnostic proposals, and after NMS (0.6) and score filtering (0.15) the top-100 instance regions are kept; each is fused by ROIAlign plus mask pooling into one region token. The three component queries of the SSA attend over this set: the battery query and the detonator query pick up large weights on their respective instances, while the main-charge query finds no region to aggregate and receives no effective support. The three prediction heads then yield a presence vector with battery and detonator near 1 and main charge near 0, link probabilities in which the battery–detonator pair is comparatively compatible while the two links involving the main charge stay low, and a scene risk that lands near 0.5. The seven scalars become seven safety tokens, are concatenated with the image/region/text tokens, and the LLM answers "missing component: main charge; risk score 0.5." By contrast, if all three components were present in the same scene, the link probabilities would rise overall and the risk score would be pushed close to 1.0, and the answer would switch from "missing part" to "could form a functional IED" — risk, presence, and links move in the same direction in the output, which is precisely the relational consistency that explicit state injection is meant to produce.

Loss & Training¶

Training proceeds in three stages so that segmentation learning is decoupled from structured multimodal reasoning. Stage 1 (segmentation-aware proposal learning) trains RF-DETR in a class-agnostic manner, treating all key components as a single threat class, with the loss \(\mathcal{L}_{S1} = \mathcal{L}_{\text{det}} + \mathcal{L}_{\text{seg}}\), where \(\mathcal{L}_{\text{det}}\) is the standard DETR set-prediction loss (classification, box regression, GIoU) and \(\mathcal{L}_{\text{seg}}\) combines mask BCE and Dice terms; only RF-DETR is optimized, all multimodal modules stay frozen, and the run is 12 epochs, with the resulting proposals fixed for later stages. Stage 2 (structured multimodal alignment) freezes RF-DETR and the LLM and updates only the SSA and the projection layers, supervising language generation and structured safety prediction jointly — an autoregressive negative log-likelihood on the language side and an L1 regression on presence, link matrix, and scene risk on the structured side:

\[\mathcal{L}_{S2} = -\sum_t \log P_\theta\!\left(y_t \mid y_{<t}, I, Z, \hat{\mathbf{p}}, \hat{L}, \hat{r}\right) + \lambda\,\mathcal{L}_{\text{struct}}(\hat{\mathbf{p}}, \hat{L}, \hat{r})\]

(⚠️ The LaTeX of this equation is corrupted in the PDF extraction; the form above is a structural illustration rewritten from the prose, and the weight \(\lambda\) together with the exact form of each term should be taken from the original paper.) Stage 3 (instruction fine-tuning) starts from the Stage-2 checkpoint, enables LoRA adapters (rank 16) in the LLM attention layers while RF-DETR stays frozen, and keeps the same objective, but gradients now flow through the LoRA parameters to improve instruction adherence and the alignment between structured predictions and generated explanations. Optimization is AdamW with mixed precision on two A100 GPUs; stages 2 and 3 run for 1 epoch each. At inference the configuration is DINOv2-L/14 with 448×448 inputs, RF-DETR at seg-2xlarge with 6 decoder layers, up to 300 proposals → NMS 0.6 → score 0.15 → top-100 regions, with seven safety tokens injected in the end.

Key Experimental Results¶

Main Results¶

Evaluation is on the self-built Falcon-X: fine-tuned methods are trained and evaluated on the official train/test splits, while zero-shot methods are evaluated on the test split only. Table 1 covers the grounding tasks of Layers I and II, where RS, PS, and RPS denote referring segmentation, panoptic segmentation, and referring panoptic segmentation, and RFG is the paper's new referring functional grounding — locating all components that could jointly form a functional device, rather than instances of one category.

Method	RS cIoU	RS mIoU	PS cIoU	PS mIoU	RPS cIoU	RPS mIoU	RFG cIoU	RFG mIoU
Sa2Va (zero-shot)	14.96	32.78	35.53	54.14	17.50	37.98	14.26	20.64
Sting-Bee (fine-tuned)	18.72	26.81	31.05	33.49	15.40	19.59	12.72	14.96
Groma (fine-tuned)	14.25	21.99	36.73	13.78	17.94	24.10	20.07	29.55
Falcon	25.86	35.39	14.37	38.02	21.74	35.30	50.45	69.58

(The parenthetical Δ annotations in parts of the original table look misaligned with their columns, so only the values themselves are kept here; the RFG column matches the paper's prose, which reports gains of +30.38 cIoU and +40.03 mIoU.)

Table 2 covers compositional semantic grounding from Layer II and relational safety metrics from Layer III. CPC is component presence check, MCI missing component identification, FC functional completeness, SRL scene risk level, PCS potential component sets, and CLR component link risk; CPC / MCI / PCS report accuracy and F1, the rest report error (lower is better).

Method	CPC Acc / F1	MCI Acc / F1	FC MAE / RMSE	SRL MAE	PCS Acc / F1	CLR MAE
Sa2Va (zero-shot)	58.40 / 46.20	12.60 / 15.80	0.60 / –	0.40	13.56 / 54.90	–
Sting-Bee (fine-tuned)	98.00 / 98.00	93.45 / 96.31	0.031 / 0.36	0.23	14.89 / 56.34	0.24
Groma (fine-tuned)	98.00 / 98.00	97.10 / 98.60	0.019 / 0.13	–	14.90 / 57.06	–
Falcon	98.10 / 97.97	94.75 / 97.33	0.017 / 0.09	0.02	15.10 / 57.69	0.005

Scene understanding and VQA (Table 2 of the paper) are not transcribed here, but their conclusion is blunt: zero-shot general-purpose models transfer poorly to the X-ray domain (for instance Groma zero-shot reaches caption BLEU 18.8 and VQA BLEU 3.32), and even the domain-adapted Sting-Bee only reaches caption BLEU 24.61 zero-shot. Once fine-tuned on Falcon-X, caption and VQA metrics saturate across the board (VQA BLEU around 99.9, METEOR around 91), with Falcon ahead only marginally (caption BLEU 40.92, ROUGE-L 47.74, CIDEr 0.064; VQA BLEU 99.93, METEOR 91.98). In other words, conventional scene understanding in this domain is largely a matter of domain adaptation and is not the difficulty the paper targets.

Ablation Study¶

Table 3 switches the SSA's three prediction heads on and off one at a time (R = risk, P = component presence, L = link), all evaluated on referring functional grounding (RFG).

Variant (R / P / L)	RFG cIoU	vs. no heads	RFG mIoU	vs. no heads
✗ ✗ ✗ (a region-grounded VLM with no structured signal)	35.82	–	43.43	–
✓ ✗ ✗	39.33	+3.51	47.98	+4.55
✓ ✓ ✗	45.52	+6.19	58.60	+10.62
✓ ✓ ✓	50.45	+4.93	69.58	+10.98

Table 4 separates perception error from reasoning error with Oracle settings: Oracle-Reasoning substitutes ground-truth relational assignments over the same predicted regions (perception fixed, isolating the decision layer), while Oracle-Perception substitutes ground-truth masks for predicted regions but keeps the learned reasoning (a perception upper bound).

Mode	Perception	Reasoning	RFG cIoU	Δ	RFG mIoU	Δ
Falcon	Pred	Pred	50.45	–	69.58	–
Oracle-Reasoning	Pred	GT	59.61	+9.16	79.53	+9.95
Oracle-Perception	GT	Pred	79.49	+29.04	83.42	+14.84

Key Findings¶

The link head is the primary driver of functional grounding. With all structured heads off, RFG is only 35.82 / 43.43; enabling the risk head alone adds just 3.51 / 4.55, adding presence takes it to 45.52 / 58.60, and switching on the link head adds another 4.93 / 10.98. Risk and presence are single-point scalars; links are the only quantity that carries the relation between components — this ablation directly supports the paper's claim that the gains come from relational supervision rather than better object classification.
Perception remains a larger bottleneck than reasoning. The reasoning headroom is 9.16 cIoU / 9.95 mIoU (replacing relational assignments with ground truth); on top of that, replacing predicted regions with ground-truth masks adds a further 19.88 cIoU / 3.89 mIoU. Even with perfect relational reasoning, segmentation quality caps the overall result, so the SSA cannot substitute for a better segmenter.
Presence recognition saturates; the relational metrics are the watershed. After fine-tuning, CPC sits at about 98% for everyone, so "is there a battery" is not hard; yet zero-shot models reach only 12%–14% MCI accuracy, and after fine-tuning FC MAE still ranges from 0.031 (Sting-Bee) to 0.017 (Falcon) and CLR MAE from 0.24 to 0.005. The gains concentrate on compatibility-dependent metrics, consistent with the explanation that structured modeling is doing the work.
The paper does not win everywhere, and the authors say so. On the appearance-driven PS / RPS tasks Falcon does not consistently beat fine-tuned baselines (PS cIoU 14.37 is below Groma fine-tuned at 36.73), and on missing component identification (MCI) Groma fine-tuned (97.10 / 98.60) also exceeds Falcon (94.75 / 97.33). This contrast actually strengthens the paper's thesis: once the target shifts from "what does it look like" to "how do the parts combine," the benefit of structured modeling shows up.
Logical consistency (PCS) is hard for everyone. Potential-component-set accuracy is around 14%–15% for all methods with F1 near 57, so moving from predicted presence and links to "which sets are feasible" is only marginally improved even by this paper's method.

Highlights & Insights¶

Recasting "threat" from a category label into relational quantities, and having the model predict those quantities, is the most transferable idea here. Any judgment of the form "harmless alone, harmful in combination" — hazardous-material compatibility, drug interactions, cascading equipment failures — fits this template of component presence plus a type-level compatibility matrix plus scene risk; the paper itself notes that swapping the component vocabulary and compatibility rules extends it to other modular threats.
The semantic bottleneck pushes relational reasoning back onto the perception side. In Stage 2 the detector and the LLM are frozen and only the SSA is trained, so the only new information the LLM receives is seven scalars; this replaces a slogan about "teaching the LLM to reason" with an architectural constraint. The benefit is diagnosability: ablation can pinpoint which head matters instead of reporting a single end-to-end number.
Counterfactuals use mask-guided inpainting rather than zero-filling, preventing the model from taking the shortcut of recognizing artifacts; and the risk labels deliberately preserve "missing a part yet still high risk," cutting the trivial derivation of risk from the presence vector. Both are data-construction insights directly reusable for any task that needs missing-part or missing-modality supervision.
The Oracle ablation separates perception headroom from reasoning headroom, a diagnostic well worth copying when evaluating structured multimodal systems: first show how much ground-truth relational assignments buy, then how much ground-truth masks buy on top, and the reader sees immediately which side the bottleneck is on.
The biggest "aha" is that the paper does not win with a bigger model: attaching a very light adapter in front of a frozen 7B LLM lifts functional grounding from 20 cIoU to 50 cIoU, suggesting that what structured signal gets injected matters more than how large the backbone is for this class of task.

Limitations & Future Work¶

The component vocabulary is narrow and compatibility is predefined. The type set is only {battery, detonator, main charge}, and the compatibility matrix \(L\) is predefined at the type level as an annotation scaffold; it cannot express instance-level or context-dependent compatibility (say, how well a particular battery matches a particular detonator). The paper acknowledges that swapping the vocabulary extends it but runs no experiment to verify that.
The data is mostly synthetic. There are about 7,000 real base images, and most of the roughly 50,000 total are counterfactual variants; the distribution shift between synthetic samples and real missing-component scenes is never evaluated separately (e.g., whether the model exploits inpainting traces), which is the most important gap for the benchmark's credibility.
The subjectivity of the risk labels makes absolute cross-method comparisons delicate. The score \(r\) is provided by the same group of experts and explicitly allows "a part is missing yet risk is high," so the absolute MAE depends on the annotation convention, and no inter-annotator agreement is reported.
Single view, with dual-energy information unused. The data comes from dual-energy X-ray scans, but the method resizes inputs into a uniform 448×448 image pathway; material properties are never explicitly exploited by the architecture, even though they would be valuable for distinguishing a battery from similarly shaped metal objects.
Only one threat template is validated. Cross-dataset generalization lives in the appendix and is not reported in the main text; cross-domain transfer (other modular threats, industrial or medical X-ray) is entirely unverified.
Concrete improvement directions: upgrade the type-level compatibility matrix to learned instance-level compatibility (for example, letting instance slots attend to each other in the link head instead of concatenating type embeddings); treat the perception bottleneck as a first-class problem by jointly fine-tuning the detector or adopting a stronger open-vocabulary segmenter; and add a synthetic-trace detection probe to the benchmark as a negative control, proving the model is not identifying missing components by artifacts.

vs STCray / Sting-Bee: they bring image-text paired data and a domain-adapted VLM to X-ray, but their tasks remain captioning, target localization, and VQA, with threat attributed to single objects. This paper keeps those tasks as the Layer I baseline but pushes evaluation to "does the component assembly hold," adding functional completeness, missing-component identification, referring functional grounding, and relational consistency analysis, and widening supervision from text to a structured safety state.
vs SIXray / OPIXray / HiXray / PIDray and similar detection benchmarks: these are closed-set, object-level, and aimed at intact prohibited items, measuring "was it detected." Falcon-X is the smallest (about 7,000 base images) but the only benchmark that jointly provides dense masks, multimodal tasks, and structured functional-threat supervision — the paper's comparison table gives "explicitly models dismantled components" its own column.
vs region-level MLLMs such as Groma / LISA / GLaMM / KOSMOS-2: they use region or box tokens so the LLM can refer and segment, but nothing explicitly supervises relations between regions. This paper inserts a fixed relational state between the region set and the LLM, changing "regions → language" into "regions → relational state → language," at the cost of a few intermediate quantities that must be calibrated accurately.
vs neuro-symbolic compositional reasoning (explicit program or automaton approaches such as NS-CL and NAVER): those express compositional logic as a parsable symbolic program, whereas this paper uses a soft-symbolic scheme of a type-level compatibility template plus differentiable prediction heads. No program parser is needed and it trains end to end, but compatibility therefore cannot vary with context, and no traceable reasoning chain is produced.
Takeaway: the paradigm of "predict a structured relational state, then inject it as tokens into a frozen LLM" transfers directly to video anomaly detection (temporal combinations of people, objects, and actions) and industrial inspection (defects in multi-part assemblies). The key is to write "the condition under which the combination holds" as a fixed-dimensional, supervisable, ablatable intermediate quantity rather than hoping an LLM infers it from implicit attention.

Rating¶

Novelty: ⭐⭐⭐⭐ [Formalizing safety threats as relational quantities (presence + type-level compatibility + risk) and injecting them explicitly into an LLM is a new problem definition and benchmark; the method itself is a sound composition of existing components.]
Experimental Thoroughness: ⭐⭐⭐⭐ [The main tables cover 7 baselines across 9 task types, plus a head ablation and an Oracle perception/reasoning decoupling ablation; however there are only ~7,000 base images, synthetic samples dominate, and only one threat template is validated.]
Writing Quality: ⭐⭐⭐⭐ [The problem formalization is clear and the task layers are well organized, and the authors honestly report not leading on PS / RPS / MCI; some formulas are corrupted after extraction and the Δ annotations in the tables have column-alignment issues.]
Value: ⭐⭐⭐⭐ [It offers a reusable evaluation paradigm for compositional safety reasoning and a clean structured-injection recipe, with direct relevance to deploying VLMs in safety-critical settings.]