RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios

Conference: ECCV 2026
Paper: ECCV 2026 official page
Code: https://github.com/crazyxiaoxi/RGBT-GroundBench
Area: Object Detection (Visual Grounding)
Keywords: Visual Grounding, RGB-Thermal Multimodal, Benchmark Dataset, Cross-Spectral Fusion, Low-Light Robustness

TL;DR

The paper builds RGBT-GroundBench, the first large-scale RGB-Thermal visual grounding benchmark for complex real-world scenarios (21,535 RGB-TIR pairs, 38,760 referred instances, three-level annotations), defines a unified protocol supporting RGB-only, TIR-only, and RGB+TIR inputs, benchmarks 11 VG models under it, and proposes the reference baseline RGBT-VGNet, which attains the best [email protected] on all three sub-datasets.

Background & Motivation

Visual grounding (VG) asks a model to localize the object described by a natural-language expression and is a foundational capability for human-robot interaction, embodied agents, autonomous driving, and assistive technology. The field advanced rapidly over the past decade on the back of large-scale benchmarks such as RefCOCO/+/g, ReferIt, and Flickr30K, together with vision-language pretraining paradigms like CLIP, BEiT-3, and LLaVA; methods such as TransVG, QRNet, D-MDETR, FSVG, CLIP-VG, and HiVG have made "find the object the sentence names" fairly accurate. Yet these benchmarks share three blind spots. Scene complexity is limited: images come from clean environments and targets are typically centered and salient, which does not reflect the clutter and dynamics of real scenes. Object diversity is insufficient: small, distant, and heavily occluded targets are barely represented, so robustness in safety-critical settings like autonomous driving and embodied perception cannot be assessed. And the modality design is RGB-centric — whereas real perception frequently happens at night, under low illumination, or in rain and fog, exactly when RGB cues are unreliable. The numbers make the point: RefCOCO has a 0.0% small-object share, and RefCOCOg contains only 10.7% weak-light samples with an average query length of 8.47 words, so these benchmarks can hardly test whether a model still works when the sensor degrades.

The most direct complement is thermal infrared (TIR), which is illumination-invariant and keeps providing stable heat signatures at night, in low light, and in bad weather. RGB-Thermal fusion is already mature in object detection and semantic segmentation, but once a language condition is added the area is essentially empty; the alternative route of RGB-D grounding does not fit either, since its formulation targets 3D environments and does not transfer directly to 2D localization, while depth sensing itself degrades under low light and adverse weather and therefore cannot serve as a robust complement. What actually blocks the direction is the evaluation protocol: RGBT perception is either measured with mAP on detection datasets or mIoU on segmentation datasets, and nobody can answer whether wiring thermal input into a grounding model helps, and under which conditions, because there are neither paired RGB-TIR referring annotations nor a way to compare RGB-only, TIR-only, and RGB+TIR fairly under the same preprocessing, training routine, and metric.

This paper closes both gaps at once. Starting from three already aligned multimodal detection datasets (FLIR, M3FD, MFAD), it filters the data, generates referring expressions with an MLLM, and refines them through hierarchical random-sampling human verification, producing three-level fine-grained annotations, from which difficulty-oriented TestA/TestB/TestC subsets are carved out. A unified protocol then runs 11 representative VG models across all three modality settings, and the results yield four observations: scene complexity dominates accuracy, LoRA-based methods resist complex scenes better, low illumination is the largest failure mode, and MLLMs cannot yet fuse cross-spectral cues reliably. Core idea: use spatially paired RGB-TIR detection data plus MLLM-generated, human-calibrated referring expressions to move visual grounding from an RGB-only task to a benchmark that supports cross-spectral comparison and diagnosis by operating condition, and instantiate a reliability-aware fusion baseline that makes "which modality to trust" explicit.

Method

The "method" here has two halves: the first turns RGBT visual grounding into a task that can actually be evaluated (where the data comes from, how annotations are produced, how the protocol is defined), and the second supplies a reproducible reference baseline, RGBT-VGNet, under that protocol. The two are chained — the benchmark first exposes phenomena such as "low illumination hurts most" and "LoRA-based methods hold up best in complex scenes," and only then does the baseline target them with dedicated modules.

Overall Architecture

The overall flow is "data → annotation → splits and protocol → evaluation → baseline." On the data side, raw image pairs are collected from the paired multimodal detection datasets FLIR, M3FD, and MFAD, cleaned by three filtering rules, captioned by Qwen3-VL for referring expressions and attribute labels, verified and corrected through hierarchical random sampling, and finally split into train/val/test following the source splits, with difficulty-oriented TestA/TestB/TestC subsets carved out of test. On the protocol side, eight model families — TransVG, MMCA, D-MDETR, CLIP-VG, FSVG, AttBalance, HiVG, and OneRef — are ported into a single codebase that supports RGB-only, TIR-only, and RGB+TIR inputs. On the model side, RGBT-VGNet sends the RGB and TIR streams through CLIP vision encoders with shared weights, applies Asymmetric Modality Adaptation, Language-Aware Visual Synergy, and Tri-Prior Fusion to obtain fused visual tokens, and feeds them together with the text tokens and a learnable regression token [Reg] into a VL-Transformer, whose lightweight MLP head regresses the bounding box.

In scale, RGBT-GroundBench contains over 40K images (21,535 RGB-TIR pairs) and 38,760 object instances with referring expressions, annotated at three levels: scene (13 types — urban, intersection, residential, suburban, highway, campus, bridge, parking lot, rural, tunnel, market, waterfront, industrial), environment (4 illumination levels, very weak/weak/normal/strong, and 4 weather conditions, cloudy/sunny/rainy/foggy), and object (size, small/normal, and occlusion, no-or-partial/heavy). The contrast with existing benchmarks is stark:

Benchmark	Modality	Typical res.	#Instances	Weak-light	Small object	Query length
Flickr30K	RGB	500×375	276,000	0.9%	0.0%	1.59
ReferIt	RGB	480×360	96,654	2.77%	0.0%	3.45
RefCOCO	RGB	640×480	50,000	6.1%	0.0%	3.49
RefCOCO+	RGB	640×480	49,856	2.8%	12.0%	3.58
RefCOCOg	RGB	640×480	49,822	10.7%	45.4%	8.47
RGBT-GroundBench	RGB+TIR	640×512	38,760	43.2%	56.8%	14.24

The paper also stresses the off-central object distribution: in classical VG benchmarks targets cluster near the image center, whereas objects in RGBT-GroundBench sit noticeably closer to the edges, mirroring how targets appear at the fringe of the frame in real driving and inspection scenes. On the resource side, the authors commit to full public release with no hidden test set or private server: annotations and splits ship with the paper, the dataset is hosted on HuggingFace (JiawenXi/RGBT-Ground-Dataset), code and checkpoints are on GitHub, and the source datasets keep their original licenses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["FLIR / M3FD / MFAD<br/>RGB-TIR detection data"] --> B["Data construction pipeline"]
    B --> C["Three-level fine-grained annotation<br/>and difficulty-oriented protocol"]
    C --> D["Asymmetric Modality Adaptation (AMA)"]
    D --> E["Language-Aware Visual Synergy (LAVS)"]
    E --> F["Illumination-Local-Global Tri-Prior Fusion (TPF)"]
    F --> G["[Reg] regression box"]

Key Designs

1. Data construction pipeline: from detection data to verifiable referring expressions

The hard part of referring data is not writing a sentence but making that sentence uniquely identify one box. The pipeline has four stages: collection, filtering, captioning, and human quality control. Collection reuses FLIR, M3FD, and MFAD, which already provide spatially aligned (or weakly aligned) RGB-TIR pairs and object-level boxes. Filtering imposes three rules — drop extremely small targets that cover only a few pixels and cannot support reliable annotation, discard image pairs with severe spatial misalignment between RGB and TIR, and remove long-tailed or underrepresented categories (the paper names dog in FLIR and lamp in M3FD) to avoid class imbalance; among the survivors, only the largest instance of each category per image pair is kept so that annotation clarity and cross-modality consistency are preserved. Captioning runs Qwen3-VL with carefully designed task prompts to emit two label types in one pass: object referring expressions, and scene-, environment-, and object-level attributes. The prompts cover four concerns — appearance details and key features, contextual relationships, how to distinguish similar objects of the same class, and concise, clear language.

Generation is followed by quality control: hierarchical random sampling over scene, illumination, weather, object size, and occlusion, after which annotators check for hallucination, ambiguity, incorrect attributes, box-expression mismatch, and severe grammar problems, fixing them by rewrite, relabeling, or invalidation. The reported quality indicators are concrete: 15% of instances are reviewed, 5% of expressions are edited, 0.5% are removed as invalid, and 95% of the reviewed expressions correctly refer to the intended target; an annotation is accepted only when the expression uniquely identifies one target and stays consistent with the final box and the cross-modal attributes. Beyond human spot checks, the paper adds a statistical consistency check: the lighting-weather cross statistics below follow plausible physical trends — weak light falls almost entirely on rainy and cloudy scenes while strong light is almost exclusively sunny — and since such "too clever" statistical relations are exactly where MLLM-generated labels tend to break, matching them indicates the annotations carry no systematic drift.

Illumination / Weather	Foggy	Rainy	Sunny	Cloudy
Very weak	71 (0.33%)	29 (0.13%)	0 (0.00%)	101 (0.47%)
Weak	2,754 (12.79%)	1,298 (6.03%)	14 (0.07%)	4,704 (21.84%)
Normal	521 (2.42%)	150 (0.70%)	2,191 (10.19%)	6,157 (28.59%)
Strong	4 (0.02%)	0 (0.00%)	3,152 (14.64%)	389 (1.81%)

2. Three-level fine-grained annotation and a difficulty-oriented protocol: making "hard" a sliceable dimension

Multimodal imagery alone does not make a diagnosable benchmark; the key is turning difficulty into switches that can be counted independently. The annotation scheme pins every instance to the scene, environment, and object axes at once, so a drop in accuracy can be traced to a specific illumination level, weather condition, or scale instead of collapsing into one global mean. By source, the data forms RefFLIR (9,712 instances), RefM3FD (7,548), and RefMFAD (21,500), totalling 38,760 with 26,604 for training, 2,032 for validation, and 10,124 for testing. On top of the test set, three diagnostic subsets are carved out: TestA covers normal-size targets under normal and strong illumination, TestB targets nighttime scenes with weak and very-weak illumination, and TestC focuses on small-size targets. These are condition-filtered subsets rather than a partition of test — for instance RefFLIR has 2,104 test instances while TestA/B/C contain 837/640/968, and the three sum to more than the test split, meaning they overlap (a small target in weak light can belong to both TestB and TestC); ⚠️ refer to the original paper and supplementary material for the exact overlap convention. The metric follows the VG community's [email protected]: a prediction counts as correct when the IoU between the predicted and ground-truth box exceeds 0.5. Results are reported per split (val/test/TestA/TestB/TestC), with [email protected] deferred to the supplementary material.

The protocol itself is part of the contribution. Eight model families — TransVG, MMCA, D-MDETR, CLIP-VG, FSVG, AttBalance, HiVG, and OneRef — are adapted into one codebase with shared preprocessing, training routine, and metrics, and each can consume RGB-only, TIR-only, or RGB+TIR input (multimodal variants carry the MV- prefix). This looks like engineering, but it is the precondition for the paper's conclusions: only when preprocessing and epoch budgets match do cross-model claims such as "LoRA-based methods are more robust" or "multimodal input gains X points" stop being artifacts of differing training recipes.

3. Asymmetric Modality Adaptation (AMA): giving the TIR branch more adaptation capacity

Models like CLIP have RGB-biased visual representations, so feeding a thermal image into their frozen vision encoder crosses a wide modality gap. Existing LoRA fine-tuning recipes usually apply the same configuration to both modalities, implicitly assuming they sit equally far from the pretraining distribution and ignoring that bias. AMA instead performs low-rank decomposition on the attention projection weights of each vision encoder layer and assigns the two streams different adaptation capacity:

\[W_{v}^{l}=W_{E_v}^{l}+\alpha_{v}^{l}A_{v}^{l}B_{v}^{l},\qquad W_{t}^{l}=W_{E_t}^{l}+\alpha_{t}^{l}A_{t}^{l}B_{t}^{l}\]

(Both streams share the same frozen CLIP initialization; \(A\) and \(B\) are learnable low-rank matrices and \(\alpha\) is a scaling factor. The cached PDF text dropped the addition signs, so the form here follows standard LoRA — ⚠️ refer to the original paper.) The decisive setting is the asymmetric rank \(r_v \le r_t\): the RGB branch keeps a smaller rank to preserve the pretrained vision-language alignment, while the TIR branch gets a larger one, on the grounds that TIR deviates further from the pretraining domain and needs more degrees of freedom to align. The paper additionally adopts HiVG's hierarchical design so that low-level structural and high-level semantic representations are aligned progressively. What makes this design notable is that it is derived from an observation rather than invented first: the benchmark evaluation found LoRA-based models (HiVG) markedly more stable in complex scenes, and only then did the authors make that adaptation scheme the backbone of the baseline.

4. Language-Aware Visual Synergy (LAVS): using text as a query to select features from both streams

What separates RGBT grounding from RGBT detection is that the target is specified by language, so which regions of the two visual streams should be fused, and with what weight, ought to be decided by textual semantics rather than by letting the two streams align blindly. LAVS treats the text embedding as a query that retrieves from RGB and TIR separately: the text features produce \(Q_s\), each stream produces \(K\) and \(V\), and after computing the language-guided cross-modal attention map, the features are updated residually:

\[A_{\text{attn}_v}^{l}=\sigma\!\left(\frac{Q_s^{l}(K_v^{l})^{\top}}{\sqrt{d}}\right),\qquad F_v^{l}=f_v^{l}+\left(A_{\text{attn}_v}^{l}\right)^{\top}\left(A_{\text{attn}_v}^{l}V_v^{l}\right)\]

The TIR stream yields \(A_{\text{attn}_t}^{l}\) and \(F_t^{l}\) the same way. Right-multiplying by \(V\) and then left-multiplying by the transposed attention map is not just formalism — it keeps the updated feature dimension identical to the original patch features, so the result can be handed straight to the fusion module. After both streams have been "washed" by language once, a cross-attention between modalities exchanges information, and linear projections \(P_v\) and \(P_t\) map each into a unified grounding space. The effect is that the text decides up front which regions are worth fusing, so the fusion module receives semantically aligned inputs instead of raw concatenated features.

5. Illumination-Local-Global Tri-Prior Fusion (TPF): deciding whom to trust by reliability

RGB degrades in low light while TIR remains usable, which is the entire reason for a second modality; but if "when to trust TIR" is delegated to a learned gate, it is easily mis-trained on a small dataset. TPF instead writes reliability as an explicit prior. Each RGB patch is converted to grayscale, pooled onto the patch grid, and min-max normalized to \([0,1]\), giving the illumination intensity \(i_p\), from which the illumination quality is defined by the distance to the optimal illumination point 0.5:

\[q_p = 1-|2i_p-1|\]

The closer the brightness to mid-gray, the more trustworthy the RGB patch. Next come the local-global semantic priors: the layer-normalized patch tokens of both streams plus their element-wise difference go into a small MLP followed by a sigmoid, producing a per-patch prior \(g_p\); the same treatment on the globally pooled features, averaged across channels, gives a whole-image prior \(b\) broadcast to all patches. The three priors are multiplied with equal weight around the neutral point 0.5 and passed through a sigmoid to yield the RGB reliability weight \(w_p\) and the fused token:

\[w_p=\sigma\big((i_p-0.5)\,q_p\,(g_p-0.5)\,(b-0.5)\big),\qquad z_p^{l}=w_p\,t_{v,p}^{l}+(1-w_p)\,t_{t,p}^{l}+g^{\mathrm{ch}}\odot\left(t_{v,p}^{l}-t_{t,p}^{l}\right)\]

where \(g^{\mathrm{ch}}\) is the channel-wise gate produced by the semantic MLP. The expression reads clearly: \(w_p\) near 1 means the patch's RGB brightness and semantics are both credible and RGB dominates, while near 0 it switches to TIR under weak light or suspicious RGB semantics; the last term adds back a semantic-gated difference so the complementary information is not lost. The authors are explicit about the module's role — it is a lightweight heuristic fusion, meant to instantiate the cross-spectral complementarity observation under a controlled protocol rather than to model illumination on its own; equal weighting around the 0.5 neutral point also avoids introducing extra hyper-parameters.

Loss & Training

All models train at 224×224 with the same augmentation policy, batch size 8, AdamW, learning rate \(1\times10^{-3}\), 120 epochs, on RTX 4090 GPUs. All CLIP encoders (vision and text) stay frozen to preserve the pretrained vision-language alignment, so only the LoRA branches, the LAVS/TPF modules, the projections, and the regression head are trained; adapted baselines follow the default hyper-parameters of their original implementations where applicable. The regression target comes from the [Reg] token through a lightweight MLP, \(\hat{B}=(\hat{x},\hat{y},\hat{w},\hat{h})\). The main text does not state the concrete box regression loss (a combination of L1/GIoU is the usual choice in this unified framework), ⚠️ refer to the original paper and the open-source code.

Key Experimental Results

Main Results

Main results under the three input settings ([email protected], %, test split) — zero-shot transfer, in-domain training with a single visual modality, and in-domain training with multimodal input:

Method	Input modality	RefFLIR test	RefM3FD test	RefMFAD test
HiVG-L (zero-shot)	RGB	44.46	53.10	40.42
HiVG-L (zero-shot)	TIR	23.80	28.89	23.11
HiVG-B (in-domain)	RGB	66.65	68.15	64.02
HiVG-B (in-domain)	TIR	64.07	61.77	62.63
MV-TransVG	RGB+TIR	46.06	48.04	53.84
MV-D-MDETR	RGB+TIR	47.42	46.50	57.31
MV-HiVG-B	RGB+TIR	69.20	72.35	67.04
MV-HiVG-L	RGB+TIR	72.50	68.67	61.31
MV-OneRef-L	RGB+TIR	60.89	69.70	62.73
RGBT-VGNet (Ours)	RGB+TIR	72.43	74.89	67.58

On the three difficulty subsets RGBT-VGNet also sits in the top band: TestA (normal/strong light, normal size) reaches 89.64/94.16/91.29, TestB (weak/very-weak light, nighttime) 69.22/80.75/64.80, and TestC (small targets) 52.53/55.52/51.18, in the order RefFLIR/RefM3FD/RefMFAD; the paper summarizes this as "[email protected] exceeding 91%, 64%, and 49% on TestA, TestB, and TestC."

Ablation Study

Adding the three components one by one ([email protected], %):

AMA	LAVS	TPF	RefFLIR val / test / testB / testC	RefM3FD val / test / testB / testC	RefMFAD val / test / testB / testC
			55.42 / 46.19 / 41.56 / 22.81	54.16 / 57.57 / 65.63 / 34.46	53.50 / 54.63 / 51.91 / 36.62
✓			74.01 / 71.17 / 66.25 / 49.69	70.83 / 72.53 / 80.16 / 50.43	68.07 / 65.27 / 62.19 / 47.96
✓	✓		73.68 / 72.65 / 67.19 / 52.23	73.21 / 74.34 / 81.93 / 53.63	67.83 / 66.62 / 64.07 / 49.76
✓	✓	✓	75.33 / 72.43 / 69.22 / 52.53	73.21 / 74.89 / 80.75 / 55.52	68.31 / 67.58 / 64.80 / 51.18

Replacing TPF with other RGBT fusion strategies under the same protocol:

Fusion method	Venue	RefFLIR val	RefM3FD val	RefMFAD val	RefFLIR testC	RefM3FD testC	RefMFAD testC
Naive Add	-	73.68	70.24	67.91	46.86	51.46	49.41
CMX	TITS'24	64.31	60.71	53.74	38.87	38.15	32.33
LIF	ICCV'25	74.34	69.64	67.59	49.70	51.30	49.96
TPF (Ours)	-	75.33	73.21	68.31	52.53	55.52	51.18

Key Findings

• Modality adaptation is a prerequisite, not an option. Zero-shot transfer collapses for every model, with the paper reporting an average degradation of about 30% (RGB) and 50% (TIR) relative to in-domain training. In the ablation, adding AMA alone lifts RefFLIR test from 46.19 to 71.17 (+24.98), the largest single-step gain of any component: CLIP representations are RGB-biased, and without first aligning the TIR branch the fusion modules downstream receive inputs that are already misaligned.
• LAVS and TPF pay off on the hard cells and give back a little on the easy ones. Adding LAVS raises RefFLIR testB from 66.25 to 67.19 and testC from 49.69 to 52.23, while val slips slightly from 74.01 to 73.68; adding TPF raises RefM3FD testC from 53.63 to 55.52 and RefFLIR testB to 69.22, yet RefFLIR testA falls back from 91.31 to 89.64. That trade — hard samples bought with easy ones — is consistent with TPF's reliability weighting, whose purpose is to rescue weak light and small targets.
• Multimodal input does help, but the paper's "about 10%" claim needs care. From the HiVG/OneRef rows that can be checked directly, RGB+TIR improves over unimodal RGB by roughly 1-4 points (e.g. RefM3FD test: HiVG-B 68.15→72.35; RefMFAD test: HiVG-B 64.02→67.04). "About 10%" is more plausibly an average over all models and all subsets, ⚠️ so the exact scope should be checked against the original paper and supplementary material. What is confirmed is that the gains concentrate on testB/testC — weak light and small objects are the conditions that benefit most.
• The fusion comparison shows explicit priors beating a fancier fusion network. The representative RGBT fusion baseline CMX trails across all three sub-datasets (RefFLIR val only 64.31, 11 points below TPF), while the structurally simpler Naive Add and LIF beat TPF in a few individual cells (e.g. LIF's RefFLIR testA 90.00) but are clearly less consistent across datasets.
• Scene complexity correlates strongly with grounding accuracy, and LoRA-based methods resist it better. Models are steady on TestA-style cells with good light and normal scale, and drop noticeably in cluttered scenes with distractors, occlusion, or small off-center targets; the LoRA-adapted HiVG family fluctuates less across scenes than fully fine-tuned or non-LoRA methods. The authors infer that parameter-efficient adaptation does more for robustness under domain shift than architecture design does.
• Low illumination is the long-neglected failure mode. TestB is where every model loses the most; thermal cues mitigate but do not fix it, and small targets combined with adverse weather are the hardest case of all — precisely the reason this condition was made its own subset.
• MLLMs are not ready for cross-spectral grounding. The paper additionally evaluates GLM-4.6V, Kimi-K2.5, and Qwen3.5-Plus at both 224×224 and original resolution, concluding that naively adding thermal input does not yield consistent localization gains and that performance is markedly sensitive to input resolution; the detailed numbers live in the supplementary material.

Highlights & Insights

• Publishing the quality indicators is far more credible than claiming "high quality." The set "15% reviewed, 5% edited, 0.5% invalidated, 95% of reviewed expressions correct," combined with the physical-consistency check on the lighting-weather cross statistics, gives two-layer verification: humans guarantee individual expressions are unambiguous, statistics guarantee the overall distribution does not drift. This "MLLM generation + hierarchical spot checks + statistical validation" recipe transfers to any setting where an MLLM is used to manufacture data.
• Difficulty is decomposed into sliceable dimensions rather than an adjective. The three-level annotation lets a conclusion like "low illumination hurts" land on a specific cell (TestB) instead of drowning in a global mean, and the TestA/B/C logic (good light and normal scale / nighttime weak light / small targets) is a template that can be reused for robustness evaluation in almost any other task.
• The unified protocol is the most underrated contribution of a benchmark paper. Porting eight model families into one codebase with shared training and metrics while freely switching among three input modalities is what makes cross-model claims like "multimodal helps" and "LoRA is more robust" stand up; otherwise the reader cannot tell method differences from training-recipe differences.
• TPF's reliability weight is a light, reusable trick. It writes "should this patch's RGB be trusted" as an equally weighted product of illumination quality, a semantic prior, and a global prior, then squashes it with a sigmoid around a fixed neutral point of 0.5 and no extra hyper-parameters. This "normalize to [0,1], treat 0.5 as neutral, multiply the priors" pattern ports directly to any two-branch or two-modality fusion.
• Benchmark observations feed straight into baseline design, closing the loop. AMA comes from "LoRA-based methods are more robust," TPF from "low illumination hurts most," and LAVS exists to give TPF semantically aligned inputs; no module is arbitrary, each traces back to an experimental conclusion — a writing pattern worth imitating.
• The trade-offs are reported honestly. The ablation table does not hide that TPF slightly lowers TestA, and that detail is in fact consistent with the narrative that reliability-aware fusion favors hard operating conditions.

Limitations & Future Work

• The data comes from only three detection datasets (FLIR, M3FD, MFAD), with categories dominated by pedestrians and vehicles, so scene and semantic diversity is bounded; expressions are MLLM-generated and only 15% were manually reviewed, leaving the remaining 85% backed only indirectly by statistical validation.
• RGB and TIR are assumed spatially aligned or weakly aligned. Real deployment involves misregistration and missing modalities (RGB-only or TIR-only sessions), which are exactly the common cases in practice and are not covered.
• TPF's priors are fairly hand-crafted: all three are normalized to \([0,1]\), treated as neutral at 0.5, and multiplied with equal weight. That neutral-point assumption may not hold across sensors or exposure policies, and no sensitivity analysis over the weighting form is provided.
• Many fine-grained conclusions depend on the supplementary material: scene, weather, and occlusion slices beyond TestA/B/C, [email protected], and the full MLLM numbers are absent from the main text, which gives only four distilled findings, leaving readers unable to verify them independently.
• The main text does not state the box regression loss or further training details (e.g. the concrete LoRA ranks and \(\alpha\) values), so reproduction depends on the open-source code.
• Promising directions: replace the hand-crafted reliability product with a learnable or calibrated gate; add missing-modality and misregistration settings to the protocol; bring MLLM localization heads (e.g. attention-to-mask approaches) into the same protocol for comparison; and extend to more categories and finer weather/occlusion slices.

Related Work & Insights

• vs RefCOCO/+/g, ReferIt, Flickr30K: these are RGB-only benchmarks built from clean environments with centered, salient targets, where the small-object share is 0 or low (RefCOCO 0.0%, RefCOCO+ 12.0%) and weak-light samples top out at RefCOCOg's 10.7%. This paper raises modality (RGB+TIR), condition coverage (43.2% weak light, 56.8% small objects), and language complexity (14.24 words on average, versus RefCOCOg's 8.47) at once, at the cost of an instance count roughly three-quarters of RefCOCO's.
• vs RGB-D grounding (Refer-it-in-RGBD, Cross3DVG, Unit3D): that line extends grounding to 3D, a different task formulation that does not transfer to 2D box regression; more importantly, depth sensing degrades under low light and adverse weather too, so it cannot be the illumination-invariant complement. That is precisely why this paper picks thermal.
• vs RGBT detection and segmentation (C2Former, D3T, UniRGB-IR, ABMDRNet, etc.): these works have explored fusion architectures extensively, but their outputs are language-free boxes or masks and the model never has to understand which object is meant. This paper brings a language condition onto the same dual-modality input, and the failure modes shift accordingly — from misses to grounding the wrong object.
• vs the evaluated VG models (HiVG, OneRef, CLIP-VG, TransVG, ...): they serve both as comparisons and as component sources. HiVG's hierarchical LoRA idea is borrowed by AMA but recast with asymmetric RGB/TIR ranks; after multimodal adaptation (MV- variants) OneRef and CLIP-VG become RGBT-VGNet's direct competitors, yet still trail the proposed baseline overall.
• Takeaway: the research path of "build a benchmark to expose failure modes, design modules that target those modes, then validate them back under the same protocol" applies equally to other cross-modal tasks (RGB-event, RGB-depth, multispectral), especially where one modality's benefit has so far remained anecdotal.

Rating

• Novelty: ⭐⭐⭐⭐ The first RGBT visual grounding benchmark, with three-level annotations and a unified three-modality protocol that the area genuinely lacked; the method side (AMA/LAVS/TPF) is combinatorial rather than a single-module breakthrough.
• Experimental Thoroughness: ⭐⭐⭐⭐ 11 models × 3 modalities × 3 sub-datasets × 5 splits, with component ablations and a fusion comparison, plus zero-shot/in-domain/multimodal tiers; the deduction is that the MLLM evaluation, weather and scene slices, and [email protected] all sit in the supplementary material, so the main text cannot verify itself.
• Writing Quality: ⭐⭐⭐ Structure and figures are clear and the link between observations and designs is well told; however key equations are corrupted in the cached extraction, and many conclusions rest on supplementary material, leaving gaps the reader must fill in.
• Value: ⭐⭐⭐⭐⭐ Data, annotations, splits, code, checkpoints, and evaluation scripts are all public with no hidden test set, and the robustness diagnostic slices (TestA/B/C) can be reused directly by follow-up work, giving the cross-spectral grounding direction an immediate push.

Related Papers

• [CVPR 2026] UAV-CB: A Complex-Background RGB-T Dataset and Local Frequency Bridge Network for UAV Detection
• [AAAI 2026] Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
• [ECCV 2026] GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT Tracking
• [CVPR 2026] Beyond Duality: A Hybrid Framework of Leveraging Shared and Private Features for RGB-Event Object Detection
• [ECCV 2026] REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling