ECCV 2026 Multimodal VLM generalised visual grounding referring expression comprehension agricultural vision hierarchical multi-label learning multimodal dataset

Multi-label Instance-level Generalised Visual Grounding in Agriculture¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/MHaghighat98/WeedVG-gRefCW
Area: Multimodal VLM
Keywords: generalised visual grounding, referring expression comprehension, agricultural vision, hierarchical multi-label learning, multimodal dataset

TL;DR¶

The paper builds gRef-CW, the first generalised visual grounding benchmark for agricultural field scenes (8,034 high-resolution images, 82,592 template-based referring annotations including image-level and instance-level negatives), and proposes Weed-VG, a modular framework on top of GroundingDINO that splits grounding into global existence detection and instance relevance scoring bound by a hierarchical constraint, reaching 62.42% Top-1 and 78.35% Neg-Acc on the test set while four general grounding baselines peak at 34.88% Top-1 and below 26% Neg-Acc in zero-shot evaluation.

Background & Motivation¶

Vision-language models in agriculture can already write image captions and answer visual questions: benchmarks such as AgriBench, AgroBench, AgEval, AgMMU and MIRAGE cover crop disease recognition and agronomic reasoning. Visual grounding (VG) — localising an object in an image from a natural-language query — is essentially untouched in this domain. This is not a peripheral capability: grounding is a core component of fine-grained vision-language understanding, it feeds back into VQA by anchoring answers to specific regions, and it maps directly onto real precision-agriculture needs — deciding whether a given crop or weed is present in a patch of field, how many there are, and which individual plants they are, which in turn supports selective weeding, variable-rate fertilisation, irrigation and harvesting. Conventional object detectors are limited to a predefined category set, whereas generalised Visual Grounding (gVG) allows one query to refer to zero or multiple targets, matching this "first ask whether, then ask which plants" mode of work.

The problem is that transferring existing gVG methods to field imagery collapses. GroundingDINO reports only 33.9 mAP on agricultural images containing weeds; on AgroBench, open-source VLMs perform close to chance at weed identification, with the best model at 55.17% accuracy; instance-aware methods such as InstanceVG often fail to localise small objects. Behind these numbers are three concrete, intertwined difficulties. First, crops and weeds look highly similar at the seedling stage, and to a general model they are entirely novel classes never seen in training, so colour, shape and texture — object-level features — cannot separate them; the only usable cue is instance-level information: where this plant is and how big it is. Second, the scale range is extreme: instance area covers only 0.01%–0.97% of the image, 84.7% of gRef-CW instances are tiny or small, and standard IoU regression losses have unstable gradients and slow convergence at that magnitude. Third, fields are dense and crowded: 30.7% of images contain more than 10 instances and some exceed 30, while an absent target is the norm rather than the exception — the model must learn to abstain instead of grabbing whichever plant looks similar.

Studying these difficulties systematically requires a benchmark that can measure them, and that is exactly the missing piece: existing agricultural multimodal benchmarks evaluate recognition and reasoning, none is designed around a VG protocol, and none writes "the target may not be in the image" into its annotations. This paper therefore does two things. It converts the CropOrWeed9 subset of CropAndWeed into gRef-CW, generating a template-based "(Size) (Category) in the (Position)" referring expression for every distinguishable instance and synthesising negatives with two strategies — Replace (swap the category) and Swap (interchange two attributes within the same category, size or position) — so that "this image contains no such target" becomes an evaluable label. It then splits gVG explicitly into two levels: an image-level sentence vocabulary decides whether the referred category is present at all (Level-0 existence detection), and instance relevance is scored per proposal under that condition (Level-1 instance relevance), with a max-form hierarchical constraint tying the two together — if existence is not learned, instance-level loss is held up. Core idea: decompose generalised visual grounding into image-level existence detection and instance-level relevance ranking, use \(L_{\text{lvl1}}^{\text{constrained}} = \max(L_{\text{lvl1}}, L_{\text{lvl0}})\) so that "a target that does not exist cannot be localised", and pair it with distance- and size-aware matching plus interpolation-driven box regression to stabilise proposal optimisation under extreme scale variation.

Method¶

Overall Architecture¶

Weed-VG is a modular framework: its localisation power comes from an off-the-shelf grounding model and the paper only adds two-level scoring on top plus a more stable matching/regression strategy — which means the base model is swappable (GroundingDINO is used to instantiate it) without redesigning a detector. The input is a field image and two texts: a general text (e.g. "plant or vegetation") that drives the grounding model to produce proposals, and a referential text (e.g. "medium weed in the top right") that is the actual query. Visual features of the proposals and the textual features are projected into a shared embedding space, passed through Multi-Head Cross-Attention (MHCA) and an FFN to obtain aligned multimodal relevance logits, and then handled by the Hierarchical Relevance Scoring (HRS) module at two levels: Level-0 performs multiclass classification over a fixed vocabulary of image-level sentences, deciding whether the referred category appears in the image at all; Level-1 assigns each proposal a referring score that fuses sentence-level and word-level similarity through a learnable weight. During training, a hierarchical constraint uses the existence loss as a lower bound on the instance-level loss, enforcing logical consistency (something that does not exist cannot be localised); the localisation branch assigns proposals to ground truth with a distance- and size-aware matching cost and refines boxes with interpolation-driven InterpIoU. At inference, the model first emits its existence decision for the query, and only if the target exists does it rank proposals by referring score to produce Top-1/Top-k — otherwise it outputs background (a correct rejection). This is what simultaneously determines its Recall and its Neg-Acc. On the data side, gRef-CW supplies the two-level labels: image-level "does this category occur" and instance-level "which plant this expression refers to".

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["gRef-CW: multi-level text annotations<br/>and negative construction"] --> B["Image + general text<br/>→ grounding model proposals"]
    B --> C["Two-level relevance scoring<br/>existence and instance relevance"]
    C --> D["Hierarchical constraint<br/>existence as instance-level lower bound"]
    D --> E["Distance/size-aware matching<br/>and interpolation regression"]
    E --> F["Existence decision + instance boxes<br/>Top-1 / Top-k / Neg-Acc"]

Key Designs¶

1. gRef-CW: multi-level text annotations and negative construction

What agricultural VG lacks first is not a model but data: without a benchmark carrying negatives and instance-level referrals, there is no way to measure whether a model can abstain. gRef-CW is built from the CropOrWeed9 subset of CropAndWeed, with labels mapped to eight crop types (Maize, Sugar beet, Bean, Pea, Sunflower, Soy, Potato, Pumpkin) plus a single unified weed class — this subset was chosen because the original work found joint training across all crops with one weed category separates crops from weeds better than single-crop models. Annotation follows four steps: filter out instances smaller than 16×16 pixels that humans cannot recognise either; extract three attributes per remaining instance — category, position (the 3×3 grid cell containing the box centre, e.g. top-left / bottom-center), and size (four levels by box area: tiny < 2k px², small 2k–20k px², medium 20k–208k px², large ≥ 208k px²); fill the attributes into a fixed template "(Size) (Category) in the (Position)" to produce positive referring expressions; finally synthesise negatives with two strategies — Replace substitutes the category word (crop → weed), Swap interchanges two attributes within the same category (small → large, top left → bottom right).

This annotation solves two problems at once. Putting position and size into the sentence injects the one reliable discriminative cue beyond appearance: two seedlings look identical, but "the tiny weed in the bottom right" is unique in language. Negatives turn abstention into a trainable, measurable capability: for the test set, roughly one third of the 11,997 candidates are sampled for each of three manipulations — category replacement, size swapping and position swapping — yielding 9,186 negatives (3,706 category, 3,294 size, 2,186 position) with the rest kept positive; negatives share the same annotation format as positives, so presence and absence can be evaluated under one metric suite. The final dataset contains 8,034 images, 78,288 instances and 82,592 annotations (78,288 instance-level plus 4,304 image-level), split 70:15:15 with balanced representation of images containing only crops, only weeds, both, or neither. Compared with the RefCOCO family its images are larger (side length ≈ 1,445), instances smaller (normalised area 0.01–0.97), sentences longer (6.34 words on average), and the category set narrower but far more long-tailed — 84.7% of instances are tiny/small and 30.7% of images hold more than 10 instances, which is precisely why general models lose points here. (The 11,997 candidates and the three negative counts 3,706/3,294/2,186 are not evenly divisible in the original, ⚠️ refer to the original paper.)

2. Two-level relevance scoring: existence and instance relevance

Standard referring grounding answers only "which box"; under the generalised setting the question becomes two: is there any such target in this image, and if so, which ones. HRS splits these into two levels sharing the same visual-textual features. Level-0 casts existence as multiclass classification over the image-level sentence vocabulary: for each sentence \(t_k\) in the vocabulary, the maximum score over all proposals serves as the pooled logit, and a softmax over the vocabulary gives the image-level class probability.

\[s_{\text{pool}}(k) = \max_{j} s(v_j, t_k), \qquad P(k \mid I) = \text{Softmax}\big(s_{\text{pool}}(k)\big)\]

Max rather than mean is the right pooling because the judgement "this category is present" should hold as soon as one region matches the sentence strongly — exactly the semantics of existence detection. Level-1 computes a referring score per proposal by fusing two granularities: sentence-level similarity compresses the sentence into one vector and takes a temperature-scaled cosine similarity against the proposal feature (learnable temperature, initialised at 0.07), carrying overall semantics; word-level similarity compares the proposal feature against each word token to form a similarity matrix and max-pools it, anchoring words such as "tiny", "weed" and "bottom right" onto boxes. The weight between them is not fixed but generated from the global textual feature through an MLP followed by a sigmoid:

\[S_{\text{ref}} = w_s \cdot S_{\text{sent}} + (1 - w_s) \cdot \text{MaxPool}(S_{\text{word}})\]

That weight matters because different sentences carry their discriminative content in different places — in "tiny weed in the bottom right" the category word, the size word and the position word all contribute, and the model has to decide for itself whether to trust the whole sentence or one token this time. Instance level is optimised with a BCE loss, which naturally accommodates the multi-positive case of one query matching zero, one or several instances; the sentence-level feature is obtained by valid-mask max pooling over word embeddings, keeping each token's maximum response across channels. (Operators in Equation (2) are missing in the cached text; they are restored here by semantics, ⚠️ refer to the original paper.)

3. Hierarchical constraint: existence as the instance-level lower bound

Splitting the two levels creates a new risk: instance ranking can be learned "on its own", so a proposal may receive a high referring score even when the image-level existence decision is completely wrong — for a negative expression this means handing over another crop plant in the background as the target. The paper writes that logical dependency into the loss as a hard constraint: the instance-level loss may not fall below the image-level loss, i.e. the maximum of the two is taken as a lower bound.

\[L_{\text{lvl1}}^{\text{constrained}} = \max\big(L_{\text{lvl1}},\, L_{\text{lvl0}}\big)\]

The gradient effect is that while existence is still poor (\(L_{\text{lvl0}}\) large) the instance-level loss is held up and cannot keep decreasing, which effectively postpones Level-1 optimisation until existence has stabilised; once existence is reliable the max degenerates into an ordinary instance-level loss and no longer interferes with ranking. The ablation is clean evidence: removing this term drops Neg-Acc from 78.35 to 41.60 (nearly halved) while Top-1 (59.87), [email protected] (53.87) and mIoU (55.83) barely move — the constraint carries the abstention capability specifically, rather than buying it by sacrificing localisation quality. This design also explains why "multi-label" is a keyword of the paper: the supervision itself is hierarchical and multi-label, and the image-level existence label must stay consistent with the instance-level relevance labels.

4. Distance/size-aware matching and interpolation regression

The localisation branch faces the same scale problem: predicted and ground-truth boxes frequently barely overlap, at which point IoU-based losses have near-zero gradients and converge slowly and unstably. A joint matching cost first assigns proposals to ground-truth instances, accounting for overlap, centre distance and relative scale together:

\[C_{ij} = \big(1 - \text{IoU}(P_i, G_j)\big) + \lambda_c \, \lVert \mathbf{c}(P_i) - \mathbf{c}(G_j) \rVert^2 + \lambda_s \cdot \text{SizeDiff}(P_i, G_j)\]

The first term is standard IoU overlap, the second penalises the squared L2 distance between box centres, and the third captures relative width-height discrepancy. The weights \(\lambda_c = 2.0\) and \(\lambda_s = 0.5\) are chosen empirically: under extreme scale variation centre error tends to dominate matching, so centre alignment is prioritised while some size awareness is retained. Regression itself is interpolation-driven: an intermediate box is linearly interpolated between the prediction and the ground truth, and an auxiliary IoU term is built from it.

\[B_{\text{int}} = (1 - \alpha) B_{\text{pred}} + \alpha B_{\text{gt}}, \qquad L_{\text{InterpIoU}} = L_{\text{IoU}}(B_{\text{pred}}, B_{\text{gt}}) + L_{\text{IoU}}(B_{\text{int}}, B_{\text{gt}})\]

\(\alpha = 0.99\) follows the original InterpIoU implementation (⚠️ coefficient as in the original paper). The key is the \(L_{\text{IoU}}(B_{\text{int}}, B_{\text{gt}})\) term: even when prediction and ground truth hardly overlap, the interpolated box still overlaps the ground truth substantially, so the gradient does not vanish but pushes the prediction smoothly toward the target — stabilising gradients at the 0.01%-area magnitude without any handcrafted geometric penalty. The ablation shows that swapping this whole package (interpolation regression plus matching) back to GroundingDINO's original GIoU costs 9.53 mIoU and 13.27 Top-1 on the test set, with small instances hit hardest.

A Worked Example¶

Take one test image holding 20–30 instances (the 21–30 density bin of gRef-CW) and the query "tiny weed in the bottom right". The system first has GroundingDINO produce several hundred proposals driven by the general text "plant or vegetation"; the query is encoded into word tokens plus one sentence vector, and each proposal's visual feature is projected with them, passed through MHCA and the FFN to obtain aligned multimodal features. Two things then happen in parallel. On the Level-1 side, each proposal gets a temperature-scaled cosine similarity against the whole sentence, \(S_{\text{sent}}\), and a max-pooled similarity against each word token, \(S_{\text{word}}\); the weight \(w_s\) produced by the MLP+sigmoid from the global textual feature fuses them into \(S_{\text{ref}}\). On the Level-0 side, the image-level sentence vocabulary ("No crops are present in this image", "No weeds are present in this image", "No crops or weeds are visible in this image", and so on) is compared against all proposals, each sentence takes its maximum response, and a softmax yields the decision of whether any weed is present.

Two cases now diverge. If this is a negative expression (say the original sentence was "tiny crop in the bottom right" and the category was swapped to weed), Level-0 gives a very low weed-existence probability, the hierarchical constraint holds down the instance-level scores for that image, and the model outputs "no target" — this sample is counted as correct in Neg-Acc, instead of settling for pointing at another ground-truth plant the way the baselines do. If it is a positive expression, the existence check passes and the proposal with the highest \(S_{\text{ref}}\) becomes Top-1, counted as a hit when its IoU with the ground truth reaches 0.5; all proposals and their scores feed [email protected] and mIoU (Top-5 asks whether a correct box appears among the top-ranked candidates). This example also exposes the cost-benefit balance of the two-level design: a correct instance is usually already in the proposal set ([email protected] is not low), and the hard part is ranking it first — which is why the gap between Top-1 and [email protected] is the metric this paper is really attacking.

Loss & Training¶

The total loss adds the hierarchical multi-label constraint loss and the box regression loss (the plus signs in Equations (6) and (7) are missing in the cached text; restored here by semantics, ⚠️ refer to the original paper):

\[L_{\text{total}} = L_{\text{HMCE}} + L_{\text{InterpIoU}}, \qquad L_{\text{HMCE}} = \lambda_0 L_{\text{lvl0}} + \lambda_1 L_{\text{lvl1}}^{\text{constrained}}\]

The two weights adapt to image type, reflecting where supervision should focus: multi-referent (mixed) images use \((\lambda_0, \lambda_1) = (1.0, 2.5)\) to prioritise instance discrimination; single-category images use \((1.0, 2.0)\); and empty images use \((1.0, 0)\), optimising existence alone — since no target is present, imposing an instance-level constraint would only inject noise.

Training runs in two stages because end-to-end joint training caused a visible degradation in regression performance. Stage one initialises from the original grounding checkpoint and fine-tunes only the last decoder layer and the box head with InterpIoU for 100 epochs, settling the box regression first; stage two trains only the projection/attention layers and the HRS module with two-level labels for 60 epochs, focusing on hierarchical semantics and instance-level discrimination. The batch size is 4, one sample per image type (crops only / weeds only / both / neither); because images containing both crops and weeds have no negative-sentence annotations and hierarchical labels cannot be empty, training introduces a special [EMPTY] token as a placeholder. Optimisation uses AdamW with a cosine annealing schedule and an initial learning rate of \(2 \times 10^{-4}\); augmentation adds Copy–Paste for instance diversity along with random rotations and colour jittering to simulate viewpoint, illumination and plant-appearance variation. The maximum encoder input length at test time is 64 tokens. Inference costs 10 FPS (97.6 ms per image) and 1.5 GB of GPU memory, with experiments on NVIDIA A100 and RTX PRO 6000 Blackwell GPUs.

Key Experimental Results¶

Main Results¶

The evaluation protocol deserves stating first, because this paper compares differently from the usual setup. Three dimensions characterise capability on gRef-CW: [email protected] (retrieval — is the target instance somewhere in the proposal set), Top-k accuracy (ranking — Top-1 requires the highest-scoring box to reach IoU ≥ 0.5 with the ground truth), and mIoU (localisation quality — how tight the box is). Because the baselines are highly sensitive to score thresholds and miss many instances on gRef-CW, the authors re-implemented them to evaluate all of their proposals given an instance sentence rather than only those above a threshold. The paper additionally introduces Negative Accuracy (Neg-Acc), a threshold-free GIoU-based metric evaluated only on sentences where no target exists; a prediction is correct when its maximum GIoU with any ground-truth box is ≤ 0, i.e. the model genuinely refers to background. Neg-Acc is meaningless on its own and must be read together with Recall — the pair forms a test of conditional understanding: detect when the target is present and abstain when it is absent, both at once. One boundary condition must be stated too: the four baselines are evaluated zero-shot (methods other than HRS cannot be fine-tuned on multi-level labels, so a direct methodological comparison would be unfair), whereas Weed-VG is trained on gRef-CW. This table therefore measures the domain gap and establishes an improved agricultural baseline, not a like-for-like method ranking.

Model (Test set)	Top-1	Top-5	[email protected]	mIoU	Neg-Acc
MDETR	10.16	12.97	7.78	54.19	3.32
GroundingDINO-T	11.92	31.99	20.34	17.44	2.88
GroundingDINO-L	20.38	43.49	28.73	23.68	7.52
SAM3	34.88	66.80	46.65	32.76	25.53
Weed-VG (Ours)	62.42	82.45	55.44	57.25	78.35

The validation set follows the same trend (Weed-VG: Top-1 63.01, Top-5 81.64, [email protected] 55.71, mIoU 58.12).

Breaking the test set down by instance scale pinpoints the failure. The table below reports the two bins that best expose the scale gap (Top-1 accuracy, tiny and large):

Model	Crop Tiny	Crop Large	Weed Tiny	Weed Large	Crop tiny→large gap
MDETR	— (no valid detection at this scale)	11.71	— (same)	13.09	—
GroundingDINO-T	0.00	31.64	0.31	43.32	31.6
GroundingDINO-L	1.67	59.44	0.83	65.20	57.8
SAM3	13.42	71.33	16.17	78.62	57.9
Weed-VG (Ours)	54.66	71.90	53.44	76.79	17.2

Scene density points at the same root cause, unstable ranking: in sparse scenes (1–10 instances) SAM3 reaches 49.58 crop mIoU (Test), but above 30 instances it falls to 25.81 and [email protected] collapses from 84.77 to 31.26, i.e. proposals interfere with each other once there are many of them. Weed-VG still holds 47.58 crop mIoU and 50.00 weed mIoU in the most crowded >30-instance scenes, with [email protected] of 44.42 and 31.71 respectively.

Ablation Study¶

Config (Test set)	Top-1	Top-5	[email protected]	mIoU	Neg-Acc	Note
Sentence-only	49.29	74.06	50.53	49.35	74.12	Sentence-level similarity only; Top-1 drops 13.13, mIoU 7.9
Word-only	36.19	67.61	44.20	39.22	71.55	Word-level cues only; drops even further
w/o Query Projection	33.20	58.02	38.43	35.39	69.83	Removing the shared-space projection; largest Top-1/mIoU collapse
w/o Hierarchical Constraint	59.87	80.57	53.87	55.83	41.60	Localisation/recall almost unchanged, Neg-Acc halved
w/o InterpIoU	49.15	73.28	44.61	47.72	75.88	Back to GIoU; mIoU drops 9.53, Top-1 drops 13.27
Full Model	62.42	82.45	55.44	57.25	78.35	Full model

Key Findings¶

The shared text-vision projection contributes the most: removing Query Projection drops Top-1 from 62.42 to 33.20 (-29.22) and mIoU by 21.86, the worst of all ablations. Instance grounding depends heavily on projecting visual and textual features into a shared learnable space that captures both image-level and instance-level context; without the projection every downstream similarity is computed on misaligned representations.
Sentence-level and word-level cues are both indispensable: sentence-only loses 13.13 Top-1 and word-only falls to 36.19, so neither branch alone supports grounding. The reading is that sentence level supplies "what this expression refers to overall" while word level supplies anchors such as "tiny" and "bottom right" that land on specific boxes — complementary, not redundant.
The hierarchical constraint is the sole source of abstention ability: removing it drops Neg-Acc from 78.35 to 41.60 while Top-1, [email protected] and mIoU barely move. It is a very cheap constraint with a very specific payoff — it does not improve localisation, it only lets the model dare to say "not here" when the target is absent.
Interpolation regression pays off mostly on small objects: reverting to GIoU costs 9.53 mIoU and 13.27 Top-1, with the paper reporting the degradation as particularly pronounced on small instances, consistent with the motivation that standard IoU gradients fail when area share drops to 0.01%.
Scale, not ranking alone, is the biggest weakness of general models: GDINO-T scores 0.00 Top-1 on tiny crops and GDINO-L only 1.67; SAM3 raises tiny to 13.42 through dense proposals but reaches 71.33 on large, a 58-point gap. Weed-VG cuts the crop tiny→large gap to 17 points, which is why it lifts Top-1, [email protected] and mIoU together: it fixes both "cannot retrieve" and "ranks wrong" rather than spiking a single metric.
A counter-intuitive observation: MDETR's mIoU (54.19) is far above GroundingDINO-L's (23.68) while its [email protected] (7.78) is a quarter of the latter's. MDETR draws accurate boxes once it retrieves the right instance, but it almost never does; reading mIoU jointly with [email protected] is necessary here, since either alone leads to the wrong conclusion.

Highlights & Insights¶

Making abstention a first-class metric: Neg-Acc defines "correctly referring to background" via GIoU ≤ 0, sidestepping threshold choice, and it only works when read alongside Recall. Compared with reporting Recall/AP alone, this pair directly exposes the illusion of "flooding the scene with proposals to inflate recall" — baselines' Neg-Acc of roughly 3–7.5% is the quantified evidence of exactly that illusion.
A max constraint rather than a weighted sum: \(\max(L_{\text{lvl1}}, L_{\text{lvl0}})\) is gradient-wise equivalent to "freeze instance-level optimisation while existence is unlearned", costs nothing to implement, needs no extra hyper-parameter, and the ablation shows it barely perturbs localisation metrics. This way of encoding a logical dependency in the loss form transfers to any "decide-then-rank" cascade, such as existence gating in retrieval or hierarchical constraints in multi-label classification.
Putting position and size into the language template instead of the architecture: gRef-CW feeds instance-level cues to the model through the minimal template "(Size) (Category) in the (Position)", sidestepping the difficulty of making a model guess which plant from pixels alone. This is directly reusable in any setting with small intra-class variation that must be separated by spatial attributes — remote sensing, medical slides, industrial inspection.
The two faces of template-based annotation: the cost is poor linguistic diversity and no way to evaluate free-form or richer contextual expressions (which the authors admit), but the benefit is fully programmatic annotation with controllable attributes, which is what makes it possible to manufacture three precise negative types via Replace/Swap — something free-form annotation cannot do. Controllability and naturalness are an explicit trade-off here.

Limitations & Future Work¶

Dependence on base-model proposal quality: Weed-VG does not generate proposals but consumes those of a grounding model. The paper explicitly acknowledges reliance on initial proposal quality — if the base model never boxes that tiny seedling, no amount of HRS accuracy can rank it. Stage two's ceiling is set by stage one.
Degradation in extremely dense scenes: above 30 instances, crop mIoU is 47.58 and weed [email protected] only 31.71, clearly below sparse scenes (63.92 / 69.02). Occlusion and mutual interference among proposals are mitigated, not solved.
The annotation format limits what can be evaluated: template sentences have only three slots — size, category, position — so freer expressions ("the weed that looks a bit yellow on the lower right") and richer semantics such as growth stage or context cannot be tested. The authors suggest porting this multi-level annotation scheme to standard benchmarks, but the annotation cost is significant.
Narrow category and environment coverage: only 8 crop types plus one unified weed class, and all images come from CropAndWeed with top-down viewpoints; cross-region, cross-season and cross-cropping-system generalisation is untested. The paper's future work lists larger-scale agriculture-specific datasets for this reason.
Reservations about the comparison protocol: baselines are zero-shot while Weed-VG is trained on gRef-CW, and the authors themselves avoid a direct methodological comparison "to ensure fairness". Table 1 should therefore be read as a quantification of the domain gap and a baseline reference, not as evidence that HRS beats other gVG designs. The baseline set is also only four models (MDETR, GroundingDINO-T/L, SAM3) with no measured numbers for GLIP-style or InstanceVG-style methods.
Concrete improvement directions: (1) let stage two feed back into proposal generation (e.g. return HRS scores to the proposal stage or run iterative refinement) instead of consuming it one-way; (2) use a stronger text encoder or free-text template expansion to reduce the linguistic bias of template sentences; (3) extend the Level-0 vocabulary from "category presence" to "attribute-combination presence" so existence detection also handles attribute-level negatives — of the three negative types, position swapping is precisely what Level-0 is least sensitive to.

vs classic referring datasets (RefCOCO / RefCOCO+ / RefCOCOg / ReferItGame): they assume exactly one target per query and contain no negatives, whereas gRef-CW writes "zero targets" into the labels, uses larger images (≈ 1,445), smaller instances (normalised area down to 0.01), longer sentences and higher instance density (30.7% of images with more than 10 instances). The cost is template-generated rather than human-written language, which is also why the paper states its metrics are not directly comparable with those benchmarks.
vs open-set detectors such as GroundingDINO / GLIP: they perform open-vocabulary detection from text prompts and serve as Weed-VG's base model and baselines; the difference is that this paper does not modify the detector but adds two-level relevance scoring plus matching/regression on top. In Table 1, scaling GroundingDINO from T to L raises Top-1 from 11.92 to 20.38 yet still trails SAM3, showing that enlarging the detector alone cannot resolve instance ambiguity — exactly the gap a ranking-side design such as HRS is meant to fill.
vs MDETR: MDETR also performs multimodal alignment, but its handling of negatives and gVG differs; on gRef-CW it exhibits an extreme "accurate but unfindable" profile (mIoU 54.19 / [email protected] 7.78), meaning its alignment cannot pick out the right plant among dense same-category instances.
vs the gREC / gRES line (e.g. InstanceVG): these share the "zero or multiple targets" setting, but the paper notes they often fail on small objects and cannot consume its multi-level labels directly (lacking a structure like HRS), which is why no direct method comparison is made.
vs agricultural multimodal benchmarks (AgriBench / AgroBench / AgMMU / MIRAGE): they evaluate recognition and reasoning, not grounding. The transferable lesson is that in any domain with small intra-class variation that must be separated by instance-level attributes, establishing the domain benchmark first (with negatives) and then designing the ranking-side module tends to work better than porting a general model directly.

Rating¶

Novelty: ⭐⭐⭐⭐ First agricultural generalised visual grounding benchmark, plus a clean and effective split of gVG into existence and relevance bound by a max constraint; but the individual components (contrastive similarity, hierarchical multi-label learning, InterpIoU) are combinations and transfers of existing techniques.
Experimental Thoroughness: ⭐⭐⭐⭐ Main results, scale stratification, density stratification, negative-type breakdown and a full ablation, with a well-argued Neg-Acc metric; points off because there are only four baselines, all zero-shot and thus not directly comparable with the trained Weed-VG, and there is no cross-dataset generalisation study.
Writing Quality: ⭐⭐⭐⭐ Motivation and dataset construction are concrete (attribute thresholds and negative counts included) and the architecture figure is clear; but several equations in the cached full text have corrupted symbols ((2), (6), (7)) and the negative counts do not divide the candidate total evenly, so some details require checking against the original.
Value: ⭐⭐⭐⭐ Directly relevant to selective weeding and crop monitoring in precision agriculture, and both the dataset and the Neg-Acc metric are reusable by follow-up work; limited by narrow category and scene coverage and template-based annotation, so cross-domain adoption needs larger datasets.