Background Blurring Matters: Improving Visual Grounding by Merging Text-Irrelevant Tokens¶

Conference: ECCV 2026
Paper: Official page · PDF
Code: https://github.com/Mr-Bigworth/ToB
Area: Multimodal VLM
Keywords: Visual grounding, background suppression, token merging, textual relevance, multi-level feature fusion

TL;DR¶

ToB preferentially merges visually similar but text-irrelevant tokens in deeper visual encoder layers, preserving fine-grained target representations and improving a DINOv2-B/BERT-B baseline from 76.05% to 80.36% on RefCOCOg test-u while increasing inference speed from 20.80 to 21.57 fps.

Background & Motivation¶

Visual grounding takes an image and a referring expression and predicts the bounding box of the described object. Conventional dual-branch Transformers encode the image and text separately before fusing them in a decoder, so the visual encoding process does not necessarily know which object the user wants. Subsequent methods introduce language-conditioned weights, cross-modal attention, or feature modulation earlier in the visual encoder, but primarily change individual token representations or attention strengths.

This paper focuses on a more specific issue: uniform image patching creates many background tokens, and even tokens receiving little attention still participate in softmax normalization and feature aggregation. When the target is small, the background dominates token counts, consuming computation and potentially interfering with target representations. However, directly importing compression based only on visual similarity is also problematic: two similar-looking objects may be the target and a distractor, and merging them can erase the distinction specified by the expression.

Consequently, background should be defined relative to the query rather than fixed by visual content alone. Nor should every low-relevance token simply be discarded, because localization may still require context. Core idea: incorporate textual relevance into visual token pairing, consolidate redundant background into fewer representations, and reuse the textual weights to strengthen the target information that remains.

Method¶

Overall Architecture¶

The model encodes images with DINOv2-B and expressions with BERT-B, inserting Token Blurring (ToB) modules into the last six visual encoder layers. Each module generates a textual weight for every visual token, selects pairs using visual similarity and textual irrelevance, and textually reweights the shortened visual sequence. Blurring happens in the feature sequence, not through Gaussian blur applied to the input image, and does not require a supplied target mask.

At the encoder output, features from the last three visual layers are concatenated and passed with language features to a six-stage cascade decoder. A learnable [REG] token progressively gathers target information, and a regression head predicts a bounding box. ToB therefore changes the sequence length processed by subsequent attention, rather than merely adding another attention score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and referring expression"] --> Enc["DINOv2-B and BERT-B<br/>Separate encoding"]
    Enc --> Weight["Text-aware weights"]
    Weight --> Merge["Joint matching and weighted merging<br/>Repeated in the last six visual layers"]
    Merge --> Fusion["Multi-level feature fusion"]
    Fusion --> Output["Six-stage decoder and REG<br/>Target bounding box regression"]

Key Designs¶

1. Text-aware weights: estimate how strongly each visual token relates to the full expression

The current visual features and BERT language features are independently linearly projected into a common 256-dimensional space. The model then computes dot products between every visual token and every language token. For each visual token, these correlations are averaged along the text dimension and passed through a sigmoid, producing a weight between zero and one. This weight represents average association with the expression; it is neither a foreground label derived from the ground-truth box nor a probability distribution normalized over visual tokens.

The central operation in the paper's Equation (3) is:

\[ W_i=\sigma\!\left(\operatorname{AvgPool}_{\text{text}}\left[(F_{v,i}P_{v,i})(F_lP_{l,i})^\top\right]\right). \]

The same image patch can consequently receive different weights for different queries. An expression emphasizing clothing should prioritize a person and their clothes, whereas one emphasizing a nearby item requires a different set of local details. Nevertheless, this remains a learned soft relevance estimate: averaging may dilute individual relational words, and the weights are not guaranteed to be correct. ToB is restricted to the final six visual layers specifically to avoid irreversible merging while shallow visual features are still poorly aligned with language semantics.

2. Joint matching and weighted merging: compress visual redundancy while letting text set the priorities

ToB partitions the current visual sequence into groups A and B using odd and even indices, applying the same partition to textual weights. It computes cosine similarity for every cross-group pair and uses the geometric mean of the two textual weights to describe pairwise textual relevance. The following expression combines the operations in Equations (2), (4), and (5); \(a,b\) denote candidate tokens and \(w_a,w_b\) their textual weights:

\[ M(a,b)=(1-\alpha)\cos(f_a,f_b)+\alpha\left(1-\sqrt{w_aw_b}\right). \]

High visual similarity and low pairwise textual relevance increase merging priority. Importantly, the geometric mean is not a hard requirement that both tokens be irrelevant: a very small weight at one endpoint can also lower the product. Visual similarity must therefore contribute to selection, and target tokens are not absolutely protected. The parameter \(\alpha\) balances the two terms. Figure 2 illustrates the calculation with \(\alpha=0.3\), but the implementation description does not explicitly establish that value as the default for all experiments, so the illustration should not be promoted to a confirmed reproduction setting.

Pairing is not a global one-to-one assignment. Each A token chooses its highest-scoring B token, producing one candidate edge; the top \(r\) candidate edges are then selected by score. Multiple A tokens may choose the same B token. For each recipient, the method takes the arithmetic mean of its own feature and those of all selected A tokens assigned to it, and averages their textual weights in the same way. Absorbed A tokens are removed, and the remaining sequence is rearranged according to original indices, reducing its length from \(N\) to \(N-r\). Thus, \(r\) counts removed source tokens even when the selected edges do not represent \(r\) independent two-token pairs.

Finally, each resulting feature is multiplied by its resulting textual weight before entering the next Transformer layer. ToB therefore has two effects: merging changes the number of background tokens, while multiplicative modulation changes the strength of surviving representations. Processing does not stop after averaging. Averaging rather than outright deletion also retains a compressed representation of background context, but does not preserve the independent details of every original patch.

3. Multi-level feature fusion: recover some localization cues lost through deep merging

After six successive compression steps, the deepest representation is semantically strong, but some spatial details have already been averaged away. The authors therefore concatenate outputs from the last \(k\) visual encoder layers instead of exposing only the final layer to the decoder; the default is \(k=3\). Because these layers have undergone different numbers of merging operations, earlier outputs offer finer local information and later outputs provide more compressed semantic representations. The paper describes concatenating multi-level outputs but does not detail an additional cross-layer positional alignment mechanism; an unmerging or interpolation module should not be invented to fill that gap.

More layers are not always better. Including earlier representations can reintroduce more background tokens and enlarge the decoder input. In the ablation, \(k=3\) achieves 80.36% on RefCOCOg test-u, compared with 79.69% for \(k=1\), while \(k=5\) falls to 79.25%. The six-stage cascade decoder itself follows an existing design: each stage uses [REG] to aggregate visual and linguistic information, and the last stage's box is the final prediction. This decoder is not a new token-merging rule introduced by ToB.

A Worked Example¶

Suppose the query refers to a small object. With the paper's default \(518\times518\) input and \(14\times14\) patches, the image produces \(37\times37=1369\) patch tokens. Text-relevant target details tend to remain separate, while visually similar, low-relevance background locations are preferentially averaged together. This is a conceptual walkthrough, not a claim that ground-truth boxes guide token selection.

Each default ToB module removes 64 tokens. Under the paper's patch-token counting convention, the six successive sequence lengths are 1305, 1241, 1177, 1113, 1049, and 985. Every step both shortens the sequence and recomputes textual weights from current-layer features before reweighting. The final three layers jointly supply the decoder, so prediction need not depend exclusively on the deepest, coarsest set of 985 representations. The actual starting token count is odd, whereas the equal-partition formula is written for an even count; handling odd lengths and special tokens in code cannot be inferred from this counting example.

Loss & Training¶

Training remains bounding-box regression without additional background-mask supervision. All six decoder stages receive smooth L1 and GIoU losses, weighted by 5 and 2 respectively. The paper denotes the former by \(\mathcal L_{L1}\) but explicitly calls it smooth L1 in the accompanying text.

\[ \mathcal L=\sum_{j=1}^{6}\left[5\mathcal L_{\mathrm{smooth\,L1}}(\hat b_j,b)+2\mathcal L_{\mathrm{GIoU}}(\hat b_j,b)\right]. \]

Training uses AdamW with batch size 32 and weight decay \(10^{-4}\) for 90 epochs, reducing learning rates tenfold after epoch 60. The visual and language encoders are initialized from pretrained DINOv2-B and BERT-B, both with an initial learning rate of \(10^{-5}\). The decoder uses Xavier initialization and an initial learning rate of \(10^{-4}\). Baseline and Baseline+ToB both use \(518\times518\) images, a maximum text-token length of 40, and [CLS] and [SEP] tokens.

When incorporating ToB into CLIP-VG, SimVG, and OneRef, the authors retain the original training strategies and initializations. Their respective square input resolutions are 224, 640, and 384 pixels. Plug-and-play therefore means that the module can be integrated and trained within these systems; it does not establish training-free insertion into an arbitrary existing model with identical gains.

Key Experimental Results¶

Main Results¶

The metric is [email protected] (%): a prediction is correct when its box has IoU greater than 0.5 with the ground-truth box. RefCOCO+ prohibits location words in expressions, while RefCOCOg contains longer expressions. Splits are explicitly identified below rather than pooled into a single dataset score. Results are selected from the paper's Table 1.

Model	RefCOCO val	RefCOCO testB	RefCOCO+ val	RefCOCO+ testB	RefCOCOg val-u	RefCOCOg test-u
Baseline, DINOv2-B/BERT-B	85.55	80.57	75.99	66.54	76.67	76.05
Baseline + ToB	87.73	83.91	78.73	68.81	80.39	80.36
CLIP-VG	84.29	78.43	69.55	57.62	73.18	72.54
CLIP-VG + ToB	86.03	80.50	75.23	65.36	78.45	77.88
SimVG	87.63	84.04	78.65	71.82	80.37	80.51
SimVG + ToB	88.95	84.87	80.02	73.11	82.23	81.79
OneRef	88.75	85.34	80.43	74.26	83.68	83.52
OneRef + ToB	89.51	86.01	81.76	76.05	84.21	84.64

The strongest evidence comes from before-and-after comparisons within each backbone, rather than a blanket state-of-the-art claim across different pretraining regimes. Baseline improves by \(80.39-76.67=3.72\) percentage points on RefCOCOg val-u. The improvement row in the original Table 1 says 3.74, which disagrees with the underlying values; this note uses the arithmetic difference. CLIP-VG gains 7.74 points on RefCOCO+ testB, but not every extended setting benefits: SimVG (28K), pretrained with additional data, drops from 87.94 to 87.43 on RefCOCO testB, a decrease of 0.51 points.

Ablation Study¶

The following table selects configurations from the paper's Tables 4 and 5. Accuracy columns are RefCOCOg [email protected] (%). The two groups separately vary fusion depth and per-layer merging count; they should not be interpreted as a joint grid search over all parameters.

Ablation dimension	Configuration	val-u	test-u	fps
Fusion layers	k=1	79.71	79.69	Not reported
Fusion layers	k=3	80.39	80.36	Not reported
Fusion layers	k=5	79.66	79.25	Not reported
Merged tokens per layer	r=0	78.91	79.12	20.55
Merged tokens per layer	r=16	80.05	79.78	20.88
Merged tokens per layer	r=32	80.17	79.64	21.09
Merged tokens per layer	r=64	80.39	80.36	21.57
Merged tokens per layer	r=128	79.26	78.84	22.05

The 79.12% result at r=0 is not the 76.05% result of the no-ToB Baseline. It belongs to the ToB merging-count sweep: disabling merging is not equivalent to removing the entire textual weighting pathway. The paper does not fully disentangle every operation through independent ablations, so the supported conclusion here is that these are different controls. Another detail is that test-u already decreases from r=16 to r=32; the table therefore does not support strictly monotonic accuracy improvement up to 64.

To relate localization gains to cost, the next table combines results from Tables 2 and 6. All configurations use the authors' Baseline, and speed and memory retain the paper's reporting convention.

Configuration	val-u	test-u	GFLOPs	Memory GB	fps
Baseline	76.67	76.05	118.31	19.97	20.80
Baseline + ToMe	76.41	75.25	Not reported	Not reported	22.09
Baseline + ToE	75.13	74.89	Not reported	Not reported	22.93
Baseline + ToB, r=64	80.39	80.36	113.85	19.21	21.57
Baseline + ToB, r=192	78.91	78.77	99.42	16.87	23.96
ToMe + ToB, r=96	77.53	77.07	76.84	14.78	28.71

Key Findings¶

Default ToB reduces GFLOPs by approximately 3.77% and improves fps by approximately 3.70%. Its primary benefit at this setting is accuracy, not an order-of-magnitude speedup; more aggressive compression increases speed at the expense of some localization quality.
At the same token reduction rate, ToMe and ToE are faster than ToB but less accurate than the original Baseline. Which tokens are compressed matters more than simply enabling compression; ToB does not win on raw throughput against these controls.
In Table 7, RefCOCOg test-u accuracy at IoU=0.9 rises from 26.92% to 48.38%, and small-object [email protected] increases from 65.32% to 72.03%. Small objects are defined by box area below \(128\times128\) pixels, so size thresholds from other benchmarks should not be substituted.

Highlights & Insights¶

The method extends the question of where attention should focus to which locations deserve independent representations. Background suppression becomes a structural change to the sequence, allowing accuracy and computational cost to benefit from the same operation.
The same textual weights serve both matching and output modulation. They influence which details are compressed and which information propagates afterward, although finer ablations are needed to attribute gains to these two effects separately.
Multi-level fusion partially compensates for irreversible merging. Earlier representations retain more detail, which could inform other fine-grained localization tasks, but extending the approach to dense outputs such as segmentation would require spatial mapping that this paper does not evaluate.

Limitations & Future Work¶

The authors' experiments directly show that excessive merging loses detail: r=128 reaches 78.84% on test-u, below the default 80.36%, and fusing too many layers also hurts. Allocating per-layer budgets according to object scale or relevance distributions is a possible extension, not an implemented result.
Textual relevance is a soft estimate rather than a protection constraint. Geometric means and expression-wide average pooling may mishandle relational words, local attributes, or pairs with highly unequal weights; targeted keyword-sensitivity and relational-expression evaluations would be useful.
Reproduction details remain incomplete: the main text does not fully specify the experimental default for \(\alpha\), odd-length sequences, or special-token handling, and the available full text does not state the throughput hardware and timing protocol. The code link is supplied by the authors; its repository contents were not checked online for this note.
Evidence covers box grounding on three established referring-expression datasets, without multi-seed variance or a systematic analysis of occlusion, distribution shift, and relational-reasoning failures. Equivalent gains in open-world grounding or every VLM are not established.
The latter four points are this note's analysis rather than a list of limitations explicitly acknowledged by the authors; the paper has no separate limitations section. The FitPrune comparison also adapts a LLaVA-oriented method to an encoder-decoder architecture and should not be treated as its performance ceiling in its native setting.

Compared with ToMe / ToE: the tested visual compression controls emphasize efficiency, whereas ToB puts query relevance into pairing scores to preserve details required for localization. Additional relevance computation is a cost, and default ToB has lower fps than these two controls.
Compared with LAPS / FitPrune: LAPS uses linguistic supervision for patch slimming, while FitPrune targets token pruning in multimodal large models. ToB performs better in the present VG experiments, but differences between native tasks and adapted settings prevent the blanket conclusion that pruning is always inferior to merging.
Compared with VLTVG / MMCA: these methods improve representations through iterative cross-modal reasoning or conditional adaptation. ToB's distinct addition is query-dependent reduction of visual token counts; its cascade decoder follows a VLTVG-like structure, so the module contribution should be distinguished from the inherited decoder architecture.
Relationship to SimVG / OneRef: these systems are both competing baselines and hosts for ToB. Within-host comparisons support some architectural compatibility, but the negative split result for extra-pretrained SimVG shows that gains still depend on the original model and data regime.

Rating¶

Novelty: 4/5. Directly integrating textual relevance into pairing and post-merge enhancement gives visual grounding a clear structural improvement, while building on established token-merging frameworks.
Experimental Thoroughness: 4/5. Three datasets, multiple hosts, compression controls, budget and fusion ablations, and high-IoU analysis provide useful coverage; finer component isolation and variance reporting are still missing.
Writing Quality: 3/5. The method is clearly motivated, but some reported gains, monotonicity claims, and implementation parameters require cross-checking.
Value: 4/5. A useful incremental component for language-guided visual encoders, with strong default accuracy gains and efficiency benefits that should be understood as the modest measured improvement.