T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/savya08/T-REN
Area: Multimodal VLM
Keywords: region-text alignment, multi-region queries, open-vocabulary segmentation, visual token compression, temporal aggregation

TL;DR

T-REN learns multiple regions per point and joint region-text alignment on frozen DINOv3/dino.txt, reaching 30.6 mIoU on ADE20K open-vocabulary segmentation and, in a separate video localization setting, 52.4% Recall@1 with 187.5-fold representation compression.

Background & Motivation

An image-level vision-language model can recognize whether an image relates to a phrase without accurately locating the corresponding region. Objects span many patches, while individual patches can mix foreground and background. Comparing each patch with category text therefore produces fragmented boundaries and requires storing many tokens per image. Long videos amplify both problems because the same object is represented by another collection of patches in every frame.

Previous approaches usually address these issues separately: dense CLIP alignment improvements retain patch-level representations, while token pruning or merging does not necessarily preserve complete semantic regions. REN already pools regions through point prompts and cross-attention, avoiding a separate SAM invocation for each image. However, it predicts only one token per point and does not learn region-text alignment. A point on a wheel belongs both to the wheel and to the entire car, forcing a single output to choose among valid interpretations.

T-REN is consequently a region encoder that can be queried directly with text, not simply a segmentation model followed by compression. Segmentation, image retrieval, and video localization share this representation. Core Idea: predict multiple hierarchical region candidates per point, jointly learn region pooling and text alignment, and merge redundant regions and objects across consecutive frames in visual space, so semantic alignment and compactness arise from the same training mechanism.

Method

Overall Architecture

A frozen DINOv3 ViT-L first extracts patch features. T-REN's lightweight query network converts point prompts into region tokens, and an MLP maps them into the text embedding space of DINOv3-based dino.txt. The added network has only 3.7% as many parameters as the original vision-language backbone; it does not retrain that backbone.

Image retrieval stores deduplicated region tokens, while video applications additionally store temporally aggregated track tokens. Point coordinates and frame indices accompany the representations, allowing a text match to recover its spatial and temporal support. The diagram describes the retrieval and video paths; image open-vocabulary segmentation disables region deduplication and should not be conflated with the compressed setting.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Frozen patch features<br/>and point prompts"] --> B["Multi-region query pooling"]
    B --> C["Visual-space deduplication"]
    C -->|Video| D["Temporal track aggregation"]
    C -->|Image| E["Joint region-text alignment"]
    D --> E
    E --> F["Text matching<br/>and spatiotemporal lookup"]

Key Designs

1. Multi-region query pooling: preserve part and whole interpretations at one point

Each point is encoded with Gaussian random Fourier features, which are added to \(k=3\) learned tokens. The resulting queries share a spatial location but can learn different semantic roles. A regular grid of \(P\) points produces \(P\times k\) queries at inference. These are not slots for three predetermined categories: region supervision teaches them to represent multiple entities at the same location, and redundant predictions can subsequently merge when only one meaningful region is present.

The queries pass through \(L=2\) decoder layers. Each layer first cross-attends to patch features and then applies self-attention among the three queries associated with the same point. This self-attention coordinates local alternatives rather than mixing all point queries indiscriminately. Positional encoding is re-injected afterward to preserve the spatial anchor. A final single-head cross-attention layer produces the region representation without value or output projections, keeping it a weighted aggregation in the frozen visual feature space rather than an arbitrarily transformed embedding.

Section 3.1 gives the attention weights as:

\[ a_i=\operatorname{softmax}\left(q_iK^\top/\sqrt{d_k}\right). \]

Here \(q_i\) is a region query, \(K\) contains patch keys, and \(d_k\) is the key dimension. The weights form a low-resolution soft mask over patch locations. Training encourages that mask to concentrate on the target region, whose original visual features are then pooled with those weights. A single head makes the spatial support of each token correspond to one interpretable mask rather than a mixture of several attention patterns.

2. Visual-space deduplication: learn similar representations before merging repeated regions

A regular grid covers the image but repeatedly queries the same wall or object. T-REN average-pools visual tokens when their cosine similarity exceeds \(\tau_{\mathrm{token}}=0.975\) or their cross-attention mask IoU exceeds \(\tau_{\mathrm{mask}}=0.8\). The output count adapts to scene complexity instead of enforcing the same small token budget on every image.

Merging occurs before text projection, avoiding a direct equivalence between sharing a category label and representing the same visual region. Its reliability also depends on training rather than thresholds alone: visual contrastive learning explicitly clusters predictions associated with the same ground-truth mask. Each merged token retains the original prompt coordinates for subsequent label propagation and localization. Compressing semantic features therefore does not discard all spatial information.

3. Temporal track aggregation: store a continuously visible object as one record

Per-frame merging still stores repeated descriptions of persistent video objects. For the next frame, T-REN compares its region tokens with active tracks using visual cosine similarity before text projection. Candidate pairs must exceed \(\tau_{\mathrm{track}}=0.65\), after which greedy one-to-one matching is performed. Unmatched tokens initialize new tracks. A track is active only if its object appears in the current frame, so this is not a tracker with arbitrary long-occlusion re-identification.

Each track averages its constituent region tokens and caches the associated frame indices. Video scene parsing compares the resulting track token with category text once, then propagates the assigned category to the contributing locations and frames. Last-occurrence localization additionally combines an object crop with a text query, reducing reliance on category names that cannot distinguish instances by themselves.

Section 4.3 defines the track score as the product of two cosine similarities:

\[ s_{\mathrm{track}}=s_{\mathrm{visual}}\times s_{\mathrm{textual}}. \]

\(s_{\mathrm{visual}}\) compares visual tokens from the query crop with visual track tokens, while \(s_{\mathrm{textual}}\) compares query text with text-aligned track tokens. After retaining tracks with scores above 0.18, the method returns the window of the last-ending track, not simply the highest-scoring track. This decision implements the task's requirement to identify the object's final appearance.

4. Joint region-text alignment: supervise the pooled representation with language directly

An MLP projects deduplicated region tokens into the text space for cosine matching with dino.txt text embeddings. The essential change is not merely adding a projection head: pooling and alignment are learned together. Visual contrastive learning groups predictions from the same region, while text contrastive learning associates region predictions with their category descriptions, making spatial aggregation compatible with semantic naming.

Distillation in both visual and text-aligned spaces anchors the new tokens to region targets from the frozen backbone. The text-aligned target is not just a category label: DINOv3 features are first average-pooled within a ground-truth mask and then processed by dino.txt's visual text-alignment block. Category text provides discriminative supervision, while backbone distillation anchors open-vocabulary transfer. These objectives have complementary roles.

A Worked Example

Consider a car, its wheel, and a wall, with a prompt on the tire. This illustrates the mechanism rather than adding an experimental result: the three queries can attend to the tire detail, wheel, and whole car, while a second point on the body can also produce a whole-car region. Visual deduplication merges repeated car representations but preserves a distinct wheel representation, so a text query for "wheel" need not recover local evidence from fragmented car patches.

In Visual Haystacks, the system first searches all region tokens across \(D\) images for an anchor object and selects the image containing the highest cosine-similarity match. It then checks for the target object only within that image. A maximum region-text similarity above 0.23 produces a yes answer; otherwise the answer is no. Image retrieval and subsequent object-presence reasoning are separate decisions, so retrieval Recall@1 cannot be substituted for final binary-question accuracy.

Loss & Training

Training combines five segmentation datasets: COCOStuff, OpenImagesV7, PhraseCut, Mapillary, and SA-1B. Points are sampled inside ground-truth masks with probability proportional to the square of the number of overlapping regions, emphasizing locations with part-whole structure. Each point has up to three target regions. Cosine distances between predicted and target visual region tokens form a cost matrix for one-to-one Hungarian assignment. Unmatched predictions are excluded from the loss when fewer than three targets exist, rather than forcing nonexistent entities into every slot.

The visual contrastive objective in Equation (1) treats predictions associated with the same mask as positive correspondences; Equation (2) applies bidirectional region-text contrastive learning. Equation (3) distills predictions toward the assigned visual and text-aligned targets. Equation (4) additionally applies BCE and DICE supervision to max-normalized attention masks; the authors report faster convergence without changing final performance. Equations (1) through (4) are corrupted in the cached text, so this note explains the verifiable objectives and assignments without inventing denominators, loss weights, or operators.

The cached main paper does not specify sufficient optimizer, learning-rate, epoch, or loss-weight details for reproduction. All downstream evaluations are described as zero-shot, but this refers to their evaluation setting: it does not imply an absence of segmentation supervision or exposure to related categories.

Key Experimental Results

Main Results

The values below come from Tables 2 and 3. Image segmentation uses a \(24\times24\) point grid and averages the three text-aligned tokens at each point. It constructs a \(24\times24\times C\) class-similarity map, upsamples it to \(384\times384\times C\), and selects the highest-scoring class per pixel. Token merging is disabled, leaving 576 point-level outputs. The ADE20K improvement and video compression ratios therefore do not describe the same experimental configuration.

Dataset and task	Metric	DINOv3 dino.txt	T-REN	Difference / T-REN compression
ADE20K, open-vocabulary segmentation, Table 2	mIoU	24.7	30.6	+5.9 points; merging disabled
Cityscapes, open-vocabulary segmentation, Table 2	mIoU	36.9	52.7	+15.8 points; merging disabled
Ego4D, last-occurrence localization, Table 3	Recall@1 (%)	36.8	52.4	+15.6 percentage points; 187.5-fold
Ego4D, last-occurrence localization, Table 3	tAP	14.4	26.4	+12.0 points; 187.5-fold
VSPW, video scene parsing, Table 3	mIoU	20.7	38.3	+17.6 points; 254.5-fold

Ego4D provides both text and object-crop queries. Videos average 140 seconds, are sampled at 5 FPS, and have target final-occurrence windows averaging 3 seconds. Localization is correct only when temporal IoU with the ground-truth window exceeds 0.25. Compression ratios measure token counts relative to patch representations, not runtime speedups. Table 2 also reports higher-resolution T-REN+ results of 32.0 mIoU on ADE20K and 58.7 on Cityscapes, which must not be mixed with the 384p base-model results.

Ablation Study

Table 4 independently controls whether region pooling and text alignment are trained. VH denotes Visual Haystacks; the table below retains retrieval Recall@1 at \(D=10\), not the separately reported final-question metric.

Train region pooling	Train text alignment	ADE20K mIoU	Cityscapes mIoU	VH retrieval, D=10 (%)
No	No	24.7	36.9	68.4
No	Yes	25.4	44.7	76.1
Yes	No	19.5	21.1	65.5
Yes	Yes	30.6	52.7	87.2

Text-alignment-only training supervises patches with region text. Pooling-only training instead applies the learned masks post hoc to dino.txt's text-aligned patch features. The latter falls below the untrained baseline on ADE20K, showing that correct pooling masks are insufficient: when the language-aligned patch features already cross object boundaries, post-hoc pooling still yields spatially misaligned semantic representations.

Table 5 separately tests whether compression damages VSPW representation quality. Its no-merging configuration already uses T-REN and is not the dino.txt baseline from the main comparison.

In-frame merging	Temporal merging	VSPW mIoU	Compression relative to patch baseline
No	No	38.6	1-fold
Yes	No	38.6	29.2-fold
Yes	Yes	38.3	254.5-fold

Key Findings

• Joint training adds 5.2 ADE20K mIoU over training text alignment alone; the full gain cannot be attributed entirely to language-supervision data.
• In-frame merging preserves VSPW mIoU, while adding temporal merging costs 0.3 points for greater compression. This is a small degradation, not lossless compression.
• Figure 3 reports an average retrieval Recall@1 gain of 18.4 percentage points across database sizes with average 24.4-fold token compression. Figure 5 supports multi-region queries through a single-token control; no additional precise values are inferred from the plotted curves.

Highlights & Insights

• Supervision can shape compressibility rather than leaving compression to heuristic deletion at inference. Training same-region predictions to be similar gives merging an explicit semantic basis.
• Multiple queries do more than increase the output budget. They preserve part-whole ambiguity at the representation stage and remove duplicates afterward, avoiding a hard one-point-one-entity restriction.
• Separating track tokens from lightweight spatiotemporal indices is useful for memory retrieval. High-dimensional semantics can be aggregated while coordinates and frame indices still recover the underlying evidence.

Limitations & Future Work

• The authors propose end-to-end region-model training as future work. Current results depend on a strong frozen backbone and do not establish equivalent performance when training such a structure from scratch.
• Compression mainly reduces cached and downstream representations, not the cost of extracting high-resolution backbone patches. An added 3.7% in parameters also does not imply only 3.7% extra end-to-end runtime.
• More than 4,600 training categories overlap with some of ADE20K's 150 classes. Only 5 categories are strictly unseen after excluding synonyms, while 13 others appear under synonym forms. Figure 6c provides limited transfer evidence, not evidence that every evaluation class was unseen.
• This note identifies fixed thresholds and active-track-only one-to-one matching as possible weaknesses under occlusion, appearance changes, and similar instances. The paper does not specifically quantify these failure modes; stratified occlusion-duration and instance-confusion evaluations would help.
• The paper does not add experiments on global tasks, instead arguing that their performance is preserved because the backbone's image-level representation is unchanged. This does not establish that substituting region tokens into an MLLM preserves all question-answering abilities.

Related Work & Insights

• vs REN: T-REN inherits point-query pooling but replaces a single token with multiple region candidates and adds joint text alignment. The full-model difference is not an isolated measure of the multi-token contribution; Figure 5 provides that controlled comparison.
• vs DINOv3 dino.txt: T-REN retains visual and textual knowledge while relearning region aggregation and projection. Table 4 shows that directly pooling existing text-aligned patches does not replace joint learning.
• vs SAM-guided segmentation: T-REN does not invoke SAM for mask refinement at inference, but its training data include SA-1B. The SAM-guided models in Table 2 also use different backbones and resolutions, so those conditions must accompany comparisons.

Rating

• Novelty: 4/5. Integrates multi-region queries, joint alignment, and temporal compression into one encoder, while inheriting the core pooling structure from REN.
• Experimental Thoroughness: 4/5. Covers images and videos with separate pooling, alignment, and merging ablations, but strictly unseen-category evaluation is small.
• Writing Quality: 4/5. Tasks and mechanisms are clear; corrupted cached equations and incomplete main-paper training settings limit reproduction checks.
• Value: 4/5. Directly relevant to dense visual retrieval and video-memory representations, without establishing a general MLLM acceleration result.