ECCV 2026 VLM Reasoning referring multi-object tracking multimodal large language model (MLLM) discriminative paradigm Yes/No verification temporal consistency

YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Reasoning¶

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/ggbondrighthere24/YesTrack
Area: VLM Reasoning
Keywords: referring multi-object tracking, multimodal large language model (MLLM), discriminative paradigm, Yes/No verification, temporal consistency

TL;DR¶

YesTrack does not ask the MLLM to generate target descriptions; it directly reads the logits of the Yes/No decision tokens in the vocabulary for discriminative referring verification, adds two inference-time constraints—Temporal Confidence Prior (TCP) and Temporal Reference Propagation (TRP)—and achieves SOTA with the smallest Qwen3-VL-2B-Instruct: HOTA 54.00 on Refer-KITTI and 43.75 on Refer-KITTI-V2, while the same paradigm transfers into a generic MOT tracker, YesTrack-MOT (KITTI HOTA 45.10).

Background & Motivation¶

Referring multi-object tracking (RMOT) requires continuously localizing and associating every instance in a video that matches a given natural-language expression: it must maintain identities across frames like generic multi-object tracking (MOT) while aligning trajectories with language. End-to-end methods (TempRMOT, DKGTrack, HFF-Tracker, among others) model cross-modal interactions in a unified framework; they perform well but rely on task-specific modules and train on a limited vocabulary, which can introduce dataset-dependent biases and generalize poorly to more complex real-world expressions. Two-stage methods (iKUN, ReferGPT) first run generic tracking and then filter trajectories by language. ReferGPT introduces an MLLM to generate captions aligned with the expression and matches them with a text-similarity module. This generative route improves robustness to diverse expressions, but autoregressive token-by-token decoding adds substantial latency, and converting text into referring decisions requires an extra similarity module—so architectural complexity and inference latency grow together, conflicting with the real-time demands of tracking.

A deeper tension underlies this: large models already possess discriminative potential—Reflexion and related work have used explicit Yes/No judgments for evaluation, verification, and iterative control—yet these usages take only the discrete Yes/No text output, treating the model as a hard binary classifier and discarding the continuous confidence contained in the vocabulary logits. Without continuous confidence, one can neither route "which candidates deserve more compute" nor apply soft temporal corrections.

YesTrack therefore reformulates referring as a binary image-text matching problem: the crop of a candidate trajectory, its normalized box coordinates, and the expression are fed to the MLLM, which answers only Yes or No; the system applies a softmax over the logits of these two decision tokens to obtain continuous matching confidence, at one forward pass per candidate, bypassing autoregressive decoding and post-hoc text matching entirely. Core idea: turn the MLLM from a "caption generator" into a "discriminative referring head" that outputs matching confidence directly from the Yes/No token logits, then stabilize framewise decisions and cut redundant verification with two lightweight inference-time constraints, TCP and TRP.

Method¶

Overall Architecture¶

YesTrack is a two-stage RMOT framework: an off-the-shelf MOT tracker produces candidates frame by frame (crops, identity IDs, normalized box coordinates), and an MLLM-based discriminative verifier decides whether each candidate matches the expression. Inference starts with Temporal Reference Propagation (TRP): only key frames trigger MLLM verification, while non-key frames directly inherit the most recent referring score. At key frames, Stage 1 scores each candidate in frame mode; after Temporal Confidence Prior (TCP) adjustment, candidates are routed by confidence—high-confidence candidates are kept or discarded directly, and only ambiguous candidates enter Stage 2 video mode, where a memory bank retrieves the past K frames of crops and scores for a joint decision under a final threshold γ. Both temporal components in the diagram are inference-time constraints; they do not modify the model architecture and barely touch the tracker itself.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    V["Video + referring expression"] --> T["Off-the-shelf MOT tracker<br/>Candidate crops, IDs, coords"]
    T --> TRP["Temporal Reference Propagation TRP<br/>Verify only at key frames"]
    TRP -->|Non-key frames<br/>Inherit referring score| O
    TRP -->|Key frames| S1["MLLM Yes/No verifier<br/>Stage 1 frame-mode scoring"]
    S1 --> TCP["Temporal Confidence Prior TCP<br/>Adjust matching probability"]
    TCP --> R{"Confidence routing"}
    R -->|"p≥0.8 keep<br/>p≤0.2 discard"| O
    R -->|Ambiguous candidates| S2["Stage 2 video mode<br/>Memory bank, past K frames"]
    S2 -->|"Threshold γ"| O["Output referring score"]

Key Designs¶

1. Discriminative Yes/No verifier: switch the MLLM from "generating descriptions" to "reading two tokens' logits"

The pain point of the generative route is slow token-by-token decoding and uncontrolled output wording. YesTrack constructs a unified multimodal input for each candidate: the cropped region, normalized coordinates embedded into the textual prompt (providing explicit location cues), and the referring expression, with explicit instructions to answer only Yes or No. This closed-set setup shrinks the decision space to two semantic options and sharpens the decision boundaries in the output logits. The key mechanism is to skip autoregressive decoding entirely and extract the logits of the two decision tokens directly, then apply a softmax:

\[ p_i=\frac{\exp(\ell^{\text{yes}}_i)}{\exp(\ell^{\text{yes}}_i)+\exp(\ell^{\text{no}}_i)}. \]

Compared with generative decoding, this design yields three things at once: decisions that are immune to unexpected wording and explanations, a continuous confidence that feeds routing and temporal refinement directly, and the cost of a single forward pass per candidate instead of iterative token sampling. Frame mode and video mode share the same binary format, differing only in prompt template and number of input images (templates are in the supplementary material).

2. Two-stage verification with confidence routing: pay for video-level verification only on ambiguous candidates

Stage 1 performs single-frame pairwise verification between candidate \(i\) and the expression, yielding confidence \(p_i\); after TCP adjustment, candidates are routed by interval: \(p_i \ge p_h\) is kept directly and \(p_i \le p_l\) discarded (with \(p_l=0.2\) and \(p_h=0.8\) in the experiments), while only the ambiguous band \(p_l < p_i < p_h\) proceeds to Stage 2. Stage 2 video mode retrieves this identity's past K frames of crops and referring scores from the memory bank, lets the MLLM decide jointly with the temporal context, and applies a decision threshold γ of 0.4. The rationale is direct: single-frame evidence is unreliable under occlusion or for small objects, but running video-level verification on every candidate is too costly; routing concentrates expensive temporal verification on the few candidates that are genuinely uncertain. This also explains why continuous logits rather than a discrete text answer are essential—without continuous confidence there is no "ambiguous band" to define.

3. Temporal consistency constraints: TCP stabilizes framewise decisions, TRP cuts redundant calls

Both constraints act only at inference time, without architectural changes, and transfer readily to other two-stage RMOT frameworks. TRP exploits the asymmetry that "referring decisions are more temporally stable than tracking states": trajectories must be updated every frame, but an identity's semantic relevance to a query rarely changes within a short window. Referring is therefore triggered only at a fixed interval Δ, when a new identity appears, or when the expression changes; non-key frames run only tracking and inherit the referring score from the most recent key frame. TCP exploits identity-level historical confidence: if an identity has been confidently matched for all of its recent K frames, the current prediction receives a temporal prior boost and is clipped to 1 (this formula is reconstructed from the paper's context; refer to the original):

\[ \tilde{p}_i^{(t)}=\min\bigl\{1,\;p_i^{(t)}+\lambda\cdot\mathbb{I}\{\min\nolimits_{k\in[t-K,\,t-1]}p_i^{(k)}\ge\alpha\}\bigr\}. \]

The outer min keeps the result a valid probability; the inner min ensures the prior applies only when every one of the last K frames was confidently matched, preventing one-off mismatches from being cemented. The experiments use K=3, α=0.4, and λ=0.3.

4. YesTrack-MOT: moving the same discriminative paradigm into generic MOT data association

The tracking backbone can be any off-the-shelf tracker; the paper further instantiates a deliberately simple tracker, YesTrack-MOT, by replacing embedding-based ReID with an MLLM-based pairwise identity verifier. The mechanism is identical to referring verification; the only difference is the input modality, which changes from image-text to image-image (the crop of a track from the previous frame against the crop of a current detection), again reading Yes/No logits directly for the verification probability. The pipeline has three steps: a distance gate first reduces candidate pairs—combinations whose box-center displacement exceeds a threshold δ are not verified (200 pixels in the experiments); gated pairs go through MLLM verification, the cost matrix is \(c_{jk}=1-p_{jk}\), and the Hungarian algorithm solves the one-to-one assignment; unmatched tracks are marked lost and kept for 10 frames, during which unmatched detections may re-associate with lost tracks once, and tracks are removed after exceeding the maximum lost age. The pipeline deliberately avoids sophisticated motion models, relying on the trio of gating, MLLM verification, and Hungarian assignment to achieve strong performance.

Loss & Training¶

Each input pair \((I_i, E)\) carries a binary label \(y_i\in\{0,1\}\), and a standard binary cross-entropy loss is computed from the matching probability \(p_i\) obtained by softmax over the two decision tokens; the softmax normalization is restricted to these two tokens:

\[ \mathcal{L}_i=-\left[\,y_i\log p_i+\left(1-y_i\right)\log\left(1-p_i\right)\right]. \]

The MLLM backbone is Qwen3-VL-2B-Instruct, with candidate crops resized to 320×320 before the vision encoder. Inference hyperparameters: confidence interval \([p_l,p_h]=[0.2,0.8]\), decision threshold γ=0.4, TCP with K=3/α=0.4/λ=0.3, TRP interval Δ=5 (Refer-KITTI) and 10 (Refer-KITTI-V2), and a memory bank of size 4; YesTrack-MOT uses a distance threshold of 200 pixels and keeps lost tracks for 10 frames. All experiments run on two RTX 4090 GPUs (24GB). The prompt templates and training/inference time comparisons are in the supplementary material, so the main text alone does not allow independent reproduction of the training details.

Key Experimental Results¶

Main Results¶

Refer-KITTI is built from 18 KITTI tracking sequences with 818 expressions (only 215 distinct, a vocabulary of 49 words); expressions are short, repetitive, and structurally simple. Refer-KITTI-V2 expands to all 21 sequences with 9,758 expressions (7,193 distinct, vocabulary of 617 words), covering appearance, spatial relations, motion, and inter-object interactions, while adding ambiguous expressions and no-target cases—a far larger linguistic space. TempRMOT* is a pure MOT model obtained by removing all language inputs and multimodal fusion from TempRMOT; using it as the backbone aligns the tracker and isolates the contribution of the referring strategy.

Method (Refer-KITTI)	Tracker	HOTA	DetA	AssA
DKGTrack (end-to-end)	–	52.08	41.10	66.04
TempRMOT (end-to-end)	–	51.15	39.99	65.61
iKUN (two-stage)	NeuralSORT	44.56	32.05	62.48
ReferGPT(O) (two-stage)	PC3T	46.36	36.58	59.00
YesTrack	TempRMOT*	54.00	43.91	66.57
YesTrack	YesTrack-MOT	52.96	46.84	60.03

Method (Refer-KITTI-V2)	Tracker	HOTA	DetA	AssA
TransRMOT (end-to-end)	–	31.00	19.40	49.68
TempRMOT (end-to-end)	–	34.72	22.52	53.64
DKGTrack (end-to-end)	–	35.26	23.04	54.13
iKUN (two-stage)	NeuralSORT	10.32	2.17	49.77
ReferGPT (two-stage)	PC3T	30.12	15.69	59.02
YesTrack	TempRMOT*	41.78	32.69	53.75
YesTrack	YesTrack-MOT	43.75	37.04	52.36

On Refer-KITTI, YesTrack with the TempRMOT* backbone leads on both HOTA 54.00 and AssA 66.57, surpassing the strongest end-to-end method DKGTrack by 1.92; with the YesTrack-MOT backbone it achieves the best DetA 46.84 and the best DetRe 62.72. On the harder V2, the YesTrack-MOT variant attains the best HOTA 43.75 and DetA 37.04 and also leads on DetRe 48.78 and AssPr 83.95, exceeding DKGTrack by 8.49; iKUN drops to 10.32 HOTA on V2, indicating that its referring module becomes clearly unstable in the larger linguistic space.

Ablation Study¶

Frame mode	Video mode	TCP	TRP	HOTA	DetA	AssA	Inference time
✓				52.64	42.68	65.06	40mins
	✓			53.49	43.29	66.24	2h15mins
✓	✓			53.47	43.70	65.60	1h32mins
✓	✓	✓		54.00	43.91	66.57	1h34mins
✓	✓		✓	53.54	43.69	65.88	22mins
✓	✓	✓	✓	53.66	43.27	66.71	25mins

TCP and TRP pay off in different directions. TCP alone pushes HOTA to the best 54.00 with DetA 43.91, showing that historical confidence genuinely stabilizes trajectory selection, but inference takes 1h34mins; TRP alone runs in 22mins, the fastest in the table, with HOTA 53.54 essentially intact; combining both yields the best AssA 66.71 at only 3mins over TRP alone—the accuracy-efficiency trade-off is explicitly split into two independently switchable components.
Referring capability decouples from the underlying tracker (Table 4 of the paper). With ground-truth tracklets as candidates, YesTrack reaches referential Acc 91.15 / Prec 74.14 / Recall 85.71, consistently above iKUN's 84.62 / 59.88 / 70.24; on the TempRMOT* backbone HOTA is 54.00 versus iKUN's 42.95. The no-referring baseline always has the highest AssA, confirming that referring methods only filter existing trajectories by language rather than improving association itself, so AssA is naturally capped by the underlying tracker.
Generic MOT validation (Table 5 of the paper, unified RF-DETR detections). YesTrack-MOT reaches HOTA 45.10 / DetA 40.96 on KITTI, above OC-SORT's 41.34 and other baselines, but its AssA 51.23 is below ByteTrack's 52.30, and DetPr 64.52 is far below BoT-SORT's 76.94—the gains come mainly from detection-related metrics; pure association quality is not the strongest.
Robust to noisy expressions. In qualitative comparisons, faced with misspellings ("blakc car in the left") and colloquial redundancy ("emmm.... black car is in the left side of the road"), DKGTrack and ReferGPT tend to lose the target, while YesTrack still locks onto it stably.

Highlights & Insights¶

Reading logits instead of generated text: this advances the MLLM's discriminative potential from discrete Yes/No text to continuous confidence, obtaining decisions and uncertainty in a single forward pass—the precondition that makes routing and temporal refinement possible. Any per-candidate verification task (ReID, RAG reranking, agent tool-result auditing) can reuse this trick.
TRP captures the asymmetry that semantic judgments can be sparsely sampled: geometric association must be updated every frame, but an identity's relevance to a query barely changes within a short window, so MLLM invocations drop to on the order of 1/Δ with almost no accuracy loss.
The same verification mechanism covers a second task by swapping the input modality: referring verifies image-text, YesTrack-MOT verifies image-image, with the identical BCE objective and probability extraction—evidence that "MLLM as verifier" is a composable, general component rather than an RMOT-specific trick.

Limitations & Future Work¶

Admitted by the authors: the two-stage design is constrained by the quality of the off-the-shelf tracker; under TRP, tracking or verification errors at a key frame propagate until the next re-evaluation, which can cause temporary target loss or distractor tracking; the heuristic TCP will be replaced by a learnable module, and a more tightly coupled framework is planned.
Spotted independently: all experiments use KITTI-family data (the car and person classes), so cross-dataset generalization is unverified; the referential metrics use ground-truth tracklets as candidates, which isolates tracking variance but does not reflect error coupling between the two stages in a real pipeline; even with TRP, the full benchmark still takes 22–25mins, leaving a gap to real-time driving scenarios.
Concrete improvements: adaptively shorten Δ when key-frame confidence is low to suppress error propagation; distill the MLLM's discriminative signal into a lightweight matching head to cut latency while keeping robustness.

vs ReferGPT: both are two-stage with an MLLM, but ReferGPT generates captions first and matches them by text similarity, while YesTrack reads the Yes/No logits directly. With the same Qwen3-VL-2B (ReferGPT(Q), PC3T backbone), HOTA is 32.56 versus YesTrack's 54.00—but the two use different underlying trackers, so this comparison contains both the paradigm and the backbone factors.
vs iKUN: iKUN adds a Knowledge Unification Module to align visual-language features and a Neural Kalman Filter for association; YesTrack does not retrain the tracker and achieves higher referential metrics across all four backbones, yet the common two-stage weakness on V2's large vocabulary (iKUN at 10.32) shows that language generalization still rests mainly on the MLLM itself.
vs end-to-end TempRMOT / DKGTrack: end-to-end methods train on a restricted vocabulary with task-specific modules; YesTrack delegates language understanding to the MLLM's instruction-following ability and is more robust to noisy expressions, at the cost of an AssA capped by the underlying tracker.
vs prior discriminative uses of LLMs (Reflexion, etc.): they elicit discrete Yes/No text, essentially treating the model as a hard binary classifier; YesTrack goes further by exploiting the continuous confidence in token-level logits for routing and temporal refinement—the key difference between "usable" and "effective."
Resources: The code repository comes from the paper's abstract; its online availability has not been independently verified.

Rating¶

Novelty: 4/5. Discriminative Yes/No usage is not new in itself, but the combination of decision-token logits, continuous-confidence routing, and inference-time temporal constraints is clean and effective, and the paradigm is shown to transfer to generic MOT.
Experimental Thoroughness: 4/5. Two benchmarks, mode and temporal-component ablations, cross-experiments over four trackers, and a generic MOT comparison are all present; cross-dataset validation and multi-seed statistics are missing.
Writing Quality: 4/5. Motivation and pipeline are clear with restrained formulas; prompt templates and training details are relegated to the supplementary material, leaving the main text insufficient for independent reproduction.
Value: 4/5. Provides a low-cost paradigm for deploying MLLMs discriminatively; the two-stage ceiling remains tied to the underlying tracker, and efficiency still falls short of real time.