ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search¶

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Code: https://github.com/gritYCDA/ARGOS
Area: LLM Agents / Interactive Person Retrieval
Keywords: multi-camera person search, witness interaction, spatio-temporal topology graph, tool use, turn-weighted success

TL;DR¶

ARGOS turns person search into budgeted evidence gathering: a four-module agent asks questions and invokes camera-topology tools to isolate a target, with best spatial and temporal Turn-Weighted Success scores of 0.383 and 0.590 across a benchmark of 2,691 tasks.

Background & Motivation¶

Conventional person re-identification starts with an image of the target, while text-based retrieval typically assumes that a witness supplies a sufficiently detailed description in one attempt. Neither assumption fits an incomplete account such as a person wearing a dark top and a mask. Interactive retrieval can ask for additional appearance details, but a witness may remember a location or the order of sightings more clearly than hairstyle or clothing patterns.

Spatial and temporal evidence can change the candidate pool in ways that better appearance descriptions cannot. Several people may wear white, yet only some have trajectories compatible with being seen in a warehouse and almost simultaneously near a lobby. Prior spatio-temporal ReID already uses transition distributions; ARGOS does not invent that constraint. Its contribution is to put evidence acquisition, feasibility testing, and stopping decisions under agent control. A system that eventually identifies the target after repeatedly asking unanswerable questions should not receive the same credit as one that collects the right evidence quickly.

MTMMC provides synchronized trajectories and annotations with which to construct verifiable tasks. ARGOS computes the underlying answer and optimal disambiguation sequence before introducing natural-language descriptions, separating correctness from language generation. Core idea: jointly choose appearance questions, spatial questions, and temporal-tool calls under uncertainty about what the witness can answer, so that the correct identity is isolated with few interactions.

Method¶

Overall Architecture¶

The input is a vague initial witness statement and a compatible candidate set drawn from a structured person gallery. Each iteration analyzes candidate differences, plans an action, executes a witness query or environment tool, and interprets the response to update the candidates. The output is a predicted identity; an episode ends on prediction or exhaustion of its 20-turn budget.

The evaluation boundary matters. The baseline reads a database with 24 annotated appearance attributes rather than independently detecting and tracking every person in raw video. A deterministic environment supplies topology queries and exact filtering; the LLM handles action selection and language interpretation. Candidate-to-camera mappings, the ground-truth disambiguation path, per-candidate feasibility details, and the witness's answerable attribute set are hidden from the agent.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synchronized trajectories<br/>and attribute annotations"] --> B["Separated Ground Truth<br/>and Language"]
    B --> C["Spatio-Temporal<br/>Topology Constraints"]
    C --> D["Four-Module Decision Loop<br/>Analyze, plan, act, interpret"]
    E["Initial statement<br/>and candidates"] --> D
    D -->|Question or filter, then continue| D
    D -->|Prediction or budget exhaustion| F["Turn-Weighted Evaluation"]
    F --> G["Identity result<br/>and interaction efficiency"]

Key Designs¶

1. Separated Ground Truth and Language: establish solvability before testing comprehension and active disambiguation

The construction pipeline first uses algorithms, not an LLM, to derive task validity and optimal action sequences from the attribute database, topology, and trajectories. Who selects appearance clues using information-theoretic elimination; Where constructs ambiguity within camera zones; When derives temporal feasibility from observed transitions. Only afterward does an LLM convert structured clues into conversational descriptions. This preserves checkable answers while introducing synonyms, hedging, and references that require language understanding. The source gallery contains 1,273 annotated people, observed by 16 synchronized cameras across 14 scenarios in a factory and a university campus.

The three tracks deliberately differ in interaction. Who has 989 tasks and supplies an already completed dialogue averaging 5.6 exchanges; the evaluated model parses it and predicts once, rather than generating those exchanges. Where has 550 tasks, each requiring at least one spatial question to resolve a sub-area inside a multi-camera zone before further attribute filtering. When has 1,152 tasks involving two sightings and approximate times, and requires at least one temporal elimination beyond simply checking whether a person was present. These are complementary evaluations, not three equally difficult tasks whose scores can be ranked directly.

2. Spatio-Temporal Topology Constraints: eliminate candidates using directly observed camera transitions

The Spatio-Temporal Topology Graph, or STTG, is a directed weighted graph. Camera nodes carry zone labels and natural-language sub-area descriptions; edges record a transition type, minimum/median/maximum transition times, and observation count. OVERLAP edges connect shared fields of view and define atomic zones. SOFT_ADJ edges connect non-overlapping views separated by a door or passage with transit below two seconds. TRAVEL edges represent the remaining transitions. Construction extracts transitions from tracking annotations, assigns prioritized labels with human verification, and aggregates directed edges, producing 110 factory edges and 149 campus edges.

For temporal tasks, absence at a reported camera is filtered first. Remaining candidates are checked against the empirical transition range; the text explicitly describes rejection for reversed temporal order and excessively slow transitions. Task generation uses only camera pairs with directly observed STTG edges, excluding multi-hop route inference. The environment executes check_temporal and returns the remaining candidate set, so the agent is choosing when to access grounded evidence rather than guessing mobility constraints from general language knowledge. However, an observed minimum or maximum travel time remains an empirical range, not a universal physical law.

3. Four-Module Decision Loop: distinguish a discriminative question from an answerable question

The Analyst uses query_gallery to inspect candidate attributes, analyze_dist to obtain distributions and elimination power, and get_zone_info to inspect spatial sub-areas. The Planner combines this summary with the full dialogue history to choose an appearance question, a spatial question, or a temporal check. The Interviewer executes the choice through ask_witness or check_temporal. The Interpreter converts a natural-language reply into a canonical attribute value and applies filter_attr or filter_spatial; predict submits the final identity. Together these eight tools provide inspection, interaction, filtering, and termination without giving the LLM permission to modify the ground-truth gallery.

The central strategic difficulty is information asymmetry. Although the gallery exposes 24 attributes, the deterministic witness reliably answers only visual gender, upper clothing color, and lower clothing color. Other appearance questions produce uncertainty, and the agent is not told this observable set in advance. An attribute with excellent candidate-separation power, such as hairstyle, can therefore be useless to ask about. The Planner must adapt after unsuccessful questions, while the Interpreter must distinguish a hedged but informative answer from a genuine lack of knowledge. Mapping an ambiguous reply to the wrong canonical value can irreversibly filter out the target. Spatial and temporal replies are grounded in precomputed paths, with varied wording rather than stochastic witness memory.

4. Turn-Weighted Evaluation: reward correct identification relative to each task's own optimal budget

For Where and When, Turn-Weighted Success gives a failed task zero credit and discounts a successful task if it takes more turns than its precomputed oracle sequence. Completing a task faster than the oracle count cannot earn more than one unit of credit. Using the variables defined in the paper, the metric is:

\[ \mathrm{TWS}=\frac{1}{N}\sum_{i=1}^{N}s_i\frac{\tau_i^*}{\max(\tau_i,\tau_i^*)}. \]

Here \(s_i\) is the success indicator, \(\tau_i\) is the agent's turn count, and \(\tau_i^*\) is the oracle-optimal count computed during construction. The principle resembles SPL in embodied navigation, replacing path length with interaction turns. It is not equivalent to dividing aggregate accuracy by mean turns because the weighting depends on individual tasks and their outcomes. Top-1 accuracy, cumulative success within a five-turn budget, and average turns on successful tasks are reported alongside it to expose different failure modes.

A Worked Example¶

Figure 4 follows a difficult factory case with a white-top clue and sightings in a warehouse and almost simultaneously near a lobby entrance. The initial pool has 19 people. A temporal check reduces it to three, eliminating 16 candidates. A question about clothing color layout receives an uncertain answer and leaves all three. Asking about lower clothing color then yields black and isolates the target, followed by a correct prediction after three turns.

The example shows both the information value of the tool and a remaining wasted question. Without the temporal tool, the comparison agent repeatedly asks about unobservable attributes such as age, bags, body shape, and hair color, eventually exhausting 20 turns. The progression 19 to 3 to 3 to 1 is a reported case study, not an average reduction rate across the benchmark.

Loss & Training¶

The paper introduces no new learned retrieval loss and reports no fine-tuning or reinforcement-learning procedure for these modules. It evaluates existing LLM backbones with temperature 0.0, seed 42, and a maximum of 20 turns. The contribution lies in the task, environment, tools, modular decision process, and evaluation protocol rather than a new visual encoder.

One implementation detail requires careful reading: the method describes autonomous temporal-tool scheduling, but Table 6 labels 80.6% Top-1 and 0.567 TWS as Temporal-first, while its autonomous variant obtains 83.9% and 0.572. The former pair also appears in the GPT-4o main results. The tables below preserve these distinctions rather than attributing the main-table numbers unambiguously to autonomous scheduling.

Key Experimental Results¶

Main Results¶

The following values are from Table 5. Top-1 is a percentage; average turns include only successful tasks. Higher TWS and accuracy are better, while fewer turns are better. Different oracle budgets and task constructions prevent a direct difficulty comparison between tracks.

Backbone	Where TWS	Where Top-1	Where avg. turns	When TWS	When Top-1	When avg. turns
Oracle	1.000	100.0	2.05	1.000	100.0	1.88
GPT-5.2	0.338	73.1	7.04	0.590	88.2	3.91
GPT-4o	0.323	74.5	7.49	0.567	80.6	3.91
GPT-5-mini	0.319	74.9	7.48	0.556	88.0	4.40
Claude Sonnet 4	0.383	76.0	6.71	0.548	83.6	4.25

On Who with GPT-4o, tool calling achieves 81.1% SR@1, versus 73.3% for direct ID prediction and 32.2% for the rule-based method. Direct prediction actually has higher parsing accuracy than tool calling, 93.0% versus 90.8%, despite worse retrieval. Attribute extraction accuracy alone therefore does not measure the quality of downstream candidate operations and identity selection.

Ablation Study¶

These GPT-4o component ablations come from Table 7. A dash indicates an inapplicable or unreported entry, not a zero score.

Configuration	Where TWS	Where Top-1 (%)	Where avg. turns	When TWS	When Top-1 (%)	When avg. turns
Full ARGOS	0.323	74.5	7.49	0.567	80.6	3.91
Without strategy	0.136	47.6	10.78	0.373	76.9	7.73
Without spatial tool	0.063	40.7	14.20	-	-	-
Without temporal tool	-	-	-	0.054	31.0	13.44
Single-pass, full dialogue	-	56.2	-	-	29.6	-
Single-pass, initial statement	-	64.7	-	-	11.3	-

Table 6 separately compares temporal-tool scheduling. Temporal-first obtains 80.6% Top-1, 0.567 TWS, and 3.91 average turns; autonomous scheduling obtains 83.9%, 0.572, and 4.05. Autonomous scheduling is not faster on the average-turn statistic, but its combined outcome yields slightly higher TWS. The experiment supports the informational contribution of temporal filtering rather than a requirement to invoke it at a fixed first step.

Key Findings¶

Removing the temporal tool lowers When Top-1 from 80.6% to 31.0%, a 49.6-percentage-point drop. Removing the spatial tool costs 33.8 points on Where. This demonstrates the value of structured evidence in the supplied environment, not an ability to reconstruct that evidence from raw video.
Without strategy, When accuracy falls by only 3.7 points, but successful episodes increase from 3.91 to 7.73 turns and TWS decreases from 0.567 to 0.373. The abstract's broad statement that strategy removal halves TWS should not replace these exact table values or be presented as a literal 50% reduction here.
On Where, giving the single-pass baseline the full dialogue lowers accuracy from 64.7% to 56.2%. More text is not automatically more usable evidence. The reported parsing success rates, 78.6% for Where and 94.2% for When, are consistent with a spatial-language bottleneck but do not establish that parsing explains every error.

Highlights & Insights¶

Separating algorithmic truth construction from linguistic presentation makes the benchmark auditable without removing language understanding from the task. Optimal action sequences remain available even when the surface descriptions are conversational.
A topology prior becomes evidence the agent can choose to acquire, not merely a fixed post-processing rule. The worked example shows how a temporal check can change which appearance question is useful next.
Candidate discrimination and witness answerability are different quantities. A planner that only examines database attribute distributions can spend its budget on highly discriminative questions that the witness cannot answer.

Limitations & Future Work¶

The authors acknowledge that the deterministic witness does not capture memory errors, contradictions, or misinformation. Only three appearance attributes are reliably answerable, so performance cannot be transferred directly to unconstrained human dialogue.
Coverage is limited to a factory and a campus. STTG construction assumes reliable trajectories and directly observed transitions; city-scale sparsity, missing detections, multi-hop movement, and out-of-distribution travel times remain insufficiently tested.
Task generation and the temporal tool use the same structured topology. Strong performance most directly demonstrates planning within that supplied world, not end-to-end robustness to visual recognition or topology-estimation errors.
The available cache contains the main paper and references but not the cited Appendices A, B, and C. Detailed multi-seed results, paraphrase experiments, and prompt templates cannot be fully checked here. Oracle average turns also differ slightly between benchmark statistics and the results table; this note retains the latter's 2.05 and 1.88 without inventing an explanation.
The authors explicitly recognize privacy and surveillance misuse risks and state that the benchmark uses consented public data. Real deployment requires lawful authorization, purpose restrictions, and human review of false matches; benchmark performance does not constitute deployment approval.

Compared with text-based and interactive ReID: CLIP-style alignment and LLaVA-ReID mainly address appearance descriptions and cross-modal matching. ARGOS adds spatial and temporal evidence acquisition, but relies on a structured attribute-and-trajectory environment rather than improving the visual representation itself.
Compared with spatio-temporal ReID: transition-time constraints are established prior work. ARGOS makes their invocation and associated questioning part of an interactive policy, extending the task and decision protocol rather than claiming that temporal elimination is new.
Compared with ChatIR, PlugIR, and VideoAgent: these systems refine retrieval through dialogue or use tools to locate video evidence. ARGOS focuses on eliminating candidates until a unique identity remains across cameras. Its transferable evaluation lesson is to report correctness, interaction cost, and failure types together.

Rating¶

Novelty: 4/5. The controlled combination of witness dialogue, camera topology, and tool-based decisions is useful, while the underlying ingredients have substantial prior work.
Experimental Thoroughness: 4/5. Three tracks, four backbones, and targeted tool and strategy ablations provide meaningful diagnostics; human noise, broader environments, and end-to-end visual errors remain open.
Writing Quality: 4/5. The task boundary and ablation rationale are clear, but scheduling descriptions, oracle turn statistics, and the abstract's strategy-effect wording require cross-checking against tables.
Value: 4/5. A useful testbed for active information gathering with verifiable tools and efficiency-aware evaluation, not evidence of a deployment-ready surveillance system.