UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: Human preference alignment, urban perception, concept bottleneck, multi-agent scoring, locally weighted ridge regression

TL;DR¶

UrbanAlign uses a frozen vision-language model to extract interpretable urban concepts and calibrates them against similar human-labeled image pairs, achieving 70.8% average accuracy across six Place Pulse 2.0 categories, 14.1 percentage points above zero-shot GPT-4o.

Background & Motivation¶

Judging whether a street looks safe, wealthy, or beautiful is different from recognizing buildings, trees, and vehicles. A vision-language model (VLM) can often describe these elements without knowing how local people combine them into a preference judgment. For example, recognizing abundant vegetation is a perception problem, whereas deciding whether it compensates for poor lighting requires context-dependent evaluation weights. Asking directly which street view is better compresses observation, weighting, and discrete decision-making into one generation, bringing position effects, anchoring, and inconsistent judgments into the label. Existing Siamese networks and regression models can learn from human comparisons, but often lack intermediate concepts that people can inspect. Fine-tuning and RLHF instead modify model parameters, requiring additional training without directly making the basis of each judgment auditable.

This paper retains the VLM's perceptual knowledge and replaces the final decision process that converts observations into human preference labels. The challenge is that the concepts cannot simply be an arbitrary list of cleanliness, greenery, and lighting: they must be observable, scorable, and useful for explaining human comparisons in the target category. Furthermore, the role of a concept can vary with the street context, so one globally fixed set of weights may conceal important differences. The authors therefore draw on concept bottleneck models, using semantic dimensions as intermediate representations and learning decision weights within neighborhoods of similar image pairs. Here, frozen means that VLM weights remain unchanged, not that human data are unnecessary: the reference set supplies calibration supervision, and the validation set supports concept and hyperparameter selection.

The purpose of the multi-agent chain is also not simply to vote for a more accurate winner, but to produce continuous concept scores better suited to subsequent regression. This distinction explains how a scoring chain with slightly worse raw judgments can become the strongest method after calibration. Core Idea: use the frozen VLM as an interpretable concept measurement system, retrieve similar comparisons through hybrid visual-semantic features, and calibrate those measurements to human preferences using local semantic weights.

Method¶

Overall Architecture¶

The input is a pair of street-view images and a perception category; the output is a left, right, or equal preference label, not a measurement of objective urban safety. The pipeline has three designs: Concept Mining, Multi-Agent Structured Scoring, and Local Manifold Calibration. During preparation, reliable examples are extracted from reference data to define scorable dimensions and build reference image pairs with human rating differences. At inference time, the selected dimensions are reused to score a new pair, retrieve neighbors, and solve a regression specific to that query. Concept and parameter search uses validation results only, while the test set is reserved for evaluation; this is not joint training of the three stages through back-propagation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Reference["Reference comparisons<br/>and TrueSkill ratings"] --> Mining["Concept Mining"]
    Mining --> Scoring["Multi-Agent<br/>Structured Scoring"]
    Images["Image pair and<br/>perception category"] --> Scoring
    Images --> CLIP["Frozen CLIP encoding"]
    Scoring --> Calibration["Local Manifold<br/>Calibration"]
    CLIP --> Calibration
    Reference -->|Human rating difference supervision| Calibration
    Calibration --> Output["Preference label and<br/>local concept contributions"]
    Calibration -.->|Validation feedback only| Mining

Key Designs¶

1. Concept Mining: derive measurable evaluation dimensions from reliable human comparisons

The authors first use TrueSkill to convert discrete pairwise comparisons into continuous per-image ratings while retaining rating uncertainty. High-rated examples have means above the 75th percentile, low-rated examples fall below the 25th percentile, and both groups must have uncertainty below the median. The examples therefore represent relatively confident extremes rather than arbitrary extreme images, reducing chance effects from sparse votes. The default uses 5 images per group, 10 in total; the prompt supplies these examples, their TrueSkill ratings, and CLIP embeddings compressed to 8 dimensions with PCA. The VLM returns 5–10 dimensions in JSON, each with a name, description, and visual indicators for high and low scores. For perceived wealth, dimensions such as facade quality, vegetation maintenance, pavement integrity, and vehicle quality turn an abstract category into observable street-view evidence. These dimensions later receive continuous scores from 1–10, so they need both semantic meaning and numerical usefulness rather than generic aesthetic descriptions.

A single generated concept set need not be optimal, so the authors also design a validation search conducted independently for each category. The exploration phase uses higher generation temperatures to try different dimension combinations; the convergence phase retains the current stronger set and replaces only 1–2 targeted dimensions. Each candidate passes through scoring and calibration and is evaluated by validation accuracy, not merely by whether its names sound plausible. Different categories may peak at different trials, and the final collection assembles each category's best set. Thus, end-to-end optimization here means an external search loop, not gradient updates to the frozen VLM. The search mechanism must also be distinguished from the main result: 70.8% uses the initial dimensions with optimized calibration hyperparameters, whereas the dimension-search variant reports 69.3%.

2. Multi-Agent Structured Scoring: replace immediate winner selection with continuous evidence extraction

The default backbone is GPT-4o, executing Observer, Debater, and Judge sequentially, with three VLM calls per image pair. The Observer only describes visible evidence for each dimension, avoiding an early winner decision followed by selective justification. The Debater provides arguments for both high and low scores on every dimension rather than acting as another voter. The Judge integrates the descriptions and opposing arguments to assign semantic dimension scores to both images. This produces two comparable sets of concept measurements rather than discrete answers suitable only for vote counting. Pairwise context gives the scenes a shared comparison frame and lets the debate focus on their actual differences.

The authors interpret this design as reducing dimension-score variance, not as guaranteeing that its raw winner predictions always beat single-shot prompting. That interpretation is consistent with the reported results: multi-agent raw accuracy is 60.9%, slightly below pairwise single-shot scoring at 61.7%, but its calibrated result is stronger. For downstream regression, stable input features can be more valuable than a correct isolated vote because the regressor must learn how concept differences relate to human rating differences. The paper also sums absolute differences across dimensions as comparison intensity, allowing low-intensity pairs to be labeled equal during raw scoring. This differs from the later statistical re-inference step; the raw intensity threshold is not the complete decision rule of the calibrated model. The proof of a three-role variance-reduction bound is deferred to supplementary material; because roles share a backbone and depend on preceding outputs, independent-estimator assumptions should not be treated as empirically established.

3. Local Manifold Calibration: retrieve with hybrid features but explain preferences through semantic differences

For each image pair, the system computes CLIP embeddings and concept scores and forms directed differences for both representations. CLIP uses 768-dimensional ViT-L/14 features to provide general visual similarity, while semantic scores supply category-relevant differences. Equation (4) constructs the following hybrid difference, where the bars denote the source's scaling operations and \(\alpha\) controls the relative contribution of the two components.

\[ \Delta_{\mathrm{hybrid}}(A,B)=\bigl[\alpha\,\overline{\Delta}_{\mathrm{CLIP}}(A,B),\;(1-\alpha)\,\overline{\Delta}_{\mathrm{Sem}}(A,B)\bigr]. \]

CLIP embeddings are L2-normalized, and semantic differences are divided by 10 to prevent feature scales from dominating neighbor selection. The paper describes the latter as normalization to \([0,1]\), but dividing the directed difference \(S(A)-S(B)\) by 10 can still produce negative values; the operation is retained here without repeating that interval claim. Each reference pair is also mirrored by swapping the images and negating the human rating difference, keeping comparison direction consistent with the supervision sign. For a query pair, cosine similarity in the hybrid space retrieves \(K\) reference neighbors, with \(K=20\) by default. Similarities are converted to weights through an exponential kernel, giving more similar neighbors greater influence on the local fit; the default kernel temperature is 1.0. Crucially, hybrid features determine which examples supply relevant experience, but are not all passed to the final regression.

The local design matrix \(X\) stacks only neighboring semantic differences, \(y\) contains their TrueSkill rating differences, and \(W\) is the diagonal matrix of kernel weights. Ridge regularization controls unstable weights in small neighborhoods; the clearly readable closed-form solution beside Equation (5) is:

\[ \hat{\mathbf w}=(X^{\top}WX+\lambda I)^{-1}X^{\top}Wy. \]

The query's semantic difference is then dotted with these local weights to obtain its calibrated preference difference. A dimension can have different weights in different query neighborhoods, avoiding the assumption that cleanliness, modernity, or greenery matters equally in every street comparison. Multiplying a local dimension weight by its corresponding score difference gives a contribution explanation for that prediction, not a causal effect on real social outcomes. The paper additionally performs statistical re-inference using a predicted-difference threshold and the proportion of equal labels among neighbors, with default thresholds of 0.8 and 0.6. The logical connector in cached Equation (6) is damaged, preventing confirmation of the precise Boolean combination, so no apparently exact piecewise decision rule is reconstructed here. Likewise, the objective in Equation (5) is corrupted while its closed-form solution is intact; this note retains only the expression directly verifiable from the source.

A Worked Example¶

The beautiful-category example in Figure 3 on page 12 shows calibration reversing a raw decision. Image A has a raw concept-score total of 36 and image B has 41, so direct summation would choose B. The local model does not simply add a uniform offset to the totals; it reweights concept differences using human comparisons in the neighborhood. The figure reports a calibrated difference of +10.0, changing the prediction to A and matching the human winner. This illustrates how some concept differences can offset others, but the example's local weights should not become fixed rules for all cities.

Loss & Training¶

Supervision consists of TrueSkill rating differences from the reference set, and the fitted parameters are query-neighborhood ridge weights; the VLM and CLIP backbone weights remain frozen. The default Observer, Debater, and Judge temperatures are 0.3, 0.5, and 0.1, respectively, while single-shot modes use 0.0. Default calibration uses \(K=20\), \(\lambda=1.0\), and \(\alpha=0.3\), but these are not the final settings for every category in Table 1. The main results randomly sample 1,000 configurations from a 50,000-configuration grid and select parameters per category on validation data; optimal \(K\) ranges over 10–50 and \(\alpha\) over 0.2–0.7. Concept search allows up to 15 trials and stops after 5 non-improving trials; the reported search completed 12/15 trials. Deployment still requires multiple VLM inferences, reference retrieval, and local solves, so no fine-tuning does not mean no adaptation cost.

Key Experimental Results¶

Main Results¶

Place Pulse 2.0 contains 110,688 street-view images and 1.17M comparisons across six perception categories: safety, beautiful, lively, wealthy, boring, and depressing. The main experiment selects comparisons with at least 2 independent votes, samples 449 pairs per category, and splits them into reference, validation, and test sets at 60%/20%/20%, with random seed 42. After removing human equal labels, each category has 75–78 test pairs; a model prediction of equal still counts as an error. The following excerpt from Table 1 on page 11 reports excl-equal accuracy; averages come from Section 4.2, and \(\kappa\) is Cohen's kappa, where higher values indicate greater agreement.

Perception category	GPT-4o zero-shot accuracy (%)	Same-semantic-feature Global Ridge accuracy (%)	UrbanAlign accuracy (%)	UrbanAlign \(\kappa\)
Safety	61.9	66.7	81.6	0.63
Beautiful	62.9	61.4	72.4	0.45
Lively	49.4	58.0	64.5	0.28
Wealthy	60.2	69.9	74.4	0.47
Boring	48.7	54.1	65.3	0.30
Depressing	57.3	57.3	66.7	0.33
Six-category average	56.7	61.2	70.8	0.41

UrbanAlign and the zero-shot baseline share GPT-4o, while Global Ridge uses UrbanAlign's semantic differences and reference labels, making the latter a more direct test of benefits beyond feature quality alone. The results support local adaptation, but do not imply identical gains against the strongest baseline in every category or establish objective urban safety prediction.

Ablation Study¶

The following excerpt from Table 3 on page 13 uses identical LWRR hyperparameters and changes only the scoring mechanism; values are excl-equal accuracy. The third depressing-category run of repeated single-shot scoring was incomplete, so that row averages only 5 categories, while the other rows cover 6.

Scoring method	API calls per pair	Raw accuracy (%)	Accuracy with LWRR (%)	Coverage
Mode 2: pairwise single-shot	1	61.7	63.6	6 categories
Mode 2: repeated scoring with majority vote	3	62.3	64.6	5 categories
Mode 4: Observer-Debater-Judge	3	60.9	70.8	6 categories

Multi-agent scoring beats single-shot scoring by 7.2 percentage points after calibration despite not having higher raw accuracy, showing why feature quality and immediate classification accuracy should be evaluated separately. Three repeated calls provide only supporting evidence: the missing category prevents treating 64.6% and the full six-category average as a strictly matched equal-budget comparison. Table 4 on page 14 instead lists calibrated pairwise single-shot accuracy as 51.9%, differing from Table 3's 63.6%; the available main text does not clearly explain the protocol difference. This note therefore does not merge the two tables or reproduce all claimed factorial interaction gains.

Key Findings¶

Table 5 on page 14 reports Mode 4 average accuracy rising from 60.9% to 70.8% after calibration; safety rises from 63.5% to 81.6%, a gain of 18.1 percentage points.
The 70.8% result and the concept-search result of 69.3% use different settings; dimension search should not be credited with raising the main result to 70.8%.
The backbone replacement experiment in Section 4.5 uses at least 3 votes and separately retuned calibration parameters: Qwen2.5-VL-72B reaches 72.5% and GPT-4o 72.2%, which cannot be directly compared with the main table's at-least-2-vote setting.
Page 15 additionally reports a SPECS average of 67.5% and AVA accuracy of 76.7% against a zero-shot 72.6%; these are main-text reports, and the full supplementary subgroup results were not available for this note.

Highlights & Insights¶

Separating concept measurement from preference decisions is useful. External calibration can align a frozen model's outputs more closely with humans without placing every adaptation mechanism inside model parameters.
Retrieving in a hybrid space and regressing in semantic space offers a reusable division of responsibilities. Visual appearance can select context while concept-level explanations remain available for the prediction.
Multi-agent benefits should be evaluated in the downstream use case. Comparing only raw voting accuracy can miss its potential value as a numerical feature extractor.

Limitations & Future Work¶

The authors explicitly note that urban perception labels can encode social biases. Interpretable weights can expose these biases but do not eliminate them, and looking safe is not evidence of actual crime risk.
The test sets are small relative to the validation search, motivating larger independent evaluations and cross-city testing. The stated set isolation alone does not prove complete image-level or city-level separation.
Section 4.5 gives a lively-category 95% bootstrap interval of [53.9, 73.7]%, whose lower bound is below the overall zero-shot 56.7%; this conflicts with Section 4.2's statement that all lower bounds exceed 56.7%.
The supplied cache lacks the repeatedly cited supplementary material, and some formulas and figure text are corrupted. Exact re-inference logic and theoretical assumptions require checking the original typesetting; unread proofs should not be treated as verified.
The paper reports roughly 68 dollars in API cost for 4,819 pairs and approximately 2–4 hours, while the local alternative uses 4 RTX 4090 GPUs. This cost accounting differs from the main sample of 449 pairs per category and should not be used to infer a complete deployment unit price.
Useful future tests include demographic reference-set selection, fallback behavior with insufficient neighbors, and explanation stability under concept-score perturbations. These are reader suggestions, not experiments completed in the paper.

Compared with concept bottleneck models: the interpretable intermediate representation is retained, but concept extraction uses external calls to a frozen VLM, and the predictor becomes query-dependent local regression instead of a global layer.
Compared with RLHF / DPO: this method calibrates a particular pairwise judgment task rather than modifying a generative model through preference optimization. Adaptation is more localized, but this does not align every answer the model can generate.
Compared with RAG: retrieved references determine the samples used to fit a numerical calibration function rather than being inserted into a generator as textual evidence. The transferable idea is to retrieve similar decision contexts, not just similar content.
Classification: retain Multimodal VLM; the core contribution is frozen vision-language concept scoring and preference calibration, with street views as the application rather than an autonomous-driving perception or remote-sensing recognition method.

Rating¶

Novelty: 4/5. Concept mining, structured scoring, and local regression are connected clearly, although the components have established precedents.
Experimental Thoroughness: 3/5. Same-feature controls and backbone comparisons are useful, but small tests, inconsistent ablation protocols, and unavailable supplementary evidence limit confidence.
Writing Quality: 3/5. The central argument is accessible, but some statistical claims conflict and corrupted cached formulas hinder exact reproduction.
Value: 4/5. A concrete approach to economical, interpretable domain adaptation of frozen VLMs that warrants further cross-cultural and robustness evaluation.