Finding Highlight Images In Your Albums: From Benchmark To Agentic MLLMs¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/naninasm/Finding-Highlight-Images-In-Your-Albums-From-Benchmark-To-MLLM
Area: Multimodal VLM
Keywords: album highlight recommendation, multimodal large language model, agentic workflow, highlight distribution aggregation, cognitive image attributes
TL;DR¶
To tackle high subjectivity and data scarcity in Album Highlight Recommendation (AHR), this paper establishes three benchmark datasets encompassing 1,010 albums and 97K images (including the owner-annotated C-AHR-100) and introduces Highlight4U, a two-stage agentic MLLM framework that synergizes local highlight distribution aggregation with global multimodal reference re-recommendation.
Background & Motivation¶
The ubiquitous adoption of smart mobile devices and the elimination of digital storage constraints have caused personal photo collections to expand at an unprecedented rate, resulting in substantial redundancy. In response to the burden of manual sorting, album-centric computer vision tasks such as event recognition, photo collection summarization, and album highlight recommendation (AHR) have attracted growing attention. Unlike event categorization which targets objective ground truth, AHR aims to automatically identify the most valuable and highlight-worthy images from personal albumsβphotographs that carry emotional resonance for personal reflection while excelling in aesthetic and social sharing appeal. However, determining what constitutes a "highlight" is intrinsically tied to human subjectivity and aesthetic perception, making reliable annotations exceptionally scarce and presenting a major bottleneck for the field.
Prior research has suffered from severe limitations on both the benchmark and modeling fronts. On the data side, public photo collections lack fine-grained highlight annotations, single-annotator selections suffer from severe personal bias, and whether third-party consensus can reliably transfer to the subjective taste of actual album owners has remained unverified. On the modeling side, most existing paradigms evaluate each image independently in isolation, ignoring event temporal context and semantic cross-image dynamics. Crucially, empirical evaluations show that while single-image scoring works reasonably well on top-ranked candidates (e.g., Top-5%), its performance deteriorates sharply when expanding to tail recommendations (e.g., Top-5% to 15%), failing to capture the human comparative behavior of establishing reference anchors before selecting supplementary highlights.
To overcome these dual barriers, this work introduces systematic improvements across both datasets and model architecture. On the benchmark side, the authors re-annotate the public PEC and CUFED datasets with multiple annotators and collect a private benchmark, C-AHR-100, comprising 100 personal albums annotated directly by their respective owners. On the modeling side, they present Highlight4U, an agentic framework built upon Qwen3-VL-8B-Instruct that combines fine-grained local scoring with global reference-based re-scoring. Core idea: formulate album highlight recommendation as a two-stage agentic workflow combining local continuous probability distribution regression with global multimodal reference-guided re-ranking, overcoming discrete score degradation and anchoring tail highlight retrieval.
Method¶
Overall Architecture¶
Highlight4U operates as a two-stage agentic framework moving progressively from local perception to global contextual reasoning. In the first stage (Local Stage), a fine-tuned vision-language agent evaluates individual images to generate predicted score token probabilities, which are transformed into continuous local highlight scores \(\hat{S}\) via a Highlight Distribution Aggregation (HDA) module; the highest-scoring images from each album are selected as top-\(M\) global references. In the second stage (Reference Extraction & Global Stage), the agent generates descriptive captions for these reference images to assemble a multimodal reference context \(\mathbf{R}\); guided by this reference set, the global re-scoring agent evaluates the remaining candidate images to output comparative highlight scores \(E\), yielding the final recommendation list.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Album Images<br/>{X_j}"] --> B["Local Highlight Prediction Agent<br/>Token Probabilities P"]
B --> C["Highlight Distribution Aggregation HDA<br/>Continuous Local Scores Ε"]
C --> D["Global Reference Selection<br/>Top-M Reference Images"]
D --> E["Multimodal Reference Construction<br/>Generate Captions & Build R"]
E --> F["Global Reference Re-Recommendation Agent<br/>Context-Aware Re-Scoring E"]
F --> G["Final Recommended Set<br/>Top-M References + Supplementary Highlights"]Key Designs¶
1. Fine-Grained Highlight Annotation System: Multi-Annotator Aggregation and Owner Ground Truth
Subjective cognitive attributes cannot be reliably captured by isolated binary labels from a single annotator. The authors address this by filtering and re-annotating the multi-event PEC (500 albums) and CUFED (410 albums) datasets. For each album, 5 independent annotators perform binary classification (recommended highlight ratio ~20%). Summing these 5 binary judgments yields continuous highlight scores from 0 to 5: 0 signifies consensus non-highlight, 1β2 denotes subjective selection by a minority, and 3β5 indicates consensus highlights validated by the majority. Inter-annotator agreement metrics (Gwet's AC1 > 0.6, saturating at 5 annotators) demonstrate robust cross-observer consensus. To examine real-world transfer, the authors collected C-AHR-100, where 100 volunteers each provided 100 personal smartphone images along with their own binary highlight selections, establishing the first authentic testbed for owner-level preference alignment.
2. Highlight Distribution Aggregation (HDA): Eliminating Ordinal Disconnection and Argmax Truncation
Directly prompting an MLLM to output discrete score tokens (e.g., 0β5) introduces severe ordinal modeling flaws: standard cross-entropy penalizes predicting score 4 for a ground-truth 3 equally as predicting 0, and greedy decoding discards rich uncertainty cues stored across the probability simplex. Highlight4U introduces the Highlight Distribution Aggregation (HDA) module to map the predicted token probability distribution \(\mathbf{P} = \{p_i\}_{i=0}^{5}\) into a continuous, refined local score \(\hat{S}\). Two formulations are developed: a training-free expected-value weighting strategy: $\(\hat{S} = \sum_{i=0}^{5} i \times \frac{\exp(p_i)}{\sum_{j=0}^{5}\exp(p_j)}\)$ and a parameterized variant that feeds the normalized probability vector into an MLP regression head optimized via mean squared error \(\mathcal{L}_{MSE}\). By preserving subtle probability gradients, HDA delivers a marked +6.5% boost in Top-5% precision without architectural bloat.
3. Global Reference-Based Re-Recommendation Agent: Multimodal Context for Tail Highlight Retrieval
Evaluating images in a vacuum ignores human comparative decision-making and breaks intra-album semantic consistency. While local scoring suffices for initial top candidates, it degrades sharply on tail highlights (Top-5% to 15%) due to vulnerability to isolated visual outliers. Highlight4U solves this through global reference-based re-recommendation. The top-\(M\) local highlights (\(M=5\) identified as optimal) are selected as anchors, and a dedicated captioning prompt generates textual descriptions, constructing a unified multimodal reference context \(\mathbf{R} = \{(\hat{\mathbf{X}}^i, \text{Cap}(\hat{\mathbf{X}}^i, \mathbf{Q}_2))\}_{i=0}^{M-1}\). The global agent then re-evaluates all remaining non-reference candidates conditioned on \(\mathbf{R}\), computing comparative highlight scores \(E\). The final target set of size \(\hat{N}\) is formed by combining the initial \(M\) reference images with the top \((\hat{N}-M)\) supplementary images ranked by \(E\), drastically elevating tail recall and aesthetic consistency.
A Worked Example¶
Consider a user album containing 100 vacation photos from a beach trip, with a target recommendation size of 15 highlights (Top-15%): 1. Local Scoring: Qwen3-VL processes each of the 100 images independently. For photo A (a well-composed sunset portrait with slight backlighting), token probabilities distribute across score 3 (0.4) and score 4 (0.5); HDA computes an expectation of 3.65, whereas standard discrete decoding would bluntly round it to 4. 2. Anchor Selection: The top 5 images with the highest \(\hat{S}\) (e.g., key ceremony shot, crisp group portrait on the shoreline) are selected as reference anchors. The agent generates scene captions and packages them into multimodal context \(\mathbf{R}\). 3. Global Re-Evaluation: The remaining 95 photos are evaluated against \(\mathbf{R}\). A candid photo B showing a family member splashing into waves previously received a modest local score of 2.1 due to motion blur; however, when contrasted against the beach context \(\mathbf{R}\), the global agent recognizes its high emotional valence and situational spontaneity, assigning it a strong relative score \(E=3.8\). Photo B is successfully retrieved as a supplement into the final 15 highlights.
Loss & Training¶
All experiments are executed on 8 NVIDIA A40 GPUs using LLaMA-Factory with Qwen3-VL-8B-Instruct as the backbone. Low-Rank Adaptation (LoRA) is applied with rank \(r=16\) and scaling factor \(\alpha=32\). Both local scoring and global reference agents are fine-tuned for 5 epochs with a global batch size of 256 using AdamW and a cosine learning rate schedule (maximum learning rate \(1\times 10^{-4}\), warmup ratio 0.1). Per-device batch size and gradient accumulation steps are set to (8, 4) for local scoring and (4, 8) for the global agent. Image resolutions are differentiated: local scoring uses up to \(1024 \times 1024\) pixels to preserve fine details, while global re-scoring scales images to \(512 \times 512\) pixels to accommodate multiple reference frames and target candidates within GPU memory limits.
Key Experimental Results¶
Main Results¶
The authors evaluate Highlight4U against diverse vision backbones, task-specific aesthetic models, and state-of-the-art MLLMs across PEC, CUFED, and C-AHR-100. Performance is benchmarked using Precision (P@), Recall (R@), and average Score (S@) at Top-5% and Top-15% thresholds.
| Dataset | Method | Type | P@5% | R@5% | S@5% | P@15% | R@15% | S@15% |
|---|---|---|---|---|---|---|---|---|
| PEC | ConvNeXt-B (Reg.) | Classical Vision (Reg.) | 68.8 | 22.0 | 3.20 | 54.9 | 50.8 | 2.71 |
| PEC | CLIP-ViT-B (Reg.) | Classical Vision (Reg.) | 72.4 | 23.1 | 3.28 | 54.3 | 50.6 | 2.69 |
| PEC | GPT-4o | Zero-shot MLLM | 29.7 | 10.1 | 1.70 | 28.2 | 26.6 | 1.66 |
| PEC | Qwen3-VL-8B (SFT) | Fine-tuned MLLM Baseline | 66.9 | 23.9 | 3.18 | 55.4 | 53.2 | 2.71 |
| PEC | Highlight4U (Training-free HDA) | Ours | 74.1 | 26.5 | 3.34 | 61.4 | 60.0 | 2.89 |
| PEC | Highlight4U* (MLP-HDA) | Ours | 74.4 | 26.7 | 3.33 | 61.1 | 59.8 | 2.89 |
| CUFED | CLIP-ViT-B (Reg.) | Classical Vision (Reg.) | 61.4 | 22.8 | 2.91 | 46.6 | 50.8 | 2.39 |
| CUFED | Qwen3-VL-8B (SFT) | Fine-tuned MLLM Baseline | 50.2 | 20.5 | 2.55 | 39.9 | 45.5 | 2.17 |
| CUFED | Highlight4U (Training-free HDA) | Ours | 63.1 | 26.3 | 3.03 | 53.4 | 61.3 | 2.65 |
| C-AHR-100 | GPT-4o | Zero-shot MLLM | 28.4 | 9.0 | - | 25.1 | 23.3 | - |
| C-AHR-100 | CLIP-ViT-B (Reg.) | Classical Vision (Reg.) | 31.7 | 9.8 | - | 28.5 | 26.4 | - |
| C-AHR-100 | Highlight4U (Training-free HDA) | Ours (Zero-shot Transfer) | 36.0 | 11.6 | - | 31.7 | 29.6 | - |
Note: On the private C-AHR-100 dataset, models trained strictly on public PEC consensus annotations are evaluated zero-shot against ground-truth labels provided by actual album owners.
Ablation Study¶
Component contributions evaluated on the PEC dataset:
| Baseline | Fine-tune (SFT) | HDA | HDA* (MLP) | Ref-Image | Ref-Text | P@5% | R@5% | P@15% | R@15% | Note |
|---|---|---|---|---|---|---|---|---|---|---|
| β | 28.6 | 9.8 | 26.1 | 24.6 | Vanilla zero-shot Qwen3-VL judgment | |||||
| β | β | 66.9 | 23.9 | 55.4 | 53.2 | Standard SFT (discrete classification) | ||||
| β | β | β | 73.4 | 26.2 | 58.4 | 56.6 | + Training-free HDA probability expectation | |||
| β | β | β | 73.5 | 26.4 | 58.6 | 56.8 | + MLP-based HDA regression | |||
| β | β | β | β | 73.3 | 26.2 | 60.0 | 58.7 | + Visual-only global references | ||
| β | β | β | β | β | 74.1 | 26.5 | 61.4 | 60.0 | Full model with multimodal references | |
| β | β | β | β | β | 74.4 | 26.7 | 61.1 | 59.8 | Full model with MLP variant |
Key Findings¶
- Discrete Classification Bottleneck vs. HDA Gain: Direct SFT with discrete classification yields a modest P@5% of 66.9%, underperforming the classical CLIP regression baseline (72.4%). Integrating the training-free HDA module propels P@5% to 73.4% (+6.5%), confirming that probability distributions preserve vital ordinal nuance and subjective uncertainty.
- Global References Anchor Tail Recall: Restricting the model to local scoring leads to vulnerability against prediction noise when expanding recall, achieving only 56.6% R@15%. Introducing multimodal global references boosts R@15% to 60.0% (+9.4% over CLIP), validating that reference anchors effectively mitigate intra-album redundancy and drift.
- Reference Capacity Trade-Off (\(M\)): Sensitivity analyses reveal that \(M=1\) or \(2\) fails to supply sufficient scene context, whereas \(M \ge 9\) introduces noisy references and fragments visual attention. Setting \(M=5\) achieves the sweet spot between contextual richness and computational cost.
- Transfer from Consensus to Personal Taste: On the private C-AHR-100 benchmark, Highlight4U attains 36.0% P@5% and 31.7% P@15%, clearly outperforming commercial GPT-4o (28.4% and 25.1%) and specialized visual backbones, verifying that public highlight perception aligns with and transfers effectively to private owner preferences.
Highlights & Insights¶
- Summation-Based De-biasing and Owner Benchmark: Using the sum of 5 independent binary ratings transforms noisy subjective labels into an objective 0β5 highlight spectrum, while the newly introduced C-AHR-100 dataset provides the community with a rigorous benchmark for owner-aligned evaluation.
- Training-Free HDA with Multimodal Re-ranking: Rather than designing cumbersome loss functions, the framework leverages soft token distributions to bypass classification discretization and harnesses LLM-generated captions to anchor global visual re-scoring.
- High-Level Semantics Over Low-Level Texture: The empirical observation that CLIP significantly outperforms DINOv2 underscores that album highlight selection is governed by affective narrative and scene semantics rather than pixel-level geometric fidelity, cementing MLLMs as the ideal foundation for AHR.
Limitations & Future Work¶
- Author-Acknowledged Limitations: To satisfy GPU memory bounds during multi-image attention, the global reference agent must downscale input resolution from \(1024 \times 1024\) to \(512 \times 512\) pixels, and the reference set size is constrained to a fixed \(M=5\), limiting global context in extremely large collections.
- Independent Observations: The current pipeline treats albums as unordered sets, ignoring temporal timestamps, burst photo intervals, and GPS geolocation clustering that could inherently filter near-duplicate poses. Furthermore, few-shot in-context personal preference adaptation remains unaddressed.
- Future Directions: Exploring long-context multi-image architectures with dynamic temporal-aware reference clustering, as well as lightweight on-device test-time adaptation (TTA) guided by user feedback.
Related Work & Insights¶
- vs Album Event Recognition and Summarization (Bossard et al., ICCV 2013; Yu et al., EMNLP 2017): Event recognition targets objective category labels, and summarization prioritizes storyline completeness; AHR specifically optimizes for aesthetic appeal, emotional value, and social shareability across varying user perspectives.
- vs Image Quality Assessment and Aesthetics (Q-Align, Wu et al., 2023; AVA Dataset): General IQA/IAA models evaluate isolated image quality without album context. In the experiments, Q-Align and CLIP-IQA underperform substantially, proving that album highlights depend critically on relative contrast and thematic coherence rather than standalone technical quality.
Rating¶
- Novelty: βββββ Establishes a pioneering multi-annotator and personal-owner benchmark while designing an intuitive local-to-global agentic recommendation pipeline.
- Experimental Thoroughness: βββββ Rigorous validation across three distinct datasets with thorough comparisons spanning CNNs, ViTs, aesthetic models, and MLLMs.
- Writing Quality: βββββ Well-structured narrative, mathematically sound distribution aggregation formulations, and clear motivation throughout.
- Value: βββββ Highly impactful for mobile phone album curation, automatic social story creation, and the broader study of subjective visual cognition.