GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks¶

Conference: ECCV 2026
Paper: ECCV Official Link
Area: Segmentation
Keywords: Error Slice Discovery, Vision-Language Models, Model Robustness, Object Detection and Segmentation Evaluation, Statistical Hypothesis Verification

TL;DR¶

Addressing the failure of existing slice discovery methods to capture contextual and relational instance-level errors, GH-ESD formulates error slice identification as grounded natural-language hypothesis generation coupled with monotonic sensitivity statistical verification, improving Precision@10 by 0.10 on the new GESD benchmark.

Background & Motivation¶

Modern deep vision models achieve impressive aggregate benchmark scores in object detection and instance segmentation, yet their operational reliability is frequently undermined by systematic failures on semantically coherent subsets of data, commonly known as error slices. Identifying and diagnosing these error slices is critical for revealing model structural weaknesses, enhancing robustness in high-stakes domains such as autonomous driving and robotics, and establishing dependable model evaluation protocols.

However, existing slice discovery approaches predominantly treat error slices as clusters in latent representation space or combinations of predefined discrete attributes. While effective for surfacing spurious correlations in global image classification, these formulations fall short for instance-level tasks where failures are often driven by contextual, relational, and spatially grounded visual patterns. Extending slice discovery to object detection via bag-of-tags representations suffers from combinatorial explosion and fails to capture relational directionality and visual similarity. Furthermore, evaluating slices via image-level error metrics severely dilutes local failure signals with unrelated correct detections across the image, while the community lacks a dedicated, spatially grounded instance-level benchmark.

The angle of attack in this paper is to leverage large language model (LLM) world knowledge combined with vision-language model (VLM) grounded visual cues to construct an expressive hypothesis space, while replacing brittle error-rate thresholding with a monotonic performance sensitivity test. Core idea: reframe instance-level error slice discovery as a generate-and-verify paradigm that constructs grounded relational hypotheses and filters out spurious correlations and VLM hallucinations via sliding-window monotonic trend statistical verification.

Method¶

Overall Architecture¶

GH-ESD follows a grounded hypothesis generate-and-verify pipeline comprising three core stages: grounded hypothesis space construction, instance-level hypothesis matching, and statistical hypothesis verification. First, the framework combines top-down LLM knowledge-driven task priors with bottom-up region-grounded descriptions to synthesize a structured natural-language hypothesis space \(\mathcal{H}\). Next, a pretrained grounded VLM processes each evaluation unit (covering false positives, false negatives, localization/segmentation discrepancy regions, or full images) alongside spatial prompts to compute normalized continuous matching scores. Finally, a statistical verification module evaluates whether model error rates scale monotonically with hypothesis matching intensity across sliding windows, retaining only validated systematic error slices.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Evaluation Units<br/>FP/FN/Localization Discrepancies & Images"] --> B["Grounded Hypothesis Space Construction<br/>Knowledge-Driven Priors + Data-Driven Synthesis"]
    B --> C["Instance-Level Hypothesis Matching<br/>VLM Spatial Prompting & Logit Normalization"]
    C --> D["Statistical Hypothesis Verification<br/>Sliding-Window Local Regression & Slope Filtering"]
    D --> E["Systematic Error Slices<br/>Interpretable & Actionable for Model Repair"]

Key Designs¶

1. Grounded Hypothesis Space Construction: Integrating Task Priors and Distributional Artifacts Conventional methods rely on unconstrained global captions or rigid discrete tag sets, unable to represent contextual relations such as "a bicycle partially occluded by a person" or "a dumpster visually resembling a truck." GH-ESD constructs a dual-source hypothesis space: top-down knowledge-driven hypotheses leverage LLMs to enumerate intrinsic task difficulties (e.g., appearance ambiguity, scale variation, object interaction, harsh lighting), while bottom-up data-driven hypotheses prompt a VLM to generate attribute-guided region captions on sampled data. An LLM subsequently infers recurring attribute-value pairs from these grounded captions and synthesizes well-formed natural-language hypotheses, ensuring broad coverage of both fundamental visual challenges and dataset-specific long-tail configurations.

2. Instance-Level Hypothesis Matching: Continuous Soft Scoring via Grounded VLM Logits Rather than forcing binary membership decisions \(g(u) \in \{0, 1\}\) that are vulnerable to boundary noise, GH-ESD quantifies the degree to which an evaluation unit \(u\) satisfies hypothesis \(h\) through a continuous matching score \(c^{(h)}(u) \in [0, 1]\). Using a pretrained VLM equipped with visual grounding capabilities (e.g., Qwen2.5-VL), the model is supplied with visual bounding boxes or points along with hypothesis prompt \(\mathcal{P}(h, u)\). By extracting vocabulary logits for affirmative ('yes') and negative ('no') tokens, the continuous matching score is formulated as:

\[c^{(h)}(u) = \frac{\exp(z(\text{'yes'} \mid \mathcal{P}(h, u)))}{\exp(z(\text{'yes'} \mid \mathcal{P}(h, u))) + \exp(z(\text{'no'} \mid \mathcal{P}(h, u)))}\]

This probabilistic formulation maps VLM discriminative responses into a continuous spectrum, avoiding information loss from premature hard thresholding and enabling fine-grained sensitivity analysis.

3. Statistical Hypothesis Verification: Mitigating Hallucinations via Monotonic Sensitivity Testing In open-ended hypothesis spaces, simple subgroup error-rate thresholding frequently flags spurious matches or VLM hallucinations. Under the principle that genuine failure modes induce higher error probabilities as hypothesis conditions are more strongly satisfied, GH-ESD verifies hypotheses via monotonic trend analysis. The evaluation restricts analysis to high-confidence units \(U_\gamma = \{u_i \mid c^{(h)}(u_i) > \gamma\}\) (default \(\gamma = 0.5\)) sorted in descending order of matching score. Within sliding windows \(w\), local linear regression is computed between matching scores and task performance metrics:

\[\text{slope}_w = \operatorname{Slope}\big(c^{(h)}(u), \ell(u)\big)_{u \in w}\]

The maximum slope across all valid windows defines the trend statistic \(\text{Trend}(h) = \max_{w \in \mathcal{W}} \text{slope}_w\). Hypotheses satisfying \(\text{Trend}(h) > \tau_{\text{trend}}\) (default \(\tau_{\text{trend}} = 0.2\)) are confirmed as systematic error slices. Alternatively, a multi-scale Spearman rank-correlation test (\(\rho \ge 0.5, p < 0.05\)) can be instantiated under the same monotonic sensitivity principle, providing rigorous multiple-hypothesis testing under Benjamini–Hochberg FDR control.

Key Experimental Results¶

Main Results¶

To benchmark instance-level slice discovery, the authors introduced GESD, constructed over 12K images from COCO, KITTI, and a face detection dataset. Errors were aggregated across YOLO, Faster R-CNN, RetinaNet, DETR, and Mask R-CNN, yielding 42 expert-annotated grounded slices (21 detection, 21 segmentation). Below are the Precision@k results on the GESD detection benchmark:

Method	P@5	P@10	P@20	Note
FACTS*	0.35	0.28	0.20	Feature clustering baseline adapted to image level
HiBug*	0.34	0.31	0.26	Extended to instance-level attribute tagging
HiBug2 (GPT-5.2)	0.73	0.63	0.53	Attribute combination search baseline
HiBug2 (Gemini-2.5-Pro)	0.77	0.63	0.52	Baseline with matching LLM backbone
GH-ESD (Ours)	0.78	0.73	0.66	Outperforms best baseline by +0.10 on P@10

Ablation Study¶

Ablation experiments evaluate the core components of GH-ESD on GESD:

Config / Component	Metric	Note
Natural-Language Hypotheses (102 slices)	Precision: 0.74 / Recall: 0.90	Compact, structurally coherent representations
Atomic Tag Combinations (HiBug2, 3666 slices)	Precision: 0.36 / Recall: 0.71	Combinatorial explosion, lacks relational context
Knowledge-Driven Alone (KD-only)	P@10 = 0.62	Captures intrinsic task challenges (14% overlap with DD)
Data-Driven Alone (DD-only)	P@10 = 0.21	Identifies dataset-specific contextual bias
Instance-Level Grounding (partially hidden faces)	P@10 = 0.80 (Trend slope = 1.32)	Eliminates dilution from unrelated correct instances
Image-Level Aggregation (partially hidden faces)	P@10 = 0.60 (Trend slope = 0.84)	Dilutes localized error signal
Monotonic Trend Verification (Sliding slope)	Slice Verification F1 = 0.77	+3.7% F1 over error rate thresholding, robust to thresholds
Monotonic Trend Verification (Spearman rank test)	Slice Verification F1 = 0.78	Robust under Benjamini–Hochberg FDR multiple testing

Key Findings¶

Spatial Grounding Eliminates Dilution: On the "faces partially hidden by objects" slice, instance-level conditioning boosts P@10 from 0.60 to 0.80 and strengthens the trend slope from 0.84 to 1.32 compared to image-level evaluation, verifying that non-erroneous instances within the same image dilute critical error signals.
Knowledge and Data Driven Synergy: While KD-only achieves P@10 of 0.62 and DD-only achieves 0.21, their union attains 0.73 with only 14% hypothesis overlap, proving substantial complementarity between conceptual task priors and empirical distribution analysis.
Targeted Model Repair Validation: Utilizing the discovered slice "bicycle partially occluded by a person" to guide GroupDRO reweighting during Faster R-CNN fine-tuning on COCO improves bicycle mAP from 27.20 to 33.59 and mAR from 39.33 to 47.09 (compared to standard fine-tuning at 28.58 mAP), confirming actionable utility for model debugging.

Highlights & Insights¶

From Discrete Tags to Continuous Grounded Hypotheses: By reformulating slice discovery as continuous VLM soft matching followed by monotonic regression, GH-ESD sidesteps combinatorial search explosion while retaining compositional context and spatial relational directionality.
First Dedicated Instance-Level Benchmark: GESD provides 42 meticulously verified spatial slices derived from 1500 person-hours of expert annotation, bridging the long-standing gap between synthetic image-level benchmarks and real-world perception failures.
Demonstrated Closed-Loop Debugging Utility: The discovered error slices provide human-interpretable diagnostic insights and directly facilitate targeted data reweighting, demonstrating concrete improvements in downstream model fine-tuning.

Limitations & Future Work¶

Dependency on Evaluation Data Distribution: The statistical verification module relies on adequate sample density; failure modes that are severely underrepresented in the evaluation split may fail to yield statistically significant monotonic trends.
Open-Ended Hypothesis Completeness: Automated hypothesis generation cannot provide theoretical guarantees of exhaustive failure mode coverage, especially for novel deployment environments.
Future Directions: Integrating domain-specific structured knowledge graphs, developing adaptive hypothesis refinement mechanisms, and incorporating interactive human-in-the-loop debugging workflows.

vs FACTS / Domino: Embedding-clustering methods operate in latent feature space, primarily suited for image-level classification, and lack semantic interpretability and spatial relational reasoning. GH-ESD introduces explicit natural language hypotheses and spatial bounding box grounding.
vs HiBug / HiBug2: Attribute-based tagging models failure modes as conjunctions of independent tags, suffering from combinatorial explosion and missing spatial directionality. GH-ESD formulates structured relational hypotheses and performs continuous VLM matching with statistical trend verification.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering instance-level grounded error slice discovery with continuous monotonic sensitivity verification.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation on the novel GESD benchmark, classical bias datasets, and downstream model repair.
Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, insightful motivation, and rigorous ablation analyses.
Value: ⭐⭐⭐⭐⭐ Offers an actionable framework and valuable benchmark for diagnosing and repairing object detection and segmentation systems.