Skip to content

Discovering Hidden Gems in Model Repositories

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Model Search, Hidden Gems, Model Trees, Multi-Armed Bandit, Sequential Halving

TL;DR

This paper exposes the pervasive presence of overlooked, low-download "hidden gems" across open model hubs that consistently outperform popular foundation baselines, and introduces an accelerated Sequential Halving algorithm with correlated sampling and aggressive pruning that discovers top models with a 50× reduction in evaluation queries.

Background & Motivation

Public model hubs such as Hugging Face have democratized community access to millions of fine-tuned language models and specialized adapters. However, as these public repositories balloon into vast ecosystems, selecting the optimal model for a given downstream task has become a fundamental bottleneck. Conventional discovery mechanisms rely on developer documentation, but model cards are frequently missing, incomplete, or lack standardized benchmark evaluations, leaving more than 90% of uploaded models without any functional evaluation signals. Consequently, the vast majority of practitioners and enterprise users default to the well-known foundation models or official instruction fine-tunes located at the root of the Model Tree. This creates an extreme centralization of usage where a mere 0.0015% of hosted models account for over 95% of aggregate monthly downloads, while the long tail remains virtually unvisited.

This staggering concentration poses a crucial foundational question: does community usage reflect an "Efficient Discovery" hypothesis where collective adoption naturally converges to the superior models, or is it governed by an "Information Asymmetry" hypothesis where elite models remain buried simply because exhaustive testing is impossible? Validating this empirically is computationally intractable at scale, as evaluating thousands of fine-tunes across standardized benchmarks would demand billions of forward inference passes. Furthermore, fine-tuned checkpoints in weight space do not follow smooth or predictable performance gradients along tree lineage, meaning that simple heuristics based on download counts or graph centrality fail to uncover top performers.

To address this challenge, the paper formalizes model selection under fixed compute constraints by framing the repository search as a Multi-Armed Bandit problem under a fixed budget. By developing an active exploration strategy that requires minimal evaluations per candidate, the authors systematically test whether long-tail superior models exist. Core idea: by mathematically defining "Hidden Gems" as unpopular models that strictly dominate the popular consensus, the authors introduce an accelerated Sequential Halving algorithm combining correlated query sampling and aggressive early-round pruning to uncover top-tier models with 50× query acceleration.

Method

Overall Architecture

The proposed discovery framework operates over a candidate model tree \(\mathcal{T} = \{m_1, \dots, m_K\}\) and an evaluation dataset \(\mathcal{D}\) under a strict global budget \(B = N \times K\), where \(N \in \{10, 50\}\) denotes the average queries allowed per candidate. To isolate capability from inference cost, candidates are grouped within families sharing common ancestral architectures (e.g., Qwen-3B, Qwen-7B, Mistral-7B, Llama3.1-8B). The framework employs round-based elimination with synchronized test queries, leveraging the empirical property that low-quality models dominate open repositories to rapidly reallocate queries toward top-performing candidates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Candidate Model Pool T and Dataset D"] --> B["Correlated Sampling Variance Reduction<br/>Shared query batch qs across all active models"]
    B --> C["Model Accuracy Estimation<br/>Cumulative evaluation on Q and empirical ranking"]
    C --> D["Aggressive Early-Round Pool Pruning<br/>Direct cut to fixed 100 candidate models"]
    D --> E["Adaptive Multi-Round Budget Schedule<br/>Iterative 50% candidate halving and query doubling"]
    E --> F["Retrieve Best Performing Model m*"]

Key Designs

1. Correlated Sampling Variance Reduction: Eliminating Inter-Candidate Query Difficulty Discrepancies In standard Multi-Armed Bandit and Sequential Halving formulations, queries are drawn independently at random for each active arm. However, real-world evaluation benchmarks comprise a heterogeneous mixture of trivial instances and unsolvable edge cases. If Candidate A happens to receive easy questions while Candidate B is assigned difficult ones, empirical score differences will be heavily biased by query difficulty rather than actual model capabilities. To neutralize this noise, the method enforces Correlated Sampling: in each round \(s\), a shared batch of unseen queries \(q_s \subset \mathcal{D}_{pool}\) is sampled, and every surviving candidate in \(\mathcal{P}_s\) is evaluated on the exact same set of prompts. By maintaining a shared evaluation set \(\mathcal{Q}\), the variance of the performance difference estimator between any two models is drastically reduced without consuming extra queries.

2. Aggressive Early-Round Pool Pruning: Slashing the Defective Long Tail to Conserve High-Tier Compute Empirical quality distributions across open model repositories reveal severe left skewness: the vast majority of community uploads are defective, undertrained, corrupted, or suffer from catastrophic forgetting, with 60%–90% of models scoring more than 10% below the top performers. Standard Sequential Halving discards only 50% of surviving candidates per round, thereby spending substantial budget refining the score estimates of obviously inadequate models. The authors replace this conservative scheme with an aggressive "fail-fast" first elimination step, immediately pruning the candidate pool from several hundred models down to a fixed set of exactly 100 candidates (retaining approximately the top 20%). This eliminates wasted computation on broken models and preserves the remaining budget for fine-grained discrimination between viable high-capacity fine-tunes.

3. Adaptive Multi-Round Budget Schedule: Rebalancing Initial Exploration and Late-Stage Refinement Under extreme query limitations (e.g., \(N=10\) queries per candidate on average), naive uniform splitting across rounds leaves too few samples for reliable early-stage pruning, causing fatal false-positive eliminations of genuine gems. The authors structure a specialized multi-round schedule that dedicates 60% of the entire budget to the first round (\(s_1 = 6\) queries per candidate when \(N=10\), and \(s_1 = 30\) when \(N=50\)), providing sufficient statistical signal to avoid misclassifying elite models during the initial heavy cut. Across subsequent rounds, as the candidate pool iteratively halves (\(100 \to 50 \to 25 \dots\)), the query count per candidate escalates aggressively (e.g., for \(N=10\): 6, 16, 30, 60, 120; for \(N=50\): 30, 75, 150, 300, 600 queries). This ensures that the final contending models receive hundreds of shared queries to reliably resolve tiny margin advantages.

Benchmark & Evaluation Protocol

The method is systematically benchmarked across four major open model trees: Qwen2.5-3B (479 evaluated models), Qwen2.5-7B (650 evaluated models), Mistral-7B (280 evaluated models), and Llama3.1-8B (617 evaluated models), spanning both full fine-tunes and LoRA adapters. The primary benchmark is RouterBench, comprising 2,500 diverse evaluation queries across MBPP (coding), Winogrande (commonsense), MMLU (multitask reasoning), ARC-Challenge (reasoning), and GSM8K (grade-school math). Domain generalization is additionally tested on MedQA. Evaluated algorithms include Uniform Allocation, UCB, UCB-E, UCB-StdDev, Top-Two Thompson Sampling (TTTS), Successive Rejects, and Bayesian Elimination (BayesElim), averaged over 100 random runs.

Key Experimental Results

Main Results

The comprehensive evaluation confirms that "hidden gems"—defined as models outside the top 1% download bracket that achieve top 1% performance and strictly outperform the best popular baseline—exist in every single evaluated model tree, delivering massive performance leaps at zero extra parameter or inference cost.

Model Tree Task Domain Popular Base Accuracy Hidden Gem Accuracy Accuracy Gain Identified Gem Model & Monthly Downloads
Qwen-3B GSM8K (Math) 83.5% 89.0% +5.5% watermelonhjg/...-EN-Zero (9 downloads)
Qwen-3B MBPP (Code) 68.6% 73.3% +4.7% GiuLeo01/FortranCodeGen-3B... (8 downloads)
Qwen-3B RouterBench (General) 71.6% 73.2% +1.6% watermelonhjg/...-EN-Zero (9 downloads)
Qwen-7B GSM8K (Math) 89.7% 92.1% +2.4% zwhe99/DeepMath-Zero-Math-7B (47 downloads)
Qwen-7B MBPP (Code) 80.9% 82.1% +1.2% mlx-community/...-abliterated-v1 (6 downloads)
Qwen-7B RouterBench (General) 78.3% 79.1% +0.8% langfeng01/GiGPO-Qwen2.5-7B... (80 downloads)
Mistral-7B GSM8K (Math) 40.6% 80.7% +40.1% mlfoundations-dev/oh-mistral-bs512... (4 downloads)
Mistral-7B MBPP (Code) 47.0% 58.8% +11.8% mlfoundations-dev/mistral_7b_0-3_oh-... (5 downloads)
Mistral-7B RouterBench (General) 55.6% 69.6% +14.0% mlfoundations-dev/...-gpt-4o-mini (1 download)
Llama3.1-8B GSM8K (Math) 83.2% 96.0% +12.8% nvidia/OpenMath2-Llama3.1-8B (930 downloads)
Llama3.1-8B MBPP (Code) 65.6% 68.8% +3.2% plandes/sdoh-llama-3-1-8b (2 downloads)
Llama3.1-8B RouterBench (General) 71.3% 74.7% +3.4% aisingapore/Llama-SEA-LION-v3-8B-IT (2.8k downloads)

Ablation Study

Ablation across budget tiers (\(N=10\) and \(N=50\) queries per model) demonstrates the individual and combined impact of correlated sampling and the custom pruning schedule on true model rank (lower is better; 1.0 is ground-truth optimal) and top-1 retrieved accuracy across 100 repetitions.

Search Method Variant Budget (Queries/Model) Qwen-3B Rank / Acc Qwen-7B Rank / Acc Mistral-7B Rank / Acc Llama-8B Rank / Acc
Random Selection - 233.8 / 0.588 318.4 / 0.599 144.6 / 0.431 270.8 / 0.601
Best Base Model - 56.5 / 0.716 30.0 / 0.783 62.0 / 0.556 41.0 / 0.713
Standard Sequential Halving 10 69.3 / 0.714 62.9 / 0.777 15.2 / 0.684 75.2 / 0.710
Ours w. Standard SH Scheduler 10 51.9 / 0.717 45.4 / 0.780 12.7 / 0.684 56.2 / 0.718
Ours w/o Correlated Sampling 10 12.6 / 0.725 10.4 / 0.788 2.8 / 0.694 11.1 / 0.725
Ours (Full Method) 10 11.3 / 0.726 15.8 / 0.786 2.6 / 0.694 15.4 / 0.725
Standard Sequential Halving 50 41.0 / 0.719 29.6 / 0.784 7.7 / 0.690 29.9 / 0.720
Ours w. Standard SH Scheduler 50 10.6 / 0.726 13.5 / 0.787 5.3 / 0.692 11.2 / 0.731
Ours w/o Correlated Sampling 50 5.4 / 0.728 5.5 / 0.789 2.0 / 0.695 6.5 / 0.729
Ours (Full Method) 50 3.5 / 0.729 3.6 / 0.790 1.6 / 0.695 3.0 / 0.736
Best Available (Oracle Upper Bound) Full (2,500) 1.0 / 0.732 1.0 / 0.791 1.0 / 0.696 1.0 / 0.747

Key Findings

  • Rebuttal of Efficient Market Selection: Usage concentration is purely driven by information friction. Superior fine-tunes regularly reside in the deep long tail; in Llama3.1-8B, an obscure fine-tune with under 1,000 monthly downloads elevates GSM8K accuracy from 83.2% to 96.0% (+12.8%), outperforming base models with millions of downloads.
  • Complete Inefficacy of Metadata Search: Out of 24 verified hidden gems across tasks and trees, 19 possess no benchmark documentation whatsoever. 3 report irrelevant benchmark scores (e.g., Southeast Asian regional metrics on a math model), and only 2 could conceivably be identified via text-based model card queries.
  • Budget Scaling and Convergence: Under an extremely tight query regime (\(N=10\)), the custom schedule achieves rank improvements of over 30 positions compared to standard SH. When budget scales to \(N=50\), correlated sampling suppresses query noise, driving the algorithm to reliably converge into the top 3 best models in every model family.

Highlights & Insights

  • Formulates the open-source model discovery challenge rigorously through the lens of best-arm identification, demonstrating that simple search heuristics based on popularity or tree topologies fail systematically.
  • Introduces correlated evaluation across bandit arms, capitalizing on the unique machine learning setting where evaluation inputs can be perfectly synchronized across competing models to cancel out query difficulty variance.
  • Provides actionable empirical insights for model hub maintainers to deploy automated background gem discovery, enabling automatic documentation generation and surface-level promotion of long-tail community assets.

Limitations & Future Work

  • The initial filtering step still requires instantiating and querying every single candidate checkpoint, which remains computationally demanding if scaling to tens of thousands of models without prior weight-space filtering.
  • The investigation is restricted to individual standalone models, leaving compound combinations—such as weight merges or dynamic ensembles of complementary gems—unexplored due to combinatorial search explosion.
  • Model discovery is task-contingent: finding hidden gems for specialized or unseen domains necessitates running the search protocol on a representative prompt distribution for that specific domain.
  • vs Efficient Model Benchmarking (Perlitz et al., 2024; Ashury-Tahan et al., 2024): Existing adaptive evaluation benchmarks focus on positioning a single newly uploaded model into a calibrated, static leaderboard with minimal sample queries; this work tackles cold-start population discovery over an unranked, unstructured pool of hundreds of models simultaneously.
  • vs Weight-Space Learning (Kahana et al., 2025b; Horwitz et al., 2025a): Weight-space representations seek zero-shot functional prediction directly from neural weights, but current methods are confined to small network sizes and simple benchmarks; the proposed active search directly operates on functional outputs, guaranteeing reliable convergence on large foundation model families.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the formalization and systematic proof of the hidden gems phenomenon in open model hubs]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Empirically evaluates 2,000+ real-world checkpoints across 4 foundation trees, backed by extensive ablations and 100-run repetitions]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional narrative clarity, rigorous problem formulation, and highly informative visual analytics]
  • Value: ⭐⭐⭐⭐⭐ [Offers a practical, highly accelerated search framework for model hubs, practitioners, and enterprise routing pipelines]