Skip to content

T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning

Conference: NeurIPS 2025 arXiv: 2506.01317 Code: GitHub Area: LLM Alignment Keywords: Instruction Tuning, Data Selection, Token-Level Informativeness, Robustness, IFD Score

TL;DR

This paper proposes T-SHIRT, a data selection framework that introduces Selective IFD (considering only informative tokens) and a hierarchical selection strategy (preferring samples with high neighborhood consistency). Fine-tuning on only 5% of data selected by T-SHIRT surpasses training on the full dataset, while the selection process requires only GPT-2 and 40 minutes on a single GPU.

Background & Motivation

Instruction fine-tuning (SFT) is a critical step for enabling LLMs to follow user instructions effectively. The "superficial alignment hypothesis" from LIMA suggests that data quality matters far more than quantity, with 1,000 high-quality samples matching large-scale fine-tuning.

Existing data selection methods suffer from two key deficiencies:

Sample-level evaluation ignores token-level information: All existing methods (IFD, Deita, DS2, etc.) score samples holistically, yet recent research shows that not all tokens are equally important in instruction tuning—only a small subset of response tokens are genuinely influenced by the instruction.

Scoring robustness is overlooked: IFD scores are highly sensitive to semantically-preserving minor input perturbations (e.g., synonym substitution)—high scores may stem from surface lexical features rather than genuine semantic quality.

A key illustrative case: when all tokens in a response have \(\Delta_t \approx 0.01\), IFD \(\approx 0.99\) (seemingly high quality), yet no token actually depends on the instruction—a misleadingly high score.

Method

Overall Architecture

T-SHIRT (Token-Selective HIeRarchical Data Selection for Instruction Tuning) comprises two novel components: 1. Selective IFD (S-IFD): Token-level informativeness-aware quality scoring 2. Hierarchical Selection Strategy: Robustness-aware selection based on neighborhood consistency

Key Design 1: Selective IFD

The paper first analyzes the token-level decomposition of the original IFD. Define \(\Delta_t = \log P_{\theta'}(y_t|y_{<t}, x) - \log P_{\theta'}(y_t|y_{<t})\), so that:

\[\text{IFD}(x,y) = \exp\left\{-\frac{1}{T}\sum_{t=1}^T \Delta_t\right\}\]

\(|\Delta_t|\) measures the influence of instruction \(x\) on generating token \(y_t\). A key finding: over 20% of response tokens have \(|\Delta_t| \leq 0.01\) (22% when computed with GPT-2, 28% with Llama-3.1-8B), meaning these tokens are equally predictable with or without the instruction.

S-IFD retains only the top \(k\%\) most informative tokens:

\[\text{S-IFD}_k(x,y) = \exp\left\{-\frac{1}{\sum_t w_t}\sum_{t=1}^T w_t \Delta_t\right\}\]

where \(w_t = 1\) if \(|\Delta_t|\) ranks in the top \(k\%\) across the entire dataset, and \(w_t = 0\) otherwise.

Key Design 2: Hierarchical Selection Strategy

Problem: IFD/S-IFD scores are not robust to semantically-preserving input perturbations. For instance, replacing "average" with "mean" in an instruction can cause a large drop in IFD score.

Solution: Generate a neighborhood for each sample via token embedding perturbations and evaluate the quality distribution over this neighborhood.

For sample \((x, y)\), \(M\) perturbed versions are generated by adding uniform noise \(\delta \sim \mathcal{U}(-\epsilon, \epsilon)\) to the token embeddings, where \(\epsilon = \alpha / \sqrt{(L+T)d}\). The following are computed:

\[\hat{\mu}(x,y) = \frac{1}{M}\sum_{i=1}^M \text{S-IFD}_k(x + \delta_x^{(i)}, y + \delta_y^{(i)})\]
\[\hat{\sigma}^2(x,y) = \frac{1}{M}\sum_{i=1}^M (\text{S-IFD}_k(\ldots) - \hat{\mu})^2\]

Hierarchical Selection: 1. First select the top \(\gamma b\) samples by neighborhood mean \(\hat{\mu}\) (\(\gamma > 1\) is an oversampling factor). 2. Then select the final \(b\) samples with the lowest neighborhood variance \(\hat{\sigma}^2\).

Intuition: Good training samples (e.g., point A in the paper's figure) should exhibit high neighborhood mean and low neighborhood variance—not only individually high quality, but stably so rather than accidentally high-scoring due to surface features.

Loss & Training

After data selection, standard SFT is applied: learning rate 2e-5, trained for 3 epochs (Alpaca-GPT-4) or 2 epochs (Magpie). The selection process uses GPT-2 to compute S-IFD, with hyperparameters \(k = 50\%\) or \(75\%\), \(\gamma = 2\), \(\alpha = 5\), \(M = 30\).

Key Experimental Results

Main Results: Alpaca-GPT-4 (5% Data Selected)

Average scores across 8 benchmarks on Llama-3.1-8B:

Method \(\mu_{\text{open}}\) \(\mu_{\text{llm}}\) \(\mu_{\text{all}}\) Cost
Full (100%) 53.93 7.15 42.23 —
Random (5%) 55.05 6.82 42.99 None
Longest 59.31 9.36 46.82 None
Deita 58.07 7.97 45.55 API
DS2 57.85 8.60 45.54 API
IFD 60.01 7.70 46.94 GPT-2
T-SHIRT (\(k=75\)) 60.91 8.12 47.71 GPT-2

On Qwen-2.5-7B: T-SHIRT achieves \(\mu_{\text{all}} = 57.52\)–\(57.60\) vs. IFD 56.81 and Full 52.50.

Using only 5% of data surpasses full-data training by 5.10–5.48 points!

Magpie Dataset (10k selected from 300k)

Method \(\mu_{\text{open}}\) \(\mu_{\text{llm}}\) \(\mu_{\text{all}}\)
Longest 68.26 23.50 57.07
IFD 68.96 25.30 58.05
T-SHIRT (\(k=50\)) 69.21 26.16 58.45

T-SHIRT remains effective on larger, higher-quality datasets.

Ablation Study

Components S-IFD Hierarchical Llama \(\mu_{\text{open}}\) Qwen \(\mu_{\text{open}}\)
IFD Baseline ✗ ✗ 60.01 70.48
+S-IFD ✓ ✗ 60.66 70.78
+Hierarchical ✗ ✓ 60.84 70.78
T-SHIRT ✓ ✓ 60.91 70.91

Each component contributes independently, and their combination yields the best performance.

Efficiency Comparison

Method Runtime API Required
Longest ~0h No
IFD 0.2h No
T-SHIRT (\(M=30\)) 0.7h No
Deita 1.9h Yes
DS2 2.6h Yes

Key Findings

  1. API-driven selection is not necessarily superior: Deita and DS2 use the GPT-4o-mini API yet underperform T-SHIRT, which relies solely on GPT-2.
  2. Selecting samples with high-variance neighborhoods causes a sharp performance drop of 2.29–4.86 points, validating the importance of robustness.
  3. The optimal token selection ratio \(k\) varies by model (75% for Llama, 50% for Qwen), but any value below 100% yields significant improvements.
  4. Binary token weights (0/1) outperform soft weights (1.5:1 or 2:1); completely discarding uninformative tokens is the optimal strategy.
  5. As few as \(M = 10\) perturbations approach near-optimal performance, further improving efficiency.

Highlights & Insights

  1. Pioneering the token-level perspective: This is the first data selection work to introduce token-level analysis, revealing that >20% of response tokens carry no instruction-following information.
  2. Robustness as a quality dimension: Neighborhood consistency is incorporated into data selection—an insight borrowed from adversarial machine learning.
  3. Counterintuitive conclusion: Expensive API-based methods (Deita, DS2) are outperformed by the cheap GPT-2 + T-SHIRT combination, demonstrating that the evaluation dimension matters more than the evaluation model's capability.
  4. Extreme efficiency: A single GPU processes 52k samples in 40 minutes with no API cost, making the method highly practical.
  5. Complementary to SLM: T-SHIRT operates at the data preparation stage without modifying the training process, and can be combined with selective language modeling (SLM).

Limitations & Future Work

  1. Limited model scale: Experiments are conducted only on 7B–14B models; effectiveness on larger models (70B+) remains unverified.
  2. Safety not considered: Safety criteria are not incorporated into the selection process, potentially allowing harmful samples to be selected.
  3. Weak theoretical foundation: The effectiveness of hierarchical selection lacks rigorous theoretical justification.
  4. Token selection ratio requires tuning: The optimal \(k\%\) varies by model, and no automatic selection mechanism is provided.
  5. Diversity not considered: T-SHIRT focuses solely on quality without explicitly modeling sample diversity, potentially leading to homogeneous selections.
  6. Limitations of embedding perturbation: Perturbations in continuous space may not fully capture discrete semantics-preserving transformations.
  • Relation to IFD (Li et al., 2024): T-SHIRT directly extends IFD through token-level selection and robustness enhancement.
  • Distinction from RHO-1 (Lin et al., 2024): RHO-1 modifies the training process (selective loss), while T-SHIRT modifies data preparation; the two are complementary.
  • Connection to NEFTune (Jain et al., 2024): The perturbation scale \(\alpha = 5\) is adopted from NEFTune's noise injection scheme.
  • Broader implications: The ideas of token-level quality evaluation and neighborhood robustness are generalizable to pre-training data selection and RLHF preference data curation.

Rating

  • Novelty: ⭐⭐⭐⭐ — Both token-level S-IFD and robustness-aware hierarchical selection are novel contributions.
  • Theoretical Depth: ⭐⭐⭐ — The analysis is intuitive and convincing, but lacks formal theoretical support.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ — Multiple datasets, models, and baselines with comprehensive ablations; very solid.
  • Writing Quality: ⭐⭐⭐⭐⭐ — Motivating examples and figures are clear; motivation is well articulated and easy to follow.
  • Practical Value: ⭐⭐⭐⭐⭐ — Efficient, low-cost, open-sourced, and directly deployable.
  • Overall: ⭐⭐⭐⭐ (8.5/10) — A highly practical contribution achieving significant performance gains with a simple and elegant method.