Skip to content

Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: On-Policy Distillation, Multi-Constraint Instruction Following, Counterfactual Scoring, Reward Shaping, Reinforcement Learning

TL;DR

To alleviate constraint dilution in on-policy distillation where a frozen teacher's probability mass is stretched over joint prompt requirements, CC-OPD forces the teacher to rescore the student's rollout under leave-one-out constraint contexts, extracting dense per-constraint token-level likelihood differences as a clipped shaping reward that allows a 1.5B student to surpass its 7B RL-trained teacher.

Background & Motivation

Modern instruction-following language model assistants are increasingly required to satisfy multiple simultaneous constraints bundled into a single user prompt, spanning output formatting, token budgets, stylistic register, content exclusions, and domain-specific citation styles. Evaluating these systems requires verifying whether a generated response simultaneously respects every single active constraint. Despite major advances in instruction tuning, state-of-the-art models still routinely stumble when faced with multi-constraint compositions: as the number and complexity of simultaneous requirements scale up, compliance degrades sharply, making multi-constraint instruction following a primary bottleneck in post-training alignment.

Current post-training paradigms leave critical structural signals underutilized. On one side, rubric- or checklist-based reinforcement learning approaches (such as ROPD, RaR, and RTT-GRPO) introduce external verification pipelines by decomposing prompts into individual criteria, grading them via rule-based code checkers or LLM judges, and aggregating those grades into sequence- or token-level RL rewards. While fine-grained, this framework relies heavily on noisy, computationally expensive external graders, treats the policy strictly as an RL actor, and discards the rich internal probabilistic representations of strong pre-trained teacher models. On the other side, on-policy distillation (OPD) directly minimizes the reverse-KL divergence between the student's self-sampled rollout and a frozen teacher. However, standard OPD feeds the entire instruction as a single joint conditioning prompt to the teacher. As multiple constraints become concurrently active, the teacher must spread its probability mass across all requirements simultaneously, making it practically impossible to disentangle the marginal influence of any individual constraint on any given token—a fundamental pathology termed constraint dilution.

The key conceptual insight of this paper is to invert the standard supervision paradigm: rather than augmenting the student or teacher input with external scaffolding or extraneous information, the method queries the frozen teacher itself under systematic ablations of its conditioning context. Because the student's generated sequence remains fixed, contrasting the teacher's full-context log-probabilities against its leave-one-out counterfactual log-probabilities cleanly isolates the marginal credit or penalty attributable to each constraint at every single decoding step. Core idea: extract per-constraint, per-token likelihood differentials via leave-one-out counterfactual teacher scoring on identical student rollouts, summing and clipping them into an internal dense shaping reward that augments on-policy distillation without external verifiers.

Method

Overall Architecture

CC-OPD builds upon on-policy distillation with sampled-token reverse-KL divergence. Given a user query \(q\) and a constraint set \(C = \{c_1, \dots, c_N\}\) combined into an inline instruction \(\text{inst}(q, C)\), the student model \(\pi_\theta\) first samples a response rollout \(y = (y_1, \dots, y_L) \sim \pi_\theta(\cdot \mid q, C)\) conditioned on the complete instruction. Next, rather than performing a single teacher pass, the frozen teacher model \(P_T\) forced-scores the exact same rollout \(y\) across \(|C| + 1\) distinct prompt contexts: once under the full instruction context \(C\), and \(N\) times under each leave-one-out context \(C \setminus \{c_i\}\) where individual constraint spans are excised.

For every token \(y_t\) along the trajectory, the difference in the teacher's conditional log-likelihood between the full context and each ablated context provides a dense, per-token attribution signal for each constraint. These per-constraint signals are summed across all constraints and symmetrically clipped to safeguard against outlier gradients. The resulting scalar shaping term is linearly combined with the standard sampled-token OPD reward and supplied to a standard ratio-clipped PPO optimization step to update the student parameters. Throughout distillation, the student retains full contextual visibility, and no external judge or discriminator is required.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query & Constraint Set<br/>q and C = {c₁, ..., c_N}"] --> B["Student Rollout Sampling<br/>y ~ π_θ(· | q, C)"]
    B --> C["Leave-One-Out Teacher Scoring<br/>score same y under C and C \\ {c_i}"]
    C --> D["Constraint Shift Sum & Clipping<br/>sum δ_i(t) and clip to [-c, +c]"]
    D --> E["Counterfactual Reward Shaping<br/>r_t = -c_KL(t) + λ δ̂(t)"]
    E --> F["PPO Student Policy Update<br/>ratio-clipped PPO on π_θ"]

Key Designs

1. Leave-One-Out Teacher Scoring: Disentangling Probability Mass to Overcome Constraint Dilution Standard on-policy distillation collapses the entire set of requirements into a monolithic prompt, obscuring whether a given generated token \(y_t\) is supported by constraint \(c_1\), by \(c_7\), or is purely conversational filler. CC-OPD addresses this structural limitation by holding the student rollout sequence \(y\) fixed and querying the frozen teacher \(P_T\) under \(|C| + 1\) prompt variations. For each constraint \(c_i\), a counterfactual prompt is created by performing a clean byte-aligned excision of its natural-language span from \(\text{inst}(q, C)\), yielding \(\text{inst}(q, C \setminus \{c_i\})\). The teacher evaluates the log-probability of token \(y_t\) under the full set \(\ell(C; t) = \log P_T(y_t \mid q, C, y_{<t})\) and under the ablated set \(\ell(C \setminus \{c_i\}; t) = \log P_T(y_t \mid q, C \setminus \{c_i\}, y_{<t})\). The per-constraint counterfactual attribution is defined as: $\(\delta_i(t) = \ell(C; t) - \ell(C \setminus \{c_i\}; t)\)$ A positive \(\delta_i(t)\) signifies that the presence of \(c_i\) elevates the teacher's conditional probability for \(y_t\) (identifying a constraint-satisfying token), a near-zero value indicates that \(c_i\) is indifferent to \(y_t\), and a negative shift indicates that \(c_i\) penalizes \(y_t\). Unlike naive alternatives that score constraints in isolation relative to an empty context (NAO, \(\ell(\{c_i\}) - \ell(\emptyset)\)), the leave-one-out design preserves realistic multi-constraint contextual interactions and dependencies.

2. Constraint Shift Sum & Clipping: Preserving High-Order Interactions with Outlier Bounding To condense the \(N\)-dimensional attribution vector at each decoding step into a scalar learning signal suitable for RL policy gradients, CC-OPD aggregates over constraints via summation and applies symmetric hard clipping: $\(\delta(t) = \sum_{i=1}^{|C|} \delta_i(t), \quad \hat{\delta}(t) = \text{clip}(\delta(t), -c, +c)\)$ Through the lens of Möbius inversion on the Boolean lattice of constraints, a single Leave-All contrast (LA, \(\ell(C) - \ell(\emptyset)\)) weights every non-empty constraint interaction coefficient equally with weight \(1\). In contrast, the LOO summation naturally scales an order-\(k\) interaction by \(k\) (\(\sum_{T \subseteq C} |T| m(T)\)), assigning strictly higher weight to multi-constraint synergistic effects where multiple requirements jointly shape token generation. To protect policy gradients from numerical anomalies, rare lexical tokens, or pathological shifts at sequence boundaries (such as <|im_end|>), the clipping threshold \(c\) (set to \(5.0\)) bounds the maximum credit assignment to \([-c, +c]\), ensuring robust and stable optimization trajectories.

3. Counterfactual Reward Shaping: Seamless Backbone Integration with Zero-Shift Fallback The clipped counterfactual signal is additively integrated with the canonical sampled-token reverse-KL reward: $\(r_t = -\hat{c}_{\text{KL}}(t) + \lambda \hat{\delta}(t), \quad \hat{c}_{\text{KL}}(t) = \log \pi_\theta(y_t \mid h_t) - \log P_T(y_t \mid h_t)\)$ where \(\lambda\) (default \(2.0\)) governs shaping intensity. This formulation exhibits two vital properties. First, it features an exact zero-shift reduction: whenever the aggregate counterfactual shift across constraints sums to zero (\(\sum_i \delta_i(t) = 0\)), the shaping term vanishes (\(\hat{\delta}(t) = 0\)), causing \(r_t\) to reduce precisely to standard sampled-token OPD. Second, it is completely backbone-agnostic: the shaping term derives strictly from teacher-side forced scoring and does not alter the functional form of the student's policy or KL estimator, allowing seamless deployment within any token-level PPO or RL framework.

Loss & Training

Student optimization uses sampled-token PPO without a learned value network or GAE. The policy gradient objective is formulated as: $\(L_{\text{PPO}}(\theta) = -\mathbb{E}_{(q, C), y \sim \pi_{\theta_{\text{old}}}}\left[ \sum_{t=1}^L \min\left( \frac{\pi_\theta(y_t \mid h_t)}{\pi_{\theta_{\text{old}}}(y_t \mid h_t)} \hat{A}_t, \, \text{clip}\left(\frac{\pi_\theta(y_t \mid h_t)}{\pi_{\theta_{\text{old}}}(y_t \mid h_t)}, 1-\epsilon, 1+\epsilon\right) \hat{A}_t \right) \right]\)$ The per-token advantage \(\hat{A}_t\) is computed via group normalization of \(r_t\) across \(8\) rollouts per prompt with PPO clip parameter \(\epsilon = 0.2\). Distributed training is executed on Ray clusters using the verl library with AdamW optimizer, a constant learning rate of \(2 \times 10^{-6}\), and no reference policy KL penalty.

Key Experimental Results

Main Results

Evaluation spans two distinct model families: the Qwen2.5 pair (student: Qwen2.5-1.5B-Instruct; teacher: Qwen2.5-7B-Instruct adapted with GRPO-IR on HIR-16K) and the same-size Qwen3 pair (student: Qwen3-4B-Instruct-2507; teacher: Qwen3-4B-Instruct adapted with GRPO-IR). Models are tested across seven benchmarks: IFEval, IFBench, MulDimIF, ComplexBench (ComplexB), InfoBench (InfoB), FollowBench (FollowB), and CFBench (CFB), reporting strict instruction-level compliance accuracy (%).

Model / Method IFEval IFBench MulDimIF ComplexB InfoB FollowB CFB Avg.
Qwen2.5 Family (1.5B Student / 7B Teacher)
Untrained Student (Qwen2.5-1.5B-Instruct) 43.6 15.3 23.6 48.2 37.0 34.2 24.1 32.3
Teacher Model (Qwen2.5-7B + GRPO-IR) 80.5 32.1 70.1 66.7 52.1 51.9 51.2 57.8
SFT Baseline 66.7 21.6 65.4 50.8 41.9 38.2 29.6 44.9
GRPO-IR (Instruction-level Reward) 54.3 18.6 36.0 51.1 39.9 35.0 27.8 37.5
GRPO-CR (Constraint-level Mean Reward) 56.6 19.2 49.0 52.3 44.1 39.6 31.4 41.7
RTT-GRPO (Learned Token Discriminator) 52.3 18.0 38.8 50.9 39.9 37.3 26.8 37.7
Vanilla OPD (Sampled-Token) 70.7 22.8 69.5 54.4 43.0 38.7 34.4 47.6
Teacher-TopK RKL (k=32) 68.0 24.2 67.5 53.5 42.9 35.7 33.8 46.5
EOPD (Entropy-aware, k=16) 69.7 24.2 70.1 54.5 43.3 37.7 33.6 47.6
ExOPD (Extrapolation, λ=2.0) 75.0 22.4 71.3 53.8 43.7 39.1 36.5 48.9
OPD-aug-LOO (Data Augmentation) 72.1 23.7 69.1 54.3 42.5 39.0 33.8 47.8
CC-OPD (Ours) 76.0 28.2 72.9 54.8 46.4 43.8 37.8 51.4
Qwen3 Family (4B Student / 4B Teacher)
Untrained Student (Qwen3-4B-Instruct) 83.8 29.9 56.9 70.6 55.3 57.8 61.2 59.4
Teacher Model (Qwen3-4B + GRPO-IR) 83.4 32.5 74.3 71.2 57.7 58.1 62.7 62.8
Vanilla OPD 84.8 29.8 74.0 69.3 55.8 57.4 62.0 61.9
ExOPD (λ=1.25) 85.8 31.6 74.5 70.0 57.7 55.7 63.4 62.7
CC-OPD (Ours) 86.1 37.6 76.2 71.0 57.4 58.4 63.4 64.3

Ablation Study

Ablations on counterfactual prompt construction were conducted on the Qwen2.5-1.5B student under identical hyperparameters: 1. LOO (CC-OPD): Leave-one-out context \(\ell(C) - \ell(C \setminus \{c_i\})\); 2. LA (Leave-All): Single global contrast between full and empty context \(\ell(C) - \ell(\emptyset)\); 3. RSS (Random Subset): Contrasting full context with random subset removals \(C \setminus S_j\); 4. NAO (No-context Anchor Only): Single-constraint singleton contrast \(\ell(\{c_i\}) - \ell(\emptyset)\) with no shared context.

Counterfactual Design IFEval IFBench MulDimIF ComplexB InfoB FollowB CFB Avg.
LOO (CC-OPD Default) 76.0 28.2 72.9 54.8 46.4 43.8 37.8 51.4
LA (Leave-All) 75.0 27.1 71.0 53.5 45.3 43.5 38.0 50.5
RSS (Random Subsets) 75.5 28.8 72.1 52.9 44.5 42.6 37.3 50.5
NAO (No-context Anchor) 71.2 26.5 66.4 54.6 45.0 41.7 35.7 48.7

On the MulDimIF difficulty splits (L1 to L4), LOO consistently outperforms LA across all tiers: L1 reaches 89.2% vs. 86.9% (+2.3 pp); L2 reaches 75.3% vs. 74.8% (+0.6 pp); L3 reaches 67.6% vs. 65.3% (+2.2 pp); and the hardest L4 tier reaches 59.4% vs. 57.1% (+2.3 pp).

Key Findings

  • Students Crossing Teacher Ceilings: In the Qwen2.5 setting, the 1.5B student achieves 72.9% on MulDimIF, surpassing the 7B teacher (70.1%) by +2.8 pp overall, and expanding to +6.3 pp on the saturated Level-4 split (59.4% vs. 53.1%). In the same-size Qwen3 setting, the student surpasses its teacher across five out of seven benchmarks, lifting overall average accuracy from 62.8% to 64.3%.
  • Exceptional FLOPs and Step Efficiency: To match the peak smoothed performance of the vanilla OPD baseline on MulDimIF, CC-OPD requires 15.2× fewer gradient steps overall and 16.6× fewer steps on Level-4. Despite incurring \(|C| + 1\) teacher forward evaluations per step, CC-OPD uses approximately 30% of baseline FLOPs on Qwen2.5 and 53% on Qwen3 at matched accuracy.
  • Empirical Rejection of Constraint Additivity: Analyzing raw unclipped token shifts across 15,450 samples reveals a relative \(L_1\) residual of 0.918 between LOO and LA, accompanied by a 31.9% sign disagreement. This confirms that teacher responses to multiple concurrent constraints cannot be modeled by independent additive effects, validating LOO's theoretical higher-order weighting.

Highlights & Insights

  • Inverting the Supervision Direction: Standard distillation attempts to feed students or teachers richer prompts or auxiliary discriminators; CC-OPD instead ab शिविर-interrogates the frozen teacher via counterfactual context removal, extracting dense supervision without modifying the student's runtime deployment.
  • Zero-Cost Credit Assignment: By exploiting teacher log-likelihood sensitivities, the framework automatically Pinpoints constraint-critical tokens (e.g., list markers, format colons, and keyword mentions) without relying on expensive LLM judges or brittle rule verifiers.
  • Theoretical Coherence via Möbius Inversion: The method algebraically connects leave-one-out differences to cardinality-weighted interaction terms on Boolean lattices, while providing guaranteed fallback to vanilla OPD when aggregate constraint shifts nullify.

Limitations & Future Work

  • White-Box Teacher Requirement: The approach mandates white-box log-probability access to calculate conditional distributions under counterfactual prompts, precluding application to black-box proprietary APIs.
  • Span-Excision Assumption: Prompt ablation relies on clean lexical deletion of constraint clauses; non-inline constraints, template-based structures, or implicit conversational requirements remain an open challenge.
  • Per-Step Memory Overhead: Performing \(|C| + 1\) forward passes per rollout increases per-step memory footprint and forward compute, which may constrain throughput in low-memory GPU clusters.
  • vs. Canonical OPD / DistiLLM / GKD: Standard on-policy distillation operates on joint instruction prompts, suffering from constraint dilution; CC-OPD preserves reverse-KL optimization while injecting fine-grained counterfactual reward shaping that enables students to surpass their teachers.
  • vs. Rubric-based RL (ROPD, RaR, RTT-GRPO): Rubric RL relies on external verifiers to score outputs, introducing grader noise and reward hacking; CC-OPD derives native, token-level supervision directly from the teacher's internal probabilistic differentials.
  • vs. Contrastive Guidance (CFG-LM, DR-CFG): DR-CFG applies a single global conditional vs. unconditional contrast; CC-OPD fine-grains contrastive scoring into an ablated multi-constraint sensitivity map tailored for complex composite instructions.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant inversion of the distillation supervision paradigm via leave-one-out counterfactual teacher scoring.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across 2 model scales, 7 benchmarks, and validated with in-depth Möbius theoretical analysis and token heatmaps.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity, mathematically rigorous formulations, and insightful discussion of empirical dynamics.
  • Value: ⭐⭐⭐⭐⭐ Offers a highly impactful, verifier-free post-training framework for complex multi-constraint alignment.