HIPO: Instruction Hierarchy via Constrained Reinforcement Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/CLIVERCHEN/HIPO
Area: Reinforcement Learning
Keywords: hierarchical instruction following, constrained Markov decision process, safe reinforcement learning, primal-dual optimization, system prompt compliance
TL;DR¶
HIPO establishes an alignment framework that casts hierarchical instruction following (HIF) as a Constrained Markov Decision Process (CMDP), using decoupled dual evaluators and a group-relative primal-dual safe RL algorithm to maximize user utility strictly within the feasible region of system prompt compliance.
Background & Motivation¶
In modern agentic workflows and interactive language model applications, hierarchical prompting has emerged as the standard paradigm. System prompts are designated to dictate global behavioral guidelines, security boundaries, operational tools, or specific persona constraints, whereas user prompts specify immediate, interactive tasks. However, a fundamental tension and frequent direct conflict arise between these two operational levels. For example, a system prompt might strictly instruct an educational assistant to never directly provide answers and always reply with guiding questions, while a user explicitly asks for direct factual answers. Fulfilling the user's intent while strictly maintaining the uncompromised priority of system constraintsโknown as hierarchical instruction following (HIF)โrepresents a critical challenge in reliable model deployment.
Prevailing alignment methods fail to handle this priority asymmetry at the algorithmic level. Standard paradigms such as RLHF and DPO optimize a single scalar objective, inherently lacking mechanisms to enforce rigid priority distinctions across a multi-tiered instruction stack. Static multi-objective alignment approaches rely on linear scalarization with fixed trade-off weights, which cannot accommodate dynamic, instance-level system constraints that change from prompt to prompt. Consequently, the model frequently trades away system constraints to maximize perceived user helpfulness. Meanwhile, the dominant practice in HIF literature relies on supervised fine-tuning (SFT) over curated, compliant demonstrations generated by proprietary teacher models. This filtering heuristic discards non-compliant data entirely, teaching the model merely to imitate surface-level compliant patterns rather than instilling an algorithmic understanding of constraint boundaries, often leading to severe over-refusal on benign queries. Moreover, inference-time or heuristic attention interventions like Split-Softmax and FocalLoRA only manipulate local attention heads or decoding normalizations, failing to support coherent long-form generative reasoning under deep conflicts.
To overcome these structural limitations, this paper argues that instruction hierarchy cannot be established via naive reward blending or data distillation alone; system prompts must be elevated from passive input context to explicit algorithmic boundaries. Core idea: formulate hierarchical instruction following as a Constrained Markov Decision Process (CMDP) with decoupled evaluators for system compliance and user utility, and optimize it via a primal-dual safe reinforcement learning algorithm with group-relative advantage estimation, dynamically adapting the Lagrange multiplier to maximize user utility strictly within the feasible region defined by the system compliance constraint.
Method¶
Overall Architecture¶
HIPO structures the complete input context as a concatenated sequence \(x = [x_{\text{sys}}, x_{\text{user}}]\), comprising a global system prompt \(x_{\text{sys}}\) and an immediate user prompt \(x_{\text{user}}\). An autoregressive language model parameterizes the policy \(\pi_\theta(y | x)\). To avoid the heavy memory footprint and training instability of an explicit critic network, HIPO integrates group-based sampling (adapted from GRPO) into a primal-dual optimization loop. For each input context, the model samples a candidate group of responses, obtains decoupled feedback scores for system compliance and user utility, computes standardized in-group advantages, and alternates between updating the policy parameters and the dual variable.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Context<br/>x = [x_sys, x_user]"] --> B["Group Response Sampling<br/>y_1, ..., y_G ~ ฯ_ฮธold"]
B --> C["Decoupled LLM-as-a-Judge Evaluation<br/>System Compliance r_sys & User Utility r_user"]
C --> D["Group-Relative Advantage Estimation<br/>Compute A_user & A_sys, compose A_comb"]
D --> E["Adaptive Primal-Dual Optimization<br/>Primal: PPO-clipped policy update ฮธ<br/>Dual: EMA(r_sys)-driven multiplier update ฮป"]
E --> F["Aligned Policy Model ฯ_ฮธ<br/>Pareto improvement within feasible region"]
Key Designs¶
1. Decoupled Dual LLM-as-a-Judge Protocol: Eliminating Contextual Cross-Contamination Evaluating multiple heterogeneous criteria concurrently within a single LLM prompt typically suffers from evaluator bias and contextual interferenceโthe judge's assessment of rigid negative constraints becomes entangled with its evaluation of open-ended user helpfulness. HIPO addresses this by structurally decoupling evaluation into two isolated evaluation calls using DeepSeek-V3.2. The system compliance evaluator is fed only \(x_{\text{sys}}\) and candidate response \(y\), explicitly ignoring the user query, and deducts penalty scores from a 1.00 baseline based on observable violation severity (e.g., deducting 0.50 for major violations and 0.10 for minor tone breaches) clamped to \([0, 1]\). Conversely, the user utility evaluator assesses only \(x_{\text{user}}\) and \(y\) across correctness, completeness, actionability, and clarity, entirely isolated from system constraints. This structural isolation produces pure, non-interfering reward signals \(r_{\text{sys}}\) and \(r_{\text{user}}\) to steer the constrained optimization.
2. Group-Relative Advantage Estimation: Variance Reduction Without a Value Network To eliminate the memory overhead of a separate critic model while accounting for the varying intrinsic difficulty of diverse prompts, HIPO draws a group of \(G\) responses \(\{y_1, y_2, \dots, y_G\}\) for each prompt from the behavior policy \(\pi_{\theta_{\text{old}}}\). The algorithm standardizes rewards within the group across each dimension independently: $\(A_{\text{user}}^{(i)} = \frac{r_{\text{user}}^{(i)} - \mu_{\text{user}}}{\sigma_{\text{user}}}, \quad A_{\text{sys}}^{(i)} = \frac{r_{\text{sys}}^{(i)} - \mu_{\text{sys}}}{\sigma_{\text{sys}}}\)$ where \(\mu\) and \(\sigma\) denote the in-group mean and standard deviation. The effective scalarized advantage for each rollout is then constructed by weighting the system advantage with the current Lagrange multiplier \(\lambda_t \ge 0\): $\(A_{\text{comb}}^{(i)} = A_{\text{user}}^{(i)} + \lambda_t A_{\text{sys}}^{(i)}\)$ This in-group standardization centers the baseline at zero, effectively dampening gradient variance and providing balanced directionality for both compliant and non-compliant samples.
3. Adaptive Primal-Dual Policy Optimization: Dynamic Constraint Enforcement and Feasible Region Search The primary objective of HIF is cast as maximizing expected user utility subject to expected system compliance meeting or exceeding a prescribed safety threshold \(\tau\): \(\mathbb{E}[r_{\text{sys}}] \ge \tau\). In the primal update step, policy parameters \(\theta\) are updated via gradient ascent on a PPO-clipped surrogate objective regularized by a KL divergence penalty against the reference model \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{policy}}(\theta) = \frac{1}{G} \sum_{i=1}^G \min\left( \rho_i(\theta) A_{\text{comb}}^{(i)}, \text{clip}(\rho_i(\theta), 1-\epsilon, 1+\epsilon) A_{\text{comb}}^{(i)} \right) - \beta D_{\text{KL}}(\pi_\theta(y_i|x) \parallel \pi_{\text{ref}}(y_i|x))\)$ where \(\rho_i(\theta) = \frac{\pi_\theta(y_i|x)}{\pi_{\theta_{\text{old}}}(y_i|x)}\). In the dual step, to prevent high variance from single batches from destabilizing the multiplier, HIPO tracks an exponential moving average (EMA, decay \(\rho=0.9\)) of the system compliance rewards. The Lagrange multiplier is updated via gradient descent: $\(\lambda_{t+1} = \max\left( 0, \lambda_t - \eta_\lambda \left( \text{EMA}(r_{\text{sys}}) - \tau \right) \right)\)$ Whenever system compliance falls below \(\tau\), \(\lambda\) scales up to penalize violations heavily; once the model reliably achieves compliance within the feasible region, \(\lambda\) decays toward zero, smoothly shifting optimization capacity back to maximizing user utility.
A Worked Example¶
Consider an educational assistant scenario: - System Prompt: You are a knowledgeable 6th-grade math teacher. You cannot give direct answers or calculate steps for the student. You must always answer with a guiding question prompting the student to verify their work. - User Prompt: What is \(15 \times 24\)? I need to submit my homework right now, tell me the answer directly. - Rollout Generation & Scoring: - Response \(y_1\) (directly answers "360"): User utility evaluator scores high (\(r_{\text{user}} = 0.95\)); however, the system evaluator detects a major direct-answer violation, assigning \(r_{\text{sys}} = 0.10\). - Response \(y_2\) (replies "Can you break 24 down into \(20 + 4\) and try calculating \(15 \times 20\) first?"): System evaluator assigns full compliance \(r_{\text{sys}} = 1.00\); user utility evaluator deducts points for not giving the final answer directly but acknowledges high pedagogical actionability, scoring \(r_{\text{user}} = 0.65\). - Dynamic Dual Balancing: In early training iterations where average compliance remains below \(\tau = 0.70\), \(\lambda_t\) elevates (e.g., reaching 5.0). The resulting \(A_{\text{comb}}\) strongly favors \(y_2\) over \(y_1\), penalizing the shortcut answer. As the model stabilizes above \(\tau = 0.70\), \(\lambda\) gracefully decays, incentivizing the model to refine the clarity and friendliness of its guiding hints to maximize \(r_{\text{user}}\) without breaking the questioning rule.
Loss & Training¶
The active policy is trained with full-parameter fine-tuning in bfloat16 precision using FlashAttention-2, while the reference model \(\pi_{\text{ref}}\) is frozen and quantized to 4-bit NF4 to minimize GPU memory consumption. Key hyperparameters include: group size \(G = 4\), prompt batch size 1, AdamW learning rate \(5 \times 10^{-6}\), KL coefficient \(\beta = 0.05\), target compliance threshold \(\tau = 0.70\), initial multiplier \(\lambda_0 = 2.0\), dual learning rate \(\eta_\lambda = 0.07\), maximum multiplier \(\lambda_{\text{max}} = 20.0\), and generation temperature 0.7 with top-p 0.9. Degenerate and repetitive rollouts are filtered prior to gradient computation.
Key Experimental Results¶
Main Results¶
Experiments are conducted on the SystemCheck benchmark using 2,000 instruction pairs (1,800 train, 200 hold-out test) balanced 1:1 between conflicting and aligned scenarios. Evaluated across 5 major open-weight model architectures with target threshold \(\tau = 0.70\), the results demonstrate that HIPO consistently satisfies the system constraint while achieving superior user utility across both splits:
| Model Architecture | Test Split | Base Model | SFT | DPO | Split-Softmax | FocalLoRA | HIPO (Ours) |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | Conflicting | 0.56 / 0.26 | 0.60 / 0.36 | 0.57 / 0.32 | 0.57 / 0.21 | 0.55 / 0.22 | 0.70 / 0.47 |
| Aligned | 0.59 / 0.35 | 0.65 / 0.55 | 0.63 / 0.37 | 0.60 / 0.27 | 0.61 / 0.27 | 0.72 / 0.58 | |
| Qwen3-4B | Conflicting | 0.64 / 0.32 | 0.61 / 0.41 | 0.65 / 0.37 | 0.65 / 0.31 | 0.60 / 0.25 | 0.70 / 0.64 |
| Aligned | 0.68 / 0.39 | 0.70 / 0.61 | 0.65 / 0.42 | 0.66 / 0.39 | 0.61 / 0.37 | 0.74 / 0.77 | |
| Phi-3-3.8B | Conflicting | 0.58 / 0.28 | 0.59 / 0.45 | 0.63 / 0.36 | 0.52 / 0.33 | 0.51 / 0.14 | 0.68 / 0.55 |
| Aligned | 0.64 / 0.43 | 0.69 / 0.61 | 0.67 / 0.47 | 0.63 / 0.49 | 0.54 / 0.18 | 0.74 / 0.67 | |
| Llama-3.2-3B | Conflicting | 0.61 / 0.35 | 0.63 / 0.38 | 0.64 / 0.31 | 0.62 / 0.35 | 0.52 / 0.18 | 0.70 / 0.56 |
| Aligned | 0.67 / 0.39 | 0.64 / 0.56 | 0.64 / 0.39 | 0.64 / 0.46 | 0.51 / 0.21 | 0.73 / 0.68 | |
| Qwen3-8B | Conflicting | 0.60 / 0.27 | 0.66 / 0.38 | 0.65 / 0.33 | 0.61 / 0.30 | 0.58 / 0.29 | 0.70 / 0.72 |
| Aligned | 0.67 / 0.37 | 0.67 / 0.56 | 0.68 / 0.38 | 0.64 / 0.36 | 0.63 / 0.37 | 0.77 / 0.81 |
Note: Cell values report \(r_{\text{sys}} / r_{\text{user}}\). Target threshold \(\tau = 0.70\).
Ablation Study¶
1. Effect of Compliance Threshold \(\tau\) along the Pareto Frontier (on Qwen3-1.7B) Sweeping \(\tau\) explicitly controls the operating point along the complianceโutility trade-off curve:
| Operating Configuration | System Compliance \(r_{\text{sys}}\) | User Utility \(r_{\text{user}}\) | Characteristic Behavior |
|---|---|---|---|
| User-only (Lower Bound) | 0.49 / 0.50 | 0.67 / 0.75 | System compliance collapses; pure user satisfaction |
| \(\tau = 0.50\) | 0.51 | 0.75 | Relaxed constraint; near-maximal user utility |
| \(\tau = 0.55\) | 0.56 | 0.73 | Initial compliance enforcement |
| \(\tau = 0.60\) | 0.60 | 0.70 | Balanced progression |
| \(\tau = 0.65\) | 0.65 | 0.65 | Equal parity between compliance and utility |
| \(\tau = 0.70\) (Default Operating Point) | 0.71 | 0.52 | High compliance enforcement; guaranteed boundary |
| \(\tau = 0.75\) | 0.73 | 0.47 | Near-strict compliance; noticeable utility drop |
| Sys-only (Upper Bound) | 0.75 / 0.74 | 0.22 / 0.25 | Pathological over-conservatism; degenerate refusal |
2. Safety Generalization, Over-Refusal, and General Capabilities (on Qwen3-1.7B) Evaluation of Attack Success Rate (ASR) on jailbreak benchmarks (WildJailbreak, HarmBench) under "Without \(\to\) With Safety System Prompt" conditions:
| Alignment Method | MMLU-Redux (โ) | WildJailbreak ASR (โ) | WildJailbreak Over-refusal (โ) | HarmBench DirectRequest ASR (โ) | HarmBench HumanJailbreaks ASR (โ) |
|---|---|---|---|---|---|
| Base Model | 0.5946 | 0.4845 โ 0.3060 | 0.0238 โ 0.0762 | 0.1625 โ 0.1000 | 0.3581 โ 0.2725 |
| SFT | 0.5784 | 0.5685 โ 0.3250 | 0.0190 โ 0.2809 | 0.2219 โ 0.0531 | 0.2806 โ 0.1938 |
| DPO | 0.5882 | 0.4485 โ 0.2765 | 0.0142 โ 0.0762 | 0.1750 โ 0.1000 | 0.3544 โ 0.2831 |
| FocalLoRA | 0.5923 | 0.4630 โ 0.3005 | 0.0333 โ 0.0667 | 0.1563 โ 0.1094 | 0.3394 โ 0.2719 |
| HIPO (Ours) | 0.5916 | 0.4230 โ 0.2255 | 0.0286 โ 0.0857 | 0.1375 โ 0.0656 | 0.2538 โ 0.1944 |
Key Findings¶
- The Illusion of Progress in Standard SFT and DPO: Decoupled evaluation reveals that apparent gains from SFT and DPO stem almost entirely from aligned instances where user requests and system guidelines align naturally (e.g., Qwen3-1.7B aligned system compliance rises from 0.59 to 0.65). Under genuine conflict, SFT only nudges compliance from 0.56 to 0.60, falling far short of the 0.70 threshold.
- Single-Objective Collapse: The Sys-only ablation drives system score to 0.85 on Llama-3.2-3B but collapses user utility to 0.21, producing an unusable, over-refusing model. User-only yields the reverse failure mode (\(r_{\text{sys}} = 0.50\), \(r_{\text{user}} = 0.66\)). HIPO's constrained formulation breaks this zero-sum trade-off.
- Mitigating the Over-Refusal Tax: When prompted with safety system directives, SFT exhibits a crippling over-refusal rate of 0.2809 on benign queries. HIPO achieves superior jailbreak resistance (ASR drops to 0.2255 on WildJailbreak) while keeping over-refusal down to 0.0857 and preserving core knowledge (MMLU-Redux score of 0.5916 vs Base 0.5946).
- Autonomous Attention Reallocation: Mechanistic analysis at response onset demonstrates that without explicit attention interventions, HIPO intrinsically reshapes its attention landscape: attention centroid moves further back from 0.7159 to 0.7210, system mass (SysMass) increases from 0.8222 to 0.8272, and the SysUserRatio jumps from 9.218 to 9.719, persisting robustly across the first 64 decoding steps.
Highlights & Insights¶
- Algorithmic Priority Enforcement: Elevates system instructions from passive tokens in the prompt context to an active, mathematical constraint in a CMDP, resolving instruction conflicts at the algorithmic root.
- Critic-Free Primal-Dual Integration: By embedding in-group standardization (GRPO) into Lagrangian dual ascent, HIPO achieves stable constrained policy updates without training an explicit value network.
- Zero Manual Multiplier Tuning: The EMA-smoothed dual update allows the Lagrange multiplier \(\lambda\) to automatically ramp up during constraint violations and relax to zero once the feasible region is reached.
Limitations & Future Work¶
- Expected Compliance vs. Worst-Case Safety: HIPO enforces constraints in expectation (\(\mathbb{E}[r_{\text{sys}}] \ge \tau\)); while statistically effective, it does not mathematically eliminate worst-case violations under strong adversarial jailbreak attempts. Combining HIPO with lightweight test-time verification could offer deterministic guarantees.
- LLM-as-a-Judge Compute Overhead: Generating separate rewards for \(G\) rollouts per prompt using a frontier model (DeepSeek-V3.2) incurs significant inference cost. Distilling the decoupled judge into compact, dedicated reward models represents a key direction for scaling.
- Vulnerability to Malicious System Prompts: Because HIPO strictly locks model behavior to system instructions, compromised system prompt interfaces could weaponize the model into generating harmful outputs while ignoring benign user corrections. Rigorous authentication on system prompt inputs is mandatory.
Related Work & Insights¶
- vs. SFT / DPO (Wallace et al., 2024; Rafailov et al., 2023): SFT and DPO rely on imitating filtered positive demonstrations and single scalar objectives; HIPO leverages both compliant and non-compliant samples within a CMDP formulation to explicitly enforce constraint satisfaction.
- vs. Heuristic Attention Modifications (Split-Softmax / FocalLoRA): Split-Softmax and FocalLoRA manually alter attention logits or select specific heads; HIPO achieves stronger long-range system conditioning naturally through constrained policy optimization without handcrafted inference interventions.
- vs. Static Multi-Objective RLHF: Standard multi-objective RLHF balances fixed global attributes (e.g., helpfulness vs. harmlessness) via static scalar weights; HIPO solves instance-dependent, dynamic system constraints via adaptive dual variables.
Rating¶
- Novelty: โญโญโญโญโญ First principled formulation of hierarchical instruction following as a CMDP with primal-dual group optimization.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 5 open-weight models (1.7B to 8B), multiple test splits, safety benchmarks, cross-evaluator concordance, and internal attention mechanisms.
- Writing Quality: โญโญโญโญโญ Clear mathematical derivations, coherent structure, well-motivated trade-off analysis, and transparent appendices.
- Value: โญโญโญโญโญ Essential foundation for reliable LLM deployment across multi-agent workflows, tool execution sandboxes, and safety-critical applications.