Skip to content

One Instruction Is a Benchmark: End-to-End Instruction-Following Evaluation with FlexBench

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Coryyy/FlexBench
Area: LLM Reasoning
Keywords: Instruction Following, Benchmark Construction, Multi-Turn Dialogue, Precondition-Aware Evaluation, LLM-as-a-Judge

TL;DR

Addressing the disconnect between aggregate public leaderboards and concrete deployment needs, this paper presents FlexBench/FlexEval, an end-to-end framework that instantiates an instruction-specialized test suite from a single target instruction by deriving verifiable atomic dimensions, generating profile-conditioned multi-turn dialogues, and disentangling workflow progress (Coverage) from conditional correctness (Achievement) via precondition-aware tri-valued judging.

Background & Motivation

As large language models (LLMs) transition from exploratory research into mission-critical enterprise workflows, their conversational behavior is increasingly dictated by lengthy, specialized, and highly structured target instructions—such as standard operating procedures (SOPs), customer-service state machines, and multi-step branching policies. Despite this shift toward complex, real-world task execution, prevailing evaluation practices remain anchored to static, aggregate public leaderboards (such as IFEval, FollowBench, and ComplexBench). While these fixed benchmarks provide valuable macro-level comparisons across standard constraints, aggregate leaderboard rankings serve as an exceedingly weak proxy for whether a specific model can faithfully execute the bespoke instruction that a practitioner actually intends to deploy. In engineering practice, the central operational question is not “which model performs best across an arbitrary web corpus?”, but rather “which model most reliably follows this specific business instruction without skipping critical verification steps or botching conditional actions?”

This evaluation gap stems from the intrinsically instruction-conditioned nature of instruction-following capability: models exhibit stark performance shifts across different workflow topologies, conditional branching structures, and domain constraints. Empirical evaluations reveal that a model excelling on academic QA benchmarks frequently fails on multi-step third-party logistics (3PL) workflows or falters during recovery from conversational FAQ interruptions. Even worse, the top-ranked model on an aggregate benchmark is routinely surpassed by alternate models when evaluated on specific target instructions. Meanwhile, emerging dynamic benchmark generation frameworks predominantly generate evaluation items from documents, interactive agent environments, or synthetic question pools; they are fundamentally not designed to take an arbitrary, user-specified target instruction as the seed and automatically synthesize a tailored, high-fidelity, and reproducible test suite around it.

To bridge the chasm between static global rankings and bespoke deployment evaluation, this paper introduces a paradigm shift: treating any single target instruction itself as an instantiated benchmark. Core idea: We propose FlexBench/FlexEval, an end-to-end instruction-conditioned evaluation framework that takes an arbitrary user-provided target instruction, automatically derives a verifiable atomic dimension set via a four-stage verification loop, synthesizes profile-controlled multi-turn dialogue rollouts, and aggregates tri-valued judgments into two decoupled, logic-aware metrics: Coverage (measuring workflow progress) and Achievement (measuring conditional correctness under activated preconditions).

Method

Overall Architecture

The FlexBench/FlexEval framework is structured into two synergistic modules: FlexBench handles benchmark construction by taking a single seed instruction \(I\) and instantiating an instruction-specialized evaluation suite \(B(I) = (D, C)\), comprising a set of verifiable evaluation dimensions \(D = \{d_i\}_{i=1}^{|D|}\) and a multi-turn conversation corpus \(C = \{c_j\}_{j=1}^{M}\); FlexEval subsequently evaluates model execution traces \(\tau \in C\) using tri-valued LLM-as-a-judge scoring (yes, no, unknown) structured across extracted precondition logic groups. The pipeline takes an arbitrary long-form instruction as input and outputs fine-grained dimension-level compliance diagnoses alongside dual decoupled metrics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Instruction I"] --> B["Stage 1: Four-Stage Dimension Derivation<br/>Segmentation→Criteria→Verification→Modification"]
    B --> C["Stage 2: Profile-Conditioned Conversation Rollout<br/>Meta-profile simulation + instruction background"]
    C --> D["Stage 3: Precondition-Aware Tri-Valued Judging<br/>yes / no / unknown verdict labeling"]
    D --> E["Stage 4: Logic Grouping & Decoupled Aggregation<br/>Coverage progress and Achievement correctness"]

Key Designs

1. Four-Stage Verifiable Dimension Derivation: Instruction Ambiguity and Implicit Dependencies -> Structured Verification Loop -> Ensuring Dimension Atomicity and Observability
Directly extracting compliance checklists from long, complex instructions often suffers from omission of subtle conditional branches or phrasing of criteria that are unobservable in conversation traces. FlexBench implements a rigorous four-stage iterative pipeline: Instruction Segmentation splits the instruction into non-overlapping, behaviorally relevant requirement units while strictly forbidding the hallucination of external constraints; Criteria Generation rewrites each unit into an entity-explicit, conversation-observable criterion using formal conditional syntax ("if/when"); Coverage Verification employs an independent LLM verifier to audit whether the candidate dimension set comprehensively covers all observable requirements; finally, Targeted Modification addresses verifier critiques through schema-preserving revisions until full coverage is certified or the retry budget is exhausted. This yields an atomic, complete, and reproducible set of observable evaluation criteria.

2. Profile-Conditioned Multi-Turn Dialogue Rollout: Narrow Distribution and Contamination Risk -> Stratified User Simulation -> Eliciting Latent Branch Trajectories
To thoroughly stress-test evaluated models without leaking the ground-truth evaluation criteria, the conversation generation module decouples user simulation from dimension definitions. FlexBench defines a suite of meta-profiles characterizing five distinct levels of user cooperativeness, ranging from uncooperative information-withholding to proactive task collaboration. These profiles are parameterized alongside instruction-specific background facts to drive an interactive user simulator. The simulator only observes the original instruction context, its assigned persona, and dialogue history, remaining strictly blind to the derived evaluation dimensions and logic groups. Multi-turn interactions are generated across multiple rollout seeds until task completion, simulator termination, or turn budget exhaustion, ensuring trajectory diversity and behavioral fidelity.

3. Precondition-Aware Logic Grouping and Tri-Valued Judging: False-Negative Penalties on Untriggered Paths -> Gated Dependency Structuring -> Eliminating Inactive Branch Penalization
In realistic multi-turn workflows involving branching or recovery, many rules are conditionally gated or mutually exclusive (e.g., issuing an advance warning only if a requested booking date is less than five days away). Under conventional binary grading, inactive branches are erroneously scored as model failures. FlexEval introduces a tri-valued verdict space: yes (applicable and satisfied), no (applicable but violated), and unknown (insufficient evidence or precondition not triggered). A programmatic logic extractor parses dimensions into logic groups, each containing a foundational base action and its dependent branch criteria. A branch criterion is only activated for evaluation when its associated base action is explicitly attempted in the dialogue; untriggered or mutually exclusive branches remain marked as unknown and are excluded from the correctness denominator.

4. Decoupled Aggregation of Coverage and Achievement: Conflating Omission with Execution Error -> Dual Metric Formulation -> Precise Failure Attribution
To deliver actionable diagnostics for deployment engineers, FlexEval separates workflow progression from execution fidelity via two complementary metrics. Coverage measures workflow exposure—specifically, the fraction of required base actions that the evaluated model attempts (judged yes or no) across the dialogue corpus, diagnosing whether steps were skipped. Conversely, Achievement measures conditional correctness—the proportion of fully applicable branch requirements that are satisfied (yes) once triggered. This dual formulation prevents conflating the failure to reach a workflow stage with making an error within that stage, enabling practitioners to isolate whether a model suffers from premature exit or execution inaccuracy.

Benchmark & Evaluation Protocol

FlexBench/FlexEval operates as an automated, inference-time benchmark construction and evaluation system governed by closed-form statistical aggregation formulas. Given an interaction corpus \(C(I)\) and derived dimension set \(D\), let \(s_d(\tau) \in \{y, n, u\}\) represent the tri-valued verdict for dimension \(d\) on trace \(\tau\). Over the set of extracted base actions \(B\) and branch dimensions \(R\), Coverage and Achievement are formally defined as:

\[\text{Coverage}(I) = \frac{\sum_{b \in B} (N_y(b) + N_n(b))}{\sum_{b \in B} (N_y(b) + N_n(b) + N_u(b))}\]
\[\text{Achievement}(I) = \frac{\sum_{r \in R} \tilde{N}_y(r)}{\sum_{r \in R} (\tilde{N}_y(r) + \tilde{N}_n(r))}\]

where the gated branch count \(\tilde{N}_v(r)\) for verdict \(v \in \{y, n\}\) conditioned on attempted base action \(b(r)\) is given by:

\[\tilde{N}_v(r) = \sum_{\tau \in C(I)} \mathbb{I}[s_{b(r)}(\tau) \in \{y, n\}] \cdot \mathbb{I}[s_r(\tau) = v]\]

In the primary empirical validation, Claude Opus 4.6 (temperature=0) serves as the fixed backend for dimension derivation, logic extraction, and FlexEval judging; Gemini 3.1 Pro Preview (temperature=0.7) serves as the multi-turn user simulator, and Qwen-Max (temperature=0.1) operates as the dialogue completion detector. For each of the 9 seed instructions and 9 evaluated models, 48 dialogues are synthesized across controlled profile distributions, yielding 3,888 dialogues and 60,480 individual dimension-dialogue judgments.

Key Experimental Results

Main Results

The primary benchmark evaluates 9 leading proprietary and open-weights LLMs across 9 multi-turn target instructions categorized into business, academic, and personal scenarios spanning five distinct workflow topologies (C1: Linear SOP, C2: Branch/Rejoin, C3: Multi-way Routing, C4: Nested Guidance, C5: FAQ/Recovery). The results demonstrate that model suitability is strongly instruction-conditioned, with relative model superiority fluctuating across instructions.

Model FastRun (C1/C2) 3PL (C3/C2/C5) Course (C4/C2/C3/C5) Exam (C2) IT (C3) 2FA (C4/C2) Museum (C1) Travel (C3/C2) ConfReg (C5/C2) Avg. Std.
Claude Sonnet 4.5 0.279 0.504 0.468 0.603 0.436 0.542 0.703 0.514 0.525 0.508 0.110
DeepSeek-V4-Pro 0.227 0.474 0.360 0.581 0.460 0.561 0.671 0.494 0.475 0.478 0.121
Doubao-Seed-2.0-Pro 0.253 0.426 0.358 0.623 0.467 0.563 0.674 0.468 0.463 0.477 0.122
GPT-4.1 0.203 0.458 0.368 0.604 0.496 0.553 0.617 0.491 0.496 0.476 0.120
Gemini 2.5 Pro 0.211 0.524 0.336 0.576 0.495 0.540 0.601 0.497 0.500 0.476 0.117
GPT-4.1-mini 0.263 0.392 0.368 0.579 0.470 0.518 0.673 0.518 0.467 0.472 0.114
Qwen3-235B-A22B 0.234 0.348 0.343 0.537 0.459 0.530 0.564 0.529 0.515 0.451 0.108
GLM-4.5 0.135 0.403 0.322 0.528 0.396 0.509 0.638 0.429 0.467 0.425 0.133
GPT-4.1-nano 0.276 0.214 0.186 0.567 0.407 0.478 0.567 0.463 0.434 0.399 0.135

Note: Scores reflect ACHIEVEMENT judged by Claude Opus 4.6. Best performance per column is bolded.

Ablation & Diagnostic Analyses

The following table presents the joint seed-level diagnostic view of mean Achievement, mean Coverage, and Uncovered Rate (\(1 - \text{Coverage}\)) across the evaluated models.

Seed Instruction Scenario Category Workflow Topology #Dims Mean Achievement (%) Mean Coverage (%) Uncovered Rate (%)
FastRun Business C1, C2 8 23.1 100.0 0.0
3PL Business C3, C2, C5 42 41.6 26.3 73.7
Course Business C4, C2, C3, C5 24 34.6 98.2 1.8
Exam Academic C2 9 57.8 100.0 0.0
IT Academic C3 12 45.4 99.3 0.7
2FA Academic C4, C2 10 53.3 99.5 0.5
Museum Personal C1 12 63.4 88.1 11.9
Travel Personal C3, C2 13 48.9 100.0 0.0
ConfReg Personal C5, C2 10 48.2 99.9 0.1

External benchmark alignment and reliability evaluations demonstrate: - Semantic Recovery of Human Decompositions: Across 448 external instructions from InFoBench Hard and ComplexBench, FlexBench achieves an overall Precision of 79.5%, Recall of 80.6%, and F1 of 80.0%, with a granularity ratio of 1.160 indicating slightly finer atomicity than human annotations. - Preference-Based Evaluation: In randomized pairwise judging by GPT-5.4, FlexBench decompositions are preferred over human reference decompositions in 75.0% of cases (64.9% on InFoBench, 87.5% on ComplexBench), demonstrating highest superiority on evaluation suitability (74.8% vs. 24.1%) and atomicity (66.7% vs. 24.3%). - Judge Reliability: Human meta-evaluation over 360 business dialogues (8,880 dimension judgment instances) demonstrates 86.25% to 88.02% agreement with the Claude Opus 4.6 judge, achieving an overall three-label Cohen's \(\kappa\) of 0.772.

Key Findings

  • Instruction-Conditioned Leader Shifting: Although Claude Sonnet 4.5 attains the highest point-estimate mean Achievement (0.508), it is outperformed on individual seeds: Gemini 2.5 Pro leads on 3PL (0.524), Doubao leads on Exam (0.623) and 2FA (0.563), GPT-4.1 leads on IT (0.496), and Qwen leads on Travel (0.529). Bootstrap resampling (10,000 replicates) indicates that best-versus-runner-up confidence intervals span zero on 8 of 9 seeds, confirming that a single leaderboard winner cannot be assumed across workflows.
  • Topological Divergence in Workflow Coverage: While Coverage reaches near saturation (\(\ge 98\%\)) on linear or moderately branched instructions, the complex, highly branched 3PL workflow exhibits an average Coverage of only 26.3% (uncovered rate of 73.7%). Decoupled evaluation prevents penalizing models for unexercised paths, maintaining interpretability for Achievement.
  • Cross-Backend Pipeline Stability: Re-running dimension derivation, logic extraction, and judging using alternate backends (GPT-5.4 and Gemini 3.1 Pro Preview) maintains strong model-level rank consistency: Claude vs. GPT-5.4 yields a Spearman rank correlation of 0.917 (Pearson 0.984), while Claude vs. Gemini yields 0.883 (Pearson 0.976), verifying that findings are not artifacts of a single judge LLM.

Highlights & Insights

  • From Fixed Benchmarks to On-Demand Benchmark Synthesis: FlexBench establishes a paradigm where developers can deploy custom conversational SOPs and instantly synthesize an instruction-grounded, multi-turn evaluation benchmark without manual labeling.
  • Precondition-Gated Tri-Valued Judging Eliminates Branch Penalties: Incorporating an explicit unknown state for unexercised conditional paths solves the structural false-negative dilemma of multi-turn dialogue evaluation, preserving pure execution correctness scores.
  • Orthogonal Separation of Progress from Precision: By formalizing Coverage (did the model execute the workflow?) and Achievement (did the model follow applicable rules?), the framework provides root-cause diagnostic clarity for agentic systems.

Limitations & Future Work

  • Dependency on Seed Instruction Self-Sufficiency: FlexBench relies heavily on the internal consistency of the target instruction. When seed instructions contain severe semantic ambiguity or underspecified domain knowledge, the dimension derivation loop can over-decompose or misinterpret implicit constraints.
  • Inference Compute for Dense Trajectory Coverage: In branch-heavy workflows with 40+ dimensions, adequately saturating all conditional pathways requires scaling the number of simulated dialogue rollouts, increasing judging token costs.
  • Future Directions: The authors highlight extending FlexBench to multi-backend consensus judging, real-user interactive trace ingestion, multimodal dialogue rollouts, and long-horizon dependency modeling spanning 100+ conversational turns.
  • vs. IFEval / FollowBench / CFBench: Prior benchmarks rely on fixed prompt templates and predefined constraints; FlexBench automates the on-demand generation of bespoke benchmark test suites directly from any user-provided instruction.
  • vs. ComplexBench / StructFlowBench: While introducing structural dependencies, earlier works focus on static composition or simple multi-turn sequences; FlexEval provides formal precondition logic grouping and decoupled dual metrics.
  • vs. AgentIF / NetArena: Agentic benchmarks typically require rigid, sandboxed execution environments (e.g., shell terminals or network emulators); FlexBench achieves rigorous stress-testing of stateful, branching workflows within an agile conversational simulation framework.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the instruction-conditioned evaluation setting and solves the end-to-end benchmark synthesis pipeline.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluation across 9 models, 3,888 dialogues, human validation, bootstrap uncertainty quantification, and multi-backend cross-auditing.
  • Writing Quality: ⭐⭐⭐⭐⭐ Cohesive, precise narrative exposition with clear mathematical formulations and comprehensive appendices.
  • Value: ⭐⭐⭐⭐⭐ Highly actionable for AI practitioners needing custom, objective evaluations for enterprise LLM agents and conversational workflows.