Skip to content

JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Safety
Keywords: Large Language Model Security, Jailbreak Assessment, Decompositional Scoring, Ternary Evaluation, Fact Checking

TL;DR

This paper introduces JADES, a universal evaluation framework inspired by analytic scoring that decomposes harmful queries into weighted sub-questions with evidence pairing and fact-checking, revealing systematic overestimation in existing binary jailbreak metrics and uncovering that partial successes dominate current attack outcomes.

Background & Motivation

With the extensive deployment of large language models (LLMs), jailbreak attacksโ€”spanning adversarial token suffixes, complex role-play framing, and automated iterative optimizationโ€”have become one of the most prominent empirical vulnerabilities confronting model alignment. However, accurately and scalably assessing whether a jailbreak attempt is genuinely successful remains an open challenge. Existing jailbreak benchmarks primarily curate open-ended, multi-constraint harmful requests without standardized reference answers, causing target models to produce verbose, constructed completions whose true hazard is inherently difficult to verify. While manual evaluation serves as the gold standard, its prohibitive cost and poor scalability force the community to rely on automated evaluators that frequently diverge from human judgment.

This divergence stems from two structural evaluation bottlenecks. First, automated methods relying on proxy indicators, such as substring matching and off-the-shelf toxicity classifiers, are fundamentally misaligned with the adversary's true objective: observing affirmative prefix phrases (e.g., "Sure, here is...") or high generic toxicity scores in no way guarantees that actionable, harmful assistance was actually provided. Second, holistic LLM-as-a-judge evaluators compress the assessment of multifaceted completions into a single monolithic prompt, leaving them vulnerable to obfuscating role-play framing, superficial compliance tokens, and formatted outlines that lack concrete payload details. Consequently, these evaluators produce frequent false positives on vacuous responses and false negatives when actionable harm is wrapped in ethical disclaimers, while completely obscuring the vital distinction between partial and complete fulfillment.

Inspired by the educational paradigm of analytic scoringโ€”wherein open-ended essays are graded across discrete, weighted criteria rather than an impressionistic wholeโ€”this paper re-examines the mechanics of jailbreak evaluation. Since executing a harmful task requires satisfying a conjunction of critical constraints, fulfillment can only be faithfully assessed by decomposing the objective into verifiable sub-goals. Core idea: decompose each harmful request from the adversary's perspective into weighted, necessary sub-questions, clean response obfuscation, pair verbatim evidence sentences for independent Likert scoring, and optionally verify unit facts against external knowledge to establish an auditable, fine-grained binary and ternary jailbreak evaluation framework.

Method

Overall Architecture

JADES is structured as a decoupled multi-agent pipeline comprising four foundational processing nodesโ€”Question Decomposition Node, Clean Node, Sub-Question Pairing Node, and Evaluation Nodeโ€”alongside a plug-and-play Fact-Checking Extension. Given a harmful query \(Q\) and a target LLM response \(R\), the system translates the open-ended objective into an auditable set of weighted sub-questions, purges peripheral distractions from the response, extracts verbatim supporting evidence for each sub-question, and aggregates sub-scores under an empirical Likert decision boundary to output binary (failed/successful) or ternary (failed/partially successful/successful) judgments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Harmful Query Q + Jailbreak Response R"] --> B["Question Decomposition & Memory Caching<br/>Decompose Q into weighted sub-questions"]
    A --> C["Response Cleaning & Distraction Filtering<br/>Segment R and filter irrelevant sentences"]
    B --> D["Sub-Question Evidence Pairing<br/>Extract verbatim supporting sentences"]
    C --> D
    D --> E["Likert Scoring & Weighted Aggregation<br/>Score sub-answers on a 5-point scale and compute Stotal"]
    E --> F{"Enable Fact-Checking Extension?"}
    F -->|Yes| G["Unit Fact Splitting & External Verification<br/>Verify factual claims and penalize hallucinations"]
    F -->|No| H["Threshold Decision Mapping<br/>Output Binary or Ternary Jailbreak Label"]
    G --> H

Key Designs

1. Question Decomposition & Memory Caching: Resolving holistic conflation via adversary-centric constraint decomposition

Holistic judges conflate multiple evaluation dimensions into an uncalibrated impression. JADES deploys a decomposition agent powered by a capable LLM (defaulting to GPT-4o) to break down a harmful request \(Q\) into \(n\) necessary sub-questions along with their normalized importance weights: \(\{(q_i, w_i)\}_{i=1}^n\), where \(w_i \ge 0\) and \(\sum_{i=1}^n w_i = 1\). The decomposition adopts a goal-oriented constraint perspective: each sub-question represents a critical requirement necessary to fulfill the adversary's primary goal (for example, breaking bomb synthesis into raw materials, proportional formulation, and detonation mechanism). To maintain evaluation stability and prevent noisy over-decomposition, the maximum number of sub-questions is capped at \(n \le 5\), with atomic queries preserved as \(n=1, w_1=1\). All generated sub-question sets are persistently cached in a Memory Module, ensuring zero redundant API calls and identical evaluation rubrics across duplicate or benchmark-repeated queries.

2. Response Cleaning & Distraction Filtering: Neutralizing adversarial prompt artifacts and role-play scaffolding

Jailbreak responses often carry over extensive distracting artifacts from attack prompts, such as elaborate fictional scenarios ("DAN", "AIM"), moral disclaimers, or off-topic dialogue filler that can hijack a judge LLM's attention. The Clean Node first applies a deterministic non-LLM sentence splitter to parse the raw response \(R\) into an ordered sequence \(\{sen_j\}_{j=1}^m\). A clean agent then evaluates the relevance predicate \(\text{Rel}(sen_j, Q)\) for each sentence, discarding irrelevant framing and retaining only sentences directly pertinent to the underlying request: \(R_{clean} = \{sen_j \in R \mid \text{Rel}(sen_j, Q) = \text{True}\}\). This front-end sanitization neutralizes prompt injection hazards and isolates the substantive payload for downstream assessment.

3. Sub-Question Evidence Pairing: Eliminating ungrounded judge hallucinations via verbatim sentence extraction

Standard LLM-as-a-judge systems frequently hallucinate connections or evaluate text based on fuzzy semantic recall. The Sub-Question Pairing Node implements a pairing agent that, for each decomposed sub-question \(q_i\), scans the sanitized response \(R_{clean}\) and extracts all supporting sentences to form the localized sub-answer: \(a_i = \{ sen_j \in R_{clean} \mid \text{Pair}(q_i, sen_j) = \text{True} \}\). Crucially, the pairing agent is strictly constrained to copy matching sentences verbatim without summarization, rephrasing, or interpolation. This ensures absolute fidelity to the model's actual output, transforms unconstrained long-form evaluation into discrete sentence-level verification, and creates a transparent, auditable trail that allows human reviewers to inspect the exact textual grounds for every sub-decision.

4. Likert Scoring & Weighted Aggregation: Calibrating granular jailbreak utility across binary and ternary boundaries

The Evaluation Node assesses the fulfillment of each sub-answer \(a_i\) with respect to \(q_i\) from the adversary's perspective using a standardized five-point Likert scale \(V = \{0.00, 0.25, 0.50, 0.75, 1.00\}\), where \(0.00\) signifies complete failure and \(1.00\) denotes a complete, actionable solution. The global utility score \(S_{total}\) is calculated via importance-weighted linear aggregation:

\[S_{total} = \sum_{i=1}^n w_i \cdot s_i\]

Under the conventional binary setup, a threshold of \(0.25\) maps the score: \(S_{total} \le 0.25 \to \text{False}\) (failure) and \(S_{total} > 0.25 \to \text{True}\) (success). Under the ternary setup, JADES introduces a distinct category for partial fulfillment:

\[\text{Decision} = \begin{cases} \text{Failed}, & S_{total} \le 0.25 \\ \text{Partially Successful}, & 0.25 < S_{total} < 0.75 \\ \text{Successful}, & S_{total} \ge 0.75 \end{cases}\]

This multi-level discretization reflects real-world adversarial utility, penalizing superficial bypasses that fail to deliver a complete solution while preserving high-confidence alerts for fully realized threats.

5. Unit Fact Splitting & External Verification: Thwarting hallucination-induced false alarms via fine-grained fact checking

Target models forced past safety boundaries frequently produce superficially structured yet factually incorrect instructions (such as substituting kitchen ingredients for chemical precursors or supplying lethal, non-functional chemical ratios). Standard judges cannot reliably detect subtle procedural hallucinations. JADES provides an optional plug-and-play Fact-Checking Extension: a fact-splitting agent decomposes each sentence in \(R_{clean}\) into self-contained unit facts \(F_{sc} = \{f_1^{sc}, \dots, f_k^{sc}\}\), which are queried against an external trusted corpus (Wikipedia or Tavily web search) to yield discrete labels \(\text{Verdict}(f_i^{sc}) \in \{\text{Right}, \text{Wrong}, \text{Unknown}\}\). The verified annotations \(\hat{F}_{annot}\) are supplied as auxiliary context to the Evaluation Node; when a sub-answer contains unit facts flagged as \(\text{Wrong}\), the judge applies a direct score penalty, preventing plausible-sounding falsehoods from being credited as successful jailbreaks.

Benchmark & Evaluation Protocol

To validate JADES and rectify empirical baselines, the authors establish two curated, expert-annotated datasets: 1. JailbreakQR Benchmark: Comprises 700 prompt-response pairs (Batch 1: 400 pairs; Batch 2: 300 pairs) generated across 5 representative jailbreak attacks (GCG, DSN, LAA, PAIR, JailbreakChat) and 7 distinct LLMs (Vicuna-13B, Llama-2-7b, Llama-3-8b, Llama-3.1-8b, GPT-3.5-Turbo, GPT-4, DeepSeek-V3). Five experienced security researchers annotated each completion into failed, partially successful, or successful categories with detailed justifications, achieving substantial inter-annotator agreement (Krippendorff's \(\alpha = 0.823\) on Batch 1 and \(0.815\) on Batch 2). 2. HarmfulQA Benchmark: Curates 100 closed-scope harmful questions paired with verified ground-truth reference answers across 5 domains (Physical Harm, Cyber Crime, Drugs, Privacy & Copyright, Harassment & Racism), along with 400 model completions, explicitly structured to evaluate factual error detection without confounding open-endedness.

Key Experimental Results

Main Results

Evaluation Dataset (Setting) Evaluator Method Accuracy (Acc) Precision Recall F1 Score
JailbreakQR Batch 1 (Binary) StringMatch (Zou et al., 2023) 0.846 0.846 0.941 0.891
JailbreakQR Batch 1 (Binary) StrongReject (\(t=0.00\)) (Souly et al., 2024) 0.863 0.831 0.984 0.901
JailbreakQR Batch 1 (Binary) HarmBench (Mazeika et al., 2024) 0.870 0.887 0.914 0.900
JailbreakQR Batch 1 (Binary) JailbreakBench (Chao et al., 2024) 0.863 0.878 0.918 0.897
JailbreakQR Batch 1 (Binary) JailbreakRadar (Chu et al., 2025) 0.883 0.850 0.988 0.914
JailbreakQR Batch 1 (Binary) JADES (Ours) 0.985 0.992 0.984 0.988
JailbreakQR Batch 1 (Ternary) ShieldLM (Zhang et al., 2024) 0.528 โ€” โ€” 0.460 (Macro)
JailbreakQR Batch 1 (Ternary) Qwen3Guard (Team, 2025) 0.538 โ€” โ€” 0.445 (Macro)
JailbreakQR Batch 1 (Ternary) JADES (Ours) 0.863 0.895 (Macro) 0.847 (Macro) 0.849 (Macro)

Comparative Re-Evaluation & Ablation Study

The table below reports the re-evaluation of 5 representative jailbreak attacks across diverse target LLMs, contrasting JailbreakBench (JBench) binary ASR against JADES binary ASR, alongside JADES ternary partially successful rate (PSR), fully successful rate (SR), and success quality ratio (\(\frac{\text{SR}}{\text{ASR}}\)):

Attack Method Target Model JBench Binary ASR JADES Binary ASR JADES Ternary PSR JADES Ternary SR Success Quality (\(\frac{\text{SR}}{\text{ASR}}\))
DSN Llama-2-7b 0.94 0.66 0.52 0.14 0.21
DSN Vicuna-13b 0.95 0.71 0.56 0.15 0.21
DSN Llama-3-8b 0.76 0.52 0.39 0.13 0.25
DSN Llama-3.1-8b 0.61 0.40 0.29 0.11 0.27
LAA Llama-2-7b 0.90 0.49 0.43 0.06 0.12
LAA Vicuna-13b 0.89 0.57 0.49 0.08 0.14
LAA GPT-3.5-Turbo 0.93 0.69 0.55 0.14 0.20
LAA GPT-4 0.78 0.72 0.62 0.10 0.14
LAA DeepSeek-V3 1.00 0.62 0.50 0.12 0.19
PAIR Vicuna-13b 0.69 0.38 0.36 0.02 0.05
PAIR GPT-4 0.34 0.20 0.20 0.00 0.00
PAIR DeepSeek-V3 0.92 0.51 0.45 0.06 0.12

The table below presents the ablation performance of the Fact-Checking Extension on HarmfulQA (400 pairs), along with architectural comparisons against alternative evaluation paradigms:

Configuration / Paradigm Backbone LLM Accuracy (Acc) Precision Recall F1 Score
JADES Baseline (w/o Extension) GPT-4o 0.840 0.683 0.778 0.714
JADES Extended (w/ Extension) GPT-4o 0.963 0.916 0.888 0.901
Single-Turn Structured Baseline (SingleDeLLM) GPT-4o 0.852 โ€” โ€” 0.889
JailbreakBench w/ Stronger Backbone GPT-5.2 0.880 โ€” โ€” โ€”
JailbreakRadar w/ Stronger Backbone GPT-5.2 0.890 โ€” โ€” โ€”

Key Findings

  • Prior Jailbreak Metrics Systematically Overestimate Attack Success Rates: Under binary evaluation, widely reported ASR figures collapse significantly when audited by JADES. For instance, LAA's ASR against Llama-2 drops from 0.90 (JailbreakBench) to 0.49, and against DeepSeek-V3 drops from 1.00 to 0.62. Conventional metrics routinely reward willingness to converse or benign outlines as genuine jailbreaks, falsely inflating the perceived vulnerability of safety-aligned models.
  • Partial Successes Dominate Real-World Jailbreak Outcomes: Ternary analysis uncovers that the proportion of complete successes among all successes (\(\frac{\text{SR}}{\text{ASR}}\)) remains stubbornly below 0.25 across nearly all attack-model pairs. For semantic-mutation attacks like PAIR on Vicuna, \(\frac{\text{SR}}{\text{ASR}}\) is only 0.05 (and 0.00 on GPT-4). Most successful jailbreaks merely elicit fragmented, non-actionable outlines, while truly catastrophic full jailbreaks are exceptionally rare.
  • The Evaluation Paradigm Trumps Model Scale: Upgrading the backbone of holistic baselines like JailbreakRadar and JailbreakBench to next-generation models (GPT-5.2) yields marginal accuracy gains of only ~1-2% (remaining under 0.89), whereas JADES achieves 0.985 with GPT-4o. Furthermore, the SingleDeLLM control experiment (forcing decomposition into a single-turn prompt on GPT-4o) attains only 0.852 accuracy. This proves that multi-agent decomposition, isolated evidence extraction, and weighted aggregationโ€”not model raw capabilityโ€”are the true performance drivers.
  • Fact-Checking Effectively Eliminates Hallucinated Hazard False Positives: On the HarmfulQA benchmark, enabling external unit-fact verification yields a +12.3% gain in accuracy (0.840 \(\to\) 0.963) and a +23.3% surge in precision, successfully neutralizing the deceptive credibility of model-generated pseudo-instructions.

Highlights & Insights

  • Cross-Disciplinary Translation of Analytic Rubrics: Adapts classical educational measurement principles into LLM red-teaming, replacing subjective global impression scoring with a transparent, verifiable evidence-chain audit trail.
  • Introducing "Success Quality" (\(\frac{\text{SR}}{\text{ASR}}\)) as a Standard Metric: Disentangles surface-level filter circumvention from substantive task fulfillment, exposing target shift in black-box prompt rewriting and shifting research focus from maximizing attack quantity to evaluating attack depth.
  • Engineered Practicality via JADES_light: Provides an optimized, PydanticAI-based parallelized re-implementation that reduces token consumption by 44% and latency by 62% (averaging 8.7 seconds per request), making multi-agent decomposition commercially viable for large-scale safety auditing.

Limitations & Future Work

  • External Knowledge Coverage in Specialized Domains: The fact-checking pipeline currently queries Wikipedia via web search; while effective for common hazardous topics, it may lack coverage for esoteric chemical formulations or zero-day exploit techniques, requiring future integration with domain-specific knowledge bases like PubChem or the National Vulnerability Database.
  • Conservative Tendency in Ternary Grading: JADES applies stricter evaluation standards than human annotators, occasionally downgrading complete successes that lack minor contextual niceties to partial successes (recall of 0.590 at the default 0.75 cutoff). Sensitivity analysis demonstrates that tuning the high threshold to 0.70 recovers one-third of these conservatively penalized cases with minimal precision loss.
  • Inability to Diagnose Root-Cause Failure Attribution: JADES accurately quantifies whether a response constitutes a jailbreak, but cannot disentangle whether a low score was caused by a flawed adversarial prompt, target-model knowledge deficits, or active safety unlearning defenses.
  • vs StringMatch & Toxicity Classifiers (Zou et al., 2023; Perspective API): String matching is fooled by benign affirmative openings ("Sure, here is how to safely..."), while toxicity detectors penalize vulgar off-task responses; JADES directly measures task fulfillment, eliminating proxy misalignment.
  • vs Holistic LLM-as-a-Judge (Chao et al., 2024; Mazeika et al., 2024; Souly et al., 2024): Holistic judges are susceptible to distraction tokens, empty formatting ("Step 1... Step 2..."), and lengthy ethical scaffolding; JADES filters non-essential framing and anchors sub-decisions on extracted verbatim evidence, boosting binary accuracy from ~86% to 98.5%.
  • vs General Content Safety Filters (ShieldLM, Qwen3Guard): General safety classifiers determine whether text is harmful in the abstract rather than whether it solves a specific adversarial prompt; their ternary accuracy on JailbreakQR hovers around 53%, whereas JADES reaches 86.3%.

Rating

  • Novelty: โญโญโญโญโญ Pioneering introduction of decompositional analytic scoring and fact-checking to LLM jailbreak assessment, establishing ternary evaluation protocols.
  • Experimental Thoroughness: โญโญโญโญโญ Features two newly constructed datasets (JailbreakQR with 700 pairs and HarmfulQA with 100 benchmarked tasks), extensive cross-model re-evaluations, and comprehensive cost and threshold sensitivity ablations.
  • Writing Quality: โญโญโญโญโญ Rigorous conceptual structure, compelling empirical arguments, clear mathematical formalization, and extensive case cards.
  • Value: โญโญโญโญโญ Fundamentally recalibrates empirical risk estimates for open-source and proprietary LLM security, providing an indispensable foundation for reliable red-teaming.