Skip to content

DIANOIA: Diagnostic Decomposition and Joint Optimization for Multi-Agent Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/sachiel321/DIANOIA4MALLM
Area: LLM Reasoning
Keywords: Multi-Agent Systems, Reasoning Gain Decomposition, Role Diversity, Execution Feedback, Potential Game Synthesis

TL;DR

DIANOIA formalizes multi-agent reasoning gain into three measurable channels—coverage, fidelity, and synthesis—and implements them via a four-phase architecture (role-diverse proposers, grounded execution/verification, cross-review, and closed-loop synthesis) that matches or exceeds state-of-the-art multi-agent baselines while requiring approximately 5× fewer tokens on code generation.

Background & Motivation

Multi-Agent Systems (MAS) powered by Large Language Models (LLMs) have achieved prominent empirical success across complex mathematical reasoning, programmatic code synthesis, and interactive tool-use environments. Researchers have introduced diverse topological paradigms—such as tree-of-thought exploration, multi-round debates, and multi-layer agent mixtures—to scale test-time inference compute beyond single-agent Chain-of-Thought prompting. Despite these empirical gains, the architectural design of multi-agent workflows remains overwhelmingly heuristic and craft-driven: when approaching an unseen downstream task, practitioners lack formal principles to predict which multi-agent paradigm will succeed, or to diagnose why a deployed system underperforms.

The fundamental tension stems from the fact that existing multi-agent paradigms disproportionately focus on only one or two isolated dimensions of reasoning gain, leaving unaddressed bottlenecks that trigger catastrophic degradation under distinct task topologies. Specifically, exploration-centric approaches like Self-Consistency expand solution-space coverage via parallel sampling but rely on naive majority voting, which collapses when answer spaces are syntactically fragmented (as in program synthesis) or when instance-level competence falls below random guessing (\(p_i < 0.5\)). Conversely, aggregation-centric systems such as ungrounded multi-agent debate or layered synthesis (Mixture-of-Agents) foster discursive refinement but operate on "cheap talk"; without grounding in external objective execution feedback, they remain susceptible to agreement bias and sycophancy. Meanwhile, feedback-grounded reflective agents often search sequentially along a single path, severely constrained by low candidate exploration. Because these performance levers interact multiplicatively, improvements along a single axis encounter steep diminishing returns unless all components are coordinated.

To overcome this fragmentation, the field requires a diagnostic framework that bridges rigorous mathematical characterization with actionable system engineering. The core idea is to factorize the expected multi-agent reasoning gain into three conceptually orthogonal and empirically measurable channels—Exploration Coverage (\(G_{\text{explore}}\)), Information Fidelity (\(G_{\text{info}}\)), and Aggregation Synthesis (\(G_{\text{aggr}}\))—and to operationalize this diagnostic protocol into DIANOIA, a four-phase system that jointly optimizes role-diverse proposal generation, grounded execution cross-review, and closed-loop potential-game synthesis.

Method

Overall Architecture

DIANOIA formalizes multi-agent reasoning as a tuple \((\mathcal{X}, \mathcal{T}, \mathcal{Q}, K, \mathcal{E}, f)\), where an input \(x \in \mathcal{X}\) is solved by \(K\) proposing agents generating candidate trajectories \(\tau \in \mathcal{T}\), evaluated by an execution or pseudo-verification environment \(\mathcal{E}\), and synthesized by an aggregation operator \(f\). Drawing on collective intelligence and ensemble learning, the framework establishes an exact definitional identity factorizing expected quality into coverage and efficiency: \(\mathbb{E}[\mathcal{Q}(\tau_{\text{MAS}})] = C_K \cdot \eta(f, s)\), leading to a conditional subadditive upper bound \(\mathbb{E}[\mathcal{Q}(\tau_{\text{MAS}})] - p \le G_{\text{explore}} + G_{\text{info}} + G_{\text{aggr}}\). DIANOIA implements these theoretical prescriptions through four tightly coupled phases: role-differentiated generation for coverage, execution sandboxing and evidence-based cross-review for information fidelity, and iterative closed-loop repair for aggregation efficiency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Input x"] --> B["Phase 1: Role-Diverse Propose<br/>Minimalist / Skeptic / Explorer generate τ(k)"]
    B --> C["Phase 2: Grounded Execution<br/>Deterministic sandbox / pseudo-verifier e(k)"]
    C --> D["Phase 3: Evidence Cross-Review<br/>Structured failure diagnosis & repair hints v(k)"]
    D --> E["Phase 4: Closed-Loop Potential-Game Synthesis<br/>Trajectory grafting & re-execution validation τ*"]
    E -->|Validation failed & t < T| E
    E -->|Validation passed or budget reached| F["Final Verified Solution"]

Key Designs

1. Role-Diverse Propose: Inducing Negative Pairwise Correlation to Maximize Coverage

Addressing the limitation that standard independent (IID) sampling with high temperatures causes agents to share common reasoning biases and produce duplicate erroneous trajectories, this design enforces structural diversity among the \(K=3\) proposing agents. DIANOIA instantiates three complementary inductive biases: the Minimalist pursues the shortest programmatic path using concise standard-library idioms; the Skeptic applies defensive programming with explicit assertions for boundary, null, and off-by-one conditions; and the Explorer leverages unconventional abstractions and non-standard algorithmic primitives. Under the second-order Bonferroni expansion, the coverage lower bound \(C_K = \mathbb{P}(\exists k: \mathcal{Q}(\tau^{(k)}) = 1)\) strictly surpasses the IID baseline whenever average pairwise correlation is negative (\(\bar{\rho} < 0\)): $\(C_K \ge Kp - \binom{K}{2} p^2 - \binom{K}{2} \bar{\rho} p (1 - p)\)$ Empirical measurements on MBPP confirm negative pairwise success correlations among the three roles (\(\rho_{12} \approx -0.15\), \(\rho_{13} \approx -0.18\), \(\rho_{23} \approx -0.12\), averaging \(\bar{\rho} \approx -0.15\)), generating an absolute coverage gain of roughly +8.2 percentage points over identically prompted IID agents.

2. Grounded Execution: Maximizing Mutual Information via Objective Verification

Targeting the severe information loss of purely textual self-reflection—where LLM evaluator misclassifications dilute the selection signal—this component establishes an objective feedback hierarchy. On executable tasks such as code generation and function calling, candidates are dispatched to a deterministic sandbox producing structured execution reports \(e^{(k)} = \mathcal{E}(\tau^{(k)})\) (pass/fail status, assertion errors, and tracebacks). By Theorem 3.5(a), under deterministic and complete verification, execution feedback serves as a sufficient statistic for solution quality, achieving maximal mutual information: $\(I(\mathcal{Q}; e) = H(\mathcal{Q}), \quad H(\mathcal{Q} \mid e) = 0\)$ In contrast, textual self-critique \(\sigma\) with positive false-positive rate \(\alpha\) and false-negative rate \(\beta\) retains only \(I(\mathcal{Q}; \sigma) \approx 0.446\) bits (a ~54% information loss from \(H(\mathcal{Q}) \approx 0.971\)). For non-executable abstract mathematical tasks, DIANOIA deploys a dedicated model-based pseudo-verifier (\(\sigma_v\)) generating calibrated confidence and structured error diagnoses, achieving intermediate fidelity (\(I(\mathcal{Q}; \sigma_v)/H(\mathcal{Q}) \approx 0.75\text{--}0.85\)).

3. Evidence Cross-Review: Transforming Raw Execution Traces into Actionable Prescriptions

Raw execution output reveals whether an implementation failed, but lacks semantic guidance on why it failed and how to repair it. Phase 3 assigns \(R\) independent reviewer agents per candidate to conduct evidence-grounded cross-review \(v_j^{(k)} = \mathcal{R}_j(\tau^{(k)}, e^{(k)})\). Reviewers inspect both the proposed solution and its sandbox report, localizing bugs against assertion failures, identifying validated components, and formulating concrete repair suggestions. By conditioning review on objective execution evidence rather than ungrounded textual debate, the empirical per-reviewer misclassification rate drops to \(\hat{\epsilon}_0 = 5.8\%\) on MBPP (compared to 17.5% on GSM8K). While the theoretical peer-review bound (\(R = K-1\)) guarantees an exponential misidentification decay of \(\epsilon_0^{K-1}\), DIANOIA adopts \(R=1\) as an efficient default that captures the predominant aggregation gain without incurring linear cost inflation.

4. Closed-Loop Potential-Game Synthesis: Trajectory Grafting with Finite-Step Convergence

Addressing the failure of majority voting on fragmented answer spaces as well as the inability of pure selection to recover from all-failing proposals, Phase 4 deploys a synthesis agent that performs trajectory grafting—splicing valid logic segments from failing candidates while repairing defective lines using reviewer advice. Crucially, each candidate generated during the \(T\) synthesis iterations undergoes mandatory closed-loop re-execution. The interactive synthesis process is formalized as an exact potential game with potential function: $\(\Phi(\tau) = \max_{k=1,\dots,K} \mathcal{Q}(\tau^{(k)}) + \lambda \sum_{k=1}^K \mathcal{R}_k(\tau^{(k)})\)$ Because candidate space is finite and re-execution enforces an improvement-restricted acceptance rule (\(\Delta u_i = \Delta \Phi > 0\)), best-response iterations are guaranteed to terminate in finite steps at a pure-strategy Nash equilibrium. Beyond selecting among correct initial inputs (16.4% pure selection, 78.1% selection with repair), this mechanism achieves a 5.5% pure rescue rate on MBPP where all initial proposals failed but a correct solution was synthesized from diagnostic feedback.

Benchmark & Evaluation Protocol

DIANOIA is evaluated across four benchmarks spanning distinct verification regimes: GSM8K (\(N=1319\), high baseline competence \(p\), pseudo-verifier), AIME-2025 (\(N=30\), low baseline competence \(p\), competition math), MBPP (\(N=500\), deterministic unit-test execution), and BFCL-SP (\(N=400\), Berkeley Function Calling Simple Python schema execution). All multi-agent baselines are standardized on Qwen3-30B-A3B-Instruct in zero-shot mode without thinking mode. Further cross-family replications are conducted on Qwen3.6-35B-A3B, DeepSeek-V4-Flash, Gemini-3.5-Flash, and Claude Opus 4.7. Statistical significance is assessed via 1,000 bootstrap resampling iterations (95% CI) and paired exact McNemar tests.

Key Experimental Results

Main Results

Under matched configurations and identical foundation models, DIANOIA outperforms all major multi-agent baselines across all four benchmarks, surpassing the strongest baseline while establishing new state-of-the-art accuracy levels.

Benchmark Metric Single Baseline (Qwen3-30B) Strongest MAS Baseline DIANOIA (Ours) Margin over Strongest Baseline
GSM8K (N=1319) Accuracy (%) 83.6% [81.5, 85.7] 89.8% (ReConcile) 91.1% [89.6, 92.7] +1.3 pp
AIME-2025 (N=30) Accuracy (%) 70.0% [53.3, 86.7] 86.7% (MoA) 93.3% [83.3, 100] +6.6 pp
MBPP (N=500) Accuracy (%) 76.0% [72.4, 80.0] 78.4% (Best-of-3) 84.6% [81.4, 87.0] +6.2 pp
BFCL-SP (N=400) Accuracy (%) 81.8% [78.0, 85.8] 88.8% (Two Heads) 92.3% [89.5, 94.8] +3.5 pp

Note: On MBPP, DIANOIA's 95% bootstrap confidence interval lower bound (81.4%) completely clears the upper bounds of all standard multi-agent baselines (Self-Consistency 78.0%, MoA 76.8%, Two Heads 77.2%, ReConcile 77.2%).

Ablation Study

By varying the configuration parameters \((K, R, S)\)—representing proposers, reviewers, and synthesis iterations—the component ablation isolates the marginal contribution of each reasoning channel and exposes their multiplicative coupling.

Configuration \((K, R, S)\) Activated Dimensions MBPP Acc (%) BFCL-SP Acc (%) Functional Description
(1, 0, 0) Single-Agent Baseline 76.0% 84.8% Unassisted baseline
(3, 0, 1) Explore-only (\(G_{\text{explore}}\)) 81.2% (+5.2) 89.8% (+5.0) Multi-role proposal without review
(1, 1, 1) Info-only (\(G_{\text{info}}\)) 79.8% (+3.8) 87.3% (+2.5) Single proposal with execution & review
(1, 0, 1) Aggr-only (\(G_{\text{aggr}}\)) 76.8% (+0.8) 86.1% (+1.3) Single proposal with iterative refinement
(3, 1, 1) Two-dim (Explore + Info) 83.6% (+7.6) 90.8% (+6.0) Diverse proposals + grounded review
(3, 1, 3) DIANOIA-full (All three) 84.6% (+8.6) 92.3% (+7.5) Full joint optimization

Extended Pareto and scalability analyses reveal: 1. Pareto Dominance: In the accuracy-vs-token sweep on MBPP across 39 configurations, MoA requires 7.73M tokens across 3 stacked layers to reach 84.2%, whereas DIANOIA achieves 84.6% using only 1.54M tokens—yielding a ~5× token-budget advantage at matched accuracy, and scaling to 88.8% at 7.36M tokens. 2. Cross-Model Family Robustness: On Qwen3.6-35B-A3B, DIANOIA advances MBPP from 78.4% to 88.8% (+10.4 pp) and BFCL-SP from 83.8% to 93.0% (+9.2 pp). At full MBPP scale (\(N=500\)), it lifts DeepSeek-V4-Flash from 88.8% to 98.2% and Gemini-3.5-Flash from 93.0% to 96.8%. 3. Prospective Held-Out Validation: On the long-horizon, interactive VitaBench benchmark (\(N=400\)), where bottleneck predictions were recorded prior to testing, DIANOIA increases task completion from 29.5% [25.2, 34.1] to 45.0% [40.2, 49.9] (+15.5 pp).

Key Findings

  • Synergy Coefficient Validates Subadditive Coupling: The empirical synergy coefficient \(\gamma = \frac{\Delta_{\text{joint}}}{\sum \Delta_i}\) evaluates to \(0.88\) on MBPP (\(8.6 / 9.8\)) and \(0.852\) on BFCL-SP (\(7.5 / 8.8\)). Because expected accuracy factors multiplicatively as \(\mathbb{E}[\mathcal{Q}] = C_K \cdot \eta(f, s)\), optimizing one channel increases the return on others, but realized gains are bounded by the weakest link.
  • Catastrophic Failure of Majority Voting at Low \(p_i\): On AIME-2025 where individual problem competence frequently falls below 0.5, Self-Consistency voting actively amplifies errors, degrading single-model accuracy from 70.0% down to 56.7% (-13.3 pp), directly confirming theoretical Failure Mode 2.
  • The Value of Synthesis over Selection: On MBPP, the Best-of-3 baseline using an identical execution verifier achieves 78.4%, trailing DIANOIA (84.6%) by 6.2 percentage points. This gap measures the pure synthesis gain (\(G_{\text{aggr}}\)), which is further verified by instrumenting a 5.5% pure rescue rate where initially failing candidate sets were repaired into passing code.

Highlights & Insights

  • From Ad-hoc Topology to Channel-Aware Diagnostics: DIANOIA replaces trial-and-error heuristic multi-agent design with four rigorous diagnostic rules (R1–R4). By analyzing task properties (verifier determinism, baseline competence, answer space topology), engineers can identify bottleneck channels ex-ante and allocate compute tokens where they yield maximal returns.
  • Potential Game Formulation Guarantees Stable Consensus: By formalizing multi-agent trajectory revision as an exact potential game with closed-loop execution validation, the system guarantees finite-step termination to a pure Nash equilibrium, mathematically ruling out oscillatory self-corruption in iterative LLM refinement.
  • 5× Efficiency Gain through High-Fidelity Information: While layered architectures like MoA attempt to resolve reasoning errors by stacking compute-heavy LLM synthesis layers, DIANOIA demonstrates that grounding decisions in low-cost, lossless execution traces achieves superior accuracy with an 80% reduction in token consumption.

Limitations & Future Work

  • Dependence on Environmental Verifiers: The framework's theoretical optimality guarantees (e.g., \(I(\mathcal{Q}; e) = H(\mathcal{Q})\)) strictly hold only when complete, deterministic verifiers are available. On open-ended creative writing, subjective summarization, or philosophical discourse where verification is intrinsically lossy, the framework's prescriptive precision degrades.
  • Sequential Latency Overhead: Although DIANOIA dramatically curtails total token volume, the closed-loop synthesis loop in Phase 4 introduces sequential execution dependencies \(S \cdot (L_{\text{synth}} + L_{\text{execute}})\), resulting in higher wall-clock latency compared to single-pass parallel sampling under unconstrained inference capacity.
  • Residual Reviewer Correlation: The theoretical exponential aggregation bound assumes conditional independence among reviewer judgments (A6). In practice, reviewers derived from the same base model family share implicit representation biases, bounding asymptotic error reduction below the idealized zero-floor prediction.
  • vs Self-Consistency (Wang et al., 2023): Self-Consistency isolates exploration (\(G_{\text{explore}}\)) but relies on ungrounded majority voting. It fails completely under answer fragmentation (code) or when instance success \(p_i < 0.5\); DIANOIA substitutes voting with grounded verification and potential-game synthesis, converting raw coverage into actual accuracy.
  • vs Mixture-of-Agents (MoA) (Wang et al., 2025): MoA stacks multi-agent layers to perform implicit iterative aggregation without environmental grounding. It requires 3 stacked layers and 7.73M tokens on MBPP to reach 84.2%; DIANOIA achieves 84.6% with 1.54M tokens, demonstrating that objective feedback dramatically improves Pareto compute efficiency.
  • vs AgentCoder (Huang et al., 2024): AgentCoder focuses on iterative test-driven repair using single programmer and test agents (\(G_{\text{info}}\) and \(G_{\text{aggr}}\)). DIANOIA demonstrates that when baseline model competence is moderate, pre-seeding the pipeline with role-diverse negative-correlation proposals (\(G_{\text{explore}}\)) is indispensable for overcoming local search traps.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first formal, measurable three-channel diagnostic decomposition for multi-agent reasoning gain.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage across 4 core benchmarks, 39 Pareto configurations, cross-model replications on Qwen/DeepSeek/Gemini, and prospective held-out validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear, maintaining strict mathematical rigor while candidly addressing the delta between idealized theory and measured empirical behavior.
  • Value: ⭐⭐⭐⭐⭐ Provides a long-overdue principled foundation and diagnostic roadmap for building and optimizing test-time multi-agent reasoning systems.