Skip to content

Diversity Combining for Multi-Path LLM Reasoning

Conference: NeurIPS 2026
arXiv: 2609.38829
Area: others
Keywords: multi-path reasoning, self-consistency, diversity combining, effective sample size, adaptive-K

TL;DR

Formalizes multi-path LLM reasoning as diversity combining over wireless multipath channels, proving that pairwise correctness correlation limits the vote-level effective sample size to a finite ceiling of \(1/c\) and deriving a 4-path pilot Adaptive-K rule.

Background & Motivation

Multi-path reasoning techniques such as self-consistency (SC) sample \(K\) chain-of-thought paths at a non-zero temperature and aggregate them via majority voting (MV), becoming a de facto standard for bolstering LLM problem-solving accuracy. However, practitioners and researchers consistently observe an unexplained empirical plateau: accuracy gains from additional reasoning paths diminish rapidly as \(K\) grows. Beyond a problem- and model-dependent threshold \(K^*\), sampling more paths incurs substantial computational inference cost with virtually zero accuracy gain.

Existing post-hoc explanations typically attribute this saturation qualitatively to answer-space coverage or sampling temperature, but they fail to provide a predictive model for \(K^*\) on a given architecture and benchmark. Under the classical Condorcet Jury Theorem, majority voting among independent decision-makers each with accuracy exceeding 0.5 converges to certainty as \(K \to \infty\). In contrast, compound LLM inference plateaus prematurely, reflecting the fact that paths sampled from the same underlying foundation model and prompt share systemic knowledge limitations and error modes. Consequently, path correctness across problems exhibits strong positive inter-path correlation, violating the independent-and-identically-distributed assumption.

This paper tackles this gap by framing multi-path reasoning through the lens of diversity combining from wireless communications: the ground-truth solution acts as the transmitted source signal, each sampled reasoning path serves as a noisy observation over a fading channel branch, and the final decision aggregator functions as a receiver-side diversity combiner. Core idea: formalize multi-path LLM reasoning as a diversity combining problem, prove that the pairwise correctness correlation \(c\) establishes an asymptotic vote-level effective sample size ceiling of \(1/c\), justify uniform majority voting via Generalized Least Squares (GLS) under path exchangeability, and derive a closed-form Adaptive-K budget allocation rule requiring only a 4-path pilot.

Method

Overall Architecture

The proposed framework establishes a rigorous theoretical pipeline linking continuous latent representations, discrete correctness indicators, statistical effective diversity orders, and adaptive inference budgets. The end-to-end workflow comprises four main stages: 1. Channel Modeling and Binary Collapse: Formulates multi-path outputs as latent channel vectors corrupted by path gains and additive Gaussian noise, and projects text-extracted answers onto binary correctness indicators relative to gold solutions. 2. Diversity Order and Saturation Diagnostics: Quantifies the degradation in independent information using covariance participation ratio (latent rank) and Kish design effects (vote level), deriving an analytical ceiling on effective sample size. 3. GLS-Optimal Combining and Vote Optimality: Employs generalized least squares to analyze minimum-MSE linear combinations of embeddings, demonstrating that uniform weighting is strictly optimal under path exchangeability and thus validating standard majority voting. 4. Adaptive-K Path Budget Allocation: Calculates the marginal diversity gain with respect to path budget \(K\), matching it to a marginal gain threshold to estimate the optimal budget \(K^*\) from a lightweight 4-path pilot.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query & Prompt"] --> B["Channel Observation & Binary Collapse<br/>Sample K paths & map to correctness indicators"]
    B --> C["Diversity Order & Saturation Diagnostics<br/>Compute correlation and effective sample size ceiling"]
    C --> D["GLS Combining Analysis<br/>Verify exchangeability & uniform weighting optimality"]
    D --> E["Adaptive-K Budget Allocation<br/>Determine optimal K* from 4-path pilot"]
    E --> F["Deploy Majority Vote to Produce Final Answer"]

Key Designs

1. Channel Observation and Binary Collapse: Mapping Reasoning Paths to Noisy Branch Signals

To formalize the statistical coupling among reasoning paths sampled from the same model, the true target answer is represented by a latent embedding \(\mathbf{z}(x) \in \mathbb{R}^D\). For problem instance \(i\), the latent representation generated by the \(k\)-th reasoning path is modeled as \(\mathbf{z}_k = h_k \mathbf{z}(x) + \mathbf{n}_k\), where \(h_k > 0\) represents the path quality (channel gain) and \(\mathbf{n}_k \sim \mathcal{N}(\mathbf{0}, \sigma_n^2 \mathbf{I}_D)\) represents generative noise. To bridge continuous channel models with discrete black-box outputs, the framework operates on binary-collapsed correctness indicators \(Y_i^{(k)} = \mathbf{1}\{a_i^{(k)} = x_i\}\). Aggregating across instances provides a direct empirical estimator of pairwise correctness correlation: $\(\hat{c}_{jk} = \frac{\frac{1}{n}\sum_{i=1}^{n}Y_i^{(j)}Y_i^{(k)} - \bar{p}^2}{\bar{p}(1-\bar{p})}\)$ where \(\bar{p}\) is the mean per-path accuracy, and \(\hat{c}\) is the mean over all path pairs. This metric is computed exclusively from observable output predictions without requiring internal model activations or gradients.

2. Diversity Order and Saturation Diagnostics: Deriving the Finite Effective Sample Size Ceiling

To explain why majority voting gains saturate rapidly, the paper distinguishes latent covariance effective rank from the vote-level effective sample size. For an equicorrelated latent covariance matrix \(\mathbf{R}\) with correlation \(\rho\), the participation ratio yields \(K_{\text{eff}}^{\text{rank}} = K / (1 + (K - 1)\rho^2)\). Crucially, on the discrete decision layer, adapting the classical Kish design effect yields the vote-level effective sample size: $\(K_{\text{eff}}^{\text{vote}} = \frac{K}{1 + (K - 1)c}\)$ As \(K \to \infty\), \(K_{\text{eff}}^{\text{vote}}\) saturates at the finite asymptotic ceiling \(1/c\). For instance, on GSM8K with \(\hat{c} \approx 0.6\), the effective sample size is fundamentally bounded by \(1 / 0.6 \approx 1.67\), regardless of how many paths are generated. Furthermore, the framework introduces a Beta-Binomial distribution over vote counts, capturing the true overdispersed voting performance where standard independence-assuming models catastrophically fail.

3. GLS Combining Analysis: Grounding Majority Voting and Heterogeneous Opportunities

To resolve whether non-uniform weighting or confidence filtering can reliably outperform plain majority voting, the paper applies generalized least squares (GLS). Under the unbiased estimation constraint \(\mathbf{g}^\top \mathbf{w} = 1\), the minimum-MSE linear combiner is given by \(\mathbf{w}_* = \boldsymbol{\Sigma}^{-1}\mathbf{g} / (\mathbf{g}^\top \boldsymbol{\Sigma}^{-1}\mathbf{g})\). Under standard self-consistency, identical prompts ensure that paths are exchangeable and equicorrelated (\(\boldsymbol{\Sigma}\) is symmetric with constant off-diagonals and \(\mathbf{g} = \mathbf{1}\)), causing optimal weights to collapse to the uniform distribution \(\mathbf{w}_* = \frac{1}{K}\mathbf{1}\). This proves that uniform majority voting is theoretically optimal for exchangeable paths. Conversely, when prompt templates break exchangeability, path heterogeneity creates fertile ground for non-uniform weighting and pruning.

4. Adaptive-K Budget Allocation: Pilot-Guided Test-Time Compute Scaling

To avoid overspending on redundant sampling, the paper computes the marginal diversity gain \(\partial K_{\text{eff}}^{\text{vote}} / \partial K = (1 - c) / (1 + (K - 1)c)^2\). Equating this derivative to an acceptable marginal threshold \(\varepsilon\) (defaulting to 0.025, or 2.5% marginal gain) yields a closed-form solution for the optimal path budget \(K^*\): $\(K^* = \left\lceil \frac{\sqrt{(1 - \hat{c})/\varepsilon} - 1}{\hat{c}} + 1 \right\rceil\)$ In deployment, running a small pilot with \(K_0 = 4\) paths across 50โ€“100 offline instances yields an accurate estimate of \(\hat{c}\), fixing \(K^*\) for full deployment. This strategy preserves peak majority-voting gains while cutting computational cost drastically.

Quantitative Experiments

Main Results

On GSM8K, the effective diversity diagnostics and majority-vote accuracy were evaluated across model families, comparing the Beta-Binomial (BB) model against the independence-binomial model (Binom) at \(K=32\).

Model \(\hat{c}_{K=4}\) \(K_{\text{eff}}^{\text{vote}} (K=4)\) \(K_{\text{eff}}^{\text{vote}} (K=32)\) Ceiling \(1/\hat{c}\) Actual MV@32 BB Pred. MV@32 Binom Pred. MV@32
Qwen2.5-7B 0.60 1.42 1.67 1.71 (98%) 81.9% 81.1% 100.0%
Llama-3.1-8B 0.53 1.55 1.94 2.21 (96%) 79.3% 77.8% 99.9%
Mistral-7B 0.45 1.71 2.13 2.22 (96%) 42.4% 41.3% 20.6%

Ablation Studies

Evaluation of the Adaptive-K decision rule across distinct domains (Math, open QA, binary NLU), measuring accuracy retention relative to MV@32 and net inference compute cost (including the 4-path pilot).

Task Domain / Benchmark Model Pilot \(\hat{c}_{K=4}\) Selected \(K^*\) MV@\(K^*\) MV@32 Retained (%) Net Cost (%)
GSM8K (Math) Qwen2.5-7B 0.60 6 81.6% 81.9% 100% 31%
GSM8K (Math) Llama-3.1-8B 0.53 8 78.1% 79.3% 98% 38%
GSM8K (Math) Mistral-7B 0.45 10 43.6% 42.4% 103% 44%
HotpotQA (QA) Llama-3.1-8B 0.61 6 50.2% 52.2% 96% 31%
BoolQ (NLU) Llama-3.1-8B 0.79 4 80.2% 80.3% 100% 25%

Key Findings

  • Rapid Saturation Near Theoretical Ceilings: Across all model families, \(K_{\text{eff}}^{\text{vote}}\) reaches 96%โ€“98% of its asymptotic ceiling \(1/\hat{c}\) at \(K=32\), validating why adding paths beyond 8โ€“16 yields near-zero return.
  • Answer-Space Gated Decorrelation: Prompt-template (PT) perturbations reduce path correlation in 55 of 57 valid cells across 12 benchmarks (mean reduction -38%). The magnitude tracks answer-space openness: open-text QA (-67%) decorrelates significantly more than bounded choice (-35% to -49%), which in turn exceeds code (-22%) and math (-9% to -29%) with Mann-Whitney \(p < 10^{-3}\), mirroring rich-scattering vs. line-of-sight propagation in wireless systems.
  • Extreme Compute Efficiency: The Adaptive-K rule retains 96%โ€“103% of full \(K=32\) accuracy while slashing compute requirements to 25%โ€“44% of the fixed budget.

Highlights & Insights

  • Interdisciplinary Theoretical Synthesis: Successfully imports diversity combining and wireless multipath fading channel theory into compound LLM reasoning, providing a concrete mathematical basis for diminishing returns.
  • Closed-Form Saturation Diagnostics: Establishes \(1/c\) as the hard ceiling on effective sample size and replaces divergent independence assumptions with well-calibrated Beta-Binomial models (within 1.5 pp error).
  • Plug-and-Play Practical Tooling: Derives a training-free, label-free (at deployment) Adaptive-K rule calibrated via an inexpensive 4-path pilot that operates entirely on black-box outputs.

Limitations & Future Work

  • Global Allocation Lacks Instance-Level Dynamism: The current formulation calibrates a single global \(K^*\) per task configuration, unable to allocate fewer paths to trivial queries and more to challenging edge cases.
  • Binary Collapse Disregards Error Fragmentation: Binary correctness collapses the rich distribution of incorrect answers in large multiclass spaces, losing information about wrong-answer dispersion.
  • Heterogeneous Weighting Requires Calibration: Non-uniform weights under prompt-template diversity require validation-set tuning or hidden-state access (GLS-R), which remains challenging in purely zero-shot, API-only environments.