Skip to content

AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Spoken Dialogue State Tracking, Audio Verification, Cross-turn Agreement, Slot-Restricted Operators, Error Correction

TL;DR

Addressing persistent ASR entity corruption and the acoustic blind spots of text-only state editors in spoken dialogue state tracking, AVERT proposes a value-level adjudication pipeline that marries cross-turn consensus with a lightweight audio verifier and a lexical attestation gate, raising the JGA of a 1B system to 40.13% on SpokenWOZ without retraining either the base model or editor.

Background & Motivation

Spoken Dialogue State Tracking (Spoken DST) aims to infer a cumulative structured state—represented as a set of slot-value pairs—directly from continuous conversational speech. Compared to text-based DST, the core difficulty stems from the fact that acoustic evidence is inherently noisy, causing Automatic Speech Recognition (ASR) errors to heavily concentrate on informational entity tokens such as proper names, locations, times, and numbers. Crucially, dialogue state accumulates sequentially over multi-turn interactions, meaning that an early acoustic distortion is continually propagated forward as historical context, severely degrading downstream task-oriented agent executions.

Existing approaches to spoken DST broadly bifurcate into two paradigms: text-only post-editing pipelines operating on top of initial ASR hypotheses (e.g., CorrectionLM, RSP-DST), and end-to-end speech-LLMs that consume raw speech histories directly. However, strong text editors rely strictly on ASR transcripts and are blind to original acoustic evidence, inherently leaving behind three unrecoverable failure modes: values predicted inconsistently across turns due to sporadic transcript perturbations, omitted slots dropped by recognizers, and hallucinated or corrupted values that the underlying audio does not substantiate. Conversely, feeding full spoken histories into an end-to-end LLM incurs an audio token context window and computational overhead that balloon linearly with dialogue length, while still failing to explicitly verify individual candidate values against acoustic evidence.

The tension lies in the structural generation of dialogue states versus the fine-grained acoustic verification of candidate entities. While text editors excel at handling grammatical flow and global schema structure from cumulative transcripts, raw audio should not be forced to regenerate the entire state from scratch; instead, it is most reliable when deployed as a local discriminator to confirm or reject specific candidate entity values. Core idea: Freeze both the base spoken-DST model and the text editor, and introduce a lightweight audio-conditioned verifier alongside a lexical attestation gate to adjudicate candidate values across turns via three slot-restricted operators (Vote, Add, Swap).

Method

Overall Architecture

The AVERT pipeline operates in three distinct stages, augmenting two frozen baseline components with a lightweight posterior verification module. At any turn \(t\), given the current audio turn and prior dialogue history, a base spoken-DST model \(R\) (combining a WavLM-large encoder, an adapter connector, and a LoRA-adapted OLMo-2-1B decoder) first emits an ASR transcript \(a_t\) alongside a first-pass state. Next, a frozen text-only editor \(E\) (a separate LoRA-adapted OLMo-2-1B) processes the cumulative transcript \(A_t = a_1 \oplus \dots \oplus a_t\) and the first-pass state to rectify text-evident defects, producing an edited state \(v_E\). Finally, AVERT executes value-level acoustic adjudication: it gathers all distinct non-empty candidate values across all past turns predicted by both \(R\) and \(E\) into a candidate set \(\mathcal{V}\), computes an audio-weighted consensus score \(p(v)\) using an audio-conditioned verifier \(\sigma(j, v)\), and sequentially applies three slot-restricted operators—Vote, Add, and Swap—under the constraint of a lexical attestation gate \(\alpha(v)\) to yield the final robust state.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current-turn Audio + Text History"] --> B["Base Spoken-DST Model R<br/>WavLM + Connector + 1B LLM"]
    B --> C["Text Editor E<br/>Cumulative At + First-pass State Revision"]
    C --> D["Cross-turn Candidate Collection & Audio-Weighted Scoring<br/>Aggregate R & E Candidates Scored by σ"]
    D --> E["Lexical Attestation Gate Constraint<br/>First-token Substring Match α as Guardrail"]
    E --> F["Slot-Restricted Adjudication Operators<br/>Targeted Rectification via Vote, Add, Swap"]
    F --> G["Final Robust Dialogue State"]

Key Designs

1. Cross-turn Candidate Collection & Audio-Weighted Scoring: Grounding Consensus in Acoustic Evidence When text editors encounter multi-turn dialogues, isolated acoustic noise in a single turn frequently causes state value flickering; simultaneously, naive frequency-based majority voting fails when recognizers consistently repeat a phonetically plausible error. AVERT addresses this by collecting all historical predictions \(\hat{v}_x(j)\) for slot \(s\) across past turns \(j \le t\) from both models \(x \in \{E, R\}\), forming a unique candidate set \(\mathcal{V}\). To score candidates against raw audio, AVERT reuses the frozen WavLM-large speech encoder and connector from \(R\), extracting frame embeddings \(\{h_i\}\) from turn \(j\)'s audio and condensing them into a turn representation \(u_j = \sum_i a_i h_i\) via a learned-query attention pool (\(a_i \propto \exp(q^\top h_i)\)). The pooled audio vector \(u_j\), a learned slot embedding \(e_s\), and a mean-pooled token embedding \(e_v\) of the candidate value are concatenated and passed through a three-layer GELU MLP to output support probability \(\sigma(j, v) \in [0, 1]\): $\(\sigma(j, v) = \text{sigmoid}\left(\text{MLP}\left([u_j;\, e_s;\, e_v]\right)\right)\)$ The composite confidence score \(p(v)\) sums these acoustic support probabilities across all occurrences where candidate \(v\) was predicted: $\(p(v) = \sum_{j \le t} \sum_{x \in \{E, R\}} \sigma(j, v) \cdot \mathbb{I}[v = \hat{v}_x(j)]\)$ By weighting occurrences with continuous acoustic confidence rather than counting them uniformly, this formulation suppresses weakly grounded hallucinations and cleanly resolves frequency-based voting ties.

2. Lexical Attestation Gate Constraint: Preventing Hallucination Drift via Substring Checking While the acoustic verifier \(\sigma\) provides continuous confidence, relying on acoustic matching alone to synthesize new slot-value pairs risks inserting spurious entities. AVERT constructs a lexical attestation indicator \(\alpha(v) = \mathbb{I}[\text{first-token}(v) \in A_t]\), which checks whether the initial token of candidate \(v\) appears as an exact substring in the cumulative ASR transcript \(A_t\). The design chooses first-token attestation over full-string matching because multi-token proper entities are typically only partially corrupted by ASR recognizers (e.g., "leicester" transcribed as "lester"); strict full-string matching would discard numerous valid, recoverable candidates. Crucially, \(\alpha(v)\) is not integrated as a soft ranking feature; rather, it functions as an uncompromising binary gating safety barrier that regulates whether candidates may be inserted or substituted during Add and Swap operations.

3. Slot-Restricted Adjudication Operators: Targeted Rectification of Distinct Failure Modes Applying correction heuristics globally across all slots risks corrupting stable categorical and boolean values. AVERT evaluates error-reduction trade-offs on the development set to pre-select domain-specific slot subsets for three sequential operators: - Vote (Consensus Adjudication): Resolves turn-to-turn value inconsistency. If slot \(s\) is already present in the editor's output (\(v_E \neq \emptyset\)) and belongs to the Vote whitelist (12 entity slots, e.g., train-destination), the current value is replaced by the global consensus winner \(v^\star = \arg\max_{v \in \mathcal{V}} p(v)\). - Add (Omission Recovery): Recovers slots inadvertently dropped by ASR or the editor. If the slot is omitted (\(v_E = \emptyset\)) and belongs to the Add whitelist (17 slots), it is inserted if the top-scoring candidate is lexically attested and sufficiently confident: \(\alpha(v^\star) = 1\) and \(p(v^\star) > \tau\) (with threshold \(\tau=0.3\)). - Swap (Acoustic Replacement): Rectifies ungrounded entity corruptions. If the active value is unattested (\(\hat{v} \neq \emptyset\) and \(\alpha(\hat{v}) = 0\)) and falls into the Swap whitelist (5 high-risk entity slots, e.g., profile-name), it is swapped with the highest-scoring attested candidate \(\arg\max_{v \in \mathcal{V}: \alpha(v)=1} p(v)\). Restricting operators to verified slot subsets ensures that categorical attributes (e.g., hotel-parking) remain untouched by entity-centric voting.

Loss & Training

Both base spoken-DST model \(R\) and text editor \(E\) remain completely frozen throughout AVERT. The only trainable parameters are the attention query vector \(q\), slot embeddings \(e_s\), and the three-layer MLP of verifier \(\sigma\). Training data for \(\sigma\) is generated by sampling \(K=8\) candidate values per (turn, slot) from editor \(E\) on the SpokenWOZ training split, labeling matches against ground-truth values as positive (\(y=1\)) and all others as negative (\(y=0\)). To mitigate severe class imbalance, \(\sigma\) is optimized using focal binary cross-entropy: $\(\mathcal{L}_{\text{verifier}} = -\alpha_t (1 - p_t)^\gamma \log(p_t)\)$ with focusing parameter \(\gamma = 2.0\). The insertion threshold \(\tau = 0.3\) is calibrated on the development set. Verifier training requires only a single NVIDIA A100 GPU, adding a mere 56 ms per turn at inference (approximately 2% of the base model's latency) and maintaining a peak memory footprint of roughly 11.8 GB.

Key Experimental Results

Main Results

On the official SpokenWOZ test set under strictly causal per-turn evaluation, AVERT is benchmarked against published 1B-class baselines. Following standard protocol, named-entity slots are evaluated using normalized Levenshtein distance at a 0.90 similarity threshold (fuzzy match), while all remaining slots require exact string equality.

Model / System Category Decoder Parameters Spoken History Context Test JGA (%)
SPACE + WavLM (Si et al., 2023) 1B-class Single turn 25.65
Whisper + T5 (Druart et al., 2023) 1B-class Single turn 24.10
UBAR + GenWOZ (Gulzar et al., 2025) 1B-class Single turn 25.90
Sedláček et al. (2025) 1B Single turn 34.66
El Ghazal et al. (2026), compressed history 1B Compressed audio history 36.49
El Ghazal et al. (2026), full history 1B Full audio history 39.32
Base Model R (First-pass) 1B Single turn audio + text history 33.04
Editor E (Baseline w/o AVERT) 1B Cumulative ASR transcript 38.34
E + AVERT (Full Proposed Pipeline) 2 × 1B (R+E) + verifier Single turn audio + text history 40.13
Sedláček et al. (2025) 9B Single turn 42.17
El Ghazal et al. (2026), full history 9B Full audio history 45.52

Under strict exact matching without fuzzy tolerance (w/o fuzzy), the performance hierarchy remains consistent: Base \(R\) records 31.73%, Editor \(E\) achieves 36.12%, and \(E + \text{AVERT}\) reaches 37.66% (+1.54% absolute improvement), confirming that the gains reflect true entity recoveries rather than boundary leniency.

Ablation Study

The contribution of individual scoring signals and cumulative operators is evaluated on the SpokenWOZ test set.

Ablation Configuration / Variant Test JGA (%) \(\Delta\) vs. Full AVERT \(\Delta\) vs. Editor E
Editor E Baseline (no AVERT) 38.34 -1.79 —
AVERT (Full Pipeline) 40.13 — +1.79
Signal: without \(\sigma\) (uniform-weighted agreement) 39.74 -0.40 +1.40
Signal: \(\sigma\) with audio zeroed out 39.80 -0.33 +1.46
Signal: without \(\alpha\) attestation gate 38.95 -1.18 +0.61
Operator: + Vote alone 39.42 -0.71 +1.07
Operator: + Vote + Add 39.97 -0.16 +1.62
Operator: + Vote + Add + Swap (Full) 40.13 — +1.79
Restriction: B1 (unrestricted global Add + Swap) 38.14 -1.99 -0.20
Restriction: B2 (unrestricted global Vote + Add + Swap) 36.53 -3.60 -1.81

Key Findings

  • Acoustic signals drive verifier efficacy: Zeroing out the audio representation in \(\sigma\) drops JGA from 40.13% to 39.80% (barely 0.06% above the model without \(\sigma\) entirely), confirming that the verifier actively exploits acoustic frame features rather than memorizing slot-value priors.
  • Unrestricted voting induces severe negative transfer: As demonstrated in variant B2, eliminating slot restrictions and voting over all slots causes JGA to collapse to 36.53% (-3.60 points), falling significantly below the baseline editor (38.34%). This collapse is driven by overwriting correct categorical and boolean slot values (erroneous categorical/boolean changes skyrocket from 114 to 675).
  • Gains scale monotonically with dialogue length: The relative improvement of AVERT over editor \(E\) expands from +0.6% on turns 1–5 to +19.4% on turns 31+. In late-stage turns where the text editor degrades to 11.92% accuracy, cross-turn audio consensus provides an essential error-recovery backbone without inflating audio context windows.

Highlights & Insights

  • Audio as a Local Discriminator rather than a Global Generator: Decoupling acoustic verification from global sequence generation avoids the quadratic context blowup of audio-LLMs while outperforming 1B end-to-end full-audio systems at negligible additional computational cost (56 ms/turn).
  • Synergy between Continuous Acoustic Scoring and Discrete Lexical Gating: Continuous confidence scores \(\sigma(j, v)\) naturally break frequency-based voting ties, while discrete first-token attestation \(\alpha(v)\) prevents hallucinated entity insertion during state expansion.
  • Strict Slot-Subdomain Specialization: Demonstrating that generic heuristic voting across heterogeneous dialogue ontologies is harmful; restricting operations strictly to noise-prone entity slots safeguards categorical attributes.

Limitations & Future Work

  • Literal Attestation Ceiling: The first-token attestation gate requires at least partial representation in the cumulative transcript. Because only 21.0% of the editor's erroneous slots have their reference values attested in self-ASR transcripts, the remaining 79.0% represent a hard theoretical ceiling that necessitates soft phonetic or generative acoustic grounding in future iterations.
  • Supervised Slot-Subset Calibration: The selection of operational slot subsets and the Add threshold \(\tau\) depends on labeled development data, limiting immediate zero-shot deployment across unfamiliar schemas.
  • Two-Decoder Latency Overhead: Although avoiding joint retraining, chaining two separate 1B decoders sequentially incurs a cumulative latency of 4,148 ms per turn, which remains higher than a unified, single-pass compact model.
  • vs. Full-Audio Speech-LLMs (El Ghazal et al., 2026): They feed concatenated multi-turn raw speech directly into an LLM decoder, leading to severe computational scaling bottlenecks; AVERT maintains a compact text-only conversational history and queries audio only via a localized, single-turn scalar verifier.
  • vs. Transcript-Only Self-Correction (CorrectionLM, RSP-DST): Pure text editors cannot recover entities that recognizers consistently omit or phonetic misspellings that appear plausible in text; AVERT grounds post-hoc edits in raw acoustic evidence.
  • vs. ASR N-Best Hypothesis Selectors (ClozeGER, DeRAGEC): Prior cloze-style models select among ASR candidates at the lexical string level before dialogue state parsing; AVERT operates directly on downstream, schema-grounded candidate values across multi-turn trajectories.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Elegantly repositions audio as a post-hoc value verifier and formalizes three slot-restricted operators for multi-turn error recovery.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous causal benchmarking, thorough component and restriction ablations, bootstrap confidence intervals, and granular error typology breakdowns.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear structural exposition, transparent operational definitions, and reproducible formulation details.
  • Value: ⭐⭐⭐⭐☆ Highly pragmatic and cost-effective architectural blueprint for deploying spoken conversational assistants in compute-constrained environments.