HyLaT: Efficient Multi-Agent Communication via Hybrid Latent–Text Protocol¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/xymou/hylat
Area: LLM Agent
Keywords: multi-agent communication, latent-space reasoning, hybrid protocol, two-stage co-training, agent collaboration
TL;DR¶
HyLaT introduces a dual-channel hybrid communication protocol for multi-agent systems, funneling verbose intermediate reasoning through continuous latent vectors to eliminate token overhead while keeping critical conclusions in natural text for interpretability and interoperability, achieving competitive collaborative performance at single-agent operational costs.
Background & Motivation¶
Large language model (LLM)-based multi-agent systems have demonstrated remarkable potential in collaborative reasoning, complex planning, and human social simulation. In these multi-agent ecosystems, communication protocols serve as the fundamental backbone for information exchange, goal alignment, and collective decision-making across extended interactions. However, existing communication paradigms remain largely confined to single-channel architectures, succumbing to the fundamental "agent communication trilemma" where efficiency, portability, and versatility cannot be simultaneously satisfied. Text-based communication protocols are inherently human-readable and model-agnostic, but generating extensive chains of thought, detailed explanations, and arguments across multiple rounds leads to explosive token consumption and prohibitive latency bottlenecks. Conversely, emerging latent-space communication approaches exchange internal representations to achieve dense, compact information transfer, but they render inter-agent communication completely opaque to external human inspection and have predominantly been restricted to one-shot or unidirectional pipelines, failing to support interactive multi-turn dialogues in open settings.
This persistent tension between interpretability and efficiency stems from a structural mismatch: forcing heterogeneous communicative signals into a homogeneous medium. Grounded in multi-channel communication theory, agent messages are naturally composed of two functionally distinct categories of information. First, elaborate cognitive signals—such as step-by-step intermediate deductions, counterfactual reflections, and contextual elaborations—are dense, machine-centric processes well-suited for continuous latent representations. Second, concise critical signals—such as final commitments, selected answers, and summary agreements—must remain transparent, human-auditable, and interoperable across differing agent protocols. Forcing both into pure text inflates overhead, while forcing both into pure latent space discards interpretability and multi-round stability.
To resolve this trilemma, the authors reconsider channel architecture through channel multiplicity. Core idea: HyLaT introduces a dual-channel hybrid latent-text communication protocol that routes verbose cognitive reasoning through a compact continuous latent channel while preserving concise critical decisions in natural language, realized via a two-stage training scheme comprising single-agent hybrid generation learning and multi-agent interactive co-training.
Method¶
Overall Architecture¶
HyLaT coordinates a multi-agent system comprising \(N\) agents communicating over \(T\) interaction rounds. At each communication turn, an agent's response is decoupled into two complementary components: a continuous latent component \(Z\) containing \(k\) vectors delimited by learnable special tokens ⟨bot⟩ and ⟨eot⟩, followed by an autoregressively decoded discrete textual component \(Y\) conveying the final decision or commitment. To empower pre-trained LLMs to both produce such hybrid outputs and interpret peers' continuous representations across interactive turns, HyLaT employs a progressive two-stage training paradigm. Stage 1 (Single-Agent Hybrid Generation Learning) aligns internal hidden activations between a hybrid generation branch and a pure-text branch at a key target position, transferring explicit reasoning capacity into the continuous latent space. Stage 2 (Multi-Agent Interactive Co-Training) immerses agents into multi-turn, multi-agent debate and decomposition dialogues, enabling the shared backbone to jointly process peers' latent tokens alongside textual history.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Context & Peer Responses<br/>Multi-turn multi-agent dialogue inputs"] --> B["Dual-Channel Response Generation<br/>⟨bot⟩ + k latent vectors + ⟨eot⟩ + discrete text"]
B --> C["Single-Agent Hybrid Generation Learning<br/>Parallel forward passes & cross-channel hidden alignment"]
C --> D["Multi-Agent Interactive Co-Training<br/>End-to-end peer latent interpretation & round-by-round co-learning"]
D --> E["Collaborative Multi-Agent Output<br/>Low token overhead, transparency & cross-system interoperation"]
Key Designs¶
1. Dual-Channel Response Format: Disentangling Cognitive Reasoning from Action Commitments
Standard multi-agent frameworks conflate procedural rationale and final answers into a unified textual stream. HyLaT explicitly partitions the output of agent \(i\) at turn \(t\) into a continuous latent part and a discrete text sequence:
$\(o_i^{(t)} = [\langle\text{bot}\rangle, z_1, \dots, z_k, \langle\text{eot}\rangle] \parallel [y_1, \dots, y_m]\)$
The latent component \(Z = \{z_1, \dots, z_k\}\) consists of \(k\) continuous hidden states autoregressively generated from the model's final layer (fixed to \(k=6\) in practice), encoding dense cognitive reasoning. The textual component \(Y = \{y_1, \dots, y_m\}\) begins after ⟨eot⟩ and generates concise conclusions (e.g., "My answer is A"). This design compresses inter-agent bandwidth by orders of magnitude while preserving essential human-observable and protocol-agnostic decision checkpoints in plain text.
2. Single-Agent Hybrid Generation Learning: Cross-Channel Layer-Wise Alignment Because ground-truth continuous vectors do not exist, standard teacher-forcing cannot supervise the latent component directly. HyLaT overcomes this by running two parallel forward passes during Stage 1 over single-agent QA instances equipped with reasoning explanations \(e\) and final answers \(y\). The hybrid branch minimizes the language modeling loss on \(y\) conditioned on its own generated latent vectors \(Z\): \(\mathcal{L}_{\text{hybrid}} = -\frac{1}{|y|}\sum_i \log P(y_i \mid y_{1:i-1}, x, Z)\). Meanwhile, an auxiliary pure-text branch trains on \(r = [e, y]\) via standard teacher forcing: \(\mathcal{L}_{\text{text}} = -\frac{1}{|r|}\sum_i \log P(r_i \mid r_{1:i-1}, x)\). Crucially, the semantic content of \(e\) is transferred into the latent vectors by minimizing the mean squared error between the hidden activations of both branches across all \(L\) Transformer layers at the key target anchor token immediately preceding the final answer: $\(\mathcal{L}_{\text{align}} = \frac{1}{L} \sum_{l=1}^L \|\text{sg}(h_l^T) - h_l^S\|_2^2\)$ where \(h_l^T\) and \(h_l^S\) represent the activations in layer \(l\) for the text and hybrid passes, and \(\text{sg}(\cdot)\) is the stop-gradient operator. This anchor alignment forces the \(k\) latent vectors to capture the complete semantic gist of the explicit explanation.
3. Multi-Agent Interactive Co-Training: Peer Latent Interpretation and Joint Convergence Single-agent alignment alone does not teach an agent how to ingest latent vectors emitted by others. Stage 2 exposes the model to multi-agent, multi-turn dialogues where each agent's input incorporates peers' previous dual-channel responses: \(x_i^{(t)} = [x_i^{(t-1)}, \bigoplus_{j \neq i} o_j^{(t-1)}]\). The training data encompasses two collaborative workflows: refinement dialogues (where a weaker 1B model and a strong model engage in iterative debate to correct initial errors) and decomposition dialogues (where multi-hop questions are partitioned across agents and synthesized). By training jointly across all communicating agents on the aggregated multi-agent objective, agents learn to attend to peers' latent vectors as contextual keys and values while relying on intermediate text summaries as stable semantic anchors, preventing compounding representational drift across rounds.
Loss & Training¶
The Stage 1 training objective is formulated as: $\(\mathcal{L}_{\text{stage1}} = \alpha \mathcal{L}_{\text{hybrid}} + \beta \mathcal{L}_{\text{text}} + \gamma \mathcal{L}_{\text{align}}\)$ with hyperparameters \(\alpha=1, \beta=1, \gamma=20\). In Stage 2, the loss is averaged across all \(N\) communicating agents: $\(\mathcal{L}_{\text{stage2}} = \frac{1}{N} \sum_{i=1}^N \left( \alpha \mathcal{L}_{\text{hybrid}}^{(i)} + \beta \mathcal{L}_{\text{text}}^{(i)} + \gamma \mathcal{L}_{\text{align}}^{(i)} \right)\)$ For optimization, LoRA fine-tuning (rank \(r=128\), \(\alpha=32\)) is applied to all attention projection layers of the Llama-3.2-1B-Instruct backbone (and verified on Llama-3.2-3B and Qwen2-1.5B). Training is conducted across 8 NVIDIA H200 GPUs using AdamW with cosine learning rate scheduling (peak LR \(8\times 10^{-4}\) in Stage 1 and \(3\times 10^{-4}\) in Stage 2) at batch size 128. For heterogeneous multi-agent communication (e.g., Llama-3.2-1B collaborating with Qwen2-1.5B), a lightweight MLP adapter is placed before the input layer of each agent to project received cross-family latent vectors into its native semantic space.
Key Experimental Results¶
Main Results¶
The primary evaluation assesses multi-agent debate (MAD) across 5 in-domain benchmarks (CommonsenseQA, StrategyQA, SocialIQA, WorldTree, PubMedQA) and 3 out-of-domain benchmarks (MedQA, ARC-Easy, ARC-Challenge), using \(N=3\) agents communicating over \(T=2\) rounds with Llama-3.2-1B-Instruct.
| Paradigm | Method | In-Domain Avg. Acc | In-Domain Maj. Acc | Out-of-Domain Avg. Acc | Out-of-Domain Maj. Acc | # Tokens / Query | Wall-Clock Time (s) |
|---|---|---|---|---|---|---|---|
| Training-Free Single-Agent | Single-Agent | 53.33 | 53.33 | 34.33 | 34.33 | 113.59 | 1.25 |
| Training-Free Text-Based | NL (Natural Language) | 57.24 | 60.20 | 36.59 | 37.45 | 1247.50 | 13.15 |
| Training-Free Text-Based | AutoForm | 57.03 | 58.80 | 37.85 | 38.44 | 1717.60 | 18.18 |
| Training-Free Text-Based | EcoLANG | 55.20 | 58.13 | 38.04 | 38.45 | 960.26 | 10.04 |
| Training-Free Text-Based | SDE | 54.45 | 56.67 | 36.15 | 36.89 | 1316.20 | 14.48 |
| Training-Free Latent-Space | Cipher | 57.11 | 59.20 | 37.60 | 38.22 | 1137.89 | 11.58 |
| Training-Free Latent-Space | LatentMAS-V | 43.56 | 44.07 | 29.11 | 29.74 | 324.43 | 3.61 |
| Training-Based | TextFullT | 71.09 | 72.66 | 40.59 | 40.67 | 505.03 | 5.47 |
| Training-Based | LatentFullT | 66.91 | 67.06 | 38.26 | 38.33 | 57.00 | 0.70 |
| Hybrid Protocol (Ours) | HyLaT | 69.53 | 69.80 | 38.33 | 38.55 | 72.01 | 1.47 |
Note: Aggregate in-domain and out-of-domain accuracies reflect the mean across the evaluated benchmark sets from Table 1. On SocialIQA, HyLaT reaches 82.56% (Avg.) / 83.00% (Maj.), outperforming all baseline models.
Ablation Study¶
The ablation study evaluates channel configurations and training stages across all benchmark datasets (Table 2):
| Configuration | Avg. Acc (%) | Maj. Acc (%) | Token Count (# token) | Wall-Clock Time (s) | Format Error (%) | Note |
|---|---|---|---|---|---|---|
| HyLaT (Full Framework) | 57.83 | 58.08 | 72.01 | 1.47 | 0.00 | Full dual-channel design with two-stage co-training |
| Channel Variant: w/ pure text | 56.93 | 57.00 | 639.18 | 13.18 | 3.28 | Text throughout all rounds; 8.9x higher latency |
| Channel Variant: w/ pure latent | 50.28 | 50.33 | 56.09 | 1.16 | 6.12 | Latent only in intermediate rounds; -7.75% accuracy drop |
| Stage Variant: w/o Stage 1 | 38.43 | 38.58 | 102.59 | 1.38 | 9.13 | Skips single-agent alignment; severe representation noise |
| Stage Variant: w/o Stage 2 | 43.64 | 43.96 | 136.10 | 2.87 | 22.27 | Skips multi-agent co-learning; 22.27% format error rate |
Key Findings¶
- Crucial Complementarity of Dual Channels: Restricting communication to pure text causes token usage and latency to surge by nearly an order of magnitude (from 72 to 639 tokens, 1.47s to 13.18s). Restricting intermediate rounds to pure latent tokens causes majority accuracy to drop sharply by 7.75% with a 6.12% format failure rate, confirming that intermediate natural text acts as a vital semantic anchor across rounds.
- Interdependent Two-Stage Learning: Omitting Stage 1 reduces accuracy by nearly 20 percentage points due to noisy, unanchored latent states. Omitting Stage 2 produces a 22.27% format error rate, demonstrating that interpreting peer continuous vectors requires explicit multi-agent interaction co-training.
- Robustness Against Latent Perturbation: Injecting Gaussian noise up to \(\sigma=0.5\) into latent vectors yields minimal performance degradation, as the discrete textual channel reliably protects the final commitment from catastrophic corruption.
- Inherent Latent Geometry: Principal component analysis (PCA) shows that the first two principal components explain 70.5% of the total variance across datasets. Embeddings form well-separated clusters corresponding to distinct semantic domains without explicit domain supervision.
Highlights & Insights¶
- Resolution of the Communication Trilemma: By drawing inspiration from classical multi-channel communication theory, HyLaT successfully decouples cognitive reasoning from actionable commitments, achieving an optimal trade-off between machine efficiency and human interpretability.
- Zero-Annotation Semantic Distillation: Aligning cross-branch hidden representations at the pre-answer anchor token allows effective supervision of continuous latent vectors directly from text rationale datasets, circumventing manual vector labeling.
- Heterogeneous Model Interoperability: With a simple linear/MLP adapter bridging representational dimensions, HyLaT enables seamless latent-text collaboration between distinct model architectures (e.g., Llama and Qwen) without degrading group consensus capabilities.
Limitations & Future Work¶
- Model Parameter Scaling: Current experiments are primarily restricted to 1B and 3B models due to computational limits; validating the protocol on 8B+ or 70B models remains an open goal.
- Incompatibility with High-Throughput Engines: Non-standard generation involving dynamic injection of continuous vectors into KV-caches prevents out-of-the-box deployment with modern serving backends like vLLM.
- Limited Stylistic Diversity in Text Channel: Training data is dominated by QA debate and decomposition templates with fixed prompt formatting, leaving rich conversational behaviors (such as negotiation and socio-emotional exchanges) underexplored.
Related Work & Insights¶
- vs. Prompt-Level Compression (EcoLANG, AutoForm): Textual prompt compression still incurs significant token generation costs; HyLaT bypasses token generation bottlenecks via continuous vector channels, operating at single-agent costs.
- vs. Pure Latent Collaboration (LatentMAS-V, Cipher): Pure latent schemes forfeit human oversight and suffer from error compounding in multi-turn dialogues; HyLaT retains explicit text anchors to ensure stability and user auditing.
- vs. Internal Latent Reasoning (CoDi, Sim-CoT): While previous works explored latent tokens for single-agent chain-of-thought compression, HyLaT successfully extends continuous reasoning across distributed multi-agent collaborative networks.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant dual-channel formulation resolving the efficiency-interpretability trilemma with two-stage co-training.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation spanning 8 benchmarks, heterogeneous pairs, human trust games, noise injections, attention analysis, and latent PCA probing.
- Writing Quality: ⭐⭐⭐⭐⭐ Clearly structured, conceptually sound, and supported by thorough visualizations.
- Value: ⭐⭐⭐⭐⭐ Provides an impactful blueprint for scaling large-scale multi-agent coordination under practical bandwidth limits.