XBridge: Entity-Grounded Latent Bridge for Heterogeneous LLM Communication¶
Conference: NeurIPS 2026
arXiv: 2608.11676
Code: https://github.com/WooseongYang/XBridge
Area: Others
Keywords: Heterogeneous LLM Communication, Multi-Agent Systems, Latent Bridge, Entity Grounding, Decode-Free Communication
TL;DR¶
XBridge introduces the first decode-free dual-channel communication protocol for heterogeneous LLM multi-agent systems, aligning discrete entity identities via deterministic lexical anchor mapping while querying latent contextual enrichment through gated cross-attention, outperforming natural language communication by 14.3โ21.4 pp across three model families and seven benchmarks with 11ร lower latency.
Background & Motivation¶
Heterogeneous multi-agent systems (MAS) powered by diverse LLM families demonstrate clear advantages over homogeneous configurations on complex reasoning tasks, because distinct tokenizers, training corpora, and architectural inductive biases help alleviate redundant majority-voting degeneration and shared blind spots. However, this representational diversity inevitably introduces a fundamental bottleneck: collaborating agents must exchange mission-critical information across structurally incompatible latent spaces.
Existing communication paradigms are fundamentally constrained by an efficiency-fidelity trade-off. Text-based protocols like natural language communication (NLComm) are natively architecture-agnostic, but they require costly autoregressive decoding at every communication stepโincurring order-of-magnitude latency overheads while compressing rich internal representations into lossy textual summaries. Conversely, latent-level protocols such as KV cache sharing and hidden-state injection typically presume homogeneous architectures with matching hidden dimensions, attention heads, and positional encodings. When continuous learned projections or adapters are deployed across disparate model families, they suffer from the "entity grounding problem": the continuous latent bottleneck undergoes rare-token compression collapse, reliably transmitting abstract contextual semantics while completely discarding discrete entity identities such as names, dates, and domain-specific identifiers (causing bridge-only F1 scores to plummet to approximately 30%).
The critical obstacle in heterogeneous multi-agent collaboration lies in simultaneously eliminating the severe decoding latency of text while preserving exact lexical entity identities alongside deep contextual semantics. The core idea is to decouple heterogeneous communication into discrete entity anchors and continuous contextual enrichment, employing deterministic Lexical Anchor Mapping (LAM) to provide lossless receiver-native token embeddings as structural anchors, combined with an inserted Latent Enrichment Bridge (LEB) that lets the receiver query the sender's deep hidden states via gated cross-attention.
Method¶
Overall Architecture¶
XBridge operates under an asymmetric communication protocol: the sender model \(M_S\) observes the full context document \(C\), while the receiver model \(M_R\) observes only the question \(Q\), tasking \(M_R\) with generating the target answer \(A\) conditioned on transmitted signals. Throughout this pipeline, the sender executes a single forward pass without autoregressive decoding and remains frozen; the receiver backbone is likewise kept completely frozen, training only lightweight inserted cross-attention bridge modules.
The overall architecture integrates two complementary channels: discrete Lexical Anchor Mapping (LAM) and continuous Latent Enrichment Bridge (LEB). The sender's single forward pass simultaneously yields raw context tokens \(c_S\) and its last-layer hidden states \(H_S\). LAM deterministically projects \(c_S\) into the receiver's vocabulary and maps them to receiver-native embeddings \(e_{\text{ctx}}\), which are prepended to the receiver's question input. Meanwhile, LEB places lightweight gated cross-attention modules after four selected receiver layers, allowing the receiver's intermediate hidden states to serve as queries that extract relevant contextual signals from \(H_S\). The receiver's native self-attention subsequently binds the continuous contextual representations to the discrete lexical anchors.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Context Document C & Question Q<br/>Sender Ms observes C / Receiver Mr observes Q"] --> B["Sender Single Forward Pass<br/>Extracts Hidden States Hs & Context Tokens Cs"]
B --> C["Lexical Anchor Mapping LAM<br/>Cross-Vocab Mapping + Native Embedding Lookup"]
B --> D["Latent Enrichment Bridge LEB<br/>4 Interleaved Gated Cross-Attention Modules"]
C --> E["Entity-Grounded Bridge Integration<br/>Self-Attention Binds Contextual Enrichment to Anchors"]
D --> E
E --> F["Answer Generation A<br/>Receiver Generates Target Tokens Autoregressively"]
Key Designs¶
1. Lexical Anchor Mapping (LAM): Deterministic Discrete Entity Grounding To resolve rare-token compression collapse where continuous projections erase exact entity identities, LAM establishes an offline precomputed, deterministic mapping function \(\phi: \mathcal{V}_S \to \mathcal{V}_R^*\). For tokens whose surface text strings exist identically in both vocabularies, \(\phi\) executes an instantaneous direct ID-to-ID lookup (e.g., achieving a 97.1% direct match rate between Llama and Qwen). For unaligned tokens, a lossless string fallback decodes the token into its surface string and immediately re-tokenizes it using the receiver's native tokenizer (for instance, mapping the multi-digit token "199" into discrete tokens [1, 9, 9]). The mapped IDs are then converted into receiver embeddings via the receiver's frozen embedding matrix \(E_R\), producing \(e_{\text{ctx}} = [E_R[\phi(c_1)], \dots, E_R[\phi(c_{T_C})]]\) which is prepended to the question input \(x_R = [e_{\text{ctx}} ; E_R(Q)]\). This operation adds negligible latency (<1 ms) while guaranteeing that transmitted tokens reside strictly within the receiver's native representation manifold.
2. Latent Enrichment Bridge (LEB): Receiver-Driven Context Retrieval While LAM guarantees entity fidelity, raw token embeddings lack the deep contextual reasoning and relational understanding developed across the sender's full transformer stack. Directly injecting the sender's final hidden states \(H_S\) into the receiver creates severe out-of-distribution activation mismatch. LEB addresses this by inserting \(M=4\) lightweight gated cross-attention modules after receiver layers \(\ell \in \{7, 14, 21, 28\}\) of its 28 total layers. Each module projects the receiver's intermediate hidden state \(h_R^{(\ell)}\) into a Query, while projecting the sender's last-layer states \(H_S\) into Keys and Values, performing cross-dimensional translation and context retrieval: $\(Q^{(\ell)} = W_Q^{(\ell)} \cdot \mathrm{LN}_R^{(\ell)}(h_R^{(\ell)}), \quad K^{(\ell)} = W_K^{(\ell)} \cdot \mathrm{LN}_S^{(\ell)}(H_S), \quad V^{(\ell)} = W_V^{(\ell)} \cdot \mathrm{LN}_S^{(\ell)}(H_S)\)$ $\(A^{(\ell)} = \mathrm{softmax}\left(\frac{Q^{(\ell)} K^{(\ell)\top}}{\sqrt{d_k}}\right) \cdot V^{(\ell)}\)$ The retrieved contextual update is injected back into the receiver's residual stream via a learnable hyperbolic tangent gate: $\(h_R^{\prime(\ell)} = h_R^{(\ell)} + \tanh(\alpha^{(\ell)}) \cdot A^{(\ell)}\)$ The gating parameter is initialized with a warm start \(\alpha^{(\ell)} = 1.0\) (\(\tanh(1.0) \approx 0.76\)), enabling immediate utilization of cross-architecture latent signals under tight data budgets. Each module requires only ~66M parameters, totaling ~264M parameters (just 3.8% of a 7B receiver backbone).
3. Entity-Grounded Integration & Orthogonal Role Separation: Disentangling "Who" from "How" XBridge requires no specialized fusion layers, relying entirely on the receiver's inherent self-attention. As LAM-derived discrete entity embeddings traverse the receiver's early layers, they shape \(h_R^{(\ell)}\) at each depth. Consequently, when LEB computes queries at layer \(\ell\), the Query already encapsulates where specific entities are located, steering cross-attention to retrieve relevant sender contextual states. An entity perturbation experiment on HotpotQA rigorously demonstrates this orthogonal separation: swapping the LEB input while holding LAM constant yields zero change in the generated entity identity (45 original vs. 0 swapped); in contrast, swapping the LAM input immediately causes the generated entity to flip to the perturbed entity (37 swapped vs. 4โ5 original). This confirms that LAM deterministically establishes which entities appear in the output, while LEB governs how the receiver reasons over them.
Loss & Training¶
The training protocol for LEB is remarkably parameter-efficient and fast. Both the sender and receiver model backbones remain frozen throughout training. Because the sender's outputs (\(c_S\) token IDs and final hidden states \(H_S\)) and receiver embeddings \(e_{\text{ctx}}\) are deterministic for any sample, they are precomputed and cached offline.
The bridge modules are trained end-to-end using standard next-token prediction (NTP) cross-entropy loss over the target answer tokens: $\(\mathcal{L}(\theta_{\text{bridge}}) = -\sum_{t=1}^{|A|} \log P_R(a_t \mid a_{<t}, e_{\text{ctx}}, Q; \theta_{\text{bridge}})\)$ Regarding training data composition, training solely on a single task (e.g., 20K HotpotQA samples) causes catastrophic degradation on unseen tasks (Countries F1 drops to 27.7%). In contrast, a compact, balanced dataset of only 587 samples uniformly distributed across the seven benchmark domains converges in under 10 minutes on a single GPU. Once trained, the bridge generalizes robustly across all evaluation tasks without any downstream adaptation.
Key Experimental Results¶
Main Results¶
Evaluation spans three distinct model familiesโLlama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3โacross seven QA and reasoning benchmarks. Performance is measured via F1 score (%).
| Sender \(\to\) Receiver | Method | Countries | Tipsheets | HotpotQA | QASPER | MuSiQue | MFldQA | 2Wiki | Avg F1 | Relative Latency |
|---|---|---|---|---|---|---|---|---|---|---|
| Llama-8B \(\to\) Qwen-7B | NoComm (Receiver only) | 0.0 | 66.5 | 24.6 | 7.1 | 10.6 | 17.3 | 19.3 | 20.8 | \(1.0\times\) (0.09s) |
| FullComm (Direct Context) | 50.8 | 98.5 | 78.4 | 31.3 | 31.5 | 48.2 | 47.6 | 55.2 | \(1.4\times\) (0.13s) | |
| NLComm\(_{\text{hetero}}\) (Text Comm) | 22.6 | 96.3 | 68.5 | 33.7 | 32.5 | 24.8 | 14.2 | 41.8 | \(18.9\times\) (1.70s) | |
| XBridge (Ours) | 72.5 | 99.8 | 78.8 | 47.5 | 48.2 | 44.2 | 51.1 | 63.2 | \(1.7\times\) (0.15s) | |
| XBridge Gain vs Second-Best | +49.9 | +3.5 | +10.3 | +13.8 | +15.7 | +19.4 | +36.9 | +21.4 | 11.3ร speedup | |
| Qwen-7B \(\to\) Llama-8B | NoComm | 0.0 | 11.0 | 31.4 | 3.8 | 2.6 | 11.2 | 12.9 | 10.4 | - |
| FullComm | 67.0 | 95.0 | 82.9 | 36.7 | 41.3 | 53.3 | 51.8 | 61.2 | - | |
| NLComm\(_{\text{hetero}}\) | 56.2 | 94.8 | 76.8 | 34.1 | 33.5 | 21.7 | 16.3 | 47.6 | - | |
| XBridge (Ours) | 62.0 | 99.0 | 80.6 | 46.6 | 62.8 | 40.3 | 42.1 | 61.9 | - | |
| XBridge Gain vs Second-Best | +5.8 | +4.2 | +3.8 | +12.5 | +29.3 | +18.6 | +25.8 | +14.3 | - | |
| Mistral-7B \(\to\) Qwen-7B | NoComm | 0.0 | 66.5 | 24.6 | 7.1 | 10.6 | 17.3 | 19.3 | 20.8 | - |
| FullComm | 50.8 | 98.5 | 78.4 | 31.3 | 31.5 | 48.2 | 47.6 | 55.2 | - | |
| NLComm\(_{\text{hetero}}\) | 43.6 | 96.0 | 66.1 | 31.8 | 28.3 | 27.0 | 15.9 | 44.1 | - | |
| XBridge (Ours) | 70.0 | 100.0 | 84.9 | 50.6 | 65.0 | 35.8 | 48.7 | 65.0 | - | |
| XBridge Gain vs Second-Best | +26.4 | +4.0 | +18.8 | +18.8 | +36.7 | +8.8 | +32.8 | +20.9 | - |
Ablation Study¶
Ablations on HotpotQA (Llama-3.1-8B \(\to\) Qwen2.5-7B) evaluate core channel contributions, module density, initialization schemes, and homogeneous baselines.
| Ablation Variant | HotpotQA F1 (%) | 7-Task Avg F1 (%) | Relative Margin | Mechanism Analysis |
|---|---|---|---|---|
| Full XBridge | 78.8 | 63.2 | Baseline | LAM discrete anchors + 4-layer LEB latent enrichment |
| w/o LAM (XBridge\(_{\text{-LAM}}\), latent-only) | 30.3 | - | -48.5 | Suffers rare-token compression collapse; loses entity anchors |
| w/o LEB (XBridge\(_{\text{-LEB}}\), token-only) | 56.5 | 44.5 | -22.3 | Preserves lexical tokens but lacks sender contextual states |
| No Communication (NoComm) | 24.6 | 20.8 | -54.2 | Lower reference bound; receiver sees only question |
| Number of modules: 1 module (66M) | 66.3 | - | -12.5 | Insufficient parameter capacity for cross-architecture translation |
| Number of modules: 2 modules (132M) | 74.6 | - | -4.2 | Suboptimal cross-layer contextual coverage |
| Number of modules: 7 modules (462M) | 75.8 | - | -3.0 | Overfitting under compact 587-sample training budget |
| Gate init \(\alpha = 0.0\) (Flamingo default) | 72.7 | - | -6.1 | Cold start slows convergence under small-data regime |
| Gate init \(\alpha = 1.0\) (Ours warm start) | 78.8 | - | 0.0 | Accelerates convergence and maximizes representation alignment |
| Homogeneous setting: KVComm (KV-sharing) | 65.3 | 51.1 | -12.0 | In Qwen \(\to\) Qwen, XBridge\(_{\text{homo}}\) (63.1%) wins on 6 of 7 tasks |
Key Findings¶
- Dual-Channel Synergy: Removing LAM causes performance to plummet from 78.8% to 30.3%, verifying that pure continuous latent transfer cannot maintain entity fidelity; removing LEB drops performance by 22.3 pp, proving raw token mappings alone fail to capture multi-hop dependencies. The combined gain (+54.2 pp) far exceeds the sum of isolated components, validating the core principle of entity-grounded latent enrichment.
- Surpassing Full-Context Skyline: XBridge consistently surpasses the theoretical upper reference FullComm across all pairs (e.g., Mistral \(\to\) Qwen reaches 65.0% vs. FullComm's 55.2%). This occurs because the receiver not only accesses the mapped context tokens but simultaneously receives the sender's 32-layer integrated contextual representations directly into its intermediate layers, bypassing the need to reconstruct complex multi-hop reasoning chains from scratch.
- Reasoning Complexity Dependency: LEB provides the largest boost on tasks demanding implicit world knowledge inference (Countries +35.8 pp) and multi-hop compositional reasoning (HotpotQA, 2WikiMQA, MuSiQue +19 to +22 pp). On single-sentence extraction over long documents (MultiFieldQA), LEB adds only +7.0 pp, reflecting that latent enrichment specifically benefits non-trivial relational synthesis.
Highlights & Insights¶
- Decoupling Identity from Reasoning: XBridge resolves the persistent dilemma in continuous cross-model communication by letting deterministic table lookup resolve "what entities exist" while continuous cross-attention resolves "how entities interact", completely bypassing token compression collapse.
- Ultra-Efficient Micro-Tuning: Comprising only 264M parameters, LEB trains in under 10 minutes on 587 balanced samples on a single GPU using precomputed hidden-state caches, requiring zero fine-tuning of the sender or receiver LLMs.
- Zero-Shot Multi-Agent Composability: Independently trained bridges (Llama-to-Qwen and Mistral-to-Qwen) compose out-of-the-box at inference time without joint tuning, allowing a single Qwen receiver to integrate multi-document reasoning from dual senders and achieving 70.4% F1 on HotpotQA (outperforming both single-sender FullComm at 67.0% and dual-sender NLComm at 56.8%).
Limitations & Future Work¶
- Unidirectional and Single-Turn Scope: The current protocol is validated exclusively on one-way, single-exchange communication. Extending this to multi-turn latent dialogue or peer-to-peer negotiation requires dynamic state synchronization mechanisms.
- Scaling Bottleneck for Larger Senders: When scaling the sender to 14B (hidden dimension 5120), the 587-sample data budget becomes a bottleneck, causing performance to plateau at 62.1%; scaling the balanced training set or exploring contrastive pre-training represents an important next step.
- Multi-Agent Network Expansion: Extending the decode-free cross-attention bus to many-agent collaborative environments and interactive code/tool generation settings remains an open avenue.
Related Work & Insights¶
- vs Natural Language Communication (NLComm): NLComm relies on text summaries, incurring 11ร higher inference latency due to autoregressive decoding while compressing nuanced hidden states into coarse text; XBridge achieves single-pass decode-free transfer with higher fidelity.
- vs KV-Cache Sharing (KVComm): KVComm strictly mandates architectural homogeneity (identical hidden dimensions and head counts); XBridge bridges cross-architecture discrepancies via learned cross-attention while outperforming KVComm on 6 of 7 tasks in homogeneous setups.
- vs Continuous Projection Protocols (C2C / CIPHER / MoT): Existing continuous protocols either require identical tokenizers or presume symmetric inputs where the receiver already sees the full context; XBridge resolves the realistic asymmetric setting where only the sender observes the document, introducing explicit discrete anchors to prevent rare-token compression collapse.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneers the identification of entity grounding collapse in heterogeneous LLM communication and provides an elegant, decode-free dual-channel solution]
- Experimental Thoroughness: โญโญโญโญโญ [Comprehensive evaluations across three leading model families, seven diverse benchmarks, causal entity perturbation, and zero-shot multi-agent composition]
- Writing Quality: โญโญโญโญโญ [Mathematically rigorous problem formulation, crystal-clear conceptual narratives, and tightly integrated ablation analyses]
- Value: โญโญโญโญโญ [Lays an essential architectural foundation for high-throughput, low-latency cross-model communication in heterogeneous multi-agent systems]