Not All Queries Are Equal: Query-Type-Aware Mixture-of-Experts for Temporal Knowledge Graph Forecasting¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/wooden070/QTI-MoE
Area: LLM Reasoning
Keywords: Temporal Knowledge Graph Forecasting, Mixture-of-Experts, Inductive Biases, Query-Aware Routing, Knowledge Extrapolation
TL;DR¶
QTI-MoE uncovers the severe performance drop of monolithic temporal knowledge graph forecasters in sparse and high-diversity historical regimes, categorizes queries into sparse, recurrent, and ambiguous types with offline LLM-refined soft labels, and routes queries to three specialized experts with complementary memory inductive biases using pure query embeddings at test time.
Background & Motivation¶
Temporal Knowledge Graphs (TKGs) encode real-world events as timestamped relational facts \((s, r, o, t)\), tracking how entities interact dynamically over continuous timelines. TKG forecasting aims to predict missing entities for future queries \((s, r, ?, t_q)\) given historical snapshots \(G_{<t_q}\), serving as a core foundation for foreign-affairs analysis, social crisis forecasting, and geopolitical conflict early warning. Existing research has evolved along two primary paradigms: structure-based methods (such as RE-GCN, HisMatch, and LogCL) that model multi-relational evolution via graph convolutions and recurrent neural architectures, and large language model (LLM) based approaches (such as TiRGN-ICL, GenTKG, and SRM-LLM) that cast forecasting as semantic extrapolation over linearized historical event sequences.
Despite architectural divergence, virtually all existing methods operate on a shared premise: routing every query through a monolithic reasoning pipeline. Even recent multi-expert formulations like MESH confine routing within a single query—deciding whether to trust structural or semantic representations—while treating all forecasting queries as interchangeable. However, queries sharing identical predicate structures often rest on starkly disparate historical evidence: some queries possess sparse records where the target entity has rarely appeared; others exhibit stable recurrent repetitions; and many present ambiguous contexts packed with competing candidate entities. A stratified empirical audit of representative forecasters on ICEWS14 reveals an inverted-U performance curve, where accuracy severely deteriorates in low-frequency and high-diversity regimes, indicating that no single monolithic pipeline can satisfy conflicting inductive biases across diverse query regimes.
Forcing a single model to accommodate all historical patterns inevitably triggers negative transfer across incompatible inductive biases and causes catastrophic degradation on boundary samples. A principled solution is to decouple forecasting queries according to their historical prefix geometry into specialized expert pathways, routing queries dynamically based on their temporal state representations. Core idea: characterize temporal queries into sparse, recurrent, and ambiguous regimes via prefix statistics with offline frozen LLM boundary refinement, instantiate three specialized experts sharing a graph backbone but equipped with complementary memory pathways, and dynamically mix them via a query-aware router using test-time query representations without any online LLM inference.
Method¶
Overall Architecture¶
The QTI-MoE framework operates through three coordinated stages: 1. Query-Type Soft Label Construction (Step 1): In offline preprocessing on the training set, three prefix statistics (event frequency, candidate diversity, and target occurrence count) are mapped to low/mid/high memberships via Sigmoids to construct rule-based soft labels \(\hat{q}_{rule}\). A frozen LLM (InternLM2-7B) refines uncertain boundary queries where rule confidence is ambiguous, yielding fused soft labels \(\hat{q}(s, r)\); 2. Type-Specialized Experts (Step 2): Three independent experts \(\{ \mathcal{E}_{sp}, \mathcal{E}_{re}, \mathcal{E}_{am} \}\) share a recurrent R-GCN + GRU + ConvTransE structural backbone but maintain decoupled parameters and distinct memory mechanisms: exponential recency decay for sparse queries, count-based repetition accumulation for recurrent queries, and candidate pool log-diversity normalization for ambiguous queries. An internal query-conditioned gate \(\alpha_\tau\) blends the structural and memory distributions; 3. Mixture-of-Experts Query-Aware Routing (Step 3): During inference, neither soft labels, privileged target statistics, nor LLMs are accessed. A lightweight router consumes the concatenated subject hidden state and relation embedding \(f_{sr} = [h_s; e_r]\) to predict a normalized mixing distribution \(\pi = (\pi_{sp}, \pi_{re}, \pi_{am})\), synthesizing the final candidate entity probability distribution \(p_{final}\) in log space.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Query (s, r, ?, tq)<br/>and Historical Event Snapshots"] --> B["Query-Type Soft Label Construction<br/>Prefix Statistics + Frozen LLM Refinement"]
B -.->|Offline Training Signal L_distill| E["Mixture-of-Experts Query-Aware Routing<br/>Query Embedding fsr Drives Dynamic Routing π"]
A --> C["Shared Structural Backbone Encoding<br/>R-GCN + GRU + ConvTransE"]
C --> D["Type-Specialized Complementary Memory<br/>Sparse Recency / Recurrent Count / Ambiguous Diversity"]
D --> E
E --> F["Log-Space Mixture Distribution<br/>Final Predicted Entity Ranking p_final"]
Key Designs¶
1. Query-Type Soft Label Construction: Multi-Factor Geometry and Selective LLM Refinement
Hard rule partitioning introduces abrupt boundary artifacts, while querying LLMs across massive temporal graphs incurs prohibitive latency. QTI-MoE introduces a hybrid offline soft labeling scheme. For each training query \((s, r)\), it extracts three strict temporal prefix scalars: event frequency \(f_{sr}\), candidate diversity \(d_{sr}\), and target occurrence count \(g^*_{sr}\). Each scalar is transformed into low, medium, and high membership degrees via Sigmoid functions calibrated against dataset percentiles. The rule score for each type is computed via geometric mean: sparse requires all three scalars to be jointly low, recurrent requires moderate frequency/diversity with high target recurrence, and ambiguous demands jointly high frequency and diversity. The geometric formulation guarantees that a type score collapses to zero whenever any defining prerequisite fails. After \(\ell_1\) normalization, base rule distribution \(\hat{q}_{rule}\) is produced. Only on boundary subset \(\mathcal{B}\)—where top-two rule probabilities differ by less than \(\epsilon\) or statistics lie near threshold boundaries—an instruction containing relation surface text and scaled statistics is passed to a frozen InternLM2-7B. The resulting next-token probability distribution \(\hat{q}_{llm}\) over label letters \(\{A, B, C\}\) is blended with the rule: $\(\hat{q}(s, r) = (1 - w) \hat{q}_{rule} + w \hat{q}_{llm}\)$ The blending weight \(w\) is zero outside \(\mathcal{B}\), and within \(\mathcal{B}\) it is bounded by the product of LLM confidence and its disagreement with \(\hat{q}_{rule}\). Crucially, \(g^*_{sr}\) acts solely as privileged training supervision during offline label generation and is never exposed to the router or inference pipeline.
2. Type-Specialized Complementary Memory: Decoupled Inductive Biases
Uniform memory modules cannot simultaneously prioritize transient recency and suppress candidate interference. QTI-MoE outfits each expert with a dedicated memory aggregation operator over identical extracted prefix evidence \(\mathcal{I}_{sr} = \{(\Delta t_i, o_i, c_i)\}_i\): - Sparse Expert (\(\mathcal{E}_{sp}\)): For queries lacking historical density, recent interactions offer disproportionately strong clues. It applies learnable exponential decay \(\tau_{sp} > 0\) paired with logarithmic frequency dampening: $\(m_{sp}(o) = \sum_{i \in \mathcal{I}_{sr}: o_i = o} e^{-\Delta t_i / \tau_{sp}} \log(1 + c_i)\)$ - Recurrent Expert (\(\mathcal{E}_{re}\)): In periodic or repetitive interactions, total co-occurrence count provides far more stability than individual timestamps. It aggregates raw historical occurrences without expert-specific learnable parameters: $\(m_{re}(o) = \log(1 + \tilde{c}_o), \quad \tilde{c}_o = \sum_{i \in \mathcal{I}_{sr}: o_i = o} c_i\)$ - Ambiguous Expert (\(\mathcal{E}_{am}\)): In queries with high candidate cardinality, raw repetition counts bias predictions toward frequent false-positive distractors. It scales the count score by the log-diversity of distinct candidate entities \(\mathcal{U}_{sr} = \{o_i \mid i \in \mathcal{I}_{sr}\}\): $\(m_{am}(o) = \log\left(1 + \frac{\tilde{c}_o}{\log(1 + |\mathcal{U}_{sr}|)}\right)\)$ Each memory vector \(m_\tau\) is converted to \(p_\tau^{mem} = \text{softmax}(m_\tau)\) and blended with the structural backbone prediction \(p_\tau^{str}\) via an expert-specific query-conditioned gate \(\alpha_\tau = \sigma(\text{MLP}_\tau([h_s^\tau; e_r^\tau]))\): \(p_\tau = (1 - \alpha_\tau) p_\tau^{str} + \alpha_\tau p_\tau^{mem}\). When historical evidence is absent, memory scores vanish and the gate naturally falls back to pure structural prediction.
3. Mixture-of-Experts Query-Aware Routing: History-Conditioned Embeddings without Online LLM Calls
To eliminate online LLM overhead and prevent expert representation collapse, QTI-MoE introduces a lightweight neural router that maps query embedding \(f_{sr} = [h_s; e_r]\) to an expert mixture distribution \(\pi\): $\(\pi = \text{softmax}(\text{MLP}_{rt}(f_{sr}) / T)\)$ Here, \(T > 0\) is a learnable temperature parameter, and the router output bias is initialized to the dataset-level prior \(\log \bar{q}\). Because subject state \(h_s = H_{t_q-1}[s]\) is dynamically generated by the GRU temporal encoder over historical snapshots \(G_{t_q-L}, \dots, G_{t_q-1}\), the router responds to the evolving temporal shape of the subject rather than static entity identity. The final mixture distribution is aggregated in log space: \(p_{final} = \sum_{\tau \in \{sp, re, am\}} \pi_\tau p_\tau\).
Loss & Training¶
To prevent the classic MoE failure mode where one expert dominates while others collapse, QTI-MoE is trained end-to-end under a composite objective balancing three supervision terms: $\(\mathcal{L} = \mathcal{L}_{pred} + \lambda_d \mathcal{L}_{distill} + \lambda_u \mathcal{L}_{uniform}\)$ 1. Task Prediction Loss: \(\mathcal{L}_{pred} = -\log p_{final}(o^*)\) computes the negative log-likelihood of the ground-truth tail entity \(o^*\) on the routed ensemble, driving end-to-end task accuracy; 2. Type Distillation Loss: \(\mathcal{L}_{distill} = \sum_{\tau \in \{sp, re, am\}} \hat{q}_\tau \text{NLL}_\tau(o^*)\) channels gradient mass directly to the expert matching the query's soft label \(\hat{q}\), enforcing sharp functional specialization; 3. Uniform Baseline Loss: \(\mathcal{L}_{uniform} = \frac{1}{3} \sum_{\tau \in \{sp, re, am\}} \text{NLL}_\tau(o^*)\) provides an unweighted regularizing gradient across all three experts, preventing non-dominant experts from losing general competency on off-type queries.
Hyperparameter defaults are set to \(\lambda_d = 0.6\), \(\lambda_u = 0.2\), and label fusion weight \(w = 0.2\). Each structural backbone uses a 2-layer R-GCN with hidden dimension 200, 100 relation bases, and dropout 0.2, followed by ConvTransE. Models are optimized using Adam (\(lr = 5\times 10^{-4}\)) for 30 epochs on eight 48GB NVIDIA GPUs.
Key Experimental Results¶
Experiments are conducted on four standard TKG benchmarks: ICEWS14, ICEWS05-15, ICEWS18, and GDELT. All metrics are reported under the standard time-aware filtered setting using MRR and Hits@{1, 3, 10} (%).
Main Results¶
| Model | ICEWS14 MRR | ICEWS14 H@1 | ICEWS14 H@10 | ICEWS05-15 MRR | ICEWS05-15 H@1 | ICEWS18 MRR | ICEWS18 H@1 | GDELT MRR | GDELT H@1 | GDELT H@10 |
|---|---|---|---|---|---|---|---|---|---|---|
| RE-GCN (2021) | 41.50 | 30.86 | 62.47 | 46.41 | 35.17 | 30.55 | 20.00 | 19.31 | 11.99 | 33.59 |
| HisMatch (2022) | 46.42 | 35.91 | 66.84 | 52.85 | 42.01 | 33.99 | 23.91 | 22.01 | 14.45 | 36.61 |
| LogCL (2024) | 48.90 | 37.80 | 70.26 | 57.04 | 46.07 | 35.67 | 24.53 | 23.75 | 14.64 | 42.33 |
| DiffuTKG (2024) | 48.50 | 36.40 | 72.70 | 52.70 | 40.30 | 36.70 | 25.70 | 25.10 | 16.30 | 42.30 |
| DiMNet (2025) | 45.72 | 34.41 | 67.99 | 58.93 | 48.55 | 34.13 | 23.29 | 21.93 | 14.03 | 37.49 |
| MESH (2025) | 44.36 | — | 64.21 | 48.66 | — | 33.96 | — | — | — | — |
| SRM-LLM (2025) | 50.84 | 39.80 | 71.92 | 57.97 | 46.98 | 37.23 | 25.92 | 25.23 | 15.92 | — |
| QTI-MoE (Ours) | 51.53 | 40.48 | 72.86 | 59.27 | 48.82 | 38.45 | 27.07 | 27.53 | 17.46 | 48.36 |
| Margin vs. Best Baseline | +0.69 | +0.68 | +0.16 | +0.34 | +0.27 | +1.22 | +1.15 | +2.30 | +0.86 | +6.03 |
Ablation Study¶
| Configuration | ICEWS14 MRR | ICEWS14 H@1 | ICEWS05-15 MRR | ICEWS18 MRR | GDELT MRR | GDELT H@10 | Key Insight & Mechanism |
|---|---|---|---|---|---|---|---|
| Full QTI-MoE | 51.53 | 40.48 | 59.27 | 38.45 | 27.53 | 48.36 | Optimal configuration with 3 specialized experts and dynamic router |
| w/o MoE | 50.79 | 39.78 | 58.49 | 37.57 | 24.24 | 43.54 | Monolithic shared backbone drops 3.29% MRR on GDELT |
| w/o sparse expert | 50.09 | 38.89 | 58.14 | 37.03 | 23.74 | 42.65 | Lacks exponential decay; degrades in early/rare event histories |
| w/o recurrent expert | 47.73 | 36.30 | 56.03 | 35.62 | 23.13 | 41.54 | Largest single drop across datasets (-3.80% MRR on ICEWS14) |
| w/o ambiguous expert | 49.43 | 38.22 | 56.95 | 35.84 | 23.28 | 42.07 | Removing candidate log-diversity dampening invites severe distractors |
| w/o \(\mathcal{L}_{distill}\) | 49.24 | 37.95 | 57.81 | 37.76 | 27.15 | 48.26 | Eliminates soft-label guidance; experts fail to specialize |
| w/o \(\mathcal{L}_{uniform}\) | 50.27 | 39.14 | 56.28 | 37.12 | 26.58 | 47.87 | Non-dominant experts drift on off-type queries (-2.99% MRR on ICEWS05-15) |
| w/o LLM refinement | 50.38 | 38.98 | 57.65 | 37.75 | 24.81 | 44.17 | Pure rule labels misclassify boundaries; GDELT H@10 drops 4.19% |
Parameter-Budget Controls¶
To verify that QTI-MoE's gains do not merely stem from having three backbones, two parameter-matched uniform ensembles were evaluated: - Ens-3 (3 backbones averaged uniformly, matching QTI-MoE parameters): Achieves 48.04% MRR on ICEWS14 and 35.84% on ICEWS18. QTI-MoE outperforms Ens-3 by +3.49% and +2.61% MRR at identical parameter count. - Ens-6 (6 backbones averaged uniformly, 2x QTI-MoE parameters): Obtains 47.95% MRR on ICEWS14 and 35.13% on ICEWS18, exhibiting slight degradation compared to Ens-3. - Takeaway: Gains are driven entirely by inductive bias specialization and query-type routing, not parameter scale.
Key Findings¶
- Performance Margins Widen on Sparse, High-Diversity Benchmarks: On dense, long-span ICEWS05-15, QTI-MoE leads by 0.34% MRR; on GDELT (15-minute resolution with extreme candidate diversity and sparse event histories), the margin expands to +2.30% MRR and +6.03% Hits@10. Cross-query routing delivers the largest dividends where query historical shapes vary most widely.
- Dominance in Tail Regimes: Stratified percentile sweeps demonstrate that each expert substantially dominates the other two in its target regime (e.g., Sparse expert leads by ~0.10 Hits@1 in the top-5% most sparse queries on ICEWS18, while other experts collapse near zero). QTI-MoE matches or exceeds the top-performing expert across all percentile thresholds.
- Training & Inference Efficiency: On ICEWS14, QTI-MoE trains at 10.15 min/epoch (faster than SRM-LLM at 13.1 min and RETIA at 93.68 min) and tests in 0.46 min/epoch. Because LLM processing is strictly restricted to one-time offline preprocessing, online latency remains comparable to lightweight GCN forecasters.
Highlights & Insights¶
- Paradigm Shift from Within-Query to Across-Query Routing: Unlike prior MoE models (e.g., MESH) that route between modalities within a single query, QTI-MoE is the first to identify that queries with identical predicates require fundamentally distinct inductive biases based on their temporal history shape.
- Complete Isolation of Privileged Knowledge and LLM Costs: Future target counts and heavy LLM token likelihood calculations are quarantined strictly within offline training set preprocessing, enabling the inference router to operate purely on recurrent entity embeddings \(f_{sr}\) with millisecond latency.
- Logarithmic Diversity Regularization as an Adaptive Temperature: Normalizing raw recurrence count by \(\log(1 + |\mathcal{U}_{sr}|)\) in the ambiguous expert elegantly preserves frequency-based rank ordering while flattening confidence distributions as candidate cardinality explodes, neutralizing near-miss distractor bias.
Limitations & Future Work¶
- Discrete Tri-Partite Taxonomy: Partitioning the continuum of temporal histories into sparse, recurrent, and ambiguous categories may force non-standard historical patterns (e.g., sudden burstiness or multi-period dormancy) into imperfect nearest categories. Expanding toward dynamic expert allocation or continuous inductive bias manifolds is a promising next step.
- Offline LLM Dependency for Boundary Refinement: Constructing soft labels requires GPU access to a 7B-parameter LLM for boundary samples. Although an alternative lightweight MLP recovers the majority of performance (+51.13% vs +51.53% MRR on ICEWS14), cold-start scenarios with unseen relation schemas remain constrained by LLM semantic priors.
- Transductive Evaluation Protocol: The current benchmarks operate under transductive settings where all test entities are seen during training. Adapting regime-specialized memory operators to inductive settings with emerging entities (unseen nodes) remains open for exploration.
Related Work & Insights¶
- vs Structure-Based Forecasters (RE-GCN, HisMatch, LogCL): Traditional GCN/RNN models enforce uniform temporal aggregation windows across all queries, resulting in the inverted-U failure pattern. QTI-MoE adopts LogCL's robust structural backbone but augments it with decoupled memory pathways that eliminate failure modes at both the sparse and ambiguous extremes.
- vs LLM-Based Generative Forecasters (TiRGN-ICL, GenTKG, SRM-LLM): Online LLM generation suffers from extreme computational latency, hallucinations, and inability to enforce topological constraints. QTI-MoE confines the LLM to an offline boundary judge, outperforming SRM-LLM while achieving 30x faster inference.
- vs MESH (Structural-Semantic Multi-Expert): MESH queries "which modality (graph vs text) should I trust for this query?", whereas QTI-MoE asks "what historical evidence regime does this query inhabit, and which memory inductive bias does it require?". The two frameworks operate on orthogonal axes and can be seamlessly combined.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First work to identify the inverted-U performance failure of monolithic TKG models across frequency/diversity spectra and introduce an across-query historical MoE paradigm.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 standard datasets, 12 competitive baselines, full ablations, parameter-controlled ensemble baselines, and percentile robustness scans.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, elegant mathematical formulations, and rigorous alignment between theoretical motivation and empirical validation.
- Value: ⭐⭐⭐⭐⭐ Provides a highly practical, high-throughput, and theoretically grounded architecture for temporal knowledge graph forecasting and complex temporal reasoning.