Enhancing LLMs in Predictive Political QA with Semi-Structured Data¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zhouzihan-liu/PSL
Area: Social Computing
Keywords: predictive political QA, semi-structured data, political stance inference, graph convolutional networks, collaborative filtering
TL;DR¶
Addressing the fundamental gap where predictive political question answering cannot be resolved via direct factual lookup, PSL introduces a dual-view framework that extracts fine-grained issue stances from semi-structured profiles in a semantic view, aggregates high-order interaction graph representations in a vector view, and aligns them via collaborative filtering to significantly enhance LLM decision forecasting.
Background & Motivation¶
Political question answering is undergoing a critical transition from conventional retrospective factual inquiry toward predictive political question answering. In real-world governance analysis, policy forecasting, and legislative intelligence, analysts and citizens are rarely interested in static historical trivia; rather, they require models to forecast how a specific political actor (such as a legislative representative or diplomatic official) will vote on an upcoming bill or respond to an evolving international incident. While large language models (LLMs) possess vast internalized commonsense and procedural reasoning abilities, their parametric knowledge remains brittle and incomplete when dealing with specialized, long-tail political actors and dynamic agendas. Existing augmentation techniques generally pursue two main paradigms: actor-profile-based simulation (e.g., PAA), which concatenates historical biographical backgrounds and roll-call records into natural language personas for role-playing, and knowledge graph (KG) evidence injection (e.g., PEG, KAPING), which retrieves entity-relation triples from curated political networks.
However, both paradigms treat external resources merely as static, knowledge-based factual evidence. In predictive political QA, the target behavior has not yet occurred, and external records serve solely as indirect historical clues rather than explicit answers. Consequently, direct factual retrieval fails to supply the decisive predictive signal. Profile-based role-playing flattens extensive behavioral histories into text, making it extremely difficult to capture implicit, group-level collaborative influence among legislators who operate across committees and party alliances. Conversely, KG-based methods rigidly compress rich legislative bills, policy statements, and diplomatic negotiations into discrete relational triples \((h, r, t)\), inevitably stripping away nuanced textual context such as specific amendments, statutory exceptions, and policy scopes that dictate an actor's decision.
To overcome the dual failure modes of text redundancy and graph over-compression, this paper argues that semi-structured political data (such as legislative records from LegiScan and diplomatic events from ICEWS) uniquely preserve fine-grained textual context while preserving explicit behavioral links between actors. Within these records lie two indispensable and complementary predictive signals: issue-specific actor stances reflecting subjective ideological orientations, and high-order structure signals capturing indirect dependencies and group influence across political interaction networks. Core idea: develop PSL, a dual-view framework that extracts semantic issue stances through ladder retrieval and task-specific model distillation, aggregates high-order structural influences across an actor interaction graph in a vector view, and synergizes both signals via collaborative filtering and LoRA fine-tuning for grounded predictive reasoning.
Method¶
Overall Architecture¶
The PSL (Predictive Stance and Structure Learning) framework operates over semi-structured political behavior profiles to answer predictive questions \(q\) regarding the future decisions of a political actor. The overall pipeline decouples evidence acquisition into a semantic view and a vector view to jointly empower downstream LLM reasoning. In the semantic view, PSL parses the question into a target actor and a question focus, executes ladder retrieval to hierarchically narrow down relevant records across multi-attribute profiles, and distills a small language model (SLM) to infer concise, issue-aligned stance narratives. In the vector view, PSL constructs an actor interaction bipartite graph based on shared voting and diplomatic events, performs multi-layer graph convolutions to capture higher-order group influences, and maps actor representations into a collaborative space aligned with the question focus. Finally, a hybrid instruction prompt containing both explicit stance text and collaborative embeddings is fed into a LoRA-adapted LLM to generate the final prediction.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Predictive Question Input<br/>Decompose Actor & Focus"] --> B["Semi-Structured Profiles & Ladder Retrieval<br/>Hierarchical Multi-Attribute Candidate Pruning"]
B --> C["Task-Distilled Semantic Stance Inference<br/>SLM Extracts Concise Issue Preferences"]
A --> D["Actor Interaction Graph Structural Aggregation<br/>GCN Propagates Multi-Hop Group Influence"]
D --> E["Collaborative Filtering & LoRA Adaptation<br/>MLP Space Alignment & Instruction Tuning"]
C --> F["Dual-View Hybrid Prompt Reasoning<br/>LoRA-adapted LLM Predicts Target Outcome"]
E --> F
Key Designs¶
1. Semi-Structured Profiles and Ladder Retrieval: High-dimensional records overwhelm flat dense retrieval -> Iterative multi-attribute filtering -> Eliminates noise while retaining fine-grained legislative context
Standard dense passage retrieval models (e.g., DPR, ColBERT) struggle with political records because each entry contains heterogeneous attributes, including bill titles, full legislative summaries, event dates, and recorded roll-call choices, where naive concatenation introduces substantial lexical noise. To tackle this, PSL structures political history into human-centered JSON profiles \(\text{file}(P)\). For each actor \(p\), profile \(\text{file}(p)\) comprises individual records \(r_p\), with key-value attributes \(a_{r_p}\) (using BART-summarized bill descriptions to curb token overhead). Given question \(q\), an LLM first extracts the target actor \(p^*\) and the core semantic question focus \(q^*\) (e.g., the specific bill text under vote). Instead of performing a single-shot vector search, PSL introduces Ladder Retrieval (LR), an iterative filtering process across \(n\) rounds. Starting from all records \(R^{(0)} = \text{file}(p_s)\), round \(i\) selects the top-\(k_i\) candidates from \(R^{(i-1)}\) based on semantic similarity between \(q^*\) and the record's \(i\)-th attribute:
$\(R^{(i)} = \text{LR}(R^{(i-1)}, q^*, k_i)\)$
After \(n\) iterations (empirically configured with \(n=3, k_n=5\)), the model yields an ultra-compact candidate set \(R^{(n)}\), accurately isolating the most contextually relevant historical records while filtering extraneous procedural noise.
2. Task-Distilled Semantic Stance Inference: Raw historical facts lack explicit ideological direction and large teacher models incur heavy latency -> Distill teacher reasoning into a compact student model -> Produces rapid, consistent issue-aligned stance evidence
Raw historical votes indicate past facts but do not explicitly articulate an actor's underlying values or ideological trajectory on emergent issues. Furthermore, invoking proprietary foundation models (like GPT-4o-mini) to generate intermediate reasoning chains for every query imposes prohibitive latency and computational costs. PSL resolves this dilemma by distilling specialized stance extraction capabilities from a large teacher LLM into an efficient small language model (Flan-T5-Small). By transforming historical records in profiles into synthetic questions, the teacher is prompted to summarize whether the actor demonstrates consistent support, regulatory skepticism, or partisan allegiance toward the given focus topic. The student SLM is trained via token-level cross-entropy loss against the teacher's softened output distribution. During inference, the frozen SLM instantaneously translates \(R^{(n)}\) and \(q\) into structured natural-language stance descriptions \(S\):
$\(R^{(n)}, q \xrightarrow{\text{SLM}} S\)$
This design captures longitudinal behavioral consistency in human-readable prose while ensuring millisecond-level execution speed.
3. Actor Interaction Graph Structural Aggregation: Isolated profiles fail to reflect network-wide political coalition dynamics -> Formulate a signed bipartite interaction graph and perform symmetric normalized convolutions -> Embeds multi-hop collaborative dependencies into actor vectors
Individual legislative decisions are rarely made in isolation; they are heavily influenced by party coalitions, caucuses, and informal diplomatic alignments. PSL captures these collective dynamics by converting shared profile records into a global actor interaction graph \(G_I = \{(p, e_{pr}, r)\} \subseteq P \times E \times R\). Edges link political actors \(P\) to specific records \(R\). The edge weights \(e_{pr}\) are signed: in roll-call voting, \(e_{pr} = +1\) if actor \(p\) voted "Yea" and \(-1\) if "Nay"; in diplomatic interactions, \(e_{pr} = +1\) indicates documented contact, while absent ties remain unlinked. To propagate group influence without introducing heavy non-linear over-smoothing, PSL employs a parameter-free symmetric normalized graph convolution aggregator:
$\(e_p^{(i+1)} = \sum_{r \in \mathcal{N}_p} \frac{e_{pr}}{\sqrt{|\mathcal{N}_p||\mathcal{N}_r|}} e_r^{(i)}, \quad e_r^{(i+1)} = \sum_{p \in \mathcal{N}_r} \frac{e_{pr}}{\sqrt{|\mathcal{N}_p||\mathcal{N}_r|}} e_p^{(i)}\)$
Initial record embeddings \(e_r\) are derived from a frozen Sentence-BERT model, and initial actor representations \(e_p^{(0)}\) aggregate their immediate 1-hop records. Following \(l=2\) propagation layers, PSL concatenates the initial and final representations via a Jumping Knowledge scheme, yielding \(\hat{e}_p = e_p^{(0)} \parallel e_p^{(l)}\), which balances localized personal history with broad structural network context.
4. Collaborative Filtering and LoRA Adaptation: Continuous graph embeddings cannot be decoded by frozen autoregressive attention -> Shared multi-layer perceptron projection optimized by BPR loss and lightweight parameter tuning -> Enables LLMs to digest collaborative vectors as implicit structural filters
Continuous topological embeddings lie outside an LLM's natural language token space, and unconditioned graph representations contain broad structural signals irrelevant to the specific bill at hand. To resolve this impedance mismatch, PSL adapts collaborative filtering principles: the political actor is conceptualized as a "user" and the question focus as an "item." After dimension-aligning the question focus vector as \(\hat{e}_{q^*} = e_{q^*} \parallel e_{q^*}\), both \(\hat{e}_{p_s}\) and \(\hat{e}_{q^*}\) are projected through a shared multi-layer perceptron (MLP with layers \(1536 \to 1024 \to 512 \to 256 \to 64 \to 32\)) into collaborative embeddings:
$\(\hat{e}_{p_s}^*, \hat{e}_{q^*}^* = \text{MLP}(\hat{e}_{p_s}, \hat{e}_{q^*})\)$
The MLP is trained using Bayesian Personalized Ranking (BPR) loss over 500,000 historical actor-record interactions, maximizing the inner-product score margin between observed positive voting records \(r^+ \in \mathcal{N}_p^+\) and negative/unobserved records \(r^- \in \mathcal{N}_p^-\):
$\(\mathcal{L} = -\sum_{p \in P} \sum_{r^+ \in \mathcal{N}_p^+} \sum_{r^- \in \mathcal{N}_p^-} \ln \sigma(y_{pr^+} - y_{pr^-}) + \lambda \|\Theta\|_2^2\)$
Subsequently, 800 paired collaborative vectors are packaged into natural-language instruction templates to adapt the target LLM (e.g., Llama-3.1-8B) via LoRA fine-tuning. This teaches the LLM to interpret the collaborative vectors as an implicit attention filter that highlights structurally pertinent allies while suppressing unrelated network noise.
Retrieval Augmentation and Prediction Policy¶
During downstream inference, PSL unifies explicit semantic evidence and implicit topological signals within a comprehensive hybrid instruction prompt. Given an incoming predictive query \(q\): 1. Semantic Channel Execution: The query \(q\) is parsed into actor \(p^*\) and focus \(q^*\); Ladder Retrieval fetches top-\(k\) records \(R^{(n)}\), which Flan-T5-Small converts into an ideological stance summary \(S\). 2. Vector Channel Execution: The actor's multi-hop representation \(\hat{e}_{p_s}\) and the focus embedding \(\hat{e}_{q^*}\) are transformed by the trained MLP into collaborative vectors \(\hat{e}_{p_s}^*\) and \(\hat{e}_{q^*}^*\). 3. Hybrid Generation: The prompt \(T(q, S, \hat{e}_{p_s}^*, \hat{e}_{q^*}^*)\) formats the background context, candidate options, explicit stance rationale, and collaborative tokens. The LoRA-adapted model \(\text{LLM}_{\Phi + \Phi'}\) generates the final prediction: $\(\text{Answer} = \text{LLM}_{\Phi + \Phi'}(T(q, S, \hat{e}_{p_s}^*, \hat{e}_{q^*}^*))\)$ Post-processing regular expressions cleanly extract the selected discrete option (e.g., "Yea" vs. "Nay"), delivering a robust and verifiable prediction.
Key Experimental Results¶
Main Results¶
PSL was rigorously benchmarked across three diverse political datasets: RCVP (roll-call vote prediction in the U.S. Congress, evaluated via Macro F1), ICEWS (international crisis event classification with four target categories, evaluated via Accuracy), and StaId (U.S. state-level ideological policy stance binary classification, evaluated via Macro F1). The table below details comparative performance across three mainstream open-source foundation models:
| Method | RCVP (Llama-8B) | RCVP (Mistral-7B) | RCVP (Deepseek-7B) | ICEWS (Llama-8B) | ICEWS (Mistral-7B) | ICEWS (Deepseek-7B) | StaId (Llama-8B) | StaId (Mistral-7B) | StaId (Deepseek-7B) |
|---|---|---|---|---|---|---|---|---|---|
| Vanilla | 27.36 | 27.33 | 28.03 | 18.42 | 19.18 | 19.67 | 35.00 | 33.36 | 29.59 |
| GKP (ACL'22) | 23.07 | 26.29 | 24.28 | 7.09 | 14.37 | 9.65 | 38.26 | 33.04 | 34.76 |
| RECITE (ICLR'23) | 25.08 | 23.38 | 14.90 | 14.89 | 16.95 | 11.81 | 31.88 | 29.47 | 27.49 |
| LangChain | 31.10 | 35.52 | 24.37 | 8.06 | 18.06 | 19.06 | 27.71 | 29.51 | 30.07 |
| InstructRAG (ICLR'25) | 29.84 | 21.64 | 26.81 | 11.57 | 7.71 | 18.21 | 31.53 | 28.30 | 32.97 |
| KAPING (ACL'23) | 23.15 | 23.96 | 23.59 | 20.52 | 22.52 | 15.36 | 36.20 | 30.28 | 34.22 |
| MindMap (ACL'24) | 18.66 | 23.02 | 21.54 | 12.70 | 22.01 | 21.70 | 34.72 | 31.97 | 31.51 |
| PAA (AAAI'25) | 44.35 | 41.78 | 42.35 | 31.35 | 18.99 | 24.53 | 36.34 | 27.41 | 39.89 |
| PSL (Ours) | 55.92 | 50.30 | 43.59 | 57.81 | 54.69 | 53.42 | 48.50 | 48.83 | 50.58 |
Under a standardized GPT-3.5-Turbo backbone, PSL similarly outperforms all competing systems across the board: achieving 50.23 on RCVP (vs. 41.21 for the strongest baseline PEGexp_GTR, +9.02 gain), 45.89 on ICEWS (vs. 28.60 for PEGexp_GTR, +17.29 gain), and 49.90 on StaId (vs. 46.21 for PEGexp_GTR, +3.69 gain).
Ablation Study¶
To isolate the contribution of semantic stance reasoning versus high-order vector structure learning, systematic ablations were conducted using Llama-3.1-8B-Instruct (detailed below):
| Configuration Variant | RCVP (Macro F1) | ICEWS (Accuracy) | StaId (Macro F1) | Description & Key Impact |
|---|---|---|---|---|
| Full PSL Framework | 55.92 | 57.81 | 48.50 | Dual-view synergy with distilled stance and collaborative vectors |
| w/o Actor Stance Acquisition | 46.15 | 39.07 | 46.17 | Removes stance text; ICEWS drops severely by -18.74, RCVP drops -9.77 |
| w/o High-Order Structure Signals | 39.35 | 49.02 | 46.35 | Removes collaborative embeddings; RCVP collapses by -16.57, ICEWS drops -8.79 |
| PSL_TF-IDF (w/o Stance Inference) | 52.07 | 56.70 | 46.45 | Raw TF-IDF lexical retrieval without distilled stance inference |
| PSL_BM25 (w/o Stance Inference) | 51.87 | 57.22 | 45.94 | Standard BM25 lexical ranking without multi-attribute structure awareness |
| PSL_Embedding (w/o Stance Inference) | 52.96 | 57.16 | 46.69 | Standard single-stage dense sentence embedding retrieval |
| PSL_LR (w/o Stance Inference) | 53.81 | 57.34 | 47.10 | Ladder Retrieval injecting raw records; tops all retrieval baselines |
Key Findings¶
- High complementarity between stance and structure: On legislative voting (RCVP), removing structural signals produces the most catastrophic performance collapse (-16.57 drop in Macro F1), underscoring that legislative roll calls are strongly bound by network-level caucus allegiances. Conversely, on international crises (ICEWS), dropping semantic stance inference induces the largest drop (-18.74 drop in Accuracy), proving that foreign policy responses rely primarily on consistent ideological positions.
- Ladder Retrieval outperforms flat retrieval: Across all three benchmarks, Ladder Retrieval (PSL_LR w/o SI) consistently outperforms BM25, TF-IDF, and dense sentence embeddings, proving that multi-stage attribute-guided narrowing effectively eliminates irrelevant record attributes.
- Graph depth exhibits an optimal threshold: Empirical evaluation across propagation depths shows that accuracy peaks at 2 layers for RCVP/ICEWS and 3 layers for StaId. Deeper message passing causes graph over-smoothing, erasing fine-grained behavioral distinctions between legislators of differing political factions.
- Expressing network structure in text induces severe degradation: When neighbor interaction histories are verbalized into natural-language prompt context instead of vector embeddings, accuracy declines steeply across all datasets (dropping to 37–38 on RCVP, compared to 55.92 with collaborative vectors). This confirms that long textual descriptions of network ties create severe context clutter and reasoning interference.
Highlights & Insights¶
- From static factual lookup to inference-oriented evidence: The paper establishes a vital conceptual shift for predictive QA: when the future answer does not exist in any external document, retrieval-augmented systems must synthesize subjective stances and structural network priors rather than injecting raw historical facts.
- Collaborative filtering bridges graph representations to LLM attention: Treating political actors as users and bills as items under a BPR ranking objective provides a mathematically principled mechanism to project continuous graph embeddings into a space directly interpretable by LoRA-tuned LLMs as implicit structural filters.
- Exceptional operational efficiency via student distillation: Distilling stance extraction into Flan-T5-Small achieves high predictive performance while requiring only a 2.48 relative runtime factor over vanilla GPT-3.5 inference, substantially outperforming path-reasoning baselines such as MindMap (13.9 relative runtime).
Limitations & Future Work¶
- Geographic and institutional concentration: The empirical validation is primarily anchored in the U.S. presidential-congressional framework and specific bilateral diplomatic datasets; evaluating the framework across parliamentary coalitions or multi-party proportional systems remains open.
- Static fusion weights across heterogeneous queries: The dual-view fusion currently relies on a static hybrid prompt template, lacking an adaptive gating mechanism to dynamically shift weight between ideological stance and peer pressure depending on bill controversy.
- Future directions: Integrating reinforcement learning with human feedback (RLHF) or routing gates to adjust stance-vs-structure weighting per instance, and expanding semi-structured behavioral profiles to include committee hearings, campaign PAC contributions, and public social media discourse.
Related Work & Insights¶
- vs PAA (AAAI 2025): PAA prompts LLMs to simulate roll calls purely through text-based persona profiles. While effective on RCVP (44.35), it fails to capture higher-order coalition constraints. PSL achieves 55.92 on RCVP by explicitly encoding network topology via graph convolutions and collaborative filtering.
- vs PEG / KAPING (WWW 2024 / ACL 2023): Knowledge-graph methods discard legislative prose to form discrete \((h, r, t)\) triples. PSL's semi-structured foundation preserves detailed bill summaries and issue-specific nuances, nearly doubling prediction accuracy on the ICEWS benchmark (57.81 vs. 28.60 for PEGexp_GTR).
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates predictive political QA with a novel dual-view paradigm; the collaborative filtering mapping for structure-aware LLM reasoning is highly original.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across three real-world benchmarks, four LLM backbones, complete ablations, and efficiency analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative structure, well-defined mathematical notation, and compelling motivating arguments.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational standard for applying LLMs to computational political science, policy forecasting, and structured graph-augmented reasoning.