Skip to content

CodeTD: Topology of Attention Detects Hallucinations in Code LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/VoronkovaDasha/CodeTD
Area: NLP Understanding
Keywords: Code LLMs, Hallucination Detection, Topological Data Analysis (TDA), Attention Maps, Pre-execution Verification

TL;DR

CodeTD models the attention maps of Code LLMs as weighted graphs, leverages persistent homology and Manifold Topology Divergence (MTD) to extract multi-scale 0-dimensional connectivity and 1-dimensional detour cycle features between prompt and generation tokens, enabling highly accurate, execution-free code hallucination detection and best-of-N reranking.

Background & Motivation

Code Large Language Models (such as CodeLlama, StarCoder2, DeepSeek-Coder, and Qwen2.5-Coder) have fundamentally transformed modern software development, yet they remain susceptible to subtle hallucinations. Unlike general NLP tasks where hallucinations often manifest as factually incorrect phrases, errors in LLM-generated code are overwhelmingly functional and logical: syntax errors are remarkably rare in modern Code LLMs (only a single-digit percentage), whereas approximately 78% of failed generations correspond to logical errors where the syntax is flawless but the code fails to transform inputs as specified by the prompt, ultimately triggering unit test assertion errors or introducing latent security and memory vulnerabilities.

Verifying code correctness before deployment in mission-critical and automated systems typically relies on running functional test suites. However, in real-world software engineering, comprehensive functional tests are frequently unavailable, incomplete, expensive to construct, or unsafe to execute due to severe sandbox security risks in environments involving multi-threading, distributed microservices, or privileged system routines. Existing static code analysis tools like Pylint fail to detect high-level semantic misalignments, while LLM-based self-evaluation approaches (such as Self-Eval or CodeJudge) impose prohibitive multi-step inference latency or suffer from unreliable judgment. Furthermore, scalar statistics extracted from attention distributions (such as entropy, max eigenvalues, or average token log-probabilities) capture only localized uncertainty and completely overlook the non-local geometric connectivity between the problem specification and the synthesized program.

The core tension lies in the requirement that functionally correct code generation demands continuous, structural grounding in the prompt's constraint manifold; when an LLM hallucinates, its attention flow structurally detaches from the prompt tokens. A simple thresholding of attention matrices introduces discontinuous topological jumps and relies on sensitive, non-transferable hyperparameters. This paper resolves this dilemma by modeling multi-head causal attention as a weighted bipartite-like graph and leveraging persistent homology tools from Topological Data Analysis (TDA). Core idea: convert causal attention maps into dissimilarity-weighted graphs between prompt and generation tokens, extract asymmetric Manifold Topology Divergence (MTD) features across 0-dimensional connected components and 1-dimensional detour cycles, and train an execution-free gradient-boosted classifier for pre-execution code verification and candidate reranking.

Method

Overall Architecture

CodeTD aims to determine the functional correctness of generated code strictly prior to execution using only the internal attention matrices produced during a single forward pass of the Code LLM. The end-to-end framework operates in five coordinated stages: first, the problem prompt \(P\) and the generated code \(G\) are concatenated and passed through the Code LLM, capturing raw multi-head self-attention maps across all layers and heads; second, each attention map is mapped to a fully connected undirected weighted graph where attention affinities are transformed into angular-like dissimilarity edge weights; third, using the prompt subgraph as a base, a filtration process is executed to compute 0-dimensional and 1-dimensional Cross-Barcodes tracking the emergence and collapse of topological structures; fourth, the birth-death intervals are integrated to compute scale-invariant, normalized MTD features augmented with diagonal self-attention statistics; finally, the joint feature vector is classified by a lightweight XGBoost model to produce the probability of code correctness for hallucination filtering or best-of-N candidate selection.

The overall operational pipeline is illustrated in the diagram below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Prompt P + Generated Code G"] --> B["Attention-to-Graph Dissimilarity Filtration<br/>Invert attention weights into dissimilarity metric"]
    B --> C["Asymmetric Multi-Scale Topological Divergence<br/>Track H0 connected components and H1 detour cycles"]
    C --> D["Structured Sparse Classifier & Mitigation Decision<br/>XGBoost auto-selects top 5% critical attention heads"]
    D --> E["Pre-Execution Hallucination Detection & Best-of-N Reranking"]

Key Designs

1. Attention-to-Graph Dissimilarity Filtration: Overcoming the instability of discrete thresholding on raw attention maps

Traditional attention analysis relies on heuristic thresholding to binarize attention weights into sparse adjacency matrices, or computes global spectral summaries like matrix determinants and eigenvalues. However, fixed thresholding requires manual calibration and causes abrupt, discontinuous topological changes under minor weight perturbations, failing to adapt across diverse prompt lengths and generation contexts.

To resolve this limitation, CodeTD interprets each causal self-attention matrix \(A \in [0, 1]^{n \times n}\) across every layer and head as an undirected fully connected graph \(G = (V, E, W)\) over \(n = |P| + |G|\) tokens. To turn high attention affinities into close geometric proximities, edge weights are defined via a dissimilarity transformation: $\(w_{i,j} = 1 - a_{i,j} \quad (i > j)\)$ Under the Manifold Topology Divergence (MTD) framework, the filtration begins with an initial graph \(G_0\) containing all vertices and all internal edges between prompt tokens \(P\). The remaining edges are then sequentially added in ascending order of their dissimilarity weight \(w_{i,j}\) (i.e., descending order of attention strength \(a_{i,j}\)). This ordering guarantees that tokens exhibiting the strongest mutual attention merge earliest in the filtration. By the persistent homology stability theorem, Cross-Barcodes satisfy Lipschitz continuity under infinity-norm weight perturbations (\(d_B(B(W), B(W')) \le \|W - W'\|_\infty\)), providing CodeTD with robust mathematical invariance against attention noise.

2. Asymmetric Multi-Scale Topological Divergence: Preventing false positives from internally coherent but hallucinated code

In code generation, the relationship between specification (prompt) and implementation (generation) is inherently asymmetric: the generated code must strictly ground itself in the prompt, whereas the prompt is independently self-contained. Relying solely on symmetric graph metrics or 0-dimensional connected components can be deceived by internally consistent code that completely diverges from the prompt: the generated tokens may attend strongly to each other (forming tight local clusters) while being completely detached from the problem constraints.

CodeTD introduces a dual persistent homology formulation capturing both 0-dimensional components (\(H_0\)) and 1-dimensional cyclic voids (\(H_1\)). For any homological dimension \(k\), the filtration records the birth times \(b_i\) and death times \(d_i\) of topological features into a multi-set called \(\text{CrossBarcode}_k\), computing the integrated topological divergence: $\(\text{MTD}_k = \sum_{(b_i, d_i) \in \text{CrossBarcode}_k} (d_i - b_i)\)$ At the \(H_0\) level, \(\text{MTD}_0(P, G)\) corresponds to the integral of the number of disconnected generation-side components waiting to merge into the prompt subgraph across filtration thresholds, measuring the total multiscale grounding cost. At the \(H_1\) level, because the prompt clique complex is contractible (\(\dim H_1(K(G_0)) = 0\)), all non-trivial 1-cycles represent "detour loops" (\(p_1 \to g_1 \to \dots \to g_k \to p_2\)) traversing the generated code before reconnecting to the prompt. In correct code that tightly adheres to prompt semantics, these detour loops collapse almost immediately, yielding a very small \(\text{MTD}_1\). In hallucinated code that drifts semantically, detour loops persist over broad weight intervals, producing large \(\text{MTD}_1\) values. To eliminate token length bias, topological metrics are normalized by vertex cardinalities and paired with diagonal self-attention averages: $\(\mathbf{f}_{\text{topo}} = \left[ \frac{\text{MTD}_0(P, G)}{|G|}, \frac{\text{MTD}_0(G, P)}{|P|}, \frac{\text{MTD}_1(P, G)}{|G|}, \frac{\text{MTD}_1(G, P)}{|P|}, \frac{1}{|P|}\sum_{i \in P} a_{i,i}, \frac{1}{|G|}\sum_{i \in G} a_{i,i} \right]\)$ This compact 6-dimensional descriptor per attention head captures both boundary grounding and cyclic semantic consistency.

3. Structured Sparse Classifier & Mitigation Decision: Eliminating full-attention storage overhead and enabling zero-overhead deployment

Storing dense attention matrices across all layers and heads for large LLMs requires prohibitive disk and GPU memory footprints (up to 1,718 GB on MBPP benchmarks), rendering offline attention caching impractical. Furthermore, heuristic hand-crafted rules that pre-select a tiny set of isolated heads fail to model cooperative multi-layer interactions.

CodeTD computes all topological scalar features on-the-fly during generation and immediately discards the raw attention tensors, shrinking memory consumption to a negligible fraction (approx. 190 MB on HumanEval). A global XGBoost classifier is trained across all extracted head features. Empirical feature importance analyses reveal that the trained tree model exhibits natural structured sparsity: pruning down to the top 5% of attention features results in virtually zero performance degradation. For downstream application, the predicted probabilities of correctness serve as a pre-execution safety filter and provide an effective scoring function for Best-of-N sampling, boosting top-candidate accuracy without requiring sandboxed code execution.

Key Experimental Results

Main Results

The primary evaluation spans HumanEval (HE), MBPP, and the library-rich benchmark BigCodeBench (BCB) across 10 Code LLMs ranging from 1.5B to 34B parameters under sampling temperature \(T=0.8\). Performance is reported using 5-fold stratified group cross-validation (grouping distinct generations of the same prompt into the same fold to ensure zero problem overlap between train and test sets). The table below details ROC-AUC scores across models and baselines:

Model Method HumanEval (HE) MBPP BigCodeBench (BCB)
DeepSeek-Coder-6.7B Prompt Length 58.0 ± 6.7 49.5 ± 4.1 54.9 ± 2.3
Generation Length 58.5 ± 3.7 55.4 ± 1.7 54.7 ± 2.7
Mean Log. Prob. 68.9 ± 3.0 60.7 ± 2.2 50.1 ± 2.9
Pylint (Static Analysis) 54.7 ± 1.1 52.8 ± 0.9 61.2 ± 1.1
CodeT5-base ft. 63.1 ± 4.7 52.8 ± 3.2 54.5 ± 2.8
Self-Eval 57.3 ± 4.5 51.0 ± 0.8 52.9 ± 1.6
AttnLogDet 78.8 ± 3.6 77.3 ± 5.1 67.5 ± 4.8
AttnEigvals 75.1 ± 3.2 75.9 ± 3.5 67.6 ± 1.9
LapEigvals 81.7 ± 1.9 77.6 ± 4.5 64.6 ± 0.5
CodeTD (Ours) 86.4 ± 1.8 81.2 ± 3.7 69.2 ± 2.8
StarCoder2-7B Prompt Length 56.3 ± 5.9 55.2 ± 4.0 51.3 ± 6.5
Generation Length 58.9 ± 2.4 59.5 ± 1.2 61.7 ± 3.1
Mean Log. Prob. 69.4 ± 1.7 61.9 ± 3.6 79.4 ± 3.3
LapEigvals 80.6 ± 2.9 79.3 ± 2.8 85.3 ± 2.4
CodeTD (Ours) 82.5 ± 2.1 81.4 ± 3.6 83.7 ± 4.3
CodeLlama-7B Prompt Length 61.6 ± 4.4 59.1 ± 4.2 51.3 ± 4.3
Generation Length 60.1 ± 5.3 60.8 ± 2.5 58.7 ± 2.4
Mean Log. Prob. 64.1 ± 2.0 61.0 ± 3.7 57.1 ± 3.5
LapEigvals 79.7 ± 2.7 73.1 ± 3.6 68.8 ± 2.9
CodeTD (Ours) 85.6 ± 3.9 83.4 ± 3.3 75.0 ± 3.4
Qwen2.5-Coder-7B Mean Log. Prob. 61.4 ± 3.6 61.6 ± 0.7 51.5 ± 3.0
LapEigvals 78.4 ± 2.6 76.8 ± 4.3 66.4 ± 0.9
CodeTD (Ours) 81.6 ± 4.7 80.9 ± 3.5 70.0 ± 1.4
Magicoder-S-DS-6.7B Mean Log. Prob. 68.8 ± 4.1 60.0 ± 2.4 54.7 ± 5.0
LapEigvals 76.7 ± 3.9 79.5 ± 1.4 66.8 ± 1.0
CodeTD (Ours) 79.3 ± 3.5 80.7 ± 2.5 68.4 ± 1.5

Ablation Study

To verify the contribution of each individual topological descriptor, an ablation study evaluated incremental feature additions using the XGBoost classifier, progressing from basic diagonal self-attention to 0-dimensional and combined 0/1-dimensional MTD features:

Model Feature Configuration HumanEval (HE) MBPP Ablation Takeaway
DeepSeek-Coder-6.7B Diag. Feat. Only 84.2 ± 1.5 81.0 ± 3.5 Baseline vertex-level self-attention
+ MTD 0-dim 85.7 ± 3.3 81.7 ± 2.7 +1.5% HE from prompt-generation connectivity
+ MTD 0,1-dim (Full) 86.4 ± 1.8 81.2 ± 3.7 Optimal performance with detour cycle tracking
StarCoder2-7B Diag. Feat. Only 83.4 ± 2.0 80.1 ± 3.3 Baseline self-attention features
+ MTD 0-dim 83.4 ± 2.9 80.4 ± 3.3 Stable component connectivity baseline
+ MTD 0,1-dim (Full) 82.5 ± 2.1 81.4 ± 3.6 Reaches best ROC-AUC on MBPP (81.4)
CodeLlama-7B Diag. Feat. Only 83.2 ± 4.3 78.8 ± 2.2 Baseline self-attention
+ MTD 0-dim 84.7 ± 3.8 81.7 ± 1.1 Substantial gain (+2.9% MBPP)
+ MTD 0,1-dim (Full) 85.0 ± 3.9 82.5 ± 1.0 +3.7% overall MBPP gain over diagonal features
Qwen2.5-Coder-7B Diag. Feat. Only 80.0 ± 2.8 76.5 ± 4.1 Baseline self-attention
+ MTD 0-dim 80.6 ± 4.0 79.4 ± 3.3 +2.9% gain on MBPP
+ MTD 0,1-dim (Full) 81.6 ± 4.7 80.9 ± 3.5 Highest ROC-AUC across both benchmarks
Magicoder-S-DS-6.7B Diag. Feat. Only 79.4 ± 2.9 80.4 ± 1.0 Baseline self-attention
+ MTD 0-dim 80.5 ± 2.9 79.5 ± 2.7 Noticeable gain on HumanEval (+1.1%)
+ MTD 0,1-dim (Full) 79.3 ± 3.5 80.7 ± 2.5 Sustained top-tier accuracy on MBPP

Key Findings

  • Substantial pass@1 Gains via Best-of-N Selection: Using CodeTD probabilities to rerank generated candidates significantly outperforms random selection across all models. On HumanEval, pass@1 improves from 28.6% to 43.3% (+14.7 pp) on StarCoder2-7B, from 39.1% to 56.7% (+17.6 pp) on DeepSeek-Coder-6.7B, and from 26.0% to 39.7% (+13.7 pp) on CodeLlama-7B.
  • Extreme Structured Sparsity: XGBoost feature importance distributions show that only ~25% of features carry non-zero importance. Retaining merely the top 5% of features across layers and heads maintains complete classification performance without noticeable metric degradation, confirming that hallucination indicators are concentrated in specialized attention heads.
  • Robust Multilingual & Cross-Benchmark Transferability: On the MultiPL-E benchmark (Java, Go, Rust, Lua), CodeTD outperforms the strongest attention baseline (LapEigvals) in roughly 84% of settings. In cross-benchmark transfer (training on HumanEval and testing on MBPP, or vice-versa), CodeTD achieves 67.7%–80.3% ROC-AUC, whereas spectral baseline LapEigvals experiences sharp drops of 13 to 20 ROC-AUC points.
  • Fine-Grained Error Classification: Training a multi-class XGBoost model on Python runtime exception types shows that CodeTD accurately identifies specific error classes: IndexError at 97.8%, RecursionError at 100%, and SyntaxError at 93.3%, while effectively separating pure logical errors (AssertionError) from runtime failures.

Highlights & Insights

  • Topological Bridge for Attention Interpretability: First work to introduce Manifold Topology Divergence (MTD) and 1-dimensional persistent homology to Code LLMs, solving the inability of scalar attention metrics to capture non-local structural cohesion.
  • Execution-Free Safety & Zero Latency: Eliminates the need for sandboxed code execution while avoiding costly secondary LLM evaluation rounds, providing a safe and lightweight pre-execution filter for production deployment.
  • Detour Cycles as Semantic Drift Signatures: Demonstrates a clean mathematical intuition where non-trivial 1-cycles in persistent homology correspond directly to detour loops in code generation that wander away from prompt specifications.

Limitations & Future Work

  • Lack of Token-Level Error Localization: CodeTD determines the overall functional correctness of a code snippet but currently cannot pinpoint the exact line or token where a bug originates.
  • Blindness to Latent Pretraining Biases: The method detects geometric manifestations of attention detachment, but cannot trace root causes stemming from spurious pretraining dataset correlations, RLHF policy bias, or temperature-induced sampling drift.
  • Streaming Verification in Agentic Loops: Future extensions can exploit incremental union-find algorithms to compute \(H_0\) persistence dynamically during token generation, halting and rolling back decoding the moment hallucination thresholds are breached.
  • vs TOHA (Bazarova et al., 2026): While TOHA investigates RAG hallucination in natural language QA using heuristic per-head selection and symmetric \(H_0\) divergence, CodeTD targets functional code correctness using asymmetric \((P, G)\) and \((G, P)\) metrics across both \(H_0\) and \(H_1\) (detour cycles), coupled with joint GBDT feature learning.
  • vs Spectral Attention Baselines (LapEigvals / AttnEigvals): Laplacian eigenvalues suffer from instability under near-degenerate matrix perturbations; in contrast, persistent homology guarantees Lipschitz stability, leading to superior cross-benchmark generalization.
  • vs LLM Self-Evaluation (Self-Eval / CodeJudge): Self-evaluation incurs heavy inference overhead and risks compounding hallucinations; CodeTD reuses existing attention maps from the primary generation pass with negligible computational footprint.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering application of TDA and 1D detour persistent homology to Code LLM hallucination detection.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 10 models, 4 benchmarks, 5 programming languages, with extensive transferability and ablation studies.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from topological motivation to mathematical definitions and practical deployment pathways.
  • Value: ⭐⭐⭐⭐⭐ Practical, lightweight, execution-free solution for pre-execution code auditing and candidate reranking in production systems.