Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention

Conference: NeurIPS 2025 arXiv: 2501.06382 Code: None Area: LLM Efficiency Keywords: spontaneous thought, topic change, self-attention theory, token priority graph, cognitive gap

TL;DR

This paper investigates, both theoretically and empirically, the dynamics of spontaneous topic changes in self-attention models. For a single-layer self-attention model, it establishes three results: (1) training on mixed topics preserves the token priority ordering of the original topic; (2) topic changes occur only when the number of low-priority tokens exceeds that of high-priority tokens; and (3) longer inputs and more ambiguous topics do not increase the probability of topic change — contrary to human cognition.

Background & Motivation

Background: Spontaneous thought is a well-studied phenomenon in human cognition, referring to abrupt, unstructured transitions to a different topic during conversation or reasoning. LLMs, by contrast, predict the next token based on statistical patterns in context and lack genuine spontaneity.

Limitations of Prior Work: Although the self-attention mechanism has been extensively analyzed theoretically — including its connections to SVMs and token priority via TPGs — the phenomenon of topic change, which is directly relevant to human cognition, has not been formally studied.

Key Challenge: Topic changes in LLMs are driven by contextual cues in the input, whereas human spontaneous thought can arise without any apparent trigger. A formal characterization of this distinction is lacking.

Goal: To formally define "topic" and "topic change" in the self-attention framework, derive the conditions governing their dynamics, and contrast the results with human cognition.

Key Insight: The Token Priority Graph (TPG) is adopted as the mathematical definition of a topic, and the self-attention → SVM convergence theory of Li et al. (2024) is used as the analytical framework.

Core Idea: Topics are defined via TPGs. It is shown that self-attention preserves topic priority after mixed-topic training, and that topic change requires low-priority tokens to appear more frequently in the input than high-priority tokens.

Method

Overall Architecture

The theoretical analysis is conducted in a simplified single-layer self-attention model trained with log-loss. A topic is defined as a set of TPGs (Definition 2), followed by definitions of topic continuation (Definition 3), ambiguous sequences (Definition 4), and topic change (Definition 5). Three main theorems are then derived. Validation is performed via RAG experiments on GPT-4o, Llama-3.3, Claude-3.7, and DeepSeek-V3.

Key Designs

1. TPG-Based Topic Definition:

   • Function: Defines a "topic" as a set of Token Priority Graphs \(\{\mathcal{G}^{(k)}\}_{k=1}^K\).
   • Mechanism: Strongly connected components (SCCs) in the TPG capture subsets of tokens with equal priority, while directed edges between SCCs encode strict priority ordering. An input sequence belongs to a topic if and only if all its tokens appear in that topic's TPG.
   • Design Motivation: This definition is both mathematically rigorous and intuitively natural, corresponding to associative memory models in neuroscience where concepts are nodes and associations are edges.

2. Theorem 2: Priority Preservation:

   • Function: Proves that mixed-topic training does not alter the token priority ordering of the original topic.
   • Mechanism: After training model \(\tilde{\mathbf{W}}_{ab}\) on a mixed dataset, the attention weight ordering for input sequences belonging to topic A is identical to that of a model trained solely on A — equal priorities remain equal, and strict priorities preserve their relative order.
   • Design Motivation: This explains why LLMs generally remain on-topic; mixed training does not corrupt the learned topic structure.

3. Theorem 3: Necessary Condition for Topic Change:

   • Function: Proves that topic change can only occur when low-priority tokens appear more frequently in the input than all high-priority tokens.
   • Mechanism: If tokens in the highest-priority SCC appear most frequently in the input, topic change is impossible. A topic change requires the existence of a non-highest-priority token \(x_j\) whose occurrence count exceeds that of every highest-priority token \(x_i\).
   • Design Motivation: Intuitively, this corresponds to the scenario where repetitive off-topic content dominates the context, potentially causing the model to drift.

4. Theorem 4: Effects of Input Length and Topic Ambiguity:

   • Function: Proves that (1) the probability of topic change approaches zero as input length increases, and (2) increased topic ambiguity does not raise the probability of topic change.
   • Mechanism: Input tokens are modeled as i.i.d. samples. When the sampling probability of the highest-priority token exceeds that of others, the law of large numbers guarantees its dominance in long sequences.
   • Design Motivation: This reveals a fundamental divergence from human cognition — humans are more prone to mind-wandering in prolonged discussions and more likely to switch topics when thematic associations are denser.

Experimental Validation

RAG experiments on GPT-4o, Llama-3.3, Claude-3.7, and DeepSeek-V3 are conducted to validate Theorem 4. One hundred arXiv papers serve as distinct topics; cosine similarity of generated text is measured with and without mixed context.

Key Experimental Results

Theoretical Validation (Single-Layer Self-Attention Simulation)

Input Length	Topic Maintained	Ambiguous Sequences	Topic Change
T=4	~60%	~25%	~15%
T=64	~85%	~10%	~5%
T=512	~95%	~4%	~1%

Frontier LLM Experiments (RAG Validation)

LLM	Cosine Sim vs. Input Length	Cosine Sim vs. Ambiguity
GPT-4o	Monotonically increasing ↑	Does not decrease
Llama-3.3	Monotonically increasing ↑	Does not decrease
Claude-3.7	Monotonically increasing ↑	Does not decrease
DeepSeek-V3	Monotonically increasing ↑	Does not decrease

Key Findings

• The necessary condition in Theorem 3 holds in 99.98% of simulation cases (the 0.02% exception is attributable to softmax approximation error).
• All four frontier LLMs exhibit behavior consistent with theoretical predictions: longer prompts and more ambiguous topics do not increase topic change.
• This stands in sharp contrast to human behavior, where prolonged discussion is associated with greater susceptibility to mind-wandering.

Highlights & Insights

• Theoretical Analysis from a Cognitive Science Perspective: Formalizing "spontaneous thought" — a concept from cognitive neuroscience — within the self-attention framework represents a rare and valuable interdisciplinary contribution.
• Elegance of TPG-Based Topic Definition: Defining topics via graph structure is both mathematically rigorous and intuitively natural, with clear correspondence to associative network models in neuroscience.
• Formal Characterization of AI vs. Human Cognition: The work explicitly identifies a qualitative divergence between LLMs and humans in topic-switching behavior — LLMs become more stable with longer context, whereas humans tend to diverge.

Limitations & Future Work

• Strong Simplifying Assumptions: Assumptions including single-layer self-attention, log-loss, and hardmax differ substantially from practical multi-layer LLMs.
• Indirect Experimental Validation: Using cosine similarity as a proxy for topic continuity is a relatively coarse measure.
• Limitations of Topic Definition: The TPG-based definition cannot capture semantic-level topic relatedness.

Related Work & Insights

• vs. Li et al. (2024) self-attention → SVM: This paper extends their theoretical framework by introducing mixed-topic settings and topic change analysis.
• vs. Ameisen et al. (2025) attribution graphs: The TPG-based topic definition echoes their work, though this paper emphasizes theoretical analysis over empirical findings.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First formal analysis of spontaneous topic changes in self-attention; highly original interdisciplinary perspective.
• Experimental Thoroughness: ⭐⭐⭐ Theoretical validation is thorough, but LLM experiments are relatively brief.
• Writing Quality: ⭐⭐⭐⭐ Theoretical presentation is well-organized; the cognitive science discussion could be deepened.
• Value: ⭐⭐⭐⭐ Reveals fundamental differences between LLMs and human cognition, with broad implications for understanding LLM behavior.

Related Papers

• [NeurIPS 2025] L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models
• [NeurIPS 2025] Let the Experts Speak: Improving Survival Prediction & Calibration via Mixture-of-Experts Heads
• [NeurIPS 2025] Scale-invariant Attention
• [NeurIPS 2025] Linear Attention for Efficient Bidirectional Sequence Modeling
• [NeurIPS 2025] Tensor Product Attention Is All You Need