CYPHERTURN: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/BarryQ/CypherTurn
Area: LLM Reasoning
Keywords: Knowledge Graph Reasoning, Multi-Turn Benchmark, Text-to-Cypher, Agentic Evaluation, Autonomy Divergence
TL;DR¶
Introduces CYPHERTURN, the first multi-turn conversational Text-to-Cypher benchmark comprising 721 sessions and 5,927 turns across 7 knowledge graphs, revealing the "Autonomy Divergence" phenomenon where frontier model rankings reorder under autonomous operation due to cascading error management.
Background & Motivation¶
Knowledge graphs and property graph databases serve as foundational infrastructure across enterprise business intelligence, scientific discovery, and complex relational modeling. With the rapid evolution of large language models, Text-to-Cypher systems that translate natural language queries into declarative Cypher queries over graph engines like Neo4j have garnered widespread research interest. However, existing benchmarks in this space, including the prominent CypherBench, ZOGRASCOPE, and SM3-Text-to-Query, operate exclusively within an isolated single-turn paradigm. Academic evaluation has predominantly tested a model's ability to translate fully self-contained questions into standalone queries, creating an acute disconnect from the real-world workflows through which human analysts investigate graph data.
In practical analytical workflows, information retrieval unfolds through iterative, multi-turn conversational sessions where subsequent queries heavily depend on preceding intermediate result sets. A business analyst might first identify departments with the highest headcounts, subsequently inquire which projects those teams handled, and follow up by asking what revenue each client generated. In such chained sequences, models must resolve implicit cross-turn anaphora (such as "those", "their", or "the top one") and carry forward intermediate node bindings and aggregation contexts. Crucially, single-turn benchmarks remain completely blind to error propagation: when a model manages its own history, an error in an early turn corrupts the contextual grounding for all downstream queries, triggering cascading session-wide failure.
While prior graph reasoning initiatives have explored interactive LLM agents (such as Think-on-Graph and Reasoning on Graphs) or sub-question graph retrieval (such as PankRAG), they continue to assume single-entry user queries without establishing a formal multi-turn conversational evaluation protocol. Consequently, the research community lacks the infrastructure to separate raw query generation competence from the capacity to manage accumulated context errors. Core idea: build CYPHERTURN, the first conversational Text-to-Cypher benchmark spanning 7 topological knowledge graphs, 13 graph-native conversational phenomena, and 721 multi-turn sessions, coupled with dual Guided and Agentic evaluation protocols to systematically quantify cross-turn error propagation and the Autonomy Divergence.
Method¶
The CYPHERTURN benchmark framework establishes a comprehensive infrastructure for multi-turn graph querying, comprising purpose-built synthetic topologies, conversational phenomenon taxonomies, persona register diversification, and complementary dual evaluation protocols.
Overall Architecture¶
The benchmark construction pipeline integrates synthetic graph modeling, programmatic session generation conditioned on conversational phenomena and user personas, dual-stage human verification on live Neo4j engines, and execution-based evaluation under both Guided (oracle context) and Agentic (autonomous context accumulation) protocols.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["7 Purpose-Built Synthetic KG Topologies<br/>Directed Cycle / Bipartite Cycle / Diamond / Hierarchy"] --> B["13 Conversational Phenomena & 9 Chain Modes<br/>Navigation / Aggregation / Filtering / Discourse Chaining"]
B --> C["10 Session Personas & Two-Stage Human Review<br/>Register Diversity + Live Neo4j Execution Verification (κ=0.83)"]
C --> D["Guided vs. Agentic Dual Evaluation Protocols<br/>Guided(Oracle Isolation) vs. Agentic(Self-Accumulation B=T×m)"]
D --> E["Five-Action Interactive Loop & Autonomy Diagnosis<br/>EXECUTE / INSPECT / SEARCH / ASK / SUBMIT + CER Tracking"]
Key Designs¶
1. Topology-Sensitive Synthetic Knowledge Graphs: Eliminating Memorization and Isolating Graph Reasoning Challenges Existing real-world KG benchmarks derived from Wikidata or DBPedia are susceptible to memorization artifacts from pre-training corpora, obscuring whether models perform true structural traversal. CYPHERTURN constructs seven synthetic knowledge graphs tailored to specific graph-theoretic challenges: Ancient Empire (directed cycles with dense numeric edge properties), Ocean Kingdom (diamond-convergence topology inducing path ambiguity), Magic Academy (bipartite cycles with dual Student-Professor paths testing entity resolution), Arcane Archive (citation self-loops on Tome nodes supporting recursive multi-hop queries), Celestial Court (arbitrary-depth bureaucratic administrative hierarchies), Stellar Colony (tree hierarchies overlaid with lateral peer-to-peer alliance edges), and Merchant Harbor (centralized multi-party invoice hub structures). All entities, relations, and numeric properties are entirely synthetic, ensuring every correct answer must be derived directly from graph pattern matching.
2. 13 Graph-Native Conversational Phenomena and 9 Chain Modes: Decoupling Contextual Dependencies and Anaphora Resolution
Multi-turn graph querying demands distinct compositional reasoning operations. CYPHERTURN categorizes 13 conversational phenomena across four functional classes: Navigation (EXPAND one-hop traversal, PIVOT rank-then-traverse, FIRST top-ranked traversal), Aggregation (AGG_AVG mean computation, AGG_MAX maximum selection, AGG_SUM summation, COUNT cardinality calculation), Filtering (VALUE_FILTER numeric thresholding, MULTI_COND compound constraints, REFINE attribute presence narrowing, FILTER general property matching), and Discourse (CONTRAST complement querying within entity types, TOPIC_SHIFT deliberate context breaks). At the structural construction layer, these map onto 9 chain modes. Apart from initial queries (None/T1) and TOPIC_SHIFT, all seven remaining modes enforce chain dependency where the prior turn's result acts as a seed set (WHERE n.name IN $prev or WITH ... ORDER BY ... LIMIT k MATCH ...), strictly measuring relational continuity.
3. 10 Session Personas and Two-Stage Human Verification: Register Diversity and Reliable Ground Truth Real users employ heterogeneous communication registers. To avoid overfitting to uniform synthetic prompts, CYPHERTURN samples uniformly from 10 distinct session personas: Impatient Executive (terse, demanding imperatives), Data Analyst (explicit column names, ordered numeric breakdowns), Novice User (vague pronouns like "those things"), Non-Native Speaker (subtle syntactic slips with recoverable semantics), and Adversarial User (skeptical phrasing challenging the system). Utterances generated from templates are stylistically polished by DeepSeek-V3.2 and rigorously audited by two Cypher-proficient annotators who independently assess query intent, unambiguous cross-turn anaphora, and persona naturalness (inter-annotator agreement \(\kappa = 0.83\)). Candidate sessions with malformed Cypher or ambiguous referents (2.3%) were rejected, and all retained gold queries were verified against live Neo4j database instances.
4. Dual Guided and Agentic Evaluation Protocols: Separating Generation Competence from Autonomous Error Management
To decouple query generation skill from historical context management, CYPHERTURN establishes two protocols over an identical 5-action space (EXECUTE_CYPHER, INSPECT_SCHEMA, SEARCH_VALUES, ASK_USER, SUBMIT_ANSWER). The Guided Protocol provides oracle context (the full schema, all prior gold natural-language utterances, and their gold execution results) with a 3-action budget per turn, isolating pure semantic translation. In contrast, the Agentic Protocol mirrors real-world deployment: the graph schema is withheld, and the model must manage an entire session under a shared action budget \(B = T \times m\) (\(m \in \{3, 5, 10\}\)), accumulating its own prediction history, discovering the schema via INSPECT_SCHEMA, and interacting with an anti-leakage persona-aware user simulator. Early mistakes directly poison downstream conversational context.
Benchmark & Evaluation Protocol¶
The benchmark evaluates performance at both turn and session granularities using four primary metrics: - Execution Accuracy (EX): Requires exact set equality of returned rows and column aliases between predicted query \(\hat{q}_t\) and gold query \(q_t\) on graph \(G\): $\(\text{EX}_t = \mathbf{1}[\text{exec}(\hat{q}_t, G) = \text{exec}(q_t, G)]\)$ - Provenance Subgraph Jaccard Similarity (PSJS): Measures structural traversal accuracy by computing the overlap between bound node element IDs \(V(\hat{q}_t)\) and \(V(q_t)\): $\(\text{PSJS}_t = \frac{|V(\hat{q}_t) \cap V(q_t)|}{|V(\hat{q}_t) \cup V(q_t)|}\)$ - Chain Error Rate (CER): Quantifies cascading failure probability conditioned on the immediately preceding turn failing: $\(\text{CER} = P(\text{EX}_t = 0 \mid \text{EX}_{t-1} = 0)\)$ - Session Exact Match (SEM): Binary session-level metric scored as 1 if and only if every turn achieves \(\text{EX} = 1\).
Key Experimental Results¶
Main Results: Guided vs. Agentic Protocols¶
Evaluations span 15 models (10 frontier API models, 2 small open-source models, and 3 fine-tuned architectures). Under Guided Oracle conditions, Claude Opus 4.7 leads at 0.647 EX. Under Agentic \(\times 3\) conditions, a consequential leaderboard reordering occurs: Gemini-3.1-Flash-Lite claims first place (0.432 EX), while Claude Opus 4.7 and GPT-5.5 tie for second, and Qwen3-235B and DeepSeek-V3.2 suffer steep cascading collapses (\(\Delta\text{EX} > -0.30\)).
| Model Tier | Model Name | Guided EX | Guided PSJS | Guided CER | Guided SEM | Agentic \(\times 3\) EX | Agentic CER | \(\Delta\text{EX}\) |
|---|---|---|---|---|---|---|---|---|
| Frontier | Claude Opus 4.7 | 0.647 | 0.846 | 0.434 | 0.046 | 0.402 | 0.712 | -0.245 |
| Frontier | GPT-5.5 | 0.625 | 0.828 | 0.424 | 0.025 | 0.401 | 0.698 | -0.224 |
| Frontier | Kimi-K2.5 | 0.602 | 0.819 | 0.469 | 0.028 | 0.335 | 0.749 | -0.267 |
| Frontier | Gemini-3.1-Flash-Lite | 0.572 | 0.744 | 0.508 | 0.017 | 0.432 | 0.695 | -0.140 |
| Frontier | Qwen3-235B | 0.513 | 0.737 | 0.466 | 0.006 | 0.199 | 0.833 | -0.314 |
| Frontier | GLM-5 | 0.506 | 0.816 | 0.496 | 0.003 | 0.274 | 0.773 | -0.232 |
| Frontier | DeepSeek-V3.2 | 0.506 | 0.729 | 0.474 | 0.004 | 0.204 | 0.835 | -0.302 |
| Frontier | ERNIE-5.0 | 0.468 | 0.676 | 0.575 | 0.003 | 0.295 | 0.830 | -0.173 |
| Frontier | MiMo-V2.5-Pro | 0.460 | 0.752 | 0.531 | 0.003 | 0.250 | 0.781 | -0.210 |
| Frontier | MiniMax-M2.7 | 0.139 | 0.379 | 0.856 | 0.000 | 0.156 | 0.862 | +0.018 |
| Small Open-Source | Llama-3.1-8B | 0.291 | 0.456 | 0.764 | 0.000 | 0.137 | 0.962 | -0.154 |
| Small Open-Source | Gemma-2-9B | 0.281 | 0.475 | 0.742 | 0.000 | 0.136 | 0.948 | -0.145 |
| Fine-tuned / Specialized | STRuCT-LLM-Novo | 0.526 | 0.763 | 0.506 | 0.015 | 0.333 | 0.766 | -0.193 |
| Fine-tuned / Specialized | text-to-cypher-gemma | 0.261 | 0.498 | 0.759 | 0.000 | 0.143 | 0.934 | -0.118 |
| Fine-tuned / Specialized | CypherRI-7B† | 0.013 | 0.283 | 0.989 | 0.000 | 0.001 | 0.999 | -0.012 |
(Note: †CypherRI-7B failed format compliance due to RL fill-in-the-middle token contamination; excluded from aggregate means)
Ablation Study and Diagnostic Experiments¶
The following table details budget scaling behavior, controlled fine-tuning paradigms (the Specialization Paradox), and protocol sensitivity variants:
| Experiment Setting | Model / Configuration | Key Metric (EX) | Empirical Observations and Mechanisms |
|---|---|---|---|
| Budget Scaling | Gemini-3.1-FL (\(\times 3 \to \times 5 \to \times 10\)) | \(0.432 \to 0.431 \to 0.430\) | Budget flatline: schema-disciplined model uses \(\approx 2\) actions/turn regardless of total budget |
| Budget Scaling | Claude Opus 4.7 (\(\times 3 \to \times 5 \to \times 10\)) | \(0.402 \to 0.429 \to 0.424\) | Plateaus at \(\times 5\); additional budget fails to close the remaining gap |
| Budget Scaling | Llama-3.1-8B (\(\times 3 \to \times 5 \to \times 10\)) | \(0.137 \to 0.203 \to 0.261\) | Significant gain: small model benefits from overcoming exploratory search starvation |
| Specialization Paradox | Gemma-2-9B (Base) | Guided: 0.281, Agentic: 0.136 | Base generalist model maintains linguistic anaphora tracking capability |
| Specialization Paradox | text-to-cypher-gemma (Single-turn SFT) | Guided: 0.261, Agentic: 0.143 | SFT degrades guided accuracy; induces hardcoded literal list anti-patterns in multi-turn |
| RL Reasoning Pair | QwQ-32B (Base Reasoning) | Guided: 0.395, Agentic: 0.324 | Base open-weight long-CoT reasoning model |
| RL Reasoning Pair | STRuCT-LLM-Novo (RL + CoT) | Guided: 0.526, Agentic: 0.333 | Graph edit distance reward + SQL/Cypher cross-transfer yields +13.1 points in Guided |
| Protocol Robustness | Agentic-PS (Persistent Schema) | Rank \(\rho = 0.915\), Top-1: GPT-5.5 (0.463) | Displaces Gemini from top-1 while frontier rank correlation remains strong |
| Protocol Robustness | Disclosed-Horizon (\(T\) & target revealed) | Rank \(\rho = 0.879\), Top-1: Claude (0.456) | Long-horizon planning signal elevates Claude; Gemini drops to 0.364 |
| Protocol Robustness | Budget-Encouraging Prompt (\(\times 10\)) | Rank \(\rho = 0.786\), Top-1: GPT-5.5 (0.399) | Encouraging full budget usage lowers accuracy in 6/7 models due to noisy retries |
Key Findings¶
- Low Performance Ceilings and Near-Zero Session Success: Even under guided oracle conditions, the top model achieves only 64.7% execution accuracy, and session-level correctness (SEM) remains below 5% (Claude Opus 4.7 solves only 33 of 721 sessions). In autonomous operation, CER reaches 0.77 across models and surpasses 0.83 for DeepSeek-V3.2 and Qwen3-235B, showing that once a turn fails, subsequent failure is almost guaranteed.
- The Autonomy Divergence: While global rank correlation across all 14 models remains high (\(\rho = 0.86\)), restricted to the frontier tier \(\rho\) drops to 0.63. Under autonomy, 11 of 45 pairwise model rankings reverse significantly: Gemini-3.1-Flash-Lite ascends from 4th to 1st, while Qwen3 collapses from 5th to near the bottom. This divergence establishes error management and historical state preservation as partially independent dimensions from raw Cypher generation.
- The Specialization Paradox: Fine-tuning Gemma-2-9B on single-turn Neo4j Text2Cypher data degrades performance from 0.281 to 0.261 EX. Single-turn supervision reinforces the assumption of query independence, prompting models to hardcode preceding results as static literals (
WHERE n.name IN ['...']) rather than using dynamic graph chaining (WITH ... MATCH). Conversely, STRuCT-LLM-Novo's multi-task RL with CoT boosts accuracy by +13.1 points over QwQ-32B. - Action Budgets Cannot Fix Strategic Deficiencies: Scaling budgets from \(\times 3\) to \(\times 10\) fails to close the autonomy gap for frontier models. Capable models self-limit to approximately 2 actions per turn; collapsing models squander extra actions on repetitive speculative executions that fail to localize the true schema or scoping error.
Highlights & Insights¶
- Conceptualizing Autonomy Divergence in Graph Systems: Highlights that raw query syntax proficiency under oracle conditions is an inadequate proxy for autonomous agent deployment, formalizing error cascading via the Chain Error Rate (CER) metric.
- Counter-Intuitive Success of Minimalist Exploration: Gemini-3.1-Flash-Lite's resilience stems from an ultra-disciplined
INSPECT_SCHEMA\(\to\)SUBMIT_ANSWERloop (83.5% inspection actions), avoiding speculativeEXECUTE_CYPHERloops that burn through budgets and compound context errors. - Syntax Contamination in Complex Chains: Case studies demonstrate that general-purpose open models (like Llama-3.1-8B) suffer severe SQL syntax intrusion (e.g., embedding
SELECT MAX(...) FROM ...inside Cypher), highlighting the necessity of domain-specific alignment for declarative graph languages.
Limitations & Future Work¶
- Synthetic Graph Topologies vs. Real-World Dirty Data: All 7 knowledge graphs are synthetic. While this guarantees topological control and avoids pretraining contamination, it lacks the messy naming conventions, null distributions, and unstructured edge properties of enterprise databases.
- Strict Metric Sensitivity to Output Aliases: Execution Accuracy requires exact equality of both row sets and returned column header names, penalizing semantically correct queries with minor alias discrepancies. Future iterations should incorporate relaxed semantic equivalence metrics.
- Monolingual Evaluation: All conversational utterances and user interactions are conducted in English. Evaluating multi-turn anaphora resolution in languages with high context omission (e.g., Chinese or Japanese) remains an open challenge.
Related Work & Insights¶
- vs CypherBench / ZOGRASCOPE: CypherBench evaluates single-turn Wikidata questions; CYPHERTURN introduces multi-turn sessions, anaphora annotations, and autonomous agent loops over Neo4j.
- vs BIRD-INTERACT / MINT: While BIRD-INTERACT established interactive evaluation for SQL, CYPHERTURN pioneers the interactive multi-turn paradigm for property graphs, designing graph-specific provenance metrics (PSJS) and error cascading metrics (CER).
- vs MTGQL: MTGQL targets financial nGQL on NebulaGraph; CYPHERTURN evaluates the industry-standard Neo4j Cypher language across 7 diverse topologies with rigorous dual-stage human verification and an agentic action budget.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First comprehensive multi-turn benchmark for Text-to-Cypher; introduces the foundational Autonomy Divergence concept.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 15 diverse models across 7 graphs, 13 phenomena, 10 personas, 3 budget tiers, and 3 protocol sensitivity ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous taxonomy, clean mathematical formulations, and compelling turn-by-turn case studies.
- Value: ⭐⭐⭐⭐⭐ Establishes a formidable open challenge and actionable diagnostic tooling for the conversational knowledge graph reasoning community.