Skip to content

TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Implicit Chain-of-Thought, Tool-Using Agents, Interleaved Reasoning, Adaptive Computation, Error Accumulation

TL;DR

TAC-CoT proposes an adaptive implicit Chain-of-Thought framework for tool-using agents by replacing verbose natural-language reasoning rationales with compact continuous latent segments while retaining textual tokens exclusively for tool calls, observations, and final answers, cutting forward passes by up to 6.8× while outperforming explicit-CoT baselines across mathematical and multi-hop QA benchmarks.

Background & Motivation

Integrating Large Language Model (LLM) agents with Chain-of-Thought (CoT) reasoning has become the prevailing paradigm for tackling complex tasks requiring mathematical problem-solving, multi-hop deliberation, and external knowledge grounding. In conventional Explicit-CoT frameworks, models verbalize every intermediate reasoning step into natural-language tokens prior to action and answer generation. While this discrete format provides human-readable transparency and integrates seamlessly with autoregressive token generation, decoding verbose intermediate rationales incurs substantial inference latency, inflated memory footprints, and severe susceptibility to error accumulation, where slight early missteps derail downstream trajectory planning and lead to catastrophic hallucination.

To overcome these generation bottlenecks, implicit CoT methods have emerged as an appealing alternative by conducting intermediate reasoning within continuous latent spaces rather than discrete token sequences. By replacing textual thoughts with compact continuous embedding representations, implicit reasoning mitigates verbalization overhead and error propagation. However, existing implicit CoT techniques remain fundamentally incompatible with tool-augmented agentic workflows due to two severe structural limitations: first, they consume a rigid, pre-allocated budget of continuous tokens in a single uninterrupted block before outputting the final answer, offering no intermediate junction to emit an executable tool call or ingest external observations; second, they enforce a static token count across all queries regardless of task difficulty or required decision rounds, causing simple queries to overthink and complex multi-hop problems to suffer from severe under-allocation of representational capacity.

The underlying tension lies between the strict requirement for explicit symbolic communication when interacting with external tools and the pressing need for compact, non-verbal latent deliberation to minimize compute and avoid textual reasoning drift. Core idea: decompose reasoning–action trajectories into an interleaved sequence where compact latent continuous segments handle intermediate deliberation, while discrete textual tokens are strictly reserved for tool invocations, environment observations, and final answers, yielding a query-adaptive, tool-empowered implicit reasoning architecture.

Method

Overall Architecture

TAC-CoT operates by establishing a dynamic interplay between continuous latent thought generation and explicit symbolic interaction throughout the agent trajectory. Given an input query, instead of autoregressively generating hundreds of discrete text tokens, the model computes an implicit reasoning segment comprising \(b\) continuous thought tokens (\(b=3\) by default). At the conclusion of each segment, a routing mechanism determines whether external assistance is needed or sufficient evidence has been gathered. If an external tool is required, the model greedily decodes an explicit tool invocation, receives the execution feedback as textual observation from the environment, and resumes latent deliberation in a subsequent implicit segment; otherwise, it consolidates previous observations and directly decodes the final answer.

The complete inference and execution pipeline is structured into four sequential phases: prefix encoding, continuous latent projection, dynamic routing selection, and explicit interaction/answer decoding:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query and Interaction Context<br/>Prompt Context: q ◦ τ"] --> B["Interleaved Implicit Trajectory Formulation<br/>Latent Segment Trajectory"]
    B --> C["Continuous Thought Token Generation<br/>Latent Projection via Projθ"]
    C --> D["Segment-End Dynamic Routing Selection<br/>Action vs Answer Logit Comparison"]
    D -->|Tool Logit > Answer Logit| E["Explicit Tool Invocation & Observation Ingestion<br/>Environment Interaction & Context Concatenation"]
    E --> B
    D -->|Answer Logit >= Tool Logit| F["Final Explicit Answer Decoding<br/>Greedy Text Generation"]

Key Designs

1. Interleaved Implicit Trajectory Formulation: Decoupling Latent Reasoning from Symbolic Interaction To resolve the structural incompatibility between purely continuous thoughts and executable environment interfaces, TAC-CoT reformulates the standard explicit trajectory \(\tau_E = (R^{(1)}, u^{(1)}, o^{(1)}, \dots, R^{(k)}, a)\) into an interleaved counterpart \(\tau_I = (C^{(1)}, u^{(1)}, o^{(1)}, \dots, C^{(k)}, a)\). Here, each lengthy natural-language rationale segment \(R^{(i)}\) comprising \(n\) discrete tokens is replaced by a compact latent segment \(C^{(i)} = (c^{(i)}_1, \dots, c^{(i)}_b)\) containing only \(b\) continuous thought vectors (\(b \ll n\)). Crucially, the explicit tool calls \(u^{(i)}\), environment observations \(o^{(i)}\), and final answer \(a\) are strictly preserved in their original textual form. This enables the agent to interface seamlessly with arbitrary external APIs and calculators while eliminating the severe decoding overhead and rhetorical verbosity of textual chain-of-thought traces.

2. Continuous Thought Token Generation: Latent Projection Bypassing Vocabulary Lookup To preserve semantic and computational coherence within the latent space without tokenization bottlenecks, continuous thought tokens \(c_t \in \mathbb{R}^d\) are derived directly from the model's internal representations rather than sampled from a discrete dictionary. Specifically, the hidden state \(h_t\) at the final transformer layer of the current sequence end is projected via an MLP projection head \(\text{Proj}_\theta\) (comprising \(\text{Linear}_{d \to d_p} \to \text{GELU} \to \text{Linear}_{d_p \to d} \to \text{LayerNorm}\)) to produce \(c_t = \text{Proj}_\theta(h_t)\). This continuous vector is directly fed back into the transformer input sequence via inputs_embeds in the subsequent autoregressive step, completely bypassing the conventional LM unembedding head and token lookup. This latent feedback loop enables high-density multi-step inference within a continuous representation manifold.

3. Segment-End Dynamic Routing Selection: Query-Adaptive Reasoning Budgeting To overcome the rigid, uniform computational budgets of prior implicit CoT frameworks, TAC-CoT introduces an adaptive routing decision at the boundary of every latent segment \(C^{(i)}\). Upon processing the final continuous token \(c_b^{(i)}\), the model inspects the output logits for the tool action trigger \(\ell_{\text{act}}\) and the answer termination trigger \(\ell_{\text{ans}}\). If \(\ell_{\text{act}} > \ell_{\text{ans}}\), the agent determines that further external evidence or programmatic calculation is necessary, triggering the greedy generation of an explicit tool call. Conversely, if \(\ell_{\text{ans}} \ge \ell_{\text{act}}\), the current segment serves as the concluding consolidation phase \(C^{(k)}\), immediately terminating the loop and yielding the final answer \(a\). Because simpler tasks resolve in a single segment while intricate multi-step queries engage multiple tool rounds, the total latent computation budget scales dynamically as \(k \times b\) in response to query complexity.

Loss & Training

Training the student model to perform robust latent deliberation without generating explicit rationales requires dense supervisory signals. TAC-CoT leverages a unified self-distillation objective where the same underlying transformer acts as both explicit teacher and implicit student:

  1. Trajectory Synthesis & Alignment: For mathematical domains with existing calculation annotations (GSM8K-Aug and GSM8K-Aug-NL), programmatic arithmetic expressions are aligned and spliced directly into natural-language rationales. For multi-hop QA (HotpotQA), trajectories are synthesized by prompting DeepSeek-V3 in a ReAct framework equipped with Wikipedia retrieval, filtering out any trajectory whose final prediction fails to match the ground truth.
  2. Student Reconstruction & Prediction Loss (\(\mathcal{L}_{\text{student}}\)): To enforce that the latent tokens truly encode the underlying multi-step reasoning steps, each explicit rationale segment \(R^{(i)}\) is partitioned into \(b\) sub-chunks \((R^{(i)}_1, \dots, R^{(i)}_b)\). A lightweight auxiliary reconstruction decoder \(D_\phi\) (active exclusively during training and discarded at inference) is tasked with reconstructing the omitted textual chunk from each corresponding latent token \(c^{(i)}_t\): $\(\mathcal{L}_{\text{implicit}} = - \sum_{i=1}^K \sum_{t=1}^b \log q_\phi(R^{(i)}_t \mid c^{(i)}_t)\)$ For explicit spans (tool calls \(u^{(i)}\) and final answer \(a\)), standard cross-entropy loss \(\mathcal{L}_{\text{explicit}}\) is applied. The complete student objective is \(\mathcal{L}_{\text{student}} = \mathcal{L}_{\text{implicit}} + \mathcal{L}_{\text{explicit}}\).
  3. Teacher Distillation Loss (\(\mathcal{L}_{\text{distill}}\)): The teacher branch is supervised on the full explicit trajectory \(\tau_E\) via standard cross-entropy \(\mathcal{L}_{\text{teacher}}\). To align the latent representations with the teacher's fully articulated reasoning states, an \(L_1\) distillation loss is enforced over the hidden states across all \(M\) layers at the answer-prompt boundary (the colon token preceding answer generation): $\(\mathcal{L}_{\text{distill}} = \frac{1}{M} \sum_{l=1}^M \| \text{sg}[h_{l,\text{teacher}}] - h_{l,\text{student}} \|_1\)$ where \(\text{sg}[\cdot]\) denotes the stop-gradient operator. The final multi-task training objective is \(\mathcal{L}_{\text{TAC-CoT}} = \mathcal{L}_{\text{student}} + \mathcal{L}_{\text{teacher}} + \mathcal{L}_{\text{distill}}\).

Key Experimental Results

Main Results

TAC-CoT is comprehensively evaluated across mathematical reasoning benchmarks (GSM8K, GSM-Hard, SVAMP) and multi-hop question answering benchmarks (HotpotQA, 2WikiMultihopQA, TriviaQA) using Qwen2.5-0.5B-Instruct, Llama-3.2-1B-Instruct, and Qwen3-4B backbones. Performance is measured via Accuracy (Acc %) / Exact Match (EM %) and the average number of forward passes per query (# Fwd).

Backbone Method Tool Use (TU) GSM8K (Acc / #Fwd) GSM-Hard (Acc / #Fwd) SVAMP (Acc / #Fwd) Math Avg (Acc / #Fwd) HotpotQA (EM / #Fwd) 2WikiMulti (EM / #Fwd) TriviaQA (EM / #Fwd) QA Avg (EM / #Fwd)
Qwen2.5-0.5B Explicit-CoT ✗ 50.9 / 117.3 13.4 / 160.0 59.0 / 61.9 41.1 / 113.1 13.2 / 951.2 21.0 / 739.1 11.3 / 891.6 15.2 / 860.6
TokenSkip ✗ 46.2 / 73.6 10.4 / 104.7 49.8 / 38.5 35.5 / 72.3 14.2 / 502.8 21.7 / 452.3 9.9 / 474.1 15.3 / 476.4
CODI ✗ 34.2 / 14.5 9.0 / 17.0 42.8 / 14.0 28.7 / 15.2 13.0 / 13.6 20.7 / 14.3 5.9 / 13.1 13.2 / 13.7
SIM-CoT ✗ 39.3 / 14.5 8.8 / 17.5 45.0 / 14.1 31.0 / 15.4 13.0 / 13.6 22.0 / 14.4 5.7 / 13.1 13.6 / 13.7
TAC-CoT ✗ 58.8 / 54.6 14.2 / 75.2 63.3 / 31.3 45.4 / 53.7 14.2 / 615.1 21.8 / 497.8 11.8 / 660.4 15.9 / 591.1
Explicit-CoT ✓ 52.4 / 109.2 30.4 / 145.7 62.4 / 58.4 48.4 / 104.4 33.1 / 309.0 40.8 / 247.8 53.3 / 293.0 42.4 / 283.3
TAC-CoT ✓ 61.2 / 43.9 32.8 / 57.0 67.7 / 26.4 53.9 / 42.4 36.2 / 49.4 46.5 / 39.8 54.8 / 43.3 45.8 / 44.2
Llama-3.2-1B Explicit-CoT ✓ 58.2 / 86.1 35.7 / 97.3 66.2 / 45.4 53.4 / 76.3 39.9 / 270.2 48.2 / 225.1 65.1 / 240.5 51.1 / 245.3
TAC-CoT ✓ 65.1 / 38.2 39.3 / 43.5 73.2 / 23.6 59.2 / 35.1 41.4 / 52.8 52.0 / 44.7 65.8 / 46.6 53.1 / 48.0
Qwen3-4B Explicit-CoT ✓ 78.0 / 115.2 49.1 / 154.3 78.5 / 61.7 68.5 / 110.4 43.4 / 272.2 59.9 / 224.5 72.3 / 235.3 58.5 / 244.0
TAC-CoT ✓ 83.4 / 41.6 54.2 / 55.9 86.0 / 23.8 74.5 / 40.4 45.9 / 43.1 60.5 / 44.9 75.5 / 37.4 60.6 / 41.8

Ablation Study & Empirical Analyses

1. Impact of Continuous Thought Tokens (# CT) per Segment (Qwen2.5-0.5B, Math Tasks)

Number of CT Tokens per Segment GSM8K (Acc) GSM-Hard (Acc) SVAMP (Acc) Average Acc (%)
0 (No Reasoning, equations only) 55.7 13.9 56.3 42.0
1 58.5 14.7 61.5 44.9
3 (TAC-CoT Default) 58.8 14.2 63.3 45.4

2. Effect of Replacement Depth on Error Accumulation (Qwen2.5-0.5B, Math Average)

Metric Explicit-CoT (0) 1 Segment Replaced 2 Segments Replaced 3 Segments Replaced 4 Segments Replaced TAC-CoT (All Implicit)
Mean Accuracy (Acc ↑ %) 41.1 41.7 42.4 43.4 44.9 45.4
Forward Passes (# Fwd ↓) 113.1 96.2 79.2 63.4 58.9 53.7

3. Hardware Wall-Clock Speedup on a Single NVIDIA L40S GPU (Qwen2.5-0.5B)

Benchmark Method Forward Passes (# Fwd) Average Latency (ms/sample) Realized Speedup
Agentic Math (Average) Explicit-CoT 104.4 1196.1 ms Baseline (1.0×)
TAC-CoT 42.4 (2.5× ↓) 531.0 ms 2.3× Speedup
Agentic QA (Average) Explicit-CoT 244.0 7654.5 ms Baseline (1.0×)
TAC-CoT 44.2 (5.5× ↓) 3187.5 ms 2.4× Speedup

Key Findings

  • Simultaneous Accuracy Superiority and Compute Efficiency: TAC-CoT is not merely an approximation or token-pruning heuristic; it consistently surpasses the fully verbalized Explicit-CoT baseline across all backbones (yielding +5.5%p on Qwen2.5-0.5B, +5.8%p on Llama-3.2-1B, and +6.0%p on Qwen3-4B in math accuracy) while simultaneously reducing forward passes by 2.3× to 6.8×.
  • Robustness Against Multi-Step Error Accumulation: Performance stratifications by reasoning depth on GSM8K reveal that for complex instances requiring 4+ steps, fixed-budget implicit baselines like SIM-CoT degrade severely, while TAC-CoT's advantage swells from +4.6 points (at 1 step) to +30.3 points (at 4 steps). Monotonically replacing explicit text chunks with continuous tokens directly suppresses over-thinking penalties and cumulative reasoning drift.
  • Enhanced Search and Multi-Tool Competence: In multi-hop QA, TAC-CoT achieves Answer Coverage @ 5 that matches or surpasses Explicit-CoT's Coverage @ 10 (e.g., 51.8% vs 48.8% on HotpotQA). On ToolBench, TAC-CoT achieves a 58.6% win rate over Explicit-CoT in complex multi-tool scenarios, demonstrating that compact latent reasoning prevents extraneous generation noise from corrupting tool invocation syntax and arguments.

Highlights & Insights

  • Interleaved Continuous–Discrete Interface: Unifying latent continuous thought tokens with explicit symbolic tokens bridges the long-standing architectural divide between implicit reasoning efficiency and agentic tool accessibility, creating an optimal division of labor.
  • Adaptive Test-Time Computation via Logit Competition: Utilizing the simple logit margin between tool execution and answer termination dynamically calibrates the reasoning horizon without requiring external meta-controllers or reinforcement learning overheads.
  • Zero-Inference-Overhead Auxiliary Supervision: The auxiliary reconstruction decoder forces continuous vectors to absorb rich semantic rationale representations during training, yet is discarded entirely at test time to retain a lean, high-throughput inference loop.

Limitations & Future Work

  • Author-Admitted Limitations: The dynamic routing selection operates strictly at the discrete segment level rather than the token level; while \(b=3\) is empirically effective, fine-grained token-level early stopping remains unexplored. Furthermore, interaction horizons in the benchmarks are capped at moderate multi-hop depths, leaving ultra-long-horizon agentic workflows open for future evaluation.
  • Independent Observations: Despite achieving up to a 6.8× reduction in forward passes, the physical wall-clock speedup on an NVIDIA L40S GPU tops out at 2.4× end-to-end (or 3.8× when isolating raw transformer generation). The remaining bottleneck stems from unoptimized token-injection execution paths and external API network latency.
  • Future Directions: Developing token-level differentiable stopping criteria; extending the interleaved latent reasoning framework to multimodal GUI agents and robotic embodied control where continuous perception-action loops dominate.
  • vs CODI / SIM-CoT: Prior implicit CoT frameworks restrict continuous thoughts to an unalterable static block immediately preceding final answer generation, structurally prohibiting any mid-trajectory tool execution and collapsing on multi-hop QA tasks; TAC-CoT fundamentally repairs this design flaw by interleaving continuous thoughts with explicit environment calls.
  • vs ReAct / Toolformer: Classical ReAct agents interleave explicit textual thoughts with actions, but repeatedly generating verbose rationales inflates latency and compounds reasoning errors; TAC-CoT compresses textual thoughts into dense latent representations while keeping the executable actions fully grounded and deterministic.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant architectural formulation bridging the longstanding incompatibility between continuous implicit reasoning and discrete tool execution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across 3 model families, 6 diverse benchmarks, extensive depth ablations, ToolBench win-rate analysis, and wall-clock hardware benchmarks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, clear exposition of trajectories and objectives, with exceptionally transparent diagnostic figures.
  • Value: ⭐⭐⭐⭐⭐ Highly practical foundation for deploying efficient, low-latency, and hallucination-resistant LLM agents on constrained edge devices and cloud endpoints.