Skip to content

TORe: Rehearsal-Free Continual Learning for Table QA via Operation Recipe Reconstruction

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/oakor/TORe
Area: LLM Reasoning
Keywords: Table Question Answering, Continual Learning, Rehearsal-Free Learning, Operation Recipe Reconstruction, Execution-Guided Synthesis

TL;DR

Addressing catastrophic forgetting caused by schema shifts and severe privacy regulations in evolving tabular environments, TORe introduces a rehearsal-free continual learning framework for Table QA that abstracts reasoning procedures into schema-agnostic Operation Recipes, identifies fine-grained structural deviations, and reconstructs both historical and consolidation knowledge via execution-verified LLM synthesis without storing raw historical data.

Background & Motivation

Large Language Models have exhibited remarkable proficiency in Table Question Answering (TableQA), enabling non-expert users to query semi-structured information using natural language. However, the vast majority of existing TableQA benchmarks and modeling techniques operate under an idealized static distribution assumption. In enterprise and production environments, tables evolve continuously with shifting schemas, emerging vertical domains, and structural rearrangements. When models are sequentially deployed across streaming task distributions with limited labeled data (fewer than 200 examples per task), they face severe performance degradation on previously mastered domains, presenting the dual challenge of rapid adaptation under data scarcity and resistance to catastrophic forgetting in TableQA Continual Learning (TableQA-CL).

Standard continual learning paradigms face prohibitive barriers when adapted to realistic TableQA scenarios. Rehearsal-based methods rely on replaying stored historical examples, which is strictly prohibited in domains governed by stringent privacy and regulatory policies, such as finance and healthcare. Program-based continual semantic parsing frameworks require executable SQL annotations or explicit grammar trees, which fail entirely on heterogeneous, semi-structured web tables that lack programmatic query engines. Furthermore, parameter-based continual learning approaches commonly assume access to oracle task IDs during inference to route task-specific adapters, an assumption that collapses in open-ended, continuous prediction streams where task boundaries are unidentified.

When human analysts encounter new table formats, they do not memorize previous cell contents verbatim; rather, they retain procedural problem-solving schemas—such as selecting conditional rows, grouping categories, and aggregating numerical fields—and readily transfer these abstract strategies to new domains. Inspired by this cognitive intuition, this work decouples logical reasoning structures from concrete semantic payloads. Core idea: abstract complex tabular reasoning chains into schema-agnostic Operation Recipes, pinpoint missing historical reasoning patterns via structural deviation analysis, and leverage execution-verified dual-track synthesis to reconstruct historical and consolidation capabilities on current-domain tables without storing any raw historical data.

Method

Overall Architecture

The TORe framework operates across three key stages: Recipe Deviation Discovery, Recipe-Guided Dual-Track Synthesis, and Task-Adaptive Representative Selection. Given a new streaming task, an external frozen LLM teacher (Meta-Llama-3.1-70B-Instruct) distills training reasoning paths into abstract Table Operation Recipes (TORs) by projecting trajectories onto 9 predefined atomic table operations and stripping away all semantic parameters. By computing the schema-aware set difference between the accumulated historical recipe bank and the current task recipes, TORe isolates the knowledge deviation \(\Delta R_t\). In the synthesis stage, the model generates Consolidation Knowledge (CK) to alleviate current-task data scarcity and projects \(\Delta R_t\) onto current schemas to synthesize Historical Knowledge (HK). An execution compiler serves as a strict quality filter, verifying intermediate table transformations and discarding hallucinated candidates. Finally, diversity-aware K-means clustering over TF-IDF feature space selects a compact, representative core dataset to fine-tune the downstream model via LoRA.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Stream Input: Current Tables & Historical Streams"] --> B["Operation Recipe Extraction & Deviation Discovery<br/>Extract 9 atomic operators & compute deviation ΔRt"]
    B --> C["Execution-Guided Dual-Track Knowledge Synthesis<br/>Consolidation (CK) & Historical (HK) with verification"]
    C --> D["Task-Adaptive Data Selection via Clustering<br/>TF-IDF representation & K-means centroid sampling"]
    D --> E["Downstream LoRA Optimization: Anti-forgetting & Forward Transfer"]

Key Designs

1. Operation Recipe Extraction & Deviation Discovery: Decoupling Abstract Operations from Surface Semantics

Conventional text-level synthesis struggles because domain vocabularies across sequential tasks exhibit virtually zero semantic overlap, causing models to overfit to superficial tokens or suffer from negative transfer. TORe establishes a comprehensive Table Operation Pool comprising 9 foundational, schema-agnostic operators: select_row, select_column, add_knowledge_column, add_inferred_column, group_column, sort_column, stitch_tables, select_table, and change_column_name. For a training instance \((T_0, Q)\), an LLM iteratively predicts operations and content-specific parameters \(((op_1, \lambda_1), \dots, (op_n, \lambda_n))\) to resolve the query. Upon completion, all content-dependent parameters \(\lambda\) are discarded, preserving only the procedural operator sequence \(r = [op_1, \dots, op_n]\) to form the task-level recipe set \(R_t\). With the accumulated historical bank \(R_{past} = \bigcup_{i=1}^{t-1} R_i\), TORe computes the schema-aware knowledge deviation \(\Delta R_t\): $\(\Delta R_t = \{r \in R_{past} \setminus R_t \mid \text{Valid}(r, \mathcal{S}(D_{current}))\}\)$ where \(\text{Valid}(\cdot)\) verifies whether the topological schema \(\mathcal{S}(D_{current})\) (e.g., presence of sortable fields, categorical attributes, single vs. multi-table layout) can physically support recipe \(r\). This mechanism isolates transferable reasoning blind spots while guaranteeing zero storage of sensitive historical data.

2. Execution-Guided Dual-Track Knowledge Synthesis: Grounded Reconstruction and Cross-Domain Projection

Unconstrained synthetic data generation is notoriously susceptible to semantic hallucinations and compounding errors that degrade downstream performance. TORe introduces a dual-track synthesis mechanism paired with execution verification. For current-task enhancement, Consolidation Knowledge (CK) is constructed by instantiating current recipes \(R_t\) on current tables, expanding limited supervision. For historical retention, Historical Knowledge (HK) projects absent historical recipes \(r \in \Delta R_t\) onto current-domain tables. If structural mismatches occur (e.g., applying a multi-table stitch_tables recipe to a single-table input), a schema transformation operator \(T_{adapt} \leftarrow \text{Transform}(T, \text{Schema}(r))\) synthesizes contextually relevant auxiliary tables. Crucially, during parameter instantiation, the LLM generates a candidate set \(\Lambda\) (\(k=3\)). The parameterized sequence \(\tilde{r} = (r, \lambda)\) is compiled and executed in a sandboxed tabular environment. Only parameter sets that execute error-free and yield a valid transformed table state \(T'\) are retained; failed attempts are immediately aborted. The verified state is then passed to the LLM to format corresponding high-fidelity QA pairs and Chain-of-Thought (CoT) rationales.

3. Task-Adaptive Data Selection via Clustering: Balancing Diversity and Representation without Overfitting

Directly pooling unchecked volumes of synthetic samples dilutes the empirical distribution of genuine annotations and induces performance collapse through redundant patterns. TORe merges the synthesized \(S_{CK}\) and \(S_{HK}\) pools and computes TF-IDF representations over the question and CoT textual representations, followed by K-means clustering. Given a fixed synthesis budget \(N_K\) (defaulting to 400, maintained at a 1 HK : 3 CK ratio), the selection selects the sample closest to each cluster centroid \(c_j\): $\(d^*_j = \arg\min_{d_i \in C_j} \| \text{TF-IDF}(d_i) - c_j \|\)$ forming the core synthetic subset \(D_{core} = \bigcup_{j=1}^{N_K} \{d^*_j\}\). Combining this curated subset with original training instances yields \(D_{train} = D_{core} \cup D_{current}\). This diversity-aware clustering prevents feature-space overconcentration and eliminates pathological outlier generations, maximizing regularization effectiveness under a constrained training budget.

Loss & Training

The downstream backbone is fine-tuned using Low-Rank Adaptation (LoRA, rank \(r=32\)) on attention and feed-forward projections while keeping backbone parameters frozen. Given the unified training mixture \(D_{train} = D_{core} \cup D_{current}\) where each example comprises tabular context \(X=(Q, D)\) and step-by-step reasoning answer \(Y\), optimization minimizes the standard autoregressive language modeling cross-entropy loss: $\(\mathcal{L}_{CL}(\theta) = - \sum_{i=1}^{|Y|} \log P_\theta(y_i \mid y_{<i}, X)\)$ The AdamW optimizer is employed with a learning rate of \(2\times 10^{-4}\). Offline synthesis utilizes a permanently frozen Meta-Llama-3.1-70B-Instruct accelerated via vLLM. Because the generator is never fine-tuned on its own synthetic outputs and recipes are strictly derived from genuine human task data, the system provably avoids recursive model collapse.

Key Experimental Results

Main Results

Evaluation was conducted on two streaming benchmarks: Mixed-QA-Stream (9 sequential single-table tasks derived from TableInstruct and SQUALL) and Mixed-Struct-Stream (9 heterogeneous multi-table tasks from MMQA and OMNISQL). Tasks feature strictly disjoint tables and fewer than 200 training examples per task. Evaluated metrics include Average Accuracy (\(ACC_a\)), Whole Accuracy on combined test sets (\(ACC_w\)), Forgetting Measure (\(FM\), lower is better), and Forward Transfer (\(FWT\)).

Model & Method Category Data Replay Mixed-QA \(ACC_a\) Mixed-QA \(FM\) (↓) Mixed-QA \(FWT\) Mixed-Struct \(ACC_a\) Mixed-Struct \(FM\) (↓) Mixed-Struct \(FWT\)
Qwen-2.5-1.5B
SFT Original 0% 23.50 0.80 13.94 23.24 5.59 20.53
RandSel Rehearsal 10% 22.59 2.57 16.93 27.99 2.29 24.37
KMeansSel Rehearsal 10% 22.49 4.37 17.80 25.74 5.48 23.30
MTL Upper Bound 100% 17.70 5.89 15.28 25.62 5.61 23.93
SEE Rehearsal+PEFT 10% 26.06 2.91 21.87 30.07 1.26 23.20
C3 (Oracle Task ID) PEFT 0% 20.05 – – 19.66 – –
Naive generate Synthesis 0% 22.90 2.21 16.82 22.44 9.59 22.45
SSR (Full) Self-Synthesis 0% 5.05 10.91 9.96 12.63 10.89 11.41
TORe (Ours) Synthesis 0% 27.51 1.03 20.20 30.43 1.85 24.24
DeepSeek-R1-8B
SFT Original 0% 38.31 2.25 23.94 31.05 12.54 27.77
RandSel Rehearsal 10% 39.34 5.22 32.17 32.81 5.76 21.41
KMeansSel Rehearsal 10% 40.62 2.53 32.06 33.17 5.07 22.94
MTL Upper Bound 100% 38.87 4.58 31.41 37.70 3.86 36.20
SEE Rehearsal+PEFT 10% 41.33 3.03 31.51 43.13 2.91 28.55
C3 (Oracle Task ID) PEFT 0% 35.74 – – 31.47 – –
Naive generate Synthesis 0% 36.56 6.87 27.24 34.04 6.98 29.71
SSR (Full) Self-Synthesis 0% 22.88 11.49 21.62 19.68 12.50 20.01
TORe (Ours) Synthesis 0% 42.46 1.99 35.62 43.95 1.86 39.18

Ablation Study

Ablations on Mixed-QA-Stream isolate the contribution of recipe guidance, clustering, knowledge ratios, and sample volume budgets \(N_K\):

Configuration / Variant Mechanism Description \(ACC_a\) \(ACC_w\) \(FM\) (↓) \(FWT\)
Full TORe Full framework (1HK:3CK, \(N_K=400\), K-means) 27.51 29.08 1.03 20.20
w/o clustering Random downsampling instead of centroid selection 26.26 27.82 3.08 16.75
random TOR Random, non-targeted operation recipe sequences 24.08 24.98 2.27 16.09
only HK Historical knowledge reconstruction only (no CK) 25.50 27.03 2.81 18.34
1HK : 3CK Balanced synergistic ratio 27.51 29.08 1.03 20.20
1HK : 1CK Equal historical and consolidation ratio 26.79 28.59 3.39 17.55
3HK : 1CK Historical-heavy synthesis ratio 26.61 28.38 2.73 18.93
only CK Consolidation knowledge only (no historical replay) 25.30 26.82 2.31 21.54
\(N_K = 1200\) Excessive synthetic data volume 26.66 28.27 3.46 20.26
\(N_K = 800\) Moderate synthetic data volume 26.76 28.27 3.71 19.48
\(N_K = 400\) Optimal synthetic data budget 27.51 29.08 1.03 20.20
\(N_K = 0\) (SFT) Zero synthetic data augmentation 23.50 24.95 0.80 13.94

Key Findings

  • Quality and Structural Fidelity Outweigh Pure Data Scale: The impact of synthetic data budget exhibits a pronounced inverted U-shaped curve. Increasing \(N_K\) from 400 to 1200 causes \(ACC_a\) to decline from 27.51 to 26.66 while \(FM\) worsens more than threefold from 1.03 to 3.46. Beyond the intrinsic structural diversity of the recipe space, excess synthetic volume injects subtle hallucinated noise that destabilizes internal representations.
  • Recipe Overlap Strongly Correlates with Continual Retention: Pearson correlation between recipe-level overlap across tasks and empirical SFT performance is exceptionally strong (\(r=0.87, p<0.001\)), whereas token-level semantic overlap shows negligible correlation (\(r=0.34, p=0.12\)). A Fisher's Z-transformation (\(Z=3.42, p<0.001\)) confirms that structural procedural recipes, rather than lexical tokens, govern cross-domain table reasoning transfer.
  • Order Robustness and Positive Forward Transfer: Under randomized task-stream orderings, TORe achieves an Order Stability score (\(\Omega\)) of 99.24%, markedly outperforming MTL (74.46%) and self-synthesis SSR (36.27%). Furthermore, on DeepSeek-R1-8B, TORe achieves 39.18% Forward Transfer (\(FWT\)), confirming that acquiring modular structural operators facilitates zero-shot adaptation to future schemas.

Highlights & Insights

  • From Concrete Instance Replay to Procedural Recipe Reconstruction: By decoupling table reasoning into 9 schema-agnostic atomic operators, TORe replaces raw data buffers with abstract algorithmic recipes. This preserves procedural problem-solving skills across disjoint domains while guaranteeing privacy compliance (domain reconstruction audit yields 18.7%, matching the 20% random guess baseline).
  • Execution Sandboxing as an Anti-Hallucination Shield: Integrating a deterministic table execution environment into the generation loop provides an ironclad filter against LLM hallucinations. Candidate parameters that fail to compile or execute are immediately discarded, elevating post-verification recipe extraction accuracy to 99.0% and preventing error accumulation across continuous updates.
  • Synergistic Balance of Consolidation and Historical Knowledge: CK establishes baseline schema familiarity for data-scarce new tasks, providing the structural anchor required for HK to successfully project historical operators. The 1 HK : 3 CK ratio strikes an optimal balance between new knowledge acquisition and historical skill retention.

Limitations & Future Work

  • Author-Acknowledged Constraints: The predefined operation pool is restricted to 9 general-purpose operators, which may not fully span highly specialized niche operations (such as internal rate of return calculations or advanced econometric tests). Additionally, recipe extraction and synthesis rely on a large 70B teacher model, creating resource bottlenecks for local edge deployment.
  • Unaddressed Challenges: The benchmarks model discrete schema shifts across task boundaries but do not emulate continuous temporal concept drift within dynamic live tables. Furthermore, for highly nested multi-table joins exceeding 5 sequential operations, LLM instantiation rejection rates reach up to 62%.
  • Future Directions: Exploring automated discovery to dynamically expand the operation pool, distilling lightweight dedicated models for recipe extraction, and extending procedural recipe abstractions to heterogeneous multi-modal structures (unifying tables, text, and knowledge graphs).
  • vs. Rehearsal-Based Methods (RandSel / KMeansSel / MTL): Rehearsal approaches require caching raw historical tables, violating data privacy regulations in confidential environments, while full replay (MTL) suffers from negative transfer under severe schema divergence; TORe retains zero historical records and surpasses rehearsal performance.
  • vs. Parameter-Isolated Methods (C3 / SEE): C3 relies on an oracle task ID during inference to route parameters, which is unavailable in open-stream environments; SEE requires physical data replay for ensemble training. TORe conducts standard inference without external task boundaries.
  • vs. Unconstrained Synthesis Methods (Naive Generate / SSR): SSR refines synthetic data using the learner's own shifting weights, causing severe recursive degradation (\(FM = 10.91\)); TORe employs a frozen generator and environment-verified compilation, establishing an analytical upper bound on error propagation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Decouples table reasoning into abstract operation recipes and execution-verified synthesis, establishing a principled rehearsal-free continual learning paradigm.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Constructs two controlled long-sequence benchmarks across multiple open-source backbones, complemented by ablation studies, order-robustness tests, and formal privacy audits.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent technical narrative, transparent problem formulation, crisp visualizations, and rigorous error analyses.
  • Value: ⭐⭐⭐⭐⭐ Offers a concrete, privacy-compliant methodology for lifelong reasoning adaptation in structured data systems and tool-augmented LLM architectures.