Skip to content

Beyond Bag-of-Words: Generative Structural Regularization for Dense Table Retrieval

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/weijieliu-cs/GSR-Table
Area: LLM Reasoning
Keywords: Table Retrieval, Dense Retrieval, Bag-of-Words Degradation, Generative Structural Regularization, Adversarial Robustness

TL;DR

This paper reveals the bag-of-words degradation phenomenon in contrastive table dense retrieval, constructs a structure-adversarial benchmark with strictly lexical-equivalent negatives via Homologous Structural Swapping, and proposes GSR-Table, which applies orthogonal attention pooling and dynamic masking reconstruction to regularize dense representations with structural constraints.

Background & Motivation

Relational tables encode structured semi-formal knowledge and serve as foundational evidence sources for complex reasoning tasks such as open-domain question answering and tabular fact verification. With the prevalence of pre-trained encoders, dense retrieval based on dual-encoder architectures optimized via contrastive learning (such as InfoNCE) has become the dominant paradigm for retrieving query-relevant tables from massive heterogeneous corpora. These systems compress queries and tabular grids into low-dimensional vector representations and compute inner-product or cosine similarities for rapid retrieval. However, in standard benchmarks like OTT-QA and NQ-Tables, hard negatives typically exhibit vast lexical and lexical-field divergence from the target tables, incentivizing contrastive objectives to exploit superficial vocabulary overlap rather than resolving authentic row-column bindings.

This optimization bias leads directly to severe "bag-of-words degradation": when presented with negative candidate tables that share identical high-frequency entity tokens and keyword values with the query but possess scrambled row-column topology, existing retrievers assign them high relevance scores, walking into shallow lexical traps. The core tension lies in the fact that compressing a two-dimensional grid into a single global vector through standard mean or CLS pooling discards row-column dependency constraints, while encoder-side inductive biases are diluted during contrastive fine-tuning. Furthermore, the community has lacked an adversarial evaluation protocol capable of strictly enforcing bag-of-words equivalence to isolate structural binding fidelity from surface token statistics.

To systematically diagnose and resolve these twin challenges, the authors first establish the Structure-Adv benchmark using Homologous Structural Swapping to generate adversarial negatives that are mathematically bag-of-words equivalent to positive tables, rendering lexical matching completely uninformative. To eliminate shortcut learning, the authors propose GSR-Table, an asymmetric encoder-decoder framework. Core idea: regularize dense table representations using text generation as a structural constraint, decoupling row and column anchors through orthogonal attention pooling and enforcing autoregressive reconstruction under a dynamic masking bottleneck to compel retrieval vectors to retain row-column topological bindings.

Method

Overall Architecture

GSR-Table adopts an encoder-decoder architecture encompassing three foundational components: multi-view structural feature aggregation, sparse routing with generative structural compensation, and tri-objective joint optimization. On the encoder side, table tokens encoded with two-dimensional positional embeddings are aggregated into decoupled row and column structural anchors via row-specific and column-specific attention pooling, which are fused via an MLP into a single global retrieval vector. During training, a dynamic masking mechanism masks a subset of informative table cells to create a generative information bottleneck, obliging the decoder to cross-attend to the decoupled row/column representations to autoregressively reconstruct the table description. At inference time, the decoder is discarded entirely, preserving standard single-vector dense retrieval efficiency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Structured Table & Natural Query"] --> B["Multi-view Structural Feature Aggregation<br/>2D Positional Encoding + Row/Col Attn Pooling"]
    B --> C["Orthogonal Decoupling & Global Fusion<br/>Extract z_row & z_col -> Project to z_global"]
    C --> D["Sparse Routing & Generative Compensation<br/>Dynamic Cell Masking + Decoder Cross-Attention Reconstruction"]
    D --> E["Tri-objective Joint End-to-End Optimization<br/>Contrastive Loss + Generative Loss + Orthogonal Loss"]
    E -->|Inference Discards Decoder| F["Dense Retrieval Representation<br/>Single-Vector Inner-Product Search"]

Key Designs

1. Multi-view Structural Feature Aggregation: Preventing Topological Collapse in Vector Pooling

Standard CLS or global mean pooling compresses two-dimensional cell states into a single vector, indiscriminately obliterating orthogonal entity-attribute coordinates. To preserve structural geometry without inflating vector dimensionality, this design introduces decoupled dual-view attention pooling over encoder cell states \(H = \{h_1, h_2, \dots, h_n\}\). It derives row-level entity/record representation \(z_{\text{row}} = \text{AttnPool}_{\text{row}}(H)\) and column-level schema/attribute representation \(z_{\text{col}} = \text{AttnPool}_{\text{col}}(H)\). These complementary views are fused via a non-linear projection: $\(z_{\text{global}} = \text{MLP}([z_{\text{row}}; z_{\text{col}}])\)$ This ensures that the final single retrieval vector \(z_{\text{global}}\) is anchored upon explicitly decoupled topological projections rather than isotropic semantic averaging.

2. Sparse Routing and Generative Structural Compensation: Enforcing Reconstruction Bottlenecks against Token Copying

Providing structural embeddings solely within the encoder does not prevent contrastive learning from discarding structural cues in favor of lexical shortcuts. If the decoder can access complete encoder representations, it reconstructs target texts through local word stitching without utilizing pooled anchors. To prevent this, dynamic masking selectively hides informative table cell tokens, producing an incomplete representation \(H_{\text{partial}} = \text{Mask}(H)\). The decoder is conditioned upon: $\(\mathcal{C} = \{z_{\text{row}}, z_{\text{col}}\} \cup H_{\text{partial}}\)$ Because critical cell values are absent from \(H_{\text{partial}}\), the decoder cannot satisfy autoregressive teacher forcing through local copying. It is pressured to extract missing relational bindings from \(z_{\text{row}}\) and \(z_{\text{col}}\), driving back-propagation gradients directly into the pooled retrieval vectors to retain row-column associations.

3. Homologous Structural Swapping: Provable Lexical Equivalence for Adversarial Diagnostics

Existing benchmarks allow models to rely on surface token frequency disparities. To isolate structural understanding, this design introduces Homologous Structural Swapping (HSS). For an evidence cell \(c_{i,j}\) supporting query \(q\) in column \(j\), another non-evidence cell \(c_{i',j}\) within the same attribute column is randomly selected and their text contents are swapped, generating candidate perturbed table \(T^-\). Because swapping occurs strictly within the same schema column, it preserves column data types while severing row-entity bindings. Crucially, it guarantees exact multiset equivalence: $\(\text{BoW}(T^+) = \text{BoW}(T^-)\)$ A closed-loop LLM evaluator filters out candidates where the query remains answerable despite the perturbation. This eliminates out-of-vocabulary signals and keyword bias, leaving topological sensitivity as the sole differentiator.

Loss & Training

GSR-Table is optimized end-to-end using a tri-objective loss function:

\[L_{\text{total}} = L_{\text{CL}} + \lambda_1 L_{\text{Gen}} + \lambda_2 L_{\text{Ortho}}\]
  1. Query-Table Contrastive Loss (\(L_{\text{CL}}\)): Aligns query vector \(q\) with positive global table vector \(z_{\text{global}}^+\) against in-batch negatives using InfoNCE with temperature \(\tau\): $\(L_{\text{CL}} = -\log \frac{\exp(\text{sim}(q, z_{\text{global}}^+)/\tau)}{\sum_i \exp(\text{sim}(q, z_{\text{global}}^i)/\tau)}\)$
  2. Generative Structural Regularization Loss (\(L_{\text{Gen}}\)): Cross-entropy objective training the decoder to reconstruct natural language table description \(Y = \{y_1, \dots, y_m\}\) given joint condition \(\mathcal{C}\): $\(L_{\text{Gen}} = -\sum_{t=1}^m \log P(y_t \mid y_{<t}, \mathcal{C})\)$
  3. Multi-View Orthogonality Constraint (\(L_{\text{Ortho}}\)): Penalizes shared directional components between row and column anchors to guarantee semantic diversity: $\(L_{\text{Ortho}} = \frac{(z_{\text{row}}^\top z_{\text{col}})^2}{\|z_{\text{row}}\|_2^2 \|z_{\text{col}}\|_2^2}\)$

Both encoder and decoder comprise 12 Transformer layers (hidden dimension 768, randomly initialized without pre-trained language model checkpoints). Training is executed on 4 NVIDIA RTX 4090 GPUs across 922K synthesized query-table pairs for 30 epochs (approximately 13 hours total) with cell masking probability 0.15, learning rate \(1 \times 10^{-4}\), \(\lambda_1 = 0.3\), and \(\lambda_2 = 0.05\). At inference, the decoder is discarded, yielding pure single-vector latency.

Key Experimental Results

Main Results

Evaluations across standard benchmarks (NQ-TABLES, OTT-QA) and the Structure-Adv adversarial benchmark (3,000 queries, 30,000 BoW-equivalent negative tables, and 320,000 background distractors) yield the following comparative performance:

Benchmark / Setting Paradigm Model MRR@10 R@1 R@5 / R@10 R@10 / R@50
Structure-Adv (Adversarial Robustness) Sparse BM25 7.72 2.84 16.65 (R@5) 38.47 (R@10)
Neural Sparse SPLADE 22.36 7.90 38.24 (R@5) 66.85 (R@10)
Dense Bi-Encoder BIBERT 26.94 12.27 44.72 (R@5) 69.36 (R@10)
Dense Bi-Encoder DPR 29.15 14.43 47.86 (R@5) 72.52 (R@10)
Structure-Aware TAPAS 34.87 19.63 58.31 (R@5) 81.44 (R@10)
Structure-Aware DTR 31.68 17.47 54.26 (R@5) 79.19 (R@10)
Late-Interaction SSDRim 38.45 22.73 62.58 (R@5) 83.76 (R@10)
Proposed GSR-Table 43.82 28.63 69.39 (R@5) 88.27 (R@10)
NQ-TABLES (Standard Open-Domain) Sparse BM25 - 18.49 36.94 (R@10) 52.61 (R@50)
Neural Sparse SPLADE - 38.43 83.13 (R@10) 95.25 (R@50)
Strong Baseline SSDRim - 45.47 84.00 (R@10) 95.05 (R@50)
Proposed GSR-Table - 46.84 86.63 (R@10) 96.46 (R@50)
OTT-QA (Standard Open-Domain) Sparse BM25 - 23.98 51.94 (R@10) 69.11 (R@50)
Neural Sparse SPLADE - 64.45 89.39 (R@10) 95.89 (R@50)
Dense Baseline TAPAS - 56.05 86.72 (R@10) 94.49 (R@50)
Proposed GSR-Table - 56.42 87.42 (R@10) 95.60 (R@50)

Ablation Study

Ablation results on NQ-TABLES and the Structure-Adv benchmark:

Configuration NQ-TABLES NDCG@10 NQ-TABLES R@1 NQ-TABLES R@10 Structure-Adv MRR@10 Structure-Adv R@1 Structure-Adv R@10 Description
GSR-Table (Full) 65.89 46.84 86.63 43.82 28.63 88.27 Full proposed model
w/o \(L_{\text{Gen}}\) 63.74 44.21 84.58 34.47 19.87 79.64 Removing generation causes -9.35 MRR@10 on Structure-Adv
w/o Dynamic Masking 64.12 44.96 85.03 37.62 22.73 82.16 Decoder degenerates into copying unmasked tokens
w/o \(L_{\text{Ortho}}\) 65.12 45.83 86.01 41.68 26.70 86.76 Loss of geometric orthogonality collapses row/col views
w/o Row/Column Views 64.93 45.54 85.55 41.52 26.17 86.17 Single global pooling token fails to capture structural axes

In the BoW-enhanced diagnostic experiment where query tokens are injected into unanswerable negatives: - BM25 achieves an attack success rate (\(\text{ASR}_{\text{BoW}}\)) of 97.00% (\(\text{Margin} = -0.312\)); - DPR suffers an \(\text{ASR}_{\text{BoW}}\) of 66.33% (\(\text{Margin} = -0.084\)); - SSDRim reaches an \(\text{ASR}_{\text{BoW}}\) of 39.00% (\(\text{Margin} = 0.071\)); - GSR-Table substantially reduces \(\text{ASR}_{\text{BoW}}\) to 20.33%, achieving a 79.67% Pairwise Accuracy and a positive score Margin of 0.133.

Key Findings

  • Generative regularization is essential for structural sensitivity: Removing \(L_{\text{Gen}}\) drops Structure-Adv R@1 by 8.76 points (from 28.63 to 19.87), proving that contrastive loss alone fails to resist lexical shortcut traps.
  • Dynamic masking enforces structural compensation: Without masking, the decoder bypasses the pooled anchors by copying verbatim cell tokens, resulting in a 6.20 point drop in MRR@10 on Structure-Adv. A 15% mask ratio effectively cuts local shortcuts.
  • Decoupled orthogonal anchors outshine single-token pooling: Replacing \(z_{\text{row}}\) and \(z_{\text{col}}\) with a single global token (w/o Row/Column Views) degrades both standard retrieval (R@1 45.54 vs 46.84) and adversarial ranking (MRR@10 41.52 vs 43.82), demonstrating that two-dimensional inductive views provide irreplaceable structural regularization.

Highlights & Insights

  • Asymmetric Training-Inference Architecture: Complex structural modeling typically incurs heavy late-interaction latency (such as multi-vector MaxSim). GSR-Table restricts the generative decoder to training time as a structural regularizer, discarding it during inference to deliver standard single-vector low-latency retrieval.
  • HSS Benchmark with Exact Bag-of-Words Equivalence: By swapping cells strictly within attribute columns, \(\text{BoW}(T^+) = \text{BoW}(T^-)\) is mathematically guaranteed. This unmasks the hidden vulnerability of contrastive retrievers to lexical shortcuts.
  • Reconstruction as Bottleneck Regularization: Using text generation not for output generation, but as an auxiliary information bottleneck to back-propagate gradients into retrieval embeddings, offers an elegant design principle extendable to graph and hierarchical document retrieval.

Limitations & Future Work

  • Restricted to Well-Formed Relational Tables: The schema-preserving swapping mechanism and dual-axis orthogonal pooling presuppose explicit row-column matrix grids, limiting direct applicability to nested hierarchies, merged multi-level headers, or sparse visual documents.
  • Synthetic Data Supervision Overhead: Training relies on 922K LLM-generated table descriptions and synthetic queries; performance may degrade if domain-specific textual descriptions cannot be easily synthesized.
  • Exclusion of Hybrid Multi-hop Contexts: Evaluated purely on tabular evidence, the method does not currently model joint reasoning over interlinked text-table hybrid passages (e.g., full OTT-QA multi-hop graph retrieval).
  • vs TAPAS / DTR (Herzig et al., 2020, 2021): TAPAS and DTR incorporate row/column index embeddings in the encoder, but these inductive biases are easily washed out when pooled into a single vector and trained exclusively with contrastive loss. GSR-Table introduces decoder-side reconstruction pressure, directly penalizing loss of structural bindings in the pooled vector.
  • vs SSDR (Jin et al., 2023): SSDR constructs fine-grained phrase and header/value multi-vectors via late interaction (MaxSim). While effective, it suffers from heavy storage and computation overhead. GSR-Table maintains single-vector efficiency (768-d) while surpassing SSDRim on Structure-Adv (MRR@10 43.82 vs 38.45).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Uncovers the fundamental bag-of-words degradation in table retrieval and formulates generative structural regularization.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous BoW-equivalent adversarial construction, closed-loop LLM filtering, human adjudication, and training-matched controls.
  • Writing Quality: ⭐⭐⭐⭐⭐ Elegant mathematical formulations, intuitive motivating examples, and meticulous ablation explanations.
  • Value: ⭐⭐⭐⭐⭐ Demonstrates that structural awareness can be embedded into single dense vectors without inference overhead, providing high practical utility for tabular RAG.