Q-CAT: A Non-Parametric Query Filtering System for Prohibited Content in Dynamic Enterprise Environments¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Query Filtering System, Non-Parametric Classification, Enterprise Data Leakage Prevention, Dynamic Policy Adaptation, Vector Retrieval & Quality Assurance
TL;DR¶
Q-CAT establishes a non-parametric, training-free query filtering framework for dynamic enterprise environments, utilizing LLM-synthesized and rubric-verified query pools alongside dense vector retrieval to achieve millisecond-level inference and instant zero-retraining adaptation under evolving security policies.
Background & Motivation¶
The pervasive integration of Large Language Models (LLMs) into enterprise workflows has dramatically boosted knowledge worker productivity across drafting, summarizing, and strategic decision-making. However, this deployment exposes organizations to acute risks of confidential data leakage when employees intentionally or inadvertent submit proprietary information into external commercial LLMs. Unlike public safety moderation benchmarks that target universal harms such as violence or hate speech, corporate confidentiality boundaries are defined by bespoke, organization-specific governance policiesโsuch as internal product specifications, undisclosed pricing strategies, asset inventories, or executive meeting agendas.
Crucially, corporate data policies in real-world business settings are inherently non-static and continuously shift across project lifecycles. An emerging product specification may constitute strictly prohibited intellectual property during early development stages, yet transition into permissible public information upon official product launch; conversely, tightening regulatory mandates can instantaneously classify previously benign internal workflows as restricted. Existing static guardrail models (e.g., LlamaGuard, QwenGuard) fail because they are constrained to fixed harm taxonomies. Supervised parametric classifiers, while initially competitive, require prohibitively expensive full-model retraining upon every policy modification, inevitably suffering from distribution shift and catastrophic forgetting. Meanwhile, direct prompt-based LLM classification incurs substantial inference costs, introduces severe response delays with long-tail latency (often tens of seconds), and exhibits high false positive rates due to overly conservative decision boundaries.
To resolve the trilemma among low latency, minimal compute budget, and agile adaptation to evolving governance, this paper abandons parametric weight updating in favor of non-parametric vector density estimation. Core idea: by constructing a high-fidelity synthetic Query Pool validated through multi-faceted rubrics and pairing it with dense vector retrieval, policy additions, deletions, label shifts, and error corrections are decoupled into instantaneous, zero-retraining database updates.
Method¶
Overall Architecture¶
The Q-CAT pipeline operates across two decoupled operational phases: the Policy Registration Step (offline/at modification time) and the User Query Inspection Step (real-time online serving). During policy registration, an administrator inputs governance policies specifying prohibited and allowed informational categories. The Policy Integrator first scans the existing registry using dense semantic similarity to prevent duplicate or conflicting definitions. Next, the Rubric Generator synthesizes six policy-specific binary evaluation questions, while the Query Generation Engine (QGE) iteratively generates synthetic positive and negative prompts conditioned on department roles, job ranks, and interaction modalities. The Quality Assurance Engine (QAE) evaluates these queries against the generated rubrics and prunes low-quality samples via threshold filtering. The approved queries are concatenated with policy descriptions, embedded, and indexed into a Query Pool powered by SQLite and FAISS. In online inspection, incoming user prompts are mapped into the shared embedding space to retrieve the top-5 nearest neighbors via inner-product search, deriving a robust binary filtering decision and policy attribution through majority voting.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Corporate Governance Policies<br/>Prohibited & Allowed Categories"] --> PI["Policy Integrator<br/>Semantic Overlap & Redundancy Check"]
PI --> RG["Rubric Generator<br/>6-Aspect Binary Evaluation Rubrics"]
PI --> QGE["Query Generation Engine<br/>Context-Aware Synthetic Queries"]
RG --> QAE["Quality Assurance Engine<br/>Rubric Scoring & Threshold Filtering"]
QGE --> QAE
QAE --> QP["Query Pool Construction<br/>Natural Text + FAISS Dense Index"]
UserQ["Incoming User Prompt"] --> EM["Embedding Model<br/>Map to Shared Latent Space"]
QP -.-> PM["Prediction Module<br/>Top-5 Cosine Retrieval & Voting"]
EM --> PM
PM --> Out["Policy Enforcement Decision<br/>Block/Allow + Violated Policy ID"]
FB["Admin Log Review / Feedback"] -.->|Zero-Retraining Hot Patch| QP
Key Designs¶
1. Policy Integrator and Rubric Generator: Resolving Policy Conflict and Establishing Objective Grounding As corporate governance taxonomies scale across departments, newly introduced rules often overlap or conflict semantically with established guidelines. The Policy Integrator utilizes a compact Qwen3-Embedding-0.6B model to embed policy names and definitions, retrieving the top-\(k\) nearest candidate policies to prompt administrator review before committing changes. Once confirmed, the Rubric Generator (powered by Qwen3-32B) synthesizes six mutually non-overlapping binary interrogative rubrics tailored to that specific category (e.g., "Does the prompt specify concrete experimental parameters or variable configurations?", "Does the query disclose date-stamped execution progress?"). Moving away from vague top-level definitions, these structured rubrics establish explicit, fine-grained semantic boundaries that directly drive subsequent data validation.
2. Context-Aware Query Generation Engine: Overcoming the Corporate Data Scarcity Cold-Start Real enterprise prompt logs containing sensitive leaks are typically confidential and legally restricted, precluding large-scale training data curation. The Query Generation Engine solves this data cold-start by synthesizing realistic positive queries (containing restricted data) and negative queries (semantically adjacent without sensitive exposure). The generation engine systematically permutes operational variables across organizational departments (R&D, Legal, Finance, HR), requester seniority levels (junior, manager, executive), and interaction formats (embedded direct queries vs. attached file summaries). Empirical ablation shows that supplying policy titles alone without noisy, bloated descriptions maximizes synthetic query fidelity, yielding the highest downstream validation pass rate of 64.4%.
3. Quality Assurance Engine: Threshold-Based Multi-Rubric Filtering against Semantic Drift Unconstrained synthetic data generation from LLMs inevitably introduces hallucinated terms, incomplete prompts, or off-topic artifacts that could distort retrieval density if indexed directly. The Quality Assurance Engine evaluates each generated prompt against the six policy-tailored rubrics, issuing a discrete binary score per criterion to yield a cumulative quality score from 0 to 6. Queries falling below an empirically determined threshold of 1 are discarded. Ablation studies confirm that threshold filtering under dynamically generated rubrics significantly outperforms hardcoded static rubrics or single-rubric baselines, pruning spurious boundary cases while preserving dense, authentic semantic coverage in the vector space.
4. Dense Query Pool and Dynamic Prediction Module: Microsecond Classification and Instant Adaptation Validated synthetic queries are concatenated with their originating policy descriptions and embedded via Qwen3-Embedding-0.6B into dense normalized representations \(\mathbf{v} = \phi(q)\), stored in SQLite and indexed with FAISS IndexFlatIP. For an arbitrary user query \(q\), the prediction module computes its embedding and retrieves the top-5 nearest entries \(\mathcal{R}_5\) using inner-product cosine similarity: $\(\hat{y} = \arg\max_{y \in \mathcal{Y}} \sum_{(q_i, \mathbf{v}_i, y_i) \in \mathcal{R}_5} \mathbf{1}[y_i = y]\)$ The final binary decision (Prohibited vs. Allowed) is derived via mapping function \(g(\hat{y})\). Crucially, this non-parametric architecture permits zero-retraining adaptation: when an enterprise reclassifies a policy from Allowed to Prohibited, or when human administrators correct an edge-case misclassification, the update requires only in-place database label modification and an instantaneous FAISS index rebuild, avoiding catastrophic forgetting and costly GPU-hours.
Benchmark & Evaluation Protocol¶
To benchmark enterprise query filtering, the authors construct two comprehensive datasets: - Core Set (750 Queries): Features 5 balanced subsets of 150 queries each: Prohibit (PR), Hard Allowed (HA), Easy Allowed (EA), Hard Empty (HE), and Easy Empty (EE), establishing a realistic 1:4 prohibited-to-allowed ratio. Real de-identified corporate queries inform PR, HA, and EA distributions, which are augmented via GPT-5/5.2 and rigorously human-annotated. EE controls are sampled from cleaned WildChat conversations. Maximum Mean Discrepancy (MMD) analysis confirms distributional separation between the Query Pool and Core Set on par with MS MARCO query-passage distances, verifying that evaluation is strictly non-self-referential. - Large Set (7,482 Queries): A 10x expansion preserving exact policy boundaries, utilized for dynamic drift evaluations without sample reuse. - 24-Round Dynamic Evolution Protocol: Spans four consecutive 6-round operational eras: Initial Deployment (R1โR6, 24 categories), Unseen Policy Addition (R7โR12, +6 holdout categories), Allowed-to-Prohibited Shift (R13โR18, 4 categories tightened), and Prohibited-to-Allowed Shift (R19โR24, 4 categories relaxed), reflecting real-world policy drift.
Key Experimental Results¶
Main Results¶
In static policy evaluation over 660 matching Core Set queries, Q-CAT achieves performance on par with massive reasoning LLMs while dominating prompt-based and non-thinking baselines. With administrator feedback incorporated, Q-CAT attains superior precision and overall accuracy:
| Model | Operating Mode | F1 (%, โ) | Recall (%, โ) | Precision (%, โ) | Policy Class. Acc. CA (%, โ) |
|---|---|---|---|---|---|
| Q-CAT (Ours) | Top-5 Retrieval (Static) | 77.7 | 90.0 | 68.4 | 86.7 |
| Q-CAT (Ours) | Top-5 Retrieval (+Feedback) | 98.3 | 96.7 | 100.0 | 93.3 |
| gpt-5-mini | Thinking | 71.0 | 99.2 | 55.4 | 99.2 |
| gpt-5.2 | Thinking | 76.5 | 99.2 | 62.3 | 99.2 |
| gpt-5.2 | Non-thinking | 81.7 | 96.7 | 70.7 | 99.2 |
| Qwen3-8B | Thinking | 81.7 | 78.3 | 85.5 | 64.2 |
| Qwen3-8B | Non-thinking | 53.0 | 44.2 | 66.3 | 71.7 |
| Qwen3-32B | Thinking | 80.5 | 87.5 | 74.5 | 76.7 |
| Qwen3-32B | Non-thinking | 64.7 | 84.2 | 52.6 | 91.7 |
| Qwen3-8B (FT) | LoRA Fine-Tuned | 93.4 | 88.3 | 99.1 | 85.0 |
Note: While fine-tuned Qwen3-8B achieves high initial static performance on in-distribution data, its performance degrades severely under dynamic policy shifts below.
Ablation Study & Empirical Analysis¶
1. Dynamic Policy Evolution on the Large Set (Zero Query Reuse Across 24 Rounds) Under strict sampling without replacement, Q-CAT is the only evaluated system demonstrating continuous positive adaptation, whereas parametric fine-tuning suffers acute performance degradation:
| Operational Era | Real-World Policy Scenario | Q-CAT (Ours) Mean F1 (%) | Qwen3-8B FT Mean F1 (%) | Margin (ฮ) |
|---|---|---|---|---|
| Era 0 (Rounds 1โ6) | Initial Deployment (24 Baseline Policies) | 85.99 | 94.63 | -8.64 |
| Era 1 (Rounds 7โ12) | Unseen Policy Addition (6 New Policies Added) | 90.43 | 85.18 | +5.25 |
| Era 2 (Rounds 13โ18) | Allowed โ Prohibited (Regulatory Tightening) | 93.98 | 69.86 | +24.12 |
| Era 3 (Rounds 19โ24) | Prohibited โ Allowed (Policy Deregulation) | 94.99 | 65.01 | +29.98 |
2. Component Latency Breakdown vs. LLM Decoding Overheads (Core Set Averages) Q-CAT delivers deterministic, millisecond-scale latency in static serving, running hundreds of times faster than reasoning-capable LLM baselines:
| System / Step | Evaluation Setting & Hardware | Avg Latency (ms) | Std Dev (ms) | Speedup vs. LLM |
|---|---|---|---|---|
| Q-CAT: End-to-End Static Serving | AMD EPYC 7763 CPU + 1ร RTX 6000 Ada | 36 | 3 | Baseline (1ร) |
| Q-CAT: Dense Embedding (\(\phi\)) | Qwen3-Embedding-0.6B | 34 | 3 | - |
| Q-CAT: FAISS Search | IndexFlatIP (\(k=5\)) | 3 | 1 | - |
| Q-CAT: End-to-End Dynamic (Update Point) | Includes DB fetching (577ms) & index build (320ms) | 933 | 80 | - |
| Qwen3-8B (SGLang) | Non-thinking Mode (2ร RTX 6000 Ada) | 358 | 851 | 0.10ร |
| Qwen3-8B (SGLang) | Thinking Mode (2ร RTX 6000 Ada) | 8,203 | 6,943 | 228ร Slower |
| Qwen3-32B (SGLang) | Non-thinking Mode (2ร RTX 6000 Ada) | 1,090 | 2,958 | 0.03ร |
| Qwen3-32B (SGLang) | Thinking Mode (2ร RTX 6000 Ada) | 21,067 | 16,555 | 585ร Slower |
3. Quality Assurance Engine Rubric and Threshold Ablation (F1 %) | QAE Threshold | Core Set: Generated | Core Set: Default | Core Set: Single | Large Set: Generated | Large Set: Default | Large Set: Single | |---|---|---|---|---|---|---| | Threshold 0 (No QAE) | 71.5 | 71.5 | 71.5 | 73.6 | 73.6 | 73.6 | | Threshold 1 (Optimal) | 73.4 | 71.5 | 72.8 | 75.0 | 73.6 | 74.7 | | Threshold 2 | 70.8 | 71.5 | - | 73.5 | 73.6 | - | | Threshold 4 | 70.2 | 71.3 | - | 73.7 | 73.5 | - | | Threshold 6 (Over-filtered) | 36.6 | 71.4 | - | 42.3 | 71.5 | - |
Key Findings¶
- Elimination of False Negative Spikes in Dynamic Drift: While parametric fine-tuning incurs severe false-negative bursts whenever policies flip between allowed and prohibited (degrading from 94.63% to 65.01% F1), Q-CAT propagates label changes instantaneously to the Query Pool, steadily driving F1 upward to 94.99%.
- Predictable Latency Profile for Enterprise Gateways: Reasoning models display substantial latency variance (e.g., Qwen3-32B-Thinking displays an average of 21,067 ms with a standard deviation of 16,555 ms), creating unmanageable request queues; Q-CAT ensures static inference within 36 ms (\(\pm 3\) ms).
- Error Typology Analysis: Residual false negatives stem primarily from lexical overlaps with adjacent policies (e.g., pilot project milestones resembling project completion reports) or domain-specific jargon; incorporating such edge cases directly into the Query Pool resolves recurring misclassifications without retraining.
Highlights & Insights¶
- Decoupling Validation from Generation via Dynamic Rubrics: Demonstrating that using simple policy names for generation paired with 6-aspect orthogonal validation rubrics achieves an optimal 64.4% pass rate, providing a scalable recipe for synthetic safety data synthesis.
- Architectural Shift from Model Weights to Semantic Memory: Converting the problem of policy enforcement into non-parametric spatial density matching completely circumvents the brittleness and catastrophic forgetting inherent to parameter-tuned safety guardrails.
- Cross-Domain and Multilingual Robustness: Validating that feedback-driven query pool expansion reliably generalizes across public English intent classification (CLINC150, reaching 0.971 F1 in Era 3) and noisy multilingual privacy moderation across 6 languages in WildChat.
Limitations & Future Work¶
- Reliance on Underlying Embedding Representational Capacity: Performance depends on the embedding model placing semantically related information queries into adjacent clusters; highly obscure corporate acronyms or adversarial obfuscations may require domain-adapted embedding backbones.
- Multi-Intent and Compound Query Parsing: The current system assumes queries target a single primary governance topic; complex user prompts interleaving multiple conflicting policies require future exploration into chunk-level intent decomposition.
- Residual False Negatives in High-Stakes Environments: While administrator feedback mitigates repeated errors, zero false negatives cannot be mathematically guaranteed; high-security deployments should combine Q-CAT with complementary output-side filters.
Related Work & Insights¶
- vs General Safety Guardrails (LlamaGuard / QwenGuard): General safety models focus on static, universal risks (hate, violence) and cannot accommodate shifting corporate confidentiality policies without costly retraining; Q-CAT targets dynamic enterprise compliance with training-free adaptability.
- vs Dynamic Agent Compliance (PolicyGuard / DynaGuard): While prior work inspects agent action trajectories or monitors output responses for leaks, Q-CAT acts directly at the input gateway level, filtering high-throughput user prompts with millisecond-level responsiveness.
Rating¶
- Novelty: โญโญโญโญ [Reimagines dynamic enterprise policy compliance as a non-parametric retrieval and rubric-audited synthetic memory system]
- Experimental Thoroughness: โญโญโญโญโญ [Evaluated across Core (750) and Large (7,482) sets with a 24-round dynamic protocol, latency profiling, and multilingual benchmarks]
- Writing Quality: โญโญโญโญโญ [Clear formal specifications, well-structured ablation studies, and comprehensive error analysis]
- Value: โญโญโญโญโญ [Addresses an immediate, critical operational bottleneck in enterprise LLM deployment with exceptional computational efficiency]