Skip to content

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/danielle34/cavewoman
Area: LLM Reasoning
Keywords: Linguistic Compression, Inference Cost, Bidirectional NLI, Prompt Compression, Output Constraint

TL;DR

CAVEWOMAN introduces a two-channel evaluation protocol across 9 LLMs and 5 benchmarks, demonstrating that output grammatical compression reduces realized API dollar cost by up to 3× without degrading task accuracy, yet over half of correct non-reasoning responses no longer semantically entail the model's unconstrained output, whereas input prompt compression paradoxically inflates net cost by up to 96%.

Background & Motivation

As large language models are increasingly integrated into production workflows, managing operational inference expenditure has become a critical operational priority. In commercial pricing models, output tokens are priced 4–6× higher than input tokens, prompting widespread interest in token-saving heuristics such as colloquial "caveman talk" (e.g., dropping functional words and grammatical scaffolding to save tokens). However, existing investigations into prompt compression and output length regulation remain siloed, and both paradigms are almost exclusively evaluated on a single task accuracy metric against reduced token budgets, completely ignoring the fundamental price asymmetry between channels and the semantic fidelity of intermediate reasoning traces.

Evaluating compression solely through the lens of answer accuracy overlooks two major failure modes. First, standard benchmarks treat any reduction in prompt tokens as cost savings, failing to recognize that when an LLM is presented with telegraphic, fragmented inputs, it often compensates by generating significantly longer, verbose explanations, which inflates the total net API dollar cost. Second, binary correctness evaluation cannot distinguish whether a compressed response arrives at the right answer via the model's original underlying reasoning or via omitted steps, altered premises, or spurious shortcuts. In applications where chain-of-thought traces are audited—such as safety compliance logs, explainable agents, and verifiable reasoning pipelines—preserving the final answer while mutating the underlying reasoning introduces severe unmonitored epistemic drift.

To resolve both blind spots simultaneously, this work establishes a rigorous unified framework evaluating prompt and response compression on identical items. Core idea: construct CAVEWOMAN, a two-channel evaluation protocol that jointly scores every question-answer generation on task accuracy with audited extraction, realized dollar cost at commercial API rates, and semantic agreement with the model's own unconstrained generation via bidirectional NLI across five nested grammatical reduction levels.

Method

Overall Architecture

The CAVEWOMAN framework implements a rigorous dual-channel controlled testing environment. The evaluation holds the underlying language model \(\mathcal{M}\) and the question \(x\) fixed while evaluating five nested grammatical reduction levels (\(\text{L}0\) through \(\text{L}4\)) across two orthogonal operational conditions.

Under Condition A (input compression), the system prompt remains neutral and identical across all levels, while the question \(x\) is filtered via a deterministic part-of-speech (POS) pipeline \(\phi_\ell(x)\) to strip function words and closed-class syntax. Under Condition B (output compression), the question \(x\) remains natural and unconstrained, while the system prompt is substituted with level-specific constraints instructing the model to respond in the corresponding grammatical register. Every resulting response is evaluated along three joint axes: task accuracy with LLM-judge recovery, realized dollar expenditure using published per-token pricing schedules, and semantic equivalence evaluated via bidirectional natural language inference (NLI) against unconstrained reference baselines, systematically partitioned into a four-way outcome matrix (C1–C4).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Input Question x & System Prompt"] --> B["Five Nested Linguistic Reduction Levels<br/>L0 Baseline / L1 Telegraphic / L2 Keyword / L3 Noun Skeleton / L4 Hard Budget"]
    B --> C["Dual-Channel Compression Architecture<br/>Condition A: Input POS Filtering vs Condition B: Output Constraint Prompting"]
    C --> D["Three-Dimensional Joint Audit Protocol<br/>Judge-Corrected Accuracy + Realized API Spend + Bidirectional NLI Verification"]
    D --> E["Four-Way Orthogonal Outcome Partition<br/>C1 Correct & Entailed / C2 Correct & Changed / C3 Wrong & Entailed / C4 Wrong & Changed"]

Key Designs

1. Five Nested Linguistic Reduction Levels: Fine-grained grammatical pruning avoiding single-step distortion

To analyze the performance degradation curve from gentle telegraphic abbreviation to radical lexical pruning, the framework constructs a nested family of POS filters \(\phi_\ell\) using spaCy 3.8 and Penn Treebank tags: - L0 (Unconstrained Baseline): Keeps all words unaltered (\(\phi_0(x) = x\)). - L1 (Telegraphic): Eliminates closed-class grammatical tags including determiners (DT), prepositions (IN), conjunctions (CC), particles (RP), infinitival to (TO), and modals (MD), retaining only content words (nouns, main verbs, adjectives, numbers). - L2 (Keyword-only): Strictly retains only nouns (\(\text{NN}^*\)), main verbs (\(\text{VB}^*\)), and cardinal numbers (\(\text{CD}\)). - L3 (Noun-phrase Skeleton): Removes all verbs, restricting output strictly to nominal phrases, compound nouns, and numbers. - L4 (Hard 15-token Budget): Applies L3 filtering and truncates to the first 15 tokens, with response generation strictly capped at 20 max tokens.

Because \(\phi_\ell(x)\) is a strict subsequence of \(\phi_{\ell-1}(x)\) for all \(\ell \ge 1\), performance drops between successive levels are isolated to the specific word categories pruned at that exact tier.

2. Dual-Channel Compression Architecture: Decoupling input prompt filtering from output response prompting

To resolve the conflation between compressing context versus compressing generation, the protocol tests both channels on the exact same benchmark items. In Condition A (input compression), the model receives \(\phi_\ell(x)\) under an unvarying neutral prompt to measure whether LLMs genuinely require grammatical cohesion to parse tasks. In Condition B (output compression), the model receives the full uncorrupted question, and the system prompt provides register instructions, task-neutral demonstrations, and an explicit Answer: <answer> convention. Furthermore, Condition B contrasts the standard step-by-step L0 prompt with an L0-neutral prompt that removes the "Reason step by step" instruction without altering the token budget, disentangling the effect of removing chain-of-thought elicitation from linguistic compression itself.

3. Three-Dimensional Joint Audit Protocol: Linking realized dollar cost to bidirectional semantic entailment

Rather than relying on ungrounded token counters and naive regular expressions, CAVEWOMAN establishes a multi-layered verification pipeline. Realized financial cost is calculated per item from exact token consumption and public API pricing: \(C = n_{\text{in}} \cdot p_{\text{in}} + n_{\text{out}} \cdot p_{\text{out}}\). Accuracy is audited via a three-tier recovery ladder: regex extraction is supplemented by an LLM recovery judge (Qwen3.5-9B), cross-validated against a second judge (Qwen2.5-32B) at 90.9% agreement and audited across four human annotators on stratified samples. Semantic agreement is measured using cross-encoder/nli-deberta-v3-base evaluating bidirectional entailment:

\[ \text{Entailment}(y_{\text{comp}}, y_{\text{ref}}) \iff (y_{\text{comp}} \models y_{\text{ref}}) \land (y_{\text{ref}} \models y_{\text{comp}}) \]

This ensures that compressed responses are only deemed faithful if they say what the unconstrained model said, cross-checked against 11 additional semantic metrics including ROUGE, BLEURT, STS, and LLM pair judges.

4. Four-Way Orthogonal Outcome Partition: Isolating correct answers from altered reasoning paths

The protocol partitions all model responses into a \(2 \times 2\) matrix across task correctness and reference entailment: - C1 (Correct + Entails): Preserves both the correct answer and the original underlying reasoning. - C2 (Correct + Non-Entail): Correct answer, but the reasoning is either completely omitted (C2 by omission) or traverses an entirely different derivation pathway (C2 by changed route). - C3 (Wrong + Entails): The model replicates its original unconstrained error. - C4 (Wrong + Non-Entail): Newly induced reasoning errors.

Standard single-metric accuracy merges C1 and C2 into a monolithic success count. CAVEWOMAN reveals that C2 constitutes the dominant share of correct answers under output compression for non-reasoning models, surfacing hidden unfaithfulness.

Benchmark & Evaluation Protocol

The main evaluation suite covers 9 diverse LLMs: six non-reasoning models (GPT-4o, GPT-5.4, Claude Haiku 4.5, Claude Sonnet 4.6, Qwen2.5-VL-7B, Gemma-4-E4B), two reasoning models that produce visible thinking traces sharing the token budget (DeepSeek-R1-Distill-Qwen-7B, Qwen3.5-9B tested with thinking on and off), and Kimi-K2.6. Models are evaluated across 11,465 items spanning five short-answer benchmarks: GSM8K (1,319 math questions), BoolQ (3,270 yes/no passage items), ARC-Easy (2,376 science reasoning items), CommonsenseQA (1,221 commonsense items), and MMLU-STEM (3,279 questions across 20 STEM disciplines). The protocol is extended to multilingual grade-school math via MGSM (250 problems across 10 languages spanning 7 scripts) and long-form document summarization on XSum and CNN/DailyMail.

Key Experimental Results

Main Results

The table below summarizes the financial cost shifts, accuracy deltas, and C2 (correct but non-entailing) shares for the four commercial API models under L1 telegraphic compression across the five benchmarks.

Model Input L1 Cost Change (%) Input L1 Acc. Delta (pp) Output L1 Cost Change (%) Output L1 Acc. Delta (pp) Output L1 C2 Share (%)
GPT-4o -2.6% -5.8 -57.0% (2.4× saving) +3.1 55.0%
Claude Sonnet 4.6 +5.1% -7.2 -41.0% (1.7× saving) -1.2 57.0%
Claude Haiku 4.5 +3.1% -6.4 -31.0% (1.4× saving) -0.8 42.0%
GPT-5.4 +18.4% -8.1 -11.0% (1.1× saving) -0.5 56.0%

Under input compression, net API spend rises on 3 of the 4 models (spiking by +96% for GPT-5.4 on CommonsenseQA) while accuracy degrades across all models. Conversely, under output compression, all models achieve substantial cost reductions (averaging 1.7× and up to 3.1× on ARC-Easy) while maintaining or improving task accuracy. However, C2 accounts for 42%–57% of all responses across the board.

Ablation Study

The table below provides controls and sensitivity analyses for Finding 2 (prevalence of C2 correct-but-changed responses) under L1 output compression across the six non-reasoning models.

Control / Ablation Setting Evaluation Metric / Benchmark Empirical Result Mechanistic Finding
Baseline Reference (L0-neutral) C2 share across 6 non-reasoning models 50.1% Main text baseline; confirms C2 is not an artifact of removing "step by step"
Baseline Reference (Step-by-step L0) C2 share across 6 non-reasoning models 51.9% Matches L0-neutral pattern; C2 remains the largest outcome cell
Length Control Audit L0 cut to L1 length and re-scored 80.4% Non-Entail Proves the gap is genuine semantic drift rather than an artifact of brevity
Natural Sampling Floor Two T=0.7 L0 draws mutual entailment 45.5% Entailment L1 entailment falls below the natural resampling floor on 15/20 model-task cells
LLM Pair Judge (Qwen3.5-9B) Semantic reasoning consistency 70% AGREE, 15% Changed route, 10% Unclear Even when ignoring style and register, 1 in 4 correct responses changes reasoning
Register Compliance Audit Response tokens in 6 function POS classes 27.6% (L0) \(\to\) 5.5% (L1) Confirms models strictly adhere to telegraphic constraints across tasks

Key Findings

  • Severe cost asymmetry across channels: Stripping function words from prompts saves 15–16 input tokens, but causes models to generate longer, compensatory explanations; because output tokens cost 4–6× more, input compression increases total dollar costs by an average of +12% across API models. In contrast, output compression directly curtails expensive output tokens, achieving 1.7× average and up to 3× realized financial savings.
  • Widespread dissociation between accuracy and semantic fidelity: Over half (50.1%) of all responses on non-reasoning models are C2 outcomes (correct but failing bidirectional NLI with the unconstrained generation). Qualitative audit reveals two mechanisms: C2 by omission (dropping reasoning entirely and returning a naked answer) and C2 by changed route (reordering intermediate computational steps or invoking disparate factual dependencies).
  • Compressibility does not correlate with unconstrained capability: Baseline unconstrained accuracy fails to predict accuracy retention under compression (Spearman \(\rho = 0.38, p = 0.35\)). While within-family scaling (Qwen2.5 3B to 32B) reduces accuracy drop on GSM8K from 46 points to 4 points, cross-family comparison shows Gemma-4-E4B gaining +11.4 pp under compression solely because shortening responses prevents it from running into decoding limits.
  • Reasoning models suffer token-budget starvation: Models allocating a single budget to hidden or visible reasoning traces (DeepSeek-R1, Qwen3.5-9B, Kimi-K2.6) collapse under output token caps; DeepSeek-R1 drops to near-zero accuracy at L4 because thinking tokens consume the budget before answer generation begins. Disabling thinking mode on Qwen3.5-9B reverses this collapse, lifting L1 accuracy from 77.6% to 88.3%.

Highlights & Insights

  • Debunking the "Caveman Prompt" cost-saving myth: By tracking actual billed dollar expenses rather than raw prompt token counts, the paper demonstrates that input-side grammatical compression frequently backfires economically due to output verbosity compensation.
  • Operationalizing the C1–C4 evaluation matrix: Distinguishing C1 (faithful correctness) from C2 (unfaithful or omitted correctness) provides an indispensable auditing framework for mission-critical deployments where reasoning auditability is as vital as the final prediction.
  • Register breakdown in long-form generation: The long-form summarization probe reveals that while L1–L3 register prompts compress short answers, they fail to control length in multi-sentence summarization (often triggering list-like enumerations); only explicit hard token budgets (L4) reliably enforce conciseness.

Limitations & Future Work

  • High sensitivity of NLI models to syntactic variation: Cross-encoder NLI judges are highly sensitive to surface syntactic phrasing, requiring reliance on L1 for formal bounds and necessitating LLM pair judges at higher compression tiers (L2–L3).
  • Absence of human reader acceptability studies: The investigation does not measure whether compressed, telegraphic outputs remain acceptable, trustworthy, and easily interpretable to end-users in real-world human-in-the-loop applications.
  • Rule-based POS filtering constraints: Input filtering relies on deterministic Penn Treebank POS taggers, which may not capture informal conversational abbreviations, typos, and natural colloquial elisions found in wild user prompts.
  • vs LLMLingua & Prompt Compression: Prior works like LLMLingua and LLMLingua-2 focus on pruning prompt tokens via small SLM perplexity or distilled classification. CAVEWOMAN shows that for single-turn reasoning, prompt pruning often induces catastrophic ambiguity (e.g., LLMLingua-2 pruning MCQ option labels, dropping ARC accuracy to 6%), proving that constraining outputs is far safer and more cost-effective.
  • vs Decoding-time Output Length Control (Hansel, Length Controls): Existing output-side methods manipulate positional encodings or token countdown counters during decoding. CAVEWOMAN demonstrates that simple zero-shot prompting using linguistic registers achieves comparable token reductions without model fine-tuning, while establishing the first systematic audit of whether compressed reasoning remains faithful.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first rigorous dual-channel economic and semantic evaluation of linguistic compression, introducing the C1–C4 outcome decomposition.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 9 models, 11,465 short-answer items, 10-language MGSM benchmarks, long-form summarization, and human audit validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, precise empirical formulations, and incisive analysis of cost asymmetries and reasoning drift.
  • Value: ⭐⭐⭐⭐⭐ High direct utility for enterprise LLM cost optimization, API pricing strategy, and faithful reasoning verification.