Skip to content

CAVEWOMAN: How Large Language Models Behave Under Extreme Context Degradation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/danielle34/cavewoman
Area: LLM Agent
Keywords: Linguistic Input and Output Compression, Cost Asymmetry, Semantic Fidelity, Bidirectional NLI Entailment, Context Degradation

TL;DR

CAVEWOMAN introduces a two-channel evaluation protocol spanning nested grammatical reduction levels, revealing that compressing prompts increases net inference costs by up to 96% while degrading accuracy, whereas compressing model outputs slashes dollar costs by up to 3ร— without losing accuracy, although over 50% of correct responses become semantically disconnected from unconstrained reasoning.

Background & Motivation

In production deployments of large language models, the colloquial heuristic "Talk short. Drop grammar. Save token." is widely promoted as a straightforward recipe for curtailing API expenses. With commercial pricing models pricing output generation 4ร— to 6ร— higher than input consumption, developers and practitioners increasingly implement ad-hoc prompt pruning, heuristic POS-dropping, or constrained generation prompts. However, research into prompt compression and output length reduction has evolved in disconnected silos, evaluating either input or output in isolation without examining the coupled economic and behavioral feedback loops between both channels.

Crucially, existing compression benchmarks evaluate degradation almost exclusively through downstream task accuracy on a reduced token budget. This paradigm suffers from two fatal blind spots. First, treating shorter input prompts as an automatic cost saving ignores compensatory verbosity: when deprived of contextual syntactic scaffolding, models frequently generate longer, more convoluted responses, causing the net dollar spend to surge due to the higher per-token price of generation. Second, a binary correct-or-wrong score cannot determine whether a compressed response preserves the semantic reasoning trajectory the model would have produced in an unconstrained setting. In real-world agent workflows, audit trails, and automated decision logs that rely on intermediate reasoning traces, this hidden decoupling poses severe safety and reliability risks.

To address these core tensions, a systematic evaluation methodology is needed to measure real-dollar expenditures alongside granular semantic fidelity across both input and output degradation vectors. Core idea: propose CAVEWOMAN, a two-channel evaluation protocol operating over a nested five-level grammatical reduction ladder, which jointly audits per-item realized dollar costs, judge-calibrated task accuracy, and bidirectional NLI semantic entailment across a four-quadrant outcome matrix to expose channel cost asymmetries and silent reasoning collapse under extreme context degradation.

Method

Overall Architecture

The CAVEWOMAN framework tests a fixed model and query distribution across two dual operational modes: Condition A (input compression) and Condition B (output compression). The end-to-end evaluation pipeline encompasses four sequential phases: POS-driven nested grammatical reduction, dual-channel controlled inference deployment, progressive three-tier answer calibration, and cross-channel realized dollar auditing coupled with bidirectional semantic entailment scoring.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Question-Answer Benchmarks & System Setup"] --> B["Nested Grammatical Reduction Ladder<br/>L0 Baseline down to L4 15-Token Budget"]
    B --> C["Dual-Channel Controlled Inference<br/>Condition A Input Filter vs Condition B Output Constraint"]
    C --> D["Three-Tier Progressive Answer Calibration<br/>Regex Extraction + LLM Judge + Blind Human Audit"]
    D --> E["Bidirectional NLI & Realized Dollar Auditing<br/>Four-Quadrant Outcome Classification (C1-C4) & Billed Cost Accounting"]

Key Designs

1. Nested Grammatical Reduction Ladder: Eliminating Cross-Level Distribution Shifts Prior prompt compression studies rely on ad-hoc heuristics or non-monotonic filters that introduce confounding distribution shifts. CAVEWOMAN leverages spaCy's Penn Treebank tagger (en_core_web_sm) to establish a mathematically nested sequence of reduction operators \(\phi_0\) through \(\phi_4\), where each higher level \(\phi_\ell(x)\) forms a strict subsequence of \(\phi_{\ell-1}(x)\): - \(L_0\) (Unconstrained Baseline): Preserves all words in natural language. - \(L_1\) (Telegraphic): Deterministically prunes all closed-class grammatical tags \(\text{g}(w) \in \{\text{DT, IN, CC, RP, TO, MD}\}\) (articles, prepositions, conjunctions, particles, infinitival to, and modals). - \(L_2\) (Keyword-Only): Retains solely nouns, main verbs, and cardinal numbers \(\text{NN}^* \cup \text{VB}^* \cup \{\text{CD}\}\). - \(L_3\) (Noun-Phrase Skeleton): Completely removes all verbs, retaining only nominal fragments and numbers \(\text{NN}^* \cup \{\text{CD}\}\). - \(L_4\) (15-Token Hard Budget): Truncates the \(L_3\) noun-phrase skeleton to its initial 15 tokens, with decoding strictly capped at 20 tokens. This strict nesting guarantees that performance and behavioral shifts across successive levels stem directly from the syntactic categories eliminated at that specific threshold.

2. Dual-Channel Controlled Inference: Isolating Causal Degradation Vectors To cleanly disentangle task instructions from syntactic constraints, CAVEWOMAN deploys symmetric channels. In Condition A (input compression), the system prompt remains permanently fixed to a neutral instruction while the input question is filtered via \(\phi_\ell(x)\), quantifying model robustness to degraded context. In Condition B (output compression), the input question is supplied unmodified in full natural language, and the system prompt is replaced by a level-specific constraint instructing the model to generate its reasoning and final answer using the exact grammar of \(\phi_\ell\). Crucially, Condition B benchmarks against both a standard step-by-step reasoning baseline (\(L_0\)) and a reasoning-instruction control (\(L_0\text{-neutral}\)) that removes the "reason step by step" directive, cleanly isolating grammatical compression from the omission of chain-of-thought prompting.

3. Three-Tier Progressive Answer Calibration: Rectifying Parsing Bias Under compressed or neutral prompts, models frequently omit explicit answer flags, causing naive regular expressions to fail catastrophically (e.g., regex parses only 4% of unconstrained Sonnet responses on CommonsenseQA), which artificially inflates compressed performance. CAVEWOMAN resolves this through a three-stage calibration ladder: initial regex extraction with a 0.01 numeric tolerance; fallback adjudication by an LLM judge (Qwen3.5-9B) assessing 90,695 edge-case responses against ground truth; and stratified blind human verification across 180 disputed cases per annotator (achieving 71.9% inter-annotator agreement). This ensures reported accuracies reflect authentic task success rather than extractor brittleness.

4. Bidirectional NLI & Four-Quadrant Outcome Classification: Auditing Silent Semantic Drift To measure whether compressed outputs maintain fidelity to unconstrained generations, the protocol computes bidirectional natural language entailment using cross-encoder/nli-deberta-v3-base (validated against 11 additional semantic metrics including MiniCheck, BART-MNLI, and STS). Each response is categorized into a \(2 \times 2\) state space: \(C_1\) (Correct + Entails \(L_0\)), \(C_2\) (Correct + Non-Entailing / Changed), \(C_3\) (Incorrect + Entails \(L_0\)), and \(C_4\) (Incorrect + Non-Entailing). The \(C_2\) cell isolates the critical failure mode where a model arrives at the correct final symbol but completely abandons or distorts the intermediate evidentiary chain, providing an essential safety audit metric.

Benchmark & Evaluation Protocol

The core evaluation covers nine leading models representing diverse architectures: commercial API models (GPT-4o, GPT-5.4, Claude Haiku 4.5, Claude Sonnet 4.6), open-weight general and vision-language backbones (Qwen2.5-VL-7B, Qwen3.5-9B, Gemma-4-E4B), an open-weight reasoning distillation model (DeepSeek-R1-Distill-Qwen-7B), and Kimi-K2.6. Benchmarks comprise five distinct short-answer datasets totaling 11,465 items per level per condition: GSM8K (math reasoning), BoolQ (passage verification), ARC-Easy (scientific QA), CommonsenseQA (commonsense reasoning), and MMLU-STEM (20 STEM subjects). Cross-lingual generalization is tested via MGSM across ten languages spanning seven writing systems, while long-form generation is probed on XSum and CNN/DailyMail.

Key Experimental Results

Main Results: Cost Asymmetry Across Channels and Benchmarks

Pricing billed token streams against May 2026 published API rates reveals sharp, opposing economic trajectories between input and output compression channels.

Evaluation Channel Dimension / Benchmark L1 (Telegraphic) Cost L2 (Keyword-Only) Cost L3 (Noun Skeleton) Cost L4 (15-Token) Cost Primary Empirical Pattern
Input Compression (Condition A) GSM8K Arithmetic +5% -2% -14% -83% Prompt pruning triggers verbose output, raising net cost
Input Compression (Condition A) BoolQ Passage QA +6% +12% +11% -77% Context loss confuses model; low levels consistently lose money
Input Compression (Condition A) ARC-Easy Science QA +15% +22% +22% -71% Semantic degradation drives compensatory explanations
Input Compression (Condition A) CommonsenseQA +36% +82% +78% -53% Extreme cost surge, increasing dollar spend by up to +96%
Input Compression (Condition A) MMLU-STEM -3% -21% -34% -87% Minimal savings achieved only alongside steep accuracy collapse
Output Compression (Condition B) GSM8K Arithmetic -29% -34% -30% -76% Output constrained; substantial cost savings without accuracy loss
Output Compression (Condition B) BoolQ Passage QA -19% -21% -19% -59% Direct answer delivery steadily compresses dollar spend
Output Compression (Condition B) ARC-Easy Science QA -41% -44% -42% -78% Dramatic reduction in boilerplate text, saving up to 3.1ร—
Output Compression (Condition B) CommonsenseQA -38% -42% -42% -78% Elimination of conversational padding yields major savings
Output Compression (Condition B) MMLU-STEM -39% -47% -47% -83% Structured telegraphic outputs slash per-item API cost

At \(L_1\) input compression, dollar spend increases on three of the four API models (+18.4% on GPT-5.4, +5.1% on Claude Sonnet 4.6, +3.1% on Claude Haiku 4.5), while only GPT-4o barely decreases (-2.6%). Conversely, under \(L_1\) output compression, cost falls by 57% on GPT-4o (a 2.4ร— reduction), 41% on Sonnet 4.6 (1.7ร—), 31% on Haiku 4.5 (1.4ร—), and 11% on GPT-5.4.

Semantic Consistency Audit: Collapse of Behavioral Fidelity Under Output Compression

Examining response distribution across the four outcome quadrants under \(L_1\) output compression (evaluated against the unconstrained \(L_0\text{-neutral}\) reference).

Model Category / Model Name C1: Correct + Entails C2: Correct + Non-Entailing C3: Wrong + Entails C4: Wrong + Non-Entailing C2 Share of Correct Responses Behavioral Analysis & Failure Modes
GPT-4o 26% 56% 2% 16% 68.3% Highly compliant; drops reasoning trace to output bare answer
GPT-5.4 35% 42% 6% 17% 54.5% Hidden reasoning consumes tokens; visible answer heavily pruned
Claude Haiku 4.5 36% 46% 6% 12% 56.1% Shifts intermediate reasoning steps, bypassing explicit setup
Claude Sonnet 4.6 36% 48% 7% 9% 57.1% Rich rhetorical scaffolding eliminated; logic flow reconstituted
Qwen2.5-VL-7B 32% 55% 3% 10% 63.2% Lower grammatical compliance; steps omitted rather than condensed
Gemma-4-E4B 18% 53% 9% 20% 74.6% Unconstrained runs hit token limits; compression resolves cutoffs
DeepSeek-R1-Distill-7B 45% 18% 27% 10% 28.6% Visible thinking trace exhausted by budget; inverse collapse pattern
Qwen3.5-9B (Thinking On) 43% 20% 27% 10% 31.7% CoT reasoning occupies shared budget, causing premature truncation
Qwen3.5-9B (Thinking Off) 10% 56% 18% 16% 84.8% With thinking disabled, model mirrors non-reasoning drift exactly

Key Findings

  1. The Cost Asymmetry Scissors Effect: Stripping closed-class words from input prompts saves 15โ€“16 input tokens, but triggers compensatory verbosity in the generated answer. Because output tokens are priced 4โ€“6ร— higher, prompt compression increases total dollar costs in 90% of model-benchmark configurations (averaging +12%, peaking at +96% on GPT-5.4 for CommonsenseQA) while consistently degrading accuracy. Conversely, output compression cuts costs by 1.7ร— on average and up to 3.1ร— without sacrificing accuracy.
  2. Pervasive Semantic Decoupling (\(C_2\) Dominance): Across all six non-reasoning models, over half (50.1%) of generated outputs at \(L_1\) reside in the \(C_2\) quadrant (correct but non-entailing). When evaluated against step-by-step \(L_0\) truncated to \(L_1\) length, non-entailing correct answers rise to 80.4%, proving that brevity alone does not explain the loss of reasoning consistency.
  3. Reasoning Trace Truncation Dilemma: Models that allocate tokens to internal thinking traces (DeepSeek-R1, Qwen3.5 with thinking on, GPT-5.4) exhibit budget starvation under output constraints. When max token budgets shrink, thinking consumes the allocation before the final answer can be emitted, dropping accuracy to zero at \(L_4\). Disabling thinking on Qwen3.5 boosts \(L_1\) accuracy by 10.7 points, but pushes \(C_2\) non-entailment to 84.8%.
  4. Model Scale and Cross-Lingual Disparities: Within the Qwen2.5 family (0.5B to 32B), larger model scale mitigates the accuracy penalty under output compression (improving from a 15-point drop at 3B to a small gain at 32B), yet semantic entailment stagnates around 41%. On MGSM across ten languages, output savings are significantly blunted in morphologically rich or low-resource scripts (e.g., Telugu, Bengali) due to severe tokenization bloat.

Highlights & Insights

  • Economic Realism Over Token Counts: Demonstrates that optimizing raw token counts is an invalid proxy for inference efficiency; real-world cost optimization must account for the pricing ratio between prompt ingestion and generation channels.
  • Controlled Baseline Isolation: By introducing \(L_0\text{-neutral}\) alongside \(L_0\), the study disentangles the specific economic contribution of deleting "reason step by step" (which cuts GPT-4o costs by 46% with zero accuracy drop) from grammatical syntax reduction.
  • Diagnostic Four-Quadrant Metric: The \(C_1\)โ€“\(C_4\) taxonomy provides a rigorous methodology to distinguish whether models solve tasks via preserved logic or degraded heuristic guessing, setting a standard for auditability in enterprise AI.

Limitations & Future Work

  • Author-Acknowledged Limitations: The DeBERTa-based NLI judge is hyper-sensitive to phrasing variations at strict reduction levels (\(L_2\)โ€“\(L_3\)), and the answer calibration judge (Qwen3.5-9B) is one of the audited models, introducing potential self-evaluation biases.
  • Scope Constraints: Input filters apply deterministic POS rules rather than naturalistic human shorthand or colloquial typist degradation. Additionally, long-form summarization experiments were restricted to two models, where style constraints failed to prevent list-based token expansion without hard numerical caps.
  • Future Directions: Extending the dual-channel protocol to long-horizon multi-turn agent tool trajectories, and designing learned compressors that jointly optimize token budgets and semantic entailment constraints.
  • vs LLMLingua / LLMLingua-2: LLMLingua prunes prompt tokens using perplexity or distilled classification; in short-answer QA, it frequently purges single-letter multiple-choice labels (e.g., "A:", "B:"), collapsing ARC-Easy accuracy to 6%. CAVEWOMAN's POS-based filter preserves critical question anchors (all options survive in 99.7% of items).
  • vs TokenSkip / Hansel: Output length control frameworks enforce budget awareness during decoding but evaluate purely on final accuracy; CAVEWOMAN reveals that aggressive length control silently severs the causal link between reasoning and conclusions.
  • Actionable Takeaways: Eliminate input prompt compression in production pipelines where queries are short. Instead, enforce structured, telegraphic output schemas while using neutral system prompts, and refrain from utilizing compressed model logs for safety audits.

Rating

  • Novelty: โญโญโญโญโญ First systematic, dual-channel protocol auditing real API billing alongside semantic entailment under context degradation.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 9 models, 5 benchmarks, 10 languages, long-form summarization, 11,465 items per cell, and verified human auditing.
  • Writing Quality: โญโญโญโญโญ Exceptionally articulate narrative, precise mathematical definitions, clean empirical presentation, and compelling insights.
  • Value: โญโญโญโญโญ Highly actionable guidance for AI system architects, directly dispelling persistent cost-optimization myths in LLM deployments.