Skip to content

LLMs Learn Better In-Context from Rules than from Examples

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/tinlaboratory/rules-vs-examples
Area: LLM Reasoning
Keywords: in-context learning, rules vs examples, instruction tuning, complex reasoning, generalization bounds

TL;DR

By systematically developing five novel, complex reasoning tasks spanning games, arithmetic, and synthetic linguistic inference, this paper demonstrates that LLMs learn far more reliably from natural language rules than from input-output examples, while adding demonstrations to rules or scaling demonstration volume fails to produce consistent performance gains.

Background & Motivation

Large language models (LLMs) exhibit striking in-context learning (ICL) capabilities, acquiring new behaviors directly from prompt contexts without updating network parameters. In contemporary practice, communicating an intended task to an LLM proceeds along two primary pathways: describing the overarching rules and instructions in natural language (rule-based learning / instruction following), or providing a sequence of input-output demonstrations (example-based learning / few-shot prompting). Recent mechanistic interpretability work reveals that even when rules and demonstrations specify the exact same latent task, language models do not converge to a single unified function vector representation, but instead mobilize partially overlapping yet functionally distinct internal circuits. This observed mechanistic divergence is frequently cited to justify prompt engineering recipes that blindly stack rules and worked examples together.

However, the presence of distinct internal mechanisms does not inherently translate into additive performance advantages when combined in practice. Prior empirical explorations of prompt formatting have largely focused on settings where natural language instructions were inadvertently underspecified, or centered on simple, single-relation tasks such as retrieval of country-capital mappings. When moving to genuinely novel tasks that require maintaining and tracking multiple latent variables, executing multi-step algebraic operations, or evaluating compositional constraints, it remains fundamentally unclear whether models can reliably induce underlying computational structures from demonstrations alone, and how task-level or learner-level properties modulate the relative efficacy of rules versus examples.

This work addresses this open question by fixing the underlying latent task and evaluating identical test problems across three strictly controlled prompt conditions: rules-only, examples-only, and combined. Core idea: by constructing five multi-step reasoning tasks across games, multi-operator arithmetic, and artificial morphosyntax under minimum-coverage demonstration guarantees, the authors demonstrate that rule-based learning systematically outperforms demonstration-based learning on complex tasks; furthermore, appending examples to unambiguous rules yields negligible statistical gain, while instruction tuning selectively magnifies rule adherence without compromising intrinsic example-based induction.

Method

The authors construct a controlled benchmark to isolate the learning efficacies of rules, examples, and their combination across 5 diverse domains, each instantiated across three rigorous difficulty tiers.

Overall Architecture

The evaluation framework programmatically generates task instances and splits them into training and test sets. For every evaluation item, the identical underlying query is framed under three prompt modes: Rules-only (providing verbal rule specifications, task constraints, and explicit output formatting instructions), Examples-only (providing a dynamically sampled demonstration set satisfying the minimum coverage criterion without explicit descriptions), and Combined (concatenating the complete rule prompt with the demonstration set). The model generates responses under greedy decoding (temperature = 0.0), which are then parsed and statistically analyzed via Linear Mixed Models (LMM) to disentangle model scale, tuning regime, and task properties.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Complex Latent Task Generation<br/>5 Domains × 3 Difficulty Tiers"] --> B["Controlled Prompt Construction<br/>Identical Test Evaluation Splits"]
    B --> C["Rules-Only Mode<br/>Verbal specifications & format bounds"]
    B --> D["Examples-Only Mode<br/>Dynamic sampling via minimum coverage"]
    B --> E["Combined Mode<br/>Full rules followed by demonstrations"]
    C --> F["Deterministic Model Evaluation<br/>14 Open-Weight Models & GPT-5.4"]
    D --> F
    E --> F
    F --> G["Linear Mixed Model Analysis<br/>Decoupling scale, tuning, & task attributes"]

Key Designs

1. Multi-Dimensional Complex Task Suite: Eliminating Monolithic Associative Shortcuts
Prior ICL evaluations commonly rely on simple relational tasks susceptible to pretraining memorization. The authors introduce 5 multi-variable synthetic tasks: ① Set Game: selecting 3 cards out of 9 on a board such that each attribute (animal, biome, food, number) satisfies a strict "all-same or all-different" rule; ② Tapatan: determining terminal outcomes (Player A wins, Player B wins, or continue) on an \(n \times n\) grid under parity-sensitive diagonal adjacency rules; ③ Operator Function: evaluating newly introduced arithmetic operators constructed by chaining primitive operations over multiple integer arguments; ④ Noun Class Agreement: judging grammaticality in a synthetic language governed by agreement across determiners, adjective suffixes, and verb affixes matching noun classes; and ⑤ Lexical Category Inference: validating a four-word sequence against semantic categories governed by conjunctive (AND) or disjunctive (OR) logical formulations.

2. Minimum Coverage Demonstration Sampling: Protecting Example-Based Learning from Coverage Deficits
To prevent few-shot prompting from failing due to superficial sample under-representation, the authors enforce a "minimum coverage" design. For every test item, demonstrations are dynamically sampled from the training split to guarantee full exposure to all valid output labels, all core relational primitives, and representative edge cases. For instance, Operator Function provides all argument permutations (\(3!=6\) in Easy, \(5!=120\) in Hard); Tapatan ensures balanced coverage across all winning alignments and draw configurations; and Lexical Category Inference includes 24 to 48 demonstrations balancing both valid and subtly perturbed negative lists. The demonstration order and selection are randomized per test item using a fixed seed, eliminating prompt ordering bias.

3. Multi-Tiered Difficulty Scaling and Rigorous Rule Disambiguation: Constraining the Hypothesis Space
Each task is parameterized across Easy, Medium, and Hard tiers by systematically expanding the number of tracked variables, the vocabulary of primitives, and the requisite composition steps (e.g., Noun Class Agreement scales from 2 noun classes and 2 agreement loci up to 6 noun classes with prefix and suffix verb markings). Concurrently, rule descriptions were iteratively refined through human validation to ensure semantic completeness and eliminate unintended ambiguities without performing prompt-level performance optimization, allowing a clean measure of models' abstract deduction capabilities.

Benchmark & Evaluation Protocol

The empirical evaluation spans 14 open-weight model checkpoints across the Gemma 3 (12B, 27B), OLMo 3/3.1 (7B, 32B), Qwen 2.5 (32B), and Qwen 3 (8B, 14B) families, covering both base and instruction-tuned variants. GPT-5.4 is evaluated via the OpenAI API as an external high-capability ceiling reference. All open-weight checkpoints are served with vLLM in BF16 precision across NVIDIA L40S and RTX A6000 hardware (~600 GPU-hours total). Statistical significance is assessed using Linear Mixed Models (LMM) with model family treated as a random intercept, regressing accuracy against fixed effects of learning mode, tuning, model size, and task difficulty.

Key Experimental Results

Main Results

The primary Linear Mixed Model (LMM) regression highlights a substantial global advantage for rule-based learning over demonstration-based learning, while adding examples to rules fails to yield meaningful gains.

Effect / Contrast Estimated Coefficient \(\beta\) (0-1 Scale) Statistical Significance Substantive Interpretation
Mode: Rules vs. Examples +0.171 \(p < 0.001\) (***) Rules improve accuracy by 17.1 percentage points over examples
Mode: Combined vs. Rules +0.012 \(p = 0.287\) (n.s.) Adding examples to rules provides negligible, non-significant gain
Tuning: Instruct vs. Base +0.122 \(p < 0.001\) (***) Instruction tuning provides a massive global accuracy lift
Model Size: Large (14-32B) vs. Small (7-12B) +0.067 \(p < 0.001\) (***) Model scale consistently benefits in-context task completion
Difficulty: Medium vs. Easy -0.073 \(p < 0.001\) (***) Step-wise degradation across increased structural complexity
Difficulty: Hard vs. Easy -0.150 \(p < 0.001\) (***) Severe monotonic performance drop on hardest task variants

Task-level contrasts demonstrate that the magnitude of rule superiority is strongly modulated by the underlying cognitive recruitment of the task:

Task Domain Rules - Examples \(\beta\) Combined - Rules \(\beta\) Dominant Cognitive Demands
Operator Function +0.401 *** -0.021 (n.s.) Symbolic algebraic computation & sequential operator application
Set Game +0.319 *** -0.044 (n.s.) Algebraic same-or-different abstraction over variable attributes
Noun Class Agreement +0.208 *** +0.029 (n.s.) Formal morphological syntax & agreement tracking
Lexical Category Inference (OR) +0.066 (n.s.) +0.021 (n.s.) Disjunctive semantic classification recruiting parametric knowledge
Tapatan +0.020 (n.s.) +0.064 (n.s.) Spatial board patterns with dense distributional surface cues
Lexical Category Inference (AND) +0.014 (n.s.) +0.020 (n.s.) Conjunctive semantic classification recruiting parametric knowledge

Fine-Grained Analysis on Semantic Logic Conditions

Mean task accuracy (%) across logic conditions (Conjunctive AND vs. Disjunctive OR) on Lexical Category Inference (Medium and Hard difficulty tiers):

Model Group Logic Condition Rules-Only Examples-Only Combined
Base models (Open-weight) Conjunctive (AND) 58.8 63.8 62.7
Base models (Open-weight) Disjunctive (OR) 58.8 56.8 58.2
Instruct models (Open-weight) Conjunctive (AND) 61.5 60.9 59.8
Instruct models (Open-weight) Disjunctive (OR) 59.3 50.2 62.3
GPT-5.4 (Frontier Reference) Conjunctive (AND) 67.0 81.7 76.9
GPT-5.4 (Frontier Reference) Disjunctive (OR) 85.2 65.8 91.2

Key Findings

  • Algebraic Abstraction Heavily Favors Rules: In tasks requiring algebraic manipulation and symbolic composition (Operator Function \(\beta = +0.401\), Set Game \(\beta = +0.319\)), rules dramatically surpass examples. Inducing multi-step mathematical operators or variable equality constraints purely from input-output mappings proves nearly intractable for LLMs.
  • Redundancy of Demonstrations in Combined Prompts: Appending worked examples to explicit rules offers virtually zero additive value (\(\beta = +0.012\), non-significant). When natural language instructions unambiguously define the hypothesis space and output constraints, additional demonstrations become redundant.
  • Instruction Tuning Amplifies Rules Without Harming Example Induction: Instruction tuning produces a dramatic surge in rule-following performance (\(\beta = +0.213, p < 0.001\)) while leaving demonstration-based learning capacity virtually unchanged (\(\beta = -0.024, p > 0.05\)). Furthermore, base models exhibit clear rule-following competence on algebraic tasks, debunking the myth that base models inherently excel only on few-shot demonstrations.
  • Disjunctive Semantics Cripple Example Induction: Under example-based learning, models struggle substantially more with disjunctive categories (OR) than conjunctive ones (AND) (a 10.7 point gap in instruct models), because positive examples in disjunctions lack mutual feature similarity. Conversely, rule-based learning explicitly articulates logical connectives, entirely bridging the AND-OR performance divide.

Highlights & Insights

  • Strictly Controlled In-Context Evaluation: By decoupling the communicative medium (rules vs. demonstrations) while strictly fixing identical latent tasks and test queries, this paper eliminates confounding factors present in earlier prompt studies.
  • Dispelling the "Rules + Examples" Orthodoxy: The common heuristic of supplying both verbose guidelines and few-shot demonstrations is shown to be statistically superfluous when rules are properly articulated, providing concrete guidance for context-efficient prompting.
  • Echoing Human Concept Acquisition: The results closely mirror classical findings in cognitive psychology: similarity-based induction struggles with disjunctive concept formation, whereas symbolic rules bypass surface distributional variances to bind abstract constraints directly.

Limitations & Future Work

  • Idealized Prompts vs. Pragmatic Noise: The study utilizes meticulously disambiguated rule prompts. In everyday deployments, user instructions frequently suffer from ambiguity, noise, or underspecification, where examples might still provide essential grounding.
  • Synthetic Domain Specialization: The benchmark focuses on synthetic games, modular arithmetic, and artificial grammar. Potential pretraining exposure to canonical game forms (e.g., standard Set or Tic-Tac-Toe) cannot be completely audited away despite counterfactual modifications.
  • Absence of Internal Circuit Localization: The investigation remains at the behavioral level; future research should trace internal attention head dynamics and MLP routing to uncover why demonstrations fail to compose additively with rule activations.
  • vs. Davidson et al. (2026): While Davidson et al. discovered distinct internal task representations for rules and examples and hypothesized complementary benefits, this paper proves that these distinct representational mechanisms do not yield additive performance gains on downstream complex tasks.
  • vs. Liu et al. (2024): Liu et al. noted that rules outperform examples when rules are correct, but focused on instruction inference. This paper broadens the scope to deep multi-variable reasoning and delineates how parametric knowledge and algebraic requirements modulate this gap.
  • vs. Agarwal et al. (2024) (Many-Shot ICL): While many-shot scaling delivers gains on standard NLP benchmarks, this paper reveals that scaling demonstrations on complex deductive reasoning often leads to flat returns or prompt truncation failures, where a single concise rule remains superior.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes a pristine, controlled paradigm to settle a foundational debate in in-context learning.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 14 open-weight checkpoints and GPT-5.4 across 5 domains, 3 difficulty tiers, and formal LMM statistical modeling.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear exposition, airtight hypothesis testing, and seamless synthesis of cognitive psychology and LLM behavior.
  • Value: ⭐⭐⭐⭐⭐ Delivers indispensable empirical insights for prompt architecture optimization and foundation model alignment.