Skip to content

From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cullena20/ModelLog
Area: Reasoning
Keywords: Language Model Evaluation, Declarative Probabilistic Evaluation, Weighted Model Counting, Semantic Constraints, Inverse Scaling

TL;DR

This paper introduces MODELLOG, a declarative probabilistic framework that translates local autoregressive token predictions into propositional variables governed by symbolic logical constraints, measuring satisfaction via calibrated weighted model counting and revealing systematic logical consistency failures alongside inverse-scaling phenomena across prominent language model families.

Background & Motivation

Pre-training in modern large language models relies almost exclusively on an elegant and minimalist objective: cross-entropy loss over next-token predictions. This simple autoregressive signal enables models to acquire broad factual knowledge and syntactic fluency, yet its supervision remains strictly local and positive over observed token continuations. Consequently, richer global semantic relations—such as mutual exclusivity between contradictory assertions, invariance across paraphrased prefixes, and coherent deductive entailment—remain entirely implicit. For instance, next-token training readily teaches a model that "alcohol" is a sensible completion to "When driving, it is not safe to drink __", but it provides no explicit mechanism preventing the model from also assigning high probability to "alcohol" under the negated prompt "When driving, it is safe to drink __".

This disparity introduces a fundamental conceptual tension: why does localized cross-entropy pre-training induce sophisticated reasoning capabilities, and how can we rigorously quantify whether pre-trained models satisfy rich semantic structures? Traditional post-hoc behavioral benchmarks rely on greedy top-1 generation accuracy, which discards the continuous probability distribution and confidence gradients of the model. Conversely, loss-based evaluations such as perplexity merely average scalar prediction errors across tokens without exposing structured semantic relations. Crucially, performing semantic reasoning directly over raw token probabilities introduces severe interpretability hurdles: raw probabilities are confounded by local vocabulary branching factors (where semantically valid continuations receive artificially small likelihoods when distributed across many plausible alternatives), and loose logical formulas can be satisfied spuriously simply because their Boolean truth tables have many satisfying assignments.

To resolve these challenges, this paper formulates pre-training evaluation through the lens of statistical relational learning and neuro-symbolic logic. Core idea: express evaluation targets as declarative symbolic theories over Boolean token-prediction events, calibrate raw probabilities to reflect local live-region membership via nucleus entropy, and measure semantic support using weighted model counting relative to an uninformed prior baseline, thereby yielding an interpretable, differentiable metric that connects behavioral evaluation directly to learning gradients.

Method

Overall Architecture

MODELLOG treats evaluation benchmarks as compositional, structured semantic objects. The end-to-end framework proceeds through five interconnected steps: it first maps autoregressive next-token predictions across contrasting prompt contexts into symbolic Boolean proposition atoms; it then defines a declarative theory \(F\) of logical constraints (e.g., transitivity, implication, negation consistency, or mutual exclusivity) using standard propositional connectives; next, it rescales raw token probabilities into valid-event Bernoulli probabilities using nucleus-entropy calibration to account for local branching sizes; following calibration, it performs exact weighted model counting (WMC) over all satisfying interpretations of \(F\) to compute model satisfaction, contrasting this with an uninformed uniform prior \(\mathrm{WMC}_0(F)\) to compute probabilistic consistency \(\rho\) and constraint accuracy \(\mathrm{CAcc}\); finally, treating semantic satisfaction as a differentiable loss, it analytically decomposes the learning gradient into fit, informativeness, and directional update components.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Contexts & Candidate Tokens<br/>Multi-prompt token predictions"] --> B["Token Prediction Atomization<br/>Local outputs mapped to Boolean events"]
    B --> C["Declarative Semantic Theory Construction<br/>Formulas encoding entailment/mutex/negation"]
    C --> D["Nucleus Entropy Probability Calibration<br/>Live-region offset balances branching odds"]
    D --> E["Weighted Model Counting & Prior Evaluation<br/>Exact WMC measures semantic mass vs chance"]
    E --> F["Semantic Loss Gradient Factorization<br/>Fit ratio / informativeness boost / update vector"]

Key Designs

1. Token Prediction Atomization: converting categorical distributions into localized Boolean events
To overcome the limitation where raw next-token probabilities conflate token competition across the vocabulary with semantic validity, MODELLOG abstracts local model predictions as discrete symbolic propositions. For an autoregressive model \(M_\theta\), position context \(x_{<j}\), and candidate completion token \(w_j\), the atomic event \(M(x, w_j)\) (abbreviated as \(P\)) denotes that \(w_j\) is a valid, acceptable continuation in context \(x\). Rather than operating on a single multiclass categorical distribution over \(\mathcal{V}\), each prediction event is interpreted under a Bernoulli semantics: in any valid truth assignment or possible world \(I\), the event either holds (True) or does not hold (False). This abstraction decouples probability mass allocation from binary semantic validity and enables evaluating joint logical constraints across multiple distinct prefixes.

2. Declarative Semantic Theory Construction: formalizing inter-prediction relations as logical constraints
To evaluate structured behavioral properties beyond isolated token accuracy, MODELLOG represents semantic relations across contexts as Boolean formulas \(F\) over prediction variables using standard operators (\(\land, \lor, \neg, \rightarrow, \leftrightarrow, \oplus\)). Paraphrase symmetry is represented by biconditionals \(M(x_1, y) \leftrightarrow M(x_2, y)\); negation consistency enforces exclusive-or constraints \(M(x, y) \oplus M(\neg x, y)\); and mutual exclusivity prevents contradictory attributes from co-occurring \(M(x, y_1) \oplus M(x, y_2)\). Significantly, the standard language model likelihood objective is formally proven to be a maximal conjunctive boundary case of this framework: for a token sequence \(x = w_1, \dots, w_n\), the likelihood formula (Formula 1) is: $\(F_\ell(x) := \bigwedge_{j=1}^n M(x, w_j)\)$ Under raw token probability weighting \(P_\theta(M(x, w_j)) = P_{M_\theta}(w_j \mid x_{<j})\), the satisfaction probability simplifies to \(P_\theta(F_\ell(x)) = \prod_{j=1}^n P_{M_\theta}(w_j \mid x_{<j})\), perfectly recovering autoregressive sequence likelihood. This bridges standard language modeling and structured logic under a single theoretical tent.

3. Nucleus Entropy Probability Calibration: correcting for context-dependent vocabulary dispersion
To resolve the pathology where semantically acceptable completions in open-ended contexts receive minuscule probabilities simply because mass is partitioned across many synonyms, the authors introduce nucleus-entropy calibration. For candidate token \(w_j\) in context \(x\), the calibrated event probability \(\hat{P}_{\theta_C}(M(x, w_j))\) is derived by applying a dynamic positive offset \(C\) to the log-odds: $\(\hat{P}_{\theta_C}(M(x, w_j)) := \sigma\left(\mathrm{logit}\left(P_{M_\theta}(w_j \mid x)\right) + C\right) \propto e^C P_{M_\theta}(w_j \mid x)\)$ Here, \(C = H(\mathrm{top}_p(P_{M_\theta}(\cdot \mid x)))\) corresponds to the Shannon entropy of the nucleus (top-\(p\) tokens, e.g., \(p=0.95\)) of the local distribution. In highly uncertain or negative contexts where many words are valid continuations, the nucleus entropy \(H\) is elevated, meaning \(e^H\) reflects a large effective candidate volume that appropriately boosts the odds of valid tokens; in constrained contexts, \(C\) remains small. Calibrated probabilities thus accurately reflect membership within the model's effective "live" region of continuations.

4. Weighted Model Counting and Prior Evaluation: eliminating chance satisfaction via constraint informativeness
To prevent syntactically loose formulas (such as large disjunctions with many satisfying interpretations) from scoring artificially high, MODELLOG computes exact satisfaction through Weighted Model Counting (WMC). Given formula \(F\), model satisfaction \(P_\theta(F)\) aggregates the probability of all satisfying truth assignments \(I \in \mathcal{I}(F)\): $\(\mathrm{WMC}(F; \theta) := \sum_{I \in \mathcal{I}(F)} \prod_{P: I(P)=\top} P_\theta(P) \prod_{P: I(P)=\bot} (1 - P_\theta(P))\)$ To establish an unbiased baseline, the framework defines the uninformed prior \(\mathrm{WMC}_0(F) := |\mathcal{I}(F)| / 2^{|\mathrm{vars}(F)|}\) under uniform weights (\(P_0(P)=0.5\)), and formalizes constraint informativeness as \(I(F) := -\log_2 \mathrm{WMC}_0(F)\). The likelihood formula \(F_\ell\) represents a complete minterm with \(I(F_\ell) = n\) (Proposition 2), whereas common logical implications admit multiple models and have lower informativeness. Normalized probabilistic consistency \(\rho\) then quantifies the fraction of possible gain over chance: $\(\rho(F; \theta_C) = \frac{\mathrm{WMC}(F; \theta_C) - \mathrm{WMC}_0(F)}{1 - \mathrm{WMC}_0(F)}\)$ complemented by discrete constraint accuracy \(\mathrm{CAcc}(F; \theta_C) = \mathbf{1}[\mathrm{WMC}(F; \theta_C) > \mathrm{WMC}_0(F)]\).

Loss & Training

Because satisfaction probabilities computed via WMC are differentiable polynomials with respect to individual variable probabilities, MODELLOG connects evaluation directly to training via the semantic loss \(\ell_{\mathrm{sl}}(F; \theta) := -\log P_\theta(F)\). The authors establish a fundamental gradient factorization identity (Proposition 5): $\(\nabla_\theta \ell_{\mathrm{sl}}(F; \theta) = -A(F; \theta) \cdot 2^{I(F)} \cdot \nabla_\theta \mathrm{WMC}(F; \theta)\)$ where \(A(F; \theta) := \frac{\mathrm{WMC}_0(F)}{\mathrm{WMC}(F; \theta)}\) is a model-fit ratio indicating under-satisfaction (\(A > 1\)) or over-satisfaction (\(A < 1\)) relative to chance; \(2^{I(F)}\) is an exponential gradient amplification term directly governed by the formula's intrinsic logical informativeness; and \(\nabla_\theta \mathrm{WMC}(F; \theta)\) supplies the local parameter update direction. This theorem elucidates why likelihood training is an extraordinarily strong pre-training signal: standard cross-entropy optimizes a maximally informative conjunction (\(I(F)=n\)), generating exponential gradient scale, whereas weaker logical constraints may suffer from insufficient gradient magnitude unless strategically weighted.

Key Experimental Results

Main Results: PROBCT Diagnostic Benchmark

The authors construct PROBCT (Probabilistic Consistency Tests) from Open English WordNet and YAGO knowledge bases, spanning five logical constraint categories. Datasets were filtered via GPT-5o-mini and verified through independent human annotations (human precision \(\ge 0.88\)). Below are the main constraint accuracy scores (\(\mathrm{CAcc} \%\)) across Gemma-3, Llama-3, and Qwen3 model families, averaged across nucleus thresholds \(p \in \{0.80, 0.85, 0.90, 0.95\}\).

Model Family & Size Transitivity
\(I(F) \approx 0.193\)
Forward Implication
\(I(F) \approx 0.415\)
Negation Consistency
\(I(F) \approx 1.0\)
Entity Mutex
\(I(F) \approx 1.0\)
Spatial Mutex
\(I(F) \approx 1.0\)
Gemma-3-270M 81.0 (±2.3) 79.7 (±0.5) 18.5 (±1.4) 48.7 (±0.8) 18.5 (±4.8)
Gemma-3-1B-pt 69.0 (±1.5) 65.1 (±1.5) 15.9 (±0.8) 57.3 (±3.9) 12.4 (±4.8)
Gemma-3-4B-pt 46.9 (±1.1) 50.8 (±1.7) 18.2 (±0.9) 56.8 (±5.2) 19.9 (±6.6)
Gemma-3-12B-pt 40.4 (±1.6) 45.2 (±2.0) 19.2 (±0.9) 54.7 (±5.4) 23.8 (±7.7)
Llama-3.2-1B 52.7 (±2.4) 68.7 (±1.8) 23.1 (±2.1) 64.0 (±3.8) 31.5 (±7.4)
Llama-3.2-3B 45.5 (±1.3) 60.8 (±2.1) 21.6 (±2.6) 61.6 (±6.0) 25.3 (±6.9)
Llama-3.1-8B 35.2 (±1.5) 50.0 (±1.2) 22.7 (±1.3) 60.0 (±5.9) 16.1 (±6.4)
Qwen3-0.6B 76.4 (±0.5) 82.8 (±1.5) 17.2 (±0.6) 58.2 (±0.7) 9.1 (±3.2)
Qwen3-1.7B 64.2 (±1.2) 75.3 (±1.6) 18.9 (±3.1) 62.9 (±2.1) 25.9 (±7.2)
Qwen3-4B 60.8 (±3.2) 73.0 (±2.6) 19.3 (±0.6) 67.7 (±5.0) 16.6 (±5.1)
Qwen3-8B 53.7 (±2.2) 69.6 (±2.7) 20.8 (±0.1) 68.2 (±4.3) 14.8 (±5.3)
Qwen3-14B 56.4 (±2.1) 64.1 (±3.6) 20.9 (±1.2) 65.9 (±5.6) 12.6 (±5.4)

Ablation & Mechanism Analysis: Inverse Scaling in Forward Implication

On Forward Implication (\(P_1 \rightarrow P_2\)) and Transitivity, models systematically exhibit inverse scaling: smaller models outperform larger models. The table below inspects the underlying probability allocations for \(P_1 \rightarrow P_2\) (where specific premise \(P_1\) entails broader conclusion \(P_2\), e.g., "A Golden Retriever is a dog \(\rightarrow\) A Golden Retriever is a mammal"). Here, \(\Delta = P_{\theta_C}(P_2) - P_{\theta_C}(P_1)\) represents the mean probability gap, and \(P(P_2 > P_1)\) indicates the percentage of examples where the broader consequence probability matches or exceeds the premise.

Model Premise \(P_{\theta_C}(P_1)\) Conclusion \(P_{\theta_C}(P_2)\) Gap \(\Delta = P_2 - P_1\) Consequence Win \(P(P_2 > P_1) (\%)\) Accuracy \(\mathrm{CAcc} (\%)\)
Gemma-3-270M 0.53 (±0.03) 0.70 (±0.02) +0.17 (±0.01) 64.7 (±0.2) 75.4 (±1.2)
Gemma-3-1B-pt 0.60 (±0.03) 0.58 (±0.04) -0.02 (±0.00) 49.5 (±0.6) 57.8 (±3.3)
Gemma-3-4B-pt 0.69 (±0.04) 0.50 (±0.04) -0.19 (±0.00) 34.7 (±0.4) 44.1 (±3.4)
Gemma-3-12B-pt 0.70 (±0.04) 0.45 (±0.04) -0.24 (±0.01) 30.1 (±0.5) 38.3 (±4.3)
Llama-3.2-1B 0.73 (±0.03) 0.66 (±0.04) -0.06 (±0.01) 45.2 (±0.8) 64.3 (±2.8)
Llama-3.2-3B 0.67 (±0.04) 0.57 (±0.04) -0.10 (±0.00) 41.5 (±0.7) 54.5 (±3.7)
Llama-3.1-8B 0.70 (±0.03) 0.50 (±0.04) -0.20 (±0.01) 33.2 (±1.0) 43.3 (±4.1)
Qwen3-0.6B 0.59 (±0.03) 0.76 (±0.03) +0.17 (±0.00) 67.1 (±0.4) 80.1 (±2.5)
Qwen3-1.7B 0.64 (±0.03) 0.72 (±0.03) +0.08 (±0.00) 60.0 (±0.3) 73.1 (±3.1)
Qwen3-4B 0.66 (±0.03) 0.69 (±0.04) +0.03 (±0.01) 54.8 (±0.7) 69.8 (±4.4)
Qwen3-8B 0.68 (±0.03) 0.67 (±0.04) -0.01 (±0.00) 51.3 (±0.8) 67.5 (±4.3)
Qwen3-14B 0.65 (±0.04) 0.62 (±0.05) -0.03 (±0.01) 46.5 (±1.1) 59.8 (±5.2)

Key Findings

  • Severe logical inconsistency in pre-trained models: Models consistently fail on negation consistency and spatial exclusivity tasks, achieving \(\mathrm{CAcc} \le 23.8\%\) across families. A large portion of examples exhibit negative \(\rho\) scores, meaning the models satisfy semantic constraints less frequently than a completely uninformed uniform prior.
  • Inverse scaling is driven by factual probability mass redistribution: In deductive reasoning rules like transitivity and implication, smaller models succeed more often because they assign higher baseline probability to generic, high-level hypernyms (e.g., "fruit", "mammal"). As parameter scale increases, models memorize rare, fine-grained entities, dramatically inflating premise probability \(P_1\) while consequence probability \(P_2\) is diluted across broader candidate sets, resulting in \(P_1 > P_2\) and violating the logical constraint.
  • Calibration invariance across methods: Testing alternative calibration formulations (full vocabulary entropy calibration and nucleus count calibration) across varying \(p \in [0.8, 0.95]\) demonstrates that relative task difficulty, inverse-scaling trends, and low negation consistency scores remain completely stable.

Highlights & Insights

  • Formal unification of next-token prediction and symbolic logic: Proving that the standard autoregressive sequence likelihood is identical to the satisfaction probability of a maximally informative minterm formula (\(I(F_\ell)=n\)) mathematically bridges statistical relational learning and language modeling.
  • Analytical insight into cross-entropy efficacy via gradient scaling: The gradient identity reveals that the gradient of semantic loss is scaled exponentially by constraint informativeness \(2^{I(F)}\). This explains why next-token cross-entropy provides such an intensely coherent learning signal compared to looser semantic losses.
  • Reusable diagnostic framework: The fit ratio \(A(F; \theta)\) and informativeness \(I(F)\) provide concrete tools for active data curriculum design and pruning—allowing researchers to identify training examples that supply maximal, non-contradictory gradient updates.

Limitations & Future Work

  • Single-token prediction scope: MODELLOG currently operates on single-token substitutions using factorized Bernoulli probabilities, without extending to multi-token entity spans, free-form text generations, or autoregressive chain-of-thought traces.
  • Synthetic template benchmarks: PROBCT data is synthetically derived from WordNet and YAGO knowledge graphs. While filtered with LLM-as-a-judge and human verification, it lacks the full pragmatics, syntactic diversity, and contextual richness of real-world discourse.
  • Empirical pre-training validation: While the gradient factorization establishes a theoretical link between evaluation metrics and learning dynamics, large-scale model pre-training directly guided by these constraint losses was not executed.
  • vs Behavioral Probing Benchmarks (LAMA, oLMpics, BECEL): Prior probing methods rely on discrete generation accuracy or raw log-likelihood ratios, which are heavily distorted by uncalibrated vocabulary priors. MODELLOG leverages exact weighted model counting and subtracts chance baselines, providing a graded, calibrated probabilistic measure of consistency.
  • vs Neuro-Symbolic Learning (Semantic Loss, DeepProbLog): Existing neuro-symbolic methods primarily use logical constraints as loss functions or output layers in specialized architectures. MODELLOG repurposes these tools as a formal evaluation paradigm for diagnosing off-the-shelf pre-trained foundation models.
  • vs Selective Modeling (Rho-1): While Rho-1 empirically selects training tokens based on loss perplexity improvements, MODELLOG's threshold disjunction relaxations (\(F_{\ge k}\)) and monotonicity properties offer a principled, symbolic foundation for sub-token selection and loss bounds.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates language model evaluation as declarative probabilistic inference and formally links likelihood objectives to maximal conjunctive minterms.
  • Experimental Thoroughness: ⭐⭐⭐⭐☆ Rigorous evaluation across 3 major model families and 12 model checkpoints, backed by comprehensive inverse-scaling and calibration ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly articulate, combining elegant mathematical derivations with intuitive conceptual clarity and insightful figure designs.
  • Value: ⭐⭐⭐⭐⭐ Provides profound theoretical clarification on why cross-entropy works while revealing critical logical deficits and inverse-scaling trends in modern LLMs.