Skip to content

Isolated Causal Effects of Natural Language

Conference: ICML2025
arXiv: 2410.14812
Code: GitHub
Area: Causal Inference / Natural Language Processing
Keywords: Isolated Causal Effect, Omitted Variable Bias, Doubly Robust Estimation, Text Causal Inference, Non-focal Language Approximation

TL;DR

Proposes a formal estimation framework for the "Isolated Causal Effect," which isolates the causal effect of focal language attributes from correlated non-focal language using a doubly robust estimator and omitted variable bias (OVB) sensitivity analysis.

Background & Motivation

With the ubiquity of language technologies, understanding how linguistic variations affect readers' perceptions and behaviors has become crucial. For example, do factual errors in machine-generated text affect reader beliefs? Does a therapist's rapport-building language improve patient mental health?

However, language is highly aliased—focal attributes (e.g., factual errors) often co-occur with other attributes (e.g., inflammatory language). Without isolating focal attributes from correlated attributes, the estimated causal effect may confound both, making it impossible to determine which issue to address.

Existing text causal inference methods predominantly estimate the "Natural Causal Effect," which is the combined effect of the focal attribute and its naturally correlated attributes. This paper proposes estimating the "Isolated Causal Effect"—measuring solely the causal effect of the focal attribute itself—by averaging over all possible variations of the non-focal language.

Method

Problem Formulation

Given a text dataset \(D = \{(X_i, Y_i)\}\), text is parameterized as \(X = \{a(X), a^{\mathsf{c}}(X)\}\):

  • \(a(X) \in \{0, 1\}\): Focal intervention, representing the linguistic attribute under study.
  • \(a^{\mathsf{c}}(X) \in \mathbb{R}^d\): Non-focal language, representing all parts of the text other than the focal attribute.

Definition of Isolated Causal Effect: Let \(P^*\) be the target distribution of non-focal language, the isolated causal effect is defined as:

\[\tau^* = \mathbb{E}_{Y(\cdot) \sim \mathcal{G}} \left[ \mathbb{E}_{a^{\mathsf{c}}(X)^* \sim P^*} \left[ Y(a(X)=1, a^{\mathsf{c}}(X)^*) - Y(a(X)=0, a^{\mathsf{c}}(X)^*) \right] \right]\]

Core idea: Under both treatment conditions, the non-focal language is forced to follow the same target distribution \(P^*\), thereby isolating the pure effect of the focal attribute.

Three Key Assumptions

  1. Consistency: Observed outcomes equal their corresponding potential outcomes, \(Y = Y(a(X), a^{\mathsf{c}}(X))\).
  2. No Unmeasured Confounding: \(Y(x) \perp\!\!\perp a(X) | a^{\mathsf{c}}(X)\)
  3. Overlap: \(0 < P(a(X)=1 | a^{\mathsf{c}}(X)) < 1\)

Identification and Estimation

By transitioning the non-focal language from the natural distribution \(P\) to the target distribution \(P^*\) via importance weighting, the transition importance weight is defined as:

\[\gamma(a', a^{\mathsf{c}}(X)) = \frac{(2a'-1) P^*(a^{\mathsf{c}}(X))}{P(a^{\mathsf{c}}(X)) P(a(X)=a' | a^{\mathsf{c}}(X))}\]

Combining this with the outcome model \(g(a', a^{\mathsf{c}}(X)) = \mathbb{E}[Y(a', a^{\mathsf{c}}(X))]\), a doubly robust estimator is constructed:

\[\hat{\tau}_{DR} = \underbrace{\frac{1}{m}\sum_{j} [\hat{g}(1, a_s^{\mathsf{c}}(X_j)) - \hat{g}(0, a_s^{\mathsf{c}}(X_j))]}_{\text{Outcome Model Term}} + \underbrace{\frac{1}{n}\sum_{i} \hat{\gamma}(a(X_i), a_s^{\mathsf{c}}(X_i))(Y_i - \hat{g}(a(X_i), a_s^{\mathsf{c}}(X_i)))}_{\text{IPW Augmentation Term}}\]

Doubly robust guarantee: As long as either the weight \(\gamma\) or the outcome model \(g\) is correctly specified, the estimator remains unbiased.

Two Practical Target Distributions

  • IATE (Isolated Average Treatment Effect): \(P^* = P\), suitable for general scenarios.
  • IATT (Isolated ATT on the Treated): \(P^* = P(a^{\mathsf{c}}(X) | a(X)=1)\), which is more robust to overlap violations.

OVB Sensitivity Analysis

Because non-focal language must be represented by a low-dimensional approximation \(a_s^{\mathsf{c}}(X)\) (e.g., LM embeddings), information loss leads to omitted variable bias (OVB). Two diagnostic metrics are defined:

  • Fidelity \(\sigma^2 = \mathbb{E}[(Y - g(a(X), a_s^{\mathsf{c}}(X)))^2]\): The gap between the short model and the true outcome model.
  • Overlap \(\nu^2 = \mathbb{E}[\gamma(a(X), a_s^{\mathsf{c}}(X))^2]\): The extremeness of the importance weights.

OVB upper bound: \(|\tau_{DR_s} - \tau^*|^2 \leq \sigma^2 \nu^2 C_Y^2 C_D^2\)

Robustness Value (\(RV\)): \(RV = |\tau_{DR_s}| / (\sigma \nu)\), representing the capacity of the effect estimate to tolerate OVB (the larger, the better).

Fidelity-Overlap Trade-off

High-dimensional representations (e.g., LM embeddings) lead to high fidelity but are prone to overlap violations; low-dimensional representations (e.g., lexicons) lead to good overlap but low fidelity. SVD dimensionality reduction can effectively balance the two.

Key Experimental Results

Datasets

Dataset Type Treatment Outcome Characteristics
Amazon Semi-synthetic 10 LIWC lexicon categories Number of helpful votes True effects are known, controllable evaluation
SvT (Reddit) Real-world Weight-loss drug type Whether weight loss > 5% Has clinical trial benchmarks

Amazon Dataset Results

  • As the dimension of the non-focal language increases, the isolated effect estimate gradually approaches the true value.
  • \(\sigma^2\) decreases with dimension (fidelity improves), while \(\nu^2\) increases (overlap worsens).
  • The robustness value increases monotonically, showing that the gain in fidelity outweighs the loss in overlap.

SvT Dataset Key Results

Non-focal Language Representation Effect Estimate \(\hat{\sigma}^2\) \(\hat{\nu}^2\) Robustness Value Performance
SenteCon-Empath Closest to ground truth Moderate Normal Highest ✅ Best
LLM Prompting Conservative positive value — — High ✅ Robust
MiniLM Close to ground truth — Elevated Moderate ⚠️ Overlap issues
MPNet — — Extremely high_ Low ❌ Severe overlap violation
RoBERTa+SVD Improved Unchanged Improved Boosted ✅ SVD effective

OVB Calibration Analysis

Stepwise removal of known correlated categories (movement, science, exercise, healing) from SenteCon-Empath still yields positive lower bounds for the effect estimates; even when masking key information such as drug names and weights, the lower bounds remain positive, demonstrating the robustness of the estimation.

Highlights & Insights

  1. Outstanding Conceptual Contribution: Formally defines "isolated causal effect" for the first time and distinguishes it from "natural causal effect," providing a more precise estimand for text causal inference.
  2. Practical OVB Framework: The three-indicator system (fidelity-overlap-robustness value) can evaluate the quality of estimation when the true effect is unknown, possessing strong practical guidance significance.
  3. SVD Dimension Reduction Discovery: Simple SVD post-processing can significantly improve the overlap problem of high-dimensional representations while maintaining fidelity, offering low cost and high yield.
  4. The application of the doubly robust estimator in language causal inference provides strong theoretical guarantees and high practicality.
  5. SenteCon representations achieve robustness comparable to carefully designed LLM prompting without requiring task-specific designs.

Limitations & Future Work

  1. Handles only text-internal confounding: Assumes all confounders are contained within the text, neglecting external confounders (e.g., annotator information).
  2. Assumes focal attribute \(a(\cdot)\) is known: If \(a(\cdot)\) needs to be estimated, estimation errors would introduce additional bias.
  3. Wide confidence intervals: On real-world data (SvT), 95% confidence intervals for all estimates include 0, indicating insufficient statistical significance.
  4. Lacks an automated method for choosing non-focal language approximations, currently relying on manual comparison of various representations.
  5. Limited experimental scale: Evaluated on only two datasets, without covering larger-scale or multilingual scenarios.
  6. Has not explored methods to learn optimal representations that directly minimize the fidelity-overlap trade-off.
  • Egami et al. (2022): Codebook framework, where \(a(\cdot)\) in this paper is the codebook function.
  • Fong & Grimmer (2023): Randomized text experiments, programmatically generating text.
  • Pryzant et al. (2021): Representing confounders using Transformer embeddings.
  • Dhawan et al. (2024): Utilizing LLM prompting to extract discrete variables for causal estimation.
  • Chernozhukov et al. (2024): Non-parametric OVB bounds, directly extended to language scenarios in this paper.

Rating

  • Novelty: ⭐⭐⭐⭐ — The concept of isolated causal effect is novel, and the OVB sensitivity analysis is a first in text-based causal inference.
  • Experimental Thoroughness: ⭐⭐⭐⭐ — Semi-synthetic and real-world data, with thorough comparisons across various representations, though confidence intervals are wide.
  • Writing Quality: ⭐⭐⭐⭐⭐ — Rigorous formulation, clear logic, and highly informative tables/figures.
  • Value: ⭐⭐⭐⭐ — Provides key theoretical tools and practical guidelines for the intersection of NLP and causal inference.