Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.1290/
Code: https://github.com/kinit-sk/steering-synth-gen
Area: LLM Reasoning
Keywords: Activation Steering, Low-Resource Language Generation, Synthetic Data Augmentation, Quality Steering Vectors, Language Steering Vectors
TL;DR¶
This paper introduces activation steering into low-resource synthetic data generation, deriving novel "Quality steering vectors" by contrasting human-authored and round-trip backtranslated text alongside Language steering vectors; steering early Transformer layers consistently enhances sample diversity and downstream classifier performance across 11 typologically diverse languages.
Background & Motivation¶
Large language models have emerged as highly effective engines for synthetic data generation and data augmentation across diverse NLP tasks such as sentiment analysis, intent detection, and topic classification. In low-resource language environments where labeled training corpora are severely constrained, leveraging LLMs to synthesize task data offers a pragmatic avenue for bolstering downstream model performance. The prevailing state-of-the-art methodology heavily relies on few-shot in-context prompting using curated target-language demonstrations. However, incorporating multi-turn demonstrations in prompt contexts inevitably incurs substantial inference latency, inflates per-sample compute expenditure, and frequently induces severe lexical anchoring—trapping generated outputs within narrow, repetitive vocabulary subspaces derived directly from the prompt examples.
In parallel, representation engineering and activation steering have emerged as promising mechanisms for modulating the internal activations of language models during the forward pass without updating model parameters. While prior literature has successfully leveraged steering vectors to govern discrete semantic attributes such as truthfulness, refusal, toxicity, or language identity switching, existing multilingual investigations have largely confined their attention to cultural fact retrieval or literal language switching rather than the generative quality, well-formedness, and lexical diversity of synthesized datasets. The core bottleneck in low-resource data synthesis is not merely enforcing a target script, but preventing synthetic artifacts, unnatural syntactic constructions, and degraded phrasing from poisoning downstream training.
To circumvent the computational overhead and diversity penalties imposed by prompt-level demonstrations, this paper explores activation steering as a lightweight, training-free intervention for multilingual synthetic data generation. The authors develop two complementary steering strategies: Language Steering, which isolates language identity via one-vs-rest contrastive activation subtraction, and Quality Steering, which systematically isolates natural well-formedness by contrasting authentic human writing against multi-round backtranslated text. Core idea: by injecting normalized quality and language steering vectors into the early residual stream layers of LLMs during forward decoding, models are steered toward a natural human-authored manifold, breaking lexical anchoring and yielding superior downstream fine-tuning utility without requiring prompt modifications.
Method¶
Overall Architecture¶
The end-to-end framework consists of four sequential stages: 1. Steering Vector Extraction: Extracting residual stream hidden states from frozen transformer blocks across layers using task-agnostic multilingual corpora (FLORES and BOUQUET). Language steering vectors \(\hat{\mathbf{v}}_{l, L}\) are obtained via one-vs-rest contrastive averaging, while Quality steering vectors \(\hat{\mathbf{v}}_{l, L, \text{qual}}\) are derived by contrasting original human-written text against two-round backtranslated machine text generated via NLLB-200-600M. 2. Inference-Time Activation Intervention: During autoregressive decoding, linear intervention is applied to the residual stream at target layer \(l\): \(\mathbf{h}_l \leftarrow \mathbf{h}_l + \alpha \cdot \hat{\mathbf{v}}_l\), with layer-wise \(L_2\)-normalized steering vectors scaled by intervention multiplier \(\alpha\). 3. Controlled Synthetic Dataset Generation: Generating labeled synthetic datasets for topic classification and sentiment analysis across 11 typologically diverse languages under both zero-shot and few-shot prompting conditions. 4. Downstream Classifier Evaluation & Diversity Auditing: Fine-tuning an XLM-RoBERTa encoder model on the generated synthetic datasets and testing on authentic human benchmarks (SIB-200 and sentiment sets), accompanied by a four-dimensional geometric evaluation of latent manifold diversity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multilingual Parallel Corpora<br/>(FLORES + BOUQUET)"] --> B["Stage 1: One-vs-Rest Language Steering Vector Extraction"]
A --> C["Stage 2: Backtranslation Contrastive Quality Vector Extraction<br/>(Human vs. NLLB Backtranslated)"]
B --> D["Stage 3: Residual Stream Layer-wise Injection & Generation<br/>(Early/Middle/Late Layer Steering)"]
C --> D
D --> E["Stage 4: Downstream Fine-Tuning Evaluation & Diversity Metrics<br/>(XLM-R Fine-Tuning + 4 Geometric Diversity Metrics)"]
Key Designs¶
1. One-vs-Rest Language Steering Vector Extraction: Isolating Target Linguistic Identity from Semantic Priors Transformer residual streams simultaneously encode semantic concepts, task context, and syntactic structures. To isolate the pure linguistic identity of a target language \(L\) from shared cross-lingual semantic representations, this design computes a one-vs-rest contrastive vector across all transformer layers. For each language \(L \in \mathcal{L}\), token-level hidden states \(\mathbf{a}_{l,i,j}\) are accumulated over all non-padding tokens \(N_L\) to obtain the language-specific centroid \(\boldsymbol{\mu}_{l,L}\). The raw steering vector \(\mathbf{v}_{l,L}\) is constructed by subtracting the composite centroid of all remaining languages from \(\boldsymbol{\mu}_{l,L}\), followed by layer-wise \(L_2\) normalization: $$ \mathbf{v}{l, L} = \boldsymbol{\mu}} - \frac{1}{|\mathcal{L}|-1} \sum_{L' \in \mathcal{L}, L' \neq L} \boldsymbol{\mu{l, L'}, \quad \hat{\mathbf{v}}} = \frac{\mathbf{v{l, L}}{|\mathbf{v} $$ Enforcing }|_2\(L_2\) normalization ensures consistent intervention magnitudes across different transformer depths and disparate token distributions, preventing activation scale explosions in deeper layers.
2. Backtranslation Contrastive Quality Vector Extraction: Decoupling Fluency from Language Specificity Prior representation engineering efforts have lacked mechanisms to steer general text well-formedness and grammatical authenticity. This work introduces Quality steering vectors by leveraging paired corpora \(D = \{d_{\text{good}}, d_{\text{bad}}\}\), where \(d_{\text{good}}\) contains authentic human-authored sentences from FLORES and BOUQUET, and \(d_{\text{bad}}\) comprises parallel counterparts degraded via two rounds of pivot backtranslation (source \(\rightarrow\) English \(\rightarrow\) Chinese \(\rightarrow\) English \(\rightarrow\) source) using the distilled NLLB-200-600M model. The quality direction vector is computed as the difference between the mean activation centroids: $$ \mathbf{v}{l, L, \text{qual}} = \boldsymbol{\mu}}}} - \boldsymbol{\mu{l, L, d}}}, \quad \hat{\mathbf{v}{l, L, \text{qual}} = \frac{\mathbf{v} $$ Because the source and backtranslated sentences share the exact underlying semantic message, the subtraction cancels out topic and content features, isolating the precise geometric direction pointing toward authentic human writing. A linear probe pre-experiment confirms that logistic classifiers separate }}}{|\mathbf{v}_{l, L, \text{qual}}|_2\(d_{\text{good}}\) and \(d_{\text{bad}}\) with \(>75\%\) accuracy starting from the earliest layers.
3. Residual Stream Layer-wise Injection & Generation: Early Layer Intervention Prevents Semantic Collapse To establish the optimal structural locus for steering, interventions are systematically evaluated at early (\(\sim 21\%\) depth, e.g., L9/L10 in Gemma, L7/L17 in Llama), middle (\(\sim 48\%\)), and late (\(\sim 74\%\)) processing stages across four instruction-tuned open LLMs (Gemma-2-9B, Gemma-2-27B, Llama-3.1-8B, and Llama-3.1-70B). Interventions applied at early layers target the consolidation of syntactic structures and linguistic identity before high-level concepts solidify, allowing the steering signal to propagate smoothly through the rest of the network. Conversely, interventions at late layers interfere with next-token probability calibration, causing degenerated repetitive text. Crucially, the optimal scaling multiplier \(\alpha\) diverges radically across architectures: Gemma models feature logit soft-capping and large activation norms, requiring robust steering (\(\alpha \in \{25, 50, 75, 100\}\)), whereas Llama models are exceptionally sensitive, functioning optimally at \(\alpha \in \{1, 2, 3, 4\}\) and collapsing under higher intensities.
4. Downstream Fine-Tuning Evaluation & Diversity Metrics: Closing the Validation Loop via Empirical Classification and Geometry To rigorously verify whether steering produces genuinely superior synthetic data, the pipeline establishes a dual-tier evaluation framework. First, synthetic data generated per configuration (50 samples per label for sentiment analysis, 20 samples per label for topic classification) is used to fine-tune an XLM-RoBERTa-base encoder from scratch across 20 random seeds, evaluating generalization on human test sets. Second, the generated corpora are evaluated across four geometric diversity metrics: Lexical Diversity (unique character trigram ratio), Embedding Diversity (intra-class pairwise cosine distance mapped via Qwen3-Embedding-4B), Isocontour Radius (90th percentile distance from the global dataset centroid), and Homogeneity (KDE-based structural density uniformity), verifying that data quality improvements arise from genuine conceptual dispersion rather than corrupt random noise.
Loss & Training¶
The data generation process operates purely at inference time through activation intervention on frozen generator LLMs, incurring zero parameter updates and zero training loss. For downstream validation, XLM-RoBERTa-base classifiers are trained with a learning rate of \(1\times 10^{-5}\), batch size of 16, AdamW optimizer, and early stopping with a patience of 10 epochs. Input sequences are normalized to lowercase with punctuation removed.
Key Experimental Results¶
Main Results¶
The methodology was comprehensively evaluated across 11 languages (Amharic am, Czech cs, Danish da, German de, Hebrew he, Indonesian id, Javanese jv, Maltese mt, Slovak sk, Slovenian sl, Sundanese su) spanning Indo-European, Afro-Asiatic, and Austronesian families. Downstream F1 score differences relative to unsteered zero-shot baselines are summarized below:
| Generator LLM | Steering Method | Layer Depth | Best Mean \(\Delta\text{F1}\) (Zero-Shot) | Best Mean \(\Delta\text{F1}\) (Few-Shot) | Statistically Significant Gain Rate |
|---|---|---|---|---|---|
| Gemma-2-9B | Quality Steering | Early (L9) | +4.55 | +2.99 | 45.46% (Zero-Shot) / 36.36% (Few-Shot) |
| Gemma-2-9B | Language Steering | Early (L9) | +3.07 | +1.22 | 34.09% (Zero-Shot) / 29.55% (Few-Shot) |
| Llama-3.1-8B | Quality Steering | Early (L7) | +3.70 | +1.66 | 45.46% (Zero-Shot) / 36.36% (Few-Shot) |
| Llama-3.1-8B | Language Steering | Early (L7) | +2.73 | +1.40 | 34.09% (Zero-Shot) / 29.55% (Few-Shot) |
| Llama-3.1-70B | Quality Steering | Early (L17) | +2.53 | +2.91 | 45.46% (Zero-Shot) / 36.36% (Few-Shot) |
| Llama-3.1-70B | Language Steering | Early (L17) | +2.63 | +2.33 | 34.09% (Zero-Shot) / 29.55% (Few-Shot) |
| Gemma-2-27B | Quality Steering | Early (L10) | +2.29 | +2.04 | 45.46% (Zero-Shot) / 36.36% (Few-Shot) |
| Gemma-2-27B | Language Steering | Early (L10) | +2.28 | +2.20 | 34.09% (Zero-Shot) / 29.55% (Few-Shot) |
In zero-shot synthesis (Table 1), Quality steering achieved double-digit F1 leaps in under-represented languages: Gemma-2-9B at L9 boosted Slovak (sk) by +11.77 F1 and Slovenian (sl) by +10.25 F1; Llama-3.1-8B at L7 boosted Amharic (am) by +13.13 F1.
Ablation Study¶
The table below contrasts intervention depths (Early vs. Middle vs. Late) and the ablation of choosing a single fixed \(\alpha\) per layer (simulating scenarios without a target validation set, Tables 14 & 15):
| Configuration / Intervention Locus | Steering Method | Zero-Shot Positive Rate | Zero-Shot Mean \(\Delta\text{F1}\) | Few-Shot Positive Rate | Few-Shot Mean \(\Delta\text{F1}\) |
|---|---|---|---|---|---|
| Early Layers (~21% depth) | Quality | 97.73% | +3.27 | 97.73% | +2.40 |
| Middle Layers (~48% depth) | Quality | 95.45% | +2.57 | 81.82% | +1.73 |
| Late Layers (~74% depth) | Quality | 93.18% | +2.56 | 86.36% | +1.69 |
| Early Layers (~21% depth) | Language | 86.36% | +2.68 | 77.27% | +1.79 |
| Middle Layers (~48% depth) | Language | 88.64% | +2.68 | 84.09% | +2.09 |
| Late Layers (~74% depth) | Language | 90.91% | +2.54 | 97.73% | +1.94 |
| Fixed Single \(\alpha\) per Layer (Early) | Quality | 79.54% | +2.13 | 68.18% | +0.95 |
| Fixed Single \(\alpha\) per Layer (Early) | Language | 72.73% | +1.26 | 54.55% | +0.48 |
Key Findings¶
- Early Layers Constitute the Primary Intervention Locus for Quality: Early-layer Quality steering produced downstream performance gains in 97.73% of test configurations under both zero-shot and few-shot conditions, substantially outperforming middle and late layer interventions. Early layers represent the regime where syntactic framing is established before conceptual features crystallize.
- Polarity Inversion Between Quality and Language Steering Vectors: Cosine similarity analysis revealed that in Llama models, Language and Quality vectors exhibit near-perfect negative correlation (\(\approx -1.0\)) for European and Austronesian languages, whereas Semitic languages (Amharic, Hebrew, Maltese) show strong positive correlation (\(\approx +1.0\)). This geometric divergence indicates that Language vectors often incorporate rare vocabulary or dialectal idiosyncrasies that oppose the smooth, generalized human-like manifold captured by Quality steering.
- Substantial Diversity Expansion Without Structural Degeneration: Under zero-shot generation, activation steering markedly increased output diversity (e.g., Gemma-2-9B at L9 Quality steering improved Embedding Diversity by +52.83% and Isocontour Radius by +4.09%). Positive homogeneity shifts confirm that the expanded space remains cohesively structured rather than degraded into disconnected noise.
- Demonstration Saturation Dampens Few-Shot Steering Impact: In few-shot prompting, human in-context demonstrations already orient the model's representations toward the human-authored manifold, leading to smaller marginal gains from steering. However, early Quality steering still provided consistent downstream improvements in \(>97\%\) of cases.
Highlights & Insights¶
- Repurposing Backtranslation as a Latent Geometric Normal: Rather than relying on heuristic quality classifiers or rejection sampling, the authors innovatively transform two-stage pivot backtranslation into paired activation differentials, capturing an unadulterated "human well-formedness" direction in latent space.
- Overcoming Lexical Anchoring at Constant Inference Cost: Conventional few-shot methods constrain model generations around prompt examples while incurring heavy context-length compute costs. Activation steering modifies internal states in constant time, expanding the generation envelope (Isocontour Radius) and producing diverse samples unconstrained by example tokens.
- Systematic Mapping of Inter-Architectural Steering Sensitivity: The paper elucidates why Gemma models can absorb massive steering vectors (\(\alpha=100\)) without output degeneration (due to Query-Key normalization and logit soft-capping), while standard Llama architectures collapse past \(\alpha=4\), establishing crucial design boundaries for representation engineering practitioners.
Limitations & Future Work¶
- Constrained Task Diversity: Evaluations are exclusively restricted to text classification tasks (sentiment analysis and topic classification); whether activation steering benefits open-ended generation, reasoning chains (CoT), or summarization in low-resource regimes remains unverified.
- Dependence on Upstream Translation Encoders: The quality vector's efficacy is inherently bounded by the translation capabilities of NLLB-200-600M. For extremely under-resourced vernaculars where machine translation models produce garbled output, backtranslation differentials may capture degenerate noise rather than linguistic quality.
- Absence of an Analytical Hyperparameter Prediction Framework: Optimal steering intensity \(\alpha\) remains empirical and model-dependent, spanning several orders of magnitude. The field currently lacks an analytical formulation to deduce the ideal \((\alpha, \text{layer})\) coordinate a priori without validation sweep data.
Related Work & Insights¶
- vs. Anikina et al. (2025) Low-Resource Few-Shot Prompting: Anikina et al. established target-language few-shot prompting as the gold standard for low-resource generation; this paper demonstrates that zero-shot activation steering matches or exceeds few-shot downstream utility while avoiding prompt token overhead and lexical anchoring.
- vs. Turner et al. (2024) / Zou et al. (2023) Activation Engineering: Earlier representation engineering focused on single-attribute steering (honesty, sentiment, toxicity, or language identity); this work represents the first application of activation steering to synthetic data generation and introduces the first general-purpose Quality steering vector.
- vs. Ghussin et al. (2026b) Multilingual Cultural Steering: Ghussin et al. leveraged steering vectors for cultural knowledge elicitation; the present work reveals the critical geometric orthogonality and polarity inversion between language identity vectors and quality vectors, advancing the theoretical understanding of multilingual latent spaces.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering application of representation engineering to synthetic data augmentation; ingenious backtranslation-based quality vector extraction.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 4 open LLMs, 11 typologically diverse languages, 3 layer depths, 4 geometric diversity metrics, and over 500 GPU hours.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, structured empirical exposition, and deep geometric insights into latent vector polarities.
- Value: ⭐⭐⭐⭐☆ Highly impactful for cost-effective low-resource NLP augmentation, though its utility for multi-hop complex reasoning tasks requires further empirical validation.