Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/jmichaelov/target-language-control
Area: Multilingual & Translation
Keywords: activation steering, optimal control, cross-lingual generation, target-language control, representation geometry
TL;DR¶
This paper formulates cross-lingual target-language text generation as an optimal control problem over hidden activation trajectories, deriving a closed-form Linear Semantic Control (LISECO) projection law and establishing a unified evaluation metricโadhered coherenceโto achieve reliable, prompt-free language forcing across multilingual language models.
Background & Motivation¶
Multilingual large language models have achieved significant progress in recent years, yet their performance remains highly skewed toward high-resource languages such as English, which dominate both pretraining corpora and latent representation space. When prompted to generate text in a lower-resource target language or tasked with extended reasoning chains, state-of-the-art models frequently suffer from "language drift," spontaneously reverting to generating English continuations.
The core tension underlying this failure mode lies in the delicate coupling between language-specific representations and conceptual semantic representations. Existing activation steering techniques, such as the difference-in-means method (DIFFMEANS) or activation addition (ACTADD), apply a fixed heuristic steering vector uniformly across intermediate activations. Because Transformer representations are highly anisotropic, blindly adding an uncalibrated shift pushes hidden states into unnatural, out-of-distribution subspaces. Consequently, while the model may nominally flip its output token distribution toward the target language, it often suffers severe syntactic degeneration, infinite repetitive loops, or complete semantic detachment. Furthermore, existing evaluations typically measure only superficial language identification accuracy, failing to penalize degenerated or unfaithful outputs.
To achieve robust, prompt-free cross-lingual switching without fine-tuning model parameters, this paper recasts activation steering through the lens of dynamical systems and optimal control theory. Core idea: formulate the propagation of activations across Transformer layers as a discrete-time dynamical system, apply the Semantic Hub Hypothesis to derive a closed-form minimal-disturbance projection control law (LISECO) onto target-language decision boundaries, and evaluate generated outputs under a tripartite metric of adhered coherence combining language adherence, linguistic well-formedness, and semantic fidelity.
Method¶
Overall Architecture¶
The proposed approach conceptualizes the layerwise feedforward propagation of Transformer representations as a discrete-time dynamical system and reshapes its activation trajectory via optimal control inputs at selected layers. An input prompt in a source language is initially mapped by token embeddings into a latent state, which evolves sequentially across subsequent layers. The target-language generation pipeline operates through three key stages: first, lightweight linear probes are fitted at each layer to identify hyperplanes demarcating target-language probability regions; second, a closed-form optimal control law dynamically calculates the minimal orthogonal projection required to steer non-adhering states onto the target boundary; finally, the adjusted trajectory passes through the unembedding head to generate fluent, semantically grounded text in the target language without explicit prompting.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input source text sequence s"] --> B["Multi-Layer Dynamical Trajectory Modeling<br/>x_ฯ+1 = โ_ฯ+1(x_ฯ)"]
B --> C["Layerwise Linear Probe Identification<br/>demarcate target probability boundary"]
C --> D["Closed-Form Minimal Disturbance Control<br/>orthogonal projection to decision boundary"]
D --> E["Prompt-Free Cross-Lingual Generation<br/>disentangle language while preserving semantics"]
E --> F["Output fluent target-language continuation"]
Key Designs¶
1. Multi-Layer Dynamical Trajectory Modeling: Formulating Representation Flow as a Dynamical System Conventional activation steering methods typically treat deep Transformers as static black boxes, arbitrarily injecting a fixed shift into a single ad-hoc layer while ignoring the progressive layerwise disentanglement and transformation of semantic versus language-specific features. To ground cross-lingual intervention on rigorous mathematical foundations, this paper formalizes the layerwise computation of a Transformer as a discrete-time dynamical system: $\(x_0 = E(s), \quad \tilde{x}_\tau = x_\tau + \theta_\tau(x_\tau), \quad x_{\tau+1} = \ell_{\tau+1}(\tilde{x}_\tau), \quad y = U(x_T)\)$ where \(x_\tau \in \mathbb{R}^d\) denotes the last token's latent state after layer \(\tau\), and \(\theta_\tau\) represents the state-dependent control input. By casting language steering into a trajectory tracking problem under boundary constraints, this design enables coordinated multi-layer interventions that steer activation flow toward target-language regions while preserving the shared conceptual backbone necessary for cross-lingual generalization.
2. Layerwise Linear Probe Identification: Mapping Language Regions under the Linear Representation Hypothesis Before designing control inputs, one must locate the subspace corresponding to the target language in latent space. Drawing on the Linear Representation Hypothesis, language attributes are hypothesized to occupy linearly separable regions across intermediate Transformer layers. For each layer \(\tau\), a lightweight linear probe \(f_\tau(x_\tau) = \text{softmax}(W_\tau^T x_\tau)\) is trained to map latent representations to categorical language probabilities. The parameter matrix \(W_\tau \in \mathbb{R}^{d \times |L|}\) is optimized via cross-entropy loss over the final token activations of parallel sentences from the FLORES development set. Consequently, the target-language region \(\mathcal{R}_t\) is defined as the pre-image satisfying a specified probability interval: $\(\mathcal{R}_t = \{x \in \mathbb{R}^d \mid \alpha_{\min} \le \sigma(W_\tau^T x) \le \alpha_{\max}\}\)$ This geometric characterization converts abstract language membership into concrete polyhedral constraints within high-dimensional activation space.
3. Closed-Form Minimal Disturbance Control: Grounding Minimal Projections in the Semantic Hub Hypothesis Standard steering approaches such as DIFFMEANS and ACTADD utilize static shift vectors \(\theta_\tau = \kappa z_\tau\), which are completely agnostic to the current activation state \(x_\tau\) and frequently cause catastrophic under-shooting or over-shooting. According to the Semantic Hub Hypothesis, semantically equivalent representations in different languages lie closer to each other in activation space than semantically unrelated content. Therefore, the most semantic-preserving path into the target language region is the nearest orthogonal projection point. The control problem is formulated as a constrained quadratic minimization: $\(\min_{\theta_\tau} \|\theta_\tau\|_2^2 \quad \text{s.t.} \quad \alpha_{\min} \le \sigma(W_\tau^T (x_\tau + \theta_\tau)) \le \alpha_{\max}\)$ Following Cheng et al. (2024), this problem admits an analytical closed-form solution: when the current activation already resides within the target bounds, no disturbance is applied (\(\theta_\tau^* = 0\)); otherwise, the intervention is scaled precisely along the probe normal vector \(W_\tau\): $\(\theta_\tau^* = \frac{\log(\frac{1}{\alpha_{\min}} - 1) - W_\tau^T x_\tau}{\|W_\tau\|_2^2} W_\tau \quad (\text{when } \sigma(W_\tau^T x_\tau) < \alpha_{\min})\)$ This guarantees that perturbed activations enter the target manifold with minimal perturbation energy, protecting semantic integrity.
4. Adhered Coherence Evaluation Metric: Unified Multi-Desiderata Quality Assessment Prior literature often relied solely on language identification classifiers, erroneously awarding top scores to degenerative outputs such as repetitive single-word loops. This work introduces a principled tripartite metric comprising Language Adherence, Linguistic Coherence, and Semantic Coherence. For a generated continuation \(s_{\text{gen}}\), language adherence is verified via a fastText classifier. Semantic coherence is established when the cosine similarity between the LaBSE embedding of \(s_{\text{gen}}\) and the true reference ending \(s_{\text{true}}\) exceeds that with the incorrect ending \(s_{\text{false}}\). Linguistic coherence is bounded using \(n\)-gram log-perplexity \(L_t \le \mu(L_b) + 2\sigma(L_b)\) against a unsteered target-language baseline, preventing neural judge confounds on repetitive degenerate text. The overall adhered coherence metric \(C \in \{0, 1\}\) represents the logical conjunction of all three criteria: $\(C = \mathbb{I}\left[\arg\max_i \hat{p}(L_i) = t \;\land\; \text{sim}(s_{\text{gen}}, s_{\text{true}}) > \text{sim}(s_{\text{gen}}, s_{\text{false}}) \;\land\; L_t \le \mu(L_b) + 2\sigma(L_b)\right]\)$
A Worked Example¶
Consider an English input story prefix from XSTORYCLOZE: "A girl bought a new bike. She rode it to the park.", with the goal of forcing Spanish (spa) continuation without changing the prompt: 1. During forward propagation, the final token's layerwise activations \(x_\tau\) yield an initial Spanish probe probability \(\sigma(W_\tau^T x_\tau) \approx 0.02\). 2. With target threshold \(\alpha_{\min} = 1 - 10^{-8}\), the model determines that the hidden state lies outside the target region and computes \(\theta_\tau^*\) as the exact orthogonal distance to the decision hyperplane. 3. Adding \(\theta_\tau^*\) shifts the latent state into \(\tilde{x}_\tau\), instantly lifting probe probability above the threshold while leaving all orthogonal semantic dimensions unperturbed. 4. The unprompted autoregressive decoder continues in grammatically sound Spanish: "Ella se divirtiรณ mucho allรญ." 5. The evaluation pipeline assesses the output: fastText identifies Spanish; LaBSE yields \(\text{sim}(s_{\text{gen}}, s_{\text{true}}) = 0.81 > 0.35 = \text{sim}(s_{\text{gen}}, s_{\text{false}})\); and the \(n\)-gram log-perplexity is \(3.42 \le 4.10\). All three conditions are fulfilled, registering an adhered coherence score of 1.
Key Experimental Results¶
Main Results¶
The primary evaluation tests unprompted cross-lingual language forcing on the held-out XSTORYCLOZE benchmark across 12 cross-lingual pairs (spanning English, Spanish, Hindi, and Chinese, totaling 1,512 items per pair). The benchmark compares DIFFMEANS (tuned to model and optimal single layer) against multi-layer LISECO (both with a single fixed universal hyperparameter setting and tuned per model).
| Model Architecture | Metric | DIFFMEANS (Model-Tuned) | LISECO (Universal Setting) | LISECO (Model-Tuned) | Performance Trend |
|---|---|---|---|---|---|
| BLOOM 7.1B | Adhered Coherence | ~0.11 | ~0.14 | ~0.15 | +36.4% (LISECO wins) |
| EuroLLM 9B | Adhered Coherence | ~0.24 | ~0.16 | ~0.18 | -25.0% (DIFFMEANS wins) |
| Gemma-3 1B | Adhered Coherence | ~0.19 | ~0.13 | ~0.14 | -26.3% (DIFFMEANS wins) |
| Llama-3.1 8B | Adhered Coherence | ~0.21 | ~0.25 | ~0.29 | +38.1% (LISECO wins) |
| Llama-3.2 3B | Adhered Coherence | ~0.24 | ~0.27 | ~0.31 | +29.2% (LISECO wins) |
| Qwen-2.5 7B | Adhered Coherence | ~0.23 | ~0.28 | ~0.29 | +26.1% (LISECO wins) |
| Qwen-3 8B Base | Adhered Coherence | ~0.27 | ~0.26 | ~0.28 | Comparable |
| Tiny-Aya-Base 3B | Adhered Coherence | ~0.17 | ~0.21 | ~0.23 | +35.3% (LISECO wins) |
Ablation Study¶
Ablations on Llama 3.2 1B across all English, Spanish, and Hindi pairs dissect the impact of intervention scope (single-layer vs. multi-layer), injection starting depth, and hyperparameter magnitudes across each component of adhered coherence.
| Steering Method & Layer Scope | Hyperparameter Setting | Language Adherence | Linguistic Coherence | Semantic Coherence | Adhered Coherence | Mechanism & Impact |
|---|---|---|---|---|---|---|
| LISECO: Multi-Layer (all layers) | \(\alpha_{\text{inner}} = 10^{-5}\) | ~0.88 | ~0.92 | ~0.57 | ~0.36 | Optimal balance across all three criteria |
| LISECO: Multi-Layer (all layers) | \(\alpha_{\text{inner}} = 10^{-1}\) | ~0.42 | ~0.95 | ~0.59 | ~0.18 | Boundary too weak; insufficient steering |
| LISECO: Single-Layer (single layer) | \(\alpha_{\text{inner}} = 10^{-5}\), late layer | ~0.55 | ~0.91 | ~0.55 | ~0.22 | Single layer lacks momentum across depth |
| DIFFMEANS: Multi-Layer (all layers) | \(c = 1.0\) | ~0.72 | ~0.40 | ~0.48 | ~0.12 | Cumulative static drift causes syntactic collapse |
| DIFFMEANS: Single-Layer (single layer) | \(c = 1.0\), late 1/3 layer | ~0.62 | ~0.89 | ~0.54 | ~0.24 | Confining drift to late layers avoids collapse |
| ACTADD: Single-Layer (single layer) | \(c = 1.0\) | ~0.45 | ~0.88 | ~0.52 | ~0.15 | Endonym word pairs generalize poorly |
Key Findings¶
- Multi-Layer Feasibility via State-Dependent Control: While applying DIFFMEANS or ACTADD across all layers destroys fluency (plunging linguistic coherence below 40%), LISECO safely operates across all hidden layers because perturbations are dynamically scaled to minimal projection distances.
- Robustness and Hyperparameter Generalization: A single LISECO configuration (\(\alpha_{\text{inner}} = 10^{-8}\) across all layers except static embeddings) matches or outperforms model-tuned DIFFMEANS on 5 of the 8 tested models without requiring per-model layer searching.
- Cross-Lingual Asymmetries and English Centrality: Across all architectures, steering into English (\(x \to \text{eng}\)) proves substantially easier than steering away from English into lower-resource target languages, highlighting how pretraining disproportionately concentrates representational volume around English.
Highlights & Insights¶
- From Heuristic Addition to Principled Optimal Control: Replaces uncalibrated vector addition with rigorous trajectory control grounded in the geometry of representation space.
- Microsecond Closed-Form Projection: Achieves exact constraint satisfaction via a closed-form analytical solution that incurs negligible latency overhead during autoregressive generation.
- Tripartite Evaluation Standard: Introduces adhered coherence to prevent the false positive evaluation of degenerated or unfaithful text in controlled generation research.
Limitations & Future Work¶
- Causal Validity of Linear Probes: LISECO assumes that linearly separable probe boundaries reflect causal mechanisms of language generation; if a probe learns spurious correlations, projections fail to guide generation.
- Binary vs. Multiclass Polyhedral Geometry: The current closed-form formulation relies on binary half-space constraints, which can induce edge-case overlaps when extended to dozens of languages simultaneously.
- Task Diversity: Due to heavy computational constraints (~7,000 GPU hours), empirical verification was restricted to 12 language pairs on story continuation tasks without explicit translation instructions.
Related Work & Insights¶
- vs. DIFFMEANS (Rimsky et al., 2024; Lu et al., 2025): DIFFMEANS adds a static difference vector regardless of current activation location, risking over-steering; LISECO applies adaptive, minimal projections that vanish when activations already adhere to the target.
- vs. INCLINE (Wang et al., 2025b): INCLINE relies on a learned cross-lingual mapping matrix modulated by a fixed scaling step; LISECO provides closed-form optimality guarantees with exact boundary satisfaction.
- vs. Neuron Manipulation (Gurgurov et al., 2025): Neuron-level boosting intervenes only on sparse subsets of coordinates (<5%); LISECO operates across continuous distributed representation space.
Rating¶
- Novelty: โญโญโญโญโญ Formulates target-language generation as dynamic trajectory optimal control with a closed-form projection law.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 8 major open LLMs across 12 language pairs with comprehensive layerwise and hyperparameter ablations (~7,000 GPU hours).
- Writing Quality: โญโญโญโญโญ Rigorous mathematical exposition paired with an intuitive, self-contained evaluation framework.
- Value: โญโญโญโญโญ Offers a foundational theoretical and practical blueprint for non-invasive, controllable language generation in multilingual models.