thinking in different spaces domain specific latent geometry survives cross arch
content_hash: 5b3e056fa8a1476f ---# Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation
Conference: EMNLP 2026
Paper: ACL Anthology
Area: NLP Understanding
Keywords: Cross-Architecture Representation Alignment, Activation Steering, Latent Space Geometry, Double Dissociation, Domain Subspace
TL;DR¶
By learning an affine projection between independently trained heterogeneous LLMs and intervening in the student residual stream during inference, this work demonstrates that cross-architecture latent geometry can be linearly aligned for zero-weight-update behavioral correction, while proving that factual recall and mathematical reasoning occupy geometrically orthogonal domain subspaces.
Background & Motivation¶
The striking convergence of modern large language models on benchmark tasks raises a foundational representation-learning question: do independently trained models with distinct architectures converge toward topologically isomorphic conceptual manifolds in their latent spaces? Existing representation engineering and activation steering paradigms (such as contrastive activation addition and fixed steering vectors) predominantly operate within a single, isolated network, fundamentally presuming a self-contained coordinate system. Conversely, while model stitching demonstrates that inserting linear adapters between early layers of one model and late layers of another produces functional hybrids, it relies on static architectural pipelines or training-time adapters, failing to address whether heterogeneous networks can dynamically steer and correct each other during real-time autoregressive generation.
The core tension in cross-architecture steering lies in reconciling architectural heterogeneity with the delicacy of dynamic autoregressive inference. Different model families (such as Llama, Mistral, Gemma, Qwen, and Phi) differ radically in hidden layer dimensions, total depth, tokenization vocabularies, and initialization distributions; injecting raw hidden states causes severe numerical instability and desynchronizes downstream decoding. Crucially, attempting alignment via over-parameterized nonlinear neural architectures (e.g., deep autoencoders) risks distorting the intrinsic geometry of latent spaces, obscuring whether downstream steerability originates from genuine conceptual compatibility or from the brute-force expressivity of an artificial mapper.
To evaluate the shared-manifold hypothesis while preserving geometric interpretability, this work adheres strictly to a minimalist affine alignment paradigm—restricting cross-model translation to global rotation, uniform scaling, and translation—coupled with dynamic residual stream substitution. Core idea: learn a global affine projection matrix via dense Ridge regression across heterogeneous latent activations, and perform norm-calibrated dynamic residual substitution with tunable intervention strength during inference, achieving weight-free reasoning correction while rigorously testing the geometric orthogonality of domain-specific subspaces.
Method¶
Overall Architecture¶
The input to the system consists of paired activation states extracted from a teacher model and a student model processing identical prompt prefixes; the output is the student model's generated text steered in real time by the translated teacher representations. The framework comprises two distinct phases: offline affine geometric alignment and online residual stream intervention. First, the final-token hidden representations across calibration questions are extracted and L2-normalized to solve an optimal affine projection via Ridge regression. Subsequently, during student autoregressive generation on an identified opportunity set (questions answered correctly by the teacher and incorrectly by the student), the student's residual stream is intercepted at a chosen layer, norm-rescaled, and dynamically updated using a convex combination or extrapolation with the projected teacher state.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prompt Question: {q} Answer:"] --> B["Dual Model Forward Pass<br/>Extract last-token hidden states $h_T$ and $h_S$"]
B --> C["Affine Manifold Mapping Estimation<br/>L2 normalization + dense Ridge regression for W and b"]
C --> D["Temporal Depth Matching<br/>Teacher deep extraction $l_T \approx 0.90$ to student mid-late $l_S \approx 0.75$"]
D --> E["Dynamic Residual Stream Intervention<br/>Norm matching + linear blend/extrapolation $h_{\text{final}}$"]
E --> F["Controlled Autoregressive Decoding<br/>Deterministic greedy sampling generates final answer"]
Key Designs¶
1. Affine Manifold Mapping Estimation: Verifying shared latent geometry via global linear transformations
To prevent nonlinear adapters from fabricating artificial alignment, the mapping between student hidden state \(h_S \in \mathbb{R}^{d_S}\) and teacher hidden state \(h_T \in \mathbb{R}^{d_T}\) is strictly constrained to an affine transformation:
where \(W \in \mathbb{R}^{d_S \times d_T}\) executes a global rotation and scaling of the conceptual manifold, and \(b \in \mathbb{R}^{d_S}\) absorbs mean coordinate shifts between distinct embedding spaces. To eliminate layer-specific magnitude variances that stem from architectural design rather than semantic content, all activation vectors are unit L2-normalized:
The objective is solved using dense Ridge regression with a fixed L2 regularization penalty (\(\lambda = 0.1\)):
This design deliberately rejects sparse penalties (L1 Lasso), confirming that conceptual alignment relies holistically on dense, distributed representations across the entire activation vector. Furthermore, orthogonal Procrustes and shuffled permutation controls rule out artifacts of pure rotation or marginal distributions.
2. Dynamic Residual Stream Intervention: Tunable extrapolation and norm-adaptive state rewriting
Injecting the projected teacher vector \(\hat{h}_S = W \tilde{h}_T + b\) directly into the student model's residual stream poses stability risks if magnitude discrepancies disrupt internal layer dynamics. The intervention hook intercepts the target layer's output, rescales \(\hat{h}_S\) to precisely match the L2 norm of the student's current residual vector, and computes the modified state parameterized by intervention coefficient \(\alpha\):
When \(\alpha \in (0, 1)\), this performs smooth state interpolation. At \(\alpha = 1\), the student representation is entirely overwritten. For extrapolative coefficients \(\alpha > 1\) (such as \(\alpha = 2\)), the algebraic identity reveals:
Here, the intervention simultaneously injects the target teacher representation and subtracts the student's erroneous internal state, implementing active error negation. This mechanism proves vital when a student model exhibits substantial representational inertia that mere substitution cannot overcome.
3. Temporal Depth Matching: Relative semantic handoff across heterogeneous depths
Because the evaluated models differ drastically in total depth (e.g., 80 layers in Llama-70B vs. 16 layers in Llama-1B), layer selection cannot rely on absolute layer indices. The injection pipeline parameterizes candidate layers by relative depth \(l \in [0, 1]\). Building on probing literature showing that transformers process syntax in lower layers, semantic abstraction in middle layers, and vocabulary distribution in late layers, the framework operationalizes a semantic handoff hypothesis: extracting activations from deep teacher layers (\(l_T \approx 0.90\)) where reasoning trajectories are fully resolved, and injecting them into mid-to-late student layers (\(l_S \approx 0.75\)) where the student transitions from abstract representations to token decoding.
Loss & Training¶
The projection matrix is computed analytically via closed-form Ridge regression on a 70/30 train/test split. During the intervention phase, all weights of both teacher and student models remain completely frozen—no gradients, fine-tuning, or backpropagation are involved. Autoregressive inference employs greedy decoding (do_sample=False), capping generation at 50 new tokens for TruthfulQA and 100 new tokens for GSM8K.
Key Experimental Results¶
Main Results¶
The evaluation spans 5 teacher models (Llama-3-70B, Llama-3-8B, Mistral-7B, Gemma-2-9B, Granite-8B-Code) and 4 student models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B, Phi-3-Mini), yielding 20 heterogeneous pairings. Below are the geometric alignment goodness-of-fit (\(R^2\) at \(l_T = l_S = 0.75\)) and peak behavioral correction rates (\(\Delta\)) across optimal layer and \(\alpha\) configurations:
| Dataset | Metric | 20-Pair Average | Peak Pairing | Lowest Pairing |
|---|---|---|---|---|
| TruthfulQA (Verbal Reasoning) | Goodness-of-Fit \(R^2\) | 0.343 (peak ~0.50) | 0.684 (Llama-8B \(\to\) Llama-1B) | 0.278 (Granite-8B \(\to\) Phi-3) |
| TruthfulQA (Verbal Reasoning) | Peak Correction Rate \(\Delta\) (%) | 25.2% | 50.0% (Granite-8B \(\to\) Phi-3) | 14.0% (Gemma-9B \(\to\) Llama-1B) |
| GSM8K (Math Reasoning) | Goodness-of-Fit \(R^2\) | 0.256 (peak ~0.40) | 0.351 (Granite-8B \(\to\) Phi-3) | 0.176 (Mistral-7B \(\to\) Qwen-1.5B) |
| GSM8K (Math Reasoning) | Peak Correction Rate \(\Delta\) (%) | 25.5% | 43.3% (Gemma-9B \(\to\) Qwen-1.5B) | 8.5% (Granite-8B \(\to\) Phi-3) |
Note: The correction rate \(\Delta\) reports the percentage of questions in the opportunity set (where the teacher is correct and the baseline student incorrect) that are converted to correct outputs under the intervention.
Ablation Study¶
Table 2 details the projection performance under alternative mapping constraints averaged across all 20 pairings at \(l_T = l_S = 0.75\):
| Config | Regularization & Constraints | TruthfulQA \(R^2\) | GSM8K \(R^2\) | Note |
|---|---|---|---|---|
| Ridge Regression (Full) | L2 penalty (\(\lambda=0.1\), distributed) | 0.343 | 0.256 | Dense affine alignment baseline |
| Lasso Regression | L1 penalty (\(\lambda=0.0001\), sparse) | 0.014 | — | Near-total collapse, proving distributed encoding |
| Orthogonal Procrustes | Orthogonal constraint (\(W^\top W = I\)) | -0.557 | -1.290 | Pure rotation fails drastically; scaling/bias are vital |
| Permutation Control | L2 with shuffled \(H_S\) rows | -0.274 | -0.218 | Baseline under destroyed semantic correspondence |
Table 3 presents the universal double dissociation between verbal (TruthfulQA) and mathematical (GSM8K) reasoning domains across all 20 pairings:
| Training Domain \(\to\) Evaluation Domain | Mean \(R^2\) | Min \(R^2\) | Max \(R^2\) | Finding |
|---|---|---|---|---|
| TruthfulQA in-domain | 0.343 | 0.278 | 0.446 | In-domain linear alignment effective |
| TruthfulQA \(\to\) GSM8K cross-domain transfer | -3.831 | -8.170 | -1.807 | Catastrophic collapse (active negative interference) |
| GSM8K in-domain | 0.256 | 0.176 | 0.351 | In-domain mathematical geometry aligned |
| GSM8K \(\to\) TruthfulQA cross-domain transfer | -1.729 | -3.227 | -1.081 | Complete transfer failure; domains are orthogonal |
Key Findings¶
- Decoupling of geometric fit and behavioral steering: The Pearson correlation between held-out \(R^2\) and downstream correction rate \(\Delta\) is near zero (\(r = -0.071\)). High explained variance does not imply behavioral efficacy; directional precision matters far more than fitting global variance.
- Complete inversion of student steerability rankings: Phi-3-Mini is the most steerable student on TruthfulQA (\(\Delta = 50.0\%\)), but becomes the least steerable student on GSM8K (\(\Delta = 23.7\%\)). In contrast, Llama-1B is the hardest student to steer on verbal tasks (\(\Delta = 16.7\%\)), but among the most steerable on arithmetic tasks (\(\Delta = 40.5\%\)). Steerability is a domain-specific property governed by subspace geometry rather than universal model plasticity.
- Fragility of mathematical representations: On GSM8K, the optimal coefficient \(\alpha\) collapses to the sub-interpolation regime (\(\alpha = 0.25\) in 99.4% of configurations), and correction rate drops to 0.0% for any \(\alpha \ge 0.8\). Mathematical representations tolerate minimal displacement before multi-step reasoning is destroyed.
- Universal domain modularity via double dissociation: Confirmed across all 20 pairings (and across all 16 layer depth combinations for Mistral-7B \(\to\) Qwen-1.5B), cross-domain transfer produces catastrophic collapse (\(R^2 < 0\)). Verbal and mathematical representations occupy geometrically orthogonal subspaces in latent activation space.
Highlights & Insights¶
- Linear existence proof of shared latent geometry: A single affine projection accounts for roughly 50% of activation variance across completely independent architectures, empirically proving that independently trained models converge to topologically compatible manifolds.
- Active error negation in the extrapolative regime: At \(\alpha = 2\), the mathematical reformulation reveals that the intervention explicitly subtracts the student's original activation while reinforcing the teacher vector, effectively overcoming residual stream inertia.
- Hierarchical vulnerability of mid-network layers: Transfer interference peaks sharply when injecting mismatched domain representations into mid-network layers (\(l_S \approx 0.50\), where \(R^2\) plunges to -11.5), directly localizing domain-specific semantic construction to intermediate transformer layers.
Limitations & Future Work¶
- Short-form generation benchmark scope: Experiments are limited to short-form factual recall and arithmetic answers (50–100 tokens) using deterministic matching; extending single-layer residual substitution to long-trace reasoning models (e.g., DeepSeek-R1, o1) remains unverified.
- Coarse layer grid resolution: Layer selection is sampled across only four relative depths (0.25, 0.50, 0.75, 0.90), leaving fine-grained transitions unobserved.
- Quantization noise in the largest teacher: The 70B teacher was loaded in 4-bit NF4 format due to GPU constraints, introducing rounding noise into extracted activations and lowering its effective steering potential.
- White-box architecture constraint: The method requires direct forward-hook access to internal hidden states, rendering it inapplicable to proprietary black-box API models.
Related Work & Insights¶
- vs Model Stitching (Bansal et al., 2021): Model stitching statically merges model halves via training-time adapters; this work achieves real-time, non-destructive dynamic steering in the residual stream during autoregressive decoding.
- vs Intra-Model Steering (Turner et al., 2024; Panickssery et al., 2024): CAA and activation engineering operate strictly within a single network's coordinate frame; this paper bridges the cross-architecture coordinate divide via linear projection.
- vs ExpertSteer (Wang et al., 2025): ExpertSteer employs complex nonlinear autoencoders and additive vectors; this approach relies on pure affine transformation and direct residual substitution, ensuring transparent geometric interpretability.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First demonstration of cross-architecture latent projection and weight-free dynamic residual steering.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ 20 heterogeneous model pairings, 2,560 conditions, rigorous permutation/sparsity/rotation controls, and universal double dissociation.
- Writing Quality: ⭐⭐⭐⭐⭐ Elegant mathematical formalization, highly rigorous empirical deductions, and lucid prose.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational framework for training-free multi-model inference collaboration and mechanistic interpretability.