A Novel Semantic Manifold Alignment Attack against Embedding-to-Embedding Obfuscation in Privacy-Preserving LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Safety
Keywords: Privacy-Preserving Inference, Embedding Obfuscation, Manifold Alignment, Semantic Reconstruction, Gromov-Wasserstein
TL;DR¶
By conceptualizing the obfuscated vector stream as an unknown foreign tokenizer-language, Proxy Manifold Alignment (PMA) leverages Gromov-Wasserstein optimal transport and mutual nearest neighbor refinement over Word2Vec-induced proxy semantic manifolds to achieve over 80โ99 ROUGE-L plaintext reconstruction without model weights, parallel data, or original embedding values.
Background & Motivation¶
With the pervasive integration of Large Language Models (LLMs) into security-critical and privacy-sensitive sectors such as clinical diagnosis, algorithmic finance, and legal advisory, the Machine Learning as a Service (MLaaS) deployment paradigm has introduced pressing confidentiality dilemmas. Because end users are typically forced to offload heavy inference computations to cloud infrastructure, sensitive user queries and proprietary model completions are immediately exposed to honest-but-curious service providers. While classical cryptographic privacy-enhancing technologiesโnotably Multi-Party Computation (MPC), Fully Homomorphic Encryption (FHE), and Trusted Execution Environments (TEE)โguarantee rigorous theoretical security boundaries, their astronomical computational and communication overheads render them impractical for interactive latency-sensitive applications; state-of-the-art GPU-accelerated FHE solutions still require multiple seconds to generate a single token.
In response, the NLP and security communities have actively pursued lightweight alternatives collectively termed Embedding-to-Embedding Obfuscation (E2EO), exemplified by ObfusLM, SentinelLMs, AlienLM, CodeCipher, and STIP. Unlike differential privacy or noise injection strategies that perturb representations at the cost of irreversible utility degradation, E2EO schemes apply deterministic, one-to-one transformations that map plaintext token embeddings onto fixed ciphertext representations locally. This design allows the cloud model to retain near-plaintext generation capabilities through minimal downstream adaptation, thereby promising an optimal equilibrium between user privacy and inference utility. Because the resulting ciphertext vectors fundamentally corrupt the ambient numerical geometry of the original embedding space, existing nearest-neighbor inversion attacks (e.g., KNN), supervised decoders (e.g., InvBert), and one-dimensional token frequency attacks (TFA) fail entirely.
However, this perceived robustness is an illusion grounded in security by obscurity. Because E2EO relies fundamentally on deterministic bijective token substitution, it completely preserves the underlying linguistic structure, discrete token ordering, and higher-order syntactic and semantic dependencies of the input sequence. In essence, the obfuscated representation acts as a massive token-level substitution cipher. Even when embedding numerical values are entirely randomized, the structural manifold formed by word co-occurrences remains invariant. Core idea: treat the obfuscated vector stream as an unknown foreign "tokenizer-language," independently build proxy semantic spaces from the obfuscated stream and an auxiliary public corpus using word co-occurrence modeling, and align their underlying semantic manifolds via Gromov-Wasserstein optimal transport and iterative mutual nearest neighbor refinement to translate ciphertext representations back to plaintext without accessing model weights or original embeddings.
Method¶
Overall Architecture¶
The Proxy Manifold Alignment (PMA) attack reformulates ciphertext-to-plaintext reconstruction as an unsupervised bilingual lexicon induction problem between an unknown tokenizer-language and standard plaintext. Assuming an honest-but-curious adversary that only intercepts the obfuscated vector stream \(Z\) and knows the publicly available tokenizer \(\mathcal{T}\) of the target model alongside generic public corpora, PMA operates across three progressive phases: (1) constructing parallel proxy semantic spaces using continuous bag-of-words modeling to extract latent geometry from empirical co-occurrence statistics; (2) performing initial structural alignment over high-frequency anchor subspaces via Gromov-Wasserstein optimal transport; and (3) computing a closed-form global orthogonal mapping and refining it iteratively with mutual nearest neighbors to eliminate high-dimensional hubness and solidify the one-to-one decoding dictionary.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
subgraph S1["Phase 1: Constructing Proxy Semantic Spaces"]
direction TB
A["Intercept Obfuscated Stream Z<br/>Treated as foreign token stream"] --> B["CBOW Word2Vec Training<br/>Ciphertext Proxy Space XC"]
C["Public Reference Corpus<br/>Tokenized via same tokenizer T"] --> D["CBOW Word2Vec Training<br/>Plaintext Proxy Space XP"]
end
subgraph S2["Phase 2: Initial Anchor Manifold Alignment"]
direction TB
B & D --> E["Select Top-K Frequent Symbols<br/>Form Anchor Subspaces AP & AC"]
E --> F["Compute Pairwise Manifold Distances<br/>Cosine Distance Matrices MP & MC"]
F --> G["Gromov-Wasserstein Optimal Transport<br/>Solve Structural Soft Coupling ฮ(0)"]
end
subgraph S3["Phase 3: Global Alignment and Refinement"]
direction TB
G --> H["Orthogonal Procrustes Analysis (SVD)<br/>Expand Anchors to Global Mapping Q0"]
H --> I["Mutual Nearest Neighbor (MNN) Filtering<br/>Eliminate High-Dimensional Hubs"]
I --> J["Iterative Orthogonal Refinement Qr<br/>Converge to 1:1 Inversion Dictionary"]
end
J --> K["Plaintext Inversion Output<br/>Reconstruct User Plaintext Sequence"]
Key Designs¶
1. Constructing Proxy Semantic Spaces: Bypassing Numerical Invariance via Co-occurrence Manifolds
Prior inversion attacks fail because they depend on preserving local metric neighborhoods between the original and obfuscated embeddingsโa property deliberately eliminated by E2EO transformations. PMA completely divorces itself from numerical values and instead treats each unique obfuscated vector as an abstract symbol within an unknown language. According to semantic space recovery theory, word embedding algorithms do not fabricate semantic structures; rather, when the corpus size \(m\) and vocabulary size \(n\) scale, the empirical negative log-cooccurrence count converges asymptotically to the squared Euclidean distance in latent space:
The attacker trains a Continuous Bag-of-Words (CBOW) Word2Vec model on the collected ciphertext stream to obtain the ciphertext proxy semantic matrix \(X_C \in \mathbb{R}^{|V| \times d}\) (\(d=300\)). Symmetrically, tokenizing a public reference corpus with the known target tokenizer \(\mathcal{T}\) and training an identical CBOW model yields the plaintext proxy semantic matrix \(X_P \in \mathbb{R}^{|V| \times d}\). Because natural language constraints and the tokenization vocabulary are shared, these independently trained semantic vector clouds form approximately isometric Riemannian manifolds.
2. Initial Anchor Manifold Alignment: High-Frequency Zipfian Priors and Gromov-Wasserstein Transport
Attempting to align entire vocabularies directly is severely impeded by sampling noise in the long tail of rare tokens and excessive quadratic complexity. Relying on Zipf's law, which dictates that the co-occurrence relationships and ranking distributions of core high-frequency words remain remarkably consistent across natural language corpora, PMA restricts initial alignment to the top-\(K\) most frequent symbols (\(K=2000\)). After isolating \(\ell_2\)-normalized anchor subspaces \(\tilde{X}_P[A_P]\) and \(\tilde{X}_C[A_C]\), intra-space pairwise semantic distance matrices are formulated:
To align these two unaligned distance structures without coordinate correspondence, PMA solves the entropic Gromov-Wasserstein (GW) optimal transport problem:
where \(\Gamma\) represents the soft alignment coupling between anchor sets. Because GW evaluates structural distortion of intra-space relational geometries rather than absolute Euclidean positions, Mรฉmoli's topological stability theorem ensures that local non-linear distortions remain strictly bounded, yielding a robust initial coupling \(\Gamma^{(0)}\).
3. Global Alignment Expansion and Mutual Nearest Neighbor Refinement: Defeating High-Dimensional Hubness
To extrapolate the anchor coupling \(\Gamma^{(0)}\) to the entire vocabulary, PMA first solves an orthogonal Procrustes problem via Singular Value Decomposition (SVD):
producing a global orthogonal rotation \(Q_0\). However, high-dimensional vector spaces inherently suffer from the hubness phenomenon, where a tiny subset of vectors acts as the nearest neighbor to an unnaturally large number of target tokens. Unconstrained nearest-neighbor matching would cause many-to-one collapse, violating the one-to-one physical nature of E2EO. PMA counters this by implementing an iterative Mutual Nearest Neighbor (MNN) self-learning protocol. At round \(r\), assignment functions \(\pi_P\) and \(\pi_C\) identify bidirectional nearest neighbors:
Only mutually matching pairs \(P^{(r)} = \{(i,j) \mid \pi_P(i)=j, \pi_C(j)=i\}\) with corresponding indices \(I_P^{(r)}\) and \(I_C^{(r)}\) are retained to recompute the orthogonal transformation:
This iterative refinement filters out spurious hub connections, expands the alignment across the full vocabulary within 5 rounds, and enables definitive standard nearest-neighbor inversion back to plaintext tokens.
Key Experimental Results¶
Main Results¶
The authors evaluated PMA across SST-2 and Alpaca using 6 representative model architectures spanning BPE, WordPiece, and SentencePiece tokenizers against 5 state-of-the-art E2EO mechanisms and a static random mapping baseline. The table below excerpts key recovery metrics across diverse obfuscation defenses:
| Dataset | Target Model | Vocab Size | ObfusLM (ROUGE-L / Top-1 / Top-3) | SentinelLMs (ROUGE-L / Top-1 / Top-3) | Random Mapping (ROUGE-L / Top-1 / Top-3) | Avg Time (min) |
|---|---|---|---|---|---|---|
| SST-2 | BERT-base | 30k | 98.8 / 89.6 / 93.4 | 98.8 / 89.7 / 93.5 | 98.8 / 89.7 / 93.5 | 3.38 |
| SST-2 | LLaMA-2-7B | 32k | 99.5 / 94.7 / 96.2 | 99.6 / 94.7 / 96.3 | 99.5 / 94.7 / 96.3 | 3.43 |
| SST-2 | LLaMA-3-8B | 128k | 97.0 / 78.0 / 83.5 | 97.1 / 77.8 / 83.5 | 97.1 / 77.8 / 83.6 | 3.42 |
| SST-2 | Gemma-7B | 256k | 95.4 / 70.4 / 77.5 | 95.3 / 70.4 / 77.5 | 95.2 / 70.4 / 77.5 | 3.45 |
| Alpaca | BERT-base | 30k | 93.8 / 45.3 / 51.8 | 94.8 / 45.5 / 51.7 | 94.6 / 45.5 / 52.0 | 4.17 |
| Alpaca | LLaMA-2-7B | 32k | 96.8 / 57.4 / 63.8 | 97.0 / 57.7 / 63.9 | 96.5 / 57.8 / 64.1 | 4.29 |
| Alpaca | LLaMA-3-8B | 128k | 89.8 / 28.5 / 34.5 | 90.2 / 28.6 / 34.6 | 89.9 / 28.5 / 34.5 | 4.77 |
| Alpaca | Gemma-7B | 256k | 89.6 / 21.6 / 26.3 | 90.2 / 21.6 / 26.3 | 89.4 / 21.6 / 26.5 | 4.92 |
When benchmarked directly against competing attack paradigms on ObfusLM, PMA demonstrated absolute dominance: on SST-2, baseline attacks KNN, TFA, InvBert, and KNN+SDA scored 24.69, 26.30, 35.59, and 41.47 ROUGE-L respectively, whereas PMA attained 98.80; on Alpaca, KNN, TFA, and ERA achieved 27.47, 32.40, and 35.53 ROUGE-L, while PMA reached 89.77 (28.49 Top-1, 34.50 Top-3).
Ablation Study¶
The paper rigorously analyzed the impact of embedding architectures, manifold alignment algorithms, and cross-domain reference corpora:
| Evaluation Dimension / Configuration | Test Setting | ROUGE-L | Top-1 | Time (min) | Key Finding & Mechanism |
|---|---|---|---|---|---|
| Word Embedding Architecture (Table 7) | |||||
| - Skip-gram | C4 \(\to\) Hacker News | 55.03 | - | - | Highly sensitive to sparse co-occurrences under limited observations |
| - GloVe | C4 \(\to\) Hacker News | 83.63 | - | - | Stable matrix factorization, but slightly trailing CBOW in local non-linear fidelity |
| - CBOW (Ours) | C4 \(\to\) Hacker News | 86.90 | - | 4.94 | Delivers the most resilient semantic manifold estimation for natural text |
| Bilingual Alignment Algorithm (Table 8) | |||||
| - MUSE (Adversarial) | C4 \(\to\) Hacker News | 0.03 | - | 7.33 | Complete optimization collapse due to discrete distribution mismatch |
| - VecMap (Self-learning) | C4 \(\to\) Hacker News | 86.70 | - | 64.12 | Matches accuracy but requires prohibitive full-vocabulary search and repeated SVD |
| - Gromov-Wasserstein (Ours) | C4 \(\to\) Hacker News | 86.90 | - | 4.94 | Achieves optimal efficiency-accuracy trade-off via anchor-level transport |
| Cross-Domain Reference Transfer (Table 5) | |||||
| - Cross-News (CNN News \(\leftarrow\) C4) | Real news text | 93.0 | 68.1 | - | General web crawl captures sufficient semantic geometry for news text |
| - Cross-Finance (Hacker News \(\leftarrow\) C4) | Mixed finance | 86.9 | 36.2 | - | General text provides robust sequence-level semantic recovery |
| - Cross-Finance (Hacker News \(\leftarrow\) C4-Fin) | In-domain finance | 94.1 | 54.1 | - | Domain matching enhances token-level Top-1 precision by +17.9% |
| - Cross-Code (Stack V2 \(\leftarrow\) C4) | Source code | 4.9 | 0.02 | - | Catastrophic failure due to disjoint semantic structures between code and prose |
| - Cross-Code (Stack V2 \(\leftarrow\) CodeParrot) | Source code | 99.1 | 51.7 | - | Restoring in-domain code reference recovers full performance |
Key Findings¶
- Phase Transition at 2 Million Tokens: Plaintext reconstruction undergoes a rapid phase transition, saturating after observing roughly 2 million ciphertext tokens across diverse benchmarks (Figure 4). At this scale, empirical distances among high-frequency tokens stabilize geometrically.
- Hyperparameter Robustness: Varying anchor size \(K\) from 1,000 to 3,000 yields negligible performance fluctuations across LLaMA-2, RoBERTa, LLaMA-3, and Gemma (Figure 5). Furthermore, iterative refinement converges reliably by the 5th round, requiring only 3โ5 minutes on a single commodity GPU (NVIDIA L20).
- Immunity to Obfuscation Mechanism: Regardless of whether E2EO schemes apply metric-preserving geometric transforms (SentinelLMs), clustering permutations (ObfusLM), or purely random static replacements (Random Mapping), PMA achieves virtually identical recovery scores (differences < 0.5%), proving that altering numerical values provides zero security against structural manifold alignment.
Highlights & Insights¶
- Reframing Cryptanalysis as Cross-Lingual Lexicon Induction: The central insight is recognizing that deterministic E2EO schemes function as large-scale substitution ciphers that preserve sentence topology. Treating the obfuscated stream as a foreign tokenizer-language bypasses all numerical defenses.
- Self-Supervised Domain Adaptation: When attacking specialized verticals (e.g., source code or finance), attackers can evaluate the Gromov-Wasserstein distance itself as an intrinsic metric of manifold discrepancy, switching reference corpora dynamically to recover performance (e.g., elevating code recovery from 4.9% to 99.1%).
- Exposing the High-Utility Fallacy of Bijective Obfuscation: The study exposes an inescapable trade-off in privacy-preserving inference: any lightweight obfuscation scheme that insists on preserving one-to-one token mappings to maintain server-side model utility inevitably leaks the complete semantic manifold of the private text.
Limitations & Future Work¶
- Reliance on Bijective Mappings: PMA is explicitly formulated to defeat one-to-one substitution mechanisms; it cannot directly break dynamic one-to-many mappings, stochastic multi-token routing, or homomorphic additive noise.
- Rare Tokens and High-Entropy Sequences: Recovery on extreme long-tail tokens, typographical anomalies, and high-entropy sequences (e.g., cryptographic keys or random alphanumerics) remains vulnerable to alignment ambiguity due to the absence of dense co-occurrence contexts.
Related Work & Insights¶
- vs Embedding Inversion Attacks (KNN, InvBert, ERA): Traditional inversion relies strictly on preserved local Euclidean coordinates and supervised training data; PMA operates entirely unsupervised without model weights or coordinate preservation.
- vs Token Frequency Attacks (TFA): TFA relies on fragile one-dimensional unigram frequency ranks, which degrade severely on large vocabularies with long-tail distributions; PMA exploits high-dimensional 300-d geometric manifold structures, delivering vastly superior precision.
Rating¶
- Novelty: โญโญโญโญโญ Formulates an original, mathematically principled unsupervised translation perspective against E2EO.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage across 6 obfuscators, 6 LLM architectures, 7 datasets, and extensive cross-domain/cross-tokenizer ablations.
- Writing Quality: โญโญโญโญโญ Exemplary mathematical rigor bridging empirical NLP, Gromov-Wasserstein optimal transport, and empirical metric convergence.
- Value: โญโญโญโญโญ Crucial security wake-up call dismantling the false sense of security offered by deterministic embedding obfuscation.