Manifold Embedding: A Point-to-Hyperplane Approach¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://aclanthology.org/2026.emnlp-main.1532/
Area: LLM Reasoning
Keywords: Manifold embedding, Non-Euclidean neural networks, Point-to-hyperplane distance, Constant curvature geometry, Parameter-efficient fine-tuning
TL;DR¶
This paper introduces the Point-to-Hyperplane Distance Map (PHDM), an auditable, parameter-free Euclidean-to-manifold mapping that interprets each Euclidean feature scalar as an exact signed geodesic distance to a canonical orthogonal hyperplane, providing closed-form solutions across four constant-curvature geometries that consistently outperform exponential map baselines in LLM arithmetic reasoning, image classification, and graph learning.
Background & Motivation¶
Non-Euclidean Riemannian manifolds—predominantly hyperbolic spaces (the Lorentz hyperboloid and Poincaré ball) for negative curvature and spherical spaces (the embedded sphere and projected sphere) for positive curvature—have emerged as powerful inductive biases across representation learning, capturing latent tree-like hierarchies, cyclic dependencies, and asymmetric relationships with minimal distortion. In contemporary manifold-valued architectures such as Hyperbolic LoRA (HypLoRA) for large language models, Hyperbolic CNNs (HCNN), and Riemannian graph transformers (Hypformer), a foundational preliminary step is mapping continuous representations from an upstream Euclidean encoder onto the target manifold surface. However, virtually all established systems rely on the origin exponential map (\(\text{exp}_{\mathbf{0}}(v)\)) or ambient projection operators, both of which operate primarily through local Euclidean tangent spaces or unconstrained ambient ambient geometry before imposing structural constraints.
This reliance on tangent-space operators exposes a conceptual and geometric disconnect. In classical Euclidean networks, the scalar components of a fully connected layer output inherently represent signed orthogonal distances to coordinate hyperplanes. When ported to non-Euclidean manifolds via an exponential map, these scalar outputs are instead treated as unconstrained tangent-vector components along radial geodesics. Because the Riemannian logarithmic map is not an isometry across arbitrary pairs of manifold points, tangent-space pseudo-distances deviate substantially from true intrinsic geodesic distances on the manifold, frequently precipitating geometric distortion, representation saturation, and catastrophic numerical collapse near manifold boundaries. While previous work explored point-to-hyperplane formulations in output classification layers (e.g., HNN++ and RMLR), hyperplanes were treated as learnable decision boundaries rather than foundational building blocks for parameter-free input embedding.
To eliminate this fundamental mapping distortion, the authors re-examine the geometric duality between Euclidean projections and manifold coordinates. By aligning the scalar outputs of an upstream linear transformation with the intrinsic signed distances to a set of fixed, canonical coordinate hyperplanes, the manifold embedding problem can be formulated as a system of distance equations. Core idea: interpret Euclidean feature components as exact signed geodesic distances to canonical coordinate hyperplanes on constant-curvature manifolds, deriving closed-form analytic solutions (PHDM) across Lorentz, Poincaré, spherical, and projected-spherical models to replace exponential maps with a parameter-free, auditable, and distance-preserving input interface.
Method¶
Overall Architecture¶
PHDM cleanly decouples upstream trainable score generation from intrinsic manifold coordinate decoding. Given an input representation \(x \in \mathbb{R}^d\), any upstream Euclidean module (e.g., a standard linear layer or low-rank adapter) computes an intermediate score vector \(v(x) = Wx + b \in \mathbb{R}^n\). Rather than interpreting \(v(x)\) as a tangent vector at the origin, PHDM enforces a set of exact signed point-to-hyperplane distance constraints \(d_{\mathcal{M}}^{\pm}(y, \mathcal{H}_{e_{(k)}, 0}^{\mathcal{M}}) = v_k(x)\) with respect to canonical orthogonal hyperplanes \(\mathcal{H}_{e_{(k)}, 0}^{\mathcal{M}}\). The target manifold coordinate \(y\) is then reconstructed in closed form across two constant-curvature families—radius models (\(\mathcal{M}_K^n\)) and stereographic models (\(\text{st}_K^n\))—and subsequently processed by downstream manifold layers before projecting back into the Euclidean residual stream via Riemannian multinomial logistic regression (RMLR).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Euclidean input feature x"] --> B["Upstream linear projection<br/>vk(x) = ⟨wk, x⟩ - bk"]
B --> C["Canonical hyperplane distance semantics<br/>Enforce exact signed geodesic distances"]
C --> D["Dual-family closed-form reconstruction<br/>Radius and stereographic analytic solvers"]
D --> E["Parameter-free modular drop-in<br/>Zero-overhead replacement for ExpMap"]
E --> F["Downstream manifold layers and PEFT residual"]
Key Designs¶
1. Canonical Hyperplane Distance Semantics: From Tangent Pseudo-Distances to True Geodesic Distances
Addressing the geometric distortion inherent in origin exponential maps, this design anchors the interpretation of Euclidean feature outputs to intrinsic Riemannian distance metrics. In a standard Euclidean fully connected layer, the \(k\)-th coordinate is intrinsically an oriented Euclidean distance to the hyperplane normal to \(e_{(k)}\): $\(v_k(x) = \text{sign}(\langle e_{(k)}, y \rangle) d(y, \mathcal{H}_{e_{(k)}, 0}) = \langle w_k, x \rangle - b_k\)$ Prior methods employing \(\text{exp}_{\mathbf{0}}(u)\) only recover signed tangent-space pseudo-distances \(\tilde{d}_{\mathcal{M}}^{\pm}(y, \mathcal{H}_{e_{(k)}, 0}^{\mathcal{M}}) = \langle \text{log}_{\mathbf{0}}(y), \bar{e}_{(k)} \rangle_{\mathbf{0}}\), which diverge from true intrinsic infimum distances \(d_{\mathcal{M}}(y, \mathcal{H}) = \inf_{z \in \mathcal{H}} d_{\mathcal{M}}(y, z)\) under non-zero curvature. PHDM resolves this discrepancy by strictly enforcing: $\(d_{\mathcal{M}}^{\pm}(y, \mathcal{H}_{e_{(k)}, 0}^{\mathcal{M}}) = v_k(x), \quad k = 1, \dots, n\)$ This condition guarantees that every scalar component is an auditable, verifiable physical distance on the manifold, effectively transforming the Euclidean-to-manifold bridge from a heuristic radial projection into an exact, geometrically consistent coordinate reconstruction.
2. Dual-Family Closed-Form Reconstruction: Unified Analytic Solutions Across Four Geometries
Addressing the computational bottleneck of solving coupled non-linear distance equations over curved geometries, the authors derive exact, closed-form analytic solutions for four constant-curvature manifolds unified into two structural families. Define the curvature-aware distance-inversion kernel \(r_K(v) \in \mathbb{R}^n\): for curvature \(K < 0\), \(r_K(v) = (-K)^{-1/2} \sinh(\sqrt{-K} v)\); for curvature \(K > 0\), \(r_K(v) = K^{-1/2} \sin(\sqrt{K} v)\). - Radius Model Family \(\mathcal{M}_K^n\) (Lorentz hyperboloid \(\mathcal{L}_K^n\) for \(K < 0\), ambient sphere \(\mathcal{S}_K^n\) for \(K > 0\)): Defined as \(\{x \in \mathbb{R}^{n+1} \mid \langle x, x \rangle_K = 1/K\}\), the canonical hyperplanes are \(\mathcal{H}_{e_{(k)}, 0} = \{y \in \mathcal{M}_K^n \mid \langle e_{(k)}, y \rangle_K = 0\}\). The principal signed distance equation is \(d^{\pm}(y, \mathcal{H}_{e_{(k)}, 0}) = \frac{1}{\sqrt{|K|}} \sin_K^{-1}(\sqrt{|K|} y_k) = v_k\), yielding spatial coordinates \(y_k = r_{K, k}(v)\) directly. The remaining time/pole coordinate is uniquely recovered via the manifold inner product constraint \(\langle y, y \rangle_K = 1/K\): $\(\text{PHDM}_{\mathcal{M}_K}(v) = \left[ \sqrt{\frac{1}{K} - \text{sgn}(K) \|r_K(v)\|_2^2}, \; r_K(v) \right]\)$ - Stereographic Model Family \(\text{st}_K^n\) (Poincaré ball \(\mathcal{P}_K^n\) for \(K < 0\), projected sphere \(\mathcal{D}_K^n\) for \(K > 0\)): With conformal metric \(g_x^K = (\lambda_x^K)^2 I\), the signed distance is \(d^{\pm}(y, \mathcal{H}_{e_{(k)}, 0}) = \frac{1}{\sqrt{|K|}} \sin_K^{-1}\left(\frac{2\sqrt{|K|}y_k}{1 + K\|y\|_2^2}\right) = v_k\). Vectorizing yields \(\frac{2y}{1 + K\|y\|_2^2} = r_K(v)\). Because \(y\) must lie along the ray spanned by \(r_K(v)\), setting \(y = \alpha r_K(v)\) with \(s = \|r_K(v)\|_2^2\) produces the quadratic equation \(K s \alpha^2 - 2\alpha + 1 = 0\). Selecting the admissible branch root satisfying \(y = 0\) when \(v = 0\) gives \(\alpha = \frac{1}{1 + \sqrt{1 - Ks}}\), leading to the compact unified formulation: $\(\text{PHDM}_{\text{st}_K}(v) = \frac{r_K(v)}{1 + \sqrt{1 - K\|r_K(v)\|_2^2}}\)$ Both derivations mathematically establish existence, uniqueness, and injectivity on their stated feasible domains, ensuring high-throughput forward and backward propagation without iterative solvers.
3. Parameter-Free Modular Drop-in: Seamless Replacement for Parameter-Efficient Tuning
Addressing the practical difficulty of deploying non-Euclidean layers without introducing architectural instability or parameter inflation, PHDM operates as a strictly parameter-free drop-in module. It requires no learnable hyperplane normals, scale parameters, or offsets. In PEFT architectures like HypLoRA, standard low-rank matrices project representations down to a low-dimensional bottleneck \(z \in \mathbb{R}^r\); PHDM directly maps \(z\) into hyperbolic space by treating \(z_k\) as canonical distances, where hyperbolic transformations are executed before mapping back via RMLR to update the Euclidean residual stream. Because PHDM introduces zero additional trainable parameters, any downstream performance improvement can be attributed solely to the restored geometric consistency of the input layer.
Loss & Training¶
- Controlled Replacement Paradigm: In all comparative benchmarks, the upstream encoder, downstream manifold layers, loss functions, learning rate schedules, and parameter budgets are identical to the respective baselines.
- Feasibility Domain Constraints: For positive curvature (\(K > 0\)), exact signed-distance preservation requires the component-wise condition \(\sqrt{K}|v_k| \le \pi/2\) and the joint constraint \(\sum_{k=1}^n \sin^2(\sqrt{K} v_k) \le 1\). If inputs exceed this principal feasible domain, radial scaling or norm clipping is utilized to preserve numerical stability.
- Hardware & Implementation: All training and inference runs were conducted on NVIDIA A100 64GB GPUs, with statistics averaged across five independent random seeds.
Key Experimental Results¶
Main Results¶
In the primary NLP evaluation on LLM arithmetic reasoning, PHDM LoRA was evaluated against standard Euclidean LoRA and HypLoRA across four prominent open-weight LLMs (LLaMA-13B, Gemma-7B, LLaMA3-8B, and Qwen2.5-7B) under matched parameter budgets (~0.67%–0.79% trainable parameters). Micro-average accuracy (M.AVG) reflects the dataset sample distributions across MAWPS, SVAMP, GSM8K, and AQuA:
| Base Model | PEFT Method | Trainable Params (%) | MAWPS (8.5%) | SVAMP (35.6%) | GSM8K (46.9%) | AQuA (9.0%) | M.AVG |
|---|---|---|---|---|---|---|---|
| LLaMA-13B | LoRA | 0.67 | 83.50 | 54.70 | 48.50 | 18.50 | 51.00 |
| LLaMA-13B | HypLoRA | 0.67 | 83.20 | 54.80 | 49.00 | 21.50 | 51.50 |
| LLaMA-13B | PHDM LoRA (Ours) | 0.67 | 84.45 | 61.80 | 46.02 | 22.83 | 52.79 |
| Gemma-7B | LoRA | 0.79 | 91.60 | 76.20 | 66.30 | 28.90 | 68.60 |
| Gemma-7B | HypLoRA | 0.79 | 89.50 | 78.70 | 69.50 | 32.70 | 71.20 |
| Gemma-7B | PHDM LoRA (Ours) | 0.79 | 90.34 | 79.00 | 74.37 | 56.69 | 75.77 |
| LLaMA3-8B | LoRA | 0.70 | 92.70 | 78.90 | 70.80 | 30.40 | 71.90 |
| LLaMA3-8B | HypLoRA | 0.70 | 91.60 | 80.50 | 74.00 | 34.20 | 74.20 |
| LLaMA3-8B | PHDM LoRA (Ours) | 0.70 | 91.60 | 78.00 | 76.35 | 34.65 | 74.46 |
| Qwen2.5-7B | LoRA | 0.71 | 90.80 | 84.40 | 78.60 | 68.10 | 80.80 |
| Qwen2.5-7B | HypLoRA | 0.71 | 91.20 | 92.20 | 87.90 | 71.60 | 88.30 |
| Qwen2.5-7B | PHDM LoRA (Ours) | 0.71 | 92.86 | 93.50 | 88.78 | 74.41 | 89.51 |
In graph learning benchmarks across four distinct target geometries, PHDM was applied to Hypformer (node classification) and RResNet (Cora link prediction):
| Task / Benchmark | Architecture Config | Lorentz | Poincaré | Sphere | Projected Sphere |
|---|---|---|---|---|---|
| Node Classification (Airport F1 %) | Hypformer Baseline | 95.00 ± 0.50 | 55.34 ± 8.31 | 96.29 ± 0.28 | 95.29 ± 0.21 |
| Node Classification (Airport F1 %) | Hypformer + PHDM | 96.03 ± 0.34 | 97.33 ± 0.40 | 97.86 ± 0.16 | 97.75 ± 0.16 |
| Cora Link Prediction (ROC AUC %) | RResNet Baseline | 90.68 ± 0.88 | 88.90 ± 0.20 | 90.70 ± 1.00 | 86.63 ± 3.37 |
| Cora Link Prediction (ROC AUC %) | RResNet + PHDM | 91.67 ± 0.15 | 91.42 ± 0.45 | 93.09 ± 0.44 | 89.70 ± 0.52 |
| Cora Link Prediction (ROC AUC %) | RResNet Graph Baseline | 90.12 ± 1.09 | 87.60 ± 0.90 | 91.70 ± 0.40 | 86.89 ± 0.92 |
| Cora Link Prediction (ROC AUC %) | RResNet Graph + PHDM | 91.32 ± 0.75 | 88.67 ± 1.29 | 92.24 ± 0.29 | 88.49 ± 0.50 |
Ablation Study & Operator Stability¶
To assess numerical robustness, a controlled benchmark evaluated the round-trip reconstruction accuracy across 4096 vectors sampled in 16 dimensions under FP32 precision:
| Mapping Operator | Geometry | Sampling Regime | Relative Recovery Error | Numerical Failure / Domain Violation (%) |
|---|---|---|---|---|
| PHDM (Ours) | Lorentz (\(K=-1\)) | Random radius \(r=10/50\) | 6.02 × 10⁻⁹ | 0.00% |
| Exp0 / Log0 Map | Lorentz (\(K=-1\)) | Random radius \(r=10/50\) | 5.39 × 10⁻¹ | 0.00% |
| PHDM (Ours) | Poincaré (\(K=-1\)) | Random radius \(r=10/50\) | 1.94 × 10⁻⁶ | 54.10% (boundary overflow) |
| Exp0 / Log0 Map | Poincaré (\(K=-1\)) | Random radius \(r=10/50\) | 1.36 × 10⁻¹ | 65.41% (boundary overflow) |
| PHDM (Ours) | Sphere (\(K=+1\)) | Feasible domain \(\rho=0.99\) | 1.46 × 10⁻⁸ | 0.00% |
| Exp0 / Log0 Map | Sphere (\(K=+1\)) | Feasible domain \(\rho=0.99\) | 6.28 × 10⁻⁸ | 0.00% |
| PHDM (Ours) | Proj. Sphere (\(K=+1\)) | Feasible domain \(\rho=0.99\) | 4.85 × 10⁻⁸ | 0.00% |
| Exp0 / Log0 Map | Proj. Sphere (\(K=+1\)) | Feasible domain \(\rho=0.99\) | 5.50 × 10⁻⁸ | 0.00% |
Fine-grained computational efficiency profiling during LLM tuning on an NVIDIA A100 64GB GPU reveals: - Train step latency: PHDM LoRA takes 777.4 ms compared to 784.6 ms for HypLoRA. - Train throughput: PHDM LoRA achieves 2634.3 tokens/s, representing a +7.5% throughput gain over HypLoRA (2451.6 tokens/s). - Prefill throughput: Reaches 8927.0 tokens/s (higher than both LoRA at 8886.1 and HypLoRA at 8867.1). - Peak training memory: Requires 44.34 GiB, saving 0.73 GiB of VRAM relative to HypLoRA (45.07 GiB).
Key Findings¶
- Superior Multi-Step Reasoning Capacity: PHDM LoRA achieved the highest micro-average accuracy across all four tested open-source LLMs. On Gemma-7B, micro-average score rose by +4.57 points, led by a massive +23.99 point gain on the complex AQuA algebraic reasoning benchmark (56.69% vs. 32.70%).
- Prevention of Boundary Optimization Collapse: In Airport graph node classification under Hypformer, the Poincaré exponential map baseline experienced severe numerical degradation (55.34 ± 8.31 F1), whereas Poincaré PHDM stabilized optimization and achieved 97.33 ± 0.40 F1 (+41.99 points), validating its distance-preserving robustness against boundary saturation.
- Zero Computational Overhead: By eliminating trigonometric series approximations and norm-clipping heuristics present in exponential maps, PHDM accelerates ResNet-18 image classification by 2.5%–3.5% and improves LLM training throughput by 7.5% while reducing peak memory by 0.73 GiB.
- Robustness to Curvature Sensitivity: Evaluating Cora link prediction across a 20-fold curvature magnitude span (\(|K| \in [0.5, 10]\)) showed minimal performance fluctuations (ROC-AUC variance below 0.85 across all four geometries), confirming robust practical tuning stability.
Highlights & Insights¶
- Exact Auditable Semantics: In contrast to heuristic tangent-space injections, PHDM coordinates can be verified analytically by directly evaluating signed distances back to canonical coordinate hyperplanes, establishing fully auditable input layers.
- Parameter-Free Plug-and-Play Generality: Operating purely as a parameter-free closed-form transformation, PHDM can immediately replace exponential maps across vision, NLP, and graph neural network backbones without modifying downstream architectures.
- Unifying Exponential Maps as Pseudo-Distance Approximations: The theoretical derivation rigorously reveals that origin exponential maps merely satisfy tangent-space pseudo-distance constraints, clarifying why true-distance PHDM achieves superior geometric fidelity.
Limitations & Future Work¶
- Compact Feasibility Restrictions in Positive Curvature: Spherical and projected-spherical models are exact only within their compact principal branch. Exceeding this domain necessitates heuristic clipping or radial scaling, which temporarily relaxes strict distance preservation.
- Restricted to Input Interface: PHDM operates solely as an input decoder; it does not replace intermediate Riemannian operations such as manifold attention or parallel transport, which remain bound to underlying architectural constraints.
- Future Directions: Extending the point-to-hyperplane distance framework to multi-sheet Riemannian coverings, adaptive product-space manifolds, and mixed-curvature multimodal embeddings.
Related Work & Insights¶
- vs Origin Exponential Maps (ExpMap₀, Ganea et al., 2018; Chami et al., 2019): Exponential maps project Euclidean vectors along radial geodesics in the tangent space, which only satisfies tangent pseudo-distances; PHDM directly reconstructs coordinates from true signed geodesic distances, eliminating boundary saturation and distortion.
- vs HNN++ Transformation Layers (Shimizu et al., 2021): HNN++ demonstrated the utility of point-to-hyperplane distances within Poincaré-to-Poincaré internal transformations using learned decision hyperplanes; PHDM fixes hyperplanes canonically, deriving parameter-free closed forms for Euclidean-to-manifold input decoding across both negative and positive constant-curvature spaces.
- vs Riemannian Multinomial Logistic Regression (RMLR, Chen et al., 2024): RMLR maps manifold features back to Euclidean logits via point-to-hyperplane distance scoring; PHDM forms the exact inverse geometric counterpart at the input layer, enabling end-to-end distance-preserving manifold pipelines.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an elegant, closed-form inverse distance mapping for four constant-curvature geometries, theoretically establishing exponential maps as pseudo-distance special cases.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across four LLMs, three vision datasets, four graph benchmarks, numerical stability sweeps, and fine-grained runtime/memory profiling.
- Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, with transparent proofs and comprehensive analysis spanning both negative and positive curvature families.
- Value: ⭐⭐⭐⭐⭐ An effortless, zero-parameter drop-in upgrade for any deep learning system that maps Euclidean features into non-Euclidean geometries.