Spectral-Sphere-Constrained Hyper-Connections

Conference: NeurIPS2026 (list assignment; this note follows arXiv v2)
arXiv: 2603.20896
Code: https://github.com/6zHAOyi/s2HC
Area: Pretraining / Optimization & Theory
Keywords: hyper-connections, spectral norm constraint, mean preservation, dynamic stream mixing, language model pretraining

TL;DR

s²HC replaces nonnegative doubly stochastic constraints on multi-stream residual matrices with a mean-preserving spectral norm sphere, using dynamic rotations and bounded scaling in the zero-sum subspace to enable non-degenerate mixing and achieving average accuracy of 47.1, 50.2, and 50.7 across eight benchmarks on three language models pretrained from scratch.

Background & Motivation

A standard residual connection preserves an identity path while adding the output of an attention or MLP branch to existing features. Hyper-Connections (HC) expand this path into parallel feature streams and exchange information through input-dependent matrices. The additional freedom concerns connectivity rather than simply a wider attention block. However, products of unconstrained matrices across depth can amplify signals. mHC therefore requires nonnegative mixing matrices with unit row and column sums, preserving the cross-stream mean while fixing the matrix spectral norm to 1.

This constraint has two kinds of cost. Computationally, finitely many Sinkhorn–Knopp (SK) iterations only approximate double stochasticity. mHC-lite enforces it exactly through convex combinations of all permutation matrices, but its parameterization grows factorially; KromHC uses Kronecker factorization for efficiency at the expense of a more restricted representation space. Representationally, this paper observes that the trained matrices in Qwen3 remain close to their identity initialization, with stream-similarity trajectories resembling a baseline without residual mixing. The proposed explanation is that, under standard mixing conditions, doubly stochastic operations tend to attenuate off-mean directions together, potentially encouraging the model to avoid mixing to prevent homogenization. This is an empirically supported mechanism hypothesis, not a theorem that every doubly stochastic matrix must degenerate; permutation matrices provide non-attenuating counterexamples.

The paper consequently separates stability from entrywise nonnegativity and controls the mixing spectrum directly. Row and column constraints preserve the shared mean direction, while the remaining directions can rotate, flip, or selectively attenuate without linear norm amplification. Core idea: parameterize mean preservation separately from deviation mixing, bounding only the maximum gain in the deviation subspace so that multi-stream interaction need not retreat toward the identity to remain stable.

Method

Overall Architecture

The input consists of multiple hidden-state streams for each token, written as \(X_l\in\mathbb{R}^{n\times C}\), where \(n\) is the stream count and \(C\) is the width of one stream. s²HC retains the two paths of HC: one mixes residual streams, while the other aggregates them into an attention or MLP input and writes its output back to the streams. Only residual-matrix generation changes; branch read/write gates follow mHC.

The new components proceed through “Mean–Deviation Decoupling,” “Dynamic Spectral Parameterization,” and then “Residual–Branch Merge.” Decoupling supplies a fixed mean projection and zero-sum basis. Dynamic parameterization generates two rotations and a set of scaling coefficients from the current hidden state, then constructs a valid residual matrix. This generation runs during both training and inference; the method does not learn a single global matrix for direct reuse at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    X["Current token<br/>multi-stream hidden state"] --> B["Mean–Deviation Decoupling<br/>fixed mean projection and zero-sum basis"]
    B --> C["Dynamic Spectral Parameterization<br/>rotations and bounded scaling"]
    X -->|generated from normalized state| C
    C -->|current residual matrix| D["Residual–Branch Merge<br/>mixing and branch read/write"]
    X -->|original features| D
    D --> Y["Next-layer multi-stream state"]
    Y -.->|training only via language model head| L["Language modeling loss"]
    T["Next-token target"] -.->|training supervision only| L

Key Designs

1. Mean–Deviation Decoupling: preserve the shared component and reserve mixing freedom for stream differences

Unit row sums leave an input shared by all streams unchanged, while unit column sums preserve the average across output streams. Together, these constraints separate the mean direction from zero-sum deviations. The authors use \(J=\frac{1}{n}\mathbf{1}_n\mathbf{1}_n^{\top}\) for the mean projection and write \(\mathcal{H}_l^{\mathrm{res}}=J+\mathcal{H}_l^{\mathrm{disp}}\), with \(\mathcal{H}_l^{\mathrm{disp}}\mathbf{1}_n=0\) and \(\mathbf{1}_n^{\top}\mathcal{H}_l^{\mathrm{disp}}=0\). The displacement here is a linear operator on cross-stream deviations, not an additional constant injected into the features.

For any feature channel, \(J\) retains only the common mean, while the displacement handles differences orthogonal to it. Their outputs remain in mutually orthogonal subspaces. The key relation is therefore \(\|\mathcal{H}_l^{\mathrm{res}}\|_2=\max(1,\|\mathcal{H}_l^{\mathrm{disp}}\|_2)\). The apparently difficult requirement of an exactly unit spectral norm reduces to bounding the displacement norm by 1. Allowing negative entries does not invalidate this relation or residual-mixing mean preservation.

This explains why the method does not simply divide an arbitrary matrix by its largest singular value: that could disturb the mean direction. Instead, it fixes that direction and learns only in its orthogonal complement. A fixed truncated Helmert matrix \(U_{\mathcal{Z}}\in\mathbb{R}^{n\times(n-1)}\) supplies an orthonormal basis for the zero-sum subspace. It is neither a data-learned projection nor an operation that averages away all stream differences.

2. Dynamic Spectral Parameterization: learn mixing directions separately from attenuation strength

In this basis, the displacement is equivalent to a small \((n-1)\times(n-1)\) matrix. The final construction is \(\mathcal{H}_l^{\mathrm{res}}=J+(U_{\mathcal{Z}}U_l^{\mathrm{core}})\Sigma_l(U_{\mathcal{Z}}V_l^{\mathrm{core}})^{\top}\). Both core matrices are orthogonal: the right factor selects deviation directions to read, the diagonal controls their gains, and the left factor determines how they are written back across streams. Separate input and output rotations avoid restricting the operator to symmetric scaling.

The implementation first flattens and RMSNorm-normalizes all streams of the current token. Three linear projections then generate two sets of rotation parameters and one set of diagonal coefficients. Rotation parameters pass through tanh with learnable scales and magnitude gates, populate the upper triangle of a skew-symmetric matrix, and receive opposite signs below the diagonal. A Cayley transform produces each orthogonal core matrix. Each rotation requires only \(k=\frac{1}{2}(n-1)(n-2)\) independent outputs; the diagonal has \(n-1\) outputs whose tanh activation bounds their absolute values by 1. This generates a matrix in a spectral-factorization form rather than running a numerical SVD of an arbitrary matrix for every token.

The paper calls the tanh diagonal entries “singular values,” but they can be negative. They are more precisely signed diagonal coefficients; the actual singular values are their absolute values, and signs can be absorbed into the corresponding directions. Norm control remains valid, while negative signs allow direction flips. Selective attenuation is not eliminated: the method removes the coupling imposed by nonnegative double stochasticity, but deviation directions can still only be preserved or attenuated, not amplified.

Theoretical and implementation coverage should be distinguished. Appendix D.4 proves completeness for an abstract representation using arbitrary orthogonal core matrices and a diagonal spectrum. The standard Cayley transform with finite parameters does not cover every orthogonal matrix: rotations with eigenvalue \(-1\) are not directly attainable through this chart, and reflections are not its direct outputs. Signed diagonal coefficients may compensate for some restrictions, but the abstract SVD argument does not by itself establish finite-parameter coverage of the entire closed target set by the specific dynamic generator.

3. Residual–Branch Merge: retain HC read/write interfaces rather than widening every computational block

The full update remains \(X_{l+1}=\mathcal{H}_l^{\mathrm{res}}X_l+(\mathcal{H}_l^{\mathrm{post}})^{\top}\mathcal{F}(\mathcal{H}_l^{\mathrm{pre}}X_l,\mathcal{W}_l)\). The residual path mixes the original multi-stream features directly. The computational branch dynamically aggregates them into one stream through the pre gate, applies the existing attention or MLP, and distributes the result through the post gate. Consequently, mean preservation describes residual mixing only; newly added branch content can still change the next-layer feature mean.

The pre gate uses sigmoid and the post gate uses twice sigmoid, both generated from normalized hidden states as in mHC. This makes the main comparison relatively focused on the residual-matrix constraint rather than simultaneous branch-network changes. These gates, however, are not themselves controlled by the new spectral constraint.

Multiplicative closure in Appendix D.1 establishes that, with the generated matrices held fixed, their residual-mixing product still preserves the mean and has spectral norm 1. This concerns the linear residual path, not a bound on the full network Jacobian. Because the matrices depend on the input, differentiation includes generator derivatives; nonlinear branches and dynamic pre/post gates introduce further terms. Thus, the non-amplification of fixed linear residual mixing is justified, whereas an input-dependent network being universally free from exploding gradients does not follow from that proof.

A Worked Example

In the four-stream main-experiment configuration, each token has four vectors of width \(C\). Besides the fixed mean direction, three independent deviation directions remain. Each rotation produces 3 parameters and diagonal scaling produces another 3, yielding 9 dynamic residual parameters that reconstruct a \(4\times4\) matrix.

If the actual singular values of the three deviation directions are close to 1, 1, and 0.77, the shared mean remains unchanged, two selected deviation directions are approximately preserved, and the third is reduced to 0.77 of its amplitude. These numbers illustrate the spectral behavior described in Figure 5; they do not imply identical coefficients at every layer. Streams can recombine, potentially with negative weights, while the residual linear operator does not amplify the Euclidean norm.

The pre gate then reads the four streams into one attention or MLP input. The post gate writes the result back, which is added to the mixed residual state. Rotations, scaling, and read/write gates are regenerated when the token or layer input changes. Inference retains this procedure without requiring ground-truth answers for gating.

Loss & Training

The paper changes connectivity and introduces no auxiliary loss; training remains autoregressive language modeling. The experiments do not fine-tune pretrained Qwen or Gemma weights. They reuse the architectures and train from scratch on FineWebEdu. Qwen3-0.6B has approximately 0.75B parameters after untying word embeddings, while Gemma3-1B and Qwen2.5-1.5B retain tied embeddings. Their token budgets are 15B, 20B, and 30B, respectively.

Initialization sets both Cayley core matrices to identity, diagonal biases to 4, projection weights to zero, rotation magnitude gates to 1, and all three scales to 0.01. The paper describes this as exact identity residual initialization, but \(\tanh(4)<1\) for finite real parameters, so the stated formula yields only a near-identity matrix. The full text does not clarify whether numerical saturation or additional implementation handling makes it exact; the description alone does not establish exact identity.

Training uses AdamW, bfloat16, cosine learning rate decay, 1000 iterations of linear warmup, and gradient clipping at 1.0. Sequence length is 2048. Qwen3 starts at \(6\times10^{-4}\), while the other two start at \(4\times10^{-4}\); minimum learning rates are one tenth of the corresponding initial rates. Appendix Table 4 gives global batch sizes of 512, 512, and 1024.

Evaluation uses lm-eval harness: PIQA, BoolQ, MMLU, and TruthfulQA are 0-shot; WinoGrande is 5-shot; HellaSwag is 10-shot; ARC-E/C are 25-shot. MMLU specifically uses 0-shot to avoid exceeding the context length, so these scores should not be treated as conventional 5-shot MMLU results.

Key Experimental Results

Main Results

The following table selects eight-benchmark average accuracy from Table 1, in percent. Gains are percentage points against the best doubly stochastic HC baseline for the same architecture, not against an external leaderboard SOTA.

Architecture trained from scratch	RC	mHC	mHC-lite	KromHC	s²HC	Gain over best doubly stochastic HC
Qwen3, approximately 0.75B	46.1	46.1	45.8	45.3	47.1	+1.0
Gemma3, 1B	48.9	49.4	49.9	50.0	50.2	+0.2
Qwen2.5, 1.5B	49.0	49.5	49.6	49.5	50.7	+1.1

All HC capability comparisons use 4 streams. Average results improve on all three models, but individual tasks do not always improve: Gemma3 ARC-C is 36.0 versus KromHC's 36.6, and Qwen3 TruthfulQA is 34.4 versus RC's 36.7.

Numerical reporting uncertainty: the eight displayed Qwen2.5 s²HC scores are 55.2, 62.2, 71.6, 51.5, 66.5, 37.0, 25.1, and 35.7, whose simple average is 50.6, whereas the paper reports 50.7 in the Avg. column. This note retains the reported 50.7 and corresponding +1.1 rather than correcting the table. The paper does not explain whether the difference arises from undisplayed precision or the averaging convention.

Ablation Study

The following results are selected from Appendix Tables 2 and 3, in accuracy percent. “Fixed identity” retains multiple streams and dynamic pre/post gates but removes learnable residual mixing. “Orthogonal” generates an orthogonal matrix directly in the full stream space without preserving the mean direction.

Architecture	Residual mixing	ARC-E	ARC-C	MMLU
Qwen3	Fixed identity	58.9	31.1	24.0
Qwen3	Orthogonal	59.8	30.4	24.3
Qwen3	s²HC	59.8	30.9	25.6
Gemma3	Fixed identity	67.0	35.4	24.3
Gemma3	Orthogonal	65.7	33.8	25.2
Gemma3	s²HC	67.5	36.0	25.3
Qwen2.5	Fixed identity	65.3	35.1	25.2
Qwen2.5	Orthogonal	66.5	33.7	26.2
Qwen2.5	s²HC	66.5	37.0	25.1

Dynamic mixing does not beat fixed identity on every entry: Qwen3 ARC-C decreases from 31.1 to 30.9, and Qwen2.5 MMLU decreases from 25.2 to 25.1. The orthogonal ablation changes both mean preservation and spectral attenuation, so its outcome cannot be attributed solely to allowing singular values below 1.

Key Findings

• Figure 5 shows two leading Qwen3 deviation singular values close to 1 and the smallest falling to 0.77, supporting selective preservation or attenuation rather than a requirement that streams become increasingly dissimilar everywhere.
• Figure 7 uses 1024 samples: after 20 SK iterations, mHC single-layer column sums can peak at 2.0, with outliers reaching 3.6 after 28-layer composition. s²HC and exact doubly stochastic methods keep cumulative spectral norms near 1. These remain residual-matrix statistics, not full-network Jacobian measurements.
• Throughput experiments use 4 H200 GPUs, sequence length 2048, and global batch size 512. At 7 streams, mHC-lite introduces approximately 2B auxiliary parameters and runs out of memory, versus approximately 26M for mHC and 20M for s²HC. This establishes parameterization scalability, not universally better capability with more streams.
• The authors note that mHC throughput comes from a PyTorch reimplementation and may underestimate specialized-kernel performance. The throughput micro-batch of 8 and the Qwen2.5 main-training micro-batch of 4 in the appendix belong to different experimental settings, not one universal configuration.

Highlights & Insights

• Stability requires an invariant mean direction and controlled operator gain, not necessarily entrywise nonnegativity. Separating the component that must remain unchanged before learning the remaining subspace gives a more interpretable constraint than whole-matrix normalization.
• Non-degenerate mixing and non-amplification can coexist: rotations redistribute information, while selective attenuation filters particular deviations. Stable gradients or near-identity matrices alone do not reveal whether multi-stream connectivity actually uses its additional freedom.
• The two rotations and diagonal spectrum produce \((n-1)^2\) outputs in total, with projection weight size \(nC(n-1)^2\), cubic in stream count for fixed \(C\). This removes factorial growth but not the storage and bandwidth cost of multi-stream states.

Limitations & Future Work

• The authors acknowledge persistent memory-access overhead and leave larger models and token budgets for future work. Capability experiments cover approximately 0.75B, 1B, and 1.5B architectures with 4 streams, not large-scale long-duration pretraining.
• Tables provide no multi-seed error bars or significance tests. The Gemma3 average gain is only 0.2 percentage points, making repeated runs particularly important for assessing robustness.
• An exact linear spectral constraint does not guarantee stability of the full dynamic network, and training also uses gradient clipping. Future analysis should separate contributions from matrix generators, read/write gates, and nonlinear branches to the total Jacobian.
• Abstract representation completeness, finite Cayley-chart coverage, and whether tanh initialization exactly reaches identity require clearer theoretical and implementation explanations. Alternative orthogonal parameterizations and explicitly near-identity initialization warrant comparison.
• The orthogonal ablation lacks a control that preserves the mean while fixing all deviation singular values to 1. Such a control would isolate selective attenuation more cleanly.

Related Work & Insights

• vs HC: HC allows dynamic residual mixing, but unconstrained matrices may amplify signals. s²HC retains dynamic mixing while directly bounding linear gain outside the mean direction.
• vs mHC: mHC obtains spectral control indirectly through nonnegative doubly stochastic matrices and SK iterations. s²HC removes nonnegativity and uses a fixed zero-sum basis with bounded spectrum for algebraic constraints, although floating-point error still matters.
• vs mHC-lite / KromHC: The former achieves exact double stochasticity through permutation mixtures, while the latter reduces cost through Kronecker factorization. s²HC needs neither permutation enumeration nor conveniently factorable stream counts, but its projection parameters still grow cubically with stream count.
• Research insight: Preserving shared information and allowing directional recombination can be treated as separate constraints in other multi-branch mixing modules. The task-specific shared direction must be identified rather than automatically copying the all-ones vector.

Rating

• Novelty: 4/5. Separates double stochasticity into mean preservation and deviation-spectrum control with a concrete dynamic parameterization.
• Experimental Thoroughness: 3/5. Three architectures and two targeted ablations are useful, but scale, repeated seeds, and isolated controls remain limited.
• Writing Quality: 3/5. Geometry and empirical evidence align well, but singular-value terminology, exact initialization, and stability claims need qualification.
• Value: 4/5. Offers a compact multi-stream connectivity design worth further pretraining and systems validation.