Parameter symmetries determine representational geometry in overparameterized nonlinear networks

Conference: NeurIPS2026
arXiv: 2609.39078
Code: https://github.com/mrvnthss/symmetries-representational-geometry
Area: Interpretability
Keywords: parameter symmetries, representational geometry, overparameterization, symmetry orbits, minimum-norm selection

TL;DR

For one-hidden-layer nonlinear networks, this paper reduces parameter symmetries preserving the global function to feature addition, duplication, and scaling, proves that they can substantially reshape representational geometry, and gives sufficient conditions for minimum-norm selection to restore geometric identifiability.

This note covers arXiv v1, dated 2026-09-30; the conference field follows the supplied archive assignment, while the cached manuscript does not independently state acceptance.

Background & Motivation

Representational similarity analysis organizes hidden responses to a set of inputs into a similarity matrix and uses alignment between models, or between models and brain data, to infer computational mechanisms. The difficulty is that the similarity matrix observes hidden activity, whereas network output also depends on readout weights: a neuron can be highly active yet contribute nothing to the output, and the same computation can be distributed across different numbers and scales of neurons. Consequently, networks implementing exactly the same function over the entire input domain need not have similar hidden geometries. Previous deep-linear-network analyses established a double dissociation between function and representation, but nonlinear networks introduce activation boundaries and activation-specific algebraic symmetries that require a different treatment.

Merely noting that parameters are nonunique is insufficient: neuron permutations change parameters without changing the Gram matrix of hidden activity. The relevant questions are which function-preserving transformations change geometry, which features must persist, how much geometric freedom additional width creates, and which constraints remove that freedom. The paper addresses these questions in a tractable one-hidden-layer model, distinguishing agreement on finite-sample predictions from global functional equivalence and avoiding the claim that failure of one similarity measure invalidates every interpretability method.

The analysis fixes a parameter symmetry orbit: implementations connected by a specified set of function-preserving transformations are considered together, and their hidden features and geometries are characterized. This approach supports exact counterexamples while exposing the implementation constraints needed for identifiability. Core Idea: geometric ambiguity arises from adding and reweighting features; only constraints on both feature-generation and readout costs, together with removal of residual orientation-allocation freedom, can make a function identify a unique geometry within a given symmetry orbit.

Method

Overall Architecture

The model is a fully connected one-hidden-layer network with incoming weights and biases at each hidden neuron and a linear combination of hidden activities at the output. For a common set of probe inputs, activities form a hidden-activation matrix \(\mathbf H\) with neurons as rows and inputs as columns; the uncentered Gram matrix \(\mathbf M=\mathbf H^\top\mathbf H\) describes representational geometry. Each neuron contributes a rank-one matrix, so feature identity, multiplicity, and magnitude directly affect geometry.

The argument develops orbit invariants, feature primitives, an essential–auxiliary decomposition of geometry, and minimum-norm selection rules in that order. The main-text width-growth and norm-selection results concern nonlinear, positively homogeneous activations of degree one; the appendices extend the symmetry taxonomy and feature structure to other activation classes. This is a mechanism and theorem analysis rather than a multi-module training system, so a network pipeline diagram would misrepresent the proof structure.

Function preservation requires identical outputs over the entire input domain, not merely identical training labels. An irreducible implementation has minimum width within the orbit generated by the cataloged symmetries; an overparameterized implementation has a narrower counterpart in that orbit, rather than necessarily having more parameters than samples. The orbit is also not automatically the set of all implementations realizing the same function: the paper explicitly does not claim that the cataloged symmetries exhaust the full functional fiber.

Key Designs

1. Orbit invariants: aggregate computational contributions by activation boundary

For nonlinear, positively one-homogeneous activations such as ReLU, the decomposition \(\sigma(z)=\delta|z|+mz\), with \(\delta\neq0\), separates each neuron’s output into nonlinear and affine components. Combining incoming weights and bias into \(\overline{\mathbf w}_j\), any nonzero multiple of the same vector defines the same activation hyperplane and belongs to the same parameter class \(q\). The class aggregates effective readouts as \(\boldsymbol\beta_q=\sum_{j\in\mathcal J_q}\|\overline{\mathbf w}_j\|\mathbf a_j\): this preserves the nonlinear computational contribution associated with the boundary, but not the scale or division of labor among its copies. The remaining affine contributions and constant-neuron outputs form the global residual \(\mathbf r\).

Proposition 3.3 gives a necessary and sufficient orbit criterion: two implementations with the same activation are symmetry-equivalent exactly when every class aggregate and the global residual agree; absent classes have zero coefficients. Classes with nonzero coefficients are essential and cannot disappear entirely within the orbit. An essential class does not fix a particular ReLU feature orientation, however: opposite orientations can be exchanged through activation-dependent group transformations whose residuals cancel collectively. Appendix B.15 extends the criterion to other nonlinear activations using activation-specific parameter classes and coefficients.

The proof does more than appeal to abstract nonuniqueness: its sufficiency construction combines the target network with readout-negated copies of the original network to form an auxiliary implementation. Matching invariants ensure cancellation of its nonlinear contributions and total residual; after adjoining it, removing each original neuron together with its canceling copy leaves the target implementation. For linear or constant groups, joint operations preserve the function only when their total residual cancels; adding each group separately is not necessarily a symmetry.

2. Feature primitives: translate parameter changes into additions and geometric weights

Ignoring row permutations, which leave Gram matrices unchanged, every cataloged parameter symmetry acts on hidden activity through three primitives and their inverses. Addition appends feature rows, duplication repeats existing rows, and scaling multiplies rows by nonzero scalars. For example, duplicating a neuron while splitting its readout preserves output but accumulates repeated rank-one contributions; increasing ReLU incoming parameters while reciprocally decreasing the readout continuously changes the contribution’s weight at fixed width. Sign flips for odd activations only change feature signs and leave their rank-one Gram contributions unchanged, so not every feature scaling changes geometry.

Proposition 4.2 shows that every implementation in a positively one-homogeneous orbit retains at least one orientation of each essential class, together with features from nonessential classes and constant neurons, followed by duplication and positive scaling. Substituting this structure into the Gram matrix yields the essential and auxiliary decomposition in Proposition 5.1:

\[ \mathbf M= \underbrace{\sum_{\ell=1}^{|\mathcal I|}\gamma_\ell\mathbf f_\ell\mathbf f_\ell^\top}_{\text{essential}} + \underbrace{\sum_{k=1}^{K}\gamma_{|\mathcal I|+k}\mathbf u_k\mathbf u_k^\top}_{\text{auxiliary}}, \qquad \boldsymbol\gamma=\mathbf D_{\boldsymbol\nu}^\top\boldsymbol\alpha^2. \]

Here \(\mathcal I\) indexes represented essential-class orientations, \(\mathbf D_{\boldsymbol\nu}\) records duplication counts, and \(\boldsymbol\alpha^2\) is the elementwise square. For any realized feature configuration, positive homogeneity allows arbitrary strictly positive geometric weights while keeping the features, duplication counts, and width fixed. Activations without this scaling symmetry can still change integer weights through duplication, but this does not establish arbitrary continuous reweighting. Auxiliary also does not mean individually removable: some such neurons can contribute to the conserved residual and require simultaneous compensation by other groups.

3. Width growth and limits: similarity ranges depend on activation and probe inputs

The paper compares geometries using Pearson correlation \(\rho\) between strict upper-triangular entries of two representational similarity matrices (RSMs), requiring these entries to be nonconstant in both matrices. For reference matrix \(\mathbf N\), the attainable correlations at fixed width form \(\mathcal S_{N_h}(\mathbf N)\), with spread \(\Delta_{N_h}=\sup\mathcal S_{N_h}-\inf\mathcal S_{N_h}\); the corresponding all-width spread is \(\Delta_\infty\). Proposition 5.2 is stronger than saying additional neurons might increase freedom: for positively one-homogeneous activations, each additional neuron strictly increases the spread whenever the orbit-wide limit has not yet been reached.

The proof adds a zero-readout neuron, introducing any single-neuron feature realizable with that activation on the probe inputs without changing the output. Scaling the original network then produces a positive multiple of its geometry plus a nonnegative rank-one contribution from the new feature. Writing upper-triangular correlation as cosine similarity after removing the diagonal and the off-diagonal mean makes it possible to identify a feature from a wider network that improves the current similarity bound.

Proposition 5.3 uses the projected feature cone to show that limiting upper and lower correlations depend only on the activation, probe inputs, and reference geometry, not on the function realized by the orbit. Conic Carathéodory bounds the required auxiliary contributions by the projected-space dimension \(d\leq P(P-1)/2-1\), so widths at least \(N_h^\star+d\), where \(N_h^\star\) is the orbit’s minimum width, already have the all-width infimum and supremum. Reaching the limit here means equality of infima and suprema, not necessarily exact attainment by finite parameters; it also does not establish that every intermediate correlation is attainable.

If the augmented input matrix satisfies \(\operatorname{rank}(\overline{\mathbf X})=P\), every reference geometry admits correlations arbitrarily close to \(-1\) and \(1\) within every orbit; sufficient width is \(N_h\geq N_h^\star+P(P-1)/2-1\). This condition requires affine independence of probe inputs and generally needs \(P\leq N_i+1\); high input dimensionality alone does not ensure it for an arbitrary real dataset. Without this condition, the appendix gives a counterexample with one-dimensional inputs \((-1,0,1)\) and reference feature \((1,0,1)^\top\): every ReLU network has nonpositive correlation with that reference, regardless of width or function.

4. Minimum-norm selection: removing auxiliary geometry still leaves orientation freedom

The authors consider two implementation-selection rules within a fixed-width orbit, rather than propose a new supervised learning algorithm:

\[ \Omega_W=\|\mathbf W\|_F^2+\|\mathbf b\|^2+\|\mathbf A\|_F^2, \qquad \Omega_H=\|\mathbf H\|_F^2+\|\mathbf A\|_F^2. \]

Both penalize the costs of generating and reading out features, preventing computational contributions from being concealed by arbitrary activity amplification and readout shrinkage. For the weight norm, the arithmetic–geometric mean and triangle inequalities yield the lower bound \(2\sum_{q\in\mathcal E}\|\boldsymbol\beta_q\|\). Equality requires zero parameters outside essential classes, balanced incoming-parameter and readout norms for essential neurons, and effective readouts aligned with their class aggregate instead of wasting norm through cancellation.

The remaining freedom distributes each essential class’s effective readout across its positive and negative orientations: the positive fraction is \(t_q\in[0,1]\) and the negative fraction is \(1-t_q\). These allocations must reproduce the original affine residual over the entire domain and respect the width budget; an interior fraction needs two oriented neurons, whereas an endpoint needs only one. Proposition 6.1 requires the feasible split set \(\mathcal T_{N_h}\) to be nonempty before guaranteeing that the lower bound is attained and auxiliary geometry disappears. The total geometric weight of each essential class is then \(\|\boldsymbol\beta_q\|\), but its allocation between orientations can still vary with \(t_q\).

Corollary 6.2 additionally requires \(m\neq0\) and linear independence of the class-specific residual matrices \(\boldsymbol\beta_q\overline{\mathbf w}_q^\top\); the residual then uniquely determines all fractions, yielding a unique minimum-weight-norm geometry unchanged at larger widths. This is sufficient rather than universal: purely even positively homogeneous activations can have unique geometry because opposite orientations generate identical features; conversely, the ReLU tent function retains multiple geometries after minimum-weight-norm selection. Thus, the claim that norm constraints restore identifiability must retain feasibility and remaining-freedom conditions.

The representation-norm result additionally depends on the particular probe samples. Appendix F.6 requires the smaller norm \(g_q\) of each essential class’s two unit-orientation features to be strictly positive and requires a residual-compatible, width-feasible allocation using only lower-activity-norm orientations; the class geometric weight is then \(\|\boldsymbol\beta_q\|/g_q\). If a globally essential feature is inactive on every probe input, increasing incoming parameters while decreasing the readout can approach an infimum without any finite minimizer, making existence checks necessary.

A Worked Example

Consider the two XOR representatives in Appendix G: solution 2 computes the absolute diagonal projection minus \(1/\sqrt2\), whereas solution 5 computes the negative absolute anti-diagonal projection plus \(1/\sqrt2\). They produce identical signed logits on the four XOR points but implement different functions over the full two-dimensional domain. This first distinguishes solving the same task from realizing the same global function.

Adding a zero-readout feature from solution 5 to solution 2 preserves solution 2’s global output while introducing a rank-one contribution toward the reference geometry. Duplicating or positively scaling that auxiliary feature can make it dominate representational similarity, so geometric proximity to solution 5 does not imply computational proximity to solution 5. Applying minimum-weight-norm selection under the stated conditions removes those auxiliary contributions; the residual fixes solution 2’s two essential-class orientation fractions at \(1/2\), giving a unique geometry within its orbit from width 2 onward. Uniqueness holds separately within each orbit, not as a single shared geometry across all six analytical solutions.

Key Experimental Results

The paper is primarily theoretical but not entirely without empirical experiments: Appendix G contains a small XOR initialization sweep and clustering analysis supporting the illustrated solutions. There are no large-scale real-data benchmarks, SOTA comparisons, or standard module-removal ablations; the tables summarize verifiable numerical evidence and theoretical condition analyses separately.

Main Results

Item	Reported setting or result	Interpretation boundary
XOR data and network	4 two-dimensional inputs, 2 ReLU hidden neurons, scalar output	The additional output bias is fixed; theory applies to the hidden-layer output after subtracting it
Initialization sweep	1000 independent initializations, parameters from \(\mathcal U(-1/\sqrt2,+1/\sqrt2)\)	Specific to this initialization and training protocol
Optimization budget	Full-batch Adam, learning rate 0.1, \(10^7\) steps	Not a comparative optimizer benchmark
Convergence criterion	Final BCE below \(10^{-12}\); 296 converged runs	Below-threshold loss is not exact zero loss
Six solution frequencies	Solutions 5, 2, 4, 6, 1, 3 have 88, 83, 34, 34, 32, 25 runs	Clustering removes permutation and positive scaling; counts sum to 296
Analytical representative logits	Magnitude \(1/\sqrt2\) on all four inputs	Correct classification does not give zero BCE at finite parameters

Ablation Study

This table reports theoretical conditions and counterexamples, not empirical ablations.

Condition or counterexample	Guarantee or observation	Essential boundary
Positive one-homogeneity, nonempty similarity sets	Correlation spread strictly grows with width until saturation	General activations do not inherit the continuous-scaling proof
\(\operatorname{rank}(\overline{\mathbf X})=P\)	Every orbit has correlation infimum \(-1\) and supremum \(1\)	\(N_h\geq N_h^\star+P(P-1)/2-1\) suffices; endpoints may only be approached
Same-activation reference network of width \(L\)	Supremum is \(1\)	\(N_h\geq N_h^\star+\min\{d,L\}\) suffices; functions need not agree
Weight norm with \(\mathcal T_{N_h}\neq\varnothing\)	Minimum \(2\sum_q\|\boldsymbol\beta_q\|\), no auxiliary geometry	Orientation allocation can remain nonunique
Also \(m\neq0\) and independent residual matrices	Unique minimum-weight-norm RSM unchanged at larger widths	Sufficient conditions, not satisfied by every function
ReLU tent-function counterexample	Two minimum-norm geometries at width 3; a one-parameter family at widths at least 6	Both endpoints have objective \(4+4\sqrt2\); minimum norm alone is insufficient
Inactive essential-feature counterexample	\(f(x)=\operatorname{ReLU}(x-2)\) on probes \((-1,0,1)\) has \(\inf\Omega_H=0\)	Scaling gives \(\Omega_H=\alpha^{-2}\to0\), with no finite minimizer

Key Findings

• All six analytical XOR solutions satisfy Corollary 6.2, so each orbit has a unique minimum-weight-norm geometry from width 2 onward; this is compatible with substantial unconstrained geometric variation.
• Solutions 2 and 5 agree on classification but differ globally; zero-readout feature addition demonstrates the separate claim that geometry can change even with the global function strictly fixed.
• Correctly classifying analytical representatives have finite logits; Appendix G increases logits to make BCE approach zero, changing the function rather than applying the reciprocal, function-preserving scaling symmetry used earlier.
• Captions describe near-zero and near-perfect correlations without corresponding precise coefficients in the cached text; no decimal values are fabricated here.

Highlights & Insights

• Mapping parameter-group operations onto three feature primitives separates permutation invariance, integer reweighting by duplication, and continuous reweighting by homogeneous scaling rather than conflating all parameter nonuniqueness.
• Essential versus auxiliary structure is defined by orbit invariants, not activity variance or visual prominence. High-amplitude activity can have zero output contribution, motivating simultaneous inspection of readout pathways.
• Equality conditions in the minimum-norm proof provide an explicit functional-calibration mechanism: essential-class geometric weights follow effective computational contributions rather than arbitrary implementation scales.
• The projected feature cone explains strong limiting counterexamples while preserving genuine restrictions imposed by input geometry; the result is not an unconditional claim that every network can match arbitrary brain data.

Limitations & Future Work

• The model is a fully connected one-hidden-layer nonlinear network; cross-layer degeneracies, attention, normalization, and modern large-model architectures are outside the proved scope.
• Symmetry orbits are not proved to cover every functionally equivalent implementation, so orbit-wise uniqueness cannot automatically become uniqueness over the whole functional fiber.
• Minimum norm is an implementation-selection principle; the paper does not prove that ordinary Adam, weight decay, or practical training finds the global minimum-norm implementation within an orbit.
• Pearson results use strict upper-triangular entries of an uncentered Gram matrix; replacing this with another RSA metric or centered kernel alignment (CKA) does not preserve the stated bounds automatically.
• XOR experiments support the mechanism and representative solutions in small networks; the six classes describe dominant outcomes of this sweep, not an exhaustive classification of all two-neuron XOR solutions.
• Extensions could investigate approximate functional equivalence, finite norm budgets, deeper networks, and whether practical training approximately satisfies the theoretical balance conditions; these are proposed directions, not completed results.

Related Work & Insights

• vs Braun et al. (2025): Earlier work analyzes function–representation dissociation and selection rules in deep linear networks; this paper uses activation algebra to obtain analogous results in one-hidden-layer nonlinear networks, not arbitrary deep nonlinear networks.
• vs Şimşek et al. (2021): Earlier work studies overparameterized loss landscapes and symmetries under specific teacher–student assumptions; this paper focuses on hidden activity and RSMs, separating essential computation from auxiliary geometry.
• vs Martinelli et al. (2024): The paper inherits and refines activation-dependent symmetry groups with nondegeneracy and global-residual constraints, then derives their feature-level effects.
• Implications for representational similarity analysis: Geometric agreement should be treated as evidence requiring implementation assumptions, not a sufficient criterion for computational agreement; activity scales and functional contributions can first be checked for balance.
• Implications for universal-representation hypotheses: Greater task complexity need not narrow implementation space unconditionally; when model width also increases, symmetry-induced ambiguity can expand unless an implementation-selection mechanism is specified.

Rating

• Novelty: 4/5. Connects nonlinear parameter symmetries, feature decomposition, width growth, and conditional geometric identifiability in a precise analysis.
• Experimental Thoroughness: 3/5. Detailed proofs and counterexamples, but empirical support is limited to XOR without real-model validation.
• Writing Quality: 4/5. Clear main argument with important feasibility and nonexistence boundaries in the appendices, requiring substantial mathematical background.
• Value: 4/5. Provides checkable assumptions for representation interpretation and model–brain comparison rather than only negative counterexamples.