Weight Feedback Computes the Exact Jacobian Transpose in Modern Deep Networks¶
Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://github.com/jlshen025/pcax
Area: Optimization & Theory
Keywords: predictive coding, bio-plausible learning, weight feedback, Jacobian transpose, credit assignment
TL;DR¶
Revealing that the exact Jacobian transpose factors into purely local quantities under frozen normalization statistics for modern deep layers, this paper introduces WF-Act-PC—an autograd-free predictive coding framework that exhibits positive depth scaling and matches or outperforms tuned backpropagation on deep convolutional networks.
Background & Motivation¶
Backpropagation (BP) calculates synaptic weight updates by transporting error signals through layer-wise Jacobian transposes \(J^\top\), providing unparalleled empirical optimization across deep architectures. However, from the perspective of neuroscience and biologically plausible computation, BP hinges on two fundamental physical implausibilities: the weight symmetry problem, which demands that backward feedback synaptic weights precisely mirror feedforward connections, and the update locking problem, which requires global backward error propagation across all layers before any local synaptic plasticity can occur. Predictive Coding (PC) has emerged as one of the most compelling bio-plausible alternatives by minimizing variational free energy via strictly local weight updates, successfully resolving update locking. Yet, during inter-layer error transport, classical PC still relies entirely on automatic differentiation engines (autograd) to compute vector-Jacobian products (VJPs)—the last remaining non-local operation in the framework.
Conversely, Feedback Alignment (FA) discards exact Jacobian transposes altogether by projecting error signals through fixed random feedback matrices \(B\), but catastrophically fails on deep convolutional neural networks (CNNs). This persistent failure has led to a widely accepted consensus that computing exact inter-layer error routing without a global backward pass is an insurmountable bottleneck for weight-feedback architectures. Consequently, prior efforts resorted to complex auxiliary backward pathways, such as Difference Target Propagation or learned feedback matrices.
This paper tackles the challenge from a simple yet historically overlooked angle: for standard deep layers of the form \(f(x) = \text{Act}(\text{Norm}(L(x)))\), evaluating normalization statistics in frozen evaluation mode during the inner inference loop collapses the normalization Jacobian into a fixed diagonal matrix. Under this setting, the exact Jacobian transpose \(J^\top\) factors into three locally accessible terms—the symmetric linear operator \(L^\top\), the pointwise activation derivative \(\sigma'\), and the channel-wise normalization gain \(s\). Core idea: exploit the diagonal collapse of frozen normalization Jacobians to factorize exact error transport into local activation derivatives and homeostatic gains under symmetric synaptic feedback, introducing WF-Act-PC—the first autograd-free predictive coding method that exhibits positive depth scaling.
Method¶
Overall Architecture¶
WF-Act-PC eliminates autograd computational graph dependencies during predictive coding error transport. The learning pipeline alternates between an inner inference phase and an outer weight learning phase. During the forward pass, feedforward signals generate layer-wise pre-activations and cache channel-wise normalization gains. During inference, the network optimizes layer prediction errors \(\varepsilon_l\) directly via a top-to-bottom Gauss-Seidel sweep, evaluating energy gradients through local closed-form factorizations and relaxing with local RMSProp updates. At equilibrium, synaptic weights are updated via local Hebbian rules followed by a slow-timescale soft spectral norm control.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input sample x & forward pre-activations"] --> B["Local Jacobian transpose factorization<br/>Symmetric synapse L_T + Local gain s + Activation derivative σ'"]
B --> C["Gauss-Seidel top-to-bottom error inference<br/>Local RMSProp eliminates autograd VJP graph"]
C --> D["Local weight updates<br/>Equilibrium variational free energy Hebbian learning"]
D --> E["Slow-timescale soft spectral norm clipping<br/>Controls synaptic gain for dynamical contractivity"]
E --> F["Converged predictions & diagnostic gap Δ"]
Key Designs¶
1. Local Jacobian transpose factorization: collapsing error transport to local synaptic and homeostatic products Classical feedback alignment reduced backward transport to \(W^\top v\), omitting nonlinear activation derivatives and normalization scaling, which caused severe performance degradation in deep CNNs. This paper observes that for any composite layer \(f(x) = \text{Act}(\text{Norm}(L(x)))\), expanding \(J = J_{\text{Act}} \cdot J_{\text{Norm}} \cdot J_L\) yields a purely local decomposition when normalization statistics are frozen. Specifically, running normalization in evaluation mode during inference fixes the running statistics, collapsing \(J_{\text{Norm}}\) to a diagonal scaling matrix \(J_{\text{Norm}} = \text{diag}(s)\), where \(s = \gamma / \sigma_{\text{run}}\) (e.g., for BatchNorm, \(\gamma\) is the affine scale and \(\sigma_{\text{run}}\) is running standard deviation). Combined with the pointwise activation derivative \(J_{\text{Act}} = \text{diag}(\sigma'(\text{pre-act}))\) and linear adjoint \(L^\top\), the exact Jacobian transpose is recovered locally: $\(J^\top v = L^\top \left( s \odot \sigma'(\text{pre-act}) \odot v \right)\)$ All quantities are locally accessible at the synapse—corresponding biologically to an STDP eligibility trace (\(\sigma'\)) and homeostatic divisive gain control (\(s\)). For residual connections \(f(x) = f_{\text{plain}}(x) + x\), the adjoint transpose decomposes into \(J^\top v = J^\top_{\text{plain}} v + v\), preserving strict locality across both plain and identity branches.
2. Gauss-Seidel top-to-bottom error inference: triangular dependency relaxation in error coordinates Optimizing latent state variables \(\mu_l\) in classical PC often causes exponential inter-layer signal decay. Following error-based PC, WF-Act-PC reparametrizes the network to optimize prediction errors \(\varepsilon_l = \mu_l - f_l(\mu_{l-1})\) directly under variational free energy \(E = \frac{1}{2}\sum_l \|\varepsilon_l\|^2 + \mathcal{L}(f_L(\mu_{L-1}), y)\). Exploiting the lower-triangular dependency of feedforward architectures, a sequential top-to-bottom Gauss-Seidel sweep computes the exact local error gradient starting from the top classification error \(\varepsilon_L = \tilde{y} - \text{softmax}(h_L)\): $\(g_l = \frac{\partial E}{\partial \varepsilon_l} = \varepsilon_l - L^\top_{l+1} \left( s_{l+1} \odot \sigma'(\text{pre-act}_{l+1}) \odot \varepsilon_{l+1} \right)\)$ Each layer updates \(\varepsilon_l\) using a local RMSProp state \(r_l \leftarrow (1-\alpha) r_l + \alpha g_l^2\) with step size \(\eta_\varepsilon\). The training health of the inference loop is monitored via the convergence gap \(\Delta = E_{\text{local}} - \frac{1}{2}\sum_l \|\varepsilon_l\|^2 \ge 0\), providing an online zero-overhead convergence diagnostic.
3. Slow-timescale soft spectral norm clipping: bounding maximum singular values against gradient explosion To guarantee contractive dynamics and stable convergence across multiple recurrent inference iterations, the spectral norm of layer transformations must remain bounded. Rather than applying hard clipping that rescales all singular directions uniformly, WF-Act-PC applies a soft rank-1 correction on the slow learning timescale after each batch weight update whenever \(\sigma_{\max}(W_l) > \tau\): $\(W_l \leftarrow W_l - (\sigma_{\max}(W_l) - \tau) u_l v_l^\top\)$ Here, \((u_l, v_l)\) are the leading singular vectors estimated by 20 power iteration steps. This operation is decoupled from the per-sample fast inference loop, acting analogously to homeostatic synaptic scaling. Disabling this module incurs only a 0.85 pp drop, confirming that the primary engine of WF-Act-PC is the exact local factorization rather than artificial regularizations.
A Worked Example¶
Consider a standard convolutional block (Conv + BN + GELU) with input channels \(C_{\text{in}}\) and output channels \(C_{\text{out}}\), processing input feature map \(h_{l-1} \in \mathbb{R}^{B \times C_{\text{in}} \times H \times W}\): 1. Forward pass: Compute convolution \(z = L(h_{l-1})\), evaluate frozen normalization \(\text{pre-act} = s \odot z + b\), and activate via \(h_l = \text{GELU}(\text{pre-act})\). The per-channel gain \(s_c = \gamma_c / \sigma_{\text{run}, c}\) is cached locally. 2. Top-to-bottom inference: Receiving error \(\varepsilon_{l} \in \mathbb{R}^{B \times C_{\text{out}} \times H \times W}\) from layer \(l+1\), layer \(l\) evaluates the pointwise derivative \(\sigma'(\text{pre-act})\), computes modulated error \(u = s \odot \sigma'(\text{pre-act}) \odot \varepsilon_l\), applies transposed convolution \(L^\top(u)\), and computes local energy gradient \(g_{l-1}\). 3. Local weight update: After \(T=20\) inference iterations, the system settles into an energy minimum. Weights are updated purely based on local pre- and post-synaptic activities \(\nabla_{W_l} E_{\text{local}} = h_{l-1}^\top \varepsilon_l\), completely bypassing autograd backward graph retention.
Loss & Training¶
- Inner-loop optimization: RMSProp decay \(\alpha_{\text{rms}} = 0.2\), learning rate \(\eta_\varepsilon = 0.1\), iterations \(T = 20\), and top-layer error norm clipping \(\|v_{\text{top}}\| \le 5.0\).
- Outer-loop weight learning: AdamW optimizer, base learning rate \(\eta_w = 7 \times 10^{-4}\) with 2-epoch linear warmup followed by cosine decay to \(10^{-6}\), and weight decay \(10^{-4}\). On CIFAR-100, CutMix/Mixup and label smoothing 0.1 are incorporated, with architecture-specific learning rates \(\eta_w \in [1 \times 10^{-3}, 2 \times 10^{-3}]\). Soft spectral norm clipping threshold is set to \(\tau = 3.0\).
Key Experimental Results¶
Main Results¶
Experiments follow the standardized PCX benchmark protocol (50 epochs, 5 seeds, mean ± std) on CIFAR-10 and CIFAR-100 across VGG-5/7/9 (Conv+GELU+MaxPool, \(s=1\)) and ResNet-18 architectures, comparing against classical PC baselines and comparably tuned backpropagation (BP tuned).
| Dataset | Architecture | WF-Act-PC (Ours) | iPC (Classical PC) | ePC (Error PC) | BP (tuned) | Note |
|---|---|---|---|---|---|---|
| CIFAR-10 | VGG-5 | 88.25 ± 0.31% | 85.51 ± 0.12% | 88.27 ± 0.18% | 88.03 ± 0.18% | Matches tuned BP (+0.22 pp) |
| CIFAR-10 | VGG-7 | 92.15 ± 0.72% | 80.15 ± 0.21% | 88.84 ± 0.31% | 91.39 ± 0.05% | Outperforms tuned BP (+0.76 pp) |
| CIFAR-10 | VGG-9 | 93.57 ± 0.48% | 79.02 ± 0.21% | 86.81 ± 0.09% | 92.43 ± 0.29% | Outperforms tuned BP (+1.14 pp) |
| CIFAR-10 | ResNet-18 | 92.76 ± 0.85% | 70.44 ± 0.81% | 91.73 ± 0.21% | 91.54 ± 0.36% | Outperforms tuned BP (+1.22 pp) |
| CIFAR-100 | VGG-5 | 65.64 ± 0.57% | 56.07 ± 0.16% | 63.39 ± 0.25% | 64.09 ± 0.14% | Outperforms tuned BP (+1.55 pp) |
| CIFAR-100 | VGG-7 | 68.75 ± 0.64% | 43.99 ± 0.30% | 58.62 ± 0.20% | 69.00 ± 0.30% | Ties tuned BP (-0.25 pp) |
| CIFAR-100 | VGG-9 | 70.32 ± 0.53% | 44.76 ± 0.40% | 60.65 ± 0.25% | 71.90 ± 0.32% | Slightly trails tuned BP (-1.58 pp) |
| CIFAR-100 | ResNet-18 | 71.07 ± 0.71% | 29.45 ± 1.36% | 69.47 ± 0.32% | 70.22 ± 0.27% | Outperforms tuned BP (+0.85 pp) |
Ablation Study¶
Ablations on VGG-5 / CIFAR-10 quantify the impact of transport mechanisms, inference iterations, and normalization factors (Table A: VGG-5 transport & inner-loop parameters; Table B: BN-VGG-5 isolating each correction factor when \(s \neq 1\)).
Table A: VGG-5/CIFAR-10 Transport & Hyperparameter Ablation
| Group | Variant | Acc (%) | s/epoch | Note |
|---|---|---|---|---|
| Transport | Random B (FA-style) | 27.42 ± 0.83% | 22.0 | Complete collapse on deep CNNs |
| Transport | \(W^\top\) only (no \(\sigma'\)) | 81.62 ± 0.47% | 20.1 | Omitting activation derivative causes -6.63 pp drop |
| Transport | \(W^\top + \sigma'\) (WF-Act-PC) | 88.25 ± 0.31% | 22.1 | Exact local transport formulation |
| Transport | Autograd \(J^\top\) (jax.vjp) | 89.01 ± 0.24% | — | Reference upper bound; local formula is within 0.76 pp |
| Inner Loop | \(T = 1\) | 87.64 ± 0.38% | 6.4 | Near BP speed, already surpasses iPC (85.51%) |
| Inner Loop | \(T = 5\) | 88.07 ± 0.29% | 10.8 | Steady convergence gain |
| Inner Loop | \(T = 10\) | 88.29 ± 0.33% | 15.5 | Accuracy reaches plateau |
| Inner Loop | \(T = 20\) (default) | 88.25 ± 0.31% | 22.1 | Stable baseline across all architectures |
| Inner Loop | \(T = 40\) | 72.01 ± 0.91% | 33.2 | RMSProp denominator shrinkage induces divergence |
| Stability | SNC disabled | 87.40 ± 0.52% | 20.6 | Minor -0.85 pp drop confirms robustness |
Table B: BN-VGG-5/CIFAR-10 Normalization Gain \(s \neq 1\) Ablation
| Transport Variant | Acc (%) | Margin vs. Full | Note |
|---|---|---|---|
| \(W^\top\) only (no \(\sigma'\), no \(s\)) | 83.74 ± 0.14% | -3.86 pp | Lacks both correction factors |
| \(W^\top \cdot \sigma'\) (no \(s\)) | 86.03 ± 0.22% | -1.57 pp | Lacks channel gain modulation |
| \(W^\top \cdot s\) (no \(\sigma'\)) | 84.48 ± 0.20% | -3.12 pp | Lacks pointwise activation derivative |
| \(W^\top \cdot \sigma' \cdot s\) (Full local) | 87.60 ± 0.23% | 0.00 pp | Empirically confirms Observation 1 |
| Autograd \(J^\top\) (jax.vjp) | 89.82 ± 0.27% | +2.22 pp | Residual gap stem from MaxPool unpooling approximation |
Key Findings¶
- The only PC method exhibiting positive depth scaling: On both CIFAR-10 and CIFAR-100, while prior PC methods suffer catastrophic degradation with depth (iPC collapses to 70.44% and 29.45% on ResNet-18), WF-Act-PC scales monotonically, improving from 88.25% to 93.57% on CIFAR-10 and from 65.64% to 71.07% on CIFAR-100.
- Activation derivative \(\sigma'\) is the critical factor: Restoring \(\sigma'\) onto \(W^\top\) delivers an immediate +6.63 pp gain (81.62% → 88.25%), pinpointing the missing derivative as the primary driver behind FA's historical failure.
- Efficient inference trade-off: Setting \(T=1\) in the Gauss-Seidel loop requires only 6.4 s/epoch (near BP wall-clock speed) while delivering 87.64% accuracy on VGG-5/CIFAR-10, demonstrating that high-iteration convergence is not mandatory for strong predictive performance.
Highlights & Insights¶
- Theoretical closure on local Jacobian transpose factorization: The paper establishes that frozen normalization collapses the Jacobian to a fixed diagonal, reducing the full backward operator into three locally available biological signals (symmetric synaptic weights, homeostatic normalization gains, and STDP-like activation derivatives).
- Implicit regularization via Gauss-Seidel energy relaxation: WF-Act-PC surpasses tuned BP on deep networks (VGG-7/9, ResNet-18 on CIFAR-10), supported by the hypothesis that iterative inference acts as an implicit denoiser that finds flatter, more generalizable local minima.
- Zero auxiliary feedback parameters: Unlike target propagation or learned feedback alignment, WF-Act-PC requires zero additional feedback parameters, enabling \(O(1)\) activation memory per layer and straightforward mapping to neuromorphic hardware.
Limitations & Future Work¶
- Limitations acknowledged by authors:
- Weight symmetry (\(L^\top\) mirrors forward operator \(L\)) remains an active assumption shared with standard PC and BP;
- Soft spectral norm clipping relies on power iteration on the slow timescale, which is not synapse-local;
- Nearest-neighbour unpooling for MaxPool introduces an approximate error (~1.25 pp loss compared to exact argmax caching).
- Practical limitations observed:
- Inner-loop iterations (\(T=20\)) introduce an approximate 3× wall-clock overhead on standard single GPU hardware;
- Evaluations focus primarily on CIFAR-10/100 and Tiny-ImageNet, leaving full-scale ImageNet-1K and large vision-language transformers as open empirical questions.
- Future directions: Integrating WF-Act-PC with asymmetric feedback mechanisms (e.g., Kolen-Pollack dynamics) to eliminate weight symmetry; extending local factorization to LayerNorm in self-attention transformers.
Related Work & Insights¶
- vs Feedback Alignment (FA / DFA): FA replaces \(J^\top\) with a fixed random matrix \(B\) and fails on deep CNNs. This paper demonstrates that weight feedback fails not because \(W^\top\) is used, but because the local corrections \(\sigma'\) and \(s\) were omitted.
- vs Incremental Predictive Coding (iPC): iPC relies on global autograd VJPs and suffers from severe depth collapse; WF-Act-PC eliminates autograd dependencies while achieving positive depth scaling.
- vs Error-based Predictive Coding (ePC): ePC introduced error coordinates to overcome signal decay but retained autograd Jacobian transposes; WF-Act-PC adopts the error coordinates while providing a closed-form local factorization.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an elegant, closed-form local factorization of the Jacobian transpose under frozen normalization.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous multi-seed evaluations across multiple architectures on CIFAR-10/100 and Tiny-ImageNet with clean ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation, balanced biological motivation, and transparent disclosure of assumptions.
- Value: ⭐⭐⭐⭐⭐ Demonstrates for the first time that local weight-feedback predictive coding can match or exceed tuned backpropagation on deep convolutional networks.