CaRe: Critical Parameter Rectification for Efficient Visual Modeling¶

Conference: ECCV2026
Paper: ECCV Official Page · PDF
Code: https://github.com/lime-j/care
Area: Model Compression / Efficient Vision Backbones
Keywords: structural reparameterization, parameter-space gating, sparse correction, linear attention, multi-stage fusion

TL;DR¶

CaRe expresses a standard weight as a dense base plus a learned Softplus-gated correction, concentrating training-time corrections on important entries and merging them into an ordinary layer before deployment; CaReNet-M1 achieves 73.6% ImageNet-1K Top-1 at 33,203 images/s on an A100, outperforming EfficientViM-M1 by 0.7 percentage points.

Background & Motivation¶

Efficient vision models must do more than reduce FLOPs: they must execute quickly on real hardware. Structural reparameterization offers a complementary route, using a richer parameter representation during training and folding it into simple operators for deployment. In RepVGG, for example, an additional 1×1 convolution and identity branch can be embedded into the main 3×3 kernel, but their corrections are confined to the kernel center. The architecture determines the spatial support in advance.

CaRe asks why important corrections should always occupy the center. Giving every weight a learnable residual would remove this restriction, but the base and auxiliary tensors could then divide the task arbitrarily and become two redundant dense representations. The authors seek corrections that can choose their own locations while experiencing an optimization bias toward sparsity. This constraint concerns the extra correction, not a requirement that the final network be sparse, and it does not freeze the base weights.

The paper combines the regularization properties of multiplicative factorization with parameter-space gating, then evaluates deployment benefits through CaReNet. Core idea: represent training weights as a dense base plus a smoothly gated correction, concentrate corrective capacity on a small set of parameters, and merge the representation completely before deployment because its gate is input-independent.

Method¶

Overall Architecture¶

CaRe is a weight parameterization, whereas CaReNet is a vision backbone built around it; these are distinct contributions. During training, CaRe constructs effective weights inside convolutional or linear layers, while images still undergo conventional feature computation. CaReNet starts with a convolutional stem that downsamples by a factor of 16, then uses 3 pyramid stages with Simple Attention for local and global mixing and Progressive Stage Fusion for classification predictions.

Before deployment, the effective weights are computed once and the auxiliary parameters are removed. Parameter gates need not be evaluated for each image. The diagram separates the training-parameter branch from the image-feature branch: the feature gate inside Simple Attention remains at inference and must not be confused with the mergeable parameter gate.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Parameters["Training parameters"] --> CaRe["Critical Parameter Rectification<br/>Training factors and deployment merge"]
    Image["Image and convolutional stem"] --> Attention["Simple Attention<br/>Repeated within pyramid stages"]
    CaRe -->|Supplies layer weights| Attention
    Attention --> Fusion["Progressive Stage Fusion"]
    Fusion --> Prediction["Classification prediction"]

Key Designs¶

1. Critical Parameter Rectification: allocate extra capacity to learned locations

A plain additive decomposition increases freedom without preventing its components from learning similar dense representations. The authors first examine a multiplicative factorization of a scalar correction: for a fixed product, the minimum squared penalty on the two factors is proportional to the absolute correction. Here \(a\) and \(g\) are the factors and \(\Delta\) is the effective correction. This identity applies to linear multiplicative factorization; it is not an equivalence proof for the complete Softplus model.

\[ \min_{a,g:\,ag=\Delta}(a^2+g^2)=2|\Delta|. \]

Applied elementwise, an \(L_2\) penalty on the factors creates an \(L_1\)-type bias on effective corrections, motivating a preference against correcting every entry. However, two signed factors can represent the same positive correction through either a positive-positive or a negative-negative pair, entangling their sign and magnitude roles. CaRe turns one factor into a strictly positive, smooth, monotonic Softplus gate:

\[ W=W_{\mathrm{base}}+\Delta W, \qquad \Delta W=W_{\mathrm{aux}}\odot\operatorname{Softplus}(W_{\mathrm{gate}}), \qquad \operatorname{Softplus}(g)=\log(1+e^g). \]

All three tensors have the same shape. The auxiliary tensor determines the correction sign, the positive gate modulates its strength, and the dense base preserves an unrestricted representation path. A negative gate parameter can make Softplus small, but the output is never exactly zero for finite inputs. Thus, sparsity here primarily means concentrated effective corrections, not binary masks or exact zero weights. Suppression of non-critical entries emerges from task optimization together with regularization; it should not be described as weight decay directly driving gate parameters toward negative infinity.

The parameter gate is shared across samples but changes during training; static does not mean frozen. At inference, evaluating the expression once produces a single standard weight tensor. No input-dependent conditional branch or sparse hardware support is required. The paper applies CaRe to depthwise convolutions and FFNs in CaReBlocks and discusses extending it to convolutional and linear layers in other backbones. Zero extra overhead refers to the deployment representation of the same layer, not a claim that CaReNet and EfficientViM have identical whole-network FLOPs.

2. Simple Attention: replace pairwise spatial interaction with a compact global state

Starting from EfficientViM, CaReNet simplifies the HSM-SSD mixer into global linear aggregation followed by output gating. Features are organized by batch, channel, and spatial position. One projection branch produces keys and queries, with a 3×3 depthwise convolution and SiLU injecting local spatial bias. A second branch produces value features and a gating signal, separating information aggregation from output modulation.

Aggregation first combines values and keys over spatial positions into a global state whose size is expanded channels × state dimension. Queries then distribute that state back to individual positions. This avoids constructing an attention matrix quadratic in the number of spatial positions; with channel and state dimensions fixed, computation grows linearly with sequence length. The redistributed features are multiplied elementwise by the SiLU-transformed output gate and passed through an output projection. This gate is input-dependent GLU-style feature modulation that remains in the deployed network.

The cached extraction of Eq. (4) does not fully preserve the transpose and normalization factors, and Eq. (5) also has missing symbols. Accordingly, this note describes only the computation order supported by the prose and tensor shapes, without inventing an exact scaling equation. The paper states that replacing the EfficientViM-M1 mixer achieves comparable performance, but the selected result tables do not separately quantify every macro-level change. The headline improvement therefore cannot be attributed entirely to CaRe.

3. Progressive Stage Fusion: keep shallow spatial information involved

EfficientViM combines multi-stage outputs using learnable weights and softmax, which the authors observe can become overly biased toward the final stage early in training. PSF instead uses fixed increasing coefficients, preserving contributions from shallow spatial cues while assigning greater weight to deeper semantics. The explicitly reported coefficients are \([0.2,0.4,0.6,0.8]\); the ablation compares final-stage-only, uniform, and several increasing configurations.

A reproducibility boundary matters here: the architecture text describes 3 backbone stages, but the fusion vector contains 4 entries. The current text cache does not clearly map all prediction heads to these 4 coefficients or provide the complete fusion equation. It would therefore be unjustified to assert that the first coefficient belongs to the stem or to interpret the vector as normalized probabilities. The supported conclusion is that the reported fixed progressive configuration outperforms final-stage-only and uniform configurations.

A Worked Example¶

For the reported M1 configuration, a 224×224 image becomes a 14×14 initial grid after the stem's factor-16 downsampling. The 3 stages use channels \([128,192,320]\), block counts \([2,2,2]\), and state dimensions \([64,32,16]\). During training, each applicable layer constructs its effective weight from base, auxiliary, and gate parameters. Features then undergo local convolution, global-state aggregation, and output modulation, with PSF used at the classification end.

Deployment leaves the image-processing sequence unchanged. The difference occurs during model preparation: the three parameter tensors are merged into ordinary layer weights. Deleting the gated correction is not an approximate substitute for merging, because merging preserves its numerical contribution while deletion discards learned representations. The ablation below directly tests this distinction.

Loss & Training¶

CaRe does not introduce a new supervised task; its focus is weight parameterization and the resulting optimization bias. The cache does not enumerate the complete classification objective or every augmentation setting, only stating that the recipe follows EfficientViM. Distillation terms or additional sparsity losses should not be invented. The main models are trained from scratch for 300 epochs, with a separate M4-256 result using 450 epochs.

Training uses AdamW, a peak learning rate of \(2.5\times10^{-3}\), and cosine decay to \(2.0\times10^{-5}\). It includes 20 warmup epochs, 10 cooldown epochs, gradient clipping at a maximum norm of 0.02, and EMA decay of 0.9995. The authors report TPU v4-8 training, a per-device batch size of 512, and a total batch size of 2048; these reported values are retained without inferring an unspecified device mapping. COCO fine-tuning starts from the 300-epoch pretrained checkpoint and uses AdamW at \(1.0\times10^{-4}\) for 12 epochs.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1. Top-1 is in %, throughput is in images/s, and the benchmark hardware is an NVIDIA A100-PCIE-40GB. Parameters and FLOPs are in millions. This throughput measurement should not be equated with phone latency.

Model	Input	Top-1	Throughput	Parameters	FLOPs
EfficientViM-M1	224×224	72.9	33,033	6.7	239
CaReNet-M1	224×224	73.6	33,203	6.7	264
EfficientViM-M2	224×224	75.4	28,249	13.9	355
CaReNet-M2	224×224	76.0	27,180	13.9	391
EfficientViM-M3	224×224	77.6	19,038	16.6	656
CaReNet-M3	224×224	78.3	19,249	16.9	727
EfficientViM-M4	256×256	79.4	13,625	19.6	1111
CaReNet-M4	224×224	79.8	14,576	19.6	964
CaReNet-M4-256	256×256	80.4	13,404	19.6	1223
CaReNet-M4-256-450ep	256×256	80.6	13,404	19.6	1223

M1 and M3 each improve by 0.7 percentage points. M2 improves by 0.6 percentage points but has lower throughput, so not every variant is simultaneously faster. At the same 256×256 resolution, M4 improves Top-1 by 1.0 percentage point while slightly reducing throughput. The 450-epoch version reaches 80.6%, but its longer training is not an equal-budget comparison.

COCO-2017 results come from Table 2: with RetinaNet, CaReNet-M4 scores 39.2 AP versus 38.8 for EfficientViM-M4, while small-object AP is 21.9 versus 22.1. With Mask R-CNN, both obtain 39.3 box AP, and mask AP is 36.0 versus 35.8. The downstream evidence therefore supports modest gains rather than dominance on every metric.

Ablation Study¶

The following selection from Table 3 compares weight-composition rules. Each \(P\) denotes an independently learned parameter tensor. The standard-layer baseline here is 72.7%, not the 72.9% EfficientViM-M1 result in the main comparison.

Weight composition	Top-1 (%)	Change from standard layer (percentage points)
\(W=P_1\)	72.7	0.0
\(W=P_1\odot\phi(P_2)\)	69.5	-3.2
\(W=P_1+P_3\)	73.1	+0.4
\(W=P_{3\times3}+P_{1\times1}+P_{\mathrm{id}}\)	73.2	+0.5
\(W=P_1\odot\phi(P_2)+P_3\odot\phi(P_4)\)	70.1	-2.6
Remove the gated correction after CaRe training	60.1	-12.6
\(W=P_1\odot\phi(P_2)+P_3\), full CaRe	73.6	+0.9

Tables 5 and 6 provide the following additional ablations. These groups change different variables and should be interpreted within their respective groups.

Ablation group	Configuration	Top-1 (%)
Gate activation	Identity	72.9
Gate activation	ReLU	73.1
Gate activation	GELU	73.5
Gate activation	Softplus	73.6
Stage fusion	Final stage only \([0,0,0,1.0]\)	73.1
Stage fusion	Uniform \([0.25,0.25,0.25,0.25]\)	72.9
Stage fusion	Simple increasing \([0.1,0.2,0.4,0.8]\)	73.2
Stage fusion	PSF \([0.2,0.4,0.6,0.8]\)	73.6

Key Findings¶

The dense base is not easily replaced: a gated product alone reaches 69.5%, and two gated branches reach 70.1%, both below full CaRe at 73.6%.
Removing the correction reduces accuracy from 73.6% to 60.1%, a drop of 13.5 percentage points. The table's -12.6 is relative to the 72.7% standard layer, not full CaRe. This establishes usefulness of the correction, but is not a sufficient causal test for identifying the most critical subset.
Figure 5 measures the fraction of large corrections using \(|\Delta W|\geq0.05\max(|W|)\). That fraction falls rapidly within approximately 40 epochs, while the share of total correction mass carried by the largest 10% of entries increases. The cache does not provide reliable curve endpoints, so no exact sparsity rate is reported here.
Under the DeiT recipe, Table 4 reports ResNet-18 at 69.1%→69.3%, ViT-Tiny at 72.2%→72.4%, and PoolFormerV2-S12 at 77.7%→78.1%. Their inference parameter counts remain 11.7M, 5.7M, and 11.9M, respectively. Transferability is supported, but the gains are small.

Highlights & Insights¶

Placing extra training capacity in the parameter representation rather than an input-dependent branch is what permits deployment without extra operators. A reusable principle is to ask whether added computation can be evaluated before an input arrives, instead of merely counting training branches.
Sparse corrections and a sparse model are different concepts. CaRe retains a dense backbone and uses correction concentration to explain optimization benefits; its execution speed does not depend on sparse operators.
Keeping an ungated base path matters more than adding further gated branches. The composition ablation suggests that the inductive bias of the representation is more informative than the number of parameter branches alone.

Limitations & Future Work¶

Theoretical scope: the \(L_2\)-to-\(L_1\) equivalence for linear products does not directly extend to Softplus or prove that practical AdamW training finds the same minima. Further analysis should address nonlinear gates and the actual optimizer.
Evidence scope: no multi-seed variance is reported, and gains of 0.2 to 0.4 percentage points on general backbones need stability checks. Removing the entire auxiliary tensor does not establish optimality of the large-correction ranking; a matched-count random-removal control would help.
Cost scope: zero extra inference overhead does not mean zero extra training cost. Training memory, optimizer state, wall-clock time, and actual mobile-device latency are not sufficiently quantified in the current cache.
Reproducibility scope: attention equations have extraction damage, and the mapping between 3 stages and 4 fusion coefficients is unclear. These details require the original PDF or code rather than assumptions based on familiarity. This note uses the cache only and does not verify accessibility of the code link.
Task scope: experiments mainly cover ImageNet and COCO, with no dedicated autonomous-driving evaluation and no improvement on small-object detection. Higher resolutions, different hardware, and domain transfer remain useful tests of applicability.

RepVGG / ACNet / DBB: improve optimization through training-time branches and inference-time merging. CaRe starts from RepVGG's center-supported correction to learn flexible support, but this should not be generalized into a claim that all structural reparameterization methods modify only the center.
EfficientViM / MambaOut: the former supplies the backbone starting point, while the latter motivates reconsidering whether state-space machinery is necessary. CaReNet changes the mixer, fusion strategy, and weight parameterization together; composition ablations and cross-backbone results are more appropriate for judging CaRe itself.
LoRA / DoRA: primarily address adaptation of pretrained models, whereas CaRe emphasizes joint training from scratch and does not constrain updates to low rank. Auxiliary representations can be merged in both cases, but their training assumptions and constraints differ.
SE / GLU / CondConv: perform input-dependent modulation of features or weights, generally preventing collapse into one fixed weight for all inputs. CaRe can coexist with these mechanisms, as illustrated by CaReNet's retained output feature gate.

Rating¶

Novelty: 4/5. The combination of sparsity-biased factorization, Softplus parameter gating, and mergeable weights is clear, although the constituent ideas have precedents.
Experimental Thoroughness: 4/5. Classification, detection, segmentation, composition, and activation studies are substantial, but variance, training cost, and mobile-device validation are missing.
Writing Quality: 3/5. The main argument is accessible, but the theoretical scope of nonlinear sparsity and fusion details need clarification; damaged cached equations further limit reproducibility.
Value: 4/5. Useful for efficient vision models that retain standard deployment operators, although small gains on general backbones still require task-specific evaluation.