Quick ViTs: Speeding up Vision Transformers through Equivariance

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/davnords/octic-vits
Area: Model Compression
Keywords: Vision Transformer, octic group, equivariance, Fourier decomposition, arithmetic intensity

TL;DR

Quick ViTs turns rotation and reflection equivariance into structured sparsity and weight sharing in Fourier-domain linear layers, allowing H8(ViT-H/14) to reach 85.0% rather than 84.6% supervised ImageNet-1K Top-1 accuracy with 102.3G rather than 167.8G FLOPs, while explaining why theoretical FLOP savings do not imply an equal runtime speedup.

Background & Motivation

A ViT splits an image into square patches and applies shared network operations to the resulting tokens. This sharing guarantees token-permutation equivariance, but does not specify how feature channels should change when the pixels inside a patch rotate or reflect. Edges, corners, and textures recur in different orientations; sharing their geometric computations could avoid learning fully independent mappings for each orientation. However, conventional equivariant networks often introduce additional orientation channels and complicated operators, increasing execution costs alongside geometric constraints. Stronger symmetry alone therefore does not establish suitability for large-scale visual backbones.

The paper builds directly on Flopping for FLOPs: with the right representation basis, equivariance can remove computation instead of adding it. Where that earlier approach used reflection symmetry, Quick ViTs extends the construction to rotations and reflections of square patches, forming the eight-element dihedral group \(D_8\). This group comprises four right-angle rotations and their combinations with reflection; it does not provide arbitrary-angle rotation equivariance. In the corresponding group Fourier domain, different irreducible representation types cannot mix freely, turning dense channel mappings into several small matrix multiplications. The important questions are whether this restriction harms ImageNet representational capacity and whether fewer operations translate into higher GPU throughput.

These questions are connected: preserving symmetry everywhere may discard useful image-orientation cues, while small matrix operations can become bandwidth limited. The authors therefore investigate hybrid architectures that break symmetry late in the network, together with arithmetic intensity, which relates operations to data movement. The paper thus tests the practical boundaries of an equivariant component under established training recipes and hardware conditions. Core Idea: perform early visual computation in Fourier representations of \(D_8\) to reduce channel-mixing costs, retain or break late-stage symmetry according to the task, and use arithmetic intensity to explain when computational savings become runtime gains.

Method

Overall Architecture

The input is a square image, and the output is a visual representation for classification or dense prediction. “Equivariant Input Construction” makes patch embeddings, positional encodings, and the class token obey a consistent geometric transformation rule. “Fourier-Domain Equivariant Computation” then preserves that rule in the first \(k\) Transformer blocks while replacing dense linear layers with smaller matrices. Finally, “Task-Dependent Symmetry Conversion” selects the I8 or H8 path, with ordinary Transformer operations in the remaining \(l-k\) blocks. The main training experiments use \(k=l/2\); the throughput benchmark replacing every block is a different configuration whose speed cannot be assigned to the main experiments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    input["Square image"] --> embedding["Equivariant Input<br/>Construction"]
    embedding --> blocks["Fourier-Domain Equivariant<br/>Computation: first k blocks"]
    blocks --> transition["Task-Dependent<br/>Symmetry Conversion"]
    transition -->|I8: tokenwise invariantization| invariant["Ordinary blocks on invariant channels"]
    transition -->|H8: break equivariance constraints| hybrid["Ordinary blocks on original channels"]
    invariant --> output["Classification or dense prediction features"]
    hybrid --> output

Channel invariance in I8 does not collapse the entire feature map into one vector: spatial tokens still permute when the image rotates. It can therefore preserve the spatial structure needed for dense prediction, while extracting CLS for classification produces an invariant prediction. H8 instead permits later blocks to exploit orientation information and provides no equivalent rotation-invariance guarantee for its final output.

Key Designs

1. Equivariant Input Construction: handle token positions and within-patch orientation together

Rotating an image does two things: it permutes patch positions and rotates the pixels inside every patch. Ordinary Transformers automatically handle only the former; Quick ViTs assigns a group representation to channels to describe the latter. Writing \(x\) as a channels-by-tokens feature matrix and \(b\) as a network block, the constraint in the paper's Equation (9) can be expressed as:

\[ b\!\left(\rho_{\mathrm{chan}}(g)x\rho_{\mathrm{token}}(g)^T\right) =\rho_{\mathrm{chan}}(g)b(x)\rho_{\mathrm{token}}(g)^T, \qquad g\in D_8. \]

This requires transforming before feature computation to agree with computing features first and then transforming them by the prescribed rule; it does not require every intermediate vector to stay unchanged. PatchEmbed uses a convolution whose kernel size and stride both equal the patch size, with weight-sharing constraints that produce the required representation types. When the embedding dimension \(C\) is divisible by 8, channels are organized as repeated eight-dimensional Fourier representations. Each contains four one-dimensional types, \(A_1,A_2,B_1,B_2\), and two copies of the two-dimensional type \(E\). These labels describe how features transform geometrically; they are not eight independent networks or eight evaluations of the image.

Positional encodings cannot be learned without constraints, since that would immediately break the established equivariance. Encodings at symmetry-related positions are constrained to match after applying the corresponding channel transformation. They still vary across positions, so the method does not guarantee translation equivariance; translation and rotation-reflection constraints are distinct. The CLS token has no ordinary patch position in the image plane, and the authors permit only its \(A_1\) component to be nonzero. It consequently enters as an invariant type without introducing orientation information inconsistent with the chosen group representation.

2. Fourier-Domain Equivariant Computation: turn representation constraints into fewer matrix operations

Schur's lemma restricts equivariant linear maps: different irreducible representation types cannot freely exchange information. A mapping from \(C\) channels to \(C\) channels consequently needs only four square matrices of width \(C/8\) and one of width \(C/4\). The latter shares weights across the two coordinate directions of the two-dimensional \(E\) representation, but must still apply the multiplication twice. This explains why an 8-fold parameter reduction differs from a 5.33-fold operation reduction. Following the multiplication counts in Section 3.3, equal-width linear layers satisfy the following relations; counting full multiply-add FLOPs preserves the ratio:

\[ \frac{N_{\mathrm{param,eq}}}{N_{\mathrm{param,dense}}} =\frac{4(C/8)^2+(C/4)^2}{C^2}=\frac18, \qquad \frac{N_{\mathrm{mult,eq}}}{N_{\mathrm{mult,dense}}} =\frac{4(C/8)^2+2(C/4)^2}{C^2}=\frac3{16}. \]

These mappings replace attention projections and MLP linear layers, rather than deleting tokens or approximating the attention matrix. The MLP retains its channel expansion, GELU, and channel contraction, and residual connections remain in place. However, GELU cannot simply act pointwise on arbitrary Fourier components, because general representation transformations do not commute with pointwise nonlinearities. The authors first transform features into the regular representation domain, where group actions are permutations, apply GELU, and transform back into the Fourier domain. Here, “domain” refers to representations of the eight-element group, not the image plane's two-dimensional frequency spectrum.

The implementation fuses the forward and inverse transforms with GELU in one Triton kernel to reduce intermediate writes and kernel launches. Most other operations use PyTorch and torch.compile, without requiring an entirely specialized inference system for the backbone. LayerNorm is also made equivariant; the main text describes centering by irreducible type followed by normalization using a norm across types. Attention retains its dot-product structure because the inner product of query and key is invariant when both transform under the same orthogonal representation. Attention weights remain consistent for corresponding token pairs, allowing the weighted sum of values to preserve equivariance.

Fewer FLOPs do not automatically imply better hardware utilization. Arithmetic intensity measures FLOPs per byte transferred; small matrix blocks can remove operations while still requiring substantial activation traffic. At smaller feature dimensions, an equivariant layer can move from compute-bound to bandwidth-bound execution and fail to realize the theoretical 5.33-fold speedup. Section 4.1 also distinguishes small and large token batches: once weight reads are amortized, input and output activation transfers become more important. With sufficiently wide channels, both types of linear layer become compute bound, and operation counts more directly determine their speed. This motivates reporting accuracy, FLOPs, throughput, and memory rather than parameter count alone.

3. Task-Dependent Symmetry Conversion: decide whether later blocks retain orientation cues

The I8 path first converts each token's steerable channels into invariant channels and then passes them to ordinary blocks. The main text uses power-spectrum invariantization to remove channel responses to group transformations while retaining quantities useful for recognition. Spatial tokens still permute with image transformations, so ordinary blocks' token-permutation equivariance preserves the subsequent spatial correspondence. Reading CLS at the end produces a \(D_8\)-invariant classification result. The precise power-spectrum construction and comparisons with other invariants are placed in Appendix E, which is absent from the supplied full text; no exact formula or dimensional expansion is guessed here.

H8 skips that invariantization and simply reinterprets the existing channels as ordinary features for subsequent blocks. This deliberately removes geometric constraints, allowing the model to exploit distributional patterns such as images usually being upright on ImageNet. It is not an inference-time rotation detector that chooses a branch: I8 and H8 are separately trained architectural choices. The authors generally find higher accuracy for H8, whereas I8 is more stable on a rotated validation set without training on those rotation augmentations. More equivariant blocks are consequently not always better: early invariantization can discard orientation information that the task still needs.

A Worked Example

Consider ViT-L/16 in the supervised experiments: its total depth is 24, and the main configuration uses 12 equivariant blocks followed by 12 ordinary blocks. An image containing a diagonal edge first passes through constrained patch kernels to produce channels with explicitly assigned representation types. Rotating the entire image by 90 degrees requires moving token positions and transforming channels according to their types, not just rearranging tokens. The first 12 blocks preserve this correspondence structurally, without separately evaluating and averaging several rotated views. With I8, invariantization removes channel-orientation responses, and final classification remains unchanged under that group rotation. With H8, the last 12 blocks may recombine features according to orientation, potentially improving accuracy on ordinary images but no longer promising identical outputs after rotation. This example explains the architecture; it is not an additional single-image experiment reported by the paper.

Loss & Training

Quick ViTs primarily changes the backbone architecture rather than introducing a geometric-consistency loss. Supervised experiments follow DeiT III, self-supervised experiments follow DINOv2, and the authors explicitly report no hyperparameter retuning. Both groups train on ImageNet-1K; the DINOv2 comparisons use baselines trained by the authors rather than directly comparing public checkpoints with different pretraining scales. Self-supervised evaluation freezes features and uses linear and k-NN evaluation for classification and semantic segmentation. Training uses bf16; the linear-layer hardware analysis additionally compares fp16 and fp32 to expose precision-dependent bottlenecks. Inference retains the selected I8 or H8 architecture without an additional teacher branch or multi-orientation ensemble.

Key Experimental Results

Main Results

The following subset of Table 3, page 13, uses DeiT III and ImageNet-1K Top-1 evaluation, with bf16 throughput; the rotation column reports accuracy changes in percentage points without training on those rotation augmentations.

Model	Parameters (M)	FLOPs (G)	Throughput (images/s)	Peak Memory (MB)	Top-1 (%)	Rotation Change (ppts.)
ViT-H/14	632.1	167.8	569	3285	84.6	-12.6
I8(ViT-H/14)	362.3	104.0	657	2249	84.7	0.0
H8(ViT-H/14)	355.8	102.3	660	2223	85.0	-13.4

H8 reduces FLOPs by approximately 39% and raises Top-1 by 0.4 percentage points, but its throughput is only about 1.16 times the baseline. I8 has slightly lower standard classification accuracy than H8 but avoids the accuracy drop in this rotation test, demonstrating a concrete functional trade-off.

The following subset of Table 4, page 13, evaluates frozen features after DINOv2-recipe pretraining on ImageNet-1K; IN1K reports accuracy, while ADE20K and VOC2012 report mIoU, all expressed as percentages.

Model	FLOPs (G)	IN1K Linear	IN1K k-NN	ADE20K Linear	ADE20K k-NN	VOC2012 Linear	VOC2012 k-NN
ViT-H/16	127.7	81.7	81.0	34.7	30.6	70.7	60.9
I8(ViT-H/16)	77.7	81.9	80.9	33.9	29.2	70.6	61.2
H8(ViT-H/16)	77.5	82.2	81.4	35.1	31.1	70.8	61.7

H8 slightly exceeds the baseline on all these ViT-H frozen-feature metrics, but this does not establish uniformly better downstream quality for every equivariant variant. For example, I8 reduces ADE20K linear mIoU from 34.7 to 33.9 and k-NN mIoU from 30.6 to 29.2.

Ablation Study

The following ablation is reported in the text of Section 5.3, page 14, without a separate table number: ViT-B, the DeiT III recipe, and ImageNet-1K classification, keeping the H8 architecture unchanged apart from the patch-embedding constraint.

Config	Top-1 (%)	Factor Tested
Unconstrained linear patch embedding	82.4	Preserves subsequent block structure without guaranteed input equivariance
Equivariant patch embedding in H8(ViT-B)	83.0	Establishes a consistent geometric representation from the input

The 0.6-percentage-point difference supports geometrically consistent representations over merely retaining the same block-diagonal structure; it is not a complete, strictly parameter-matched causal decomposition. Figure 4b, page 11, also varies the number of equivariant blocks in ViT-L, motivating the choice of half the blocks; the cached figure does not support reliable pointwise extraction, so no curve-value table is invented.

Key Findings

• Table 2, page 12: on an A100-80GB with a token batch of \(64\times196\), fp16, and 1024-to-4096 channels, the standard layer takes 0.47 ms and the equivariant layer takes 0.19 ms; the table reports a 2.6-fold speedup, with timings already rounded.
• The same table reports a 5.33-fold equivariant-layer speedup for fp32 and 8192-to-32768 channels. This is a linear-layer microbenchmark, not the speed of a trained end-to-end ViT.
• Table 1, page 10, reports 3.54-fold throughput for ViT-22B with all blocks equivariant, but that large model only participates in efficiency benchmarking and is not trained and evaluated for accuracy in this work.

Highlights & Insights

• Equivariance is not an added regularizer: it directly restricts valid parameter connections. In a suitable basis, geometric knowledge becomes matrix operations that can be skipped.
• H8 combines early sharing of geometric patterns with late use of dataset-specific orientation preferences. This fits natural-image classification better than requiring strict invariance at every layer.
• Arithmetic-intensity analysis explains why impressive FLOP reductions may produce only modest throughput gains. It makes architectural benefits interpretable in terms of channel width, numerical precision, and hardware bandwidth.

Limitations & Future Work

• The authors systematically study only \(D_8\), without validating arbitrary rotations, translations, or larger transformation groups; discrete-symmetry results do not establish general geometric robustness.
• Actual training stops at ViT-H, representation-type channel allocations are not extensively ablated, and hyperparameters are not retuned. Larger-model efficiency benchmarks do not replace task-quality evidence.
• Strict invariantization may discard orientation information needed for semantic segmentation, making I8's ADE20K declines in Table 4 important; H8 instead loses strict rotation robustness.
• The supplied full-text cache contains the main paper and references but not Appendices A through F; the power-spectrum implementation, white-blood-cell experiments, and additional invariantization ablations cannot be individually verified from it.
• Reader interpretation: task-dependent equivariant depth and combinations with pruning or quantization merit investigation, but the paper provides no experimental conclusions about those combinations.

Related Work & Insights

• Versus Flopping for FLOPs: the earlier work reduces computation through reflection equivariance; Quick ViTs adds rotations, arithmetic-intensity analysis, and self-supervised downstream evaluation. It is not the first demonstration that equivariance can save computation.
• Versus Steerable CNNs: both constrain channel transformations through group representations, but Quick ViTs targets expensive Transformer linear layers and does not inherit convolutional translation-equivariance guarantees.
• Versus DeiT III and DINOv2: these are the adopted training recipes, not losses superseded by a new objective; Quick ViTs tests whether its backbone modification remains compatible with established training pipelines.
• Research implication: image matching and multi-view reconstruction may place greater value on geometric consistency. The authors identify these as future directions; current classification and segmentation results do not yet establish benefits for them.

Rating

• Novelty: 4/5. Extending reflection-based efficiency to the octic group yields clear structural savings, while building on an existing line of efficient equivariant networks.
• Experimental Thoroughness: 4/5. The study covers supervised and self-supervised learning, segmentation, rotation tests, and hardware microbenchmarks, with limits around very-large-scale training and appendix verification.
• Writing Quality: 4/5. Representation theory, implementation, and hardware bottlenecks connect clearly, but power-spectrum details depend on an appendix and extracted equations contain formatting damage.
• Value: 4/5. The work offers reusable structural constraints for efficient visual backbones, while deployment gains still require measurement at the intended width, precision, and hardware.