Accurate Zero-shot Quantization via Hierarchical Teacher-Assistant Distillation¶
Conference: ECCV 2026
Paper: Official ECCV page · Paper PDF
Code: https://github.com/snudm-starlab/ZEST
Area: Model Compression
Keywords: zero-shot quantization, teacher assistant, knowledge distillation, BatchNorm statistics, block-wise reconstruction
TL;DR¶
Zest connects a full-precision CNN to a low-bit student without original training images through an intermediate-precision teacher assistant, quantization-aware image recalibration, and block-wise reconstruction, reaching 70.50% ImageNet-1K Top-1 accuracy for ResNet-18 W4A4, 0.51 percentage points below the reported full-precision baseline.
Background & Motivation¶
Zero-shot quantization assumes access to a pretrained model, not to its original training data. A deployment team may inspect weights and stored batch-normalization statistics while being unable to use the images that originally trained the network. Existing approaches therefore optimize synthetic inputs or train a generator to reproduce the teacher's internal statistics, then use those inputs for quantization and knowledge distillation. This solves the problem of obtaining calibration data, but it does not ensure that the data suit a student restricted to three- or four-bit representations.
Aggressive quantization changes internal distributions through rounding and clipping. Some distinctions expressed by the full-precision teacher may fall inside numerical intervals that the student can no longer distinguish. If synthetic images are optimized exclusively for the full-precision network, the student must learn from mismatched inputs while chasing a numerically distant supervisor. The paper links this difficulty to unstable optimization and shrinking activation variance in deep layers, rather than treating synthetic image realism as the only bottleneck.
Zest applies the teacher-assistant idea along the precision axis instead of changing network depth. An intermediate-precision network first learns from the original teacher, then helps adjust the existing synthetic images and supervises the final student. Core idea: use an intermediate-bit teacher assistant to align both the calibration inputs and the reconstruction targets with the quantized domain, rather than asking the low-bit student to imitate the full-precision teacher directly.
Method¶
Overall Architecture¶
The inputs are a full-precision CNN with stored BatchNorm (BN) statistics and a target bit-width; the output is a low-bit student with the same network structure. Training constructs a Teacher-Assistant Bridge, applies Quantization-Aware Image Recalibration, and performs Block-Wise Reconstruction of the student. The diagram follows student construction; the same block-wise reconstruction mechanism is also used while preparing the assistant, not only in the final stage.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Full-precision CNN<br/>Weights and BN statistics"] --> B["Teacher-Assistant Bridge<br/>Synthetic data and intermediate bits"]
B --> C["Quantization-Aware<br/>Image Recalibration"]
C --> D["Block-Wise Reconstruction<br/>Assistant supervises student"]
A -->|Direct quantization initializes student| D
D --> E["Low-bit CNN<br/>Standalone inference"]
The hierarchy describes supervision, not a requirement that the student inherit the assistant's quantized weights. Algorithm 1 explicitly initializes the final student by quantizing the original full-precision model to the target precision. It then trains that student against assistant activations using recalibrated images. Once training finishes, inference requires the student alone, not a cascade of all three networks.
Key Designs¶
1. Teacher-Assistant Bridge: split a large precision gap into two transitions
The teacher, assistant, and student retain the same layer count and corresponding blocks, while their numerical precision differs. The default assistant uses W8A8, and the target students use W4A4 or W3A3; W/A denotes weight/activation bit-width. The assistant should preserve useful semantic information while already exposing the restricted dynamic range of integer representations. A full-precision target may contain fine-grained responses that the student cannot represent, whereas an intermediate-bit target is intended to remain informative without being numerically as distant.
Building this assistant begins with a generator conditioned on random noise and uniformly sampled class labels. The generator is trained so that its images induce BN means and variances matching the full-precision teacher's stored statistics. After generator training, it is frozen and produces the base synthetic calibration images. The teacher is quantized to intermediate precision, and its resulting assistant is fine-tuned block by block to reproduce teacher activations on those images. The presence of class conditioning should not be taken as evidence for an additional classification loss that the method section does not explicitly specify.
Higher assistant precision is not automatically better. Table 3 reports ResNet-18 W4A4 student accuracy of 69.15% with a W16A16 assistant, 70.50% with W8A8, 70.34% with W6A6, and 69.19% with W4A4. This supports a useful intermediate precision in the tested setting, not a universal rule that eight bits is optimal for every architecture or quantizer. The authors also state that one assistant works better than multiple assistants, but place that comparison in separate supplementary material unavailable in the local cache. No unverified numbers from that comparison are reproduced here.
2. Quantization-Aware Image Recalibration: preserve the starting semantics while correcting numerical mismatch
Base images were optimized for the full-precision network and may therefore elicit unsuitable responses around quantized clipping boundaries or shifted activation means. Zest copies these images into optimizable inputs and directly updates their pixels against the assistant's BN statistics. It does not train a new generator from scratch on the quantized assistant, nor does it resume optimization of the already frozen generator. The assistant supplies the statistical target for this second, quantization-aware calibration stage.
This separates semantic initialization from numerical adaptation. The full-precision teacher supplies an information-rich starting point, and the assistant subsequently adjusts how those images interact with limited-precision computations. Recalibration uses fewer iterations than the initial generator training: 500 versus 4,000 in the reported implementation. The intended effect is a restrained numerical adjustment rather than replacing the original semantic structure. The relevant success criterion is better calibration for student reconstruction, not necessarily greater visual realism of the images.
The recalibration target is still the intermediate-bit assistant, not the exact final student's distribution. Consequently, the method uses a nearby numerical domain as a proxy rather than guaranteeing coverage of every response specific to three-bit computation. The image-refinement ablation supports its usefulness, but the main paper does not separately quantify how much each type of clipping or distribution error is corrected.
3. Block-Wise Reconstruction: align intermediate responses before errors accumulate deeper in the network
Supervising only the final classifier output allows errors from different layers to interact and potentially conceal one another. Zest instead optimizes corresponding blocks sequentially from the beginning to the end of the network, matching their output activations. In the first training stage, the full-precision teacher supervises the assistant on base synthetic images. In the second stage, the assistant supervises the student on recalibrated images. Their shared structure makes block correspondence straightforward without a cross-architecture feature adapter.
The main training mechanism is intermediate activation reconstruction, not merely the output-level KL distillation introduced in the preliminaries. The Zest-B ablation specifically replaces block-wise reconstruction with end-to-end reconstruction over supervisor and learner final logits. For ResNet-18 W4A4, this variant reaches 69.99%, compared with 70.50% for the complete method, a 0.51-percentage-point difference. That comparison supports local reconstruction, although classification accuracy alone does not establish that every form of deep activation collapse has been eliminated.
Loss & Training¶
Two families of objectives drive the pipeline: BN mean/variance matching for synthesis and image recalibration, and corresponding-block activation matching for network fine-tuning. Equations (1) and (3)–(6) in the local text contain extraction-damaged operators, norms, or subscripts. This note therefore explains the mechanisms supported by the surrounding prose without inventing complete loss formulas. The exact norm conventions, coefficients, and implementation details require checking the original equations or author code.
The assistant uses standard W8A8 min-max quantization, while the student uses Learned Step Size Quantization (LSQ). Section 4.1 reports 1,024 synthetic images, 4,000 generator iterations, 500 image-recalibration iterations, and 20,000 reconstruction iterations per block with batch size 32. Thus, “zero-shot” means no access to real training data, not optimization-free or negligible-cost calibration. The paper retains an eight-bit first layer, with Section 4.2 specifically noting the first-layer weight exception; the W4A4/W3A3 labels should not be interpreted as uniform precision at every layer without exceptions.
Key Experimental Results¶
Main Results¶
The following selected entries come from Tables 1 and 2 and report ImageNet-1K validation Top-1 accuracy in percent, with higher values better. The validation set contains 50,000 images; these are reported paper results, not independently rerun comparisons under matched compute budgets. W/A indicates target weight/activation precision and must be read with the first-layer exception above.
| Architecture | W/A | Genie | GenQ | Zest | Gain over the better selected baseline |
|---|---|---|---|---|---|
| ResNet-18 | 4/4 | 69.66 | 69.77 | 70.50 | +0.73 pp |
| ResNet-50 | 4/4 | 75.59 | 75.50 | 76.32 | +0.73 pp |
| MobileNetV2 | 4/4 | 68.38 | 68.96 | 70.71 | +1.75 pp |
| ResNet-18 | 3/3 | 66.16 | 68.18 | 68.45 | +0.27 pp |
| ResNet-50 | 3/3 | 71.61 | 73.99 | 75.28 | +1.29 pp |
| MobileNetV2 | 3/3 | 57.54 | 59.15 | 67.74 | +8.59 pp |
The full-precision ResNet-18 baseline in Table 1 is 71.01%, making the W4A4 gap 0.51 percentage points rather than a 0.51% relative decline. Other methods appear in the original table, including IntraQ at 69.10% for MobileNetV2 W4A4. Accordingly, the gain column compares only the selected Genie and GenQ entries, not the strongest result among every prior method.
Ablation Study¶
Table 4 uses ResNet-18 W4A4 on the ImageNet-1K validation set; higher Top-1 accuracy is better. Drops below are calculated against the complete method in that same table.
| Configuration | Top-1 (%) | Drop from full method | Change |
|---|---|---|---|
| Full Zest | 70.50 | 0.00 pp | Assistant, recalibration, block-wise reconstruction |
| Zest-T | 70.23 | 0.27 pp | Remove the teacher-assistant bridge |
| Zest-R | 69.94 | 0.56 pp | Use base images without recalibration |
| Zest-B | 69.99 | 0.51 pp | Use end-to-end final-logit reconstruction |
Key Findings¶
- Removing image recalibration produces the largest single-component drop, 0.56 percentage points. An assistant network alone does not replace numerical adaptation of the calibration inputs.
- MobileNetV2 W3A3 benefits much more than ResNet-18 W3A3 in the selected comparison, highlighting architecture-dependent sensitivity to low-bit errors.
- Section 4.4 separately reports 70.42% ± 0.08% across five independent ResNet-18 W4A4 training runs. This must not be conflated with the main table's 70.50% or presented as if the latter were the five-run mean.
- Assistant precision has a non-monotonic effect in Table 3. That supports the tested intermediate integer representation, not universal optimality across assistant configurations.
Highlights & Insights¶
- The assistant changes both the supervisor and the training inputs. Connecting these two roles is more comprehensive than simply replacing the original teacher.
- Numerical precision can create a distillation capacity gap even when architecture is unchanged. Shared structure also makes intermediate-block supervision easier to interpret.
- Reusing synthetic images avoids restarting synthesis from a less expressive quantized network. The transferable principle is to preserve an informative starting point and then adapt it to deployment-specific numerical constraints.
Limitations & Future Work¶
- Author-stated direction: The conclusion proposes extending the pipeline to vision transformers and large language models. Current evidence concerns BN-equipped CNNs and does not establish compatibility with other normalization structures.
- Reader observation: Two reconstruction stages and 20,000 iterations per block add offline work. The main paper does not provide end-to-end runtime, peak-memory, or actual device-speedup comparisons.
- Evidence boundary: Detailed comparisons involving multiple assistants are assigned to separate supplementary material. The local 17-page cache contains the main paper and references, not that numerical evidence.
- Metric interpretation: Section 4.4 defines its smoothness index as inverse gradient variance but describes the improvement in percentage points without a clear normalization in the readable text. A precise cross-experiment smoothness gain should not be inferred from that wording.
- Strength of mechanism evidence: Optimization curves and accuracy support improved stability, but the main text lacks comprehensive layer-wise activation-variance measurements. Preventing activation collapse is an author interpretation, not an exhaustively verified guarantee.
Related Work & Insights¶
- Versus ZeroQ and GDFQ: All exploit knowledge stored in a pretrained network to synthesize calibration inputs. Zest emphasizes an intermediate numerical domain and a second adjustment of existing images rather than relying exclusively on full-precision statistics.
- Versus conventional teacher-assistant distillation: Earlier assistants address architectural capacity gaps such as depth or width. Here the transition axis is bit-width, and matching architectures enable direct block-level supervision.
- Versus Genie and GenQ: These provide strong synthetic-data quantization references, but Zest's advantage varies across architectures. The large MobileNetV2 three-bit gain should not be generalized to every configuration.
Rating¶
- Novelty: 3/5. A clear extension of teacher-assistant distillation to precision hierarchies, with a useful but largely compositional recalibration design.
- Experimental Thoroughness: 4/5. Three CNNs, two target precisions, component ablations, and repeated runs are covered; cost and broader model-family evaluations are missing.
- Writing Quality: 3/5. The algorithm order is clear, but some metric conventions need clarification, and local equation extraction limits exact verification.
- Value: 4/5. A practical path for low-bit CNN quantization without original data, subject to its architectural assumptions and offline optimization cost.