Aggregating Cross-Domain Knowledge via Learnable Tokens for Multi-Teacher Distillation

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/VISION-SJTU/ACTok
Area: Model Compression / Visual Representation Learning
Keywords: Multi-teacher distillation, learnable tokens, cross-architecture transfer, alternating interaction, vision foundation models

TL;DR

ACTok places channel-concatenated learnable tokens and alternating cross-space/intra-space interactions after a student backbone to consolidate CLIP, MAE, and DINO knowledge using ImageNet-1K, reaching 94.4% on Caltech101 with ViT-B without universally outperforming the strongest teacher or distillation baseline.

Background & Motivation

Vision foundation models are not simply more or less accurate versions of the same representation. CLIP's image-text objective favors global semantics, MAE's reconstruction objective emphasizes local detail, and DINO models capture object-level semantic structure. Consolidating these teachers into one student could produce a more versatile visual encoder, but their supervision can interfere: accommodating one teacher may erase details another teacher needs.

RADIO-style methods combine teachers through global and local feature alignment, while DUNE adds Transformer-based projection. Nevertheless, the student must still match multiple targets directly. ACTok argues that the problem extends beyond mismatched feature dimensions to conflicting representations. More proxy images can help, but increase distillation cost; interaction designs tied to ViT internals also make CNN transfer difficult. The authors instead use ImageNet-1K and post-backbone modules to explicitly learn how teacher-specific student representations should interact before alignment.

Here, cross-domain primarily refers to teachers' pretraining objectives and representation spaces, rather than conventional domain adaptation with target-domain labels. Core idea: use learnable tokens as knowledge intermediaries, exchange and reorganize information across teacher-specific student spaces, and then apply global and local supervision instead of imposing all heterogeneous constraints directly on the bare backbone output.

Method

Overall Architecture

The student and multiple pretrained teachers extract features from the same proxy image. Student features are projected into teacher-specific spaces and augmented with learnable token channels, followed by alternating cross-space and intra-space interaction. The final representation is split into token and visual-feature branches, which receive global and teacher-specific feature supervision, respectively.

Crucially, the interaction modules process multiple projections of student features, not teacher features inserted into the student's forward pass. Teachers supply targets during distillation; downstream tasks use the student and its aggregated projected features without executing the teachers. Nor are the interaction modules necessarily disposable training attachments: the paper explicitly studies their downstream learning rate and the choice of output features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Proxy image"] --> Student["Student backbone"]
    Input --> Teachers["Pretrained teachers"]
    Student --> Tokens["Channel-token proxies<br/>Multi-space projection"]
    Tokens --> Interaction["Alternating space interaction<br/>Cross-space then intra-space"]
    Interaction --> Alignment["Dual-level teacher alignment<br/>Global and local supervision"]
    Teachers -->|Supervision targets| Alignment
    Interaction --> Output["Student projected features<br/>Downstream tasks"]

Key Designs

1. Channel-token proxies: preserve the spatial layout and place knowledge intermediaries in the channels

A CNN feature map can be viewed as one token per spatial position, while a ViT already produces a class token and patch tokens. Both backbones therefore support a common view of spatial positions plus feature channels. Teacher-specific projectors, implemented as MLPs or Transformer blocks, map student features into multiple representations. Their visual-feature channels are unified to the largest teacher feature dimension, then concatenated with learnable token channels of dimension \(d_m\). This is not the usual prompt-learning operation of adding a few extra sequence positions.

Preserving the spatial layout matters. Concatenating teacher spaces along the sequence dimension would force attention to search over potentially conflicting tokens and disrupt the regular two-dimensional grid required by CNNs. The paper identifies an \(O(M^2L^2)\) attention term for sequence concatenation, where \(M\) is the teacher count and \(L\) the number of spatial tokens. Channel concatenation does not eliminate computation for free: it moves growth to the channel side, where grouped convolutions and increased attention-head counts offer implementation options for controlling cost. The cache does not fully specify token initialization and broadcasting, so a tensor-by-tensor implementation should not be invented from it.

2. Alternating space interaction: exchange complementary information, then reorganize each teacher space

Each interaction stage first concatenates all projected student spaces and their token components along the channel dimension. A cross-space module mixes them, and channel splitting restores the individual streams. Each stream then passes independently through an intra-space module that reorganizes the newly absorbed information for alignment with its own teacher. Repeating this process creates a trainable interaction path before the alignment objectives, rather than allowing teacher signals to meet only through summed losses.

For CNNs, the cross-space module is a residual Conv-GELU-Conv block with a depthwise convolution followed by a grouped pointwise convolution; the group count grows linearly with the teacher count. For ViTs, an attention block is used with the number of heads increased with the number of teachers. Intra-space modules use the corresponding convolutional or attention structure, but share parameters across teacher spaces within each stage. Folding the teacher dimension into the batch dimension enables parallel processing. Thus, teacher-specific spaces do not imply independently parameterized intra-space modules: projections and targets distinguish teachers, while the reorganization operator is shared.

This mechanism is not a mathematical constraint guaranteeing agreement among all gradients. A more defensible interpretation is that it adds trainable intermediate capacity for handling incompatible targets. The ablations demonstrate improved optimization, but the architecture alone does not establish that negative transfer has been eliminated.

3. Dual-level teacher alignment: tokens learn global semantics while visual branches preserve teacher differences

After the final interaction stage, each stream is split by channel into an aggregated token component and a visual-feature component. The token branch uses the class-token position for ViTs or global average pooling for CNNs, then the shared linear transformation described in the paper, before squared-error alignment with the corresponding teacher's global representation. Multiple teachers constrain the tokens' aggregation ability; this does not mean untransformed teacher vectors are forced to be identical.

The visual branch uses a teacher-specific linear projector to match output dimensions and different alignment objectives. CLIP uses pooling followed by a cosine-similarity constraint; DINO uses a cosine objective, supported by the comparison with MSE in original Table 6; other teachers use squared error. Although the paper groups this branch under patch-level alignment, the CLIP branch actually includes global pooling. It would therefore be misleading to describe all three teachers as using an identical per-patch loss.

When CNN and ViT spatial resolutions differ, both sides are reshaped into two-dimensional feature maps, and bilinear interpolation downsamples the higher-resolution map to the lower-resolution one before the relevant loss is computed. This resolves geometric shape mismatches; alternating interaction addresses representation relationships. The two operations solve different problems.

A Worked Example

Consider the paper's ViT-B/16 configuration. A \(224\times224\) image yields \(14\times14=196\) patch positions plus a class token. Student outputs are projected into CLIP, MAE, and DINOv3 spaces, each augmented with 512 token channels, then processed by one cross-space/intra-space interaction pair. This increases channel capacity, not the 196 spatial positions by a factor of three.

During training, aggregated tokens receive supervision from the three teachers' global targets, while visual features retain teacher-specific objectives. After distillation, the paper generally uses features projected into the DINO space for downstream tasks. This neither reruns the DINO teacher nor establishes that the DINO head is best on every dataset: the ResNet-50 experiment in original Table 5 achieves better CUB accuracy with the classification head.

Loss & Training

Let \(\mathcal L_i^m\) denote global token alignment to teacher \(i\) and \(\mathcal L_i^f\) its visual-feature alignment loss. The overall objective described in the paper can be written as:

\[ \mathcal L=\sum_{i=1}^{M}\mathcal L_i^m+\sum_{i=1}^{M}\mathcal L_i^f. \]

Teachers receive equal weights, without a separate teacher selector or sample-confidence weighting mechanism. Equations 1 through 4 contain extraction damage in the cached text. The expression above organizes the objective according to the surrounding prose; it does not invent unverified normalization, broadcasting, or reduction details. Figure 4 refers to supplementary material for the definitions of PCD, GIR, and SCR, which are not included in this cache. The figure is therefore treated only as the authors' gradient-conflict trend analysis, without inventing metric definitions or numerical values.

Students are randomly initialized and distilled for 300 epochs on approximately 1.2 million ImageNet-1K images at resolution 224. Training uses AdamW with learning rate \(10^{-4}\), weight decay 0.01, mixed precision, and four A800 80GB GPUs. Batch size is 1024 for CNNs and ViT-B, and 512 for ViT-L. CNNs use two interaction pairs and ViTs use one. Learnable token dimensions are 1024 for ResNet-50, 512 for ConvNeXt-B and ViT-B, and 768 for ViT-L.

Downstream adaptation retains the aggregation modules and gives them a smaller learning rate, such as 0.1 times the backbone rate, to reduce forgetting on limited task data. ImageNet classification uses another 10 fine-tuning epochs, while fine-grained classification uses 100 epochs with batch size 32. ADE20K freezes the backbone for linear probing at resolution 512. NYUv2 uses a DPT decoder, batch size 16, and 25k training iterations.

Key Experimental Results

Main Results

The following results are selected from the base-scale models in original Table 1. Classification columns report accuracy in percent, ADE20K reports mIoU in percent, and NYUv2 reports RMSE, where lower is better. Data counts follow the source table: teacher rows describe pretraining data, whereas distillation rows describe distillation data. They should not be interpreted as comparable totals for the entire system.

Method	Images	ImageNet	Aircraft	Caltech101	CUB	ADE20K	NYUv2 RMSE
MAE ViT-B/16	1.2M	83.6	85.9	91.6	81.2	22.2	0.347
DINOv3 ViT-B/16	1689M	86.2	92.6	93.5	90.3	50.9	0.293
Proteus ViT-B/14	1.2M	84.9	91.0	91.9	90.5	Not reported	0.304
RADIOv2.5-B	About 1400M	84.8	83.5	91.2	85.9	48.9	Not reported
ACTok ViT-B/16	1.2M	85.6	91.1	94.4	90.8	46.5	0.300

ACTok-B improves Aircraft accuracy over RADIOv2.5-B by 7.6 percentage points, but trails by 2.4 points on ADE20K. Against DINOv3-B, it gains 0.9 points on Caltech101 and 0.5 on CUB, while remaining behind on ImageNet, Aircraft, segmentation, and depth. Proteus and RADIO numbers come from their officially reported results, with different teacher scales, student architectures, and training procedures. This is not a strictly controlled equal-budget comparison.

Ablation Study

The next table reproduces the selected module ablation from original Table 4 with a ConvNeXt-B student. Both numerical columns are distillation training losses, not downstream accuracy. The source table provides no error bars for this ablation.

Configuration	Global token loss \(\mathcal L^m\)	Feature alignment loss \(\mathcal L^f\)	Interpretation
Without learnable token	Not applicable	0.29	0.06 above the full model
Without cross-space module	0.18	0.35	Slightly lower global loss, but higher feature loss
Without intra-space module	3.56	2.60	Both losses increase substantially
Full ACTok	0.20	0.23	Reference configuration

Removing the intra-space module has the largest impact, supporting the need to reorganize each space after information exchange. Cross-space interaction does not reduce every loss, so the change from 0.18 to 0.20 should not itself be interpreted as a performance regression. Feature alignment and downstream task outcomes must also be considered.

Original Table 6 further compares DINO-branch alignment objectives for ViT-L/16. All entries are classification accuracy percentages, with averages retained as reported.

DINO alignment objective	Aircraft	Caltech101	CUB2011	Average
MSE	89.4	92.8	90.6	90.9
Cosine	92.1	93.8	91.4	92.4

Key Findings

• Architectural similarity between teachers and the student is not necessarily beneficial. In original Table 2, switching a ConvNeXt-B student from all-CNN to all-ViT teachers raises ImageNet accuracy from 83.0 to 84.1 and reduces NYUv2 RMSE from 0.361 to 0.349. CUB remains at 88.7, so not every metric improves.
• Post-backbone modules contain useful knowledge. In original Table 5, the bare ResNet-50 backbone reaches 73.8 on CUB, while DINO, MAE, and classification heads reach 77.9, 77.0, and 79.7. Discarding projection heads cannot be assumed to preserve all gains.
• A smaller downstream learning rate contributes measurable gains. In original Table 7, scaling the interaction-module rate improves CUB from 89.2 to 90.8, Caltech101 from 94.3 to 94.4, and Aircraft from 90.3 to 91.1.
• Stability from input resolution 512 to 4096 is supported mainly by cosine-similarity maps and PCA visualizations. VQA and data-scaling experiments appear as figures and qualitative descriptions in the cache, without reliably readable per-condition numbers; no numerical accuracies are supplied here.

Highlights & Insights

• Teacher conflict becomes a representation-interaction problem rather than only a loss-weighting problem. Equal teacher weights still produce gains, suggesting that trainable intermediate representations can help absorb conflicts without an elaborate teacher-weighting policy.
• Channel concatenation makes the same concept usable with CNNs and ViTs. It preserves the spatial grid and separates cross-space communication from intra-space reorganization, placing architecture adaptation after the backbone rather than rewriting its internals.
• Aggregated features are not synonymous with bare backbone features. Output-head selection and adaptation learning rates affect final performance, so evaluation and reproduction must specify the actual representation used.

Limitations & Future Work

• The authors explicitly acknowledge proxy-data bias. Object-centric ImageNet-1K does not cover the complex multi-object concepts found in DINOv3's large-scale pretraining data; the dense-prediction gap to the strongest teacher is consistent with this limitation.
• Architecture-agnostic applicability does not imply uniform performance across architectures. CNN teacher combinations remain difficult to optimize, and the ResNet-50 student's ImageNet accuracy is 78.0 versus 80.9 for its classification teacher. All-ViT ensembles only partially mitigate these issues.
• From an experimental perspective, data efficiency is not end-to-end computational efficiency. Teachers have already been trained on large datasets, and student training executes multiple teachers plus interaction modules. The main text lacks a complete comparison of parameter counts, FLOPs, latency, and total GPU hours, so unchanged inference overhead or a thousand-fold total-cost reduction cannot be claimed.
• From an evidence perspective, module ablations primarily report training losses, without corresponding full downstream evaluations or repeated-run variance. A stronger follow-up would fix teachers, student, and budget, compare direct alignment with ACTok across tasks, and test whether multi-object proxy data closes the segmentation gap.

Related Work & Insights

• Versus Proteus: Proteus also accesses foundation-model knowledge using ImageNet-1K, so a small proxy dataset is not unique to ACTok. ACTok's central distinction is coordination among multiple teachers and cross-architecture interaction, not merely using fewer images.
• Versus RADIO / RADIOv2.5: Both consolidate multiple visual teachers, with RADIOv2.5 also emphasizing multi-resolution distillation. ACTok changes information exchange before alignment and obtains stronger results on some classification tasks with fewer proxy images, but does not win universally on segmentation.
• Versus DUNE: DUNE strengthens projection to integrate heterogeneous 2D and 3D teachers, whereas ACTok emphasizes learnable tokens and alternating cross-space/intra-space reorganization. Both suggest that projection spaces can be substantive learning components rather than dimension-matching utilities.
• Connection to VGGT-style token aggregation: Multiple views have potential geometric correspondence, whereas heterogeneous teachers may have conflicting semantics. ACTok therefore chooses channel concatenation instead of copying sequence concatenation. The transferable lesson is to ask whether sources share meaningful correspondence before deciding along which dimension their tokens should interact.

Rating

• Novelty: 4/5. Channel-token intermediaries and alternating space interaction form a concrete multi-teacher design, although they build on existing projection and token-aggregation ideas.
• Experimental Thoroughness: 4/5. CNNs, ViTs, multiple downstream tasks, and key ablations are covered, but strict equal-budget comparisons and full computational-cost reporting are missing.
• Writing Quality: 3/5. The main argument is clear, but the patch-level label requires care because CLIP uses pooling, and some figure/table references are imprecise; damaged equation extraction in the cache is not counted as a flaw in the paper itself.
• Value: 4/5. The framework provides a reusable structure for consolidating foundation models with limited proxy data, provided deployment retains the aggregation modules actually used and accounts for their cost.