MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment¶

Conference: ECCV 2026
Paper: ECCV
Code: https://github.com/SamsungLabs/MOCHA
Area: Object Detection
Keywords: knowledge distillation / cross-architecture alignment / multimodal supervision / few-shot personalized detection / representation alignment

TL;DR¶

MOCHA has a frozen LLaVa teacher produce object-level multimodal embeddings conditioned on "this box plus this class name," maps the multi-scale region features of a YOLO student into that space with a channel-attention translation module, and supervises the result with a dual objective combining pointwise alignment and relational geometry preservation, lifting a 3.2M-parameter detector by 10.1 points on average over the YOLOv8n baseline across four personalized detection benchmarks at only 3 ms/image of extra inference cost.

Background & Motivation¶

Personalized object detection must adapt a detector that recognizes a generic "dog" into one that recognizes the user's own dog, with only 1-5 user-provided samples. This setting matters most on edge devices such as phones and robots, and it is also the hardest: lightweight detectors have thin semantic priors to begin with, and in the low-data regime they drift toward neural collapse, where features of different classes become nearly collinear and discriminative power disappears. Large vision-language models such as CLIP, Flamingo, and LLaVa hold strong object-level semantics, but a 7B-scale model is neither deployable on-device nor practical inside a real-time loop. Knowledge distillation is the natural answer: move the VLM's semantics into a small detector and deploy only the small model.

The difficulty is that this transfer has to cross more than the usual cross-architecture gap. Methods such as OFA and cross-architecture KD generally assume teacher and student perform the same task, projecting their multi-scale features into a shared space for dense supervision; AuXFT, the closest prior work, instead injects DINO visual features into a lightweight detector to support prototype-based classification, one of the few approaches where the teacher guides a related auxiliary task. AuXFT, however, uses purely visual cues and aligns dense intermediate activations. What a VLM actually offers is not another, easier-to-align visual feature map but the multimodal semantics that arise from "this region" together with "this class name" — precisely what makes CLIP-style representations strong in open-vocabulary and few-shot settings. The tension follows: the teacher's representation is object-level, single-scale, and language-conditioned, whereas the student's features are pixel-level, multi-scale, and vision-only. The two are separated by a modality gap (vision vs. vision-language) and by a structural mismatch (multi-scale detector feature maps vs. object-centric representations). Copying dense activations transfers little semantics while forfeiting the compute savings that compact embeddings would have bought.

This paper's angle is to narrow alignment to the granularity that actually matters: since the end goal is instance-level discrimination rather than duplicating feature maps, alignment is anchored on detection boxes — the teacher crops each box, pairs it with the class name as a textual query, and fuses the two into a compact object embedding, while the student pools the same box's features across scales into a fixed-length descriptor, with a small translation module bridging the two spaces. A relational term then constrains the global geometry of the embedding space so that pointwise alignment cannot flatten inter-class structure into "every point matched, all relations scrambled." Core idea: recast cross-architecture distillation from dense feature-map alignment into box-anchored, object-level multimodal embedding alignment, using a translation module plus a pointwise/relational dual loss to carry both the semantics and the geometry of the VLM into a lightweight detector — with neither the teacher nor any text needed at inference.

Method¶

Overall Architecture¶

MOCHA is a three-stage knowledge distillation pipeline that moves the multimodal semantics of a frozen vision-language teacher into a vision-only lightweight detector. The inputs are annotated detection data (for pretraining and distillation) plus a handful of user-provided personalization samples; the output is a small detector that at deployment is fully standalone, requiring neither the teacher nor text input. The three stages are base pretraining (train a generic detector \(m_S=l_S\circ g_S\) with standard detection objectives on COCO/OpenImages, where \(g_S\) is the backbone and \(l_S\) the detection head), feature distillation (the teacher stays frozen while the student is aligned and regularized in the object-level embedding space), and few-shot personalization (the student backbone and the translation module are frozen together, and only a prototype-based classifier is trained). The core contributions live in the second stage.

Crucially, alignment happens at the level of object-region embeddings, not dense feature maps and not detection-head outputs: on the teacher side each ground-truth box is cropped into a small image and fed to the VLM together with that box's class name, yielding a fused vector that encodes "what this region is"; on the student side the same box is pooled into a descriptor from the multi-scale feature maps. The spatial dimensions are eliminated entirely, so both sides compare the same semantic object — that is what "objects-aware" concretely means in this method.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["input image + box labels"] --> B["Multimodal object-level supervision<br/>box crop + class-name query<br/>fusion → PCA → channel norm"]
    A --> C["Region-aligned multi-scale student aggregation<br/>crop + pool + concat"]
    B --> D["Translation module t_S<br/>channel attention + MLP"]
    C --> D
    D --> E["Dual-objective distillation loss<br/>pointwise + relational"]
    E -->|freeze student and t_S| F["Few-shot personalization<br/>prototype classifier overrides labels"]

Key Designs¶

1. Multimodal object-level supervision: make the teacher answer "this region plus this class" rather than hand over visual features

What the teacher produces — semantics anchored on the box — sets the ceiling for everything that follows. Each box \(b_i\) is cropped and passed through the teacher's visual encoder (the CLIP ViT-B/32 inside LLaVa-1.5-7B) to obtain a visual embedding \(Z_{V,i}\), while the class name \(c_i\) is embedded as a textual query \(Z_{Q,i}\). Both go through the teacher's shared projection matrix \(W_T\) into a joint representation space and are passed to the frozen language model, which computes the token sequence describing the interaction between region and class; averaging along the temporal dimension yields the fused semantic \(h_i\). This \(h_i\) is language-conditioned — it does not describe "there is some texture here" but "this is a dog" — exactly what a purely visual teacher cannot supply.

Using \(h_i\) alone, however, discards appearance: the paper states explicitly that the multimodal feature does not necessarily retain appearance-level cues, yet personalized detection depends on appearance to tell this dog from that dog. MOCHA therefore concatenates the corresponding visual class token \(z_{V,i}\), rescaled by \(\gamma=\|h_i\|\) to match the textual feature's magnitude, producing a representation that carries both raw appearance and its label-conditioned semantic interpretation:

\[u_i=\mathrm{concat}\left(\gamma z_{V,i},\,h_i\right)\in\mathbb{R}^{d},\qquad d=d_z+d_h,\qquad \gamma=\|h_i\|\]

This vector is high-dimensional and redundant (\(d\) reaches 4608), so PCA is applied offline on the distillation dataset to keep the first \(d_t\) components, followed by dividing each channel by that channel's activation standard deviation \(\sigma_c\). The PCA basis is precomputed with cached teacher embeddings, so no VLM runs during training iterations; the normalization equalizes channel contributions so a few high-magnitude channels cannot dominate the distance metric — the ablation shows the normalized version clearly beating the unnormalized one in the 1-shot setting. The paper notes in the appendix that \(\sigma_c\) decays approximately hyperbolically with channel index, so it can be estimated from the index alone.

2. Region-aligned multi-scale student aggregation: turn "the same box" into a comparable descriptor on the student side

The student is a multi-scale CNN detector while the teacher emits a single object token, so scale and granularity do not match; the student side must first construct a representation that also describes that box. The construction mirrors the teacher's cropping: for each ground-truth box, the feature maps are first resized to a common size to cancel resolution differences so that the same spatial location corresponds to the same region across levels; region-aligned features \(F_{A,j,i}\) are then extracted at each scale \(j\), spatially average-pooled into fixed-length descriptors \(f_{A,j,i}\), and concatenated into \(f_{A,i}\). The box's multi-scale context is thus compressed into one vector that refers to the same pixels the teacher cropped — both sides look at the same object, which is what makes alignment meaningful.

Pooling rather than retaining the feature map is deliberate: object-level supervision needs a single region descriptor, and keeping dense activations would only introduce a dimensionality explosion and a large alignment cost. In the ablation, replacing the pooled form with its dense, unpooled equivalent brings no advantage, while concatenating multiple scales yields consistent gains over a single level, indicating that multi-level context helps refine small and occluded objects.

3. Translation module \(t_S\): cross-architecture alignment is a semantic mapping, not a linear projection

Even with matched dimensions, the student's region descriptor and the teacher's PCA target live in two heterogeneous spaces: one is a convolutional detector's latent, organized mainly along scale and texture by the detection objective; the other is a VLM's latent, organized along semantics and language conditioning by image-text alignment. \(t_S\) moves the former into the latter. It is built as one transformer encoder block, whose core is a channel-wise multi-head self-attention plus a lightweight MLP: attention models inter-channel dependencies so the network decides which channels carry transferable semantics, and the MLP adapts the representation to the target dimensionality \(d_t\). It is trained jointly with the detector rather than pretrained in a separate stage, so alignment evolves together with the detector — as the detector produces better-aligned features, the translation module moves up with it. In the personalization stage \(t_S\) is frozen and reused alongside the backbone, and user samples must pass through it to land in the same space.

4. Dual-objective distillation loss: beyond pointwise alignment, preserve the relative geometry of the embedding space

Pointwise regression alone has a subtle failure mode: the student pulls every region as close as possible to its target, but the relative arrangement among targets — which classes sit near each other, whether instances of a class cluster — can be crushed, and instance-level discrimination depends on exactly that neighborhood structure. MOCHA therefore adds a relational term \(L_{\mathrm{emb}}\): over a batch of regions, pairwise Euclidean distance matrices are computed for student features and teacher targets, the diagonal self-distances are discarded, both are turned into distributions by a softmax, and the cross-entropy between them is minimized — the student must reproduce who is near whom, not merely each point's absolute position. The negative sign before the distances means closer pairs receive larger probability mass, so the gradient concentrates on the genuinely important neighbor pairs.

The term is \(O(N^2)\) in theory, but each box corresponds to one ground-truth region and images contain few objects in practice (on OpenImages, \(N\approx8\) regions per image on average), so the overhead is negligible next to convolution. A toy experiment validates the mechanism: ten 2D points (proxies for student embeddings) are optimized to match the pairwise distance distribution of a fixed 3D reference configuration (proxy for teacher embeddings); under \(L_{\mathrm{emb}}\) alone, the agreement between the 2D top-k nearest neighbors and those of the 3D reference rises steadily and converges quickly, showing that relational supervision preserves neighborhood structure even under severe dimensionality reduction. In the real ablation, removing \(L_{\mathrm{emb}}\) costs accuracy on both datasets, making it the most sensitive component besides the pretraining strategy.

A Worked Example¶

Walking one POD scene image (the paper's feature-similarity visualization uses a POD scene with three objects) through the pipeline:

Pretraining: YOLOv8n is trained on OpenImages with standard detection objectives; starting from AuXFT weights instead leaves both generic detection and instance discrimination stronger.
Building teacher targets: roughly 8 ground-truth boxes on the image are cropped, paired with their class names, and passed through the frozen LLaVa to obtain \(h_i\); concatenation with the rescaled visual class token gives \(u_i\) (up to 4608 dimensions), which is compressed offline by PCA to 512 dimensions and divided channel-wise by \(\sigma_c\).
Alignment: the same boxes are region-pooled across the student's feature maps, concatenated, and passed through \(t_S\) into a 512-dimensional \(f'_{A,i}\), compared pointwise against \(u'_i\), with the relational term computed over all 8 regions. The detection loss still supervises box regression and classification.
Personalization: the user supplies one labeled sample (say "my mug"); the frozen backbone plus \(t_S\) turns that instance's embedding into a class prototype, and a nearest-class-mean classifier is trained. At inference the detection head's generic class prediction is overridden by the personal label. The inference path contains no LLaVa, no text prompt, and no online adaptation — only the small \(t_S\) module (about 3 ms/image on YOLOv8n, a relative increase of roughly 10%).

Loss & Training¶

The distillation-stage objective combines the detection loss with two auxiliary terms:

\[\mathcal{L}=\mathcal{L}_{\mathrm{det}}+\lambda_{\mathrm{dist}}\mathcal{L}_{\mathrm{dist}}+\lambda_{\mathrm{emb}}\mathcal{L}_{\mathrm{emb}}\]

The pointwise distillation loss \(L_{\mathrm{dist}}\) averages the \(\ell_1\) and \(\ell_2\) distances (the paper describes it as "the average of \(\ell_1\) and \(\ell_2\) distances", while the extracted equation only retains a sum over the \(n\) regions; whether a \(1/2\) factor is present ⚠️ refer to the original paper): the \(\ell_1\) term is robust to outlier regions and the \(\ell_2\) term finely closes the magnitude gap, so together they make alignment both stable and accurate.

\[\mathcal{L}_{\mathrm{dist}}=\frac{1}{n}\sum_{i=1}^{n}\left(\left\|f'_{A,i}-u'_i\right\|_1+\left\|f'_{A,i}-u'_i\right\|_2\right)\]

The relational term \(L_{\mathrm{emb}}\) is defined in Key Design 4. On the training recipe: distillation runs for 50 epochs by default, reduced to 20 when starting from AuXFT weights because convergence is faster; the PCA dimension is \(d_t=512\); detector, translation module, and both losses are optimized jointly while the teacher stays frozen and its embeddings are precomputed and cached. In personalization, backbone and \(t_S\) are entirely frozen and only the prototype classifier is trained, evaluated under 1-shot and 5-shot budgets.

Key Experimental Results¶

Main Results¶

Four personalized datasets: PerSeg and POD report end-to-end mAP50-95, while CORe50 and iCubWorld follow the AuXFT retrieval protocol (an object counts as correct if it appears among the top-ranked detector proposals). The student is YOLOv8n and the teacher LLaVa-1.5-7B, with 50 distillation epochs unless stated otherwise.

Teacher	Method	PerSeg 1s	POD 1s	POD 5s	CORe50 1s	CORe50 5s	iCubWorld 1s	iCubWorld 5s	Avg
–	YOLOv8n baseline	41.2	23.6	30.4	57.8	67.3	51.2	68.4	48.6
DINO	AuXFT	48.8	31.5	38.8	58.8	69.3	55.0	74.5	53.8
LLaVa	KL div. + MSE	50.1	27.6	30.5	53.3	65.7	61.1	78.5	52.1
LLaVa	GLIP	52.1	27.3	32.7	58.7	64.6	61.2	71.1	52.2
LLaVa	SKDF	47.0	25.8	28.4	58.4	67.3	62.7	74.1	52.0
LLaVa	MOCHA (AuXFT init)	59.1	36.3	45.9	60.9	70.6	61.4	77.0	58.7

The student-architecture ablation (Tab. 3) shows the supervision transfers to other detectors as well:

Student	No distillation	AuXFT	MOCHA (AuXFT)
YOLOv8n (≈3.2M, PerSeg 1s / iCub 1s / iCub 5s)	41.2 / 51.2 / 68.4	48.8 / 55.0 / 74.5	59.1 / 61.4 / 77.0
YOLOv11n (≈2.6M)	49.3 / 52.7 / 72.8	52.8 / 53.9 / 73.3	56.2 / 57.5 / 74.1
RT-DETR-l (≈45M)	43.5 / 45.0 / 63.5	45.2 / 46.1 / 64.0	47.0 / 47.8 / 64.4

Ablation Study¶

Component and hyper-parameter ablation (PerSeg 1-shot / iCubWorld 1-shot / iCubWorld 5-shot, all against the full model with AuXFT initialization and \(d_t=512\)):

Config	PerSeg 1s	iCub 1s	iCub 5s	Note
Full MOCHA (AuXFT, \(d_t=512\))	59.1	61.4	77.0	reference
w/o \(L_{\mathrm{emb}}\)	56.6	59.8	74.9	both datasets drop without the relational term
translator w/o attention	56.8	57.5	73.2	largest drop on iCubWorld (-3.9)
translator w/o MLP	55.7	61.4	76.6	PerSeg drops 3.4
\(d_t=256\)	56.4	61.2	75.7	over-compressed, semantics lost
\(d_t=384\)	56.2	59.6	74.7	same direction
\(d_t=1024\)	53.8	58.4	77.0	larger brings no gain and costs 5.3 on PerSeg
distillation data + color augmentation	53.3	58.4	73.4	standard YOLO augmentation hurts robustness
encoder lr ×2	53.6	59.0	75.8	scaling either side alone costs accuracy
decoder lr ×2	53.9	59.7	75.9	same direction
pretrain on OpenImages	53.8	56.8	70.4	OpenImages-only pretraining is weakest
pretrain on COCO	55.4	60.6	77.6	COCO weights already beat all prior methods
pretrain on AuXFT	59.1	61.4	77.0	best, and compatible with MOCHA's architecture

The controlled study of teacher signals (oracle setting, using ground-truth boxes and therefore measuring classification ability alone): DINO alone averages 59.9, CLIP visual tokens alone 72.7, and LLaVa's fused semantics \(h_i\) alone only 62.2, while concatenating \(h_i\) with CLIP visual tokens reaches 74.2/74.3; the corresponding distilled students score 43.2 (no distillation) / 50.7 / 48.0 / 54.0, confirming that visual and textual cues are genuinely complementary rather than the gains coming from simply switching to a stronger teacher.

Key Findings¶

Single-modality supervision is not enough; complementarity is the key: on the teacher side, textual semantics alone (\(h_i\), 62.2) are actually worse than visual tokens alone (CLIP, 72.7), yet concatenating the two (74.3) beats either, and since CLIP is already part of LLaVa the fusion adds essentially no cost. This is why \(u_i\) must retain an appearance token — semantics answer "which class" while appearance answers "which individual."
The relational term is the second most sensitive component: removing \(L_{\mathrm{emb}}\) costs 2.5 on PerSeg, 1.6 on iCubWorld 1-shot, and 2.1 on 5-shot. The paper's explanation is that pointwise alignment tends to distort the relative geometry of the embedding space, and few-shot personalization discriminates precisely through that neighborhood structure; the toy experiment (2D points fitting a 3D distance distribution) further shows relational supervision preserves top-k neighborhoods under severe compression and converges quickly.
PCA is neither tighter nor larger at will: as \(d_t\) grows from 512 to 1024 accuracy plateaus or even falls (PerSeg drops to 53.8), and shrinking to 256/384 also loses points, making 512 the balance between compactness and expressiveness; the normalized variant beats the unnormalized one especially in the 1-shot setting, showing that channel-scale alignment matters in the low-data regime.
The more YOLO-like the student, the larger the gain: MOCHA improves most over AuXFT on YOLOv8n (PerSeg +10.3); YOLOv11n still gains but by less (PerSeg +3.4), and the transformer-based RT-DETR-l only by about +1.8. The paper attributes this to RT-DETR-l being an order of magnitude larger and architecturally far from YOLO, making the translation harder to learn — though it is also closer to the teacher to begin with, so less needs to be moved.
Pretraining initialization matters a lot: with OpenImages-only pretraining MOCHA reaches just 53.8, COCO weights already suffice to surpass all prior methods (55.4), and starting from AuXFT weights is best (59.1), showing that this distillation and supervision signal compounds with an initialization that already has instance-discrimination ability.
The cost is very low: the personalization and inference stages involve no VLM, no text prompt, and no online adaptation; on YOLOv8n and PerSeg the overhead is about 3 ms/image (roughly +10%), which is the central advantage over ViLD, GLIP, and PerSAM-style methods that must carry a large model or prompts into inference.
Qualitatively, MOCHA produces tighter boxes and fewer class errors on PerSeg, whereas open-vocabulary methods such as ViLD struggle in highly personalized scenarios — open-vocabulary capability is not the same thing as instance-level discrimination.

Highlights & Insights¶

Moving the alignment granularity from feature maps to objects: this is the paper's pivotal trade-off. Since the goal is instance-level discrimination, duplicating dense activations is neither necessary nor desirable; eliminating the spatial dimensions makes both sides compare the same semantic object, turning the alignment problem from cross-resolution feature regression into compact-embedding matching and cutting both training and deployment cost along the way. Any setting that distills a large teacher into a small student for a region- or instance-level task can borrow this framing.
Treating the teacher's multimodality as label-conditioned supervision rather than a stronger feature: \(h_i\) alone is weaker than CLIP visual tokens, and the real gain comes from combining semantics with appearance. This counter-intuitive control experiment is valuable because it shows a VLM's knowledge must be taken from the part that is actually good at it.
A relational loss as a geometry preserver: replacing pointwise regression with cross-entropy over pairwise distance distributions mitigates the common distillation failure of "every point matched but the structure collapsed," and because it is computed only on ground-truth regions, the \(O(N^2)\) cost is essentially free in practice. The trick transfers to any representation-alignment task that needs to preserve structure (cross-modal retrieval, prototype networks, drift constraints in continual learning).
Decoupling personalization from training: learning semantics is delegated to server-side distillation while recognizing individuals is left to a prototype classifier on frozen features, so the device only pays for one forward pass and a nearest-class-mean lookup. This division makes personalization a zero-retraining operation, which is practical from an engineering standpoint.

Limitations & Future Work¶

Cross-architecture gains depend heavily on the student-teacher gap: on RT-DETR-l, a student that is both large and architecturally close to the teacher, the improvement is only one or two points, so the method's main battleground remains small CNN detectors such as YOLO — a limitation the authors themselves frame as suitability for on-device scenarios.
The distillation targets are ground-truth boxes: cropping by human annotation at training time ties the ceiling of teacher supervision to annotation quality and precludes using unlabeled box data; extending to open-world or semi-supervised settings would first require deciding whose predicted boxes serve as anchors and how to keep wrong boxes from contaminating the alignment targets.
Each image requires cropping several regions and running them through the VLM to precompute targets (offline and cached, but still), and the paper does not discuss the cost or engineering complexity of this step at very large scale; moreover the PCA basis and channel standard deviations \(\sigma_c\) are fitted offline on the distillation dataset, so whether those statistics still hold in a very different target domain deserves verification.
Personalization remains a closed-set "user supplies samples, labels get overridden" procedure and does not handle users adding classes at will, hierarchical relations among classes, or noisy user samples; the variance under 1-shot (standard deviations of 3-6 in the tables) also suggests it is still some way from robust.
The roughly 3 ms/image overhead on YOLOv8n comes from the translation module, a single small forward pass; the paper nonetheless reports no measured latency or memory on real edge hardware, nor a pure-vision self-supervised pretraining baseline at comparable student size, so the net benefit of multimodal supervision over simply better visual pretraining remains an open question.

vs AuXFT (IROS 2024): AuXFT is the only prior work addressing the same setting, injecting DINO's high-level visual features into a lightweight detector to support prototype classification. MOCHA differs in three ways: the supervision is multimodal (visual plus textual) rather than purely visual; the alignment targets compact region-level embeddings rather than dense intermediate activations (cheaper, and semantically denser); and an explicit relational regularizer constrains the geometry among instances. It wins by +4.9 on average and leads by more on hard domains such as PerSeg and POD where fine-grained semantic alignment is critical.
vs OFA / cross-architecture KD: these project teacher and student multi-scale features into a shared space for dense supervision, assuming both perform the same task and share task semantics. MOCHA instead has the teacher guide a related auxiliary task and aligns only object-level embeddings, freeing it from strict correspondence between resolutions and feature-map shapes.
vs GLIP / SKDF / DRKD / KL(+MSE): these represent different supervision paradigms — similarity/feature regression, relational matching, and output-space KL. Re-implemented within MOCHA's own multimodal region-level setting, KL and OFA-style supervision yield limited gains while similarity- and relation-based objectives (GLIP, SKDF, DRKD) do better but still trail MOCHA, indicating that neither pointwise regression nor pure relational matching suffices: pointwise alignment plus explicit relational regularization is the combination that makes this signal work.
vs open-vocabulary and prompt-based methods such as ViLD / GLIP / PerSAM / SwissDINO: they drive generalist models with textual or visual prompts, enabling zero-shot recognition at high inference cost and without directly addressing "which instance of this class." MOCHA uses the VLM once, offline, on the server side and removes it entirely at deployment — the opposite trade-off.

Rating¶

Novelty: ⭐⭐⭐⭐ Reframing cross-architecture distillation from feature-map alignment to object-level multimodal embedding alignment is a clean and convincing reformulation, though the translation module, relational loss, and prototype personalization are organic combinations of existing components.
Experimental Thoroughness: ⭐⭐⭐⭐ Four personalized benchmarks, three student architectures, and complete component and hyper-parameter ablations, plus a toy experiment explaining the relational term; missing measured edge-device latency and a "better visual pretraining" control baseline.
Writing Quality: ⭐⭐⭐⭐ The three stages and contributions are clearly delineated and the teacher-signal control study is informative; some equations are incomplete after extraction from the paper layout, and a few table columns require careful reading to align.
Value: ⭐⭐⭐⭐ A directly usable, low-overhead solution for on-device personalized detection, and the "object-level alignment" idea transfers to other large-teacher-to-small-student distillation tasks.