GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

Conference: ECCV2026
Paper: ECCV official page
Project: GenRecal
Area: Model Compression
Keywords: cross-tokenizer distillation, feature recalibration, teacher language head, representation alignment, small vision-language models

TL;DR

GenRecal uses a training-time Recalibrator to connect a small vision-language model's hidden features to the teacher's language head, bypassing direct cross-tokenizer correspondence and improving InternVL2.5-8B from 56.0 to 68.1 on MMMU, while removing all auxiliary modules for deployment and retaining the student's original architecture.

Background & Motivation

Transferring the capabilities of a large vision-language model (VLM) to a smaller one involves more than reducing parameter count. Conventional token-level distillation typically compares teacher and student output distributions, but different tokenizers can differ in vocabulary size, text segmentation, and token indices. The same answer may have different sequence lengths, while image-splitting strategies can also change input length. Directly applying KL divergence can therefore encounter incompatible dimensions or compare semantically unrelated tokens at the same index. Even InternVL2.5-8B and InternVL2.5-78B use InternLM2.5 and Qwen2.5, respectively, so a shared VLM family name does not establish distillation compatibility.

An intuitive alternative is to generate answers with a large model and use the text for supervised fine-tuning, but this discards its soft distributional information. Cross-vocabulary matching offers another route, yet must still handle segmentation boundaries and sequence lengths. The authors further argue that even when Qwen2-VL-7B and 72B share a tokenizer, the smaller model's representation space and language-head capacity can constrain knowledge transfer. The challenge is therefore not just a vocabulary interface: the teacher must be able to interpret the visual and linguistic information already extracted by the student.

GenRecal does not force the student to change its tokenizer. Instead, it learns a training-time route into the teacher's representation space and obtains supervision through the teacher's language head. Core Idea: let the student supply hidden question representations and the teacher supply the answer sequence's coordinate system, perform conditional generation-based alignment in a learnable Recalibrator, and pass the resulting learning signal back to the student.

Method

Overall Architecture

Training uses the same image, question, and reference answer, processed separately through each model's visual encoding pipeline and tokenizer. GenRecal first establishes a Cross-Tokenizer Recalibration path, warms it up through Teacher-Anchored Alignment, and then performs Distillation and Standalone Fine-Tuning to obtain a student that generates answers independently.

The large model participates only during training. Online inference requires neither teacher answers, the teacher's language head, nor the Recalibrator: it uses the student's original vision encoder, visual projector, language-model body, and language head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Training: image, question<br/>and reference answer"] --> Bridge["Cross-Tokenizer Recalibration"]
    Bridge --> Align["Teacher-Anchored Alignment"]
    Align --> Distill["Distillation and<br/>Standalone Fine-Tuning"]
    Labels["Training supervision:<br/>teacher distributions and labels"] -.-> Align
    Labels -.-> Distill
    Distill --> Export["Remove teacher<br/>and Recalibrator"]
    Export --> Student["Inference: original student<br/>image and question to answer"]

Key Designs

1. Cross-Tokenizer Recalibration: move the comparison before the teacher's language head

The authors divide each VLM's language component into a VLM-body and a VLM-head. The body produces contextualized hidden features, while the head projects them into the vocabulary. After running both bodies on the same question-answer sample, the recalibration path extracts the student's question features and the teacher's answer features and concatenates them into a new sequence. The question portion retains the student's visual and linguistic representations, while the answer portion naturally follows the teacher's token count and ordering, eliminating the need to match individual student and teacher answer tokens.

When hidden dimensions differ, Proj-pre first maps the teacher's answer features to the student's hidden dimension. The concatenated sequence passes through two Transformer blocks configured like the student's decoder, and Proj-post restores the teacher's hidden dimension. Only the answer portion is then passed to the teacher's language head. Both projectors are linear, but the intervening Rec-body is a nonlinear sequence model: it must enable teacher answer positions to use student question context, rather than independently transform each token's channels.

Because the concatenated sequence differs from either original sequence, the authors reassign position IDs, introduce new RoPE positional embeddings, and apply an additional layer normalization to the recalibrated output. Rec-body retains the student's causal mask, attention-head count, and FFN structure. This enables autoregressive modeling in the new sequence order while keeping the module shallow; cross-tokenizer compatibility does not mean discarding position or sequence dependencies.

2. Teacher-Anchored Alignment: learn to read the student while constraining representation drift

The first stage freezes both teacher and student and trains only the Recalibrator. The mixed path uses student question features and teacher answer features. Its objectives require the recalibrated answer features, read through the teacher's language head, to predict the correct teacher-token labels and match the teacher's original answer distribution. These provide explicit answer targets and soft teacher knowledge, respectively, rather than directly comparing two different vocabularies.

The same stage also runs a teacher-only path: both teacher question and answer features pass through the Recalibrator, with answer cross-entropy and KL supervision from the teacher distribution. Because Rec-body operates at the student's hidden dimension, inputs to this teacher-only path require Proj-pre dimension matching. This regularization supplies a teacher-side reference for the recalibrated space, preventing the mixed path from lowering its loss while drifting away from the semantics of teacher features. It is neither an additional token-level feature MSE nor a second teacher.

Crucially, the same teacher language head interprets both paths. Student weights remain completely unchanged after the first stage, so successful bridge alignment should not be mistaken for an improved standalone student. The Stage 1 checkpoint in the paper's Table 9 is indeed identical to the baseline.

3. Distillation and Standalone Fine-Tuning: retain the auxiliary path's benefits in the student

The second stage retains teacher-side cross-entropy and KL supervision through the recalibrated path and adds the student's own answer cross-entropy. The main text explicitly describes training the student VLM-body, while Table 9 summarizes the trainable components as Recalibrator + Student. The teacher remains frozen. Learning signals from the teacher head flow through the Recalibrator into student question features, updating the student body toward representations the teacher can interpret. The student's native head path also ensures it can express answers using its own tokenizer.

The third stage removes the teacher and Recalibrator and performs supervised fine-tuning (SFT) on the student alone, updating every student parameter except the vision encoder. This includes the original language body and head as well as the visual projector, further adapting the distilled representations for independent instruction following. Deployment therefore retains the original student architecture, rather than a hybrid system consisting of a small model, an adapter, and a large-model head.

A Worked Example

Consider a chart question: given a bar chart and the prompt "Which year has the highest value?", the reference answer is "2020". This is an illustrative information-flow example, not an additional experimental sample from the paper, and it makes no assumption about how either tokenizer segments the text.

1. Teacher and student independently encode the image, question, and reference answer; their question and answer token counts may differ.
2. The Recalibrator receives student question features and teacher answer features projected into the student's dimension. After Proj-post, the teacher head predicts teacher-token answer labels.
3. Stage 1 trains only the Recalibrator to learn this conditional generation relation, with the teacher-only path maintaining a teacher-side reference. Stage 2 enables student-body learning and also requires the student's native output to predict its own answer labels.
4. Stage 3 fine-tunes only the student. At deployment, the chart and question alone are supplied, and the student generates "2020" without receiving a reference answer or querying the teacher.

Loss & Training

The following summarizes the mixed-path objectives in Algorithms 1 and 3. Here, \(p_l\) is the teacher's original answer distribution, \(p_r\) is the distribution obtained by passing recalibrated features through the teacher head, and \(y_l\) denotes ground-truth answer labels under the teacher tokenizer. Stage 2 additionally uses the native student distribution \(p_s\) and student labels \(y_s\):

\[ \mathcal{L}_{\mathrm{bridge}} = \mathrm{CE}(p_r,y_l) + D_{\mathrm{KL}}(p_l\Vert p_r),\qquad \mathcal{L}_{\mathrm{stage2}} = \mathcal{L}_{\mathrm{bridge}} + \mathrm{CE}(p_s,y_s). \]

Stage 1 additionally optimizes the teacher-only path from Algorithm 2 alongside the mixed path. This is an algorithm-level summary: it does not supply temperatures, regularization weights, or token-shifting details absent from the cache, and specifically does not reconstruct the corrupted Equation (1). The main text refers to Appendix H for further regularization details, but the available cache ends with the references and does not contain that appendix.

The first two stages each use the complete 9M dataset. The main text describes Stage 3 as using 6M samples after removing general visual question answering data. Training uses AdamW, DeepSpeed ZeRO-3, and 128 A100 80GB GPUs, with the learning rate decaying linearly from \(10^{-4}\) to \(10^{-5}\) in each stage. Gradient accumulation spans 16 steps; total batch size is 8192 in Stages 1 and 3 and 4096 in Stage 2. The reported stage durations are approximately 2-4, 8-11, and 4-6 hours, depending on model size.

A reproducibility caveat remains: the data-scale analysis in Table 10 describes Stage 3 as a random approximately 70% subset of Stage 2 data, unlike the main text's removal of general visual question answering samples. These descriptions are not silently merged into a single recipe here; reproductions need to check the authors' implementation.

Key Experimental Results

Main Results

Table A selects three representative metrics from the paper's Table 1. Both GenRecal students use InternVL2.5-78B as teacher. Values are the reported benchmark scores, with higher being better; they are not all described as a common accuracy measure.

Student model	Training configuration	MathVista	MM-Vet	MMMU
Qwen2-VL-7B	Original baseline	58.2	62.0	54.1
Qwen2-VL-7B	GenRecal	68.8	70.4	65.6
InternVL2.5-8B	Original baseline	64.4	62.8	56.0
InternVL2.5-8B	GenRecal	74.9	73.2	68.1

InternVL2.5-8B gains 12.1 points on MMMU and 10.4 points on MM-Vet. However, the InternVL2.5-78B teacher scores 70.1 on MMMU in the paper's Table 2, still above the student's 68.1. Surpassing the teacher on some tasks does not establish universal superiority.

Ablation Study

Table B selects results from the paper's Table 5, fixing InternVL2.5-8B as student and InternVL2.5-78B as teacher and testing the recalibration autoregressive loss. Avg averages the 11 benchmark scores in that source table, not the four scores used in the next table.

Configuration	MathVista	MM-Vet	MMMU	Avg (11 benchmarks)
Original baseline	64.4	62.8	56.0	68.1
GenRecal without autoregressive loss, KL only	55.2	55.4	50.6	59.8
GenRecal with autoregressive loss	74.9	73.2	68.1	77.1

Table C selects stage-wise checkpoints from the paper's Table 9, using the same teacher-student pair as Table B. Avg averages only MMB, MathVista, MM-Vet, and MMMU to identify where gains emerge in training.

Checkpoint	Data for this stage	MMB	MathVista	MM-Vet	MMMU	Avg (4 benchmarks)
Original baseline	None	84.6	64.4	62.8	56.0	67.0
After Stage 1: alignment module only	9M	84.6	64.4	62.8	56.0	67.0
After Stage 2: distillation	9M	88.2	72.5	70.9	65.7	74.3
After Stage 3: full GenRecal	6M	89.5	74.9	73.2	68.1	76.4
Separate SFT-only baseline	9M	86.5	65.2	65.0	59.9	69.2

Key Findings

• Hard-target supervision matters. In Table B, KL alone yields an 11-benchmark average of 59.8, below the original baseline's 68.1. Adding autoregressive supervision raises it to 77.1, a difference of 17.3 points. This supports explicit correct-answer prediction for alignment in this setting, not a claim that every distillation method needs the same loss recipe.
• Gains precede final SFT. Table C's Stage 2 average of 74.3 exceeds the SFT-only baseline's 69.2, and final SFT raises 74.3 to 76.4. The training procedures and compute budgets differ, so this does not establish a strictly compute-matched advantage.
• Shared-tokenizer pairs also benefit. With Qwen2-VL-72B teaching the 7B student in the paper's Table 4, GenRecal scores 64.2 on MMMU versus LLaVA-KD's 58.2. This differs from Table A's 65.6 obtained with InternVL2.5-78B as teacher.

Highlights & Insights

• A shared prediction interface replaces direct index matching. The teacher head remains unchanged while student context is recalibrated into a space it can interpret. Cross-vocabulary transfer becomes conditional generation learning rather than a search for a perfect token correspondence table.
• Stabilize the auxiliary module before updating the student. The teacher-only path anchors the Recalibrator, while Stage 1 prevents simultaneous student drift. Learning the bridge first and then adapting the source representation may inform other heterogeneous transfer tasks, but requires task-specific validation.
• Training complexity is separated from deployment complexity. The Recalibrator handles heterogeneity during training and is subsequently removed. The cost does not disappear: it is concentrated in training, making the approach relevant to centralized training followed by small-model deployment.

Limitations & Future Work

• Teacher internals must be accessible. The main text explicitly requires open-weight teacher parameters and hidden features. A closed-source API returning only text cannot directly support this method; SFT remains an alternative but is not equivalent to GenRecal.
• No extra inference modules does not mean cheap training. The paper uses 128 A100 80GB GPUs and millions of samples. Its time and FLOPs comparisons apply to the reported settings and do not directly establish single-GPU reproduction costs or measured mobile-device latency.
• Averages cannot replace per-task inspection. The student does not surpass the teacher on every task. Tables B and C also average different benchmark sets, so their absolute averages are not interchangeable.
• Reproduction details require verification. Some equations and Table 6 are corrupted in the text extraction, referenced appendices are absent, and final SFT data selection is described differently in two places. This note uses only clearly readable algorithms and tables, without guessing temperatures, weights, token shifts, or missing ablation values.
• Future directions. The authors propose extending the Recalibrator to intermediate layers and multiple teacher sources. Testing transfer stability across visual token strategies and training-domain shifts would also be useful; this is a suggestion from this note rather than a validated result.

Related Work & Insights

• vs LLaVA-KD / MiniLLM / DistiLLM: These comparison methods perform logit distillation with shared token types and use different KL objectives or training strategies. GenRecal additionally learns a representation transformation before the teacher head; shared-tokenizer comparisons test whether its benefits extend beyond resolving vocabulary mismatch.
• vs ULD / MOT: The paper describes these as cross-tokenizer logit-matching approaches based on Wasserstein distance or optimal transport. GenRecal instead aligns hidden features before computing supervision in the teacher vocabulary. The distinction is the transfer interface, not the absence of KL in GenRecal.
• vs answer-level SFT: SFT requires only text labels or teacher-generated answers and suits teachers with restricted access. GenRecal needs internal features but retains distributional teacher supervision and a representation-transfer path.

Rating

• Novelty: 4/5. The contribution lies in cross-tokenizer teacher-head distillation and a stable training recipe, rather than a new Transformer module.
• Experimental Thoroughness: 4/5. Multiple teacher-student pairs, loss ablations, and stage checkpoints are covered, but open-weight access and large training budgets limit generalization of the findings.
• Writing Quality: 3/5. The main information flow and algorithms are clear, although data-selection descriptions need clarification and text extraction loses some formatting information.
• Value: 4/5. The approach offers a concrete route to compressing heterogeneous VLM capabilities, especially when changing tokenizers solely for distillation is undesirable.