From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion¶

Conference: ECCV2026
Paper: ECCV
Code: https://github.com/codefuse-ai/CLI
Area: Multimodal VLM
Keywords: vision-language model, cross-layer injection, hierarchical visual features, gated fusion, parameter-efficient fine-tuning

TL;DR¶

Two parameter-efficient modules — Adaptive Multi-Projection (per-layer LoRA projections) and Adaptive Gating Fusion (a gate computed from the decoding context) — turn the single-point "last vision layer to first LLM layer" connection of a VLM into a dynamic many-to-many bridge between multiple vision layers and multiple decoder layers, yielding consistent gains across 28 benchmarks on LLaVA-OneVision and LLaVA-1.5 (LLaVA-OV-7B: +9.7 overall, LLaVA-W +6.5, OCR +4.7, MME +3.3).

Background & Motivation¶

Today's dominant vision-language models follow the "frozen twin towers plus a lightweight bridge" recipe established by Flamingo and BLIP-2 and popularized by LLaVA: the vision encoder and the LLM keep their pretrained weights, and only a projector in between is trained, mapping the token output of the ViT's final layer into the text embedding space and prepending it to the text tokens for autoregressive decoding. The trouble is that both the ViT and the LLM refine their representations layer by layer, yet the connection between them is starkly asymmetric: the top of the vision hierarchy feeds the bottom of the language hierarchy, and the entire intermediate path is collapsed into one projector. The edges, textures, strokes, and spatial structure preserved in the ViT's early layers are discarded wholesale at this step, so the LLM has no channel through which to "zoom in" — it only ever sees a single image already compressed into a semantic summary. The paper illustrates the consequence with an ice-skate photo: a baseline that relies solely on final-layer features calls it a "roller skate," because it cannot see the wheels and therefore cannot judge whether the shoe is fit for skating.

A closer look shows that even "shallow-to-shallow, deep-to-deep" parallel alignment would not be enough. The paper identifies two dependencies that point in opposite directions: a shallow LLM layer parsing a noun (say, "shoe") needs the high-level semantic concept already formed in the deep ViT layers in order to ground that word on the right object; meanwhile a deep LLM layer performing the final judgement ("is this shoe appropriate?") must instead go back to the fine-grained structure in the early ViT layers to examine the wheels. What is needed, then, is a criss-crossed set of connections that also shifts as decoding proceeds. Prior attempts are all incomplete. DeepStack is a brute-force one-to-many broadcast: it partitions high-resolution visual tokens into groups and adds each group into fixed LLM layers element-wise, entirely blind to context. CogVLM's Visual Expert inserts a module into every LLM layer, yet all of them consume the same final-layer feature map. CogAgent statically broadcasts high-resolution features to every decoder layer, so every layer sees the same fixed level of visual information. EVLM, mPLUG-Owl3, Qwen3-VL, and the concurrent DEHVF are all hard-wired one-to-one schemes — specific ViT layers pinned to pre-configured LLM layers. However refined, each of these is a manually configured static straitjacket that predetermines how information must flow.

This paper's angle is to change the role assignment: instead of a human designing the connections, make the LLM an active observer that queries the entire visual hierarchy at each injection point, according to its own decoding context, and learns which layer's visual evidence this reasoning step requires. Core idea: sample several intermediate layers of the vision encoder into a "visual repository," align every layer's features to the text embedding space with per-layer LoRA projections, and use an attention gate driven by the current hidden state to write visual information back into the hidden state on demand and in the right amount at multiple decoder layers — turning one-to-one into a learnable many-to-many.

Method¶

Overall Architecture¶

CLI still takes an image plus a text instruction as input and still produces autoregressive decoding from the LLM; what changes is the route by which visual information enters the language model. Instead of taking only the last ViT layer, the image's token matrices are sampled from \(L_V\) layers at different depths, forming a hierarchical feature set \(\mathcal{V}=\{V_k\}_{k=1}^{L_V}\) (the paper gives layers 1, 7 and 14 as an example); this set acts as the "visual repository." Injection points are then placed at several layers of the LLM decoder, and each point receives the entire set rather than one layer: AMP first applies a layer-specific low-rank correction to every \(V_k\), aligning all levels to the text embedding dimension; AGF then uses the current hidden state as a query to compute how much each hierarchical feature should be admitted at that injection point, and writes the weighted visual information back into the hidden state at the positions of the visual tokens. The whole modification only adds LoRA matrices and gating modules without touching the vision encoder or the LLM backbone, so it plugs into different VLMs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["image + text instruction"] --> B["Multi-level sampling &<br/>multi-point injection"]
    B --> C["Adaptive Multi-Projection<br/>per-layer LoRA alignment"]
    C --> D["Adaptive Gating Fusion<br/>context-driven gated fusion"]
    D -->|repeated per injection layer| E["LLM autoregressive decoding"]

Key Designs¶

1. Multi-level sampling and multi-point injection: replacing a single connection with a "visual repository plus many query ports"

It is worth being precise about the two topologies, because "one-to-one" and "many-to-many" in the title refer to the correspondence between layers, not to the number of modules. One-to-one comes in two typical forms. The first is the LLaVA style, where the whole vision hierarchy is represented by the final layer alone: projected once, prepended to the text tokens, one single point of contact between vision and language. The second is the hard-wired style of EVLM, Qwen3-VL and DEHVF, where vision layer \(i\) is fixed to language layer \(i\) (or to a fixed group), a pairing written by hand. One-to-many is the DeepStack style, broadcasting one visual feature into many LLM layers. Both families share the same weakness: which layer's information goes where is predetermined by the designer and never varies with the input or the reasoning process.

CLI opens up both ends of the topology at once. On the vision side, token matrices are sampled from \(L_V\) layers at different depths, so the repository holds both shallow structural detail (crucial for OCR and localization) and deep semantic concepts (crucial for open-ended reasoning). On the language side, injection points are placed at multiple decoder layers, and each of them can access the complete hierarchical set. Which vision layer should serve which language layer is therefore no longer assigned by a human but left as a matching problem for the data to solve. The paper stresses that even the seemingly more sensible "parallel alignment" is insufficient: the ice-skate example shows a shallow LLM layer sometimes needing deep ViT semantics, and a deep LLM layer sometimes needing shallow ViT detail, so crossing must be allowed. The heatmap of learned gate weights (Fig. 1(b) in the paper) is the empirical evidence for exactly this: deep decoder layers query almost uniformly across all vision layers from shallow to deep, and the learned connections are non-parallel and criss-crossed.

2. Adaptive Multi-Projection: resolving the dilemma of "one projector cannot fit many distributions" with per-layer LoRA

Aligning multi-level visual features to the language space runs into an unavoidable dilemma. Reusing the single shared projector of LLaVA cannot reconcile the large distributional variance across vision layers — from low-level texture to high-level semantics — and causes severe feature misalignment. Yet the intuitive alternative of training a dedicated projector per layer is computationally prohibitive because each would have to be pretrained from scratch. The paper's solution makes the projector adaptive without retraining it: it keeps the pretrained MLP projector \(P\) and attaches a dedicated low-rank branch for each sampled vision layer \(k\), so the output is the original projection plus that layer's low-rank correction:

\[\hat{V}_k = P(V_k) + B_k A_k V_k,\qquad V_k \in \mathcal{V}\]

where \(A_k\) and \(B_k\) are low-rank matrices trained specifically for layer \(k\). Each layer thus gets a small set of parameters to adapt to its own feature distribution, while the shared \(P\) continues to supply the generic alignment learned in pretraining, at a cost of only a few percent more parameters (AMP reaches 104.37% of the baseline's parameters in the ablation, versus 110.07% for a fully fine-tuned projector plus AGF). Applying this to every \(V_k\) in the set yields a collection of hierarchical token maps \(\hat{\mathcal{V}}\) already living in the text embedding dimension, ready to be consumed by the gate. The point is not just parameter savings: it makes the division of labour between AMP and AGF explicit — first bring the layers onto a comparable scale, then let the gate choose. In the ablation the two together beat either alone, confirming that a cleaner input makes the gate more effective.

3. Adaptive Gating Fusion: letting the LLM decide what to inject and how much

After alignment, direct addition is not an option. The hidden state \(h_t\) already carries contextual information, and crudely stacking external features on top corrupts representations the LLM has already learned — DeepStack's results (a 50.7-point drop at 0.5B and 39.1 at 7B) are the price of such unfiltered injection. AGF therefore casts injection as a gated selective update, with the switch decided by the current decoding state rather than being a fixed weight. Concretely, each injection layer introduces two learnable query vectors as probes: \(q_v\) distills the essence of the hierarchical visual features and \(q_h\) distills that of the current hidden state, both through multi-head attention:

\[att_v=\text{MHA}(q_v,\hat{V}_k,\hat{V}_k),\qquad att_h=\text{MHA}(q_h,h_t,h_t)\]

The two context vectors are concatenated, passed through a linear layer and a Sigmoid to produce a gate weight \(W\in[0,1]\). A binary mask then marks which positions of \(h_t\) are visual tokens, so that non-visual positions are preserved while visual positions receive a weighted update (the formula in the source PDF is corrupted on extraction; only the mechanism is kept here, ⚠️ refer to the original paper for the exact form):

\[h'_t = h_t \odot (1-\text{mask}) + \text{mask} \odot (W \odot \hat{V}_k)\]

The key is that \(W\) is jointly determined by "how far decoding has progressed" and "what the current layer is computing": for OCR-like questions the gate can amplify high-frequency stroke information from shallow layers, for open-ended reasoning it can turn to deep semantic concepts, and when a level's low-level features are unhelpful for the task at hand it can suppress them. This is precisely why adding AGF alone lifts the total from 366.72 to 370.60 (+3.88) while adding AMP alone yields only 367.89 (+1.17) — being able to select dynamically matters more than being able to see. Gating and writing back repeat across multiple injection points, so the model "re-examines" visual evidence at varying granularities during generation, forming an iterative refinement of its visual understanding.

Loss & Training¶

The method introduces no new loss term and follows the original training protocols of LLaVA-OneVision and LLaVA-1.5 exactly, training only the new modules (the per-layer LoRA projection matrices and the gating modules). The two configurations deliberately differ in backbone and initialization so that the conclusion is credible. The LLaVA-OneVision variant uses a Qwen-2 series LLM, a Siglip-so400m-patch14-384 vision encoder and a two-layer MLP projector, and starts fine-tuning from the public checkpoint released after its "High-Quality Knowledge Learning" stage to avoid data contamination and keep the comparison fair. The LLaVA-1.5 variant uses Vicuna-7B as the LLM, CLIP-Large as the vision encoder and a two-layer MLP projector with GELU activation, starting from the pretrained checkpoint and using exactly the same training data as LLaVA-1.5. Detailed hyper-parameters are in Appendix A; the instruction tuning set consists of the single-image datasets of LLaVA-OneVision, and all evaluation runs through the LMMs-Eval framework.

Key Experimental Results¶

Main Results¶

On LLaVA-OneVision at 0.5B and 7B, a single-layer projection baseline retrained on the same data (Baseline Projector) is compared against DeepStack and Shallow-Layer Injection (SLI, which maps n vision layers sampled at a uniform stride of 8 one-to-one onto the first n decoder layers) across nine benchmarks, with larger models listed for reference.

Model	AI2D	ChartQA	DocVQA	InfoVQA	LLaVA-W	OK-VQA	GQA	Total (9)
VILA-13B	57.6	32.8	20.0	10.0	58.6	1.6	17.6	240.5
IXC-2.5-7B	39.1	81.2	90.3	68.1	63.2	29.2	57.7	574.8
InternVL-2-8B	82.2	82.5	90.0	66.5	72.7	52.1	62.7	660.9
InternVL-2-26B	83.0	84.4	90.4	68.9	90.6	48.3	65.1	686.7
LLaVA-OV-0.5B	56.5	64.5	64.1	47.5	61.7	48.4	53.7	541.2
+ DeepStack	53.2	54.0	54.6	41.0	57.1	46.2	53.3	490.5 (−50.7)
+ SLI	57.2	64.7	63.6	46.6	55.2	48.8	54.5	534.7 (−6.5)
+ CLI	56.7	65.2	64.7	47.1	61.1	48.6	53.7	542.9 (+1.7)
LLaVA-OV-7B	77.5	78.5	82.5	69.5	68.0	58.5	59.4	650.9
+ DeepStack	76.0	63.8	72.6	62.2	70.3	58.3	58.5	611.8 (−39.1)
+ SLI	77.6	77.3	79.9	67.6	68.5	58.4	59.3	645.8 (−5.1)
+ CLI	77.9	78.7	82.8	70.5	74.5	59.2	59.7	660.6 (+9.7)

On the second suite of nine benchmarks covering perception and multidisciplinary reasoning (MathVerse, MathVista, MMBench, MME, MMStar, MMMU, MMVet, SeedBench, ScienceQA) the same ordering repeats: at 0.5B, Baseline 391.0, DeepStack 380.9, SLI 393.7, CLI 395.6; at 7B, Baseline 556.2, DeepStack 539.3, SLI 551.1, CLI 559.4, driven mainly by MME 88.4 (+3.3), MMStar 56.5 (+2.4) and MMMU 47.6 (+0.6). With CLI, LLaVA-OV-7B reaches a total of 660.6, closing in on the much larger InternVL-2-8B (660.9).

Architecture-agnostic behaviour is verified on LLaVA-1.5-7B (different LLM, vision encoder and pretraining pipeline):

Model	Document understanding, 4-item partial sum	Reasoning / perception, 5-item partial sum
LLaVA-1.5-7B	468.0	433.8
+ CLI	475.5 (+7.5)	442.4 (+8.6)

Gains are visible on OK-VQA +5.2, LLaVA-W +3.0, MMBench +2.0 and MMVet +3.3, indicating that CLI does not merely add polish but can compensate for pre-existing weaknesses in a base model's vision-language alignment.

Ablation Study¶

Component ablations run on LLaVA-OV-0.5B with 50% of the instruction tuning data (a configuration justified by the scalability analysis in Appendix E.1), so the absolute scores sit below the same-scale results in the main table and are only comparable across rows:

Config	Total (9)	Params	Note
Baseline	366.72	100.00%	single-layer projection baseline
w/ AMP	367.89	102.25%	multi-level projection alone, almost no gain
w/ AGF	370.60	102.12%	gating alone, +3.88, the main source of gain
w/ AMP + AGF	371.51	104.37%	the two synergize, a further +0.91
w/ Full AMP	367.74	107.95%	fully fine-tuned projector still yields nothing
w/ Full AMP + AGF	407.01	110.07%	higher ceiling, but a costly parameter budget

Computational cost per sample on MMBench, for both training and inference:

Method	Inference Time	Inference Memory	Inference FLOPs	Training Memory	Training FLOPs
Baseline	0.66 s	241.2 MB	3.7 T	251.3 MB	5.46 T
CLI	0.77 s	244.4 MB	5.0 T	277.6 MB	7.42 T

Key Findings¶

The gate is the main engine of gain, and alignment is its prerequisite. AMP alone adds 1.17 points, AGF alone adds 3.88, and the two together add 4.79; a fully fine-tuned projector used alone again yields nothing (367.74). The numbers make the division of labour clear: AMP is not the source of gain, it brings the layers onto a comparable scale so the gate can discriminate effectively.
Unfiltered injection hurts, and it hurts badly. DeepStack loses 50.7 points at 0.5B and 39.1 at 7B, and SLI loses 6.5 and 5.1 respectively; across the 18 tasks in the two nine-benchmark suites, CLI is the only deep-fusion scheme that consistently beats the baseline. This directly refutes the intuition that more visual information can never be worse.
The gain comes from architecture, not parameter count. The authors build a Parallel control model with a comparable parameter count that implements static one-to-one injection (vision layer i to decoder layer i). It loses 6.5 points on LLaVA-Wild and shows no significant gain on MME or MMStar, a gap of more than 13 points against CLI's +6.5 on the same benchmark — ruling out "just more parameters" as an explanation.
Injection density shows a counter-intuitive non-monotonic pattern. Single-point injection is consistently worst, confirming that the information bottleneck is real; a high-density strategy (frequent injection at many layers) is best overall, matching the claim of on-demand access to the full hierarchy; but a medium density performs worse than a sparser one. The authors attribute this to the cognitive overhead of intermittent updates — the LLM is interrupted repeatedly without the near-continuous context that high density provides.
Gains are distributed in a clear task-dependent pattern. Open-ended reasoning and fine-grained perception benefit most (LLaVA-W +6.5, OCR +4.7), multidisciplinary understanding next (MME +3.3, MMStar +2.4, MMMU +0.7), and document understanding shows small cumulative gains of +1.1 at 0.5B and +1.9 at 7B. Only a few tasks dip slightly (MathVerse −1.7, ScienceQA −0.8), which the authors attribute to the gate suppressing interference when low-level features are harmful. The aggregate gain over all 28 benchmarks is +19.06.
Transfer to LLaVA-1.5 yields even larger gains (+7.5 and +8.6 on the two partial sums, versus +9.7 and +3.2 on LLaVA-OV), suggesting that the weaker a base model's original vision-language alignment is, the more room multi-level gated injection has to compensate.
Cost is manageable but not free. Inference memory grows by just 1.3% (241.2 to 244.4 MB) and inference time from 0.66 to 0.77 s, but inference FLOPs rise from 3.7T to 5.0T and training FLOPs from 5.46T to 7.42T — the compute increase deserves more attention than the memory increase.

Highlights & Insights¶

The design object shifts from "the projector" to "the connection topology." Nearly all recent progress on the vision-language bridge has been about designing a better projection module; this paper restates the problem as "who queries whom, and when," which demotes the projector to an alignment tool and hands the real decision back to the LLM. That reframing explains why it gains more than module-stacking alternatives.
Per-layer LoRA is an elegant way out of the hierarchical-projection dilemma. It sidesteps both the distribution mismatch of a shared projector and the pretraining cost of per-layer full projectors, buying the lower bound of the fully fine-tuned solution for 4.37% extra parameters. The paper is honest about the ceiling too: 371.51 versus 407.01, where the full variant scores higher but is not worth the budget.
The gate weights double as an interpretability tool. Plotting the learned gate weights as a heatmap gives direct criss-cross evidence that deep decoder layers query the whole visual hierarchy. This is one of the few works that closes the loop from design motivation to mechanism to visual evidence, making "why many-to-many is needed" a testable claim rather than a motivational assertion.
The transfer surface is broad. Any "hierarchical encoder into hierarchical consumer" architecture can swap a single-point projection for AMP plus AGF: video encoders into LLMs, audio encoders into LLMs, or other modality front ends. Video in particular has richer temporal structure across layers, where the split between shallow detail-seeking and deep semantics-seeking may be even more pronounced.

Limitations & Future Work¶

Mathematical and symbolic reasoning does not improve. MathVerse drops 1.7 points and MathVista gains are limited, indicating that multi-level visual information mainly helps tasks that need visual evidence rather than abstract symbol manipulation; the paper offers no deeper analysis of this.
The ablation coverage is narrow. Component and injection-density experiments all run on the 0.5B model with 50% of the data, and there is no systematic search over which layers receive injection points, how many layers \(L_V\) to sample, or how to pick them (the paper only cites layers 1/7/14). Reproducing this in practice requires re-tuning.
Compute scales with the number of visual tokens. Every injection layer recomputes the gate over the full hierarchical set, increasing FLOPs by roughly 35%. With high resolution or long video the visual token count grows sharply and this overhead grows with it, yet efficiency is only reported on single-image MMBench with no cost curve under larger token budgets.
The gate operates at a fairly coarse granularity. \(W\in[0,1]\) is computed independently per visual level, and whether levels are correlated — or whether the gate should be aware of how much other levels were selected — is not discussed.
Some results exist only as figures. The OCR and RefCOCO/+/g results are given only as bar charts and delta plots (Fig. 4, Fig. 5), without standalone numeric tables, so strict reproduction depends on the appendix. There is also no comparison with models such as Qwen3-VL that push one-to-one fusion to its limit under an equal training budget, so cross-paper comparisons should be made with care.

vs LLaVA-style single-layer projection: they project the final ViT layer once and prepend it to the text tokens, giving one single point of contact; this paper samples multiple depths and injects at multiple decoder layers, each with access to the full hierarchy, turning visual information from a one-off summary into a repository that can be queried repeatedly.
vs DeepStack: it is a brute-force one-to-many broadcast that partitions high-resolution tokens and adds them element-wise into fixed layers, blind to context and losing points almost across the board; this paper fuses selectively through AGF and is the only deep-fusion scheme in the comparison to consistently beat the baseline on all 18 tasks.
vs CogVLM / CogAgent: Visual Expert inserts a module at every layer but all of them consume the same final-layer feature map, and CogAgent statically broadcasts high-resolution features to every layer; both give all decoder layers the same level of visual information, whereas this paper lets different layers see different levels on demand.
vs EVLM / mPLUG-Owl3 / Qwen3-VL / DEHVF: all are hard-wired, static one-to-one schemes pinning specific ViT layers to pre-configured LLM layers (DEHVF even uses a rigid group mapping). This paper leaves the alignment to be learned by the LLM and does not restrict an injection point to a single layer; the two differ fundamentally over who decides the pairing.
vs FUSION: it deepens fusion with learnable tokens that recursively interact with textual and visual features, but still operates on a single level of visual representation and never addresses level selection.

Rating¶

Novelty: ⭐⭐⭐⭐ Reframes the widely acknowledged underuse of visual hierarchy as a connection-topology problem rather than a module problem; the many-to-many plus context gating combination is the first to be systematically proposed and visually validated, though LoRA projection and attention gating are themselves mature components, so the novelty lies in the combination and the framing.
Experimental Thoroughness: ⭐⭐⭐⭐ 28 benchmarks, two VLM architectures, an equal-parameter control and training/inference efficiency analysis, with the DeepStack and SLI controls particularly convincing; marked down because ablations run only on 0.5B with half the data, and the choice of injection sites and sampled layers lacks systematic study.
Writing Quality: ⭐⭐⭐⭐ The motivation chain is clear (opening with the ice-skate failure case, closing with the gate heatmap) and negative controls are reported honestly; but several formulas are corrupted by the PDF layout, hindering reproduction, and the LLaVA-1.5 conclusions rest on coarse partial-sum aggregates.
Value: ⭐⭐⭐⭐ Plug-and-play and parameter-efficient (+4.37% parameters, +1.3% inference memory), directly applicable to other VLMs; it gives a reproducible, empirically supported answer to how a vision encoder's hierarchical information should enter the LLM.