Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/lchen1019/DIPE
Area: Multimodal VLM
Keywords: Inter-modal distance invariance, RoPE, visual fading, long context, query anchoring

TL;DR¶

DIPE uses sequential queries for intra-modal attention and segment-anchored queries for cross-modal attention, preserving image geometry and text order while mitigating visual fading and raising the MRoPE aggregate benchmark score from 44.69 to 48.79 under 8K textual distractors.

Background & Motivation¶

A vision-language model (VLM) handling long documents, sustained question answering, or lengthy responses must repeatedly consult the original image rather than inspect it only at the beginning. Mainstream multimodal large language models (MLLMs) place image features and text tokens in one sequence, using multimodal RoPE, or MRoPE, to represent time, height, and width. All three text position components advance together, whereas image positions retain their grid coordinates, so later text queries become increasingly distant from the image in positional space. The paper calls the accompanying reduction in attention allocated to image tokens visual fading. This can cause a model to abandon specific visual evidence in favor of language priors, such as replacing an exact fee in a document with a typical conference price range.

This behavior cannot simply be attributed to difficult image content: the authors insert unrelated text between the image and question to separate the effect of cross-modal distance from reasoning over task-relevant text. Longer sequences do introduce more competitors into softmax, but the paper argues that the long-distance decay bias inherited from RoPE is another important cause. Nearby words are often more relevant in language, making that bias useful; repeatedly examining an image, however, gives no reason to treat it as progressively older evidence merely because more text has been produced. The problem is therefore not that positions should disappear, but that intra-modal structure and cross-modal evidence access need not share the same distance rule. The paper's discussion of distance decay is best understood as an inductive bias and an empirical trend, not a guarantee that attention between arbitrary queries and keys decreases strictly monotonically with distance.

DIPE preserves the text locality and image geometry already handled by MRoPE, changing only the position used by a query in cross-modal interactions. Queries within the same contiguous modality segment share its starting position when accessing another modality, while retaining their actual sequential positions when accessing the same modality. Generating more text therefore stops increasing that segment's positional distance to existing image keys, although different image patches still retain distinct key positions. Core Idea: make cross-modal query distance invariant to generation steps within a segment while preserving intra-modal order and spatial structure, rather than collapsing every token position to one constant.

Method¶

Overall Architecture¶

The input is an image-text sequence: the main experiments extract dynamic-resolution visual features with the NaFlex variant of SigLIP2-SO400M and project them into the language space through a two-layer MLP with GELU. The primary language backbone is Qwen2.5-3B; DIPE changes positional handling inside multimodal self-attention rather than adding an image retriever or a new visual encoder. The sequence is partitioned into contiguous modality segments, such as text, image, text, and another image, with sequential positions SPE and anchored positions APE generated for each segment. Here APE means Anchored Position Encoding, not absolute positional encoding, which uses the same abbreviation in the related-work discussion. Intra-modal sequence preservation produces sequential queries and keys, inter-modal query anchoring adds a second query view, and normalized fusion combines the two attention outputs. Both branches share keys and values and retain the task's causal visibility constraints; routing changes the query rotation used by visible token pairs rather than permitting access to future tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image-text sequence<br/>and modality segments"] --> Features["Visual projection and<br/>queries, keys, values"]
    Features --> Sequential["Intra-modal<br/>sequence preservation"]
    Sequential --> Anchored["Inter-modal<br/>query anchoring"]
    Anchored --> Fusion["Normalized fusion"]
    Fusion --> Output["Attention output and<br/>autoregressive answer"]

The serial steps first prepare the sequential view, then add the anchored view, and finally compute and merge the two outputs; the anchored view does not overwrite the sequential view. Training and inference use the same positional rules: inference caches SPE-rotated keys and values, while each new token produces only its current query's two views. The design adds no learnable parameters, but two masked attention calls and their fusion still incur runtime overhead.

Key Designs¶

1. Intra-modal sequence preservation: retain image geometry and language locality

For image-to-image and text-to-text attention, DIPE does not revise existing distance relationships, instead retaining the sequential positional encoding SPE supplied by MRoPE. Image-token height and width follow the patch grid, with a shared temporal component within a frame; text tokens use the same sequential index across time, height, and width. Text can therefore still distinguish word order, nearby context, and distant context, while visual tokens retain spatial neighborhoods through positional rotation. Queries and keys are both rotated with their own SPE coordinates before intra-modal attention, without new position compression or additional supervision in this branch. This preservation matters: assigning all image and text tokens a single anchor would stop distance growth but also erase necessary structural information.

Position allocation advances segment by segment, but each segment internally follows the original MRoPE rule. After processing a segment, Algorithm 1 sets the next offset to that segment's maximum SPE position plus one, preserving the order of segments in the full sequence. Other RoPE variants can replace this position allocator; DIPE does not require redesigning their rotation frequencies. DIPE is thus complementary to MRoPE-I: the latter adjusts frequency allocation, whereas the former determines which query position a modality relationship should use. Retaining the intra-modal mechanism motivates broadly comparable short-context performance, but structural preservation alone cannot guarantee no regressions on individual datasets.

2. Inter-modal query anchoring: fix the segment's query origin without moving keys

When a query and key have different modalities, DIPE replaces the query's three-dimensional position with the SPE tuple of the first token in its modality segment. Algorithm 1 broadcasts that tuple over the segment to create APE; subsequent queries in this branch no longer use their own advancing positions. Keys always retain their SPE coordinates, so different image patches still have different relative positions rather than collapsing into one equidistant point. Let \(R(p)\) denote MRoPE rotation at three-dimensional position \(p\), and let \(a(s)\) denote the first-token position of segment \(s\); the mechanism can be written as:

\[ S_{mn}=\begin{cases} \bigl(R(p_m^{\mathrm{spe}})q_m\bigr)^\top R(p_n^{\mathrm{spe}})k_n, & \mathrm{mod}(m)=\mathrm{mod}(n),\\ \bigl(R(a(s_m))q_m\bigr)^\top R(p_n^{\mathrm{spe}})k_n, & \mathrm{mod}(m)\ne\mathrm{mod}(n). \end{cases} \]

This scoring expression is organized from Section 4.1 and Algorithms 1 and 2, omitting ordinary attention scaling and masking; Equations 4 and 5 are damaged in the text extraction, so this is not a verbatim transcription. Here \(m,n\) index tokens, \(s_m\) is the query's modality segment, and \(q_m,k_n\) are content vectors before rotation. Only the positional difference between the anchor and a fixed key is invariant; query content still changes, and attention is normalized jointly with other visible keys. DIPE therefore guarantees neither identical cross-modal scores nor completely constant total visual attention. It removes the factor whereby continuing generation within one segment necessarily increases cross-modal positional distance, while dilution from additional softmax competitors remains.

The same mechanism handles interleaved image-text inputs by establishing a new anchor for each new modality segment, rather than reusing the first position of the whole sample. This retains coarse ordering between segments and makes invariance a within-segment property, not a statement that arbitrary cross-segment or cross-image token pairs share one distance. The paper applies the rule in both cross-modal directions; whether a particular attention edge exists still depends on the causal mask and input layout. Section 5.3 supports compatibility through interleaved-context tests and multi-image BLINK results, without proving that every aspect of complex multi-image temporal reasoning is unaffected.

3. Normalized fusion: two computations still implement one joint softmax

A query now needs two rotated views: its sequential view accesses same-modality keys, and its anchored view accesses cross-modal keys. Algorithm 2 calls FlexAttention with two modality masks; both branches read the same SPE keys and values and return their respective outputs and LogSumExp statistics. Simply averaging the outputs would be incorrect because each is normalized over a different key set, with different set sizes and score magnitudes. Instead, each branch must contribute according to its share of the joint softmax denominator, rather than forcing vision and text to receive equal weight. With \(\ell_{\mathrm{intra}}\) and \(\ell_{\mathrm{inter}}\) denoting the LogSumExp of each branch's masked attention logits, fusion becomes:

\[ \alpha=\sigma(\ell_{\mathrm{intra}}-\ell_{\mathrm{inter}}),\qquad O=\alpha O_{\mathrm{intra}}+(1-\alpha)O_{\mathrm{inter}}. \]

The explicit computation in Algorithm 2 on page 8 verifies this expression and clarifies Equation 6, whose extracted rendering is corrupted. Because \(e^{\ell}\) is a branch's exponential sum, this weighting restores joint normalization across the two disjoint visible-key sets. It is a normalization identity, not an additional learned gate, and no separate visual-weight predictor is trained. The implementation retains the existing key-value cache layout, avoiding re-rotation of past keys or rearrangement of the KV cache as generation grows. Cache compatibility nevertheless means that infrastructure can be reused, not that the paper has established the effectiveness of switching arbitrary existing checkpoints to DIPE only at inference time.

A Worked Example¶

Figure 2 illustrates an image with temporal component 0 and height and width coordinates from 0 to 16, followed by text whose three SPE components start at 17 and increase. A text query at sequential position 23 still uses position 23 when reading earlier text, retaining its proximity to position 22. When reading an image patch, however, its query position becomes the segment origin \((17,17,17)\), while the image key position remains unchanged. Later generated text in the same segment uses this same cross-modal anchor, so advancing its own sequential position does not push that patch farther away. For example, the cross-modal positional difference to an image key at \((0,16,16)\) does not change with generation steps within this text segment. Both branches still compute attention using content-dependent queries and their respective masks before LogSumExp-weighted fusion; this does not reuse a cached, fixed answer about the image. If another image and a new text segment follow, the new text segment receives its own origin rather than continuing to use 17. These numbers illustrate the positional rule in Figure 2 and are not additional experimental results.

Loss & Training¶

Section 5.1 uses two-stage vision-language training, first freezing the vision encoder and language model to initialize only the visual projector. The second stage performs instruction tuning and unfreezes the language model, with LLaVA-Pretrain-558K and LLaVA-NeXT-779K used for the respective stages. DIPE itself introduces no new loss, learnable positional parameters, or additional annotations; it changes the attention scoring rule used during training and generation. Backbone extensions also cover Qwen2.5-0.5B and Qwen3-1.7B, so the main experiment should not be described as a training-free upgrade to the existing Qwen2.5-VL product model. The main text points to Appendix 1.1 for detailed training settings such as learning rates, but the supplied cache ends with the references and does not contain that appendix, so specific hyperparameters are not supplied here.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1 on page 10, using Qwen2.5-3B and inserting 8K unrelated text tokens between each image and question. This Long-Context VQA stress test diagnoses visual fading rather than directly measuring integration of relevant information in real documents. Values retain the paper's percentage-scale benchmark reporting; the aggregate is the authors' cross-task average, not one uniform accuracy metric. The main text does not spell out the underlying scoring details for every task, so benchmark score is retained rather than relabeling DocVQA as exact-match accuracy.

Benchmark / aggregate	MRoPE	MRoPE + DIPE	Absolute score difference
HRBench-4K	47.63	54.50	+6.87
POPE	74.50	85.58	+11.08
CountBench	66.60	75.97	+9.37
DocVQA val	33.25	38.55	+5.30
InfoVQA val	23.60	21.84	-1.76
MMBench V1.1-EN dev	50.00	58.75	+8.75
All 19 evaluations	44.69	48.79	+4.10

The +4.10 is an absolute score increase, not a 4.10% relative improvement over 44.69; for accuracy measures, the corresponding unit is percentage points. The same table reports aggregate changes of 46.31 → 48.31 for Vanilla RoPE and 46.50 → 48.51 for MRoPE-I, showing that benefits are not exclusive to the original MRoPE. Not every combination improves: V* with MRoPE-I falls from 50.79 to 47.64, so overall effectiveness does not mean improvement on every task.

Ablation Study¶

The main text does not provide a complete component-ablation matrix individually removing SPE, APE, and fusion; this section instead uses the efficiency analysis in Table 2 on page 13 to expose computational costs. All entries below use unpacked single-sequence evaluation in milliseconds, retaining the original means and ± values; the reviewed main text does not define the precise statistic represented by ±. Full-model and individual attention-block measurements are different scopes, and forward latency must not be confused with single-token decoding latency.

Measurement scope and operation	Context length	MRoPE (ms)	MRoPE + DIPE (ms)
Full-model forward	8K	140.1 ± 0.4	144.6 ± 0.5
Full-model forward	32K	818.0 ± 1.6	839.8 ± 3.7
Full-model backward	8K	293.0 ± 0.8	322.0 ± 1.0
Full-model single-token decoding	32K	18.3 ± 0.1	19.6 ± 0.1
Attention-block forward	32K	13.56 ± 0.01	14.31 ± 0.05
Attention-block single-token decoding	32K	0.30 ± 0.01	0.38 ± 0.01

Table 2 marks full-model backward execution as OOM at both 16K and 32K for both methods, so measurable block-level performance does not establish feasible full-model training at that length. As an architectural generalization analysis, Table 3 on page 14 reports Qwen3-1.7B aggregate scores of 40.21 → 44.46, an absolute gain of 4.25 points. This supports effectiveness across backbones but does not strictly separate the contributions of positional anchoring and training adaptation.

Key Findings¶

Figure 3 on page 11 varies distractor length from 0K–32K across 9 benchmarks, with DIPE generally showing slower degradation; the extracted curves do not support precise pointwise readings, so none are invented.
Figure 4 and the accompanying text on page 12 report short-context MRoPE-I aggregate scores of 51.5 → 51.4, supporting approximate parity rather than literally zero performance loss.
Figures 7 and 8 show restored shallow-layer visual attention and reduced decay in the all-layer average; the authors also acknowledge that natural attention dilution from longer text remains.
Figure 5 adds interleaved long-context evidence through MM-NIAH image reasoning, but both methods degrade sharply at 64K, so DIPE does not remove the language backbone's context-capacity limit.

Highlights & Insights¶

Choose positions by interaction type, not only by token identity. A text query can retain its true linguistic position while using a stable anchor to read an image, targeting the problem more precisely than uniformly shortening all sequence distances.
Changing only queries is an important engineering choice. Old keys and values need not move during generation, preserving cache reuse while concentrating overhead in current dual-view queries and split attention.
Fusion preserves competition rather than forcing visual access. LogSumExp imposes no fixed visual quota, retaining content-based scoring and explaining why long-text dilution can still affect DIPE.

Limitations & Future Work¶

Author-acknowledged boundaries: visual attention still declines somewhat, extreme contexts remain constrained by backbone capacity, and Table 1 contains regressions; invariance in the title does not mean constant performance.
Evaluation boundaries: 8K unrelated distractors provide a clear diagnostic but change both positional distance and the number of competing tokens, so this setup alone cannot fully separate their causal contributions.
Reader proposal: hold text content and length fixed while changing only position indices, then compare inference-only replacement against joint training to disentangle the encoding mechanism from training adaptation.
Reproducibility boundaries: the supplied cache omits the referenced appendices, leaving training hyperparameters, interleaved-test details, and latency hardware conditions incompletely verified; several equations are corrupted, so only algorithm-verifiable mechanisms are restated rather than reconstructing the decay bound.
Citation ambiguity: the main text labels MM-NIAH with reference [22], but entry [22] lists the generic Needle-in-a-Haystack repository; this note retains the authors' evaluation name without guessing a separate MM-NIAH link or dataset version.

vs MRoPE / MRoPE-I: the former preserves three-dimensional geometry and the latter improves frequency allocation, whereas DIPE changes how cross-modal queries use positions; the problems are related but distinct, and Table 1 directly evaluates combinations.
vs V2PE / Circle-RoPE: as described in the paper's related work, these adjust visual position scale or geometric layout; DIPE is distinguished by switching a query's encoding according to key modality rather than producing only one improved static positional sequence.
vs explicit cross-modal attention architectures: DIPE adds neither a Q-Former nor a separate fusion network, instead grouping computations inside existing self-attention with modality masks; this grouping should not be mistaken for an additional visual feature extractor.
Transferable direction: tasks that repeatedly revisit fixed perceptual inputs could separate when an input appeared from how distant it should be during current access; extension to audio or video is a research suggestion, not an empirical result of this paper.

Rating¶

Novelty: 4/5. Switching query positions by modality relationship is targeted and compact, while building on existing RoPE and attention normalization.
Experimental Thoroughness: 4/5. The study covers 19 evaluations, multiple backbones, and efficiency, but lacks complete component ablations and a strict separation of positional distance from softmax dilution.
Writing Quality: 4/5. Illustrations and pseudocode clarify the mechanism, while claims such as distance invariance and strict parity require reading alongside their boundaries and counterexamples.
Value: 4/5. The method offers a composable positional change for long-context visual grounding, with adoption still requiring attention to inference overhead, task regressions, and training adaptation.