I³CL: Restoring Interactive Capabilities to Implicit In-Context Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/LzVv123456/I2CL
Area: LLM Reasoning
Keywords: In-Context Learning, Task Vector, Implicit ICL, Interactive Attention, Dynamic Routing
TL;DR¶
This paper proposes Interactive Implicit In-Context Learning (I³CL), an elegant framework that restores inter-demonstration self-attention interactions and query-aware cross-attention routing to implicit task-vector representations, eliminating the representation bottleneck of isolated encoding and uniform average pooling while achieving a +3.54% average accuracy gain over existing task-vector baselines across eight classification benchmarks, outperforming LoRA fine-tuning and delivering near zero-shot inference efficiency.
Background & Motivation¶
In-context learning (ICL) has become an essential paradigm empowering large language models (LLMs) to perform novel reasoning tasks given only a handful of demonstrative input-label examples prepended to the context prompt. Despite its versatility without parameter fine-tuning, conventional ICL suffers from acute practical vulnerabilities: its predictions fluctuate wildly under varying demonstration formats and ordering choices, and its self-attention computational overhead and memory footprint grow quadratically (\(\mathcal{O}(N^2)\)) as demonstration sequences expand. These critical drawbacks severely impede high-throughput and long-context enterprise deployment. In response, task-vector-based methods—exemplified by In-Context Vectors (ICV) and Implicit In-Context Learning (I²CL)—have emerged as a compelling alternative. By condensing demonstrations into compact latent task vectors and injecting them directly into the residual activation streams of frozen LLMs, these methods attain order invariance and achieve inference throughput approaching that of zero-shot prompting.
Nevertheless, existing task-vector methodologies achieve fast inference only by making substantial compromises in representation expressiveness, often lagging considerably behind standard few-shot prompting and Parameter-Efficient Fine-Tuning (PEFT). A fundamental investigation reveals that this performance degradation stems from the severe sacrifice of two core interactive mechanisms intrinsic to standard ICL: first, prior works encode each in-context demonstration completely in isolation, thereby discarding the rich inter-demonstration information flow that allows self-attention to summarize collective regularities across diverse examples; second, to collapse individual representations into a single task vector, existing methods universally rely on static, uniform average pooling across all demonstration vectors. This naive averaging treats every example as uniformly informative regardless of the specific test query, entirely forfeiting standard ICL's query-dependent attention mechanism that adaptively emphasizes the most relevant demonstrations during reasoning; furthermore, relying solely on the Label Last Token (LLT) induces positional bias in causal autoregressive decoders, starving representations of crucial task instructions and input semantics.
To resolve this fundamental tension, the authors challenge the prevailing assumption that task-vector methods must inherently trade model expressiveness for inference speed. Instead, they propose Interactive Implicit In-Context Learning (I³CL), a framework that systematically restores these indispensable interactive capabilities within the latent task-vector space while preserving the efficiency benefits of implicit injection. By integrating a dual-token representation with two lightweight attention-driven modules, I³CL allows demonstrations to exchange contextual insights and dynamically routes query-relevant task knowledge into the frozen model. Core idea: reintroduce inter-demonstration self-attention to extract global collective task regularities and query-aware cross-attention to dynamically route demonstration vectors, thereby restoring standard ICL's core interactive expressiveness within a compact latent task-vector space at near zero-shot inference cost.
Method¶
Overall Architecture¶
The I³CL framework decouples task adaptation into three tightly integrated stages: dual-token demonstration representation, inter-demonstration interactive encoding, and query-aware cross-attention dynamic injection. During the demonstration encoding phase, each demonstration consisting of instruction \(I\), input \(x_i\), and label \(y_i\) is fed into a shared frozen LLM. To avoid self-referential token bias, embeddings from both the Prompt Last Token (PLT) and Label Last Token (LLT) across the Multi-Head Attention (MHA) and Feed-Forward Network (MLP) layers are averaged and concatenated into an initial demonstration context matrix. Next, a lightweight inter-demonstration self-attention module enables full-connection communication across the demonstration set, distilling collaborative task regularities. During inference on a test query, query-token states interact with the demonstration matrix via cross-attention, producing a query-specific context vector. Finally, these vectors are calibrated via layer-wise learnable scalars and added into the LLM's residual streams.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Demonstrations D={x_i, y_i} and Query x_t"] --> B["Dual-Token Context Representation<br/>Average PLT and LLT embeddings into matrix C^l"]
B --> C["Inter-Demonstration Self-Attention<br/>W_Q, W_K, W_V projections model collective task regularities"]
C --> D["Query-Aware Cross-Attention Routing<br/>Query tokens interact with demonstrations to compute c*"]
D --> E["Residual Stream Injection & Calibration<br/>Learnable scalar weights λ, β scale and merge into latent space"]
E --> F["Coherent Prediction & Text Generation"]
Key Designs¶
1. Dual-Token Context Representation: Mitigating Autoregressive Positional Bias via Complementary Boundary Anchors
Prior task-vector architectures represent each demonstration purely by extracting the hidden state of its Label Last Token (LLT). However, in causal language models (CLMs) trained with autoregressive language modeling objectives, unidirectional attention masks enforce an acute positional bias wherein the LLT's attention distribution concentrates excessively upon its own position. Consequently, the resulting vector captures isolated label semantics while severely diluting the task instructions \(I\) and input context \(x_i\). To construct a semantically balanced and complete summary, I³CL introduces the Prompt Last Token (PLT)—the terminal token of \(\{I, x_i\}\)—as an indispensable complementary boundary anchor. At layer \(l\), the MHA output \(a_i^l\) and MLP output \(m_i^l\) for the \(i\)-th demonstration are computed by averaging the PLT and LLT representations:
Concatenating \(\bar{a}_i^l\) and \(\bar{m}_i^l\) yields the demonstration context vector \(c_i^l = [\bar{a}_i^l; \bar{m}_i^l] \in \mathbb{R}^{2d}\). Stacking all \(|D|\) demonstrations establishes the layer context matrix \(C^l = [c_1^l; c_2^l; \dots; c_{|D|}^l] \in \mathbb{R}^{|D| \times 2d}\), simultaneously grounding instruction constraints and mapping relationships.
2. Inter-Demonstration Self-Attention: Bridging Isolated Encoding to Synthesize Global Collective Task Regularities
In conventional ICL, contextual self-attention across multiple examples allows emergent task properties to surface through mutual constraints; a single isolated demonstration can only ever reflect idiosyncratic local mappings. Previous task-vector approaches encode each example in total isolation, preventing the emergence of coherent global task representations. I³CL resolves this limitation by introducing a parameter-efficient self-attention mechanism applied along the demonstration dimension \(|D|\). Given the layer context matrix \(C^l \in \mathbb{R}^{|D| \times 2d}\), the module projects representations using trainable weight matrices \(W_Q, W_K, W_V \in \mathbb{R}^{|D| \times |D|}\):
Operating directly on the demonstration cardinality rather than the hidden dimension \(d\) drastically restricts the parameter budget to \(3|D|^2\). This dense, inter-example cross-pollination removes noise, aligns shared task patterns, and constructs highly discriminative, mutually orthogonal representations that resist task contamination.
3. Query-Aware Cross-Attention Routing: Replacing Static Average Pooling with Adaptive Dynamic Focus
Uniform average pooling across all demonstration vectors imposes an unyielding inductive bias: it assumes all demonstrations contribute equally to every incoming query. However, standard ICL inherently relies on test queries attending preferentially to semantically correlated demonstration exemplars. To recover this adaptive capability, I³CL deploys a cross-attention routing mechanism at each Transformer layer. Denoting the test query's layer-\(l\) MHA and MLP outputs as \(a_{x_t}^l, m_{x_t}^l \in \mathbb{R}^{s \times d}\) (where \(s\) is the query sequence length), and partitioning the refined demonstration matrix \({C'}^l\) into MHA block \({c'}_a^l\) and MLP block \({c'}_m^l \in \mathbb{R}^{|D| \times d}\), pairwise token-to-demonstration affinities are normalized via Softmax across the demonstration axis:
Using these attention scores, query-conditioned context vectors are synthesized dynamically:
This enables the model to selectively route and amplify the specific demonstration features most critical to the immediate reasoning token.
4. Residual Stream Injection and Coefficient Calibration: Harmonizing Distribution Mismatch and Autoregressive Extrapolation
Directly adding externally generated task vectors into the LLM's latent space risks distorting pretrained activation distributions. To preserve generational stability while seamlessly steering the hidden representations, I³CL incorporates layer-wise learnable scaling coefficients \(\lambda_a^l, \beta_a^l, \lambda_m^l, \beta_m^l\) within a residual formulation:
All foundational LLM weights remain completely frozen. Furthermore, during open-ended multi-token generation tasks (such as style transfer and code synthesis), I³CL recalculates and injects these contextual vectors at every consecutive autoregressive decoding step, enforcing sustained task guidance across lengthy outputs.
Loss & Training¶
The fusion coefficients \(\lambda, \beta\) and projection matrices \(W_Q, W_K, W_V\) are calibrated in an offline bootstrapping stage on the task demonstration set \(D\) by minimizing label perplexity:
Where \(O_x^*\) is the final-layer query output. The total count of trainable parameters across \(L\) layers is strictly confined to \(3|D|^2 + 4L\) (e.g., merely 428 parameters for a 32-layer LLM with 10 demonstrations). Optimization proceeds for 100 epochs using AdamW (learning rates between \(2\times 10^{-4}\) and \(2\times 10^{-6}\)). Once calibrated, these parameters are frozen, ensuring high-speed inference without gradients.
Key Experimental Results¶
Main Results¶
Evaluations span four mainstream LLM backbones (LLaMA2-7B, LLaMA3-8B, GLM4-9B, and DeepSeek-R1-8B) across eight diverse classification benchmarks (SST-2, SST-5, TREC, AGNews, HS18, DBPedia, MR, Subj) in a 5-shot setting over 5 random seeds. In addition, the text style transfer benchmark ParaDetox and code synthesis benchmark MBPP+ were evaluated.
The table below reports the complete accuracy comparison on LLaMA2-7B across all eight classification datasets (directly citing Table 1 from the paper):
| Method / Dataset | SST-2 | SST-5 | TREC | AGNews | HS18 | DBPedia | MR | Subj | Average |
|---|---|---|---|---|---|---|---|---|---|
| Zero-Shot | 89.16 | 41.04 | 41.96 | 45.60 | 76.44 | 69.44 | 76.88 | 52.52 | 61.63 |
| Standard ICL | 91.00 | 44.32 | 60.08 | 85.00 | 77.92 | 71.48 | 89.56 | 56.24 | 71.95 |
| LoRA-FT | 92.15±1.16 | 48.44±1.10 | 85.36±1.11 | 86.04±1.30 | 82.48±0.99 | 95.36±0.78 | 85.72±1.23 | 87.48±1.39 | 82.88 |
| Noise-Vector | 51.80±2.04 | 23.04±2.90 | 51.28±2.89 | 26.80±1.33 | 54.92±3.17 | 8.64±0.90 | 52.44±1.40 | 17.88±1.33 | 35.85 |
| Label-Anchor | 86.00±0.84 | 38.96±1.43 | 80.36±1.43 | 83.52±1.74 | 58.72±2.10 | 73.68±2.39 | 71.32±1.85 | 56.44±1.79 | 68.63 |
| Task-Vector | 84.92±0.91 | 36.48±2.04 | 71.92±2.32 | 73.52±2.18 | 65.00±1.66 | 76.48±1.90 | 84.80±1.07 | 56.04±1.30 | 68.65 |
| ICV | 90.16±1.04 | 35.36±2.19 | 55.52±1.97 | 73.28±1.95 | 73.68±2.04 | 90.08±1.99 | 85.12±1.27 | 59.08±2.21 | 70.29 |
| I²CL | 90.28±1.32 | 43.76±1.73 | 75.44±1.92 | 84.24±1.27 | 76.40±2.04 | 91.87±1.05 | 85.52±1.56 | 76.36±1.32 | 77.98 |
| ICL-TV | 92.60±1.64 | 44.58±1.25 | 82.69±1.84 | 85.04±1.71 | 83.25±1.32 | 93.04±1.41 | 87.23±1.67 | 85.21±1.29 | 81.71 |
| AutoComp | 91.11 | 27.35 | 59.15 | 85.74 | 58.84 | 91.18 | 84.33 | 61.51 | 69.90 |
| ICAE | 89.79 | 40.25 | 51.04 | 80.47 | 67.73 | 59.93 | 86.77 | 50.44 | 65.80 |
| CEPE | 75.83 | 35.35 | 55.36 | 79.44 | 60.83 | 85.56 | 82.75 | 59.34 | 66.81 |
| I³CL (Ours) | 92.66±0.90 | 51.16±1.97 | 87.36±1.14 | 87.07±1.33 | 86.28±1.65 | 95.88±0.57 | 90.67±0.93 | 90.84±1.88 | 85.24 |
Performance highlights on non-classification benchmarks: - ParaDetox Text Style Transfer (LLaMA2-7B): I³CL reduces toxicity to 24.80%, dramatically outperforming ICV (39.54%), I²CL (57.56%), and standard ICL (66.81%), while sustaining strong semantic preservation (BERT-Score 91.74) and fluency (ROUGE-1 69.19). - MBPP+ Code Generation: I³CL attains a Pass@1 of 62.08%, exceeding standard ICL (47.20%) by +14.88%, I²CL (53.65%) by +8.43%, and ICL-TV (55.40%) by +6.68%. - Scale Comparison vs. Large Models: LLaMA2-7B equipped with I³CL (85.24% average) approaches GPT-5.5 standard ICL (86.25%) within 1.01%, even outperforming GPT-5.5 ICL on HS18 (+1.20%) and Subj (+4.43%).
Ablation Study¶
The table below delineates the cumulative performance enhancements provided by each constituent module over the I²CL baseline (directly citing Table 3 from the paper):
| Configuration / Variant | SST-2 | SST-5 | TREC | AGNews | HS18 | DBPedia | MR | Subj | Architectural Role & Impact |
|---|---|---|---|---|---|---|---|---|---|
| I²CL Baseline | 90.28 | 43.76 | 75.44 | 84.24 | 76.40 | 91.87 | 85.52 | 76.36 | Isolated LLT token with uniform average pooling |
| + PLT (Prompt Last Token) | 91.16 | 44.08 | 75.64 | 84.64 | 85.36 | 92.08 | 86.44 | 78.60 | Rectifies positional bias; huge jumps on HS18 (+8.96) |
| + Inter-Demo (Self-Attention) | 92.56 | 44.56 | 77.16 | 84.76 | 82.96 | 94.48 | 86.64 | 77.52 | Enables collaborative cross-example task mapping |
| + Demo2Query (Cross-Attention) | 92.32 | 45.32 | 86.56 | 86.44 | 82.28 | 93.52 | 87.00 | 87.20 | Strongest single module; massive gains on TREC (+11.12) |
| + All (I³CL Full Framework) | 92.66 | 51.16 | 87.36 | 87.07 | 86.28 | 95.88 | 90.67 | 90.84 | Full synergy across all modules achieves SOTA across all sets |
| I³CL w/o FC Projections | 91.40 | 44.20 | 77.60 | 85.27 | 85.15 | 92.67 | 86.27 | 79.96 | Removing WQ,WK,WV maintains baseline edge but loses capacity |
Efficiency and scalability profiling across demonstration shots on TREC using LLaMA2-7B (citing Table 5): - Calibration Overhead: In typical 5- to 30-shot scenarios, I³CL introduces \(\le 7\%\) extra calibration time relative to I²CL (1.02x to 1.07x). Even under 100 shots (1.34x), the modest offline computational investment yields an immense +11.92% accuracy dividend. - Inference Latency: Under 5- to 30-shot regimes, I³CL exhibits \(\le 1.3\%\) additional latency over I²CL (4.10x vs 4.08x baseline), massively undercutting standard ICL's computational explosion (which scales from 5.84x at 5 shots to 36.66x at 30 shots). - Peak Memory: Standard ICL triggers fatal Out-Of-Memory (OOM) failures at 100 shots. Conversely, I³CL maintains a compact memory footprint of 2.83x (compared to I²CL's 2.04x), completely preventing OOM crashes.
Key Findings¶
- Query-Aware Routing Acts as the Primary Catalyst: Ablations clearly demonstrate that the Demo2Query module contributes the most pronounced individual gains (+11.12% on TREC, +10.84% on Subj). Attention weight visualizations in Table 13 corroborate that the routing mechanism consistently directs peak attention mass onto demonstrations matching the test question's semantic category (e.g., location queries assign top weight 0.168 to location demonstrations).
- Sharp Inter-Task Orthogonality and Discriminative Power: Pairwise cosine similarity analysis reveals that I²CL exhibits severe inter-task contamination (>85% similarity between unrelated tasks like SST-2 and TREC). In contrast, I³CL drives cross-task similarity down to near-zero levels while maintaining high intra-task consistency (>95%), verifying that inter-demonstration self-attention cleanly extracts pure, orthogonal task signatures.
- Robust Monotonic Scalability Across Demonstration Volumes: While standard ICL's accuracy degrades and destabilizes as shots increase beyond 30 on the Subj benchmark, I³CL demonstrates steady, monotonic performance growth, peaking comfortably around 30 shots and retaining high accuracy through 100 shots.
Highlights & Insights¶
- Conceptual Paradigm Shift: Overturns the historical compromise of task-vector architectures, demonstrating that high-efficiency vector injection does not require abandoning inter-example interactions and query-conditioned routing.
- Dual Boundary Token Rectification: Identifies and cures the systemic self-referential positional bias of the Label Last Token in causal transformers by elegantly pairing it with the Prompt Last Token at zero extra parameter cost.
- Compact Dimension-Orthogonal Parameterization: Parameterizing the self-attention projections over the demonstration cardinality \(|D| \times |D|\) instead of hidden feature dimension \(d \times d\) restricts trainable parameters to a few hundred floats, completely safeguarding the offline calibration stage against overfitting.
- Step-Wise Autoregressive Extrapolation: Validates that latent task-vector steering extends robustly beyond categorical classification into open-ended token generation when refreshed at each decoding step.
Limitations & Future Work¶
- Inflexibility under Dynamic Retrieval Paradigms: In dynamic RAG or instance-retrieved few-shot configurations where demonstration subsets vary per query, I³CL's reliance on a 100-epoch offline calibration step creates a computational bottleneck, demanding faster online adaptation techniques.
- Latent Space Perturbation in Generative Decoders: Authors report that injected task vectors occasionally provoke repetitive or ungrammatical sequences in open-ended text generation, indicating that additive steering can perturb the fragile local geometry of pretrained language manifolds.
- Cross-Domain Parameter Specificity: Calibrated parameters show steep degradation when transferred across semantically disparate tasks (e.g., a 26.45% accuracy drop when transferring SST-5 calibrated weights to TREC), necessitating dedicated recalibration for unrelated domains.
Related Work & Insights¶
- vs I²CL (Li et al., 2025): I²CL introduced residual stream task-vector injection but suffered from isolated demonstration encoding and uniform average pooling; I³CL directly resolves both bottlenecks via self-attention and dynamic cross-attention routing.
- vs ICL-TV (Dong et al., 2026): ICL-TV addresses low-rank representational limits by brute-force stacking multiple static task vectors; I³CL demonstrates that the primary bottleneck lies in missing interaction mechanisms, achieving higher accuracy with a single dynamic vector.
- vs ICV (Liu et al., 2024): ICV focuses primarily on latent space steering for stylistic and open-ended generation; I³CL provides a more rigorous, calibrated routing framework that excels across complex discriminative reasoning and generative tasks alike.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant conceptual restoration of interactive attention inside latent task-vector spaces.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across 4 modern LLM backbones, 8 classification benchmarks, text style transfer, code generation, and rigorous efficiency/similarity profiling.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formalization, compelling analytical prose, and transparent disclosure of design trade-offs.
- Value: ⭐⭐⭐⭐⭐ Furnishes a highly practical, parameter-efficient roadmap for deploying high-throughput, order-invariant ICL systems in production.