Skip to content

LoMaR: Local Margin Preservation via Residual Subspace Adaptation for Noisy Prompt Robustness in Small Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/HongJin2828/LoMaR
Area: Audio & Speech
Keywords: Speech Robustness, Prompt Noise, Parameter-Efficient Fine-Tuning, Residual Subspace Alignment, Local Logit Margin

TL;DR

Addressing the vulnerability of on-device small language models (SLMs) to speech ASR transcription errors and mobile typing noise caused by narrow logit margins, LoMaR introduces a two-stage PEFT framework that first widens local clean decision boundaries and then restricts noise adaptation to an SVD-isolated residual subspace, significantly reducing token-level rank flips with zero extra inference latency.

Background & Motivation

Small language models (SLMs, ranging from 0.5B to 1.5B parameters) are increasingly deployed on smartphones, edge wearables, and embedded local systems due to their minimal memory footprints and rapid inference speeds. However, interactive deployments in real-world scenarios inevitably expose models to diverse forms of surface-level noise: spoken voice interactions processed through automatic speech recognition (ASR) engines introduce phonetically similar substitutions, background noise misinterpretations, and dropped tokens, while mobile touchscreen keyboards suffer from typographical errors and colloquial contractions. Although such perturbations largely preserve underlying human intent, they severely disrupt fragile internal representations of capacity-constrained SLMs, resulting in catastrophic drops in downstream task accuracy and generation consistency.

A fine-grained token-level diagnostic reveals the mathematical root cause of this fragility across model scales (from 0.5B up to 14B): under teacher-forced decoding, SLMs exhibit fundamentally narrower logit margins between the target gold token and the top-ranked non-gold competitor token compared to larger models. This narrow safety buffer is acutely evident at the initial generation token and persists across the entire sequence average. Consequently, minor feature shifts introduced by ASR acoustic perturbations or character edits easily reverse the local ranking, causing the non-gold competitor to outscore the gold token. Such local rank flips cascade uncontrollably through autoregressive decoding steps, culminating in completely erroneous task predictions.

Existing mitigation techniques remain fundamentally ill-suited for edge SLM deployment: test-time prompt rewriting or stochastic prediction smoothing incurs prohibitive inference latency and energy consumption; meanwhile, fine-tuning approaches using data augmentation or output distribution matching (such as COIN and CoIPO) diffuse the model's scarce low-rank adaptation capacity across the entire vocabulary distribution rather than reinforcing the razor-thin boundary between the gold token and its primary competitors. Furthermore, unconstrained adaptation on noisy views risks catastrophically overwriting the dominant feature representations acquired on clean data. Core idea: This paper decouples robust adaptation into clean margin expansion and residual-subspace alignmentโ€”first explicitly widening the local logit margin between the gold token and top-K competitors on clean prompts, then applying SVD to freeze the dominant update subspace and restrict noisy adaptation strictly to the residual subspace, finally refactorizing the merged weights into a standard single-branch LoRA adapter with zero inference overhead.

Method

Overall Architecture

LoMaR establishes a two-stage parameter-efficient fine-tuning paradigm tailored for on-device SLMs under semantics-preserving noisy prompts. In Stage I (Clean Margin Expansion), the model is trained on clean prompts using standard task supervision augmented with a smoothed local margin objective, which actively pushes away top-\(K\) non-gold competitors to construct a robust local decision boundary while caching the resulting safe margins. In Stage II (Subspace-Preserved Noise Adaptation), the Stage I low-rank update \(\Delta W_{\text{base}}\) is decomposed via compact Singular Value Decomposition (SVD) into an energy-dominant main update subspace and a residual adaptation subspace. By freezing the base backbone, the main update components, and the residual projection bases, the model optimizes solely the residual coefficient matrix using paired ASR and text noisy prompts to align noisy margins with cached clean margins. Upon completion, the main and residual updates are seamlessly fused and re-factorized into standard LoRA format for deployment without any structural overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Clean Prompts and Labels"] --> B["Clean Margin Expansion<br/>Supervised Tuning + Top-K Competitor Margin Pushing"]
    B --> C["SVD Energy Decomposition & Caching<br/>Partition Dominant vs Residual Subspaces + Cache Clean Margins"]
    C --> D["Residual Subspace Noise Adaptation<br/>Freeze Dominant Bases + Tune Residual Coefficients"]
    D --> E["Local Margin Alignment<br/>Noisy Local Margins Aligned to Cached Clean Margins"]
    E --> F["Standard LoRA Refactorization<br/>Sum Subspaces and Refactorize into Unified Adapter"]

Key Designs

1. Clean Margin Expansion: Smoothing Local Competitor Repulsion to Form a Decision Buffer

Standard cross-entropy loss simply encourages the gold token probability to approach one, yet does not guarantee a substantial logit advantage over runner-up candidates under out-of-distribution acoustic or textual shifts. To establish an explicit safety cushion without traversing the full vocabulary, this design dynamically identifies the top-\(K\) non-gold logits at each position \(t\) for clean input \(x^c_i\): \(\mathcal{C}^c_{i,t} = \text{TopK}_{j \in \mathcal{V} \setminus \{y_{i,t}\}} z^c_{i,t,j}\). To mitigate gradient oscillations driven by single outlier tokens, soft-max weights \(w^c_{i,t,j} \propto \exp(z^c_{i,t,j}/\tau)\) parameterize a smoothed margin objective over local logit margins \(M^c_{i,t,j} = z^c_{i,t,y_{i,t}} - z^c_{i,t,j}\):

\[l_{\text{margin},i,t} = \sum_{j \in \mathcal{C}^c_{i,t}} w^c_{i,t,j} \cdot \text{softplus}\left(m - M^c_{i,t,j}\right)\]

By selecting an empirically grounded safety threshold \(m = 4\) positioned between the first-token margin and the sequence-average margin, this objective systematically penalizes narrow margins without inducing detrimental overconfidence, establishing a protective buffer against downstream ASR and typing corruptions.

2. SVD Energy Decomposition & Residual Subspace Isolation: Preserving Dominant Clean Directions

Directly fine-tuning an existing LoRA adapter on noisy prompts tends to overwrite the primary clean task adaptation directions due to the low intrinsic dimensionality of PEFT parameters. To counter this, LoMaR computes a singular value decomposition on the Stage I update \(\Delta W_{\text{base}} = B A = U \Sigma V^\top \in \mathbb{R}^{d \times d}\). Guided by a cumulative spectral energy ratio \(\rho\) (\(\rho = 0.9\)):

\[\frac{\sum_{j=1}^k \sigma_j^2}{\sum_{j=1}^{r_{\text{base}}} \sigma_j^2} \ge \rho\]

the update space is partitioned into a frozen dominant main subspace \(\Delta W_{\text{main}} = U_m \Sigma_m V_m^\top\) and a residual adaptation subspace \(\Delta W_{\text{res}} = U_r \Sigma_r V_r^\top\). During Stage II, the pretrained weights, \(\Delta W_{\text{main}}\), and the residual orthogonal bases \(U_r, V_r\) are entirely frozen. Adaptation on noisy prompts is strictly confined to optimizing a learnable residual coefficient matrix \(M_\Theta \in \mathbb{R}^{r_{\text{res}} \times r_{\text{res}}}\), initialized from \(\Sigma_r\):

\[\Delta W_{\text{res}}(\Theta) = U_r M_\Theta V_r^\top\]

This formulation establishes an orthogonal mathematical barrier that prevents noise adaptation from perturbing the core clean reasoning subspace.

3. Local Margin Alignment: Cross-View Invariance to Eliminate Local Rank Flips

When noisy prompts \(x^n_i\) (e.g., transcripts generated via multi-condition ASR pipelines) pass through the model, conventional distribution-level divergence matching forces small models to expend capacity on irrelevant non-target tokens. Instead, LoMaR reuses the competitor set \(\mathcal{C}^c_{i,t}\) and normalized weights \(w^c_{i,t,j}\) derived from the clean counterpart, extracting the stop-gradient cached clean margin \(\hat{M}^c_{i,t,j} = \text{sg}(M^c_{i,t,j})\). The alignment objective minimizes the expected squared margin deviation:

\[\mathcal{L}_{\text{align}} = \frac{1}{N} \sum_{i=1}^N \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \mathbb{E}_{j \sim w^c_{i,t}} \left[ \left( M^n_{i,t,j} - \hat{M}^c_{i,t,j} \right)^2 \right]\]

By directly anchoring the noisy relative margin to the clean safety margin, the model prevents acoustic perturbations from causing negative margin flips, maintaining robust decision boundaries precisely where the risk of failure is highest.

4. Standard LoRA Refactorization: Folding Decoupled Subspaces for Zero-Overhead Inference

Resource-constrained mobile edge platforms cannot tolerate multi-branch architectures or runtime feature reweighting. LoMaR resolves this operational constraint by taking the final combined adapter update \(\Delta W^* = \Delta W_{\text{main}} + \Delta W_{\text{res}}(\Theta)\) at the conclusion of training and refactorizing it via truncated SVD back into a compact, standard rank-\(r\) parameterization \(B^* A^*\). Consequently, the trained weights can either be merged directly into the base model parameters \(W_0 + \Delta W^*\) or loaded via off-the-shelf single-branch LoRA inference engines, ensuring zero auxiliary inference latency or memory overhead.

Loss & Training

Stage I optimizes clean supervised cross-entropy alongside margin expansion: \(\mathcal{L}_{\text{stage1}} = \mathcal{L}_{\text{clean}} + \lambda_m \mathcal{L}_{\text{margin}}\), using hyperparameters \(\lambda_m = 0.2\), \(m = 4\), \(K = 4\), and smoothing temperature \(\tau = 1\).
Stage II optimizes noisy prompt task supervision combined with margin alignment: \(\mathcal{L}_{\text{stage2}} = \mathcal{L}_{\text{noisy}} + \lambda_a \mathcal{L}_{\text{align}}\), with \(\lambda_a = 0.1\) and energy threshold \(\rho = 0.9\).
Training was conducted on an NVIDIA RTX 4090 GPU across linear attention projection modules q_proj, k_proj, v_proj, o_proj, configured with base rank \(r=8\), scaling factor \(\alpha=16\), learning rate \(1\text{e}-4\), batch size 32, and maximum sequence length 256.

Key Experimental Results

Main Results

Experiments are conducted across small language models (anchored by Qwen2.5-0.5B) under text perturbations (DeepWordBug, TextFooler, CheckList) and speech-derived ASR noise (ASR-GLUE with low, medium, and high acoustic noise conditions). Performance Drop Rate (PDR) and Rank Flip Rate (RFR) quantify task and token-level degradation.

Table 1: Comprehensive Performance Under Text Noise (Qwen2.5-0.5B, 3-seed & 4-template mean)

Method Clean Avg DeepWordBug Avg TextFooler Avg CheckList Avg Overall Noisy Score Average PDR (%)
Base (Zero-shot) 53.02 47.94 48.29 51.28 50.14 5.43
SFT (Data Aug) 55.92 48.72 53.06 54.97 53.17 4.92
COIN (Contrastive) 53.90 49.69 51.73 51.81 51.78 3.93
CoIPO (Logit Align) 54.91 51.90 52.70 53.47 53.25 3.02
LoMaR (Ours) 57.85 55.95 56.24 57.10 56.79 1.83

On individual benchmarks under text noise, LoMaR achieves 58.61 Acc on MNLI (PDR 1.30%), 67.29 Acc on MRPC (PDR 1.65%), 74.64 Acc on QNLI (PDR 1.74%), and 35.77 F1 on SQuAD (PDR 1.49%), consistently dominating all baselines.

Table 2: ASR-GLUE Robustness Benchmark Under Speech Transcription Noise (Qwen2.5-0.5B)

Method Clean Low WER Medium WER High WER Average Score
Base 63.41 62.42 61.48 60.29 61.90
SFT 66.33 64.61 62.55 61.34 63.71
COIN 65.53 63.38 61.75 60.31 62.74
CoIPO 66.51 65.80 63.36 62.91 64.65
LoMaR (Ours) 68.03 66.33 65.03 63.97 65.84

Under acoustic noise and diverse ASR models (wav2vec2, Whisper, Conformer), LoMaR outperforms strong competitive alignment baselines such as CoIPO by 1.19 points and conventional SFT by 2.13 points on average.

Ablation Study

Ablation experiments dissect each algorithmic component on Qwen2.5-0.5B under text perturbation settings.

Table 3: Component Ablations Under Text Perturbation

Variant Configuration Overall Noisy Score Average PDR (%) Description
Full LoMaR 56.79 1.83 Complete model with margin expansion, SVD isolation, and alignment
Two-stage SFT 50.59 5.69 Sequential SFT on clean then noisy data without margin objectives
w/o Margin 52.75 3.02 Removes Stage I clean margin expansion loss \(\mathcal{L}_{\text{margin}}\) (-4.04 pts)
w/o SVD 53.77 5.02 Tunes full LoRA in Stage II without subspace freezing (PDR surges to 5.02%)
w/o Alignment 53.90 3.16 Removes Stage II local margin alignment loss \(\mathcal{L}_{\text{align}}\) (-2.89 pts)
Base Model 50.14 5.43 Unadapted base foundation model

In terms of training efficiency, LoMaR incurs only 1.02x the training time of standard SFT (throughput of 12.06 samples/s compared to 10.65 for COIN and 10.45 for CoIPO), demonstrating that the compact one-time SVD decomposition adds negligible overhead.

Key Findings

  • Decoupled Subspace Preservation Prevents Overfitting: Merely sequencing training data across two stages yields almost no gain (50.59 score). Crucially, removing SVD subspace isolation (w/o SVD) causes average PDR to spike from 1.83% to 5.02%, confirming that unconstrained noise tuning erodes essential representations learned from clean prompts.
  • Selective Rectification of the Negative-Margin Tail: Quantile analysis reveals that LoMaR does not uniformly inflate global margins. Instead, it selectively contracts the vulnerable left tail: the proportion of tokens with negative margins (where competitors outrank the target) drops from 23.64% in the Base model and 19.30% in SFT down to 16.80% in LoMaR.
  • Consistent Scaling Across Model Capacities: The performance advantages of LoMaR extend seamlessly to Llama-3.2-1B (56.55 vs. 49.31 for SFT), Qwen2.5-1.5B (70.14 vs. 66.45 for SFT), 3B (72.54 vs. 69.04), and 7B (76.42 vs. 74.36), proving broad efficacy across architectural scales.

Highlights & Insights

  • Local Margin Targeting Rather Than Global Divergence Matching: By identifying that token-level rank flips stem from narrow top-competitor margins rather than broad vocabulary discrepancies, LoMaR focuses its limited optimization capacity specifically where errors originate, achieving sharp decision boundaries.
  • Orthogonal Subspace Decoupling via Low-Rank SVD: Treating the low-rank update matrix as a functional manifoldโ€”freezing high-energy singular components and restricting noise adaptation strictly to the residual directionsโ€”provides an elegant mathematical guardrail against catastrophic forgetting during robust fine-tuning.
  • Practical On-Device Deployability: The framework unifies text and ASR speech noise resilience into a single training protocol and compiles back into an off-the-shelf single-adapter format, making it directly deployable to edge voice assistants with zero inference penalty.

Limitations & Future Work

  • Restricted to Closed-Set Deterministic Tasks: The margin expansion objective is formulated around known gold token targets, making it directly suited for classification, NLI, and extractive QA. Defining adaptive competitor margins for open-ended creative text generation remains an open challenge.
  • Synthetic ASR Generation Pipeline: While the TTS-ASR generation pipeline integrates diverse acoustic reverberations and acoustic backbones, real-world conversational speech exhibits complex overlapping speakers and rapid conversational disfluencies that require further evaluation.
  • Multi-Turn Dialog Cascades: Current evaluations primarily focus on single-turn instruction following; analyzing how residual logit margin preservation mitigates multi-step error accumulation in extended spoken dialogues represents a promising future avenue.
  • vs SFT (Data Augmentation): Standard SFT mixes clean and perturbed samples naively, causing small language models to overfit surface corruptions while diluting clean accuracy; LoMaR's two-stage decoupled paradigm achieves 3.62 points higher noisy accuracy while boosting clean performance by 1.93 points.
  • vs COIN / CoIPO (Representation & Logit Alignment): COIN enforces contrastive representation constraints while CoIPO aligns full-vocabulary logits; both spread low-rank PEFT capacity too thin. LoMaR focuses strictly on local top-\(K\) margins and residual subspace isolation, cutting the relative performance drop rate by roughly half.
  • vs PiSSA / MiLoRA (SVD-based PEFT): Prior SVD-based LoRA methods decompose pretrained model weights to guide parameter allocation; LoMaR innovatively decomposes the task-adapted incremental update matrix to decouple clean dominant knowledge from noisy residual tuning.

Rating

  • Novelty: โญโญโญโญโ˜† Pinpointing narrow local logit margins as the root cause of SLM noise vulnerability and isolating noise adaptation via post-tuning SVD represents an elegant architectural insight.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 0.5B to 7B backbones, multiple text attacks, multi-condition ASR-GLUE benchmarks, quantile breakdowns, and hyperparameter sensitivity sweeps.
  • Writing Quality: โญโญโญโญโญ Rigorous mathematical formulation, coherent narrative progression from token-level diagnostics to modular solutions, and detailed visualization of update spaces.
  • Value: โญโญโญโญโญ Provides an immediate, zero-inference-overhead recipe for enhancing the noise resilience of on-device LLMs and speech assistants.