LoMaR: Local Margin Preservation via Residual Subspace Adaptation for Noisy Prompt Robustness in Small Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/HongJin2828/LoMaR
Area: LLM Reasoning
Keywords: Small Language Models, Prompt Robustness, Local Logit Margin, Residual Subspace Adaptation, Parameter-Efficient Fine-Tuning
TL;DR¶
LoMaR tackles the noisy prompt vulnerability of on-device small language models by combining Stage I clean local logit margin expansion with Stage II SVD-decomposed residual subspace adaptation and margin alignment, significantly lowering performance drop rates and token rank flip rates without incurring any inference overhead.
Background & Motivation¶
Small language models (SLMs, typically ranging from 0.5B to 1.5B parameters) are increasingly deployed on edge and mobile devices due to their compact memory footprint and low operational latency. However, in practical deployment scenarios such as mobile virtual keyboards or voice assistants, user-provided prompts frequently deviate from clean canonical text. Mobile typing frequently introduces typographical slips, letter omissions, and colloquial syntax, while speech-based interaction inevitably suffers from acoustic noise, reverberation, and automatic speech recognition (ASR) transcription errors. Although these input perturbations largely preserve the underlying user intent, they cause substantial surface-level feature shifts that severely degrade the downstream task performance of SLMs.
Empirical evaluations across model scales reveal that smaller language models experience disproportionately severe degradation compared to their larger counterparts when subjected to identical perturbations. Analyzing this vulnerability from a token-level perspective under teacher-forced decoding uncovers the core cause: SLMs inherently exhibit much narrower local logit margins between the target gold token and the top-ranked non-gold competitor tokens. Because the first generated token often determines the task classification or reasoning trajectory in SLMs, even a minor logit shift induced by prompt noise easily triggers a local rank flip at this critical decision point, cascading into a complete task-level failure across subsequent autoregressive decoding steps.
Existing approaches to prompt robustness remain inadequate for resource-constrained SLMs. Inference-time mitigation techniques such as prompt rewriting and prediction smoothing impose substantial latency and computation overheads, negating the deployment efficiency of edge models. Meanwhile, fine-tuning approaches based on data augmentation (e.g., standard SFT on mixed clean and noisy views) or consistency alignment (e.g., COIN, CoIPO) attempt to match global hidden states or entire output distributions. In parameter-efficient fine-tuning (PEFT) where parameter updates are inherently low-rank, broad distribution matching dilutes limited parameter capacity across non-decision-critical vocabulary dimensions, while failing to protect the model's core clean representations from being overwritten by noisy updates. Core idea: decouple robust fine-tuning into a two-stage orthogonal adaptation framework that first expands and caches clean local logit margins around the gold token, and then uses SVD to freeze the dominant clean update directions while aligning noisy local margins strictly within the orthogonal residual subspace, achieving high noise resilience with zero added inference overhead.
Method¶
Overall Architecture¶
LoMaR establishes a two-stage parameter-efficient robust fine-tuning pipeline for SLMs under semantics-preserving textual and ASR noise, which seamlessly folds into a standard LoRA adapter for deployment. In Stage I, the base model is fine-tuned on clean prompts with both the supervised task loss and a novel token-level local logit margin expansion loss, widening the separation boundary between the gold token and top competing non-gold tokens while caching these clean margins. In Stage II, singular value decomposition (SVD) is performed on the Stage I LoRA update matrix to partition the parameter update space into a frozen main update subspace (retaining dominant spectral energy) and a trainable residual adaptation subspace. During Stage II, only the residual subspace coefficient matrix is optimized on noisy prompts to align noisy local margins with the cached clean margins. Finally, the main and adapted residual components are recombined and refactorized into a standard LoRA adapter structure, introducing zero architectural changes or computational overhead during deployment.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Clean and Noisy Prompt Pairs<br/>Input with Identical Target Output"] --> B["Clean Local Margin Expansion<br/>Top-K Competitor Soft-Margin Loss in Stage I"]
B --> C["Subspace Spectral Energy Partition<br/>SVD on LoRA Matrix and Freeze Main Subspace"]
C --> D["Residual Subspace Margin Alignment<br/>Train Residual Coefficients to Align Cached Margins"]
D --> E["Adapter Recombination and Refactorization<br/>Fold into Standard LoRA with Zero Inference Cost"]
Key Designs¶
1. Clean Local Margin Expansion: Eliminating Narrow Decision Boundaries at Token Level
Small language models struggle with noisy prompts primarily because their gold token logits are barely separated from top non-gold competitors, making them fragile to minor input perturbations. In Stage I, LoMaR trains a base LoRA adapter \(\Delta W_{\text{base}} = BA\) on clean inputs \(x_i^c\) while actively pushing away competing tokens. For target position \(t\) with gold token \(y_{i,t}\) and output logits \(z_{i,t}^c\), the model identifies the top-\(K\) non-gold logits to construct a local competitor set \(C_{i,t}^c = \text{TopK}_{j \in V \setminus \{y_{i,t}\}} z_{i,t,j}^c\). To prevent the optimization from over-indexing on a single outlier competitor, a temperature-scaled softmax weight \(w_{i,t,j}^c \propto \exp(z_{i,t,j}^c / \tau)\) is applied across \(C_{i,t}^c\). Defining the local logit margin as \(M_{i,t,j}^c = z_{i,t,y_{i,t}}^c - z_{i,t,j}^c\) against a preset minimum safety margin \(m\), the token-level margin expansion loss is computed via a smooth softplus penalty: $\(l_{\text{margin}}^{i,t} = \sum_{j \in C_{i,t}^c} w_{i,t,j}^c \cdot \text{softplus}\left(m - M_{i,t,j}^c\right)\)$ This objective explicitly expands the decision boundary around the gold token on clean data. At the conclusion of Stage I, the computed margins \(\hat{M}_{i,t,j}^c = \text{sg}(M_{i,t,j}^c)\) and competitor indices \(C_{i,t}^c\) are detached and cached as target references for subsequent alignment.
2. Subspace Spectral Energy Partition: Isolating Noise Adaptation from Core Clean Directions
Directly fine-tuning the entire adapter on noisy prompts in Stage II risks overwriting the dominant feature representations established during clean training, causing catastrophic degradation on clean inputs. To resolve this parameter conflict, LoMaR computes a compact SVD on the low-rank Stage I update: \(\Delta W_{\text{base}} = U \Sigma V^T\). Sorting singular values \(\sigma_j\) in descending order, the smallest rank \(k\) is selected such that the cumulative squared singular value energy reaches a threshold ratio \(\rho\) (empirically set to 0.9): $\(\frac{\sum_{j=1}^k \sigma_j^2}{\sum_{j=1}^{r_{\text{base}}} \sigma_j^2} \ge \rho\)$ The parameter space is thus partitioned into a dominant main update subspace \(\Delta W_{\text{main}} = U_m \Sigma_m V_m^T\) of rank \(k\) and an orthogonal residual adaptation subspace \(\Delta W_{\text{res}} = U_r \Sigma_r V_r^T\) of rank \(r_{\text{res}} = r_{\text{base}} - k\). During Stage II, the base model, \(\Delta W_{\text{main}}\), and projection bases \(U_r, V_r\) are entirely frozen. Only the residual coefficient matrix \(M_\Theta \in \mathbb{R}^{r_{\text{res}} \times r_{\text{res}}}\) (initialized from \(\Sigma_r\)) is updated: $\(\Delta W_{\text{res}}(\Theta) = U_r M_\Theta V_r^T\)$ This architecture restricts noisy prompt adaptation strictly to non-dominant residual directions, structurally preventing noise gradients from disturbing clean task representations.
3. Residual Subspace Margin Alignment: Precision Alignment without Exhausting PEFT Capacity
Conventional consistency methods attempt to align full output distributions or high-dimensional hidden states across clean and noisy views, which rapidly exhausts the low-rank capacity of SLMs on non-essential dimensions. Instead, LoMaR focuses adaptation solely on preserving the local ranking between the gold token and its primary competitors. In Stage II, when feeding a paired noisy prompt \(x_i^n\), the model reuses the Stage I competitor set \(C_{i,t}^c\) and weights \(w_{i,t,j}^c\) to compute the noisy local margins \(M_{i,t,j}^n = z_{i,t,y_{i,t}}^n - z_{i,t,j}^n\). The alignment loss penalizes the squared deviation between noisy margins and cached clean margins: $\(\mathcal{L}_{\text{align}} = \frac{1}{N} \sum_{i=1}^N \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \mathbb{E}_{j \sim w_{i,t}^c} \left[\left(M_{i,t,j}^n - \hat{M}_{i,t,j}^c\right)^2\right]\)$ By penalizing only the degradation of local safety margins rather than enforcing full vocabulary matching, this design provides a lightweight, highly targeted objective that perfectly suits the constrained capacity of the trainable residual subspace.
4. Adapter Recombination and Refactorization: Ensuring Zero Inference Overhead
Practical edge deployment cannot tolerate multi-branch execution or dynamic routing latency. LoMaR ensures that after training, the composite effective adapter \(\Delta W^* = \Delta W_{\text{main}} + \Delta W_{\text{res}}^*\) is merged into a singular matrix. SVD is applied to \(\Delta W^*\) to refactorize it back into compact standard LoRA matrices \(B^* \in \mathbb{R}^{d \times r}\) and \(A^* \in \mathbb{R}^{r \times d}\). The resulting adapter requires no custom runtime kernels, introduces zero extra parameters or FLOPs, and plugs directly into standard LoRA inference or parameter-merging workflows.
Loss & Training¶
LoMaR trains on a paired dataset constructed from 25 deterministic-answer FLAN subsets without requiring extra augmented samples: 1. Stage I Objective: \(\mathcal{L}_{\text{stage1}} = \mathcal{L}_{\text{clean}} + \lambda_m \mathcal{L}_{\text{margin}}\), optimized on clean prompt views. Hyperparameters: LoRA rank \(r=8\), scaling factor \(\alpha=16\), learning rate \(1\text{e}-4\), batch size \(32\), target margin \(m=4\), competitor size \(K=4\), temperature \(\tau=1.0\), and margin loss weight \(\lambda_m=0.2\). 2. Stage II Objective: \(\mathcal{L}_{\text{stage2}} = \mathcal{L}_{\text{noisy}} + \lambda_a \mathcal{L}_{\text{align}}\), optimized on paired noisy views with frozen \(\Delta W_{\text{main}}\) (\(\rho=0.9\)). Alignment loss weight is \(\lambda_a=0.1\). All models are trained on a single NVIDIA RTX 4090 GPU, maintaining throughput and wall-clock times comparable to vanilla SFT.
Key Experimental Results¶
Main Results¶
Experiments are conducted on Qwen2.5-0.5B, Qwen2.5-1.5B, and Llama-3.2-1B under both textual noise (DeepWordBug character noise, TextFooler word substitutions, and CheckList irrelevant sentence noise) and acoustic ASR noise (low, medium, and high WER levels from TTS-ASR synthesis). Benchmarks include GLUE, ASR-GLUE, SQuAD, MMLU, GSM8K, and CoQA. Key metrics include task accuracy/F1, Performance Drop Rate (PDR), and Rank Flip Rate (RFR).
The table below presents the main results of Qwen2.5-0.5B under clean prompts and three text-noise settings (directly quoted from Table 1 of the paper):
| Noise Setting | Method | MNLI (Acc / PDR) | MRPC (Acc / PDR) | QNLI (Acc / PDR) | QQP (Acc / PDR) | SQuAD (F1 / PDR) | MMLU (Acc / PDR) | Avg Score / PDR |
|---|---|---|---|---|---|---|---|---|
| Clean | Base | 48.42 / - | 66.46 / - | 70.92 / - | 66.83 / - | 34.34 / - | 31.17 / - | 53.02 / - |
| SFT | 55.29 / - | 68.04 / - | 69.83 / - | 72.58 / - | 34.10 / - | 35.67 / - | 55.92 / - | |
| COIN | 60.54 / - | 54.75 / - | 74.46 / - | 65.04 / - | 31.66 / - | 36.92 / - | 53.90 / - | |
| CoIPO | 57.13 / - | 67.75 / - | 68.79 / - | 73.83 / - | 30.30 / - | 31.67 / - | 54.91 / - | |
| LoMaR (Ours) | 59.38 / - | 68.42 / - | 75.96 / - | 71.17 / - | 36.31 / - | 35.83 / - | 57.85 / - | |
| DeepWordBug | Base | 41.50 / 14.29 | 55.54 / 16.43 | 68.46 / 3.47 | 57.50 / 13.96 | 33.99 / 1.02 | 30.67 / 1.60 | 47.94 / 9.58 |
| SFT | 46.54 / 15.83 | 53.33 / 21.62 | 70.08 / -0.36 | 63.21 / 12.91 | 34.22 / -0.35 | 24.92 / 30.14 | 48.72 / 12.88 | |
| COIN | 56.27 / 7.05 | 52.08 / 4.88 | 72.75 / 2.30 | 55.08 / 15.31 | 29.52 / 6.76 | 32.43 / 12.16 | 49.69 / 7.81 | |
| CoIPO | 53.63 / 6.13 | 63.46 / 6.33 | 68.54 / 0.36 | 70.46 / 4.56 | 28.02 / 7.52 | 27.30 / 13.80 | 51.90 / 5.48 | |
| LoMaR (Ours) | 57.83 / 2.61 | 65.33 / 4.52 | 74.67 / 1.70 | 68.96 / 3.11 | 35.41 / 2.48 | 33.50 / 6.50 | 55.95 / 3.28 | |
| TextFooler | Base | 51.50 / -6.36 | 63.71 / 4.14 | 66.71 / 5.94 | 53.63 / 19.75 | 30.59 / 10.92 | 23.58 / 24.35 | 48.29 / 8.92 |
| SFT | 56.75 / -2.64 | 66.46 / 2.32 | 66.67 / 4.53 | 65.00 / 10.44 | 31.47 / 7.71 | 32.00 / 10.29 | 53.06 / 5.11 | |
| COIN | 58.18 / 3.90 | 51.54 / 5.86 | 71.17 / 4.42 | 64.25 / 1.21 | 29.04 / 8.28 | 36.17 / 2.03 | 51.73 / 4.03 | |
| CoIPO | 55.79 / 2.35 | 63.58 / 6.15 | 66.87 / 2.79 | 72.29 / 2.09 | 29.81 / 1.62 | 27.83 / 12.13 | 52.70 / 4.02 | |
| LoMaR (Ours) | 58.00 / 2.32 | 67.04 / 2.02 | 73.29 / 3.52 | 69.58 / 2.23 | 34.84 / 4.05 | 34.67 / 3.24 | 56.24 / 2.78 | |
| CheckList | Base | 48.08 / 0.70 | 66.33 / 0.20 | 69.25 / 2.35 | 63.71 / 4.67 | 33.12 / 3.55 | 27.17 / 12.83 | 51.28 / 3.28 |
| SFT | 57.08 / -3.24 | 66.92 / 1.65 | 66.54 / 4.71 | 71.46 / 1.54 | 33.96 / 0.41 | 33.83 / 5.16 | 54.97 / 1.70 | |
| COIN | 59.17 / 2.26 | 54.25 / 0.91 | 72.63 / 2.46 | 63.92 / 1.72 | 29.95 / 5.40 | 30.92 / 16.25 | 51.81 / 3.88 | |
| CoIPO | 57.58 / -0.79 | 66.79 / 1.42 | 66.71 / 3.02 | 72.55 / 1.73 | 29.52 / 2.57 | 27.67 / 12.63 | 53.47 / 2.62 | |
| LoMaR (Ours) | 59.21 / 0.29 | 68.37 / 0.07 | 74.62 / 1.76 | 69.04 / 2.99 | 36.50 / -0.52 | 34.87 / 2.68 | 57.10 / 1.30 | |
| Text Avg | Base | 47.38 / 2.15 | 63.01 / 5.19 | 68.84 / 2.93 | 60.42 / 9.59 | 33.01 / 3.87 | 28.15 / 9.69 | 50.14 / 5.43 |
| SFT | 53.92 / 2.48 | 63.69 / 6.39 | 68.28 / 2.22 | 68.06 / 6.23 | 33.44 / 1.94 | 31.61 / 11.38 | 53.17 / 4.92 | |
| COIN | 58.54 / 3.30 | 53.16 / 2.90 | 72.75 / 2.30 | 62.07 / 4.57 | 30.04 / 5.12 | 34.11 / 7.61 | 51.78 / 3.93 | |
| CoIPO | 56.03 / 1.93 | 65.40 / 3.47 | 67.73 / 1.54 | 72.28 / 2.10 | 29.41 / 2.94 | 28.62 / 9.63 | 53.25 / 3.02 | |
| LoMaR (Ours) | 58.61 / 1.30 | 67.29 / 1.65 | 74.64 / 1.74 | 69.69 / 2.08 | 35.77 / 1.49 | 34.72 / 3.10 | 56.79 / 1.83 |
On the ASR-GLUE benchmark across Low, Medium, and High ASR noise levels (Table 2), LoMaR achieves an average score of 65.84, outperforming the Base model (61.90), SFT (63.71), COIN (62.74), and CoIPO (64.65) by clear margins.
Ablation Study¶
The ablation study systematically analyzes the contribution of the two-stage schedule, margin loss, SVD subspace partitioning, and residual alignment (quoted directly from Table 3 of the paper):
| Configuration | MNLI (Acc / PDR) | MRPC (Acc / PDR) | QNLI (Acc / PDR) | QQP (Acc / PDR) | SQuAD (F1 / PDR) | MMLU (Acc / PDR) | Overall (Avg / PDR) |
|---|---|---|---|---|---|---|---|
| Base | 47.38 / 2.15 | 63.01 / 5.19 | 68.84 / 2.93 | 60.42 / 9.59 | 33.01 / 3.87 | 28.15 / 9.69 | 50.14 / 5.43 |
| Two-stage SFT | 51.21 / 1.82 | 62.65 / 2.20 | 67.45 / 4.43 | 59.37 / 9.06 | 32.60 / 4.28 | 30.24 / 8.29 | 50.59 / 5.69 |
| w/o Margin | 57.45 / 3.32 | 60.95 / 1.76 | 70.96 / 3.13 | 58.50 / 1.75 | 32.92 / 6.37 | 35.71 / 3.28 | 52.75 / 3.02 |
| w/o SVD | 52.36 / 3.48 | 65.54 / 4.90 | 71.04 / 2.18 | 68.40 / 7.52 | 32.31 / 4.66 | 32.96 / 8.44 | 53.77 / 5.02 |
| w/o Alignment | 56.73 / 2.61 | 65.05 / 1.99 | 70.28 / 4.05 | 63.45 / 0.20 | 33.70 / 8.30 | 34.17 / 4.42 | 53.90 / 3.16 |
| LoMaR (Full Model) | 58.61 / 1.30 | 67.29 / 1.65 | 74.64 / 1.74 | 69.69 / 2.08 | 35.77 / 1.49 | 34.72 / 3.10 | 56.79 / 1.83 |
Key Findings¶
- Clean Margin Expansion is Decisive: Removing the margin loss in Stage I drops average noisy performance from 56.79 to 52.75 (-4.04 points). Logit margin quantile analysis (Figure 5) reveals that LoMaR does not inflate logits indiscriminately; rather, it specifically contracts the negative-margin tail (Q01โQ10), preventing the vulnerable tokens from being overtaken by non-gold competitors.
- SVD Subspace Freezing Prevents Catastrophic Forgetting: Without SVD subspace partitioning (
w/o SVD), the average PDR spikes to 5.02% (nearly triple that of the full model at 1.83%), and the average score drops to 53.77. SVD subspace reconstruction experiments (Table 13) confirm that the frozen main subspace \(\Delta W_{\text{main}}\) alone recovers the vast majority of clean adaptation gain (53.94 vs 50.14 for Base), validating the strategy of isolating noise adaptation to the residual subspace. - Low Training Cost and Cross-Scale Generalization: LoMaR introduces only a single SVD operation after Stage I. Its normalized training wall-clock time is just 1.02ร that of standard SFT (12.06 samples/s vs 12.57 samples/s), noticeably faster than COIN (1.13ร) and CoIPO (1.18ร). Scaling experiments show robust gains across larger models: +5.13 on Qwen2.5-1.5B, +3.50 on Qwen2.5-3B, and +2.06 on Qwen2.5-7B under text noise. Furthermore, data scaling studies (Table 5) show that increasing SFT data from 25% to 100% fails to close the robustness gap with LoMaR.
Highlights & Insights¶
- Root-Cause Diagnosis via Local Logit Margins: Rather than treating prompt sensitivity as an abstract embedding mismatch, the paper identifies token-level margin collapse as the direct mechanical trigger of local rank flips in autoregressive SLMs.
- Spectral Energy Decoupling in Low-Rank Adapters: Utilizing SVD on intermediate LoRA adapters to segregate dominant task-solving directions from residual noise-buffering dimensions provides an elegant, generalizable template for multi-objective PEFT without structural parameter bloat.
- Zero-Cost Inference Deployment: By recombining the updated residual and frozen main subspaces back into standard LoRA matrices, LoMaR guarantees zero additional latency, parameters, or GPU memory overhead during serving.
Limitations & Future Work¶
- Evaluation Limited to Deterministic Tasks: The current benchmark focuses primarily on classification, NLI, extractive question answering, and short-form math reasoning where gold tokens are unambiguous. Extending local margin preservation to open-ended text generation and conversational contexts remains an open challenge.
- Restricted to Standard LoRA Parameterization: The SVD-based subspace partitioning is tied to the low-rank factorization of LoRA. Its applicability to other PEFT variants (e.g., prefix tuning, IA3) or full-parameter fine-tuning has not been explored.
- Future Directions: Exploring dynamic adaptive safety margins (\(m\)) conditioned on token entropy and extending the framework to multimodal settings (e.g., speech-text interleaved streams) are promising avenues for future research.
Related Work & Insights¶
- vs Data Augmentation Fine-Tuning (SFT): Standard SFT merely mixes perturbed prompts into the training set, which fails to establish explicit safety boundaries and risks overfitting to observed noise distributions. LoMaR enforces explicit margin expansion and subspace isolation, achieving +3.62 higher noisy average score under identical data exposure.
- vs Representation and Consistency Alignment (COIN / CoIPO): COIN aligns hidden representations while CoIPO aligns label logits across clean and noisy pairs. Both methods expend the limited capacity of SLMs on matching non-essential vocabulary distributions. LoMaR targets only the Top-\(K\) local competitor margins, achieving better robustness with lower computational overhead.
- vs SVD-based LoRA Methods (PiSSA / MiLoRA): While PiSSA and MiLoRA utilize SVD for adapter initialization to retain pretrained capabilities, LoMaR is the first to employ SVD for two-stage parameter decoupling between clean task learning and noise adaptation.
Rating¶
- Novelty: โญโญโญโญโ [Precise token-level logit margin formulation paired with an elegant SVD residual subspace partitioning mechanism]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 3 model families, 6 parameter scales, 2 noise modalities, and comprehensive ablations]
- Writing Quality: โญโญโญโญโญ [Clear motivation, mathematically rigorous derivations, informative visualizations, and crisp structural organization]
- Value: โญโญโญโญโญ [Highly practical for on-device SLM deployment where inference latency and prompt noise resilience are paramount]