MOLM: Mixture of LoRA Markers

Conference: ICLR 2026 arXiv: 2510.00293 Code: Not released Area: Image Generation Keywords: Watermarking, LoRA, Diffusion Models, Routing Mechanism, Robustness

TL;DR

This paper proposes MOLM, a watermarking framework that reinterprets LoRA adapters as watermark carriers. A binary key-driven routing mechanism embeds verifiable and robust watermarks into a frozen generative model without per-key retraining.

Background & Motivation

• High-quality images generated by diffusion models raise concerns about authenticity and attribution.
• Existing watermarking methods face three major challenges:
• Fragility: Adversarial attacks (regeneration attacks, averaging attacks) can readily remove watermarks.
• Quality-robustness trade-off: Improving robustness typically introduces visible degradation.
• High cost: Changing the watermark key requires expensive retraining (e.g., Stable Signature requires per-key training).

Method

General Watermarking Framework

Watermarking is formalized as a key-dependent parameter perturbation applied to a frozen generative model:

\[\tilde{\mathbf{x}} = \mathcal{G}_{\Phi + \Delta\Phi(\kappa)}(\mathbf{q}, \mathbf{t})\]

where \(\Delta\Phi(\kappa)\) denotes the parameter perturbation determined by key \(\kappa\).

MOLM Routing Mechanism

1. Architecture: \(P\) LoRA adapters are added to each of \(L\) pre-selected blocks.
2. Key mapping: An \(M\)-bit binary key is divided into \(L\) non-overlapping segments \(\kappa_\ell\), each of length \(\log_2 P\) bits.
3. Routing: Each segment \(\kappa_\ell\) is converted to a decimal index \(s_\ell \in [P]\), which activates the corresponding adapter.

The operation at block \(\ell\) is:

\[\boldsymbol{h}_\ell = \mathcal{F}_\ell(\boldsymbol{h}_{\ell-1}) + \alpha \mathcal{A}_\ell^{(s_\ell)}(\boldsymbol{h}_{\ell-1})\]

Default configuration: \(L=14\) ResNet blocks (VAE decoder), \(P=4\) adapters per block, total key length \(M = 14 \times 2 = 28\) bits.

Loss & Training

Perceptual imperceptibility loss:

\[\mathcal{L}_{\text{imp}} = \mathbb{E}_{\kappa} \frac{1}{N} \sum_{n=1}^N \sum_{k=1}^K w_k \|\varphi_k(\mathcal{G}_{\Phi+\Psi(\kappa)}(\mathbf{q}, \mathbf{t}_n)) - \varphi_k(\mathcal{G}_\Phi(\mathbf{q}, \mathbf{t}_n))\|_2^2\]

Verifiability loss (binary cross-entropy):

\[\mathcal{L}_{\text{ver}} = \mathbb{E}_{T \sim \Pi} \frac{1}{NM} \sum_{n,m} [-\kappa_m \log \sigma(u_m) - (1-\kappa_m)\log(1-\sigma(u_m))]\]

Overall objective: \(\min_{\Psi, \eta} [\mathcal{L}_{\text{ver}} + \lambda \mathcal{L}_{\text{imp}}]\)

Key Experimental Results

Detection & Robustness Comparison (Stable Diffusion v1.5, MS-COCO)

Method	FID(↓)	SSIM(↑)	Clean	Crop	Rot	Resize	Bright	JPEG	Key Size
Stable Signature	29.5	0.85	0.99	0.97	0.56	0.72	0.95	0.89	48
AquaLoRA	30.5	0.63	0.95	0.91	0.45	0.91	0.72	0.94	48
WOUAF	27.8	0.73	0.98	0.96	0.85	0.71	0.98	0.98	32
MOLM	27.7	0.77	0.98	0.91	0.84	0.90	0.95	0.89	28

Adversarial Attack Robustness (After Augmented Training)

Attack Type	Parameters	Bit Acc.	FID
Cheng2020 Compression	q=1/3/6	0.94/0.95/0.97	30.1/28.9/28.7
Diffusion Regeneration	steps=30/60/100	0.85/0.85/0.82	30.2/29.9/31.2
PGD Adversarial	ε=10⁻³/10⁻²/10⁻¹	1.00/0.99/0.96	28.4/28.6/29.0
Averaging Attack (5000 images)	k=5000	≥0.96	-

Key Findings

1. MOLM achieves the best overall robustness with a smaller key (28 bits vs. 48 bits).
2. Under averaging attacks, MOLM maintains ≥0.96 bit accuracy (5000 images), while WOUAF drops below 0.90.
3. Under forgery attacks, MOLM remains at the random-guess level (≈0.5), effectively preventing forgery.
4. Training requires approximately one day on a single A100; inference introduces no additional overhead.

Highlights & Insights

1. Conceptual innovation: Redefining LoRA from a model adaptation tool to a watermark carrier is a novel and elegant perspective.
2. No per-key retraining: Capacity scales naturally by adjusting the number of routing layers and adapters.
3. Distributed redundant encoding: Mapping analysis reveals that keys are redundantly encoded across multiple blocks, enhancing robustness.
4. Sampler-agnostic: The method does not rely on a specific sampler, unlike approaches such as Tree-Ring that require deterministic sampling.

Limitations & Future Work

• Routing in the UNet leads to degraded generation quality, requiring a trade-off between key size and fidelity.
• Validation is conducted only on SD v1.5 and FLUX; generalization to more architectures remains to be tested.
• A 28-bit key capacity may be insufficient for large-scale user attribution.
• Watermarks are non-transferable when an attacker independently retrains the model (as intended by design).

Related Work & Insights

• Encoder-decoder methods: Hidden, Stable Signature
• Backdoor methods: DreamBooth fine-tuning, SleeperMark
• Generation process methods: Tree-Ring, Gaussian Shading, ROBIN
• Mixture of LoRA experts: MoLE

Rating

• Novelty: ⭐⭐⭐⭐⭐ — The conceptual reframing of LoRA as a watermark carrier is highly elegant.
• Technical Depth: ⭐⭐⭐⭐ — The framework design is complete and the attack evaluation is comprehensive.
• Experimental Thoroughness: ⭐⭐⭐⭐ — Validated across multiple attacks, datasets, and architectures.
• Value: ⭐⭐⭐⭐ — An efficient and deployable watermarking solution.

Related Papers

• [AAAI 2026] T-LoRA: Single Image Diffusion Model Customization Without Overfitting
• [ICCV 2025] MoFRR: Mixture of Diffusion Models for Face Retouching Restoration
• [CVPR 2026] Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge
• [CVPR 2026] Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
• [AAAI 2026] Mixture of Ranks with Degradation-Aware Routing for One-Step Real-World Image Super-Resolution