Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Project: Moebius
Area: Model Compression
Keywords: Image inpainting, lightweight architecture, knowledge distillation, latent diffusion, multi-granularity alignment

TL;DR

Moebius combines a lightweight LλMI backbone with adaptive multi-granularity latent distillation to build a 0.226B-parameter inpainting specialist, achieving FID 0.92 and LPIPS 0.091 on Places2 with small masks; its approximately 15.5-fold total-time advantage over FLUX.1-Fill-Dev comes from both faster individual steps and fewer default sampling steps.

Background & Motivation

Image inpainting requires more than filling a hole with similar colors: generated textures must connect to the boundary, and object structure must fit the semantics of the entire image. Large diffusion models provide broad priors but require substantial parameter and compute budgets. PixelHacker already reduces some overhead through Latent Categories Guidance (LCG) and Gated Linear Attention (GLA), yet its backbone still contains 0.862B parameters. Simply replacing convolutions with depthwise convolutions can remove precisely the structural and textural capacity that inpainting needs.

The paper frames the problem as a representational bottleneck in extremely small students rather than merely excessive sampling steps. In an early-checkpoint ablation, the baseline obtained by reducing the macro-architecture has FID 32.75; replacing only its convolutions with depthwise convolutions worsens this to 43.58. Meanwhile, the baseline uses GLA for self-attention, without a directly corresponding formulation for cross-interaction with external LCG priors. Compressing one component in isolation therefore does not produce a coherent lightweight design.

Moebius changes both information interaction and training supervision. It first ensures that the small backbone can aggregate within-image context and read external semantics, then uses a teacher to constrain student representations at different spatial scales. Core Idea: replace expensive attention interactions with compact λ mappings and compensate for structural compression through gradient-adaptive, multi-granularity latent distillation, rather than assuming that a small model inherently possesses every capability of a large generalist.

Method

Overall Architecture

The inputs are an image to be completed and a binary mask marking the missing region; the output is the completed image. Moebius retains the latent diffusion framework: a VAE encodes the masked image, the encoding and downsampled mask form the spatial reference, and a lightweight U-Net performs iterative denoising conditioned on this reference and the LCG semantic prior before VAE decoding.

The two main designs are the Lightweight LλMI Backbone and Adaptive Multi-Granularity Distillation. The former determines the student's computation at each denoising step; the latter uses a frozen PixelHacker teacher during training only. The teacher is not an additional deployment branch. Training also requires complete images to construct diffusion supervision and learn the LCG category distribution; complete images available during training must not be mistaken for inference inputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Masked image and mask"] --> B["VAE encoding and spatial reference"]
    B --> C["Lightweight LλMI Backbone"]
    P["LCG semantic prior"] --> C
    C --> D["Iterative denoising and VAE decoding"]
    D --> E["Completed image"]
    C -.->|Training representations and predictions| F["Adaptive Multi-Granularity Distillation"]
    T["Frozen PixelHacker<br/>and ground-truth supervision"] --> F
    F -.->|Training-only updates| C

Key Designs

1. Lightweight LλMI Backbone: compress within-image interaction and external-prior access together

Compression starts at the macro-architecture level: removing PixelHacker's final downsampling stage reduces the parameter count from 862M to 526M. The method then rebuilds two kinds of computation within the multi-scale U-Net. Depthwise residual blocks (DW.Res) extract spatial features, while interaction blocks cascade Local-λ, Interactive-λ, and Mix-FFN into LλMI, retaining normalization and residual paths. Rather than discarding semantic conditioning or treating every position independently, the design changes how context is summarized and queried.

Local-λ handles within-image interaction. Projections turn latent features into queries, keys, and values. Normalized keys and values are aggregated into a compact content mapping, which each position's query then reads. This avoids explicitly constructing a pairwise attention matrix whose size grows quadratically with the number of positions. A second branch applies a local positional convolution to values to form a position-dependent mapping; queries read the content and positional results, which are added together. The first branch preserves a semantic summary across positions, while the second supplies neighborhood continuity. Thus, Local in the name does not mean that the entire module sees only neighboring pixels. The paper's local window size of 15 applies to the positional aggregation branch.

Interactive-λ handles a different source of information: queries still come from the current image, but keys and values come from LCG embeddings. Because these semantic embeddings have a smaller spatial scale than the image latent features, direct interaction can lack explicit spatial correspondence. A lightweight positional embedding adds layout information to values, producing content and positional mappings for queries to read. Both λ modules summarize before querying; they differ in whether the summarized context comes from the image or an external category prior. LCG therefore remains available after compression through a less expensive cross-interaction mechanism.

Finally, Mix-FFN replaces heavier dense projections with an efficient depthwise-augmented structure, complementing DW.Res in reducing the remaining budget. In the early-checkpoint ablation, adding Mix-FFN to a model that already uses both λ modules and distillation reduces parameters from 274M to 226M. FID changes from 25.86 to 26.43, while LPIPS improves from 0.262 to 0.258. The final architecture is therefore not optimal on every metric; it preserves perceptual quality while reducing parameters further. Local-λ, Interactive-λ, and Mix-FFN in the paper's diagram are sequential substeps within this design, not separate complete models.

2. Adaptive Multi-Granularity Distillation: constrain both low-resolution structure and final latent outputs

Even with suitable interaction mechanisms, the 226M student has a substantial quality gap when trained with the standard prediction loss alone. Moebius therefore uses the officially pretrained PixelHacker as its teacher and constrains both coarse intermediate representations and fine output predictions. Coarse alignment occurs at \(16\times16\): the teacher supplies its representation after the first upsampling block, while the student supplies its representation after the last downsampling block, with element-wise squared-error alignment. The layer indices differ but their spatial scales match. Matching only identical layer indices would overlook the student's removal of a downsampling stage.

Fine alignment occurs at the final \(64\times64\) latent output. The student fits the ground-truth training target and also approaches the teacher prediction through squared error. In addition, E-LatentLPIPS measures perceptual differences directly between the student prediction and the ground-truth target in latent space. This is neither ordinary LPIPS applied after pixel-space decoding nor a separate teacher-feature loss. Avoiding high-resolution decoding along this training path saves memory, but does not eliminate VAE decoding when an image must be produced at test time.

Simply adding these losses leaves a scale mismatch: a larger loss value does not necessarily imply a proportionally larger gradient on student parameters. The method first compares the gradient norms of the task, fine-distillation, and perceptual losses at the student's final output-layer parameters, adjusting the latter two weights accordingly. It then compares the coarse loss and combined output loss at parameters immediately preceding the student's coarse feature to determine a cross-granularity weight. The two balancing levels address different issues—competing objectives at one output layer and supervision strength across spatial scales—and cannot be reduced to one shared empirical constant.

Loss & Training

Let \(\mathcal L_{C\_KD}\) denote coarse representation alignment, \(\mathcal L_{F\_KD}\) teacher–student final-prediction alignment, \(\mathcal L_{task}\) ground-truth supervision, and \(\mathcal L_{perceptual}\) the latent perceptual constraint. The training structure can be summarized as:

\[ \mathcal L_{out}=\mathcal L_{task}+w_F\mathcal L_{F\_KD}+w_P\mathcal L_{perceptual},\qquad \mathcal L_{total}=\mathcal L_{C\_KD}+w_C\mathcal L_{out}. \]

The weights are computed dynamically from gradient norms at the two parameter sets described above rather than treated as fixed hyperparameters. The cached text has damaged formatting in the weight equations, so this note retains only the objective combination cross-checked against the prose and architecture diagram, without reconstructing implementation details of the weight ratios. The source also switches somewhat unclearly between noise-prediction descriptions and \(x_0\) notation. Reproduction requires checking the precise target parameterization rather than interpreting the notation alone as clean-image regression.

Training shares the teacher's LCG embedding size, SDXL VAE encoder, and \(512\times512\) input resolution. Distillation uses Muon, weight decay 0.1, and BF16 on 16 L40S GPUs, with total batch size 768 for 138K iterations. The learning rate starts at \(2\times10^{-4}\) and is multiplied by 0.1 at both 111K and 129K iterations. The model is subsequently fine-tuned separately on Places2, CelebA-HQ, and FFHQ; it is not one unadapted student evaluated zero-shot across every benchmark.

Key Experimental Results

Main Results

FID measures the feature-distribution distance between generated and real images, while LPIPS measures perceptual differences between paired images; lower is better for both. The following table uses clearly readable values from the paper's Table 1 and does not compare scores across different masks or resolutions. Places2 Small uses 36.5K validation images with \(512\times512\) small masks; CelebA-HQ uses 3K images with \(512\times512\) large masks; FFHQ uses 10K images with \(256\times256\) LaMa-style masks.

Model	Places2 Small FID ↓	LPIPS ↓	CelebA-HQ FID ↓	LPIPS ↓	FFHQ FID ↓	LPIPS ↓
Moebius	0.92	0.091	5.39	0.122	8.15	0.231
PixelHacker	0.82	0.088	4.75	0.115	6.35	0.229
SD3.5 Large-Inp.	3.02	0.105	11.80	0.134	109.42	0.402
FLUX.1-Fill-Dev	0.94	0.099	10.13	0.141	11.19	0.268

The teacher remains better than the student on every metric in these three settings. Approaching large generalist models must not be read as outperforming the teacher. On CelebA-HQ, Moebius lowers FID from FLUX's 10.13 to 5.39, an absolute reduction of 4.74, or approximately 46.8% with FLUX as the denominator. The approximately 88% gap marked in the original table uses Moebius as the denominator and is not the conventional percentage reduction.

Model	Parameters (B) ↓	TFLOPs ↓	Single-step latency (ms) ↓	Default steps	Total time (s) ↓
Moebius	0.226	0.154	26.01	20	0.52
PixelHacker	0.862	0.338	46.89	20	0.94
SD3.5 Large-Inp.	8.057	8.657	151.02	28	4.23
FLUX.1-Fill-Dev	11.902	9.927	161.01	50	8.05

Single-step latency is measured consistently on one L40S GPU at batch size 1 and \(512\times512\). Relative to FLUX, Moebius uses approximately 1.90% of the parameters and is about 6.19 times faster per step. Its tabulated total time is about 15.5 times faster, but with 20 versus 50 steps. The total-time values match single-step latency multiplied by the step count; the table does not separately break down encoding, decoding, or other end-to-end overhead, so these figures should not be presented as a guarantee of full product-request latency.

Ablation Study

The following results come from Table 5, evaluated on Places2 Test at \(512\times512\) with 40–50% masks using an 18K training checkpoint. They are not the main results from fully trained, benchmark-specific fine-tuned models above.

Supervision configuration	FID ↓	LPIPS ↓	Note
Coarse distillation only	74.20	0.367	Intermediate alignment alone cannot determine the final output
Add fine distillation	36.17	0.291	Align the teacher's final prediction
Also add task supervision	32.59	0.273	Introduce the ground-truth target
Also add latent perceptual constraint	26.43	0.258	Full multi-granularity objective

Adding the perceptual term last lowers FID by 6.16 and LPIPS by 0.015 in absolute terms. This supports the value of latent perceptual supervision, but the incremental ablation does not independently establish that adaptive weights outperform optimally tuned fixed weights, because that direct comparison is absent.

Key Findings

• Architecture and supervision work together: in Table 2, adding full distillation to the same 226M architecture lowers FID from 33.42 to 26.43 and LPIPS from 0.312 to 0.258.
• Human preference does not show an overwhelming advantage over the teacher: in a forced-choice study with 22 participants and 50 cases per scenario, average selection rates are 31.76% for Moebius, 32.18% for PixelHacker, 23.70% for FLUX, and 12.36% for SD3.5.
• Advantages have protocol boundaries: on Places2 256, Moebius has LPIPS 0.470 versus MI-GAN's 0.394 in the table. The three selected main settings do not imply superiority under every evaluation setup.

Highlights & Insights

• Self-interaction and cross-interaction share the principle of summarizing context before querying it. Compression retains the LCG conditioning interface instead of removing it to gain speed.
• Distillation layers are matched by spatial scale rather than mechanically by layer index. This keeps intermediate supervision structurally meaningful when the teacher is deeper than the student.
• Efficiency reporting separates parameters, computation, single-step latency, and sampling steps. These quantities must be examined individually to distinguish backbone compression from sampling-budget effects.

Limitations & Future Work

• The comparison is between a benchmark-specific fine-tuned inpainting specialist and large zero-shot generalists. It establishes the cost-effectiveness of specialization, not equivalence in general editing or open-domain semantic capabilities.
• A small deployed model does not imply inexpensive training: the method needs a large teacher, multi-GPU distillation, and per-dataset fine-tuning. Inference efficiency should not be equated with a low-cost overall development process.
• The local full text provides no isolated fixed-weight comparison, breakdown of gradient-balancing overhead, or mobile-device measurement. Real-world object removal is supported mainly by qualitative examples and the preference study, leaving coverage incompletely characterized.
• The cached weight equations and target notation have the ambiguities described above. These do not prevent reading the main tables, but they limit precise reproduction from the current text alone.

Related Work & Insights

• vs PixelHacker: Moebius inherits LCG and teacher capabilities while redesigning student depth, convolutions, and interaction blocks. The deployed model becomes smaller and faster, but loses some quality in the main table.
• vs GLA and LambdaNetworks: GLA supplies within-image interaction in the original baseline, while LambdaNetworks provides related background for λ-style context aggregation. The contribution here is adapting both internal and external LCG interactions to a compressed inpainting backbone, not inventing every λ operation from scratch.
• vs diffusion-step distillation: This work primarily transfers architectural capacity; both teacher and student use 20 steps. Its total-speed difference from large generalists cannot be attributed entirely to a new sampling algorithm.

Rating

• Novelty: 3.5/5 — The contribution lies in task-specific lightweight interaction and its synergy with distillation; several basic components have precedents.
• Experimental Thoroughness: 4/5 — Multiple datasets, architecture and loss ablations, and a user study are included, but zero-shot and fine-tuned protocols differ and weight comparisons are incomplete.
• Writing Quality: 3.5/5 — The architecture evolution is clear, though percentage conventions and prediction notation can be misread.
• Value: 4/5 — Offers an explicit quality–efficiency trade-off for resource-constrained inpainting, without directly establishing a replacement for general-purpose models.