From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Planting¶

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Code: https://github.com/bigD233/D2DF
Area: Model Compression
Keywords: Consistency distillation, privileged information, video object removal, latent pseudo-drafts, temporal masked attention

TL;DR¶

D2DF first trains a multi-step teacher to refine video removal drafts, distills a one-step student using a ground-truth prior visible only to the teacher, and finally replaces the external inpainter with lightweight latent pseudo-drafts; its draft-free version achieves 31.56 dB PSNR on RORD with 1.05 seconds for prior extraction plus denoising, excluding latent encoding and decoding.

Background & Motivation¶

Video object removal must reconstruct the background hidden by an object, rather than generate any plausible video. Methods such as ProPainter propagate textures visible in other frames into missing regions, preserving evidence from the actual scene. However, large occlusions or backgrounds that never become visible can produce blurred or distorted reconstructions. Video diffusion models offer stronger generation capabilities but may insert natural-looking content that does not belong to the scene. Their iterative denoising also makes editing slow.

Instead of asking diffusion to infer the background from an empty region, this paper treats a traditional method's low-quality output as a draft. The draft need not look polished: approximate background layout and color can narrow the refinement problem. This introduces two costs, however: an external inpainter produces the draft, and a diffusion network repeatedly refines it. Even after reducing denoising to one step, draft generation can remain the main bottleneck. Few-step inference and independence from external drafts therefore require separate solutions.

The authors further observe that a teacher conditioned on an imperfect draft provides less stable intermediate trajectories, passing uncertainty to its student. Since training data include the true removal result, the teacher can use it during distillation while the student receives only deployment-available information. Core idea: use a ground-truth-privileged teacher to provide more reliable consistency targets, then compress the external draft into an internal latent prior, removing iterative denoising and standalone draft generation costs in sequence.

Method¶

Overall Architecture¶

The inputs are a video to edit and per-frame object masks; the output is a video with the hidden background reconstructed. Training has three stages: establish the Draft-Guided Teacher D-LDM; apply Prior-Privileged Consistency Distillation (PPCD) to obtain D2DF-DG, a one-step model that still requires an external draft; then introduce Self-Guided Fast Planting (SGFP) and distill D2DF-DF with D2DF-DG as its teacher, removing the external draft dependency.

The diagram shows training progression, not three networks executed for every prediction. Deploying D2DF-DG still requires an initial call to a method such as ProPainter. D2DF-DF instead encodes the masked video, uses SGFP to construct a latent pseudo-draft, and performs one denoising step followed by decoding. The true removal video used during training is not a test input. “Draft-free” means freedom from external pixel-space drafts, not the absence of all priors.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training video and masks<br/>Ground-truth removal result"] --> B["Draft-Guided Teacher"]
    B --> C["Prior-Privileged Consistency Distillation"]
    C --> D["D2DF-DG<br/>One-step draft refinement"]
    D --> E["Self-Guided Fast Planting"]
    E --> F["D2DF-DF<br/>One-step removal without external drafts"]

Key Designs¶

1. Draft-Guided Teacher: turn background generation into evidence-guided refinement

The first stage initializes D-LDM from CogVideoX-5B-I2V. ProPainter produces a low-quality removal result, which is encoded into a draft latent. The teacher also receives the masked-video latent and a noisy latent. In the paper's conditioning construction, the masked-video latent is added to the draft latent, and this sum is concatenated with the noisy latent; the three are not concatenated as independent branches. The teacher is trained with the standard diffusion noise-prediction objective and uses 50 DDIM steps for refinement.

The draft reduces uncertainty about what belongs in the missing region, rather than merely adding another high-quality input. Propagated background structure supplies spatial and semantic anchors, while the large video model repairs propagation distortions and missing details. This stage retains expensive multi-step solving to establish a refinement capability that subsequent students can learn. It does not yet solve deployment efficiency.

2. Prior-Privileged Consistency Distillation: let the teacher see ground truth while the student learns a deployable mapping

Standard consistency distillation starts from neighboring times on the same denoising trajectory and requires student predictions to reach the same clean endpoint. When the teacher sees only a flawed draft, its trajectory can shift with draft errors, producing weak supervision for one-step compression. PPCD changes the teacher's conditioning: during distillation, the latent of the true removal video replaces the teacher's draft. Ground truth is not silently supplied to the student.

The training chain is as follows. Noise is added to the ground-truth latent to obtain a later-time state. The privileged teacher uses a one-step solver to predict an earlier-time state. An exponential moving average (EMA) copy of the student predicts an endpoint target from that earlier state, while the online student predicts the endpoint from the later state. Both the EMA copy and the online student use ordinary draft conditioning; only the teacher providing the trajectory step uses ground-truth conditioning. The student thus learns to approach a more reliable target from an actually available draft, rather than inheriting the teacher's privileged input.

PPCD additionally constrains the student's prediction to approach the true removal result. The two terms discourage deviation from the teacher trajectory and convergence to a self-consistent but incorrect endpoint, respectively. The objective can be summarized with readable notation below: \(\hat z_s\) is the online student's prediction, \(\hat z_{\mathrm{EMA}}\) is the target obtained by applying the EMA copy to the privileged trajectory, and \(z_0\) is the ground-truth latent.

\[ \mathcal{L}_{\mathrm{PPCD}} =\mathbb{E}\!\left[\|\hat z_s-\hat z_{\mathrm{EMA}}\|_2^2\right] +\alpha\,\mathbb{E}\!\left[\|\hat z_s-z_0\|_2^2\right]. \]

This mechanism-level expression is reconstructed from Eq. (9) and its explanation, with locally simplified notation rather than a verbatim transcription. The local PDF text extraction damages operators in several equations; consult the original PDF for exact typesetting. Experiments use \(\alpha=1\). After this stage, D2DF-DG refines in one step but still incurs the cost of generating an external draft.

3. Self-Guided Fast Planting: make internal latents perform the external draft's role

The final stage no longer requires a complete external inpainter to generate a pixel-space video first. SGFP reads the masked-video latent directly, divides each frame's spatial grid into non-overlapping patches, flattens and linearly projects them into tokens, and passes normalized three-dimensional coordinates through a small MLP to obtain positional embeddings. The Temporal Masked Transformer (TMT) therefore knows both the spatial location and the frame of each token.

The central restriction determines which patches may supply information. Every patch can act as a query, but only valid patches from known regions can become keys and values. Queries may additionally be restricted to a spatial neighborhood of radius \(R\) across all time frames, avoiding unconstrained global attention over the entire video. Missing regions can then draw on nearby visible background instead of allowing empty tokens to reinforce invented content. After the Transformer, tokens are projected back into a spatial grid. Reconstruction fills only masked regions; unoccluded regions retain the original input latent.

The paper's notation for valid patches needs verification. It first defines mask value 1 as a known region, but subsequently marks a patch as invalid when its mask mean exceeds the threshold. This conflicts with the stated intention of excluding masked patches. The explanation here follows that explicit mechanism, without treating the inequality as an implementation specification. Reproduction should verify mask polarity against the code rather than infer the filtering direction from this formula alone.

The SGFP pseudo-draft is a coarse latent completion and need not first decode into a polished video. The paper's diagnostic decoding shows blocky textures, but also background colors and structure sufficient to guide subsequent one-step refinement. SGFP and the final generator are trained jointly using PPCD again, now with the trained D2DF-DG as teacher. A separate mean squared error constrains the pseudo-draft toward the ground-truth latent. This preserves the draft's informational role without reinstating the complete ProPainter pipeline.

Loss & Training¶

Training samples come from the RORD and ROVI training splits, yielding approximately 30k samples after extracting 25-frame clips. The base resolution is \(480\times720\). D-LDM uses a learning rate of \(5\times10^{-5}\); PPCD uses \(2\times10^{-5}\), batch size 4, and EMA decay 0.99. Sampling times are selected at intervals of 50 over 1k diffusion timesteps. This is distinct from the teacher's 50 DDIM inference steps.

SGFP uses 4 Transformer layers, spatial radius \(R=2\), patch size \(P=10\), embedding dimension 512, and threshold \(\tau=0.7\). Its parameters use a learning rate of \(10^{-4}\). The third-stage weighting ratio of draft reconstruction loss to PPCD loss is 1:5. On the large-model side, the DiT component undergoes full-parameter supervised fine-tuning. One-step refers to denoising-network calls, not a single operation for the entire system or conversion of the backbone into a small-parameter network.

Key Experimental Results¶

Main Results¶

RORD and ROVI provide training and test data, while VPLM is used for evaluation. All three provide original videos, masks, and true removal results. Higher PSNR indicates lower pixel reconstruction error; lower LPIPS indicates a smaller perceptual distance; lower VFID indicates video feature distributions closer to real videos. VFID should not be equated with framewise motion error alone. The following selection from Table 1 uses the external draft source ProPainter as a common reference, not as an assertion that it is the strongest baseline for every dataset and metric.

Dataset	Method	PSNR ↑ (dB)	VFID ↓	LPIPS ↓
RORD	ProPainter	30.97	0.230	0.1020
RORD	D2DF-DG	32.20	0.251	0.0902
RORD	D2DF-DF	31.56	0.268	0.1001
ROVI	ProPainter	40.86	0.069	0.0263
ROVI	D2DF-DG	42.41	0.062	0.0227
ROVI	D2DF-DF	42.25	0.063	0.0282
VPLM	ProPainter	41.15	0.241	0.0374
VPLM	D2DF-DG	43.31	0.191	0.0337
VPLM	D2DF-DF	43.05	0.196	0.0348

D2DF-DF improves PSNR over ProPainter by 0.59, 1.39, and 1.90 dB, respectively, but does not improve RORD VFID or ROVI LPIPS. “Better on most metrics” must not become “best on every metric.”

Efficiency is measured on an NVIDIA A100 using 25-frame videos at \(480\times720\). The following values come from Table 2. The subtotal includes only prior extraction and denoising.

Method	Prior extraction ↓ (s)	Denoising ↓ (s)	Denoising steps	Subtotal ↓ (s)
ROSE	Not used	45.76	50	45.76
DiffuEraser	2.38	2.85	2	5.13
D2DF-DG	2.38	1.03	1	3.41
D2DF-DF	0.02	1.03	1	1.05

Under this subtotal definition, D2DF-DF is approximately 43.6 times faster than ROSE. D2DF-DG and D2DF-DF both require 1.03 seconds for denoising, so their difference mainly comes from draft generation. The authors separately report complete processing, including latent encoding and decoding, within 10 seconds. The 1.05-second figure is not end-to-end latency.

Ablation Study¶

The table below selects the distillation comparison from Table 4 and the SGFP comparison from Table 5. The former concerns draft-guided refinement; the latter explicitly evaluates the final draft-free approach on RORD. Interpret changes within each group rather than treating both groups as one continuous module-addition sequence.

Group	Config	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓
Distillation objective	Direct diffusion training followed by one-step inference	31.54	0.9192	0.1014
Distillation objective	Standard consistency distillation CD	31.85	0.9296	0.1002
Distillation objective	PPCD	32.20	0.9318	0.0902
SGFP	No additional prior	29.36	0.8920	0.1426
SGFP	Transformer layers	30.73	0.9171	0.1149
SGFP	Add valid-patch filtering	30.78	0.9228	0.1093
SGFP	Add spatial windows	31.47	0.9197	0.1070
SGFP	Add masked-region-only reconstruction	31.56	0.9202	0.1001

Key Findings¶

PPCD improves PSNR over standard CD by 0.35 dB and reduces LPIPS from 0.1002 to 0.0902, supporting the effectiveness of privileged trajectory targets. These results do not theoretically prove that an ideal trajectory must be stable.
SGFP improves PSNR by 2.20 dB over the no-additional-prior baseline. Adding spatial windows improves PSNR but lowers SSIM, and the full version does not achieve the highest SSIM in its group. Component gains are therefore not monotonic across all metrics.
On zero-shot Camera-Bench evaluation, D2DF-DF has a flow warping error of \(0.27\times10^{-4}\), below MiniMax-Remover's \(0.37\times10^{-4}\). Its PSNR is 26.57 dB, however, below the latter's 26.63 dB. Flow warping error measures residual differences after aligning neighboring frames with optical flow; lower is better, and conclusions still need to be metric-specific.

Highlights & Insights¶

Privileged information improves training supervision without entering the student's deployment inputs. Rather than merely replacing a distillation loss, the design explicitly separates target quality from the conditions available to the student.
A latent pseudo-draft does not need to be a separately viewable video. It only needs to provide enough background evidence for one-step refinement, allowing the external inpainter and intermediate pixel output to be omitted.
The speed analysis separates denoising from prior generation. Once denoising takes one step, the previously secondary external-draft cost becomes the bottleneck. SGFP targets that bottleneck rather than repeatedly compressing an already short denoising stage.

Limitations & Future Work¶

The authors show failures under dynamic, complex multi-object occlusions: a retained object hidden by another object may not be fully reconstructed. The conclusion also acknowledges blurring in some one-step results.
The models do not lead on every quality metric, and removing external drafts usually sacrifices some reconstruction quality. Editing requires checking fidelity, perceptual distance, and temporal behavior together, not PSNR or speed alone.
Efficiency evidence comes from one GPU and short clips. It does not establish comprehensive long-video throughput, memory requirements, or latency on consumer hardware.
The local full-text cache contains damaged equation extraction, and the valid-mask definition is internally inconsistent. Reproduction requires checking the original PDF and implementation rather than directly adopting the extracted symbols.

vs ProPainter: ProPainter combines propagation and Transformers to produce pixel-space repairs, which D2DF-DG uses as drafts. D2DF-DF learns a cheaper latent substitute; it does not continue calling ProPainter at test time.
vs ROSE and DiffuEraser: These methods also use diffusion for video inpainting. D2DF focuses on compressing multi-step solving and reducing external-prior overhead. Perceptual quality is not determined by denoising steps alone and must be compared using dataset-specific results.
vs consistency models and LCM: These works establish direct mappings from noisy states to clean endpoints. PPCD adapts this foundation by giving the trajectory teacher a ground-truth prior while the online student and its EMA copy continue learning under ordinary conditions.

Rating¶

Novelty: 4/5. Privileged trajectory distillation and internal pseudo-drafts form a targeted combination, built on existing diffusion and consistency-model foundations.
Experimental Thoroughness: 4/5. Multiple datasets, zero-shot transfer, and stage-specific ablations are covered, but deployment-resource and long-video evidence remain limited.
Writing Quality: 3/5. Stage progression is clear, but mask notation and timing definitions require careful checking.
Value: 4/5. The work offers a practical route to one-step conditional generation and exposes the importance of prior-generation costs.