RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/RT-DETRs/RT-DETRv4
Area: Object Detection
Keywords: real-time detection, knowledge distillation, vision foundation models, deep semantic supervision, gradient modulation

TL;DR

RT-DETRv4 supervises deep AIFI features with a frozen vision foundation model during training and adapts supervision strength to the module's gradient share, raising the L variant from DEIM-L's 54.7 to 55.4 COCO AP while retaining 8.07 ms inference latency.

Background & Motivation

Real-time object detection must identify objects and output their categories and bounding boxes under a restricted compute budget. RT-DETR addresses this requirement through a lightweight backbone, a hybrid encoder, and a set-prediction decoder, but reducing computation also limits high-level semantic representations. Its encoder does not perform expensive self-attention at every scale: it applies intra-scale interaction only to the deepest features and propagates information to other scales through convolutions. The deep representation is therefore more than an isolated feature map; it influences cross-scale fusion, initial query selection, and final detection quality.

Existing detection supervision ultimately comes from category and box errors and must pass through the decoder and fusion module before reaching deep features. This indirect signal can teach a model to distinguish training objects without sufficiently suppressing nuisance information such as background texture. Vision foundation models acquire richer semantics through large-scale pretraining, but replacing a real-time detector's backbone with one may violate deployment budgets. Ordinary distillation is not an automatic solution either: CNN and ViT features differ in spatial structure, channels, and learning objectives, and indiscriminate multi-scale alignment can damage details needed for detection.

The paper retains the inference network and uses a foundation model as a training-only semantic teacher. The central questions thus become where to inject semantics and how strongly to inject them, rather than how large a model to incorporate. The authors motivate the design using the information bottleneck, but the testable evidence comes from injection-position and loss-weight ablations, not direct mutual-information measurements. Core Idea: supervise only the AIFI output that carries global semantics, propagate improvements through existing forward fusion and backward gradients, and regulate distillation strength using the module's gradient share.

Method

Overall Architecture

The input image follows the original detection path through a backbone that produces multi-scale features S3, S4, and S5. Attention-based Intra-scale Feature Interaction (AIFI) applies self-attention to the deepest feature S5 to produce F5; Cross-scale Feature Fusion (CCFF) then propagates deep semantics into multi-scale outputs. These outputs support query selection and decoder predictions, ultimately producing categories and bounding boxes. RT-DETRv4 does not redesign this inference path: it adds two training mechanisms around F5, the Deep Semantic Injector and Gradient-guided Adaptive Modulation.

The same input image also enters a frozen vision foundation model, which produces patch tokens as supervision targets. The Deep Semantic Injector (DSI) reconciles spatial and channel mismatches between teacher and student and computes a position-wise semantic alignment loss. Gradient-guided Adaptive Modulation (GAM) monitors AIFI's share of the total gradient magnitude and adjusts the DSI weight for the next epoch at each epoch boundary. After training, the teacher and projection branch are removed, leaving the original detector forward pass unchanged.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Image["Input image"] --> Backbone["Backbone: S3, S4, S5"]
    Backbone --> AIFI["AIFI: produce F5"]
    Image --> Teacher["Frozen vision foundation model<br/>Training only"]
    AIFI --> Fusion["CCFF, query selection<br/>and decoder"]
    Fusion --> Output["Categories and bounding boxes"]
    AIFI -.-> DSI["Deep Semantic Injector"]
    Teacher -.-> DSI
    DSI -.-> Loss["Weighted alignment loss<br/>and detection loss"]
    Fusion -.-> Loss
    Loss -.-> GAM["Gradient-guided<br/>Adaptive Modulation"]
    GAM -.->|Update weight for next epoch| Loss
    Loss -.->|Back-propagation| AIFI

Solid arrows denote detection or feature-extraction data flow; dashed arrows denote training supervision and weight feedback, while the teacher branch also runs only during training. The feedback in the diagram is a cross-epoch loss-weight update during training, not repeated correction of predictions at inference time. The teacher does not output pseudo bounding boxes: distillation operates in feature space, while ordinary detection labels continue to define the task.

Key Designs

1. Deep Semantic Injector: supervise the source of semantic propagation

F5 is the deep representation after global interaction in AIFI and subsequently participates in cross-scale fusion. Supervising it can directly modify the semantics propagated to other scales without requiring shallow features to imitate the teacher's abstract representation simultaneously. This differs from attaching a distillation head to each of S3, S4, and S5: shallow features still serve localization and texture modeling, so pulling all of them toward the teacher space may introduce conflicts. The paper offers this conflict as an explanation; the position ablation supports the effectiveness of F5-only alignment but does not directly measure gradient conflict to establish the entire causal account.

The teacher is typically a ViT producing a sequence of patch tokens, whereas student F5 is a spatial feature map. DSI first reshapes teacher tokens into a two-dimensional grid and interpolates it to the spatial resolution of the student feature. A lightweight projector transforms the student channel dimension into the teacher's semantic space. The projector serves only training supervision and does not replace the actual feature path into CCFF, so its additional parameters need not be deployed. Table 4 compares a 1x1 convolution, an MLP, and a Linear projector, with the best-performing linear projection selected.

Alignment targets consistency in feature direction at each spatial position rather than exact equality of every channel value. The following simplified notation restates the negative-cosine objective described in Section 3.3: \(U\) is the projected student feature, \(V\) the interpolated teacher feature, and \(H,W\) the aligned grid dimensions.

\[ \mathcal{L}_{\mathrm{DSI}}=-\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}\frac{U_{ij}\cdot V_{ij}}{\lVert U_{ij}\rVert_2\lVert V_{ij}\rVert_2}. \]

The loss emphasizes directional similarity and relaxes constraints on teacher-student feature magnitudes, making it more suitable for cross-architecture alignment than direct mean squared error. The teacher remains frozen, and student gradients are not detached at F5, so DSI updates both AIFI and the upstream backbone. During the forward pass, improved F5 influences other scales through CCFF, combining forward semantic propagation with backward representation improvement. Directly supervising F5 therefore does not mean training only AIFI, and leaving shallow features without direct alignment does not mean they cannot benefit.

2. Gradient-guided Adaptive Modulation: monitor the supervised module rather than loss values alone

A fixed distillation weight is sensitive to training stage: it may be too weak to promote semantic learning early on and too strong relative to detection later. GAM does not weight objectives directly from their scalar loss magnitudes; it observes gradients produced by the current total objective on different components' parameters. At every training step, it records the L1 gradient norm of each component and computes the target module AIFI's share of their sum. Averaging these ratios over an epoch yields the control signal for the next epoch's distillation weight. The following notation follows the textual definitions in Section 3.4, with \(\theta_{\mathrm{AIFI}}\) denoting AIFI parameters and \(\theta_C\) the parameters of component \(C\).

\[ r_t=\frac{\lVert\nabla_{\theta_{\mathrm{AIFI}}}\mathcal{L}_{\mathrm{total}}\rVert_1}{\sum_C\lVert\nabla_{\theta_C}\mathcal{L}_{\mathrm{total}}\rVert_1},\qquad \bar r_e=\frac{1}{T_e}\sum_{t=1}^{T_e}r_t. \]

\(T_e\) is the number of steps in the epoch; this is not the ratio of DSI gradients to detection gradients, because the numerator already includes both objectives' effects on AIFI. GAM defines a target interval \([\rho-\delta,\rho+\delta]\), where \(\rho\) controls the desired share and \(\delta\) the tolerance. If the mean ratio lies inside the interval, the next epoch retains the distillation weight; otherwise, an excessive ratio weakens supervision and an insufficient ratio strengthens it. The authors specify that an out-of-range update targets the farther boundary rather than the midpoint, because not all AIFI gradients originate from DSI. Epoch-level aggregation with a tolerance band avoids reacting immediately to the noise of an individual batch.

The mechanism adjusts supervision strength rather than projecting parameter gradients or removing conflicts component by component. Consequently, balanced optimization describes an empirical control effect, not a guaranteed multi-objective optimum. Equation (14) in the cache is missing comparison symbols and other content; this note retains the update direction and boundary principle supported by readable prose without guessing the piecewise formula. Exact reproduction still requires checking the original PDF or code for the initial weight, target interval, and component accounting; default values cannot be inferred from this extraction alone.

A Worked Example

Consider an image containing a pedestrian and a bicycle; this is an illustrative data-flow scenario, not an additional quantitative case from the paper. The student backbone produces representations at three scales, and AIFI first exchanges global information among the deepest positions. After spatial alignment, the teacher's patch-semantic grid supervises feature directions at corresponding locations in student F5. Detection labels still tell the decoder where the pedestrian and bicycle are; the teacher does not generate these box labels. If AIFI's mean gradient share exceeds the upper bound late in training, GAM reduces the next epoch's DSI weight so that the detection task can continue guiding task adaptation. Deployment on a camera retains only the student: no teacher forward pass, feature alignment, or GAM gradient statistics remain.

Loss & Training

The total training objective combines standard classification, bounding-box regression, and other detection losses with the DSI loss weighted by GAM. DINOv3-ViT-Base performs best in the teacher comparison, which also tests MAE-ViT-B and DINOv2-ViT-Base; this does not establish equal effectiveness for every foundation model. The main S/M/L/X results use 120/90/50/50 epochs respectively, so they should not be described as sharing one training duration. Unless otherwise stated, ablations use 36 epochs; the alignment-loss ablation uses DEIM-M with 90 epochs and 12 EMA epochs. The static-weight comparison uses DEIM-L with 50 epochs and 8 EMA epochs and should be read separately from the short-run ablations. The cache does not provide complete input-size, optimizer, or timing-precision settings, so common defaults are not supplied here.

Key Experimental Results

Main Results

Original Table 1, page 11, COCO val2017; Figure 1 identifies latency measurements as being on a T4 GPU. Parameters and GFLOPs refer to detector inference. The table prioritizes corresponding DEIM scales rather than treating YOLO results obtained under different training protocols as strictly controlled comparisons.

Model	Epochs	Parameters M	GFLOPs	Latency ms	AP
DEIM-S	120	10	25	3.66	49.0
RT-DETRv4-S	120	10	25	3.66	49.8
DEIM-M	90	19	57	5.91	52.7
RT-DETRv4-M	90	19	57	5.91	53.7
DEIM-L	50	31	91	8.07	54.7
RT-DETRv4-L	50	31	91	8.07	55.4
DEIM-X	50	62	202	12.90	56.5
RT-DETRv4-X	50	62	202	12.90	57.0

S/M/L/X improve over their DEIM counterparts by 0.8/1.0/0.7/0.5 AP points; the abstract reports corresponding speeds of 273/169/124/78 FPS. No additional overhead refers to the inference network relative to the same student detector, not zero training cost or identical latency across model scales.

Ablation Study

Original Table 2, page 12; COCO val2017 under the default 36-epoch setting. Gains are relative to each corresponding baseline and must not be mixed with the fully trained results in Table 1.

Detector	Config	AP	AP50	AP75
RT-DETRv2-L	Baseline	52.1	70.2	56.7
RT-DETRv2-L	DSI	52.3	70.4	56.4
RT-DETRv2-L	DSI + GAM	52.6	70.7	56.8
D-FINE-L	Baseline	53.1	70.8	57.4
D-FINE-L	DSI	53.2	70.8	57.7
D-FINE-L	DSI + GAM	53.4	71.1	58.0
DEIM-L	Baseline	53.8	71.4	58.5
DEIM-L	DSI	53.9	71.3	58.8
DEIM-L	DSI + GAM	54.3	71.8	59.0

Original Table 3, page 12; the DEIM-L injection-position ablation follows the default 36-epoch setting.

Injection Position	AP	AP50	AP75
No additional supervision	53.8	71.4	58.5
S3 only	53.7	71.2	58.4
S4 only	53.7	71.3	58.4
S5 only	53.8	71.3	58.5
S3 + S4 + S5	53.7	71.3	58.5
S3 + S4 + S5 + F5	53.8	71.4	58.4
F5 only	54.3	71.8	59.0

Key Findings

• On DEIM-L, DSI alone adds only 0.1 AP, while DSI with GAM yields a total gain of 0.5 AP; GAM adds 0.4 AP on top of DSI, not another 0.5 AP.
• Table 3's clearest result is that F5-only alignment beats hybrid alignment, 54.3 versus 53.8. Supervising more layers does not necessarily improve transfer.
• In Table 7, page 13, the best tested static weight is 20 at 55.1 AP; GAM reaches 55.4 AP, exceeding the best static setting by 0.3 AP.
• Table 6, page 13, reports 53.7 AP for cosine and 52.7 for MSE, but its DEIM-M baseline is 52.5 rather than Table 1's 52.7; retain their respective settings instead of erasing the baseline difference.

Highlights & Insights

• Semantic supervision is placed according to a feature's role in information propagation rather than a generic assumption that multi-layer distillation works. One training branch can thereby affect the backbone, semantic interaction, and subsequent fusion.
• GAM monitors the target module's gradient share instead of comparing losses with different scales and units. Feedback centered on the supervised module is transferable, but the target interval still requires validation for a new task.
• Improvements reside directly in the weights of the original inference graph, with no teacher deployment. This is valuable for already optimized real-time systems but does not automatically guarantee end-to-end throughput on other hardware.

Limitations & Future Work

• The authors discuss extensions to CNN detectors, CLIP, and multiple teachers, but quantitative evidence mainly covers COCO and DETR-family models; these discussions are not completed cross-task evaluations.
• Teacher inference, projection, and gradient statistics still consume training resources. The paper calls the extra training cost small but provides no comparison table for training time or peak memory, so this cannot be quantified as zero cost.
• The information bottleneck is an explanatory framework: mutual information is not directly estimated, and dedicated out-of-domain or corruption-robustness benchmarks are absent. Suppressing nuisance information is not equivalent to demonstrated universal robustness.
• Original Table 5 reports 71.9 AP50 for the DINOv3 configuration, whereas the best configurations in Tables 2, 3, and 4 report 71.8, all at 54.3 AP. The cache does not explain this difference, and this note does not silently reconcile it.
• Extraction damage in Equation (14) and missing complete GAM defaults limit cache-only reproduction. Follow-up work should check the implementation and measure training overhead and cross-domain transfer.

Related Work & Insights

• Compared with RT-DETR / RT-DETRv2 / RT-DETRv3: these provide real-time set-prediction architectures and training refinements; the addition here is external foundation-model semantic supervision, not a new inference decoder.
• Compared with D-FINE / DEIM: they emphasize regression-distribution refinement and matching optimization respectively. This method can build on them, making same-scale DEIM the most direct main-result reference rather than a weaker early-version baseline.
• Compared with FGD / DETRDistill / CrossKD: conventional detector distillation more often relies on a detection teacher, whereas this work uses general frozen VFM features without requiring teacher training on the target detection task.
• Relationship to MAE / DINOv2 / DINOv3: these supply pretrained semantics here rather than serving as deployed detectors. Replacing the teacher is a plausible experiment, but alignment location and supervision strength must be revalidated.

Rating

• Novelty: 4/5. The contribution combines targeted injection and gradient feedback for real-time DETR rather than introducing feature distillation itself.
• Experimental Thoroughness: 4/5. Multiple scales, detectors, and key ablations are covered, but training-resource and cross-domain results are missing.
• Writing Quality: 4/5. The structure and ablation logic are clear, although theoretical motivation must be distinguished from established conclusions; extraction damage is not counted as an authorial writing defect.
• Value: 4/5. Accuracy upgrades without changing the deployment graph have practical value, with gains best judged against matching-setting baselines.