CerDETR: Cell-Prior Empowered DETR for Cervical Lesion Detection¶

Conference: ECCV2026
Paper: ECCV Official Page
PDF: Full Paper
Code: https://github.com/imAzhou/CerDETR
Area: Medical Imaging
Keywords: Cervical lesion detection, multiscale cell priors, one-to-many matching, query enhancement, annotation noise

TL;DR¶

CerDETR adds a cell-prior correction branch during DINO training, combining multiscale proposals, overlap-and-containment matching, and category-scale query enhancement to improve cervical lesion detection from 30.9 to 39.1 AP on CDetector, while retaining only the main detection branch at test time.

Background & Motivation¶

Detection boxes in cervical cytology do not always represent objects of consistent size: the same abnormality class may appear as small cells or a large cell cluster, while different lesion classes can look remarkably similar. DCC-MSI strengthens multiscale spatial features, and YOLO detectors emphasize speed. DETR instead matches queries to a target set, using global context and avoiding non-maximum suppression. Yet generic DETR does not inherently resolve these cytological challenges: with limited annotations, query learning can still miss small targets or confuse lesion subtypes.

Annotation granularity adds another difficulty. Some boxes enclose individual cells, whereas others group multiple cells together. A well-positioned nucleus proposal may lie entirely inside a large lesion box but receive a low IoU because their areas differ substantially. Strict one-to-one matching neither fully exploits multiple cellular cues within a lesion nor cleanly distinguishes annotation granularity from poor proposal quality. Rather than abandoning DETR's set prediction, the paper supplies additional training structure about where cells lie, their categories, and their scales.

Existing cell segmentation models provide part of this structure, but their final probability maps are unreliable for cervical cell clusters. The authors observe that clusters with weak probability responses can still have clear internal flow fields. They therefore reconstruct priors from those fields and use a training-only branch to turn them into learning signals for the shared detector. Core idea: turn multiscale cell morphology priors into dense auxiliary supervision during training, while preventing main queries from depending on prior queries, so the benefits remain in detector weights rather than adding prior generation to inference.

Method¶

Overall Architecture¶

The input is a cytological image, and the output consists of lesion categories and bounding boxes. The main branch uses ResNet-50 and DINO: an encoder extracts image features, query selection produces standard queries, and a decoder with a prediction head performs classification and box regression. During training, an additional Prior Corrector branch makes the same decoder and prediction head classify and refine cell priors as well.

The auxiliary path performs Multiscale Prior Generation, One-to-Many ICG Matching, and Prior Query Enhancement in sequence. Its priors are not additional manual nucleus-center annotations, but boxes obtained from an existing Cellpose model and flow-field post-processing. After matching to existing lesion boxes, they enter the decoder with category and scale information. In the diagram, the prior path is training-only, whereas the main-query path exists during both training and inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Cytological image"] --> B["Multiscale Prior Generation"]
    B --> C["One-to-Many ICG Matching"]
    T["Training annotations"] --> C
    C --> D["Prior Query Enhancement"]
    A --> X["Encoded image features<br/>and main queries"]
    D -->|Training only| E["Shared decoder<br/>and prediction head"]
    X --> E
    E -->|Training| F["Joint supervision of<br/>main and auxiliary branches"]
    E -->|Main queries only at inference| G["Lesion categories and boxes"]

Key Designs¶

1. Multiscale Prior Generation: recover cellular structure missed by the probability map

Auto MPG, the Auto Multiscale Prior Generator, changes how Cellpose outputs are used rather than training a new segmentation network. Increasing the target diameter alone does not reliably cover large cell clusters: their probability responses may be near zero even when the predicted pixel flow retains internal structure. Auto MPG therefore sets separate diameters for nuclei, cytoplasm, and clusters and reads the two-dimensional flow toward cell centers, instead of treating the original probability map as the final judge of prior quality.

The flow field serves two purposes: divergence identifies candidate boundaries, while flow magnitude estimates cell-body response. The latter undergoes percentile normalization and Gaussian smoothing, after which responses at boundary locations are set to zero to separate adjacent regions. For nuclei and cytoplasm, segmentation still uses Cellpose's flow tracking; large clusters instead use connected-component analysis. Instance masks are then converted to bounding boxes. This preserves both individual-cell and clustered structures instead of forcing all abnormalities into one scale. The prose identifies absolute divergence as the boundary cue, but the pseudocode uses signed divergence in its threshold comparison. This discrepancy requires checking the implementation; the note does not turn that ambiguity into a definitive equation.

2. One-to-Many ICG Matching: do not reject contained priors solely for limited overlap

The main branch retains one-to-one Hungarian matching, preserving the training constraint of one prediction per target. The auxiliary branch allows multiple priors to match the same ground-truth box, increasing supervision density. For each prior, ICG finds the maximum IoU over ground-truth boxes. A prior meeting the high threshold is assigned to the box with the largest overlap; one below the low threshold is negative. In the intermediate range, containment is checked, and the smallest ground-truth box fully enclosing the prior is selected. The default low and high thresholds are 0.2 and 0.5.

This rule specifically addresses inconsistent cell-versus-cluster annotation granularity rather than merely relaxing matching. IoU alone may undervalue a smaller proposal belonging to a cluster; containment supplies complementary spatial evidence, while choosing the smallest enclosing box avoids unnecessarily broad targets. However, it does not correct erroneous class labels or accept every internal small box: priors below the low threshold remain negative. The main text does not specify a fallback for intermediate-IoU priors with no enclosing ground-truth box, so it does not establish a positive target for every possible prior.

3. Prior Query Enhancement: condition auxiliary queries on position, lesion category, and scale

Prior position alone cannot adequately separate morphologically similar lesion subtypes. Prior Query Enhance forms auxiliary queries by adding sinusoidal box-coordinate encodings to learnable category and scale embeddings. Categories come from ICG assignments and account for non-lesion cases; scale has three bins corresponding to nuclei, cytoplasm, and clusters. Positive priors select a scale bin from the area of their matched ground-truth box, whereas negative priors use their own box area. The scale at which a proposal was generated is therefore not necessarily identical to its final embedding scale label.

Position encoding indicates where to look, category embeddings indicate the morphology to learn, and scale embeddings indicate the spatial extent of relevant evidence. Together they participate in attention over encoded image features, with the shared prediction head learning classification and localization correction. These labels are conditions of a training-only auxiliary task, not answers available at test time. Main queries cannot read auxiliary queries through self-attention; the benefit instead passes through gradient updates to shared parameters. Category- and scale-related attention visualizations provide supporting evidence, but do not prove that every embedding has stable clinical semantics.

A Worked Example¶

Consider a lesion cluster in an image. Auto MPG may produce both smaller internal cell boxes and a box covering the cluster. If an internal box has an IoU between 0.2 and 0.5 with its best-overlap annotation and is fully enclosed by it, ICG can assign supervision through containment without requiring nearly identical outlines. If several ground-truth boxes contain it, the smallest one is selected. Priors below 0.2 do not use this supplementary matching route.

The prior then becomes a query comprising its own position encoding, the matched lesion-category embedding, and the scale embedding determined by the matched ground-truth box. The decoder trains ordinary detection queries alongside this auxiliary query, and auxiliary prediction errors update shared parameters. At test time, comparable images no longer require Cellpose or annotation-conditioned queries; only the DINO main branch with its improved learned features remains. This is an explanation of the mechanism, not a separately reported clinical case result.

Loss & Training¶

Let the main-branch loss be \(\mathcal{L}_s\) and the auxiliary prior-branch loss be \(\mathcal{L}_p\), both including classification and box regression supervision. The textual description of joint optimization can be summarized as:

\[ \mathcal{L}=\mathcal{L}_s+\lambda\mathcal{L}_p. \]

Here, \(\lambda\) controls the weight of auxiliary gradients. This is a shorthand for the prose description, not a verbatim transcription of the corrupted gradient equation in the cache. Hungarian matching determines main-branch targets, while the auxiliary branch uses targets already assigned by ICG. The shared decoder and prediction head receive both gradients, and the self-attention mask prevents main queries from accessing prior queries, avoiding dependence on information unavailable at test time.

The implementation uses MMDetection, with default target diameters of 15, 120, and 240 and 300 auxiliary priors. Training images are preprocessed with Auto MPG, and prior counts are standardized across images for batching. Prior-count handling, several training hyperparameters, and other dataset split details are delegated to an appendix absent from the local cache. Sampling rules, learning rates, loss coefficients, and scale thresholds are therefore not supplied here. Auto MPG being training-free means that the prior generator receives no task-specific training, not that the CerDETR detector requires no training.

Key Experimental Results¶

Main Results¶

AP averages average precision over IoU thresholds from 0.5 to 0.95; AP50 and AP75 fix IoU at 0.5 and 0.75. AR denotes mean average recall. All are reported as percentages, with higher values better. CDetector contains 7,410 images, including 6,666 training and 744 test images, and 11 abnormal classes. CRIC and HMCHH use a single abnormal class, whereas BCCD detects blood cells; these are not equally difficult classification tasks.

Dataset	Metric	DINO	Highest AP among other listed methods	CerDETR	Gain over that AP-best comparator
CDetector	AP ↑	30.9	DCC-MSI: 36.1	39.1	+3.0 percentage points
CRIC	AP ↑	36.8	DINO: 36.8	41.4	+4.6 percentage points
HMCHH	AP ↑	35.5	DEIM: 37.7	41.7	+4.0 percentage points
BCCD	AP ↑	61.3	DEIM: 63.2	65.8	+2.6 percentage points

The first three rows come from the paper's Table 1, and BCCD comes from Table 2. On CDetector, the AP gain over the underlying DINO baseline is 8.2 percentage points, not the 3.0-point gain over DCC-MSI. The comparison combines results adopted from existing papers with additional reproductions where code is available; it is not one fully standardized retraining exercise. Some methods also lack available results, so the best listed comparator does not exhaust all published systems.

For inference efficiency, both CerDETR and DINO report 226.5 GFLOPs and 20.1 FPS, with 47.9M and 47.6M parameters, respectively. CerDETR preserves main-branch throughput, but its parameter count is not identical, and it is not the fastest model: YOLOv12-l reaches 82.7 FPS in the same table. BCCD supports applicability to another cell detection task, not zero-shot transfer across datasets without training.

Ablation Study¶

The following sequential-addition ablation is on CDetector, from the paper's Table 3. All metrics are percentages and higher is better. Gains depend on addition order and should not be treated as independent contributions.

Config	AP ↑	AP50 ↑	AP75 ↑	AR ↑
DINO baseline	30.9	56.8	30.5	59.2
+ Simplified Prior Corrector branch	33.4	60.0	31.8	61.2
+ Multiscale Prior Generation	37.2	66.3	36.7	63.9
+ One-to-Many ICG Matching	38.4	67.5	37.8	65.4
+ Prior Query Enhancement	39.1	68.1	39.8	66.2

The simplified branch uses direct Cellpose priors, IoU matching with a 0.5 threshold, and position-only query encoding. Adding Auto MPG gives a 3.8-point AP gain, the largest step in this sequence; ICG adds 1.2, followed by 0.7 from query enhancement. In a separate prior-replacement experiment with other modules fixed, Table 4 reports 36.7 AP for direct Cellpose and 39.1 for Auto MPG. The background configurations differ: 36.7 must not be confused with the 33.4 result in the sequential ablation.

Key Findings¶

Extra auxiliary training alone does not explain the improvement. The simplified branch already helps, but flow-derived priors further improve AP substantially, identifying prior quality as an important factor.
AR rises from the baseline's 59.2 to 66.2, supporting the value of reduced missed detections. However, AR is not a substitute for patient-level screening sensitivity or clinical benefit assessment.
Increasing prior count adds training cost, and the paper discusses larger prior sets against memory use and precision. The ordering of AP values in that passage is ambiguous, so this note does not quantify that comparison or infer curve values from figure placeholders in the cache.

Highlights & Insights¶

Reusing the flow field instead of trusting segmentation probability maps is the most concrete domain-specific modification. It converts geometric information already present in the segmenter, but underused by its final masks, into detection priors.
One-to-one matching in the main branch supports duplicate-free prediction, while one-to-many matching in the auxiliary branch supplies dense supervision. Their different objectives explain why inference still avoids additional non-maximum suppression.
Category and scale priors serve only as training conditions, combined with attention isolation and parameter sharing. Their value is improving learning with annotation-side information, not making deployment depend on ground-truth categories.

Limitations & Future Work¶

The authors acknowledge mixed annotation granularity, insufficient single-cell labels, and limited dataset scale. More accurate segmentation and cell-level labels could make the currently fixed prior generator learnable, as proposed in the paper.
Multiple datasets still provide offline detection evidence rather than validation of an entire pathology workflow. Whole-slide processing, patient-level decisions, prospective multicenter testing, and clinical error costs are not adequately established by these results.
Reproducibility has limits: the referenced appendix is absent from the cache, several equations have damaged formatting, and ICG fallback behavior and divergence thresholding remain incompletely specified. Removing the auxiliary branch at test time must not be mistaken for eliminating offline prior generation or additional training cost.

vs DINO and Mr. DETR: CerDETR retains query-based detection and training-time enhancement, but ties auxiliary supervision to cell structure, lesion category, and scale rather than adding only generic detection training routes. The comparisons support the value of task-specific priors.
vs Cellpose, SAM, and CellSAM: These models supply priors or serve as replacement controls, rather than functioning as the final lesion detector. Cellpose provides the flow field, and Auto MPG modifies its post-processing to improve cluster coverage beyond direct priors.
vs DCC-MSI and DPD-Net: The former emphasizes multiscale spatial information; the latter needs additional nucleus-center annotations and does not distinguish subtypes in this setting. CerDETR trains multiclass detection using existing box annotations and generated priors, without implying that annotation noise is fully resolved.

Rating¶

Novelty: 4/5 — Cell-flow priors, containment-aware matching, and query enhancement form a targeted combination, although the system builds on existing detection and segmentation architectures.
Experimental Thoroughness: 4/5 — Multiple datasets, broad comparisons, and sequential and replacement ablations are useful, but standardized reproduction and clinical validation remain limited.
Writing Quality: 3/5 — The main pipeline is clear; matching edge cases and the cache's presentation of equations and hyperparameters require further checking.
Value: 4/5 — A practical contribution to cell detection with limited annotations, without running the auxiliary prior path at deployment; clinical value still requires validation.