Bottom-up modeling of repeated elements via single image analysis-by-synthesis

Conference: ECCV2026
Paper: ECCV Official Page
PDF: Full Paper
Authors: Syrine Kalleli, Alexei A. Efros, Mathieu Aubry
Area: Self-Supervised Learning
Keywords: Repeated element discovery, analysis-by-synthesis, object-centric learning, low-dimensional appearance prototypes, image decomposition

TL;DR

A shared two-dimensional prototype with adjustable appearance, single-image crop-based curriculum optimization, and full-image selection and refinement turn repeated-element discovery into interpretable reconstruction; on 116 curated FSC-147 images, the method achieves an image PSNR of 23.76 and a counting MAE of 14.11, without implying that the entire system is free of pretraining or scale priors.

Background & Motivation

Discovering what repeats within an image involves more than counting: it also requires identifying the appearance shared by instances and explaining their different positions and colors. Counting networks can return counts, boxes, or masks, but usually do not produce an object model that can resynthesize individual instances. Single-image generative models can capture recurring local texture statistics without decomposing the scene into objects whose rotation, color, and shape can be edited independently.

Explicit prototypes offer another route: copy and transform an object template onto a canvas, and judge the decomposition through reconstruction error. A fixed template, however, struggles with differences in pose, color, and fine appearance. Giving every instance an independent generator creates the opposite problem: arbitrary colored blobs may suffice to fit the pixels. Requiring a genuine three-dimensional model is not a free solution either, because related methods depend on relatively clean instance segmentations, and the available viewpoint variation may not support reliable geometry recovery. This paper deliberately models two-dimensional image-space elements, targeting discovery and controllable decomposition rather than geometry reconstruction.

The challenge is to retain enough shared information while allowing constrained instance differences, and to prevent background regions or object parts from becoming convenient repeated elements. Core Idea: combine shared appearance with one-dimensional instance variation into a tunable prototype, then use crop augmentation and staged reconstruction to make it explain repeated instances across locations instead of memorizing the entire image.

Method

Overall Architecture

The input is one image containing repeated instances of the same element. Outputs comprise the background, a shared element model, and each instance's existence probability, position, rotation, scale, color, and appearance parameters. Synthetic repeated images first provide initialization; a low-dimensional element model then undergoes curriculum optimization on random target-image crops, followed by merging, deduplication, and full-image refinement of sliding-window predictions. The final representation remains a composition of two-dimensional RGBA layers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synthetic repeated images"] --> B["Synthetic Localization Pretraining"]
    B --> C["Low-Dimensional Variable Prototype"]
    C --> D["Crop-Based Curriculum Optimization"]
    G["Single target image"] --> D
    D --> E["Full-Image Selection and Refinement"]
    E --> F["Repeated-instance parameters<br/>and two-dimensional reconstruction"]

Here, “bottom-up” primarily describes how elements are discovered in the target image, not learning from a randomly initialized network without external knowledge. The implementation uses a frozen DINOv3 backbone, synthetic pretraining data constructed from COCO images and segmentations, and a coarse scale prior obtained from exemplar boxes. Default single-image optimization does not add exemplar reconstruction or location supervision losses.

Key Designs

1. Synthetic Localization Pretraining: learn repeated locations before adapting detailed appearance

Jointly discovering locations and learning shapes directly from one real image can leave optimization stuck explaining the scene with colored blobs. The authors first choose a segmented COCO object and paste it repeatedly onto a uniform or heavily blurred background, varying each copy's position, scale, rotation, and color. This controls task difficulty and yields boxes and location labels from the synthesis process. Pretraining primarily supplies coarse localization rather than a detailed object library that transfers directly to real images.

The implementation extracts patch features with a frozen ViT-B/16 DINOv3 and projects them to 64 dimensions. The element decoder uses 60 instance queries and one global appearance query, processed by 3 cross-attention/self-attention blocks. Its pretraining set contains 5,000 synthetic images, with 500k iterations on uniform backgrounds followed by 500k iterations on an equal mixture of uniform and blurred backgrounds. Reconstruction, exemplar, and location supervision are all used. Consequently, “no explicit object-label losses during default target-image optimization” should not be expanded into “an entirely unsupervised training pipeline.”

2. Low-Dimensional Variable Prototype: share an appearance family rather than a fixed sprite

The global appearance query summarizes the common appearance of repeated elements in the current image. Each instance query feeds four heads: selection predicts existence probability; spatial prediction provides scale, rotation, and translation; color prediction supplies a linear RGB transformation; and appearance prediction supplies a one-dimensional code. The generator combines shared information with the instance's appearance code to produce an RGBA element, followed by spatial and color transformations. Small shape differences can therefore be explained without forcing them entirely into geometric transformations or color correction.

The one-dimensional code constrains flexibility: individual instances cannot acquire arbitrarily expressive generators and must still reuse common object structure. Each element's opacity is weighted by its existence probability, and layers are composited in a fixed order over the background. In the implementation, the background branch predicts a background color through global average pooling and an MLP. Prototype resolution is \(64\times64\), network input resolution is \(224\times224\), and reconstruction resolution is \(128\times128\). The spatial head predicts the inverse transform passed to the spatial transformer network, which the authors found more stable. The representation can absorb some appearance changes caused by three-dimensional pose, but it neither outputs three-dimensional shape nor guarantees three-dimensional consistency.

3. Crop-Based Curriculum Optimization: make the shared prototype explain different local views

Minimizing whole-image pixel error alone can produce correctly positioned blobs with meaningless shapes. Repeated random cropping and rotation require a shared element model to reconstruct objects under different local views, reducing the opportunity to memorize the entire image. A coarse scale prior determines the crop range, making the shorter object side approximately 10%–50% of the crop side. The default prior comes from dataset exemplar boxes, so annotated scale remains an input channel rather than this being a strictly prior-free setting.

Optimization order is also constrained. First, only the background is trained, preventing the element model from immediately absorbing it. Next, the remaining trainable components are optimized while appearance-head outputs are fixed at zero, forcing a common prototype across instances. Finally, instance appearance variation is enabled to explain differences around the established object structure. The stages run for 100, 500, and 2500 iterations, respectively. More flexibility is not necessarily helpful earlier: if appearance codes can immediately alter position, color, or shape, the network may bypass the dedicated spatial and color parameters and undermine interpretability.

4. Full-Image Selection and Refinement: merge instances by reconstruction contribution, not confidence alone

After crop-based training, the fixed query predictions cannot simply be applied to the full image because object scale and count can exceed the range of an individual crop. The method uses a sliding window with the average crop size and a stride of 25% of the window size, mapping predicted element transforms into global coordinates. Thus, the 60 queries limit capacity per crop, not the final whole-image object count.

Overlapping windows can predict the same instance repeatedly. Candidates with existence probability below 0.5 are discarded, and the remainder are considered in descending probability order. A candidate is retained only if it reduces reconstruction loss by more than \(10^{-5}\) and has overlap below 0.1 with previously selected elements. The main text provides this overlap threshold without explicitly defining the overlap formula there; it should not be silently relabeled as a particular IoU measure. The selected instance set is then fixed, and its parameters, the averaged background prediction, and the prototype generator are jointly refined for 200 iterations. Selection determines which instances are worth retaining; refinement determines how those instances can better explain the image.

Loss & Training

The default single-image objective combines pixelwise Huber reconstruction error, a structural similarity term, and instance sparsity regularization. The following consolidates the surrounding prose and definitions in equations (1)–(4), whose layout is corrupted in the text cache:

\[ \mathcal{L}_{\mathrm{unsup}}=\mathcal{L}_{\mathrm{Huber}}+\lambda_{\mathrm{SSIM}}\bigl(1-\mathrm{SSIM}(I,\hat I)\bigr)+\lambda_{\mathrm{reg}}\frac{1}{K}\sum_{k=1}^{K}|p_k|. \]

Here, \(I\) and \(\hat I\) are the input and reconstructed images, \(K\) is the number of element queries, and \(p_k\) is existence probability. The Huber term averages over pixels, penalizing small residuals quadratically and larger residuals linearly. Sparsity prevents the model from adding elements at no cost to paint over the background. The weights are \(\lambda_{\mathrm{SSIM}}=0.01\) and \(\lambda_{\mathrm{reg}}=0.001\), with a Huber threshold of 1. Crop optimization uses Adam with learning rate \(10^{-3}\), batch size 64, and 500 iterations of linear warmup; refinement uses learning rate \(10^{-4}\).

Optional supervision is not part of the default model. The exemplar loss requires a single element that best fits an exemplar box to reconstruct that region with the background, while raising its existence probability. This discourages splitting one object into multiple templates. The location loss uses Hungarian matching to associate predicted centers with location labels, constrains matched positions, and raises matched-instance probabilities while suppressing unmatched ones. Sparsity regularization is removed when this loss is enabled. The main text describes location pseudo-labels from a counting network as an input to this branch, but the numerical rows in Table 3 do not separately identify their provenance. Those results are therefore described conservatively as variants with additional supervision.

Key Experimental Results

Main Results

The evaluation does not use the entire FSC-147 test set. The authors select 4 images from each of 29 test categories, totaling 116 images, favoring simple backgrounds and diverse object poses and counts. Each image has a count and three exemplar boxes; TPC-268 provides qualitative prototype examples only.

“Prototype quality” is not a comparison against a ground-truth template. For each exemplar box, evaluation selects the single predicted element with the greatest mask overlap, composites it over a uniform background, and measures the resulting exemplar reconstruction. Background color is the ground-truth image's average color within the predicted background mask; no ground-truth segmentation mask is needed. Full-image evaluation similarly composites all predicted elements over a uniform background computed this way, rather than directly evaluating the unprocessed output of the original background branch. Higher PSNR and SSIM and lower LPIPS are better. Counting MAE and RMSE measure mean absolute error and root mean squared error, respectively, and are lower-is-better metrics.

Method	Counting MAE ↓	Counting RMSE ↓	Image PSNR ↑	Image LPIPS ↓	Prototype PSNR ↑	Prototype LPIPS ↓
ABC123	20.92	28.58	—	—	—	—
GeCo	7.34	13.66	—	—	—	—
TMR	4.11	6.82	—	—	—	—
TMR + SAM + Average	—	—	13.95	0.543	17.44	0.464
TMR + SpaceJAM	—	—	13.63	0.544	17.77	0.430
TMR + SAM + 3D	—	—	—	—	15.53	0.500
Ours, default model	14.11	26.93	23.76	0.242	19.40	0.353

Values are from the paper's Table 2. Relative to TMR + SpaceJAM, the method gains 10.13 dB in image PSNR and 1.63 dB in prototype PSNR, but it is not the best counter: TMR has lower MAE. Modeling baselines use TMR boxes, and some also use SAM masks, thus relying on a supervised counter and exemplar inputs. The default proposed model has the pretraining and scale prior described above, so the comparison should not be presented as using identical information.

ABC123 requires no test exemplar. The evaluation additionally uses the ground-truth count to choose the lowest-error option among its four predicted counts and their sum. Even under this protocol favorable to ABC123, the proposed method has 6.81 lower MAE. This finding concerns the curated image set, not a general counting leaderboard.

Ablation Study

Config	Counting MAE ↓	Image PSNR ↑	Image LPIPS ↓	Prototype LPIPS ↓
Full model	14.11	23.76	0.242	0.353
Without appearance head	14.99	20.58	0.384	0.432
Without sparsity regularization	15.62	22.57	0.298	0.389
Direct full-image optimization	27.54	15.84	0.493	0.490
Without synthetic pretraining	18.81	21.08	0.330	0.469
Without crop-based optimization	15.25	17.45	0.511	0.496
Without full-image refinement	14.11	20.07	0.443	0.468
With exemplar and location losses	7.07	20.00	0.343	0.330

Values are from Table 3. “Direct full-image optimization” removes both pretraining and crop-based training rather than merely changing an optimizer. The last row changes supervision conditions and is not a free improvement of the default setting.

Key Findings

• The appearance head primarily improves modeling rather than count accuracy. Removing it increases MAE by only 0.88, but lowers image PSNR by 3.18 dB and worsens prototype LPIPS from 0.353 to 0.432. Low-dimensional variation around the shared template does account for instance appearance differences.
• Crop-based optimization is particularly important for modeling: removing it lowers image PSNR by 6.31 dB. Refinement does not change the selected instance count, so its ablation retains MAE 14.11 while substantially degrading image quality.
• Additional location constraints lower MAE to 7.07 but reduce image PSNR from 23.76 to 20.00. The authors attribute this to fewer elements being used to fit the background, illustrating that better pixel reconstruction does not automatically mean more accurate object discovery.
• A uniform scale prior increases MAE by 4.53 and RMSE by 6.1. On 90% of the images, MAE increases by only 1.3; the remaining 10%, containing the smallest objects, accounts for most of the gap. The coarse scale prior matters most for very small objects.

Highlights & Insights

• One-dimensional instance appearance adds necessary variation to a fixed prototype rather than maximizing generative capacity. Limited flexibility helps keep position, color, and shape separately interpretable and editable.
• Requiring candidates to improve reconstruction makes sliding-window deduplication more than geometric suppression. It also exposes the limits of reconstruction-driven discovery: spurious objects that help explain the background may still be rewarded.
• Single-element exemplar reconstruction prevents several fragments from collectively masquerading as one complete object. It is closer to the intended object-level representation than full-image reconstruction scores alone.

Limitations & Future Work

• The authors assume non-cluttered backgrounds and moderate object aspect ratios, while the coarse scale prior limits extreme scale variation. Semantically related but visually dissimilar objects may not be grouped as one element type.
• Part–whole ambiguity remains unresolved: a symmetric object may be interpreted as two repeated parts. Exemplar and location supervision help, but change the input conditions.
• The small curated test subset favors the reconstruction assumptions. It does not establish broad superiority on cluttered scenes, the full FSC-147 dataset, or quantitative cross-dataset evaluation. The main paper does not provide complete per-image runtime for this method, so a slow three-dimensional baseline cannot establish its actual speedup factor.
• “Single-image learning” still leverages pretrained representations, synthetic localization training, and scale information. Further evaluation should separate these sources of support and broaden coverage of difficult backgrounds and tiny objects, rather than equating bottom-up optimization with an absence of priors.

Related Work & Insights

• vs DTI-Sprites / MarioNette: These also compose scenes from transformable image-space objects, but their prototypes are fixed, and their settings mainly rely on multiple images or grid-based representations. This paper emphasizes single-image adaptation, instance appearance variation, and the optimization mechanisms needed for them.
• vs SpaceJAM: Joint alignment normalizes already extracted instances into a common template but does not directly locate instances within one image. The proposed method discovers and models instances jointly and explicitly varies color and appearance. Its baseline comparison first extracts objects with TMR, an important prerequisite.
• vs Seeing a Rose in Five Thousand Ways: That method learns parametric three-dimensional models from object segmentations, with different geometric goals and input requirements. Retaining a two-dimensional representation reduces dependence on viewpoint diversity and mask quality, but gives up three-dimensional recovery.
• vs ABC123 / TMR: Counters output counts or detections, whereas this method also produces an element model that can regenerate and edit instances. Counting error and object-model quality are complementary objectives, neither of which fully substitutes for the other.

Rating

• Novelty: 4/5. Combining low-dimensional variable prototypes with a single-image crop curriculum yields an explicitly interpretable representation of real repeated elements.
• Experimental Thoroughness: 3/5. Counting, image reconstruction, prototype evaluation, and optimization ablations are substantial, but data selection, supervision sources, and practical runtime need broader analysis.
• Writing Quality: 4/5. Representation and optimization steps are clear, and qualitative decompositions are useful; broad prior-free claims require careful reading alongside the pretraining and scale settings.
• Value: 4/5. Useful for object-centric modeling of repetition, with its main value in interpretable decomposition rather than state-of-the-art counting accuracy.