Human-like Object Grouping in Self-supervised Vision Transformers¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: Segmentation
Keywords: object grouping, self-supervised vision transformer, two-dot paradigm, Gram matrix, human visual perception

TL;DR¶

This paper scales the classical two-dot same/different object judgment paradigm into a human behavioral benchmark of 255 natural COCO scenes and 1,020 trials, reads out same/different decisions and trial-wise reaction times from a simple MLP over two patch features, finds that DINO-family self-supervised ViTs align best (up to 91.9% grouping accuracy), proposes an object-centricity AUC based on ROC analysis of patch affinity maps that correlates strongly with behavioral alignment across nine models (Spearman r=0.950), and finally raises supervised models' grouping accuracy by roughly 8–18 percentage points through Gram matrix distillation.

Background & Motivation¶

Grouping visual input into coherent whole objects is a fundamental problem shared by the brain and by AI systems. Humans bring a whole set of signals to bear: part-whole relationships, Gestalt cues, and semantic priors over object categories. In visual cortex this integration is thought to be mediated by "association fields" — long-range lateral connections in retinotopic maps that link distant points. Existing models of object perception largely implement these cues and treat grouping as the gradual activation of lateral connections, but the corresponding behavioral and modeling experiments have mostly stayed on simple stimuli with clear boundaries, leaving natural scenes, where objects overlap and contours are ill-defined, largely unexamined.

Vision foundation models show a mirror image of the same phenomenon. Transformers trained with self-supervised objectives spontaneously exhibit object segmentation properties: self-attention builds dynamic contextual connections between locations, formally resembling the association fields of cortical maps. Recent large-scale model comparisons show that contour integration also emerges as training data and model scale grow, with the strongest models approaching human-level integration. Yet contour integration is still a low-level Gestalt cue, and it is a weak touchstone for human object perception — whether these emergent object properties actually agree with people in naturalistic scenes had no behavioral answer at all.

The gap persisted because a fine enough yardstick was missing. Prior two-dot experiments were either small (for example, 24 hand-picked images) or used simple stimuli, so they neither covered the hard cases of natural grouping — spanning parts, textures, and occlusions within one object — nor provided trial-by-trial human reaction times for models to align against. On the model side, there was no intrinsic, label-free measure of "how object-centric is this representation" that could be compared across models. Core idea: build a human yardstick for object grouping in natural scenes by scaling up the two-dot behavioral paradigm, map patch features to human judgments and reaction times with a minimal readout, use an affinity-map ROC measure of "object-centricity AUC" to show that object structure in the representation drives behavioral alignment, and prove that this structure can be explicitly injected into supervised models via Gram matrix distillation.

Method¶

Overall Architecture¶

The paper does not train a new vision model; it builds a comparison framework of "behavioral yardstick + representation readout + intrinsic measure + interventional validation". The inputs are natural images from COCO2017 and patch features from nine pretrained vision models (ViT-B and ConvNeXt-B). There are three outputs per model: grouping accuracy at predicting human same/different judgments, the noise-normalized Spearman correlation with trial-wise human reaction times, and the model's own object-centricity AUC. On top of this baseline matrix the work takes one further step and intervenes — distilling the Gram matrix of DINOv3 ViT-B into four supervised models and checking whether all three metrics improve together, which pushes the "object-centric structure ↔ behavioral alignment" relationship from correlation toward causation.

The pipeline has four stages: first, a human behavioral experiment on natural scenes yields 1,020 trials with reaction times as the yardstick; second, patch features are taken from each model's final layer, the features of the two dot patches are concatenated into a trial representation, and a two-layer MLP predicts the judgment and the reaction time; third, a cosine-similarity affinity map is computed for each patch and subjected to ROC analysis against object boundaries, yielding a label-free object-centricity AUC; fourth, a Gram alignment experiment tests whether this intrinsic measure can be changed by a training signal. All four analyses share the same 1,020 trials and the same patch locations, so the three metrics can be compared in one coordinate system.

Key Designs¶

1. Scaling the two-dot paradigm into a natural-scene behavioral benchmark: turning reaction time into a continuous yardstick for grouping difficulty

The logic of the two-dot paradigm is direct: place two dots on an image, ask whether they fall on the same object or two different objects, and the button-press latency reflects how hard the grouping is. The paradigm itself is not new, but it had never been scaled to natural scenes. The authors select images from COCO2017 validation: first those where an object covers the image center (guaranteeing the center dot falls inside a well-defined object boundary), then those that additionally contain a second object overlapping the first. An algorithm then places one peripheral marker for each of the four conditions (same/different × close/far), deliberately matching the distance distributions of the same-object and different-object conditions so that participants cannot answer by distance alone; only images where all four conditions admit valid locations are kept. After manual inspection — removing images where annotation inaccuracies pushed a dot outside object boundaries or where the intended object was visually ambiguous — 288 images remain (32 practice + 256 experimental); one further image is dropped during analysis because its ground truth is ambiguous, leaving 255 images × 4 conditions = 1,020 trials. Seventy-two participants took part, each seeing an image in only one condition (counterbalanced across participants); the center dot appears for 500 ms to mark fixation, the two dots then appear alone for 1,000 ms, and finally the dots are superimposed on the scene and flicker at 5 Hz to keep them visible. Gaze must stay on the center dot throughout; deviating more than 1° of visual angle terminates the trial (7% of trials were excluded this way). At the viewing distance used, the close dot sits roughly 3° and the far dot roughly 6° from the center.

This design works as a yardstick because reaction time in this task really does index grouping difficulty, and the behavioral results reconfirm it: participants are clearly faster when both dots lie on the same object (the classic same-object advantage), and only in the same-object condition does reaction time grow with dot separation — separation barely matters when the dots are on different objects. That interaction ("faster on the same object, with a distance effect specific to the same-object condition") becomes a qualitative signature a model has to reproduce. The paper also catalogues the case-by-case reasons reaction times slow down within a condition — an internal object boundary between the dots, dots on parts with different textures, dots on a narrow part of the object, dots near an object boundary, multiple objects of the same category in the scene — showing that the dataset captures genuinely hard natural grouping cases rather than a set of easy images.

2. Two-patch features plus a two-layer MLP readout: turning "grouping" into a supervised judgment and regression task

To line a model up with human behavior trial by trial, the patch-level representation must first be compressed into a "are these two dots on the same object" decision, plus a continuous prediction of how hard that decision is; inspecting attention maps gives neither trial-wise scores nor anything to correlate with human reaction times. The authors take the patch tokens from the final layer — for ViTs, patch features from the last Transformer layer; for convolutional models, the input is resized so the feature map's spatial resolution matches that of a patch-16 ViT-B, and features from the last convolutional layer are then cut into tokens of the same size. Each trial takes the features at the central and peripheral patch locations and concatenates them in that order (center first, peripheral second) into a trial representation (768-dimensional for ViT-B).

A two-layer MLP head sits on top. In the grouping experiment it is a binary classifier: to give the readout enough training data, the same dot-placement algorithm is run over the COCO2017 training set to generate roughly 30,000 trials, and accuracy is then reported on the 1,020 held-out human behavioral trials. Reaction-time prediction uses a different training protocol — nested cross-validation directly on the 1,020 behavioral trials: an outer 10-fold loop trains on 90% and tests on the held-out 10%; within each outer fold the training data is split 90/10 into train and validation for early stopping and hyper-parameter selection; and 10 random seeds are trained per outer fold and averaged to damp the variance of such a small dataset. Model performance is finally normalized by a human noise ceiling:

\[\rho_{\text{norm}}=\frac{\rho\left(\text{model-predicted RT},\ \text{mean human RT}\right)}{\rho_{\text{ceiling}}}\]

The ceiling is estimated without reference to any model: subjects are randomly split into two equal halves 20 times, the Spearman correlation between the two halves' mean reaction times is computed each time, and the 20 values are averaged. On the model side, the correlation between predicted reaction times and the mean reaction times of each of the 20 subject splits is likewise averaged; the ratio is 1.0 when a model reaches the limit that human self-consistency allows (raw, non-normalized correlations are reported in the supplementary).

The deliberate minimalism of the readout is the point: two MLP layers using only two patches means any difference between models must come from the features themselves rather than from readout capacity. Generating large-scale surrogate trials from the COCO training set avoids overfitting the 1,020 human trials, and noise-ceiling normalization both makes correlations comparable across models and honestly exposes the remaining gap to human behavior.

3. Object-centricity AUC: turning "is this organized by object" into a label-free score via ROC on affinity maps

Explaining behavioral alignment requires quantifying how strongly a model's patch representation is organized by object. The difficulty is that this cannot be supervised with segmentation labels, or one would measure segmentation ability rather than intrinsic representation structure; and inspecting affinity maps for a handful of examples cannot be compared across models. The proposed measure converts the question into a ranking evaluation: the cosine similarity between the central patch's feature and every other patch in the image forms an affinity map (higher means more similar), and the affinity values are then treated as scores for "does this patch belong to this object" in an ROC analysis. At threshold \(\theta\):

\[\mathrm{TPR}(\theta)=\frac{\#\{i:\ i\ \text{inside object},\ s_i\ge\theta\}}{\#\{i:\ i\ \text{inside object}\}},\qquad \mathrm{FPR}(\theta)=\frac{\#\{i:\ i\ \text{outside object},\ s_i\ge\theta\}}{\#\{i:\ i\ \text{outside object}\}}\]

where \(s_i\) is the cosine similarity between the central patch and patch \(i\), and inside/outside is defined by COCO instance annotations. Sweeping the threshold from high to low, within-object patches light up first and outside-object patches later, so TPR rises sharply while FPR is still low. Averaging TPR/FPR over all 1,020 trials yields one ROC curve per model, whose area is the object-centricity AUC (0.5 is chance, 1.0 is perfect organization by object). Concretely, on one example image the DINOv3 ViT-B affinity map has already reached a high TPR at a threshold of 0.42 while FPR stays low, and FPR only starts to rise once TPR approaches saturation — exactly the signature of a strong object-centric signal.

The appeal of this measure is that it relies only on the ranking of patch-feature similarities and needs no object-level supervision, so it applies unchanged to self-supervised, supervised, and reconstruction-based models alike; the ROC form is also insensitive to models differing in overall similarity scale. More importantly, it shares the same trials and patch locations as Design 2, so "behavioral performance of the readout" and "intrinsic object-centricity of the representation" live in one coordinate system — the precondition for the r=0.950 correlation reported below.

4. Gram matrix alignment: distilling object-centricity out of a self-supervised model into supervised ones

The three analyses above all yield correlational evidence: models with stronger object-centricity also align better with behavior, but a confound such as training data scale or overall model quality could be pushing both. Establishing that the Gram structure itself matters requires an intervention. The Gram matrix is defined by taking the cosine similarity between every pair of the \(h\times w\) patch tokens, giving an \((h\times w)\times(h\times w)\) matrix:

\[G_{ij}=\cos\!\left(f_i,\ f_j\right)\]

In representations with strong object structure, patches of the same object are mutually more similar, so the matrix shows block-like structure along the diagonal whose block boundaries roughly follow object boundaries — the phenomenon shown in the paper's first figure.

The intervention takes four supervised models (ViT-B and ConvNeXt-B on IN21k and IN1K) and, while training them on ImageNet classification, distills the Gram matrix of DINOv3 ViT-B as a teacher signal — that is, the trained model's inter-patch similarity structure is pushed toward the teacher's (⚠️ the paper's main text does not state the exact distillation loss; the natural form minimizes the difference between student and teacher Gram matrices, e.g. \(\|G^{(s)}-G^{(t)}\|_F^2\) — refer to the original paper and its supplementary material). The three metrics are then re-measured.

The design is convincing because it changes only the inter-patch correlation structure while holding the classification task fixed: if all three metrics improve together, the similarity structure is what matters. That is what happens — grouping accuracy rises consistently for all four models, object-centricity AUC rises consistently and approaches the level of the DINO-family self-supervised models, and behavioral alignment improves across the board; both Transformer models gain more than the ConvNeXt models on all three metrics, indicating that self-attention is a natural substrate for encoding pairwise patch relations. Notably, even models without large-scale pretraining gain substantially from Gram alignment, implying that alignment can partially compensate for limited training data. The finding also converges with DINOv3's own design, where Gram matrix anchoring was introduced as an explicit training signal to preserve and enhance dense feature quality. The supplementary material reports a CLS-token-only distillation baseline that isolates the Gram structure as the active ingredient.

A Worked Example¶

Walk one representative trial through all four steps. Suppose a cat occupies the center of the image and the peripheral dot sits on the cat's back (same-object, far condition). Behavior: the center dot marks fixation for 500 ms, the two dots appear alone for 1,000 ms, then the dots are superimposed on the scene and flicker; the participant fixates the center dot and answers "same" as quickly as possible. Trials like this are among the slower ones in the same-object/far condition, because the two dots span a stretch of the cat with different texture that sits near the contour. Readout: the model encodes the image into a sequence of patch tokens; the central and peripheral locations each yield a feature vector, which are concatenated and passed to the two-layer MLP, producing a same/different prediction and a scalar usable for reaction-time prediction. Note that the readout sees only two patches — the rest of the cat can influence it only indirectly, through its imprint on those two features. Measure: the central patch's affinity map (cosine similarity to every patch in the image) is treated as a score and swept against the COCO instance mask of the cat as ground truth. If cat-body patches are consistently more similar to the center than background or other objects, the curve bulges toward the top-left and contributes a high AUC. Intervention: if this were a supervised model, its Gram matrix might not show a clean block structure across cat-body patches; after distilling the DINOv3 Gram matrix, the block structure on the same image becomes clearer and the three metrics shift accordingly.

Loss & Training¶

Nothing is trained end to end here; training appears in two places. The first is the behavioral readout: the grouping task trains a two-layer MLP classifier on roughly 30,000 trials automatically generated from the COCO2017 training set and evaluates it on the 1,020 held-out behavioral trials, while the reaction-time task runs nested cross-validation directly on the 1,020 behavioral trials (outer 10 folds, inner 90/10 for early stopping and hyper-parameter selection, 10 random seeds averaged per fold) with the noise-ceiling normalization described above. The second is Gram alignment: the four supervised models align with DINOv3 ViT-B's Gram matrix structure on top of the ImageNet classification objective; ⚠️ the main text does not give the distillation weight, training schedule, or optimizer settings — these are in the supplementary material.

Key Experimental Results¶

Main Results¶

Table 1 of the paper reports grouping accuracy for nine models on the 1,020 held-out behavioral trials — that is, whether the MLP readout over the two patch features reproduces the participants' same/different judgments. Participants' own mean accuracy was 90% (the two numbers are not strictly the same quantity: the human figure includes the participants' own judgment noise and should not be treated as an identical baseline).

Model	Architecture	Training regime	Large dataset	Grouping accuracy
DINOv3 ViT B	Transformer	Self-supervised	Yes	91.9
DINOv2 ViT B	Transformer	Self-supervised	Yes	89.0
DINOv3 ConvNext B	Convolutional	Distilled	Yes	86.7
MAE ViT B	Transformer	Self-supervised	No	80.7
DINO ViT B	Transformer	Self-supervised	No	76.5
IN21k ViT B	Transformer	Supervised	Yes	72.2
IN1K ViT B	Transformer	Supervised	No	70.6
IN21k ConvNext B	Convolutional	Supervised	Yes	67.4
IN1K ConvNext B	Convolutional	Supervised	No	60.0

Ablation Study¶

(a) Intervention effect of Gram matrix alignment: four supervised models additionally distill the Gram matrix of DINOv3 ViT-B during ImageNet classification training, and all three metrics move together. The main text reports only the aggregate improvement across the four models (roughly 8–18 percentage points in grouping accuracy); the per-model post-alignment values appear only as bar charts in Fig. 7, so this note does not speculate about specific numbers (⚠️ refer to the original paper).

Model	Architecture	Grouping accuracy before	Grouping accuracy after	Object-centricity AUC	Behavioral alignment
IN21k ViT B	Transformer	72.2	↑ (within the +8–18 pt range)	↑ approaching DINO-family self-supervised models	↑
IN1K ViT B	Transformer	70.6	↑ (within the +8–18 pt range)	↑ approaching DINO-family self-supervised models	↑
IN21k ConvNext B	Convolutional	67.4	↑ (within the +8–18 pt range)	↑	↑ (smaller gain than Transformers)
IN1K ConvNext B	Convolutional	60.0	↑ (within the +8–18 pt range)	↑	↑ (smaller gain than Transformers)

(b) Where object-centricity comes from (based on DINOv3 ViT-B and on all nine models):

Analysis	Object	Observation
Layer depth	The 12 layers of DINOv3 ViT-B	Object-centricity strengthens with depth; the last layer has the highest AUC (ROC curves sorted by decreasing AUC)
Attention heads	The 12 heads of the last layer	Curves are broadly similar; object structure is spread across all heads rather than concentrated in a few
Cross-model link	Object-centricity AUC vs behavioral alignment over 9 models	Spearman r=0.950, p=0.0001, holding across both Transformer and convolutional architectures

Key Findings¶

Training objective explains more than architecture: among ViT-B models, the self-distilled DINOv2/DINOv3 (89.0/91.9) far exceed the supervised IN1K/IN21k models (70.6/72.2). The paper itself notes the two factors are not fully decoupled (the DINO family also uses larger-scale data and a different training pipeline), so the claim should be read as "the training objective is the main factor" rather than "architecture is irrelevant."
Reconstruction sits in the middle: MAE ViT B (80.7) falls between the DINO family and the supervised models, indicating that reconstruction-based self-supervision does learn some object structure, but less than self-distillation.
Object-centric representations are not exclusive to Transformers: DINOv3 ConvNext B (86.7) far exceeds its supervised convolutional counterparts (67.4 and 60.0), showing that such representations can be obtained in a convolutional architecture through distillation — though self-attention responds more strongly to Gram alignment.
The intrinsic measure predicts behavioral alignment: object-centricity AUC correlates strongly with noise-normalized behavioral correlation across the nine models (r=0.950, p=0.0001), and the relationship spans two architectures and three training objectives, supporting patch-level object-centricity as a general driver of behavioral alignment.
The best model reproduces the human qualitative signature: DINOv3 ViT-B's predicted reaction times also show the same-object advantage and a distance effect confined to the same-object condition, indicating it captures more than accuracy-level structure.
Depth helps, heads do not specialize: object-centricity increases with layer depth, but the last layer's attention heads contribute similarly — at odds with the intuition of a few dedicated "object heads."
A clear ceiling gap remains: the best model still falls short of the human noise ceiling, which the authors leave as room for future models.

Highlights & Insights¶

Swapping "segmentation accuracy" for "human consistency" as the criterion: no segmenter is trained and no mIoU is reported; representations are evaluated by trial-wise reaction-time correlation, which answers "does the model group objects like a person" rather than only "can it separate objects." The approach transfers to any study contrasting representation properties with human perception — depth, material, affordance judgments, and so on.
Object-centricity AUC is a label-free, architecture-agnostic diagnostic: it needs only patch similarities and existing instance annotations for one ROC pass, so it scores any backbone far more cheaply and comparably than attention-map visualization or downstream segmentation fine-tuning. It works well as a dense-feature-quality probe when choosing a backbone.
Noise-ceiling normalization makes behavioral alignment comparable: estimating a measurement ceiling from split-half subject consistency and dividing it out looks like a detail, but it is the necessary step that makes correlations comparable across models in small behavioral datasets, preventing measurement noise from being counted as model difference.
The "correlation plus intervention" structure is the paper's strongest argument: first showing that the intrinsic measure correlates strongly with behavioral alignment, then providing directional evidence through Gram distillation, and finally converging with DINOv3's own Gram anchoring design — two independent lines of evidence pointing at one mechanism is far more credible than an observational analysis alone.
A counter-intuitive closing point: the authors state that perfect object-centricity is not necessarily the goal — a representation organized entirely around object boundaries would resemble a semantic segmentation map and lose the fine-grained feature structure that supports other visual tasks. The most behaviorally aligned models may be those balancing object-level organization against feature sensitivity, a useful caution against the instinct to push every capability toward objectness.

Limitations & Future Work¶

Training objective and data scale are not fully decoupled: the DINO-family models are all trained with self-distillation on large-scale data, while the supervised baselines use DeiT3 on IN1K/IN21K; the two groups differ in data scale, training length, augmentation policy, and more. The paper acknowledges this, so "the training objective matters more than architecture" holds within this model set but still needs a stricter single-variable comparison to become a general rule.
Behavioral data are limited in scale and population: 72 undergraduates participating for course credit, 255 images, 1,020 trials — a sample that is homogeneous in age and visual experience, with no reported robustness across populations or cultures.
Ground truth depends on COCO instance annotation quality: the TPR/FPR of object-centricity AUC are entirely defined by instance masks, so imprecise annotation boundaries become measurement noise. The authors removed images with ambiguous ground truth, but residual error on fuzzy-boundary objects (fur, transparency, thin structures) is not separately quantified.
Both the readout and the measure use only the final layer: the layer and head analyses are run only on DINOv3 ViT-B, and the behavioral readout always uses the last layer; whether cross-layer concatenation, additional context patches, or attention weights instead of token similarity would change the model ordering has not been systematically explored.
Distillation details are opaque: the main text omits the exact Gram distillation form and weight, and the CLS-token baseline lives only in the supplementary material, so reproduction requires the supplementary.
Directions to extend: turning trial-wise reaction-time prediction into per-subject prediction (only mean reaction times are used now) would test whether models also capture individual differences; applying Gram alignment to larger models and more downstream tasks would show whether pushing object-centricity up costs anything on fine-grained recognition or dense prediction — a direct test of the authors' conjecture that more object-centricity is not always better.

vs the original two-dot work (Korjoukov et al., 2012): they used 24 hand-picked images to study the time course of grouping in natural scenes, drawing psychological conclusions; this paper keeps the paradigm but scales it to 255 natural images and 1,020 trials and uses it for the first time as a model evaluation yardstick — the difference is scale and purpose (a psychological phenomenon → a behavioral alignment benchmark).
vs LOST / TokenCut (unsupervised object discovery): they also exploit patch similarity graphs, but aim to produce object discoveries or segmentations and are scored on segmentation metrics; this paper does not produce masks, instead using the same similarity structure to predict trial-wise human judgments and reaction times and defining it as a cross-model intrinsic measure.
vs analyses of the emergent segmentation properties of DINO: that line mainly inspects attention maps for objectness; this paper uses cosine similarity between token features (not attention weights) for both the readout and the measure, and adds external-validity evidence at the behavioral level — a pretty attention map does not imply agreement with human judgments.
vs model-human comparison on contour integration (Lonnqvist et al., 2025): that line compares models and humans on low-level Gestalt cues and finds the strongest models approaching human level; this paper moves the question to object grouping in natural scenes and answers whether those properties still hold when objects overlap and contours are ill-defined.
vs Gram anchoring (DINOv3 itself): DINOv3 introduces Gram matrix anchoring as a training signal to preserve dense feature quality; this paper runs the reverse test — distilling DINOv3's Gram matrix into supervised models also improves object-centricity and behavioral alignment, so two lines of evidence converge on the same mechanism from opposite directions.

Rating¶

Novelty: ⭐⭐⭐⭐ Both the behavioral benchmark and the object-centricity measure are new, but the two-dot paradigm and Gram analysis each have precedents; the contribution is connecting them to the model-human alignment question for the first time.
Experimental Thoroughness: ⭐⭐⭐⭐ Nine models spanning two architectures and three training objectives, with correlational analysis and an interventional experiment complementing each other; weaknesses are the un-decoupled objective and data scale, the limited participant pool, and details relegated to the supplementary material.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation chain is clear, the definitions and normalization of the three metrics are stated cleanly, and the conclusion is honest about its own constraints (object-centricity is not always better, architecture and objective are not fully decoupled).
Value: ⭐⭐⭐⭐ It provides a reusable behavioral benchmark and a label-free diagnostic metric and pushes the "Gram structure" mechanism to the causal level, valuable for both self-supervised representation design and perceptually aligned evaluation.