EM3M: An Electron Micrograph Dataset for Microstructural Segmentation and Generation¶
Conference: ECCV2026
Paper: ECCV Paper
Dataset: UniParser/EM3M
Generative Model: UniParser/EM3M-Gen
Area: Segmentation (materials microstructure instance segmentation)
Keywords: electron micrographs, instance segmentation, structured descriptions, synthetic data, flow representation
TL;DR¶
EM3M establishes a materials microstructure benchmark with 5,091 real electron micrographs, 2,985,660 instance annotations, and nine-dimensional structured descriptions, while SDXL augmentation and the adapted flow-based UniAIMS baseline raise full-dataset [email protected] from 0.800 with real-data training to 0.809 with mixed training.
Background & Motivation¶
Particle size, shape, count, and spatial distribution influence material properties, so microstructure analysis requires separating adjacent particles rather than merely recognizing image content. Scanning electron microscopy (SEM) and transmission electron microscopy (TEM) reveal fine structures, but imaging modes, surface textures, and material systems vary substantially. A single image can contain both large particles and densely packed small ones, without the distinct object contours common in natural photographs. Expert tracing is expensive, while a model trained for one material rarely covers this heterogeneity directly. The missing ingredient is therefore not just a larger network, but sufficiently dense, reliable, and diverse instance supervision.
Existing public resources leave gaps between image scale, annotation granularity, and scientific descriptions. For example, the Aversa dataset contains 22,000 images but provides classification labels; EMPS has instance annotations but only 465 images and 11,596 instances. MMSci collects image-text pairs from papers for scientific multimodal research, yet publication figures are often edited and their captions may not describe the visible microstructure completely. Acquisition costs, specialist annotation requirements, and intellectual property restrictions also prevent straightforward replication of natural-image crowdsourcing pipelines. Free-form captions further make it difficult for a generator to control particle density, morphology, and surface texture separately.
EM3M addresses these gaps by building instance annotations and structured image-text data together, giving real supervision, controllable generation, and downstream segmentation a shared materials-image foundation. It is primarily a dataset and benchmark contribution; UniAIMS is a strong reference for studying suitable representations, not a new general theory of segmentation. Core Idea: organize real electron micrographs using expert-validated dense instance labels and nine-dimensional attribute descriptions, then supplement segmentation training with domain-adapted synthetic data to evaluate instance representations in crowded microstructures.
Method¶
Overall Architecture¶
The inputs are real images spanning materials and microscopy conditions; dataset construction produces instance masks and image-level structured text as two forms of supervision. Masks support automatic segmentation training and evaluation, while image-text pairs adapt SDXL to generate additional in-domain training images. UniAIMS takes a micrograph, predicts intermediate pixel-level representations including flows, and groups pixels into instances instead of first assigning a box or query to every particle. The generation and segmentation branches share data resources, but they are not a single joint network that must run sequentially at test time. Segmentation inference requires neither text input nor a preceding diffusion-model call; text primarily controls synthetic image attributes. The authors combine data cleaning, expert annotation, controllable synthesis, and segmentation benchmarking to ask whether these resources improve instance recognition on real images. Because the central contribution is a dataset and benchmark, the following explanation uses data flow and evaluation conditions rather than depicting an end-to-end training network not defined by the paper.
Key Designs¶
1. Real images and structured descriptions: support segmentation and conditional generation
Images come from academic collaborations, web collection, and 300 representative images selected from the public Aversa resource and newly annotated. Their final proportions are approximately 84%, 10%, and 6%, so this is not a closed dataset acquired with a single instrument. Curation uses perceptual-hash deduplication and removes low-resolution images, with the paper specifying a threshold below 200px. GPT-assisted quality checks also remove severely blurred images whose embedded text is illegible. Generation receives an additional cleaning step: a trained YOLO11 detects and removes textual metadata regions so that instrument text and scale annotations are not learned as microstructural texture. The main text does not specify whether removal uses cropping, masking, or inpainting, so no particular image-editing algorithm should be inferred.
Descriptions cover nine dimensions: material, microscopy type, color configuration, morphology, density, spatial distribution, Z-axis layering, surface texture, and diameter distribution. Gemini and GPT independently produce attributes in a shared JSON schema, cross-check each other's outputs, and submit all outputs to mandatory human review. Materials experts resolve disagreements, making language models annotation and checking assistants rather than the final source of ground truth. This structure makes particle crowding and surface roughness separately adjustable descriptions instead of implicit information buried in a caption. However, the authors describe these attributes as approximately disentangled; material, morphology, and density should not be treated as statistically independent variables. The text is image-level supervision, not an individual description for each instance or a causal mapping from composition to physical properties.
2. Expert annotation loop: accelerate tracing without letting the model define truth
Materials experts annotate a high-quality seed set, train an initial segmentation model, and use it to produce pseudo-labels for the remaining images. Diffusion-generated diversity augments training to improve the usefulness of subsequent pseudo-labels. Trained annotators correct predictions, peers review the results, and a materials expert performs final validation. The loop reduces the effort of tracing from a blank canvas rather than treating a model's first prediction as a public annotation. In extremely dense images, missing a group of small particles or merging an agglomerate changes both particle counts and size statistics. The authors therefore assess whether human intervention moves instance labels toward independently re-annotated references, not merely whether the images look clear.
The quality audit uses a stratified subset of 300 images independently re-annotated by three annotators. Pixel-level Krippendorff's Alpha (KA) measures annotation agreement, while Panoptic Quality (PQ) combines matched-mask quality with instance recognition quality. In Table 2, PQ against the reference ground truth rises from 0.553 for pseudo-labels to 0.738 after peer review and 0.784 after final validation. KA correspondingly rises from 0.695 to 0.801 and then 0.818, showing that human review is more than a procedural sign-off. These results support a substantial improvement in consistency, not a claim of error-free labels or complete elimination of model bias. The main text does not fully explain how synthetic-image masks are obtained and audited, so SDXL should not be described as automatically producing inherently correct paired instance masks.
3. Attribute-controlled SDXL augmentation: supplement domain samples, not physical measurements
The authors fine-tune SDXL with LoRA and introduce learnable category tokens initialized from rare words, optimizing their embeddings together with adapter weights. Training pairs consist of micrographs with text regions removed and structured descriptions; generation uses attribute prompts in the same format. The prompt orders microscopy type, material, morphology, texture, density, distribution, layering, diameters, and color consistently, giving each condition a stable semantic position. The order of the nine attributes in the descriptive framework is therefore not identical to their order in the actual prompt template. Category tokens capture domain appearance, while attribute words provide interpretable controls over structural variation. This adapts a natural-image generator to an unfamiliar microscopy distribution rather than introducing a new diffusion sampling algorithm.
Synthetic attributes can be sampled according to observed correlations to preserve plausible combinations commonly found in material images. The authors also explore random recombination for previously unseen appearances, explicitly acknowledging that some combinations may be physically implausible. Controllability consequently refers to visual attributes, not verified inverse design of manufacturable materials. Generation is assessed using distributional metrics and downstream real-image segmentation, with the latter offering stronger evidence of augmentation utility than example images alone. The paper also evaluates pretraining on Gen30K followed by fine-tuning with different amounts of real data; Figure 6a shows an advantage over SA1B pretraining. The text extraction does not contain complete discrete values for that plot, so exact gains at individual scales are not estimated from its axes.
4. UniAIMS flow-based baseline: represent many instances through pixels
Detection methods process proposals, while query-based methods face fixed object-query counts and matching overhead, which become costly when an image contains thousands of particles. Cellpose-style methods instead predict pixel-level flows and group pixels into instances without reserving a query slot for every object. This distinction explains the paper's bottom-up comparison group, which includes flow, distance or embedding proxies, and polygon proxies. These approaches avoid part of the training overhead of assigning supervision to many object candidates, although their shape assumptions and post-processing remain different. UniAIMS retains the Cellpose flow formulation and adapts data-scale handling and features to electron micrographs rather than claiming all underlying algorithms as new. Its adjustments remove the built-in size model, introduce scale-robust augmentation, support distributed training with large-resolution inputs, and modernize the backbone.
Removing the size model addresses substantial variation within and across images, where one typical object size cannot summarize the entire scene. Stronger features and scale augmentation target the coexistence of small particles, larger structures, and complex surface texture. Experiments equip bottom-up methods with a SAM-pretrained ViT-B to reduce backbone differences within this comparison group. Flow post-processing can still accumulate errors, and the main text notes occasional complete omission of very large instances, so this representation does not inherently solve every scale problem. The model outputs instance masks, not material chemical classes or a complete three-dimensional reconstruction of occluded structures. Detailed flow construction, size handling, and backbone implementation are deferred to supplementary Sections 8-9, which are absent from the supplied source.
Loss & Training¶
EM3M contains 5,091 images randomly divided into 4,128 training and 963 test images; here 4,128 is a training-set size, not a definition of the conference poster identifier. Detection and query-based baselines are benchmarked only on a sparse subset with fewer than 100 instances per image; bottom-up methods additionally have full-dataset results. Table 4 uses 1K through 4K for real training scales, so its 4K label should not be treated as an exact training-file count. Real-only, generated-only, and mixed training are distinct configurations: Table 3 reports 0.800 for full-dataset UniAIMS, whereas the best mixed configuration reaches 0.809. The main text describes joint adaptation of LoRA and word embeddings but does not provide a reproducible optimizer specification, learning rate, LoRA rank, or complete segmentation loss. This note therefore does not invent a full training recipe or portray each engineering adaptation as independently validated by a component ablation.
The mAP definition also requires care: predictions and ground truth are matched at an IoU threshold of 0.5, the following AP is computed per image, and values are averaged across images.
TP, FP, and FN count true-positive, false-positive, and false-negative instances; this microscopy definition is explicitly given on page 10 and is not COCO precision-recall area. [email protected] combines instance recognition and matched-mask overlap quality, so numerically similar mAP and PQ scores are not interchangeable. The main text does not fully specify when generator training is isolated from the test split; independent reproduction should check that only permitted training data are used, an unresolved audit question rather than demonstrated leakage.
Key Experimental Results¶
Main Results¶
The following selection from Table 3, page 9, uses the paper's [email protected] and [email protected] definitions; "Not evaluated" does not mean a score of zero. The sparse subset contains fewer than 100 instances per image, so differences between sparse and full columns are not changes measured on identical test samples.
| Method | Backbone | Sparse [email protected] | Sparse [email protected] | Full [email protected] | Full [email protected] |
|---|---|---|---|---|---|
| Mask R-CNN | ResNeXt101 | 0.620 | 0.610 | Not evaluated | Not evaluated |
| MaskDINO | Swin-Base | 0.395 | 0.515 | Not evaluated | Not evaluated |
| StarDist | ViT-Base | 0.670 | 0.559 | 0.668 | 0.582 |
| CellViT | ViT-Base | 0.608 | 0.625 | 0.672 | 0.600 |
| Cellpose | ViT-Base | 0.650 | 0.692 | 0.711 | 0.691 |
| UniAIMS | ViT-Base | 0.818 | 0.769 | 0.800 | 0.704 |
UniAIMS exceeds the listed Cellpose baseline by 0.089 in full-dataset mAP but only 0.013 in PQ, showing that instance detection-style scores and fine mask quality improve by different amounts. The table supports the overall configuration but cannot isolate the contributions of the backbone, scale strategy, and training implementation.
Ablation Study¶
Table 4, page 13, analyzes data scale and source rather than removing individual UniAIMS components; selected training configurations below retain evaluation on real images.
| Training data | Reported amount | [email protected] | [email protected] |
|---|---|---|---|
| Real | 1K | 0.767 | 0.682 |
| Real | 3K | 0.795 | 0.700 |
| Real | 4K | 0.800 | 0.704 |
| Generated | 5K | 0.760 | 0.669 |
| Generated | 30K | 0.791 | 0.690 |
| Generated + real | 30K + 4K | 0.809 | 0.707 |
The 0.791 mAP from 30K generated images approaches but remains below 0.795 from 3K real images, so equal per-sample efficiency is not established. Mixed training adds 0.009 mAP and 0.003 PQ over 4K real images, demonstrating supplementary value rather than a dramatic leap in precise segmentation.
Table 6, page 13, analyzes generation quality: lower FID is better, while Style is cosine similarity between pretrained DINO-ViT CLS features, with higher values preferred. Real is a domain reference: its FID compares real images with rotated, scaled, and flipped counterparts, not a distribution against itself with an expected zero score.
| Metric | Real reference | Original SDXL | Fine-tuned model |
|---|---|---|---|
| FID | 28.06 | 213.16 | 34.03 |
| Style | 0.985 | 0.669 | 0.958 |
Key Findings¶
- Domain adaptation substantially improves the visual distribution, but neither ImageNet-feature FID nor DINO style similarity directly establishes physical validity.
- Page 15 reports UniAIMS [email protected] of 0.704 but only 0.340 under stricter [email protected], identifying precise boundaries as a substantial remaining weakness.
- Table 5 gives EMPS zero-shot performance of 0.89, below its reported baseline of 0.92; the prose claim of broadly surpassing existing baselines zero-shot should not be repeated uncritically, and the table header does not clearly name its metric.
Highlights & Insights¶
- The resource combines dense instance supervision with structured text instead of pursuing image count alone. Individual contours for many particles directly support microstructural measurement in ways that a single classification label cannot.
- Independent re-annotation supports the human-in-the-loop workflow rather than relying solely on attractive mask examples. The stage-wise improvement in Table 2 makes human validation empirically inspectable.
- Synthetic data are assessed through both visual distributions and real downstream tasks. These checks complement each other but still distinguish plausible appearance from evidence sufficient for materials-science conclusions.
Limitations & Future Work¶
- The authors acknowledge persistent difficulty with extreme stacking, fibrous textures, weak boundaries, and large scale variation. Stricter overlap thresholds expose errors hidden by permissive matching metrics.
- Attributes have long-tailed distributions, and random image splits do not automatically establish generalization to new laboratories, materials, or instruments. Grouped evaluation by material and acquisition source is a reader-proposed extension.
- The main text relies on supplementary material for synthetic labels, complete training details, and further annotation protocols; the supplied source contains only main-paper pages 1-19. This limits reproduction-level interpretation without preventing verification of the reported main-table values.
- Random attribute recombination can produce physically implausible structures. Expert rules or physical constraints could help validate them, but reliable material inverse design is not demonstrated here.
- Key tables lack variance across repeated runs, leaving uncertainty around the relatively small mixed-training gain. Boundary errors and particle-size statistic errors would complement mAP with more application-relevant measures.
Related Work & Insights¶
- Compared with EMPS and Aversa: EM3M expands both instance supervision and scene complexity; image counts in classification datasets should not be equated with instance segmentation supervision.
- Compared with MMSci: the focus is experimental micrographs and expert-validated structural attributes rather than primarily publication captions; scientific question answering and microstructural measurement have different requirements.
- Compared with Cellpose and CellViT: UniAIMS adapts a flow-based framework to materials and compares it empirically on shared data; it does not introduce pixel proxies or SAM-pretrained backbones for the first time.
- Compared with general SDXL: domain image-text pairs and attribute tokens support augmentation. A transferable approach is to define auditable conditions for specialist imagery and evaluate generation through real downstream tasks.
Rating¶
- Novelty: 4/5. Dense instance labels and structured image-text resources add clear value, while the models mainly adapt established frameworks.
- Experimental Thoroughness: 4/5. Baselines, data scale, generation quality, and annotation reliability are covered, but main-text component ablations and variance reports are missing.
- Writing Quality: 4/5. The data-to-application narrative is clear, although some external-dataset claims exceed table support and important implementation details rely on supplementary material.
- Value: 5/5. The work supplies a scarce high-density public benchmark and reusable generation resources for materials microstructure instance analysis.