ATOMIC: A Domain-Specific Vision-Language Model for Transmission Electron Microscopy¶

Conference: ECCV2026
Paper: Official paper page / PDF
Code: https://github.com/SemiMIRTLab/ATOMIC
Area: Multimodal VLM
Keywords: Transmission electron microscopy, domain adaptation, literature mining, visual grounding, instruction tuning

TL;DR¶

ATOMIC retains the LLaVA-v1.5 architecture, converts published TEM subimages and captions into complementary visual and domain supervision, and performs two-stage adaptation, reaching 75.2% TEM-MCQ accuracy and 37.5% TEM-VQA Answer Word Coverage with its 7B model, without yet replacing quantitative microscopy analysis.

Background & Motivation¶

Grayscale textures, lattice fringes, diffraction spots, and scanning transmission contrast differ substantially from the object appearances found in natural images. A general-purpose vision-language model may describe scattered dark dots but interpret a nanoparticle micrograph as a snowy landscape or ordinary dirt. Even recognizing an electron micrograph does not guarantee an ability to connect local structures to crystallography, defects, or experimental methodology. Existing denoising, detection, and segmentation models handle specific tasks but do not naturally answer a researcher's successive interpretation questions.

The bottleneck is not simply a shortage of images: different questions require different sources of knowledge. Particle dispersion may be observable directly, whereas a material name, processing condition, or experimental context may appear only in an expert caption. Image-only target generation can lack terminology and context; caption-dependent generation can instead teach a model to repeat information that is not visually observable. The paper therefore treats materials-science literature as an existing source of expert knowledge and asks how supervision provenance changes visual judgment and scientific expression without changing the architecture.

Core idea: generate separate training targets for what the image shows and what the caption supplies as scientific context, then combine them for connector alignment and multi-turn instruction tuning so that a locally deployable model learns both visual evidence and domain language.

Method¶

Overall Architecture¶

ATOMIC uses a CLIP-ViT-L-336px vision encoder, an MLP connector, and a Vicuna-7B-v1.5 language model, with a 13B variant also evaluated. The contribution lies in data organization and training, not a new vision backbone: images become visual tokens, the connector projects them into language space, and the language model generates an answer conditioned on the question. The vision encoder remains frozen throughout both adaptation stages.

The offline pipeline first extracts TEM subimages from compound publication figures, then uses GPT-4.1 to generate supervision under VisionGround and DomainContext conditions. Stage 1 trains only the connector using brief and detailed descriptions; Stage 2 jointly trains the connector and language model on multi-turn conversations derived from detailed descriptions. Blend combines both supervision sources. Inference takes only the TEM image and prompt, not the parent caption. Optional downstream task fine-tuning produces a separately identified +FT variant, which must not be conflated with the base Blend model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Compound figures<br/>and captions"] --> Mining["Literature Subimage Curation"]
    Mining --> Supervision["Dual-Source Supervision<br/>VisionGround / DomainContext"]
    Supervision --> Alignment["Connector Concept Alignment<br/>Brief + Detail / Blend"]
    Alignment --> Conversation["Multi-Turn Instruction Adaptation<br/>Detail to dialogue / Blend"]
    Conversation --> Output["Image and prompt input<br/>Answer or optional task fine-tuning"]

Key Designs¶

1. Literature Subimage Curation: make sure the supervision actually concerns TEM images

The authors extract 34,073 figure-caption pairs from Nature, Nature Communications, Nature Materials, and Nature Nanotechnology. Publication figures often combine electron micrographs with plots and schematics; generating questions from an entire figure can mistake a plotted conclusion or a non-TEM panel for microscopy evidence. YOLOv11 first locates candidate subfigures, and ResNet-50 retains four modalities: CTEM, HRTEM, STEM, and SAED. This yields 9,513 figure-caption pairs and 32,564 modality-verified TEM subimages. The classifier belongs to data filtering, not the ATOMIC inference architecture.

Before generating instructions, the authors separate 29,950 adaptation images from 2,614 evaluation-construction images; the latter do not generate Stage 1 or Stage 2 supervision. Article URLs, parent figure IDs, caption associations, and crop metadata are retained for reconstruction under copyright constraints. This removes compound-figure contamination and establishes an explicit unit for training/evaluation separation. However, the paper describes an image split, which does not establish article-level separation or near-duplicate removal.

2. Dual-Source Supervision: give observable evidence and scientific context different generation conditions

Each training subimage produces four alignment targets: Brief and Detail under VisionGround, and Brief and Detail under DomainContext. Brief emphasizes core semantics with concise descriptions, while Detail supplies finer morphology, texture, and domain information. The VisionGround generator sees only the TEM image, aiming to limit textual shortcuts. The DomainContext generator additionally receives the parent figure caption, drawing on expert material names, experimental conditions, and imaging modes that a single image cannot necessarily provide. The caption conditions the teacher's answer generation; it is not an auxiliary input later supplied to the student model.

Each source produces approximately 60k Stage 1 samples, combined into Blend-120k. Image-only supervision should not be mistaken for guaranteed factual grounding, since the teacher can still misread an image. Caption-conditioned supervision does not establish chemical-composition measurement ability either, because some answer content may come from the caption rather than pixels. Its actual contribution is a controlled separation of supervision provenance that exposes different benefits for visual judgment and domain expression. Blend also increases dataset size, so the existing comparison does not fully isolate complementary sources from additional samples.

3. Connector Concept Alignment: establish TEM semantic correspondence while freezing both endpoints

Stage 1 freezes the vision encoder and language model and updates only the MLP connector. Given a TEM image and a description instruction, the model learns to produce the generated Brief or Detail target. Freezing the language endpoint concentrates adaptation on how visual features enter language space, rather than immediately using the limited domain corpus to update the entire language model. Brief targets emphasize core concepts, while detailed targets broaden fine-grained descriptions, giving the connector supervision at different granularities.

This does not mean that ATOMIC relearns atomic-scale visual features. The CLIP vision tower keeps its original parameters, so the connector can only reorganize the available representations and associate them with domain vocabulary. If fine lattice periodicity or scale-bar information has already been lost during visual encoding, this stage alone is unlikely to recover it. This boundary helps motivate attention to the reported quantitative-scale and periodic-structure failures; it is not a unique causal explanation demonstrated by the paper.

4. Multi-Turn Instruction Adaptation: turn detailed descriptions into domain conversations

Stage 2 generates multi-turn conversations from Detail responses rather than using the less informative Brief responses as its main conversation source. VisionGround and DomainContext remain separate, each forming approximately 30k conversation samples, with their mixture forming Blend-60k. The language model is now unfrozen and trained jointly with the connector, while the vision encoder stays frozen. Whereas Stage 1 primarily connects images to language concepts, Stage 2 teaches the model to select evidence in response to questions, use materials-science terminology, and express its knowledge conversationally.

DomainContext caption information enters these conversations indirectly through previously generated descriptions, not as direct inference context. The evaluation generator can also consult captions; when a question requires experimental background, its stem includes the minimum necessary context without revealing the answer. Evaluated models receive only the image and question, with no caption, article metadata, or source index. Optional +FT then uses separate downstream training QA pairs to adapt to multiple-choice or open-ended formats. Because this adds supervision, its results must be explicitly distinguished from those of the base model.

A Worked Example¶

Consider the nanoparticle TEM image in the paper's Figure 3. After extracting the subimage and passing modality filtering, VisionGround generates descriptions of visible particle distribution, size consistency, and dispersion; DomainContext can additionally draw on material and experimental information in the parent caption. The actual caption for this image is not provided here, so no particular material name or processing temperature is invented.

These descriptions first train the connector, and the Detail responses subsequently support multi-turn conversation generation. At inference, a user uploads only the image and requests a detailed description. ATOMIC recognizes a TEM image and discusses nanoparticle distribution and dispersion in domain-appropriate language, whereas general-purpose baselines produce microscope-slide or snowy-landscape interpretations. This illustrates improved domain recognition and expression, not validated particle-size measurement, elemental identification, or defect-localization accuracy.

Loss & Training¶

The paper describes supervised adaptation using generated targets and conversation answers but does not introduce a new loss function or provide fully verifiable token-masking implementation details in the main text. Accordingly, no ATOMIC-specific loss is invented here. Learning rates across the three training stages are \(10^{-3}\), \(2\times10^{-5}\), and \(2\times10^{-5}\); per-device batch sizes are 32, 16, and 16; epoch counts are 1, 3, and 1. All stages use cosine scheduling, a 0.03 warmup ratio, a maximum sequence length of 2,048, and gradient accumulation of 1.

Training uses 4 NVIDIA RTX PRO 6000 Blackwell GPUs and BF16. Stage 1 uses ZeRO-2, while Stage 2 and task fine-tuning use ZeRO-3. For Blend, Stage 1 takes approximately 2 hours, Stage 2 approximately 4.5 hours, and task fine-tuning approximately 45 minutes. These are reference times for the reported hardware and configuration, not single-GPU deployment costs or end-to-end data-construction costs.

Key Experimental Results¶

Main Results¶

The internal evaluation datasets contain 23,526 QA pairs. Final test sets contain 2,349 TEM-VQA pairs and 2,358 TEM-MCQ pairs, while another 9,405 VQA pairs and 9,414 MCQ pairs are reserved for downstream +FT. The following selection comes from Table 3; all values are percentages and higher is better. Frontier API models are zero-shot references, whereas ATOMIC has already undergone domain adaptation.

Model	VQA BLEU-1	VQA ROUGE-L	VQA METEOR	VQA AWC	MCQ accuracy
GPT-4o	21.9	26.1	29.3	40.4	92.3
Gemini-2.5-Flash-Lite	17.8	22.2	26.0	37.7	90.3
LLaVA-v1.5-7B	9.6	13.6	18.2	33.3	70.9
LLaVA-v1.5-13B	10.5	15.0	17.8	31.0	74.0
ATOMIC-7B-Blend	12.2	16.2	22.8	37.5	75.2
ATOMIC-13B-Blend	11.7	15.5	21.7	36.6	76.5

AWC measures reference-answer word coverage, not semantic correctness. Equation (1) is typographically damaged in the text cache. The following is a normalized restatement based on the adjacent definitions of reference vocabulary, token counts, and coverage, not a character-for-character reproduction of the original equation:

\[ \mathrm{AWC}=\frac{\sum_{w\in V_{\mathrm{ref}}}\min\!\left(\mathrm{count}_{\mathrm{ref}}(w),\mathrm{count}_{\mathrm{hyp}}(w)\right)}{\sum_{w\in V_{\mathrm{ref}}}\mathrm{count}_{\mathrm{ref}}(w)}. \]

\(V_{\mathrm{ref}}\) is the set of unique reference-answer tokens, and the two count terms give token frequencies in the reference and model response. Matches are capped by reference frequency. Including expected technical words raises coverage, but additional incorrect claims receive no direct precision-style penalty. Against the same-size LLaVA baseline, the 7B model improves MCQ accuracy by 4.3 percentage points and AWC by 4.2 percentage points. Its AWC gap to GPT-4o is 2.9 percentage points, which must not be interpreted as a 2.9-point gap in scientific reasoning ability.

Ablation Study¶

The following values come from Tables 5 and 6. MCQ columns report accuracy; VQA columns report the average of BLEU-1, ROUGE-L, METEOR, and AWC, all expressed as percentages. The VQA columns are not AWC alone.

7B configuration	MCQ perception	MCQ reasoning	MCQ methodology	VQA perception	VQA reasoning	VQA methodology
VisionGround	64.8	86.1	71.4	22.2	23.7	17.7
DomainContext	62.9	83.0	73.3	20.7	25.0	18.3
Blend	66.5	85.6	73.3	22.5	25.1	19.0
Blend+FT	84.6	94.3	94.5	41.9	36.9	47.9

Blend does not win every individual category: its 85.6 MCQ reasoning score is below VisionGround's 86.1. DomainContext is stronger for VQA reasoning, while VisionGround is stronger for MCQ reasoning, supporting a distinction between generating a domain-specific answer and discriminating among options using visual evidence. +FT delivers the largest gains, but it adds task-training data, so these gains cannot all be attributed to dual-source supervision.

External Validation & Quality Audit¶

The independent TEM subset of MatCha is not generated by this paper's pipeline. Selected results from Table 4 follow, with all values representing accuracy percentages. The two +FT models use identical fine-tuning conditions; this should not be interpreted as training on MatCha test questions.

Model	All 371 questions	Structure analysis, 165 questions	Defect classification, 33 questions
GPT-4o	73.3	72.1	57.6
LLaVA-v1.5-7B	35.0	39.4	18.2
LLaVA-v1.5-7B+FT	38.5	39.4	18.2
ATOMIC-7B-Blend	47.7	47.9	21.2
ATOMIC-7B-Blend+FT	56.6	52.7	21.2

Under identical +FT conditions, ATOMIC exceeds LLaVA by 18.1 percentage points, supporting domain adaptation as a better initialization. However, ATOMIC's defect classification accuracy remains 21.2%, with no improvement from task fine-tuning in this column. The separately reported property-analysis category contains only 6 questions, so its percentages should not support strong conclusions.

A TEM specialist scores 810 responses to 270 questions on a 0–10 scale. Mean scores for GPT-4o, ATOMIC-7B-Blend, and LLaVA-v1.5-7B are 6.27, 5.75, and 4.57, respectively; their AWC correlations are 0.516, 0.394, and 0.424, all with \(p<0.05\). A separate audit examines 962 Stage 2 QA pairs: 961 score at least 7, and none score at most 3. This is a sampled audit, not evidence that the complete generated corpus is error-free.

Highlights & Insights¶

Supervision provenance becomes an experimental variable. Separating image-only and caption-conditioned data exposes differences between visual judgment and domain expression. This is more informative for adaptation analysis than mixing all synthetic instructions and reporting only aggregate performance.
A fixed architecture makes the data contribution easier to examine. The 7B model outperforms the 13B model on VQA, showing that language-model scaling is not a sufficient solution in these experiments. Nevertheless, Blend contains more samples, so matched-size comparisons are still needed to quantify complementarity strictly.
External validation is more persuasive than internal lexical overlap alone. The matched +FT comparison on MatCha reduces concern about the internal question generator's style. A transferable evaluation practice is to report the effects of domain initialization separately from extra task fine-tuning.

Limitations & Future Work¶

Author-identified distribution bias: training and internal evaluation rely on publication-quality images, potentially underrepresenting noisy, artifact-heavy, or otherwise difficult laboratory acquisitions. Evaluation should include uncurated raw images across instruments and laboratories.
Author-observed failure modes: quantitative scale estimation, periodic-structure interpretation, and overgeneralization of local contrast remain systematic problems. Scale-bar reading, diffraction analysis, or higher-resolution visual adaptation are possible directions, not existing ATOMIC modules.
Experimental concerns raised in this note: Blend doubles the sample count relative to a single source, without a matched-size mixture ablation; the split description also does not establish article-level separation. Follow-up work should control the data budget, group splits by article, and check near-duplicate images.
Evaluation and teacher dependence: related GPT-based generation procedures construct both training data and internal evaluation questions, potentially sharing expression biases. The single-specialist audit provides no inter-rater agreement estimate. AWC does not detect all causal errors, motivating multi-expert review, factual constraints, and calibrated abstention evaluation.
Reproducibility boundary: the authors state that they release annotations, article URLs, figure IDs, normalized crop coordinates, the pipeline, and weights, rather than directly redistributing every publisher-owned image. This note checks only the local paper cache; current code and weight availability has not been verified online.

LLaVA-v1.5: ATOMIC inherits the vision encoder, connector, and language-model structure, turning general-purpose alignment and instruction tuning into a TEM supervision pipeline. Its principal contribution is domain-data organization rather than an architectural breakthrough.
LLaVA-Med and BiomedCLIP: these also demonstrate the value of adapting specialized images and specialized language, but target biomedical content. Materials-science TEM involves different contrast mechanisms, lattice structures, and experimental terminology, so biomedical gains are not direct evidence for TEM performance.
MicroscopyGPT: this work uses simulated STEM images of two-dimensional materials and chemical-formula inputs to predict POSCAR atomic structures; ATOMIC targets natural-language QA across multiple TEM modalities. Different input conditions and output spaces preclude direct ranking using this paper's accuracy scores.
Research direction: explicitly label dual-source supervision as visually observable, context-dependent, or insufficiently supported, and require answers to distinguish observations from inferences. This could reduce the presentation of caption-derived knowledge as pixel-derived fact, but remains a proposal without experimental verification in this paper.

Rating¶

Novelty: 3/5. Dual-source supervision is a targeted TEM adaptation design, but the architecture and two-stage training recipe build on established approaches.
Experimental Thoroughness: 4/5. Internal tasks, supervision ablations, external benchmarking, and specialist audits provide broad evidence, but matched-size mixtures and stricter source-separation documentation are missing.
Writing Quality: 4/5. The distinction between supervision generation and inference inputs is clear, and AWC is framed cautiously; broad improvement claims should be read alongside individual counterexamples.
Value: 4/5. Local QA for privacy-sensitive materials images is useful, but does not replace reliable quantitative microscopy analysis.