ECCV2026 Medical Imaging LLM Agent multi-agent co-scientist verifiable hypothesis testing statistical power evidence classification medical image analysis

VERITAS: A Multi-agent Co-scientist for Verifiable Image-Derived Hypothesis Testing¶

Conference: ECCV2026
Paper: ECCV
Code: https://github.com/LucZot/veritas
Area: Medical Imaging / LLM Agent
Keywords: multi-agent co-scientist, verifiable hypothesis testing, statistical power, evidence classification, medical image analysis

TL;DR¶

Veritas splits "natural-language clinical hypothesis to auditable conclusion" into four phases — analysis planning, segmentation, statistics, and interpretation — carried out by three role-specialized agents (PI, Imaging Specialist, Statistician) plus a Critic, and maps the observed statistics mechanically into Supported / Refuted / Underpowered / Invalid via an evidence classification operator that is independent of agent judgment; on 64 cardiac and brain-glioma MRI hypotheses spanning six complexity tiers it reaches 81.4% (frontier models) / 71.2% (locally deployed 8–30B open-weight models) majority-vote verdict accuracy and the highest rate of independently verifiable statistical outputs (86.6%).

Background & Motivation¶

Medical images encode volumetric measurements, morphological features, and their associations with patient outcomes, making them one of the most important data domains in clinical research. Yet turning image data into statistically defensible conclusions — the gold standard of scientific evidence — is a fragmented process. A neuro-oncologist who wants to test whether the more aggressive biology of glioblastoma is reflected in larger contrast-enhancing tumors than in lower-grade gliomas must cross several software tools and disciplinary boundaries, solving image segmentation, metric extraction, statistical test selection, and result interpretation in sequence, where an error in any single step contaminates the final conclusion. This coordination cost both slows discovery and, in practice, decides who gets to participate in imaging research.

Over the past two years, LLMs and multi-agent systems have begun automating literature synthesis, hypothesis generation, and experimental planning in scientific workflows, and some work validates hypotheses through sequential falsification with formal error control. Moving these systems to medical imaging hypothesis testing runs into three new obstacles. First, the pipeline must bridge very different tasks — cohort definition (picking the right patients from a dataset), vision (segmenting anatomical structures), computation (turning masks into biomarkers), and statistics (selecting, executing, and interpreting tests) — so models must reason jointly across all of them, whereas existing imaging agents mostly stop at the perception level (segmentation, retrieval, visual question answering) and never perform cohort-level inference across patients. Second, clinical conclusions demand full auditability: every step from segmentation mask to p-value calculation must be inspectable and reproducible, which collides directly with the LLM habit of emitting a natural-language summary. Third, clinical cohorts are often small, so the system needs epistemic awareness — the ability to distinguish "no effect detected" from "insufficient power to detect an effect" — otherwise underpowered analyses get mistaken for true negatives.

This paper's angle is to imitate the clinician-scientist workflow: decompose the study into role-specialized work and force every step to leave behind machine-readable, executable, traceable artifacts instead of letting one model reason end to end. Core idea: replace monolithic LLM reasoning with role-specialized agents plus a phased chain of executable artifacts plus an evaluator-derived evidence label, so that hypothesis testing on medical images is simultaneously autonomous and step-by-step auditable, and so that 8–30B local models can perform reasoning that normally requires frontier models.

Method¶

Overall Architecture¶

Formally, the setting is: given a dataset \(D=\{(x_i, m_i)\}_{i=1}^{N}\) (\(x_i\) the images of patient \(i\), \(m_i\) cohort variables, clinical covariates, and outcomes such as survival endpoints) and a natural-language hypothesis \(H\), autonomously decide whether \(H\) is Supported, Refuted, Underpowered, or Invalid given the available evidence. Unlike a traditional computational pipeline, the system does not hard-code metric definitions and test types in code; it interprets the hypothesis semantically, checks feasibility against the data, designs an analysis plan, and then executes it end to end with an audit trail.

The workflow has four phases. Phase 1 (Analysis Planning): the three role agents collaboratively produce a structured plan.json specifying the target cohorts, population restriction, required anatomical structures and observations, derived measurements, and the planned statistical analysis, together with a TESTABLE / UNTESTABLE feasibility decision (including the subtype of untestability and the missing requirements); a programmatic validator checks schema completeness and consistency against the dataset metadata before execution, and an untestable hypothesis terminates immediately as Invalid. The Statistician also derives a priori power from the queried cohort sizes under a fixed planning SESOI. Phase 2A (Segmentation): when the hypothesis needs image-derived measurements, the Imaging Specialist issues a segmentation request (patients, target structures, observations) and a SAT backend stores the resulting binary masks in a shared database — this is the vision core of the framework, and all subsequent quantitative analysis is grounded in these masks; hypotheses needing no imaging structures bypass this phase. Phase 2B (Statistical Analysis): the Statistician writes executable Python that retrieves masks and metadata through the API, computes the planned measurements, runs the test, and reports effect size δ, 95% confidence interval, p-value, and sample sizes; the code runs in a sandbox with timeout constraints, and failed or invalid attempts are revised iteratively using execution traces and validator feedback (optionally with Critic input), up to 8 attempts per round. Phase 3 (Interpretation): the agents synthesize the preceding artifacts in a structured discussion and emit a verdict ∈ {YES, NO, INCONCLUSIVE} with rationale; crucially, the evaluator-computed evidence label is deliberately withheld from the agents at this point.

Two model deployments share the same workflow and prompts: a local open-weight team (GPT-OSS-20B for PI/imaging, Qwen3-8B for discussions and critic, Qwen3-Coder-30B for coding, 8–30B in total) and a frontier team (GPT-5.2 for PI/imaging/coding, GPT-5 Mini for discussions and critic). ⚠️ Model version names are as given in the original paper, whose references point to vendor release pages rather than papers.

Auditability comes from the artifacts: each phase emits structured artifacts (hypothesis record H, plan P, segmentation requests R, masks M, code C, statistical outputs S, verdict V, evidence label L), all stored and versioned rather than summarized in natural language; the phase boundaries carry four in-workflow validity gates (feasibility check, segmentation validity check, integrity check, verdict schema check), plus a post-hoc plan-execution consistency check run by the evaluator.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Natural-language hypothesis H + imaging dataset D"] --> B["Role-specialized agents and the four-phase artifact chain<br/>PI · Imaging Specialist · Statistician · Critic"]
    B -->|TESTABLE| C["Phase 2A segmentation<br/>SAT outputs binary masks"]
    B -->|UNTESTABLE| Z["Terminate: Invalid"]
    C --> D["Dataset-API boundary and sandboxed iterative execution<br/>statistics code → δ / 95% CI / p / n"]
    D --> E["Evidence Classification Operator (ECO)<br/>four evidence labels · withheld from agents"]
    D --> F["Phase 3 structured discussion<br/>verdict: YES / NO / INCONCLUSIVE"]
    E --> G["Verdict + evidence label + versioned artifact chain"]
    F --> G

Key Designs¶

1. Role-specialized agents and the four-phase artifact chain: every step leaves machine-readable hand-off artifacts

A monolithic LLM cascades hallucinations on long, tool-heavy tasks of this kind; an independent evaluation already found that several multi-agent research frameworks fail to complete a full research cycle, with agents frequently fabricating results during implementation. Veritas responds not by swapping in a stronger model but by changing the organization: the PI owns scientific validity and feasibility, the Imaging Specialist owns segmentation targets and imaging protocols, the Statistician owns test selection, effect sizes, and power analysis, and a separate Critic reviews the execution phases (2A/2B) by default. What actually does the work is the form of the hand-off — each phase emits JSON plans, segmentation requests, Python code, statistical outputs, and verdicts rather than prose summaries, all structured, executable, and versionable, directly reusing MetaGPT's insight that structured intermediate artifacts curb cascading hallucinations. The cost is a longer workflow; the payoff is twofold. Any single step can be opened and inspected on its own, and when something goes wrong one can localize which phase produced a bad artifact. And because phases couple through machine-readable artifacts, each role only judges its own segment within a constrained context, which is what lets small models carry their share.

2. Dataset-API boundary and sandboxed iterative execution: turn "fabricating statistics" into "rewrite until it runs"

Letting an LLM read raw files and "summarize" the statistics amounts to letting it invent p-values. Veritas installs two gates. The first is a data-access boundary: agents never touch raw files during analysis and can only call a constrained API layer offering cohort discovery, per-patient metadata, observation identifiers, segmentation masks, and geometry-aware measurement utilities; the unified API boundary standardizes provenance and makes off-contract data access automatically detectable. The second is an execution boundary: every statistical conclusion must come from Python that actually ran in a sandbox, and failed or non-compliant attempts are rewritten with execution traces and validator feedback, up to 8 attempts. This loop is the system's main anti-hallucination engine — the paper's diagnostics show that adding iterative self-correction drops "hallucinated significance" (a YES verdict with p ≥ 0.05 or with no p-value at all) from 9–12% for one-shot baselines to ≤1.5%. Its limit is equally clear: the sandbox only guarantees the code runs, not that it computes the right thing. In practice the local configuration still generated mock/random data instead of real measurements in 10.9% of runs, so "executed successfully" is not "analysis is valid" — the weakest link in the framework today.

3. Evidence Classification Operator (ECO): taking the epistemic judgment away from the agents

Clinical imaging cohorts are usually small, and reading every p ≥ 0.05 as "no effect" mistakes underpowered analyses for true negatives; in a binary evaluation, a system that conservatively labels every non-significant result as "refuted" can even score spuriously high accuracy. Veritas introduces a four-label evidence classification: given the observed statistics \((p, \delta, n)\) and a pre-specified smallest effect size of interest (SESOI) \(\delta_0\), the label is a deterministic function — with \(\alpha=0.05\) and \(\pi\) the statistical power computed at \(\delta_0\) for the actual sample sizes (i.e., the probability that this study would detect an effect as large as \(\delta_0\) if one truly existed):

Label	Condition	Interpretation
Supported	\(p < \alpha\) and direction matches the hypothesis	Evidence confirms the hypothesis
Refuted	(\(p \ge \alpha\) and \(\pi \ge 0.80\)) or (\(p < \alpha\) and opposite direction)	Adequate power with no effect / significant opposite effect
Underpowered	\(p \ge \alpha\) and \(\pi < 0.80\)	Insufficient power to conclude in either direction
Invalid	Untestable or execution failure	Results unreliable

The label is computed by the evaluator-side Evidence Classification Operator and withheld from the agents until Phase 3 begins — agents only produce their own verdict and cannot reverse-engineer the label to guess the answer. This turns "is the conclusion warranted by the statistics" from a subjective judgment into a checkable mechanical mapping, and makes the Supported vs. Underpowered distinction part of the evaluation: the paper's examples are glioma_31 (1p/19q codeletion, n=13 vs. n=86, power=0.38), which must be labelled Underpowered rather than Refuted, and glioma_14 (IDH-mutant grade IV, n=28 vs. n=367, power=0.72), which sits on the borderline. Conceptually it descends from the four-outcome decision framework that formalized TOST equivalence testing, except that framework previously existed only as manual analytical guidance; here it is actually automated inside the pipeline.

A Worked Example¶

Take an L2 hypothesis on UCSF-PDGM: "IDH-wildtype gliomas have larger enhancing tumor volumes than IDH-mutant gliomas," and walk it through the pipeline.

In Phase 1 the PI confirms the hypothesis is testable on this dataset (501 subjects, IDH molecular markers available, T1-contrast sequences present), the Statistician looks up both cohort sizes in the metadata and derives a priori power under the fixed SESOI, and the Imaging Specialist identifies the four required subregions (WT / NCR / ED / ET); the plan lands in plan.json and passes the validator's schema and metadata-consistency checks. In Phase 2A the Imaging Specialist issues the segmentation request and SAT writes binary masks for the four subregions over the preoperative MRIs. In Phase 2B the Statistician's Python groups patients by IDH status, computes enhancing-tumor volume from the ET masks, runs the between-group test (with covariate adjustment where required), and prints δ, 95% CI, p, and both sample sizes; if the first run fails on a missing mask or a geometry mismatch, it is rewritten with the execution trace, up to 8 attempts. ECO then maps \((p, \delta, n)\) to a label: p < 0.05 with matching direction → Supported; opposite direction → Refuted; p ≥ 0.05 with power ≥ 0.80 → Refuted (adequate power, genuinely no effect); power < 0.80 → Underpowered. Finally, in Phase 3 the agents — unable to see the label — discuss the plan, code, and statistical outputs into YES / NO / INCONCLUSIVE.

Contrast this with the 1p/19q hypothesis behind glioma_31: n = 13 vs. n = 86, power only 0.38, p non-significant. Under binary evaluation, calling it "negative" would count as correct; the four-label framework requires the system to mark it Underpowered, i.e. to admit that this analysis cannot support a directional conclusion at all.

Key Experimental Results¶

The benchmark is a tiered hypothesis bank: 32 hypotheses per dataset, 64 in total, spanning six complexity tiers, where each hypothesis's ground truth is established from canonical reference statistics computed on the ground-truth segmentation masks and patient metadata and validated by a domain expert. Every tier mixes positive controls, negative / no-effect controls, and underpowered or untestable controls, so all four outcomes of the evidence-label framework are exercised:

Tier	Name	n	Capability probed	Pos.	Neg. / no effect	Underpowered / untestable
L0	Untestable	5	Feasibility detection	—	—	5
L1	Metadata-only	9	Statistical reasoning	3	5	1
L2	Single imaging metric	23	Vision-to-statistics pipeline	15	8	—
L3	Engineered features	6	Feature engineering	6	—	—
L4	Mixed	16	Multimodal integration	8	3	5
L5	Multivariate	5	Advanced statistical reasoning	5	—	—
Total		64		37	16	11

The two datasets cover distinct clinical domains. ACDC contains 150 subjects of short-axis cardiac cine MRI in five diagnostic groups of 30 (DCM, HCM, MINF, RV, NOR), with ED/ES phases, ground-truth segmentations of the left ventricle, right ventricular cavity, and myocardium, and metadata on height, weight, and cardiac timing; its hypotheses span L0–L4. UCSF-PDGM contains 501 preoperative diffuse glioma MRI subjects (WHO grade 2: 56, grade 3: 43, grade 4: 396) with molecular markers (IDH mutation, MGMT methylation, 1p/19q codeletion), ground-truth tumor masks for 495 subjects following the BraTS conventions, and metadata on age, sex, overall survival (395 with survival data, 228 events), and extent of resection; SAT segments four subregions (WT, NCR, ED, ET) and the hypotheses span L0–L5, including survival analyses and multivariate regression. The baselines form a five-step single-model ablation ladder: SMa direct reasoning (only pre-computed per-group summary statistics, no code), SMb code on pre-computed features (a per-patient CSV and one automatically executed script), SMc code via API (only the hypothesis and the API documentation, one end-to-end script), SMd agentic (same API, write → execute → inspect → refine, up to 3 rounds), and SMe pipeline (same API but following Veritas's exact phase agendas as a single-model chain-of-thought). The ladder isolates code execution vs. pure reasoning, pre-extracted features vs. raw API access, iterative self-correction, and structured phase guidance in turn; comparing SMe with Veritas then directly isolates the contribution of multi-agent discussion.

Main Results¶

Every method was run 10 times per hypothesis (64 hypotheses, 640 runs per method). Evidence accuracy only counts runs that produced executable statistical output, hence no such column for SMa; unless stated otherwise all metrics are computed on testable hypotheses (L1–L5), while L0 feasibility detection is reported separately. Baselines always produce output by construction, so no completion rate is reported for them.

Model family	Method	Evidence (run)	Verdict (run)	Evidence (majority vote)	Verdict (majority vote)	Compl.
Local 8–30B	SMa direct reasoning	—	56.1	—	55.9	—
Local 8–30B	SMb code on features	58.1	63.8	61.0	62.7	—
Local 8–30B	SMc code via API	39.0	63.2	40.7	59.3	—
Local 8–30B	SMd agentic	46.3	66.0	52.5	66.1	—
Local 8–30B	SMe pipeline	55.1	58.0	57.6	61.0	—
Local 8–30B	Veritas	63.4	71.4	67.8	71.2	78.1
Frontier GPT-5.2	SMa direct reasoning	—	55.9	—	55.9	—
Frontier GPT-5.2	SMb code on features	63.7	70.7	69.5	69.5	—
Frontier GPT-5.2	SMc code via API	52.9	72.8	61.0	76.3	—
Frontier GPT-5.2	SMd agentic	64.7	76.6	64.4	74.6	—
Frontier GPT-5.2	SMe pipeline	60.8	65.5	59.3	61.0	—
Frontier GPT-5.2	Veritas	70.0	81.0	76.3	81.4	87.5

L0 feasibility detection: Veritas 74.0% (local) and 100% (frontier); all baselines reach 100% except SMb-frontier at 98%.

Run-level evidence accuracy stratified by dataset and tier (verdict accuracy also listed; the L0 column is feasibility detection):

Method	ACDC Ev.	ACDC Vd.	UCSF Ev.	UCSF Vd.	L1	L2	L3	L4	L5	All evidence
Veritas (local)	91.0	87.9	36.7	49.0	54.4	87.0	51.7	49.4	30.0	63.4
SMc (GPT-5.2)	79.0	85.1	27.7	60.9	75.6	70.9	43.3	34.4	0.0	52.9
SMd (GPT-5.2)	95.9	96.6	34.7	53.9	94.4	85.2	55.0	42.5	0.0	64.7
SMe (GPT-5.2)	90.3	96.9	32.3	35.1	56.7	89.1	56.7	43.1	0.0	60.8
Veritas (frontier)	90.0	78.9	50.7	83.2	90.0	90.4	73.3	45.0	16.0	70.0

Ablation Study¶

The ablation uses the local Veritas configuration with 10 runs per hypothesis; the defaults are 16k context, Critic enabled in the execution phases, and temperature 0.2 (ablations of the vision source, i.e. ground-truth masks vs. SAT-Pro, are in the supplementary material):

Configuration	ACDC Ev.	ACDC Vd.	UCSF Ev.	UCSF Vd.	Combined Ev.	Combined Vd.
Default (16k context)	91.0	87.9	36.7	49.0	63.4	71.4
Context 8k	91.4	91.7	29.0	45.7	59.7	72.0
Context 32k	90.0	87.1	31.7	51.8	60.3	72.7
No Critic	90.7	90.1	32.0	50.5	60.8	73.2
Temperature zero	90.3	88.7	27.7	47.5	58.5	72.6

Diagnostic rates characterize how methods fail rather than only how often; note these are conditional rates over each method's own run pool, not a flat quality ranking:

Method	Overclaim ↓	False ref. ↓	Hall. sig. ↓	Synth. ↓	Lit. p ↓	Verif. ↑
SMc code via API (local)	65.1	6.7	8.7	0.2	1.4	82.3
SMd agentic (local)	25.6	10.2	1.5	0.2	12.2	53.6
SMe pipeline (local)	3.8	8.5	0.0	0.9	17.2	57.7
Veritas (local)	7.2	24.2	2.7	10.9	2.4	77.3
SMd agentic (frontier)	14.8	11.7	0.0	0.6	3.3	70.5
SMe pipeline (frontier)	13.0	7.9	0.0	0.5	10.9	68.1
Veritas (frontier)	20.0	12.6	0.4	3.6	15.0	86.6

Verifiability is the fraction of runs reporting the four core statistical outputs required for independent re-verification (test type, sample sizes, effect size, p-value).

Key Findings¶

The multi-agent gain is directly attributable. The SMe-to-Veritas comparison is the cleanest one: identical phase agendas, identical API documentation, identical sandbox, differing only in multi-agent discussion versus a single-model chain-of-thought — majority-vote verdict accuracy improves by +10.2pp locally (71.2% vs. 61.0%) and +20.4pp on the frontier (81.4% vs. 61.0%). The larger frontier gap suggests multi-agent discussion pays off more as the base model gets stronger.
Code execution is the hard threshold, and self-correction mainly fixes statistics rather than interpretation. SMa direct reasoning plateaus at ~56% verdict accuracy regardless of model scale, while adding code execution (SMb) lifts it by 8–15pp. The SMc → SMd self-correction step adds only +2.8 / +3.8pp of verdict accuracy but +7.3 / +11.8pp of evidence accuracy, i.e. iteration improves the statistical analysis, while the final interpretation step was relatively easy to begin with.
The bottleneck is computing correctly, not reasoning correctly. Verdict accuracy exceeds evidence accuracy for every method (Veritas local: 71.4% vs. 63.4%; frontier: 81.0% vs. 70.0%), most strikingly for frontier Veritas on UCSF, where verdict accuracy (83.2%) even exceeds its ACDC verdict accuracy (78.9%) while evidence accuracy is only 50.7%. That points the improvement target squarely at code generation for complex analyses such as survival models and multivariate regression.
Only Veritas achieves non-zero L5 evidence accuracy (16–30%), whereas every single-model baseline scores 0% on L5 even though their verdict accuracy there reaches up to 92% — they produce correct YES/NO guesses with no valid statistical backing whatsoever. With only 5 L5 hypotheses the numerical precision is limited, but the qualitative non-zero vs. zero gap is robust.
The four-label framework genuinely separates the two kinds of negative. Without the Underpowered label, a batch of small-subgroup UCSF hypotheses would be conflated with true negatives: glioma_31 (n=13 vs. n=86, power=0.38) and glioma_14 (n=28 vs. n=367, power=0.72) are both underpowered, and the system must admit that these analyses support no directional conclusion rather than scoring points by labelling everything negative.
Tier sensitivity is concentrated in UCSF. ACDC evidence accuracy stays at 90–91% under every ablation, i.e. cardiac hypotheses are essentially solved by the base architecture; the sensitivity lives almost entirely in UCSF-PDGM evidence accuracy: dropping context from 16k to 8k costs 7.7pp (complex survival/regression analyses no longer fit), raising it to 32k gives no uplift (a further 5.0pp down), and deterministic decoding (T=0) costs 9.0pp (complex hypotheses benefit from stochastic code exploration). Removing the Critic actually improves combined verdict by +1.8pp, indicating that the iterative code-revision loop already performs the error correction and the Critic is redundant.
The honest costs. Frontier Veritas has relatively high overclaim (20.0%) and literal-p (15.0%) rates among code-producing methods, while local Veritas has the highest false refutation (24.2%) and synthetic-data rate (10.9%). The frontier numbers follow from attempting more, and more complex, analyses; the local false refutation is not a conservative interpretation style, because ECO assigns labels mechanically — it reflects weaker code yielding null or invalid statistics on adequately powered, truly Supported hypotheses.

Highlights & Insights¶

Verifiability is turned into a quantity. Most agent evaluations report only end-to-end accuracy; Veritas additionally measures the fraction of runs emitting the four core statistical outputs (86.6% on the frontier), which makes even its mistakes traceable, diagnosable, and correctable. The paper's careful split between methods operating on raw data and methods operating on pre-computed features matters here — SMb/SMc also reach ~82% verifiability, but only because their simpler code completes more often, at the price of zero L5 evidence accuracy.
The evidence label is a decision rule with no free parameters. The label is a deterministic function of \((p, \delta, \pi)\) with nothing tunable beyond the significance level α and the SESOI, which is far more reproducible than "let an LLM judge," and removes the evaluation ambiguity common in agent benchmarks. The idea transplants to any benchmark involving statistical testing.
The ablation ladder is well designed. SMa → SMb → SMc → SMd → SMe each change exactly one variable (code, data access, iteration, phase guidance), and the last step differs from Veritas only in multi-agent discussion, so every gain can be attributed. This "same prompts, same tools, same sandbox, only the organizational structure changes" control is worth reusing when evaluating other multi-agent systems.
Structured decomposition substitutes for model scale. The local 8–30B team reaches 71.2%, the best among same-scale baselines; for clinical settings where data cannot leave the hospital and everything must run on-site, that conclusion is worth more than absolute accuracy.
A reusable catalogue of failure modes. Overclaim, false refutation, hallucinated significance, synthetic data, and hard-coded p-values decompose "the agent failed" into separately detectable patterns that any code-producing agent evaluation can borrow. The hard-coded p-value mode deserves particular attention: once self-correction suppresses hallucinated significance to ≤1.5%, some models hard-code the significance threshold in the analysis code instead, moving the failure from the output layer to the code layer.

Limitations & Future Work¶

Segmentation dependency (authors). All imaging-derived conclusions depend on segmentation quality, and the framework treats segmentation as a point estimate without propagating its uncertainty; the supplement quantifies the gap between ground-truth and SAT masks, but pushing segmentation error into the statistical inference remains future work.
Benchmark scope (authors). The evaluation covers two MRI domains and a single segmentation model; extending to CT or histopathology requires new — though widely available — segmentation models. The hypothesis bank also does not yet cover longitudinal or causal analyses.
Ground truth (authors). Ground-truth verdicts come from expert-validated reference computations on the given datasets. That was strictly necessary for mechanical verifiability, but it may reflect dataset-specific rather than population-level effects.
Read the numbers with care. Diagnostic rates are conditional on each method's own run pool, and different pools are not directly comparable; L5 has only 5 hypotheses, so the qualitative "16–30% vs. 0%" conclusion is stable while the exact values are not; and frontier Veritas's ACDC verdict accuracy (78.9%) is actually lower than several baselines (85–97%), which the authors attribute to pipeline failures on easy hypotheses — a reminder that a multi-stage workflow is pure overhead on tasks that were never hard.
My own observations. The sandbox guarantees only that code runs, not that it computes the right thing — the 10.9% synthetic-data rate of local Veritas shows the self-correction loop lacks semantic checks on the code. A direct fix is static verification in Phase 2B: check that the script really reads measurements from masks/API, does not hard-code the significance threshold, and respects the planned population restriction. A second direction is adaptive phase pruning: for L1/L2 hypotheses, Phase 2A and multi-agent discussion bring limited benefit, so a shorter path could cut cost and reduce the execution failures that hurt on easy hypotheses.

vs Popper: it validates hypotheses through sequential falsification with formal error control, but operates on tabular and text data with neither medical imaging nor power analysis; Veritas supplies exactly those two missing pieces — image grounding and epistemic power awareness.
vs MedAgent-Pro: it pairs segmentation tools with coding agents to compute clinical indicators from images, but targets per-patient diagnosis rather than population-level hypothesis testing; Veritas's unit of inference is the cohort and its output is a statistically grounded conclusion with an evidence label.
vs MESHAgents: also multi-agent orchestration for association studies in cardiovascular imaging, but it reasons over pre-extracted tabular phenotypes, does not derive measurements from images, and provides no grading of evidence strength.
vs AI Scientist / AI co-scientist (Lu et al., Gottweis et al., Robin, etc.): they automate full research cycles but mostly over text, tables, and ML experiments; Veritas's niche is unifying image-derived measurement, executable statistical validation, and power-aware evidence calibration inside one auditable pipeline.
vs the single-model baselines SMa–SMe: the ladder itself is a step-by-step rebuttal of doing science with a monolithic LLM — pure reasoning is not enough, and even after adding code execution, raw data access, iterative self-correction, and phase guidance it still trails multi-agent discussion. It also serves as a warning: SMe (a single model given the identical phase instructions) is a strong control, and any multi-agent system should run that control before claiming its gain comes from the division of labour.

Rating¶

Novelty: ⭐⭐⭐⭐ First to make a power-aware evidence label a mechanically derived first-class citizen of the pipeline, and to connect an AI co-scientist to cohort-level hypothesis testing on medical images.
Experimental Thoroughness: ⭐⭐⭐⭐ 64 hypotheses × 10 repeats × two model families × a five-step ablation ladder, plus diagnostic rates and configuration ablations; the weak spots are the 5 L5 hypotheses and ground truth drawn from dataset-internal reference computations.
Writing Quality: ⭐⭐⭐⭐ Failure modes are reported frankly, including unfavourable contradictions (lower ACDC verdict accuracy, the highest false-refutation and synthetic-data rates) with explanations.
Value: ⭐⭐⭐⭐ Directly useful for clinical research that demands auditability and on-site deployment, and the evidence-label framework transfers to any benchmark involving statistical testing.