Skip to content

LUMINA: LongitUdinal MultImodal reasoNing Agent for Medical Imaging

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/VisualReasoner/LUMINA
Area: LLM Reasoning
Keywords: Longitudinal Reasoning, Multimodal Agent, Sequential Monte Carlo, Evidence Reconciliation, Medical Imaging

TL;DR

Addressing the degradation and opacity caused by collapsing irregular longitudinal clinical histories into monolithic prompts, LUMINA formulates longitudinal reasoning as a structured single-agent state machine featuring intra-modality anchor comparisons, multi-timescale trajectory memory with Sequential Monte Carlo (SMC) compression, and bounded contradiction audits, establishing state-of-the-art results on Alzheimer's disease classification and 36-month progression forecasting across 9 MLLM backbones.

Background & Motivation

Multimodal Large Language Models (MLLMs) have demonstrated impressive perception and question-answering capabilities on static clinical inputs. However, critical real-world diagnostic decisions—especially in neurodegenerative disorders such as Alzheimer's disease (AD)—are fundamentally longitudinal. Biomarkers, structural neurodegeneration, and clinical presentations unfold across multi-year trajectories rather than within isolated clinical encounters. In standard clinical cohorts, observations arrive through irregular visit schedules, imaging modalities (e.g., T1-weighted MRI, amyloid PET, and tau PET) suffer from heterogeneous missingness, and clinicians must jointly evaluate immediate local anatomical changes against established long-term disease progression.

Despite these clinical demands, existing multimodal paradigms collapse the entire historical record, irregular visits, and current observations into a single monolithic prompt (collapsed static prompting). This single-pass formulation forces the foundational model to act as an unconstrained black box that conflates perceptual grounding, memory retention, evidence selection, and outcome prediction. Recent studies indicate that expanding context lengths impairs reasoning and fine-grained retrieval even when the context contains the necessary facts. When heterogeneous modalities provide conflicting or noisy signals over time, standard MLLMs exhibit severe recency bias, hallucinate unobserved progression, and lack mechanisms to explicitly trace or verify evidence provenance.

To address these fundamental limitations, this work reformulates longitudinal multimodal reasoning as an agentic memory-and-control problem. Rather than treating temporal history as passive context, the agent actively constructs, compresses, and audits an explicit, verifiable evidence state prior to emitting any diagnostic prediction. Core idea: reframe longitudinal multimodal reasoning into a structured inference loop that explicitly decouples current-visit visual grounding, intra-modality prior anchor comparison, Sequential Monte Carlo (SMC) importance-compressed trajectory memory, and bounded belief verification, enabling MLLMs to perform transparent, verifiable clinical reasoning over irregular clinical prefixes.

Method

Overall Architecture

LUMINA operates as a single-agent framework that maps an irregular, partially observed subject-history prefix \(X_{s,\le t}\) into an explicit evidence state \(\omega_{s,t}\) before deterministically projecting a final belief state into the target label space \(\hat{y}_{s,t} = \Pi(B_{s,t})\). Guided by a controller with a bounded action vocabulary, inference proceeds through four sequential phases: (1) inspection planning and current-visit perceptual grounding, optionally calibrated against matched healthy reference comparators; (2) intra-modality anchor retrieval and localized temporal change measurement; (3) multi-timescale trajectory review with Sequential Monte Carlo (SMC) compression and optional cross-subject analogy retrieval; and (4) multimodal evidence reconciliation, belief state formation, and bounded contradiction repair.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input: Irregular Longitudinal Prefix<br/>Xs,≤t (MRI / PET 2.5D Views)"] --> B["Inspection Planning & Current-Visit Grounding<br/>Extract visible findings Os,t and calibrated read Cs,t"]
    B --> C["Intra-Modality Anchor Comparison<br/>Retrieve same-modality anchor, compute local change Δs,t,m"]
    C --> D["Multi-Timescale Memory & SMC Compression<br/>Utility-weighted importance selection for trajectory state"]
    D --> E["Cross-Subject Memory Retrieval<br/>Retrieve analogous summaries from frozen disjoint bank"]
    E --> F["Multimodal Evidence Reconciliation & Bounded Audit<br/>Synthesize local & trajectory evidence, repair conflicts"]
    F --> G["Deterministic Label Normalization<br/>AD-continuum / 36-month progression prediction"]

Key Designs

1. Inspection Planning & Current-Visit Grounding: Isolating Visible Manifestations from Speculative Trajectories

To prevent the model from hallucinating longitudinal progression based solely on demographic priors or raw visit counts, the controller begins with an explicit Inspect phase. It inspects available metadata and imaging timestamps to determine eligible modalities and formulate targeted clinical inspection questions \(Q_{s,t}\). The subsequent Observe phase instructs the MLLM to describe only directly visible anatomical and metabolic features \(O_{s,t}\) while explicitly flagging perceptual uncertainty. When protocol-consistent healthy comparators \(R_{s,t}\) are supplied by the benchmark, a CalibrateVisualRead step normalizes scanner-dependent intensity variations and PET non-specific background binding into a calibrated observation state \(C_{s,t}\), establishing an objective perceptual baseline before any longitudinal comparison occurs.

2. Intra-Modality Anchor Comparison: Explicit Quantification of Modality-Specific Changes

Evaluating multi-year disease trajectories by inspecting unordered image bundles obscures localized, modality-specific structural degeneration. For each observed modality \(m\), LUMINA identifies the most recent valid historical study from the same subject and same modality (\(x_{s,a_m,m}\)). Through ComparePastEvidence, the agent performs a pairwise comparative evaluation, extracting change direction (e.g., bilateral hippocampal volume loss, ventricular enlargement), magnitude, confidence, and uncertainty into a structured visit delta \(\Delta_{s,t,m}\). Crucially, if a subject has historical visits but lacks a prior scan for a specific modality, LUMINA explicitly flags the anchor as unavailable rather than hallucinating longitudinal change from cross-modality inference, compiling these rigorous deltas into a visit-level evidence state \(M_{s,t}^{\text{visit}}\).

3. Multi-Timescale Memory & SMC Compression: Principled Retention of Critical Disease Turning Points

As clinical visit histories lengthen, concatenating all prior encounters overwhelms the token budget and dilutes high-signal pathological transitions. LUMINA introduces a multi-timescale memory architecture comprising short-term visit memory \(M_{s,t}^{\text{visit}}\) and long-term trajectory memory \(M_{s,<t}^{\text{traj}}\). To compress historical events under a fixed budget, LUMINA employs Sequential Monte Carlo (SMC) importance weighting. For each historical candidate event \(e \in H_{s,t}\), an explicit utility function synthesizes clinical change grade \(g(e)\), event confidence \(c(e)\), and half-life recency \(r(e) = \exp(-\ln 2 \cdot \Delta t(e)/\tau)\):

\[u(e) = \lambda_g g(e) + \lambda_c c(e) + \lambda_r r(e)\]

Utilities are normalized into categorical sampling weights \(w(e) = u(e) / \sum_{e' \in H_{s,t}} u(e')\), and the trajectory manager retains only the top-\(K\) events by weight. This SMC-inspired pruning preserves seminal disease milestones (e.g., initial conversion to amyloid positivity or abrupt tau cortical expansion) over multi-year spans while discarding redundant, clinically quiescent follow-ups.

4. Cross-Subject Memory Retrieval: Frozen, Disjoint Trajectory Analogies Without Data Leakage

When an individual's historical prefix is brief or ambiguous, clinical decision-making relies heavily on population-level disease models. LUMINA incorporates an optional cross-subject retrieval module \(M_{s,t}^{\text{cross}}\) that matches current trajectory signatures against an external memory bank. To strictly eliminate data contamination and outcome leakage, this memory bank is constructed offline from completely disjoint subjects and permanently frozen during evaluation. The retrieval index queries only non-label numerical descriptors (e.g., modality availability, trajectory variance, and normalized visit intervals), returning compact structural progression summaries and audit histories without exposing target labels, cognitive scores (MMSE/CDR), or prospective outcomes.

5. Multimodal Evidence Reconciliation & Bounded Audit: Enforcing Diagnostic Consistency Across Divergent Signals

In degenerative diseases, structural MRI and molecular PET frequently provide temporally asynchronous signals (e.g., elevated cortical tau deposition preceding detectable cortical thinning). The ReconcileMultimodalEvidence stage systematically cross-references current observations, local same-modality deltas, trajectory memory, and cross-subject analogies, formally documenting modalities that concur, diverge, or remain indeterminate. In FinalBelief, the agent constructs a non-probabilistic belief object \(B_{s,t}\) specifying the leading candidate, secondary differential, supporting/contradicting evidence chains, and decision margins. If logical contradictions are detected between belief hypotheses and the underlying evidence state, a single bounded VerifyAndRepair cycle is executed, after which a deterministic operator \(\Pi\) projects the audited belief onto the benchmark label space.

Benchmark & Evaluation Protocol

LUMINA is evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort, comprising 1,512 visit sessions across 685 subjects. Standardized volumetric 3D T1-weighted MRI, amyloid PET (AV45), and tau PET (AV-1451) acquisitions are converted into unified 2.5D slice triplets (axial, coronal, and sagittal center slices) mapped to MNI152 space. The benchmark establishes two clinical prediction tasks: - AD-Continuum Classification (380 target subjects): A 3-way diagnosis task predicting the patient's current clinical stage: Cognitively Unimpaired non-AD (CU_nonAD), Preclinical AD (Preclinical_AD), and Symptomatic AD (Symptomatic_AD). - 36-Month Progression Forecasting (401 target subjects): A 3-way prognostic task predicting the worst held-out clinical diagnosis reached over a prospective 36-month follow-up window: Stable, Mild progression, or Clear progression. All future observations, follow-up dates, cognitive scores, and diagnosis fields are strictly withheld from model inputs.

Key Experimental Results

Main Results

Evaluations span 9 vision-language backbones, including open-weight Qwen3-VL models (Thinking and Instruct variants from 2B to 8B), domain-specific medical models (MedGemma-4B, MedVLM-R1-2B), and commercial proprietary GPT-4o-mini. Table 1 reports Balanced Accuracy (with 95% bootstrap confidence intervals) and Macro-F1 across all backbones.

Backbone Method AD-Continuum Bal. Acc. (%) AD-Continuum Macro-F1 (%) Progression Bal. Acc. (%) Progression Macro-F1 (%)
Qwen3-VL-2B-Thinking Best non-agent 66.03 ± 3.62 59.40 58.42 ± 5.10 50.84
Qwen3-VL-2B-Thinking MedAgent-Pro 72.94 ± 3.21 67.36 66.85 ± 4.55 57.16
Qwen3-VL-2B-Thinking LUMINA (Ours) 75.49 ± 3.08 71.58 69.05 ± 4.41 60.72
Qwen3-VL-4B-Thinking Best non-agent 69.42 ± 3.41 64.06 61.18 ± 4.86 54.82
Qwen3-VL-4B-Thinking MedAgent-Pro 77.28 ± 2.92 72.42 70.88 ± 4.18 61.34
Qwen3-VL-4B-Thinking LUMINA (Ours) 80.38 ± 2.68 77.10 73.52 ± 4.01 66.08
Qwen3-VL-8B-Thinking Best non-agent 72.06 ± 3.22 66.18 62.15 ± 4.72 57.96
Qwen3-VL-8B-Thinking MedAgent-Pro 79.75 ± 2.77 74.02 72.96 ± 3.93 64.80
Qwen3-VL-8B-Thinking LUMINA (Ours) 83.21 ± 2.39 79.42 76.42 ± 3.67 69.10
MedGemma-4B Best non-agent 70.82 ± 3.36 63.88 64.12 ± 4.61 56.22
MedGemma-4B MedAgent-Pro 77.85 ± 2.89 72.90 72.47 ± 4.13 64.22
MedGemma-4B LUMINA (Ours) 80.59 ± 2.63 76.54 74.89 ± 3.91 68.06
MedVLM-R1-2B Best non-agent 68.24 ± 3.63 62.14 59.28 ± 5.18 54.12
MedVLM-R1-2B MedAgent-Pro 75.11 ± 3.20 69.12 69.10 ± 4.46 59.94
MedVLM-R1-2B LUMINA (Ours) 80.03 ± 3.01 74.06 72.44 ± 4.27 62.58
GPT-4o-mini Best non-agent 74.36 ± 3.11 69.36 65.04 ± 4.55 59.24
GPT-4o-mini MedAgent-Pro 82.72 ± 2.55 77.28 74.02 ± 3.81 66.18
GPT-4o-mini LUMINA (Ours) 85.10 ± 2.28 81.46 77.68 ± 3.55 71.26

Ablation Study

To systematically determine the empirical contribution of each architectural component, ablations were conducted on AD-continuum classification using both GPT-4o-mini and Qwen3-VL-8B-Thinking (Table 2), complemented by an inference call-budget analysis (Table 3).

Ablation Variant GPT-4o-mini Bal. Acc. (%) GPT-4o-mini ΔBal. Acc. GPT-4o-mini Macro-F1 (%) Qwen3-8B-Think Bal. Acc. (%) Qwen3-8B-Think ΔBal. Acc. Qwen3-8B-Think Macro-F1 (%)
LUMINA (Full) 85.10 ± 2.28 — 81.46 ± 2.82 83.21 ± 2.39 — 79.42 ± 3.01
✗ Intra-modality anchor comparison 80.42 ± 2.91 -4.68 75.86 ± 3.62 76.83 ± 3.06 -6.38 73.64 ± 3.78
✗ Trajectory memory 81.73 ± 2.76 -3.37 77.92 ± 3.34 79.04 ± 2.92 -4.17 75.18 ± 3.55
✗ Cross-subject memory bank 82.44 ± 2.65 -2.66 78.16 ± 3.21 79.19 ± 2.78 -4.02 76.62 ± 3.36
✗ SMC dynamic compression 83.02 ± 2.55 -2.08 78.82 ± 3.16 80.16 ± 2.70 -3.05 76.04 ± 3.42
✗ Evidence verification/repair 83.92 ± 2.34 -1.18 81.12 ± 2.91 81.36 ± 2.47 -1.85 79.06 ± 3.08

Under a matched computational budget of 5.7 average LLM calls per subject, static self-consistency achieves 76.1% (classification) and 66.2% (progression), whereas LUMINA reaches 85.1% and 77.7% (+9.0 and +11.5 percentage points improvement), demonstrating that gains stem from structured evidence orchestration rather than repeated stochastic sampling.

Key Findings

  • Intra-modality anchor comparison is the primary driver: Disabling same-modality anchor comparison induces the largest performance degradation (-4.68% on GPT-4o-mini and -6.38% on Qwen3-VL-8B-Thinking), demonstrating that local temporal contrast across identical imaging modalities is the fundamental bedrock of longitudinal clinical perception.
  • SMC compression preserves disease-critical mass: Trajectory memory contributes the second-largest gain (-3.37% to -4.17%), while SMC importance weighting consistently outperforms naive recency-based truncation by dynamically prioritizing seminal pathological inflection points.
  • Robust cross-domain generalization: When evaluated without modification on external transfer cohorts with cross-subject retrieval disabled, LUMINA preserves its relative superiority over MedAgent-Pro and self-consistency on Parkinson's disease classification (PPMI, 3-class: 79.7% vs. 78.5% and 71.8%) and breast cancer neoadjuvant therapy complete response prediction (ACRIN-6698, binary: 83.9% vs. 79.3% and 71.4%).

Highlights & Insights

  • Reinterpreting SMC for Agentic Working Memory: Rather than applying Sequential Monte Carlo as a heavy particle filter over unconstrained token spaces, LUMINA cleverly adapts importance sampling as a discrete event-retention heuristic, solving long-context cognitive fatigue under a bounded token budget.
  • Strict Leakage-Controlled Protocol Design: The evaluation framework rigorously prevents target contamination by enforcing subject-disjoint external memory banks and isolating post-cutoff clinical scores, establishing an auditable standard for clinical reasoning agents.
  • Transferable Principles for Stateful Multi-Step Agents: For domains involving irregular time series, multi-sensor telemetry, or noisy longitudinal observations, decomposing the reasoning trajectory into "intra-source differential + utility-compressed trajectory + single-shot bounded audit" proves vastly superior to naive long-context prompting.

Limitations & Future Work

  • Spatial Resolution Loss in 2.5D Slice Triplet Representation: Operating on three orthogonal central slices ensures computational tractability and fair comparison across generic VLMs, but inherently overlooks peripheral lesions, focal micro-bleeds, and volumetric cerebral cortex quantification.
  • Prefix-Based Retrospective Setup: The progression benchmark evaluates reasoning over historical prefix records rather than prospective clinical trials, meaning missingness patterns and visit intervals reflect retrospective trial artifacts rather than real-world unconstrained care pathways.
  • Inference Latency in Real-Time Clinical Settings: With an average of 5.7 model calls and ~4 minutes of wall-clock latency per subject, LUMINA's multi-step verification may be computationally restrictive for high-throughput, latency-critical emergency triaging.
  • vs MLLM Bundle & Visit Sequence: Direct prompting over unordered or chronologically stacked image series suffers from severe context-window dilution and hallucinated temporal continuity; LUMINA decouples the process into explicit, auditable intermediate states.
  • vs MedAgent-Pro: MedAgent-Pro offers a general multi-step medical diagnostic workflow, but its linear action space lacks specialized intra-modality change anchors and multi-timescale memory compression, causing it to fall behind on long-term disease progression modeling.
  • vs See-in-pairs (Jin et al., 2025): While See-in-pairs employs paired healthy reference scans to stabilize static interpretation, LUMINA integrates reference calibration as a preliminary sensory read, directing its primary innovation toward longitudinal within-subject temporal dynamics.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers an explicit memory-and-control framework for longitudinal clinical reasoning with SMC-guided working memory compression]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluation across 9 MLLM backbones, extensive ablations, call-budget controls, and external transfer validation on PPMI and ACRIN-6698]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally rigorous formulation of clinical state spaces, leakage audits, and schema-constrained agent operations]
  • Value: ⭐⭐⭐⭐⭐ [Establishes a critical benchmark and practical architectural blueprint for deploying LLMs in long-term disease trajectory modeling]