Towards Ultrasound-based Reliable Disease Diagnosis Using Causal Inference¶

Conference: AAAI 2026 Paper: AAAI official page · PDF
Code: https://github.com/BoLeiChen/Ceusformer
Area: Medical Imaging
Keywords: multimodal ultrasound, causal inference, back-door adjustment, front-door adjustment, breast lesion diagnosis

TL;DR¶

Given a B-mode ultrasound image and a contrast-enhanced ultrasound video from the same case, the method uses reference feature dictionaries and causal-adjustment-inspired attention when two network branches exchange morphology and perfusion information, aiming to reduce reliance on incidental associations and reporting 89.80% benign/malignant classification accuracy on its dataset of 486 breast cases.

Background & Motivation¶

B-mode ultrasound (BUS) reveals lesion morphology, margins, and local echoes, whereas contrast-enhanced ultrasound (CEUS) video captures contrast-agent perfusion and time-varying microvascular appearances. Sonographers compare these sources of evidence, and existing dual-branch models emulate this process through regions of interest, brightness curves, or cross-modal attention. However, attending to a particular region does not establish that the model relies on transferable pathological evidence.

The paper focuses on feature preferences in limited datasets: conspicuous morphology is easy to exploit, while subtle texture and microvascular changes may be overlooked; patient composition, labeling, and acquisition sources can also alter feature-label associations. The authors therefore propose addressing these preferences within a structural causal model, rather than correcting only the final classification scores. Core Idea: when BUS and CEUS exchange information, reorganize transmitted features using dictionary attention inspired by back-door and front-door adjustment, while retaining temporal and local context to reduce reliance on incidental associations.

Method¶

Overall Architecture¶

The model combines static lesion morphology with dynamic contrast-agent perfusion while aiming to reduce feature preferences introduced during fusion. It takes one BUS image and a CEUS video from the same case, sampling 7 frames by default: BUS passes through a CNN pyramid, while CEUS passes through a Transformer with temporal attention (TA) and contextual attention (CA). Cross-branch interaction bridges exchange representations during encoding.

In the paper's Figure 4, Down sends BUS features into the CEUS branch, and Up sends CEUS features back into the BUS branch. Back-door adjustment BACL or front-door adjustment FACL reorganizes the transmitted features before concatenation with the receiving branch's local features. BACL and FACL are two adjustment mechanisms, both included in the full model; this does not establish a mandatory serial chain or a fixed assignment of one mechanism to each direction.

The final BUS representation incorporates CEUS information and feeds the benign/malignant classification head. Both branches also have lesion segmentation heads supervised by masks during training. The diagram unfolds multilevel interactions into one representative Down/Up exchange: intermediate stages are a dataflow schematic, not exact layer indices or recurrent iterations. Solid arrows show forward dataflow; dashed arrows show only inputs to training losses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
  BUS["BUS image"] --> CNN["Spatiotemporal Feature Extraction<br/>BUS: CNN"]
  CEUS["7 CEUS frames"] --> Transformer["Spatiotemporal Feature Extraction<br/>CEUS: TA/CA"]
  CNN --> Down["Cross-branch Interaction Bridges: Down<br/>Alignment + Back-door Adjustment BACL<br/>or Front-door Adjustment FACL"]
  Down --> CEUSLater["Concatenate local CEUS features<br/>Continue TA/CA encoding"]
  Transformer --> CEUSLater
  CEUSLater --> Up["Cross-branch Interaction Bridges: Up<br/>Alignment + Back-door Adjustment BACL<br/>or Front-door Adjustment FACL"]
  Up --> BUSLater["Concatenate local BUS features<br/>Continue CNN encoding"]
  CNN --> BUSLater
  BUSLater --> Diagnosis["Benign/malignant classification"]
  BUSLater --> BUSMask["BUS segmentation"]
  CEUSLater --> CEUSMask["CEUS segmentation"]
  Diagnosis -.-> Loss["Training only: classification<br/>and segmentation losses"]
  BUSMask -.-> Loss
  CEUSMask -.-> Loss
  Labels["Training labels and ground-truth masks"] -.-> Loss

Key Designs¶

1. Spatiotemporal Feature Extraction: make subtle cues available to the model

The two encoders have distinct roles. The BUS branch forms a feature pyramid from \(N\) ResNet-style convolutional blocks, decreasing resolution and increasing channel count with depth while retaining lesion boundaries and local texture. The CEUS branch processes video with \(N\) Transformer blocks containing TA, CA, and an MLP. TA applies self-attention across frames at each spatial patch: for example, it compares how a location brightens before and after contrast agent arrives, organizing separate frames into a representation of perfusion changes.

CA instead addresses how a location relates to surrounding tissue. A \(3\times3\) convolution first encodes keys into static context; these encoded keys are concatenated with queries, and two consecutive \(1\times1\) convolutions generate a multi-head attention matrix. Multiplying this matrix by values produces dynamic context, which is fused with the static representation. Unlike spatial attention over isolated query-key pairs, CA uses neighboring structure when generating weights, allowing subtle perilesional changes to participate in fusion. TA captures changes over time, while CA captures spatial context, supplying complementary features for subsequent adjustment.

2. Back-door Adjustment BACL: build dictionaries of explicitly extractable factors and aggregate them

BACL addresses the tendency for transmitted representations to be dominated by the most conspicuous features of the current case, introducing reference factor representations before transmission. Each modality has separate benign and malignant dictionaries: BUS factors include shape, margins, calcification, and echo types, while CEUS factors include enhancement types. Figure 3 depicts extracting and aggregating reference features separately by class. The two class dictionaries are then concatenated, projected by a fully connected layer, and processed with LayerNorm to obtain the dictionary representation used for querying.

The hidden representation of the transmitted feature acts as a query, computing similarities to dictionary entries. Softmax turns these similarities into weights for a weighted sum of mapped dictionary features. The module adds this aggregate to a linear mapping of the current feature, then applies a fully connected layer and LayerNorm to produce the adjusted feature for the bridge. Transmission therefore combines the current case representation with information aggregated from reference factors, rather than forwarding the original feature unchanged. The authors use an additive linear approximation to avoid expensive nonlinear integration over the joint input-factor space.

The causal motivation is back-door adjustment: if \(Z\) is an adjustment set satisfying the back-door criterion, averaging over factor values can block confounding along \(X\leftarrow Z\rightarrow Y\), replacing observational conditional weights \(P(z\mid X)\) with population marginal weights \(P(z)\):

\[ P(Y\mid do(X))=\sum_z P(Y\mid X,z)P(z). \]

The theoretical target should be distinguished from the implementation approximation: attention weights still depend on the current query, so this is back-door-inspired aggregation rather than a direct estimate of the formula's input-independent true \(P(z)\).

3. Front-door Adjustment FACL: separate current-input mediator aggregation from reference-dictionary aggregation

For texture styles and microvascular appearances that are difficult to enumerate, FACL combines representations from two sources. The mediator in the upper path of Figure 3 comes from the current transmitted feature \(x\) itself: a linear mapping produces attribute representations \(m\), and self-attention selects and aggregates these attributes into an input-dependent mediator summary. The "candidate mediators" are representations within the current feature that attention can aggregate, not an unexplained external sample pool or manually annotated lesion masks.

The lower path queries the reference dictionary \(x'\): another projection of the current feature supplies queries, while benign and malignant inconspicuous-factor dictionaries supply reference keys/values for cross-attention. The main text describes these dictionaries as medical-pretrained-model representations of BUS style texture and CEUS brightness changes. They store features associated with inconspicuous factors, rather than individually measured latent variables. The upper path selects attributes to retain from the current input; the lower path aggregates information from reference representations. Their outputs are added under the authors' linear approximation to produce the FACL-adjusted transmission feature. In Eq. (6), \(g_2\) queries mediators and \(g_1\) queries references; multi-head attention implements these aggregations.

This double aggregation corresponds to two averages in the front-door formula: one over mediators that the current input can produce, and another over the reference input distribution to adjust the mediator-output relationship:

\[ \begin{aligned} P(Y\mid do(X)) &=\sum_m P(m\mid X)\,P(Y\mid do(m)),\\ P(Y\mid do(m)) &=\sum_{x'}P(Y\mid m,x')P(x'). \end{aligned} \]

The formula requires the mediator to intercept all directed input-output paths, no unblocked back-door path from input to mediator, and input to block back-door paths from mediator to output. These are front-door identification conditions; the self-attention and cross-attention paths specify a computation but do not themselves verify those conditions.

4. Cross-branch Interaction Bridges: adjust features before fusion instead of only correcting outputs

The bridges connect CNN feature maps with Transformer representations, first addressing differences in channels, resolution, and numerical scale. Down goes from BUS to CEUS: Figure 4 converts CNN features through channel transformation, average pooling, reshaping, and LayerNorm into a form compatible with CEUS representations. After BACL or FACL adjustment, the transmitted representation is concatenated with local features on the Transformer side. Up goes from CEUS to BUS: reshaping, interpolation, channel transformations, and normalization match Transformer features to CNN feature maps; the adjusted representation is concatenated on the CNN side for subsequent convolution. The bridge design uses \(1\times1\) convolutions to match channels and LayerNorm and BatchNorm to coordinate feature scales.

Together, the bridges introduce BUS morphology into video modeling and CEUS perfusion representations into the BUS branch before final classification. Unlike correcting only final class scores, they change the inputs received by subsequent encoders. Auxiliary segmentation adds region-level supervision: BUS uses a four-layer CNN/upsampling decoder, while reshaping and one-dimensional convolution turn CEUS outputs into segmentation predictions. Ground-truth masks supply training supervision, not inference inputs or FACL mediator annotations.

A Worked Example¶

Consider a breast examination awaiting a benign/malignant prediction. The BUS image and 7 sampled CEUS frames are resized to \(384\times384\). The BUS encoder produces a feature map containing boundary and texture information; the CEUS encoder retains changes across frames at different locations and incorporates neighborhood context. Focus on one CEUS feature vector to be transmitted through the Up bridge into BUS, to see what adjustment changes.

If BACL is used at this point, the vector queries the CEUS conspicuous-factor dictionary. Solely to illustrate a weighted sum, suppose the reference pool has two entries with softmax weights of 0.7 and 0.3. The aggregate is 0.7 times the first mapped feature plus 0.3 times the second; it is a feature vector, not a "70% malignant" probability. Adding it to the current vector's linear mapping, then projecting and normalizing, produces a new representation to send to BUS. These weights and the dictionary size are teaching assumptions, not experimental data from the paper.

For FACL, the two aggregation targets differ: the upper path applies self-attention to attributes mapped from the current CEUS features, while the lower path uses current features to query the CEUS inconspicuous-factor reference dictionary, then adds the summaries. Whichever adjustment is being examined, the bridge output is concatenated with local BUS features rather than replacing their morphology representation. Both benign and malignant dictionaries participate in querying, without requiring the true case label at test time. The final BUS classification head uses the fused representation for diagnosis, while both segmentation heads predict lesion regions.

Loss & Training¶

Diagnosis uses binary cross-entropy; segmentation uses cross-entropy, Dice, and IoU losses with weights of 1:1:1. Benign/malignant labels supervise the classification head, while ground-truth masks for both modalities supervise the segmentation heads. A cross-validation reproduction should construct or update dictionaries using only the training fold, preventing validation cases and labels from entering the reference pool.

The implementation uses PyTorch, one NVIDIA GeForce RTX 3090, Adam, a batch size of 4, 50 epochs, and learning rate \(10^{-4}\); the CEUS embedding dimension is 384 with 8 attention heads. Augmentation includes horizontal flipping and rotation. The default 7 frames and \(384\times384\) input resolution are selected through the parameter study. In unimodal experiments, the same modality is fed into both branches; those results do not represent retaining only one branch.

Key Experimental Results¶

Main Results¶

The self-constructed BUS-CEUS dataset contains 486 different patients, including 237 benign and 249 malignant cases, with biopsy-derived diagnostic labels and dual-modality segmentation masks supplied by sonographers. Both modalities were acquired simultaneously using a Mindray Resona R9, with original image size \(550\times582\); the main text identifies the source only as one unnamed hospital. Evaluation uses a 4:1 training/validation split and five-fold cross-validation, and mentions paired Student's t-tests (\(p<0.05\)).

The following values reproduce Table 3 on page 2783 without correction. All metrics are percentages; the reported ± values are not reinterpreted as confidence intervals.

Method	Acc	Sens	Spec	F1
TSDBN	82.22	80.72	68.42	79.77
DKG	85.31	79.02	72.27	85.03
Huang et al.	82.41	76.76	63.55	81.77
AHAF	84.53±2.72	83.37±1.59	83.90±2.07	79.47±1.93
KAMnet	85.42±2.82	74.19±2.67	89.04±2.83	83.97±2.03
DaST	86.28±2.61	82.79±1.97	79.81±2.73	82.53±1.82
UAC-T	86.33±2.81	81.62±2.49	92.36±3.43	87.28±2.79
Ours	89.80±2.58	85.08±1.80	95.83±3.04	89.36±2.00

Arithmetic within the table gives gains of 3.47 percentage points in Acc and 2.08 in F1 over UAC-T. However, rows including TSDBN, DKG, Huang et al., and DaST report Acc above both Sens and Spec, conflicting with the relationship between standard binary metrics from the same confusion matrix. Cross-fold aggregation can affect that relationship, but the main text provides insufficient detail to explain the discrepancies. The original values are retained, and these gains are treated as paper-reported results.

Unimodal evidence appears in Tables 1 and 2 on the same page: BUSI uses 647 abnormal breast images and reports Acc of 91.6±1.8 for this method; SYSU-FLL-CEUS contains 353 focal liver lesion cases and reports Acc of 94.2±1.6. These are evaluations within their respective datasets, not external cross-disease validation of a breast-trained model directly transferred to liver lesions.

Ablation Study¶

The following results come from Table 5. Every configuration retains TA; the baseline uses ordinary spatial attention before CA is introduced. No standard deviations are listed for these ablations.

Config	Acc	Sens	Spec	F1
Baseline: TA	79.25	74.61	84.17	78.83
TA + CA	83.67	78.34	88.83	82.60
TA + CA + B-Down	84.39	79.67	89.01	83.96
TA + CA + B-Down + B-Up	85.03	80.90	90.83	84.79
Above dual bridges + FACL	88.37	83.83	94.16	88.40
Above dual bridges + BACL	89.07	84.28	94.77	88.96
Above dual bridges + BACL + FACL	89.80	85.08	95.83	89.36

Key Findings¶

CA improves Acc by 4.42 percentage points over the baseline; adding both bridges on top of CA improves it by 1.36 points.
Relative to dual bridges without adjustment, FACL alone improves Acc by 3.34 points, BACL alone by 4.04, and both together by 4.77. Their combined gain is not the sum of the two individual gains.
Figure 5 qualitatively supports 7 frames and \(384\times384\) inputs; Table 4 reports 63.41M parameters and 48.28G FLOPs. These complexity measures do not substitute for measured clinical latency.

Highlights & Insights¶

Placing adjustment at cross-modal transmission can influence subsequent feature extraction instead of changing only final scores. This architectural choice is relatively straightforward to transfer to other imaging-fusion networks.
Separating conspicuous and inconspicuous factors offers a framework for examining information sources. Nevertheless, effective dictionary attention and valid causal identification remain separate propositions requiring separate tests.

Limitations & Future Work¶

Causal evidence boundary: Classification gains support module effectiveness on the evaluated data, but do not establish genuine causal identification or clinical reliability. Morphology and enhancement may also be consequences of disease, so treating them as confounders is a modeling assumption. Controlled distribution shifts, intervention data, or identification arguments are needed to test the causal graph, mediator conditions, and dictionary sufficiency.
Clinical validation scope: A single-center dataset of 486 cases and random cross-validation do not replace cross-center validation, probability calibration, or prospective clinical evaluation.
Internal metric discrepancies require per-fold confusion matrices, metric implementations, and definitions of uncertainty before between-method differences can be interpreted reliably.

Reading and reproduction boundaries: This review checked the main text and Figures 3 and 4, but not supplementary material, code, or data downloads. Dictionary sizes, pretrained models, sampling and update rules, per-fold isolation, complete loss weighting, and some tensor transformations still require confirmation from supplementary material or implementation. This describes the material available for this note, not necessarily omissions from the paper. The BACL paragraph calls a group of CEUS factors "inconspicuous," conflicting with the earlier categorization and module role; this wording ambiguity is retained.

vs DaST / UAC-T: These methods also exploit multimodal feature interaction; this paper emphasizes adding BACL/FACL during feature transmission rather than only emulating the regions or perfusion patterns attended to by sonographers.
vs HoVer-Trans / Trans-CEUS: The former incorporates anatomical knowledge and the latter uses dynamic perfusion representations. This paper additionally combines local context with adjustment-inspired fusion, but retains two branches even in unimodal comparisons.

Rating¶

These are the reader's subjective assessments of the paper: 3 means adequate, 4 strong, and 5 outstanding; half points are allowed. Writing Quality refers to the original paper, and Value concerns research contributions and potential, not clinical readiness.

Novelty: 4.5/5. Embedding front-door- and back-door-inspired modules in bidirectional feature bridges provides a clear contribution to ultrasound fusion, primarily through task-specific integration of established causal-adjustment and attention mechanisms.
Experimental Thoroughness: 4/5. Multiple datasets, unimodal and multimodal comparisons, and component ablations support classification gains, but do not directly establish causal identification; internal metric discrepancies also weaken the evidence.
Writing Quality: 4/5. The dual-branch and module diagrams clarify the implementation, while confounder categories, the correspondence between causal formulas and network approximations, and metric definitions remain ambiguous.
Value: 4.5/5. Intermediate feature adjustment and cross-modal interaction are worth reproducing, and the breast dataset has research value; the work currently supports methodological exploration, with clinical generalization requiring independent validation.