---

title: >- [Paper Note] ELDiff: When Evidential Learning Meets Text-to-Image Diffusion description: >- [ECCV2026][Image Generation][Evidential Learning] Introduces evidential deep learning and Dempster-Shafer theory to address segmentation bias and semantic overlap conflicts in text-to-image diffusion models. tags: - ECCV2026 - Image Generation - Evidential Learning - Text-to-Image Diffusion - Multi-Object Composition date: 2026-09-19 content_hash: a0acd2e23ff2b6e2

ELDiff: When Evidential Learning Meets Text-to-Image Diffusion

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/QingtaoPan/ELDiff
Area: Image Generation
Keywords: text-to-image diffusion, evidential deep learning, Dempster-Shafer theory, multi-object composition, cross-attention supervision

TL;DR

By coupling evidential deep learning (EDL) for uncertainty estimation with Dempster-Shafer theory (DST) for semantic conflict quantification, ELDiff addresses segmentation bias and overlap conflicts in multi-object text-to-image diffusion without any inference-time overhead.

Background & Motivation

Text-to-image (T2I) diffusion models have demonstrated remarkable generative fidelity across diverse domains. However, when prompts involve multiple distinct objects and intricate spatial relationships, standard models struggle to synthesize coherent scenes. Because their denoising objective predicts noise conditioned on the full text prompt as a single global embedding, models frequently suffer from concept omission, attribute bleeding, and spatial hallucination due to the lack of explicit local alignment between individual linguistic tokens and visual regions.

To alleviate this compositionality bottleneck, token-supervised diffusion paradigms such as TokenCompose have emerged. These methods utilize automated detection and segmentation models (e.g., Grounded-SAM) to extract binary masks for each noun token and enforce pixel-level consistency on cross-attention maps during fine-tuning. While effective, this supervisory regime suffers from two fundamental flaws: first, pseudo-ground-truth segmentation masks generated by SAM inevitably carry segmentation map bias, false positives, and imprecise boundaries, causing the diffusion model to overfit to spurious supervisory regions; second, supervising each noun token independently presupposes that concepts are spatially disjoint, which triggers severe semantic overlap conflicts when multiple entities interact or physically overlap (e.g., "a cat on a chair"), producing contradictory gradient updates that compromise generative quality.

Addressing both pseudo-label noise and cross-token competition requires an uncertainty-aware supervisory mechanism that can dynamically downweight unreliable label regions while resolving multi-concept spatial tension. The core idea is to map cross-attention scores to Dirichlet evidence distributions for pixel-level uncertainty estimation, and leverage the Dempster-Shafer evidence combination rule to explicitly quantify and penalize inter-token semantic conflicts during diffusion fine-tuning.

Method

Overall Architecture

ELDiff introduces an end-to-end fine-tuning framework that replaces deterministic hard-mask alignment with evidential uncertainty estimation and conflict-aware cross-attention optimization. During training, input prompts and noisy latents are fed into the diffusion backbone (e.g., U-Net), which computes cross-attention maps \(A\) corresponding to the noun tokens in the prompt. Offline binary segmentation maps \(M\) from SAM serve as the guiding targets.

The architecture comprises two complementary modules: the Pixel Evidence Loss (\(\mathcal{L}_{\text{PixEvi}}\)) routes attention values through an evidential network to build a Dirichlet distribution, quantifying epistemic uncertainty to prevent overconfidence on biased pseudo-masks; and the Token Conflict Loss (\(\mathcal{L}_{\text{TokCon}}\)) aggregates per-noun evidence via Dempster-Shafer theory (DST) to detect and penalize conflicting activations in overlapping regions while compacting intra-object attention. At inference time, ELDiff operates as a standard diffusion model with zero test-time overhead and no requirement for segmentation masks.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt + Noisy Latent z_t"] --> B["Diffusion Backbone U-Net Denoising<br/>Extract noun token cross-attention maps A"]
    B --> C["1. Pixel Evidence Loss: Mitigating Segmentation Bias<br/>MLP evidence mapping + Dirichlet Subjective Logic"]
    B --> D["2. Token Conflict Loss: Resolving Semantic Overlap<br/>DST evidence aggregation + conflict factor optimization"]
    C --> E["Joint Optimization: L_LDM + L_PixEvi + L_TokCon"]
    D --> E
    E --> F["Inference Synthesis<br/>Zero extra latency / No masks needed"]

Key Designs

1. Pixel Evidence Loss: Subjective Logic-Based Pixel Uncertainty Metric To prevent diffusion cross-attention from overfitting to biased or noisy boundaries in automatically generated SAM masks, this design reinterprets cross-attention scores as class evidence rather than deterministic predictions. For each noun cross-attention map \(A\), an MLP evidence network \(f_\phi(\cdot)\) with Softplus activation maps attention values to non-negative evidence vectors \(e_{i,j}\). These evidence values parameterize a Dirichlet distribution \(\mathcal{D}(p \mid \alpha)\) with parameters \(\alpha_{i,j}^c = e_{i,j}^c + 1\). Following Subjective Logic, the total Dirichlet strength \(S = \sum_{c=1}^C (e_{i,j}^c + 1)\) determines the allocated belief mass \(b_{i,j}^c = \frac{e_{i,j}^c}{S}\) and epistemic uncertainty \(u_{i,j} = \frac{C}{S}\).

The supervisory objective computes the expected cross-entropy under the predicted Dirichlet distribution, supplemented by a Kullback-Leibler (KL) divergence regularizer that suppresses evidence for incorrect classes:

\[\mathcal{L}_{\text{ice}} = \sum_{c=1}^C y^c \left( \psi(S) - \psi(\alpha^c) \right)\]

where \(\psi(\cdot)\) denotes the digamma function and \(y^c\) is the binary pseudo-mask label. In ambiguous or mislabeled border regions, the evidential network outputs a dispersed Dirichlet distribution corresponding to high uncertainty, preventing excessive gradient penalties and shielding the backbone from pseudo-label corruption. The overall pixel evidence loss \(\mathcal{L}_{\text{PixEvi}}\) combines standard cross-entropy, expected cross-entropy, and the KL divergence penalty.

2. Token Conflict Loss: Dempster-Shafer Combination Rule for Overlap Resolution To eliminate the semantic contradiction caused by supervising overlapping entities independently, this design leverages Dempster-Shafer Evidence Theory (DST) to quantify and decouple inter-token competition. For each noun token \(n\), basic belief assignments are defined as \(b^n = \gamma^n A^n \cdot M^n\), where \(\gamma^n \in (0, 1)\) is a learnable scaling factor. For any two interacting nouns \((v, w)\), their conflict coefficient in the overlapping spatial region \(M^v \cap M^w\) is calculated as:

\[K_{i,j}^{v,w} = b_{i,j}^v \cdot b_{i,j}^w \quad \left(\text{for } (i, j) \in M^v \cap M^w\right)\]

An inter-object conflict loss penalizes high-conflict points in overlapping zones, while an intra-object consistency loss encourages attention to concentrate inside the true target regions:

\[\mathcal{L}_{\text{inter}} = \sum_{(i,j) \in M^v \cap M^w} \frac{K_{i,j}^{v,w}}{1 - K_{i,j}^{v,w}} \cdot \sigma(K_{i,j}^{v,w} - \tau), \quad \mathcal{L}_{\text{intra}} = \sum_n (1 - b^n)^2\]

where \(\sigma\) is the Sigmoid activation and \(\tau\) denotes the conflict tolerance threshold. The non-linear penalty factor \(\frac{K}{1-K}\) escalates rapidly when both concepts claim high confidence over the same pixel, compelling the model to negotiate distinct spatial boundaries and decoupling entangled representations.

Loss & Training

The overall training loss during fine-tuning combines the standard latent diffusion loss with the two evidential objectives:

\[\mathcal{L}_{\text{ELDiff}} = \mathcal{L}_{\text{LDM}} + \mathcal{L}_{\text{PixEvi}} + \mathcal{L}_{\text{TokCon}}\]

Hyperparameters for \(\mathcal{L}_{\text{PixEvi}}\) are set to \(\lambda_1 = 5 \times 10^{-5}\), \(\lambda_2 = 5 \times 10^{-7}\), and an annealed \(\lambda_3 = 5 \times 10^{-7} \cdot \min\{1, n_{\text{epoch}}/100\}\); for \(\mathcal{L}_{\text{TokCon}}\), \(\eta_1 = \eta_2 = 1 \times 10^{-4}\). Fine-tuning is conducted on a single NVIDIA GeForce RTX 3090 GPU with the AdamW optimizer, learning rate \(5 \times 10^{-6}\), batch size 1 with 4 gradient accumulation steps, running for 24,000 steps (approximately 22 hours). Segmentation masks are precomputed once offline, requiring zero mask processing during diffusion training or inference.

Key Experimental Results

Main Results

On Stable Diffusion v1.4, ELDiff demonstrates comprehensive superiority across multi-category composition, image fidelity, and efficiency (data from original Table 1; C denotes COCO, F denotes Flickr30K):

Model (SD v1.4)	OA↑	MG3↑	MG4↑	MG5↑	FID(C)↓	FID(F)↓	CLIP(C)↑	Latency↓
SD v1.4 Baseline	0.2986	0.5074	0.1148	0.0088	22.06	57.24	0.3022	7.54s
Attend-and-Excite	0.4513	0.6510	0.2801	0.0601	21.57	57.05	0.3018	25.43s
CoMat	0.4732	0.7044	0.2566	0.0211	22.43	58.49	0.3122	7.54s
IterComp	0.4891	0.6742	0.2487	0.0167	21.68	57.66	0.3047	7.54s
TokenCompose	0.5215	0.7616	0.2881	0.0328	21.12	56.93	0.3112	7.56s
ELDiff (Ours)	0.5563	0.8041	0.3305	0.0931	19.25	55.13	0.3276	7.54s

When deployed on more advanced backbones including SD v2.1 and SD v3.5 Medium, ELDiff consistently establishes new state-of-the-art benchmarks (data from original Table 3):

Backbone & Fine-tuning	OA↑	MG3↑	MG4↑	MG5↑	FID(C)↓	CLIP(C)↑
SD v2.1 Frozen	0.4728	0.7014	0.2557	0.0327	21.79	0.3045
SD v2.1 + TokenCompose	0.6010	0.8048	0.3669	0.0571	20.77	0.3209
SD v2.1 + ELDiff (Ours)	0.7116	0.8870	0.5401	0.1392	20.09	0.3241
SD v3.5 Frozen	0.8084	0.9996	0.9696	0.7328	18.21	0.3185
SD v3.5 + TokenCompose	0.8233	0.9997	0.9839	0.7569	18.05	0.3198
SD v3.5 + ELDiff (Ours)	0.8644	0.9997	0.9964	0.8171	17.66	0.3247

Ablation Study

Ablations on SD v1.4 systematically isolate the contribution of each evidential objective (data from original Table 5):

Configuration	OA↑	MG3↑	MG4↑	MG5↑	FID(C)↓	CLIP(C)↑
SD v1.4 Baseline	0.2986	0.5074	0.1148	0.0088	22.06	0.3022
+ \(\mathcal{L}_{\text{LDM}}\) (Standard Fine-tuning)	0.3821	0.6372	0.1956	0.1897	24.57	0.3028
+ \(\mathcal{L}_{\text{LDM}} + \mathcal{L}_{\text{PixEvi}}\)	0.5466	0.7648	0.2934	0.3361	21.17	0.3178
+ \(\mathcal{L}_{\text{LDM}} + \mathcal{L}_{\text{PixEvi}} + \mathcal{L}_{\text{TokCon}}\) (Full Model)	0.5563	0.7660	0.2962	0.3530	20.84	0.3204

Key Findings

• Pixel evidence loss drives the foundational gain: Introducing \(\mathcal{L}_{\text{PixEvi}}\) improves Object Accuracy (OA) from 0.3821 to 0.5466. By dynamically capturing uncertainty across noisy mask boundaries, the model preserves cross-attention guidance while avoiding negative transfer from incorrect pseudo-labels.
• Token conflict loss unlocks high-order composition: Adding \(\mathcal{L}_{\text{TokCon}}\) boosts high-order multi-object generation (e.g., MG5 reaching 0.3530) and lowers FID from 21.17 to 20.84, demonstrating that DST conflict suppression resolves spatial concept collision and eliminates object merging.
• Zero inference-time overhead: Unlike training-free test-time guidance methods such as Attend-and-Excite (25.43s per image), ELDiff internalizes evidential constraints purely into network weights during fine-tuning, preserving the baseline generation speed (7.54s).

Highlights & Insights

• Cross-pollinating Evidential Deep Learning into T2I diffusion: Repurposing Dirichlet-based evidential modeling to estimate pixel-level uncertainty on cross-attention maps provides an elegant, principled solution to noisy pseudo-label supervision.
• Formalizing semantic conflict via Dempster-Shafer Theory: Modeling overlapping cross-attention regions as multi-source evidence conflicts and optimizing a non-linear DST penalty factor \(\frac{K}{1-K}\) offers a robust mechanism for spatial concept decoupling.
• Strong cross-architecture generalization: ELDiff delivers consistent gains across SD v1.4, SD v2.1, SDXL, SD v3.5, and the modern Qwen-Image backbone, excelling on both GenEval 2 and T2I-CompBench++.

Limitations & Future Work

• Lack of explicit attribute binding constraints: The authors acknowledge that while multi-object presence and count accuracy improve substantially, fine-grained attribute binding (binding complex color, shape, and texture to specific objects) remains suboptimal without targeted attribute-level loss terms.
• Dependence on initial detection recall: When Grounded-SAM suffers from complete detection misses or category misclassification, the evidential loss cannot reconstruct missing concepts from scratch.
• Future directions: Extending evidential formulation to joint attribute-object modeling and applying it to video and 3D generative diffusion frameworks.

Related Work & Insights

• vs TokenCompose: TokenCompose pioneered token-level cross-attention fine-tuning with Grounded-SAM masks, but treated them as ground truth and supervised tokens independently; ELDiff tackles its inherent mask bias and overlap conflicts via EDL uncertainty and DST conflict mitigation.
• vs Attend-and-Excite / SynGen: These methods rely on test-time optimization during the reverse diffusion trajectory, incurring severe latency penalties; ELDiff achieves superior compositionality via training-time supervision while retaining standard inference latency.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First principled framework to integrate evidential learning and DST conflict quantification into text-to-image diffusion.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across five major backbones (SD v1.4, v2.1, SDXL, SD v3.5, Qwen-Image), encompassing VISOR, T2I-CompBench++, GenEval 2, and user studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation with clear transitions from subjective logic to training loss functions.
• Value: ⭐⭐⭐⭐⭐ Provides a generalizable, zero-inference-overhead methodology for noisy weakly-supervised multimodal alignment.

Related Papers

• [ECCV 2026] Distribution Matching Distillation Meets Reinforcement Learning
• [ICML 2025] DDIS: When Model Knowledge Meets Diffusion Model
• [ECCV 2026] Physics Meets Perception: A Reinforcement Learning Framework for Unpaired Real-World Image Dehazing
• [ICML 2026] Divide and Conquer: Reliable Multi-View Evidential Learning for Deepfake Detection
• [CVPR 2026] When Anonymity Breaks: Identifying Models Behind Text-to-Image Leaderboards