Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representations

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/antoniopurificato/artistic_sheaf
Area: Multimodal VLM
Keywords: Art understanding, relation conditioning, sheaf theory, FiLM, graph-regularized contrastive learning

TL;DR

CANVAS modulates each artwork or text into relation-specific representations and uses a training graph's line-graph heat kernel to soften false-negative penalties, achieving the strongest image-to-text retrieval results across three art datasets without requiring full graph connectivity at inference.

Background & Motivation

A painting can support descriptions of its visible content, artistic style, and historical period, but these descriptions express different kinds of similarity. CLIP aligns all such relationships in one image-text embedding space, potentially forcing historical association and visual content into the same distance function. Its contrastive objective also treats unmatched samples as negatives, even when they are different descriptions of the same work or involve artworks sharing an artist or style. The issue is not merely a shortage of captions: the model lacks an explicit account of what a comparison is meant to compare.

Graph-augmented methods can introduce artist, date, and other contextual knowledge, but some approaches discussed in the paper depend on graph connections and associated annotations at test time. If a new artwork must first be annotated and inserted into a graph, retrieval is no longer an inductive operation on an independently supplied image or text. CANVAS separates these responsibilities: the training graph identifies related pairs, while the encoder learns to express that knowledge through independently computable, relation-conditioned representations.

Core idea: use content-and-relation-conditioned FiLM maps to change how items are compared, and turn structural proximity in the training graph into soft contrastive targets, placing relational knowledge in model parameters rather than requiring test-time graph connections.

Method

Overall Architecture

Inputs are artwork images, descriptive texts, and the relation types linking them. CANVAS constructs a typed image-text bipartite graph, encodes content and relation names with CLIP, and uses sheaf-inspired relation-conditioned FiLM to produce distinct views of each node. A separate training-supervision branch converts original graph edges into line-graph nodes and diffuses affinity with a heat kernel; these soft targets supervise cosine similarities between the relation-conditioned views.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Artworks, texts<br/>relation labels"] --> B["Typed bipartite graph"]
    B --> C["CLIP content encoding<br/>frozen relation encoding"]
    C --> D["Relation-conditioned FiLM"]
    B --> E["Line-graph heat-kernel<br/>soft supervision"]
    D --> F["Bidirectional contrastive<br/>and KL training"]
    E --> F
    F --> G["Relation-specific views<br/>retrieval and classification"]

Inference runs content encoding and relation-conditioned transformations without neighbor aggregation or access to the training graph. However, graph-free does not mean relation-free: a relation-aware comparison still needs a choice of relational view. The main text says relation types are optional and can be inferred from text, but the actual procedure points to an appendix absent from the cache. It would be unjustified to invent a relation predictor from that statement.

Key Designs

1. Typed bipartite graph: make an artwork-description relationship the training unit

Images occupy one side of the graph and texts the other; each edge carries a relation name such as content, context, or style. One image can connect to multiple texts, and a text can participate in multiple relationships. Training therefore operates on typed image-text associations rather than a fixed list of one-image-one-caption matches. CLIP image and text encoders initialize node features. Relation names also pass through a text encoder, but this relation-encoding branch remains fully frozen so that the conditioning signal does not drift together with content fine-tuning.

The consequential choice is edge-indexed computation: the same image receives separate outputs on its style and date edges. This retains a shared CLIP backbone while telling the next module what the comparison is about. The bipartite graph supplies training pairs and structural supervision; it is not a test-time adjacency list for GNN propagation. Relation-label quality consequently affects both the representation conditions and the learning targets.

2. Relation-conditioned FiLM: compare the same content in different relational coordinates

In sheaf theory, a stalk is a local vector space attached to a node or edge, and a restriction map transports node information into an edge space. CANVAS borrows the idea of comparing items after entering a relation-specific space, but implements it with FiLM rather than a complete sheaf-diffusion GNN. For either an image or a text endpoint, its feature vector is concatenated with the relation embedding and passed through a three-layer MLP with LeakyReLU activations to produce feature-wise scales and shifts. The notation below uses node for either endpoint, making explicit that each side's parameters depend on its own content; it restores the equations using the surrounding explanation:

\[ (\gamma_{node,e},\beta_{node,e})=\phi([h_{node}\oplus h_{r_e}]),\qquad \widetilde h_{e,node}=\gamma_{node,e}\odot h_{node}+\beta_{node,e}. \]

Scaling can suppress dimensions irrelevant to the requested relation, while shifting relocates the view within a relation-specific subspace. Because the map depends on both content and relation, it is not just a fixed linear head for each relation, nor is it strictly the fixed linear restriction map introduced in the mathematical preliminaries. The model stacks 3 layers and averages their outputs to obtain the final view. For a stack of \(N\) layers, the averaging rule is:

\[ \widetilde h^{out}_{e,node}=\frac{1}{N}\sum_{n=1}^{N}\widetilde h^{(n)}_{e,node}. \]

This computation never aggregates neighboring artworks or texts. Given content and a relation, a view can be computed independently; graph information acts through learned parameters rather than test-set adjacency in the forward pass. That separation directly enables processing new artworks not inserted into the training graph.

3. Line-graph heat-kernel soft supervision: related pairs should not be fully unrelated negatives

Relation-specific views alone do not fix the learning target: ordinary InfoNCE can still push apart other relational pairs involving the same artwork. CANVAS constructs the line graph \(G'=L(G)\), where each original image-text relation edge becomes a node. Two line-graph nodes connect when the original edges share an image or text endpoint. Relationships between training pairs thus become an explicit structure over which affinity can diffuse. The method computes a heat kernel from the combinatorial Laplacian \(L_{G'}=D'-A'\), applies an element-wise power, and normalizes each row:

\[ W=\exp(-\tau L_{G'}),\qquad \widetilde W_{ij}=\frac{(W_{ij})^\alpha}{\sum_k(W_{ik})^\alpha},\qquad P=\lambda\widetilde W+(1-\lambda)I. \]

The exponential is a matrix exponential. Diffusion extent is controlled by \(\tau\), distribution sharpness by \(\alpha\), and the mixture of graph proximity and strict pair identity by \(\lambda\). Different descriptions of a shared artwork can receive nonzero targets instead of reserving all positive mass for the diagonal. This does not declare every same-style pair equivalent to an exact match: the identity component remains, and the heat kernel reflects graph connectivity rather than art-historical truth itself. Incorrect or overly dense connections can therefore create inappropriate soft positives.

A Worked Example

This is an illustrative mechanism walkthrough, not an additional experiment. Suppose one painting connects to a content description and a historical-context description, producing 2 typed edges. During encoding, the image combines separately with the content and context relation embeddings, producing 2 views. Each text is likewise modulated under its own relation, so historical text need not match the image solely through depicted objects.

The 2 edges share an image and become neighbors in the line graph, allowing the heat kernel to assign them structural affinity. Training preserves each pair's identity alignment while reducing the penalty for treating the other pair as a negative. The same relational views can be computed independently for a new painting outside the graph. How to choose a view when a user supplies no relation is insufficiently specified in the cache and should not be presented as a fully verified automatic procedure.

Loss & Training

The final image-relation and text-relation matrices are row-wise L2-normalized before bidirectional cosine similarities are computed. Training combines bidirectional InfoNCE with a bidirectional KL term based on \(P\), preserving exact pairing and structural proximity respectively. From the prose describing the weights, the overall objective can be restored as:

\[ \mathcal L=(1-\eta)\mathcal L_{CLIP}+\eta\mathcal L_{KL}. \]

The cached extraction damages the KL argument separators and probability-normalization details in equation (11), as well as some operators in equation (12). KL requires probability distributions; a raw cosine-similarity matrix cannot simply be treated as one. This note therefore retains only the text-supported meaning of aligning soft targets in both directions, without inventing a temperature, softmax axis, or reverse-target renormalization rule. The FiLM, averaging, heat-kernel, and mixed-target equations above likewise restore mathematical formatting from context rather than assume intact equation extraction.

The backbone is OpenCLIP ViT-B/32 pretrained on LAION-2B. The final 3 Transformer blocks and projection heads of both content encoders are unfrozen; earlier parameters remain frozen, as does the entire relation encoder. The latent dimension is 512, the three-layer FiLM MLP has hidden widths of 512 and 256, and the stack contains 3 layers. AdamW uses a learning rate of \(10^{-5}\) and batch size 256. The training budget is 50 epochs, gradient-norm clipping is set to 1.0, and early stopping uses a patience of 5 epochs on validation loss. The validation-best model is used for testing.

The reported settings are \(\tau=0.7\), \(\alpha=1.2\), \(\lambda=0.7\), and \(\eta=0.5\). Computational cost is referred to Appendix A.4, which is absent from the cache, so added memory, latency, and heat-kernel preprocessing costs cannot be quantified here.

Key Experimental Results

Main Results

The three datasets are the newly public institutional collection HertzianaDP, SemArt+ built from SemArt with sentence-level annotations and cleaning, and WikiArt+ enriched with Wikipedia explanations. The main text states that splitting is item-disjoint to prevent leakage, but exact partition details are in the missing Appendix A.2. HertzianaDP was not publicly online during CLIP pretraining, making it the authors' primary testbed with less potential pretraining leakage. This is not an experiment that trains on one dataset and performs zero-shot transfer to another.

The table selects Recall@10 from the paper's Table 1. Values are fractions, and higher is better. Following the paper's definition, Recall@K is the fraction of queries with at least one correct match in the top K results. T2I denotes text-to-image retrieval; I2T denotes image-to-text retrieval.

Dataset	Method	T2I R@10	I2T R@10
HertzianaDP	CLIP-ft	0.069	0.084
HertzianaDP	SigLIP	0.083	0.022
HertzianaDP	SigLIP-ft	0.058	0.076
HertzianaDP	CANVAS	0.158	0.514
SemArt+	CLIP-ft	0.216	0.131
SemArt+	SigLIP	0.423	0.133
SemArt+	SigLIP-ft	0.446	0.150
SemArt+	CANVAS	0.489	0.714
WikiArt+	CLIP-ft	0.149	0.158
WikiArt+	SigLIP	0.301	0.091
WikiArt+	SigLIP-ft	0.307	0.209
WikiArt+	CANVAS	0.217	0.781

Against the strongest I2T R@10 baseline in the original table, CANVAS improves by 0.430, 0.564, and 0.572 across the three datasets, corresponding to 43.0, 56.4, and 57.2 percentage points. T2I tells a different story: on WikiArt+, 0.217 trails SigLIP-ft's 0.307 by 9.0 percentage points. The method is not best in every retrieval direction.

Ablation Study

The paper's Table 2 performs component ablations only on HertzianaDP. All four retrieval metrics from this compact table are retained below.

Configuration	T2I R@5	T2I R@10	I2T R@5	I2T R@10
CANVAS	0.099	0.158	0.386	0.514
InfoNCE only	0.002	0.004	0.525	0.541
KL only	0.098	0.156	0.385	0.490
No relation modulation	0.002	0.003	0.001	0.002
Simple multi-head projections	0.021	0.040	0.140	0.170

Removing relation modulation drops I2T R@10 from 0.514 to 0.002; replacing it with simple multi-head projections yields 0.170. These are direct evidence for content-dependent modulation. However, InfoNCE alone raises I2T R@10 to 0.541 while collapsing T2I R@10 to 0.004. The combined objective offers a bidirectional compromise, not superiority over every ablation on every metric. KL alone already reaches T2I R@10 of 0.156, close to the full model's 0.158.

Key Findings

• I2T and T2I difficulty cannot be compared directly. The authors note that an image can have multiple valid textual matches; the HertzianaDP test split contains 6,874 images and 12,145 texts, and this candidate and positive-match structure affects the directional gap.
• Relation information is not supplied identically to all baselines. CANVAS textual prompts contain relation information, whereas CLIP, SigLIP, and MSC do not receive relation types. With the automatic relation-inference details missing, the comparison should not be described as a pure architectural gain under identical inputs.
• Classification Table 3 contradicts the prose claim that CANVAS ranks first or second on every task. For HertzianaDP Artist, CANVAS scores 0.472, below ArtSAGENet at 0.808, SigLIP-ft at 0.650, and GraphCLIP at 0.588. For WikiArt+ Period, CANVAS scores 0.485, below CLIP-ft at 0.546 and GraphCLIP at 0.522. Results should be interpreted per attribute rather than through that blanket claim.

Highlights & Insights

• Putting relations into both representations and supervision addresses two distinct errors. FiLM determines the space in which comparison happens, while line-graph targets determine which other pairs should not be treated as unrelated negatives.
• A graph can shape the training objective without entering the test-time forward pass. This is a reusable pattern for collections with rich metadata whose deployed systems must handle new artworks, and distinguishes graph-based training from graph-dependent inference.
• Bidirectional ablations are more informative than a single aggregate score. InfoNCE's I2T advantage and T2I collapse suggest choosing models according to the actual query direction, rather than assuming that a more elaborate objective is universally better.

Limitations & Future Work

• The cache contains the complete main text and references but no appendices, and the text retains a "??" relation-inference reference. Automatic relation identification, exact data splits, computational costs, and additional Precision/NDCG results cannot be verified; unseen experiments should not be supplied.
• Dataset counts are internally inconsistent. HertzianaDP's 45,819 images and 86,321 texts sum to 132,140, while the stated node total is 125,267. SemArt+'s 34,770 and 62,289 sum to 97,059, while the stated total is 97,053. The cache does not explain whether different counting conventions were used; replication needs clarification.
• Sheaf theory supplies an interpretation, but the implemented maps are nonlinear, content-dependent transformations. Ablations support them over simple heads without separately establishing the necessity of strict sheaf constraints; the gains should not be attributed to a proven sheaf-consistency theorem.
• Relation labels, shared text nodes, and diffusion extent can all change the balance of false negatives and false positives. Label noise, missing relations, long-tailed relation types, and cross-collection transfer are useful further tests proposed by this note, not completed experiments from the paper.
• Main tables report no repeated-run variance, and component ablations cover only one dataset. WikiArt+ T2I and several classification attributes still lag; broader claims need equal-information controls, cross-dataset ablations, and verifiable inference costs.

Related Work & Insights

• CLIP / SigLIP: These establish a shared image-text space, while CANVAS adds relation-conditioned views and structural soft supervision. Ordinary fine-tuning is not automatically beneficial: SemArt+ CLIP-ft T2I R@10 is 0.216, below pretrained CLIP's 0.385.
• RCML: Relation-guided cross-attention learns conditioned multimodal representations. CANVAS instead uses FiLM restriction maps and additional cross-pair line-graph structure; the distinction is how relations affect representations and negative targets, not merely whether relations are used.
• ArtSAGENet / GraphCLIP: These inject graph knowledge into artwork representations or align images with knowledge graphs. CANVAS emphasizes graph-free inference; the comparison also removes ArtSAGENet's test-connectivity dependency, so its low retrieval scores should not be generalized to its original transductive setting.
• FiLM / SoftCLIP / Neural Sheaf Diffusion: These contribute related ideas in conditional modulation, soft alignment, and relation-local spaces. The reusable combination is content-dependent modulation plus structural soft targets; effects outside art, including medical imaging and other cultural-heritage tasks, require further evaluation.

Rating

• Novelty: 4/5. Relation-conditioned maps and line-graph heat-kernel supervision fit inductive artwork retrieval well, although the individual components have precedents.
• Experimental Thoroughness: 3/5. Three datasets, bidirectional retrieval, and component ablations are informative, but input asymmetry, unavailable appendices, and contradictory classification claims weaken completeness.
• Writing Quality: 3/5. The method's main thread is clear, but unresolved references, inconsistent counts, and some overgeneralized claims increase verification effort; equation damage may also stem from extraction.
• Value: 4/5. Useful for multimodal retrieval with structured metadata and new, unconnected items, but not a uniformly optimal solution across directions and attributes.