ARGENT: Adaptive Hierarchical Image-Text Representations

Conference: ECCV 2026
Paper: Official page · PDF
Project: https://hmchuong.github.io/argent
Area: Multimodal VLM / Hyperbolic Representation Learning
Keywords: hyperbolic geometry, hierarchical representations, adaptive entailment, norm regularization, hierarchical evaluation

TL;DR

ARGENT replaces unstable cone-aperture constraints in a HyCoCLIP hyperbolic dual encoder with adaptive angular entailment and norm regularization, while redesigning hierarchical evaluation through PEP; its Large model improves ImageNet accuracy by 1.7 percentage points and PEP AUC by 1.5 points over HyCoCLIP-L, although gains do not hold for every scale and metric.

Background & Motivation

CLIP can bring representations of dogs and poodles close together without necessarily distinguishing a directed taxonomic relationship from an association such as dogs and leashes. Hierarchy requires both direction and abstraction level. Hyperbolic space offers exponentially growing volume with radius, making it a useful substrate for branching concept trees. MERU introduces hyperbolic image-text alignment, and HyCoCLIP adds entailment cones to model relationships among images, crops, full captions, and phrases.

The difficulty is that a cone's aperture depends on the parent concept's distance from the origin. A more general concept is normally closer to the origin and receives a wider cone, but optimization can exploit this coupling by continually shrinking the parent norm until the aperture calculation leaves its valid domain. Clipping the aperture to a half-space prevents a numerical failure while weakening the intended hierarchy. Evaluation has a separate problem: HierarCaps retrieval can count a caption attached to another image as negative even when a generic description such as a group of people also fits the query.

The paper therefore changes both the training signal and its validation: hierarchy strength should not depend on a potentially invalid aperture, and evaluation should test cleaned positive and negative relations rather than demand one exclusive image-caption match. Core idea: directly constrain the directed entailment angle, adaptively weaken only potentially near-duplicate intra-modality pairs, stabilize geometric scale with norm regularization, and independently assess hierarchical relationships with an angular score.

Method

Overall Architecture

Inputs are images and their captions; the inherited HyCoCLIP framework also uses crops and phrases to construct compositional hierarchy. CLIP-style encoders first produce Euclidean vectors, which are mapped into Lorentz hyperbolic space through an exponential map at the origin. Training retains contrastive alignment with negative geodesic distance as similarity, alongside adaptive angular entailment, intra-modality adaptive weighting, and norm regularization.

The resulting representations support zero-shot classification and image-text retrieval, as well as a separate PEP hierarchical evaluation. PEP cleaning and scoring belong to evaluation, not additional training supervision. ARGENT does not generate images and is not a multimodal conversational model with a language decoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Images, crops<br/>captions, phrases"] --> B["Dual encoders<br/>Lorentz mapping"]
    B --> C["Adaptive Angular Entailment"]
    C --> D["Intra-Modality Adaptive Weighting"]
    D --> E["Norm Regularization<br/>with contrastive learning"]
    E --> F["Classification and retrieval"]
    E -->|Post-training evaluation| G["PEP Hierarchical Evaluation"]
    H["Cleaned HierarCaps<br/>and negative pool"] --> G

Key Designs

1. Adaptive Angular Entailment: bypass parent-norm-dependent cone apertures

Let \(x\) denote a specific concept and \(y\) a general one. Entailment direction is described by the exterior angle \(\phi(x,y)=\pi-\angle Oyx\) at the parent, where \(O\) is the origin. This is not ordinary cosine similarity: the origin establishes a radial direction from general to specific, so exchanging parent and child generally changes the angle. A well-ordered specific concept extends outward along the direction of its more general parent, producing a small exterior angle.

The original aperture depends on \(2C/(\sqrt{\kappa}\|\tilde y\|)\), where \(\tilde y\) contains the Lorentz point's spatial coordinates and curvature is \(-\kappa\). When this ratio exceeds 1, the inverse sine has no real-valued solution. With the paper's illustrative \(C=0.1\) and \(\kappa=0.1\), the required norm is approximately 0.6325 or greater. Aperture clipping leaves many relationships without a useful learning signal. AdaEnt instead penalizes the exterior angle directly with a Huber smoothing mechanism controlled by \(\beta\), removing dependence on aperture validity.

Equation (7) is corrupted in the text cache. Its exact normalization and whether the adaptive weight is applied inside or outside the Huber function cannot be recovered reliably, so this note does not invent a complete loss formula. The recoverable mechanism is angular optimization, Huber smoothing, and pair-dependent weighting as described next. Exact implementation requires checking the original PDF or code.

2. Intra-Modality Adaptive Weighting: do not force nearly identical crops and images apart

A crop covering almost the entire image can have essentially the same meaning as the image. Forcing a hierarchy between them would create an artificial distinction. ARGENT uses a weight that increases with hyperbolic geodesic distance for intra-modality pairs: near-zero distances suppress the entailment constraint, whereas larger distances restore it. Inter-modality image-text pairs always receive weight 1 because weakening these relationships would undermine crucial cross-modal constraints.

Combining the explanation around Equation (8) with Figure 4 gives the following equivalent distance-based expression for the weights. This is a semantic reconstruction rather than a character-for-character transcription of the damaged equation:

\[ h(x,y)= \begin{cases} 1-\exp[-d_{\mathcal L}(x,y)], & \text{intra-modality},\\ 1, & \text{inter-modality}. \end{cases} \]

Here \(d_{\mathcal L}\) is geodesic distance on the Lorentz manifold, and contrastive similarity is its negative. The design responds to how much distinction a pair actually supports; it does not indiscriminately downweight every difficult example. Applying adaptive weights to all pairs substantially harms PEP in the ablations, making modality type an essential condition.

3. Norm Regularization: prevent outward drift after removing aperture constraints

Direct angular optimization no longer automatically controls distance from the origin, and representations can become excessively sparse. The paper regularizes the spatial-coordinate norm as follows: the negative logarithm repels very small norms, while the quadratic term penalizes large ones.

\[ \mathcal L_{\mathrm{Reg}}(x)=\|\tilde x\|^2-\log\|\tilde x\|. \]

This is the Euclidean norm of the spatial coordinates, not a Lorentz inner-product norm. In isolation, the regularizer is minimized at \(1/\sqrt{2}\approx0.7071\). Actual embeddings also respond to contrastive and entailment objectives, so they are not all forced to that radius. HoroPCA visualizations and norm distributions show clearer text-level separation and a bimodal crop distribution for ARGENT, but these are analytical tools rather than additional network components.

4. PEP Hierarchical Evaluation: test angular scores on cleaned positive and negative relations

Kendall correlation mainly checks ordering and cannot distinguish angular margins when rankings agree. Retrieval can also incorrectly penalize semantically valid captions outside the original pairing. The authors use Qwen3-VL for image-caption verification together with Qwen3-VL-Embed and Qwen3-VL-Reranker similarity information to clean HierarCaps. They retain 522 image-caption pairs from 1,000 samples and construct a negative pool of 100 captions with the lowest retrieval degree, meaning they are rarely retrieved for other images.

For each image, all corresponding hierarchical captions are positives; fine-grained captions from the non-ambiguous pool provide negatives. Equation (9) is also damaged in the cache. The expression below restates the linear angular score using the endpoints and clipping behavior described in the text; the PDF remains authoritative for exact notation:

\[ p_{\mathrm{Ent}}(x,y)=\max\left(0,1-\frac{2\phi(x,y)}{\pi}\right). \]

An exterior angle of 0 receives score 1, and angles at or above \(\pi/2\) receive 0. Evaluation reports AUC-ROC and AP rather than checking whether the top caption belongs to the original pair. Although called a probability, this is an angular score: these metrics establish discrimination, not probability calibration.

A Worked Example

Consider an image of a group of people indoors, paired with captions at different levels of detail. After Lorentz mapping, contrastive learning aligns paired images and text, while angular entailment checks whether specific representations extend outward from general concepts. If a crop is almost identical to the image, their intra-modality distance approaches 0 and so does the adaptive weight. The corresponding image-text constraint still receives weight 1 and must not disappear with it.

Norm regularization simultaneously discourages collapse near the origin and unbounded outward drift. At evaluation, multiple correct hierarchical captions contribute positive scores instead of competing for a single correct answer. This is a mechanism illustration, not a reported numerical case study; no embedding coordinates or model scores are fabricated.

Loss & Training

The combined objective retains image-text contrastive learning and adds adaptive entailment and norm regularization:

\[ \mathcal L=\mathcal L_{\mathrm{Cont}}+\gamma_1\mathcal L_{\mathrm{AdaEnt}}+\gamma_2\mathcal L_{\mathrm{Reg}},\qquad \gamma_1=\gamma_2=0.1. \]

Main experiments use GRIT with 20.5M grounded image-text pairs and compare Small, Base, and Large models. Image inputs are \(224\times224\) with \(16\times16\) patches; text length is capped at 77 tokens. Ablations use a smaller CC3M setting constructed through the HyCoCLIP data pipeline, with 40,000 iterations processing 30M data points in total. That cumulative count does not imply 30M distinct training images.

The main text contains a supervision detail that needs verification for reproduction: the dataset paragraph says only HyCoCLIP additionally uses box information, while the architecture paragraph integrates ARGENT into the crop-and-phrase-based HyCoCLIP framework. This note explains the inherited framework without asserting that all supervision budgets are identical. The Huber threshold, optimizer, and complete training schedule are also not invented from missing information.

Key Experimental Results

Main Results

The following selection comes from Table 2. Accuracy, Recall, AUC, and AP use the paper's percentage scale, with higher values better. Average classification covers 16 datasets. The retrieval column consistently reports COCO text-to-image R@5 rather than mixing datasets or directions.

Model	ImageNet Accuracy	16-Dataset Average	COCO T2I R@5	PEP AUC	PEP AP
HyCoCLIP-S	37.3	39.2	52.0	96.8	87.7
ARGENT-S	38.8	38.4	53.2	99.3	91.2
HyCoCLIP-B	43.1	42.6	56.7	97.2	88.5
ARGENT-B	44.0	42.8	57.9	99.4	90.6
HyCoCLIP-L	43.9	44.4	57.5	98.0	89.5
ARGENT-L	45.6	45.1	58.6	99.5	90.3

Large-model improvements in these columns are +1.7, +0.7, +1.1, +1.5, and +0.8 percentage points. Small-model average classification instead falls by 0.8 points, so the narrative of near-universal improvement should not become an unconditional claim. These are comparisons against matched-scale HyCoCLIP models, not claims of superiority over CLIP at arbitrary training budgets.

Ablation Study

The following rows select HyCoCLIP-S configurations from Table 3. Cls. Acc. and Ret. R@5 retain that table's labels, which do not explicitly identify the dataset and aggregation at that location; they should not be directly compared with the COCO column above. The no-regularization row is interpreted using Section 6.3 and the matching full-configuration values because the cached toggle symbols are damaged.

Source and Configuration	Cls. Acc.	Ret. R@5	PEP AUC	PEP AP
Table 3(a): original Ent	22.9	54.8	94.6	76.2
Table 3(b): AdaEnt, no adaptive weighting	20.9	51.3	94.9	74.3
Table 3(b): AdaEnt, weight all pairs	21.6	54.9	71.1	40.2
Table 3(b): AdaEnt, weight intra-modality only	22.8	55.5	97.3	76.9
Table 3(c): remove norm regularization	20.8	52.6	96.2	75.6

The full configuration is 22.8 / 55.5 / 97.3 / 76.9 in Table 3(a)'s AdaEnt row, Table 3(b)'s Intra row, and Table 3(c)'s regularization-enabled row. These groups answer different questions and should not be interpreted as one sequential component-addition experiment.

Key Findings

• Weighting all pairs instead of intra-modality pairs alone reduces AUC by 26.2 points and AP by 36.7 points. This is a clear boundary of the design, not evidence that adaptation is universally beneficial.
• Removing norm regularization reduces classification from 22.8 to 20.8 and R@5 from 55.5 to 52.6. Correcting the angular loss alone does not guarantee downstream generalization.
• Scaling ARGENT from S to L raises ImageNet accuracy from 38.8 to 45.6 but reduces AP from 91.2 to 90.3. Downstream scaling benefits and hierarchical retrieval precision do not move together perfectly.
• In Table 1, removing potentially ambiguous captions from the top-10 similar images raises CLIPLFT Precision from 15.36 to 18.05 and Recall from 43.23 to 48.10. This supports the presence of evaluation ambiguity, but does not establish that every removed caption was manually verified.

Highlights & Insights

• Geometric suitability and optimization stability are different questions. Hyperbolic space may fit trees while a particular constraint still fails; the paper identifies a concrete aperture-domain and vanishing-signal failure mode.
• Weighting must respect pair semantics. Near-duplicate intra-modality views need less forced distinction, whereas cross-modal relationships still need their constraints; this could inform learning with local views or segments, provided the pair assumptions are revalidated.
• Before interpreting a retrieval score, verify whether its negatives are actually wrong. Candidate cleaning exposes hidden label assumptions, but cleaning rules also need to be published so selection bias can be assessed.

Limitations & Future Work

• The authors acknowledge that angular optimization can make representations sparse, requiring an extra norm regularizer and loss balancing. The experiments do not provide a formal guarantee against collapse.
• PEP retains only 522 pairs, relies on Qwen-family cleaning, and favors low-confusion captions in its negative pool. An AUC near 99.5 does not mean open-world fine-grained hierarchy is solved; manual auditing, harder negatives, and cross-dataset evaluation remain important.
• Reproduction uncertainties remain: cleaning thresholds and full training details are deferred to supplementary material, and the box-supervision description needs checking. The reported tables do not provide multi-seed variance.
• Cached Equations (7) and (9), along with ablation toggle symbols, suffer extraction damage. This note distinguishes verified mechanisms, semantic reconstructions, and unrecoverable implementation details rather than treating extraction loss as an omission by the authors.

Related Work & Insights

• Versus CLIP / ALIGN: these methods center on Euclidean image-text alignment. ARGENT adds directed hyperbolic hierarchy, at the cost of managing curved-space numerical and radial behavior.
• Versus MERU: MERU supplies the hyperbolic image-text foundation. ARGENT primarily improves entailment stability and hierarchical evaluation; it is not the first hyperbolic VLM.
• Versus HyCoCLIP: ARGENT inherits the dual-encoder and compositional hierarchy framework, replacing the key training mechanism with an angular objective, pair weighting, and norm regularization. Gains need to be interpreted alongside supervision budgets and model scale.
• Versus the original HierarCaps protocol: the multi-level caption resource is retained, but sample cleaning and the unit of evaluation change. Reporting the original protocol, cleaned PEP, and manually checked hard negatives together would reduce dependence on any one favorable evaluation.

Rating

• Novelty: 4/5. A concrete diagnosis of entailment-cone instability is combined with training changes and an evaluation protocol, while the architecture builds on existing hyperbolic VLMs.
• Experimental Thoroughness: 4/5. Three scales, 16 classification datasets, bidirectional retrieval, and component ablations provide breadth, but variance estimates and broader independent hierarchy benchmarks are missing.
• Writing Quality: 3/5. The problem chain is clear, but narrative summaries and tables do not agree everywhere, and some supervision details require cross-checking. Cache extraction damage is not counted as an author writing defect.
• Value: 4/5. Useful for stable hierarchical multimodal training and evaluation design, with no direct experimental evidence for image generation.