TaxoAgent: Utility-Aligned Policy Optimization for Hierarchical Semantic Classification¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Hierarchical Semantic Classification, Policy Optimization, Large Language Model Agent, Utility Alignment, Out-of-Scope Detection
TL;DR¶
TaxoAgent reformulates hierarchical semantic classification as an explicit coarse-to-fine decision trajectory, aligning an LLM policy with a composite non-decomposable utility spanning correctness, structural validity, prototype semantic proximity, and trajectory coherence via Group-Relative Policy Optimization (GRPO) to curb cross-branch errors and hierarchy violations.
Background & Motivation¶
Hierarchical semantic classification (HTC) is foundational for real-world taxonomy systems such as mobile app store categorization, academic literature indexing, e-commerce cataloging, and dialogue intent parsing. Unlike standard flat classification, hierarchical prediction demands that label assignments strictly conform to directed acyclic graph (DAG) topological constraints, where child categories must be supported by valid parent ancestors. Conventional HTC approaches predominantly employ local-global neural classifiers, hierarchy-aware graph neural networks, or contrastive representation learning to incorporate structural dependencies as auxiliary representations or decoding biases, yet fundamentally treat taxonomy nodes as independent label targets.
This formulation introduces a critical misalignment between conventional optimization objectives and the true operational utility of hierarchical prediction. In real-world taxonomy deployments, classification errors exhibit distinct non-decomposable severities: confusing fine-grained sibling categories under the same parent node represents a mild, semantically benign near-miss, whereas predicting a fine-grained category under an unrelated, cross-branch parent causes catastrophic errors that derail downstream search, recommendation, and safety audit workflows. Furthermore, flat likelihood training frequently produces isolated fine-grained predictions that lack parent support, causing high hierarchy inconsistency rates. While general-purpose large language models (LLMs) possess extensive semantic priors, naive zero-shot or few-shot prompting lacks strict topological governance, offers poor rejection calibration for out-of-scope (OOS) inputs, and struggles with dynamic taxonomy expansion.
To bridge this fundamental gap, this work rethinks hierarchical classification not as flat multi-label scoring or unguided text generation, but as an explicit, structured decision process over a taxonomy graph. Core idea: formulate hierarchical classification as an explicit, coarse-to-fine parent-to-child decision trajectory, evaluate candidate paths using a composite utility encompassing outcome correctness, topological validity, semantic error severity, and trajectory coherence, and optimize the LLM policy end-to-end via Group-Relative Policy Optimization (GRPO) to convert severe cross-branch failures into benign near-misses.
Method¶
Overall Architecture¶
TaxoAgent operationalizes hierarchical classification as a multi-step structured decision trajectory \(\tau = (a_p, a_c, a_v)\) guided by a rooted taxonomy graph \(G = (V, E)\). Given an input text \(x\), the LLM policy first emits coarse parent decisions \(a_p\), subsequently expands child candidate labels \(a_c\) conditioned on the activated parent context, and finally conducts a verification decision \(a_v\) that yields raw label predictions \(\tilde{Y} = (\tilde{Y}^p, \tilde{Y}^c)\). The candidate trajectory is scored by a non-decomposable composite utility combining structural verification and frozen prototype-based semantic alignment. The policy is optimized via Group-Relative Policy Optimization (GRPO) using group-normalized trajectory rewards, and finishes with a deterministic topological closure projection at inference time.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Text & Taxonomy Graph Topology"] --> B["Coarse-to-Fine Decision Trajectory Policy<br/>Sequential generation of parent, child, and verification decisions"]
B --> C["Semantic Prototype & Hierarchy Verifier<br/>Evaluates parent-child support and prototype-space semantic distance"]
D["Composite Non-Decomposable Utility Reward<br/>Jointly measures correctness, validity, semantic proximity, and coherence"]
C --> D
D --> E["Group-Relative Policy Optimization & Inference Projection<br/>Group-relative advantage updates policy and enforces closure projection"]
E --> F["Final Validated Hierarchical Classification"]
Key Designs¶
1. Coarse-to-Fine Decision Trajectory Policy: Flat decoding obscures parent-child dependencies -> Factorize into sequential parent selection, conditioned child expansion, and structured verification -> Enforce conditional dependencies and prune the search space
Conventional approaches apply single-step flat classification or monolithic text decoding, failing to condition fine-grained selections on validated parent categories. TaxoAgent parameterizes the policy as an LLM \(\pi_\theta\) that decomposes the decision sequence into an explicit autoregressive conditional flow:
$\(\pi_\theta(\tau | x, G) = \pi_\theta(a_p | x, G) \cdot \pi_\theta(a_c | x, G, a_p) \cdot \pi_\theta(a_v | x, G, a_p, a_c)\)$
The initial action \(a_p\) generates a ranked set of coarse parent categories \(\tilde{Y}^{step,p}\), providing a bounded semantic domain. The subsequent child action \(a_c\) explores fine-grained labels strictly under the induced semantic branch. The final verification action \(a_v\) outputs a parsed JSON structure containing parent_labels and child_labels, discarding out-of-vocabulary artifacts. This multi-step formulation acts as an internal reasoning chain, decomposing complex graph navigation into manageable local decisions while retaining trace-level interpretability.
2. Semantic Prototype & Hierarchy Verifier: Uniform misclassification penalties distort semantic geometry -> Construct hybrid prototypes and cross-branch penalties to gauge error severity -> Differentiate benign near-misses from catastrophic cross-branch errors
Standard cross-entropy treats all false positives identically, penalizing a near-sibling confusion identically to a completely unrelated top-level category mismatch. TaxoAgent constructs an embedding prototype space using a frozen text encoder \(f_\phi\). For each fine-grained child label \(c\), the unnormalized prototype integrates label-name semantics and empirical training instance representations: $\(q_c = \lambda_{name} f_\phi(t_c) + \frac{\lambda_{data}}{|D_c| + \epsilon} \sum_{(x_i, Y_i) \in D_c} f_\phi(x_i)\)$ Parent prototypes \(q_p\) are synthesized by interpolating parent-name embeddings with the mean normalized child prototypes \(e_c\) across all immediate children \(Ch(p)\), normalized to unit vectors \(e_\ell = q_\ell / \|q_\ell\|_2\). Given normalized text-label and label-label similarities \(s_{\ell, b} = \frac{1 + e_\ell^\top e_b}{2}\), the semantic distance is defined as \(d(\ell, B) = 1 - \max_{b \in B} s_{\ell, b}\). To penalize cross-branch errors, an asymmetry penalty factor \(\kappa \ge 0\) is applied to child errors unsupported by the true parent set: $\(D_c = \frac{1}{|\tilde{Y}^c| + \epsilon} \sum_{\tilde{c} \in \tilde{Y}^c} \left(1 + \kappa \cdot \mathbb{I}[\rho(\tilde{c}) \notin Y^p]\right) d(\tilde{c}, Y^c)\)$ The resulting semantic reward \(R_{sem} = \exp\left(-\frac{\alpha D_p + (1-\alpha) D_c}{\tau_{sem}}\right)\) provides a continuous, severity-aware training signal that discourages severe cross-branch mistakes.
3. Composite Non-Decomposable Utility Reward: Isolated component rewards induce degenerate policies -> Unify multi-aspect objectives into a normalized scalar utility -> Reconcile exact match, topological validity, semantic locality, and decision coherence
Optimizing accuracy alone ignores graph compliance, while solely rewarding structural validity permits degenerate conservative shortcuts, such as predicting empty sets. TaxoAgent balances four complementary objectives into a unified trajectory utility \(U(\tau; x, Y, G) \in [0, 1]\): $\(U(\tau; x, Y, G) = \lambda_{out} R_{out} + \lambda_{hier} R_{hier} + \lambda_{sem} R_{sem} + \lambda_{traj} R_{traj}\)$ where \(R_{out}\) measures the set F1 scores across parent predictions, child predictions, and parent-child matched pairs \((p, c)\); \(R_{hier} = 1 - \frac{\sum_{\tilde{c} \in \tilde{Y}^c} \mathbb{I}[\rho(\tilde{c}) \notin \tilde{Y}^p]}{|\tilde{Y}^c| + \epsilon}\) directly penalizes raw structural violations where child predictions lack corresponding parents in \(\tilde{Y}^p\); and trajectory coherence reward \(R_{traj} = \frac{\sum_{\tilde{c} \in \tilde{Y}^c} \mathbb{I}[\rho(\tilde{c}) \in \tilde{Y}^{step,p}]}{|\tilde{Y}^c| + \epsilon}\) enforces coarse-to-fine commitment, ensuring that fine-grained child labels strictly derive from the parent candidates declared in the initial step \(a_p\).
4. Group-Relative Policy Optimization & Inference Projection: Post-hoc re-ranking fails to internalize policy competence -> Optimize relative trajectory utility via GRPO and apply deterministic topological closure -> Enable internal self-correction with guaranteed zero violation at test time
Supervised fine-tuning (SFT) merely imitates serialized gold tokens without experiencing or correcting invalid intermediate states. TaxoAgent adopts Group-Relative Policy Optimization (GRPO). For each training instance \(x_i\), \(G\) distinct trajectories \(\{\tau_i^g\}_{g=1}^G\) are sampled from the behavioral policy \(\pi_{\theta_{old}}\). Trajectory utility rewards \(r_i^g = U(\tau_i^g)\) are normalized across the sampled group to compute the relative advantage: $\(A_i^g = \frac{r_i^g - \mu_i}{\sigma_i + \epsilon}, \quad \mu_i = \frac{1}{G}\sum_{g=1}^G r_i^g, \quad \sigma_i^2 = \frac{1}{G}\sum_{g=1}^G (r_i^g - \mu_i)^2\)$ The policy is optimized using the clipped surrogate objective with KL divergence regularization against a reference policy \(\pi_{ref}\). At inference time, after the policy generates raw predictions \(\tilde{Y}\), a deterministic projection operator \(\Pi_G(\tilde{Y}) = (\hat{Y}^p, \hat{Y}^c)\) prunes out-of-vocabulary predictions and closes the parent set \(\hat{Y}^p = (\tilde{Y}^p \cap P) \cup \{\rho(\hat{c}) : \hat{c} \in \hat{Y}^c\}\), ensuring strict structural compliance.
Loss & Training¶
The policy parameters \(\theta\) are trained end-to-end to maximize the objective: $\(J_{GRPO}(\theta) = \frac{1}{\sum_{i,g} T_{i,g}} \sum_{i,g,t} \min\left(r_{i,g,t}(\theta) A_i^g, \, \text{clip}(r_{i,g,t}(\theta), 1-\varepsilon, 1+\varepsilon) A_i^g\right) - \beta D_{KL}\left(\pi_\theta(\cdot | x_i, G) \,||\, \pi_{ref}(\cdot | x_i, G)\right)\)$ where \(r_{i,g,t}(\theta) = \frac{\pi_\theta(\tau_{i,g,t} | \dots)}{\pi_{\theta_{old}}(\tau_{i,g,t} | \dots)}\) denotes the token-level importance weight, and \(\varepsilon\) is the clipping parameter. Using Qwen3-14B as the base policy model, training is conducted across 8 A100-80GB GPUs. The framework extends seamlessly to arbitrary depths \(D > 2\) (e.g., RCV1's 4-level taxonomy) by applying level-wise ancestor penalty sums \(R_{hier}^{multi}\) and root-to-leaf path F1 evaluations.
For out-of-scope (OOS) intent detection, the trained semantic prototypes calculate an in-scope composite confidence score during inference: \(q_{max}(x) = \max_{(p,c) \in E} [\eta s_x(p) + (1-\eta) s_x(c)]\). Predictions are rejected as OOS whenever \(q_{max}(x) < \delta_{oos}\), using a threshold calibrated on the validation set.
Key Experimental Results¶
Main Results¶
TaxoAgent was comprehensively evaluated across two industrial mobile app categorization benchmarks (IOS-APP-ZH and Google-APP-EN) and two standard academic HTC benchmarks (RCV1 and Web of Science), compared against encoder baselines, graph neural networks, contrastive learning models, and general-purpose LLMs.
| Dataset | Metric (%) | TaxoAgent | Best Prior Baseline | Improvement |
|---|---|---|---|---|
| IOS-APP-ZH | Micro-F1 | 77.35 | 74.86 (HILL) | โ2.49 |
| Macro-F1 | 35.21 | 32.66 (DeepSeek-3.2) | โ2.55 | |
| Google-APP-EN | Micro-F1 | 82.50 | 81.84 (HILL) | โ0.66 |
| Macro-F1 | 80.70 | 79.01 (TELEClass) | โ1.69 | |
| RCV1 | Micro-F1 | 89.38 | 87.92 (HMCN+HMCL) | โ1.46 |
| Macro-F1 | 73.15 | 71.36 (HMCN+HMCL) | โ1.79 | |
| WOS | Micro-F1 | 89.93 | 89.17 (TELEClass) | โ0.76 |
| Macro-F1 | 83.75 | 81.77 (HILL) | โ1.98 |
Controlled Same-Backbone Comparison & Ablations¶
To isolate the algorithmic contributions of utility-aligned reinforcement learning from the capacity of the base model, all methods were benchmarked using an identical Qwen3-14B backbone. Raw Hierarchy Inconsistency Rate (Raw-HIR) is reported prior to structural projection.
| Configuration / Variant | IOS-APP-ZH Micro-F1 | IOS-APP-ZH Macro-F1 | IOS Raw-HIR (%) โ | Google-APP-EN Micro-F1 | Google-APP-EN Macro-F1 | Google Raw-HIR (%) โ |
|---|---|---|---|---|---|---|
| Qwen3-14B Raw (Direct Prompting) | 31.38 | 25.02 | 10.2 | 68.00 | 66.00 | 12.1 |
| + SFT (Supervised Fine-Tuning) | 73.51 | 31.92 | 8.3 | 73.00 | 68.00 | 10.4 |
| + SFT + Structural Projection | 73.89 | 32.18 | 8.3 | 73.55 | 68.42 | 10.4 |
| + SFT + Trajectory Prompt | 74.18 | 32.65 | 7.2 | 75.40 | 70.25 | 8.8 |
| + SFT + Verifier Rerank (4x sample) | 75.62 | 33.40 | 6.4 | 78.10 | 74.20 | 7.5 |
| + DPO (Pairwise Preference Optimization) | 76.41 | 34.36 | 5.9 | 80.10 | 78.40 | 6.8 |
| TAXOAGENT (Full Utility-Aligned GRPO) | 77.35 | 35.21 | 5.3 | 82.50 | 80.70 | 6.1 |
Key Findings¶
- Macro-F1 and Long-Tail Superiority: The performance advantage of TaxoAgent is notably pronounced on Macro-F1. Frequency-group analysis shows an average gain of +8.7 F1 points on tail labels (\(n_{train} < 50\)) on IOS-APP-ZH (compared to +4.1 on medium and +1.6 on head labels), demonstrating that prototype-guided structural rewards facilitate effective knowledge transfer to low-resource classes.
- Semantic Error Compression: Analysis of false-positive distributions reveals that TaxoAgent significantly left-shifts the error distance distribution. On IOS-APP-ZH, mean semantic error distance drops from 0.266 to 0.148, boosting the near-miss rate from 51.6% to 82.0%. On Google-APP-EN, the near-miss rate reaches 88.2%, proving that errors are predominantly confined to local siblings under the true parent branch.
- Active Trajectory Self-Correction: Alluvial flow diagnostics reveal that 70.2% of initially flawed decision units in IOS-APP-ZH are strictly resolved by the final trajectory stage (recovering missed positive labels or eliminating invalid child predictions). Counting benign near-misses, the soft correction rate attains 86.6%, indicating genuine multi-step refinement rather than trivial re-generation.
- Robust OOS Rejection: On the challenging domain-level in-domain OOS benchmark CLINC150, TaxoAgent outperforms the leading specialized baseline FCSLM by +2.17% overall accuracy, +3.64 OOS-F1 on Banking (87.93 vs. 84.29), and +3.68 OOS-F1 on Credit Cards (86.39 vs. 82.71), exhibiting sharp utility margins for boundary separation.
Highlights & Insights¶
- Taxonomy as a Reinforcement Learning Environment: Shifts the paradigm from passive representation injection or hard architectural constraints to treating taxonomy structures as interactive decision spaces governed by non-decomposable utility functions.
- Asymmetric Cross-Branch Penalty: By incorporating the asymmetric factor \(\kappa\) into the prototype semantic penalty, the system penalizes cross-domain hallucinations far more severely than local sibling mismatches, directly aligning LLM generation with industrial risk tolerance.
- Retraining-Free Taxonomy Adaptation: Accommodates evolving categories by registering lightweight seed prototypes and updating edge sets; on newly expanded app taxonomies, TaxoAgent achieves 94.1% child accuracy on IOS and 95.7% on Google Play without full model retraining.
Limitations & Future Work¶
- Dependence on Informative Label Semantics: The prototype construction heavily exploits rich text descriptions and meaningful names; its effectiveness could be curtailed in cryptically coded industrial taxonomies or noisy, uncurated ontologies.
- Inference Latency Overhead: Generating the multi-step trajectory increases average generated tokens from 80 to 630 tokens per sample, raising latency from 0.24s to 1.27s on a single A100 GPU. While appropriate for offline catalog classification, online real-time systems would require distillation.
- Future Directions: Exploring partially observed, noisy, or user-contributed taxonomies; testing across biomedical coding (ICD-10/SNOMED-CT) and complex multilingual legal classification frameworks.
Related Work & Insights¶
- vs Conventional HTC Models (HiAGM, HMCN, HiMatch): Traditional neural models use GCNs or hierarchical layers to constrain flat logit outputs, remaining prone to predicting unsupported child nodes; TaxoAgent models explicit reasoning trajectories where structural consistency is optimized via policy gradient.
- vs Contrastive Representation Approaches (HGCLR, HILL, HMCL): While contrastive models shape embedding geometries to cluster related labels, they optimize separability rather than non-decomposable utility; TaxoAgent repurposes representation prototypes as verifier reward signals during post-training.
- vs Direct LLM Prompting & Enhancement (ChatGPT-4o, DeepSeek-V3.2, TELEClass): Direct prompting displays high raw hierarchy inconsistency (up to 12.1% HIR) and weak OOS discrimination; TaxoAgent achieves superior structural compliance (5.3% Raw-HIR) and calibrated uncertainty through utility-aligned policy post-training.
Rating¶
- Novelty: โญโญโญโญโญ (Pioneering formulation of hierarchical semantic classification as an RL decision trajectory with non-decomposable utility)
- Experimental Thoroughness: โญโญโญโญโญ (Rigorous across 4 diverse HTC benchmarks, evolving taxonomies, in-domain OOS detection, controlled same-backbone baselines, and alluvial correction diagnostics)
- Writing Quality: โญโญโญโญโญ (Cohesive structure, clear theoretical formulation, informative visual diagnostics, and thorough empirical analysis)
- Value: โญโญโญโญโญ (Directly addresses high-cost cross-branch misclassifications in production taxonomy systems with strong transferability)