Skip to content

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agents
Keywords: Agentic LLMs, API Provider Auditing, Tool-use Policy, Model Fingerprinting, MMD Permutation Test

TL;DR

AgentProv is the first action-based black-box identity auditing framework for agentic LLM APIs, utilizing categorical tool-selection preferences under functionally equivalent probes and a block-decomposed MMD permutation test to catch backbone substitutions without text dependence and with strong robustness against hidden system prompt injections.

Background & Motivation

Commercial LLMs are increasingly accessed via black-box API endpoints advertising specific flagship foundation backbones. However, because downstream customers and enterprise developers cannot inspect the physical deployment infrastructure, dishonest third-party providers, resellers, and proxy wrappers frequently engage in silent model substitutions to lower inference and GPU serving costs. Deceptive practices include silently switching to smaller distilled variants, down-quantized weights, or prompt-wrapped open-source alternatives. Recent empirical investigations reveal the alarming prevalence of this problem: an audit of 31 commercial Llama endpoints discovered that 11 were serving drifting distributions, while a comprehensive survey of 17 commercial wrapper services found a 45.8% identity verification failure rate, including wholesale silent swaps such as serving GLM-4-9B under the label of GPT-5.

Existing black-box model auditing methodologies, such as Model Equality Testing (MET) based on Hamming kernels over natural language completions and Rank-based Uniformity Testing (RUT) based on log-rank score uniformity, rely strictly on the natural language text output channel. In the emerging era of agentic LLM APIs, this reliance causes structural failures. First, production agentic serving stacksโ€”such as OpenAI Function Calling, Anthropic Claude Tool Use, Gemini, and agent runtimes like LangGraph or Cloudflare Workers AIโ€”routinely suppress natural language text upon tool invocation, emitting only structured action objects (tool_calls), leaving text-channel probes with an empty generation space. Second, commercial providers routinely inject proprietary system prompts, safety guardrails, and formatting chrome around user queries; in real-world scans across 75 OpenRouter endpoints, 44% deviated from official chat templates. Because text-channel tests operate directly on literal surface tokens, even a single extraneous whitespace or a five-token introductory greeting shifts the completion distribution and triggers false rejections, mistakenly accusing honest providers of deploying substituted models.

To overcome these structural roadblocks, the authors recognize a fundamental attribute of modern agentic post-training: tool-calling capabilities are directly internalized into model weights through dedicated supervised fine-tuning and reinforcement learning pipelines rather than remaining shallow inference scaffolding. Consequently, when presented with multiple functionally equivalent tool alternatives under neutral instructions, different foundation model families exhibit distinct, highly reproducible categorical tool preferences that serve as an intrinsic behavioral signature. Most crucially, this action-level decision is largely invariant to deployment-context modifications: provider-side system prompts drastically alter surface text tokens but rarely flip the discrete choice of which tool to invoke. Core idea: bypass the fragile natural language completion channel and ground black-box API identity auditing in the intrinsic tool-selection policy, extracting action fingerprints via unbiased equivalent probes and conducting block-diagonal MMD permutation tests to achieve text-free, prompt-injection-robust API provenance verification.

Method

Overall Architecture

AgentProv formulates API accountability as a two-sample hypothesis test on the discrete tool-selection policy: \(H_0: \pi_{\text{ref}} = \pi_{\text{sus}}\) against the alternative that the policies diverge, calibrated at significance level \(\alpha = 0.05\). The auditor has black-box query access to a suspect endpoint claiming to serve model \(M\), as well as reference access to an authoritative instance of \(M\) (locally executed for open-weight models, or the official vendor API for closed-source models). The auditing pipeline flows through four sequential stages: querying both endpoints using unbiased tool-preference probes across \(K\) templates; converting returned structured tool choices into one-hot action fingerprints in a joint template-option alphabet; evaluating a squared \(L_2\) test statistic derived from an MMD formulation with a delta kernel; and conducting within-template label permutations to establish a non-parametric empirical null distribution for the binary accept/reject decision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Black-box Suspect & Reference Queries<br/>Send neutral requests with equivalent tool options"] --> B["Tool-Preference Probes<br/>Randomize schema ordering and balance descriptions"]
    B --> C["One-Hot Action Fingerprinting<br/>Deterministic bucketing into joint alphabet space"]
    C --> D["Block-Diagonal MMD Statistic Computation<br/>Delta kernel induces per-template squared L2 distance"]
    D --> E["Within-Template Permutation Testing<br/>Shuffle endpoint labels to calibrate empirical null"]
    E -->|T > Critical Threshold| F["Reject H0: Detect silent backbone substitution"]
    E -->|T <= Critical Threshold| G["Accept H0: Confirm legitimate model identity"]

Key Designs

1. Tool-Preference Probes: Bias-controlled policy elicitation over equivalent tool sets
Standard prompting induces noise and positional artifacts that distort true policy measurement. To isolate the model's internalized tool-selection prior, the probe comprises \(K\) standardized templates (with \(K=20\) in the primary setup and an expanded pool of \(K=60\)) spanning two complementary domains: general everyday tools (15 functional categories such as calendar scheduling, email dispatch, and weather checking) and Model Context Protocol (MCP) server naming conventions. Each template \(k\) defines a neutral task prompt and a schema offering functionally equivalent tool alternatives. The probe enforces three mandatory experimental controls: first, the ordering of tools in the JSON schema is randomized per query using a deterministic pseudo-random seed to eliminate first-listed position bias; second, all tool names, argument definitions, and descriptions within a template are matched in token length and syntactic structure to prevent verbosity-driven selection; third, user requests employ strictly neutral terminology (e.g., "schedule a meeting" instead of "use Google Calendar") and generic recipient handles. The auditor deterministically maps the emitted tool_calls into a predefined option set \(O_k\) (typically 4 to 6 options), mapping missing tool invocations to _no_call and syntax errors to _malformed.

2. One-Hot Action Fingerprinting: High-dimensional empirical policy representation
Because agentic decisions are discrete and span multiple probe templates, the framework defines a joint template-option alphabet \(\mathcal{Z} = \{(k, a) : k \in [K], a \in O_k\}\) with total dimension \(D = \sum_{k=1}^K |O_k|\). Drawing \(N\) independent tool-call completions per template from both the reference and suspect endpoints yields \(N_{\text{total}} = K \times N\) observations per endpoint. Each observed decision is mapped to a one-hot vector \(z_i \in \{0, 1\}^D\). Stacking these rows creates observation matrices \(Z_R, Z_S \in \{0, 1\}^{N_{\text{total}} \times D}\), and taking their column-wise means yields the empirical fingerprint vectors \(\phi_R\) and \(\phi_S\): $\(\phi = \frac{1}{N_{\text{total}}} \sum_{i=1}^{N_{\text{total}}} z_i \in \mathbb{R}^D\)$ This fingerprint vector compactly encodes the concatenation of empirical multinomial distributions across all \(K\) templates, providing a clean numerical signature of the model's multi-scenario tool preferences.

3. Block-Diagonal MMD Statistic Computation: Characteristic kernel mapping and interpretable diagnostic decomposition
To measure the divergence between reference and suspect action distributions without imposing parametric assumptions, AgentProv employs the Maximum Mean Discrepancy (MMD) metric over a Reproducing Kernel Hilbert Space (RKHS). On the finite discrete alphabet \(\mathcal{Z}\), the framework adopts the delta kernel: $\(\kappa\left((k, a), (k', a')\right) = \mathbf{1}\{k = k'\} \cdot \mathbf{1}\{a = a'\}\)$ Because the delta kernel on a finite alphabet is strictly characteristic, the population MMD vanishes if and only if the underlying policies \(\pi_{\text{ref}}\) and \(\pi_{\text{sus}}\) are identical. Under this kernel, the empirical squared MMD estimator cleanly simplifies to the squared \(L_2\) distance between the empirical fingerprint vectors: $\(T = \|\phi_R - \phi_S\|_2^2 = \sum_{k=1}^K \|\phi_R^k - \phi_S^k\|_2^2\)$ Because each row has non-zero entries exclusively in one template's coordinate block, the overall statistic decomposes into a sum of per-template squared Euclidean distances. This property allows auditors to trace any rejected hypothesis directly back to specific probe scenarios, identifying exactly which tool choices exposed the substitution.

4. Within-Template Permutation Testing: Non-parametric exact Type-I error control
In finite-sample audit regimes, asymptotic normal approximations frequently distort significance levels. To strictly enforce the nominal false positive rate \(\alpha = 0.05\) across arbitrary choices of sample size \(N\), template count \(K\), and option set sizes, the framework leverages sample exchangeability under the null hypothesis \(H_0\). Specifically, for each template \(k\), the algorithm pools the \(2N\) responses from both endpoints, shuffles the endpoint assignment labels uniformly at random, and splits them back into two surrogate sets of size \(N\). Across all \(K\) templates, the recomputed pseudo-statistics yield a sample \(T^{(r)}\). Repeating this procedure \(B = 1,000\) times yields the empirical null distribution \(\{T^{(r)}\}_{r=1}^B\). The test rejects \(H_0\) if and only if the observed test statistic \(T\) exceeds the \((1 - \alpha)\)-quantile of this permutation null distribution.

A Worked Example

Consider an auditor auditing a suspect endpoint advertising Qwen2.5-7B. The probe instantiates two templates: Template 1 asks to "Send an email" with options {gmail, outlook, yahoo, _no_call}; Template 2 asks to "Schedule a meeting" with options {gcal, zoom, calendly, _no_call}, creating an 8-dimensional space. The auditor queries each endpoint \(N=50\) times per template. The authentic Qwen2.5-7B reference displays an internalized prior favoring outlook on Template 1 (empirical proportions \([0.16, 0.68, 0.16, 0.00]\)). In contrast, the suspect endpoint secretly routes queries to a cheaper distilled model that preferentially selects gmail (\([0.72, 0.14, 0.14, 0.00]\)). Computing the squared distance between these fingerprints yields a large discrepancy \(T \approx 0.60\). Shuffling the combined 100 observations within Template 1 across 1,000 permutations yields surrogate statistics that rarely exceed \(0.08\). The observed statistic heavily exceeds the 95th-percentile critical threshold (\(p < 0.001\)), decisively rejecting \(H_0\) and flagging the silent substitution.

Key Experimental Results

Main Results

AgentProv was evaluated across 36 open-weight base checkpoints (covering 13 model families and 0.5B to 14B parameters), 5 local checkpoints subjected to injected system prompts, and 9 closed-source OpenAI models across official and third-party reseller routes.

Table 1: Model Identity Auditing and Substitution Detection at \(\alpha = 0.05\)

Evaluation Regime Evaluated Pairs (\(n\)) MET (Gao et al., 2025) RUT (Zhu et al., 2026) AgentProv (Ours)
Distinct Checkpoint Cross-Pairs 630 100.0% Rejection (630/630) 100.0% Rejection (630/630) 100.0% Rejection (630/630)
Same-Model Self-Pairs 36 0/36 False Rejections (0.0% FPR) 2/36 False Rejections (5.6% FPR) 0/36 False Rejections (0.0% FPR)
Closed-Source Official OpenAI Pairs 36 - (Requires Logits/Text) - (Requires Logits) 36/36 Rejection (100.0%)
Official OpenAI vs. Local Checkpoints 324 - - 324/324 Rejection (100.0%)
Official OpenAI vs. OpenRouter Resellers 9 - - 9/9 Acceptance (0.0% FPR)

Table 2: False Positive Rate under Hidden System Prompt Injection Control (\(\alpha = 0.05\))

Base Model System Prompt Injection Condition MET RUT Adapted LLMmap AgentProv (Ours)
Llama-3.2-1B-Instruct Empty (1-token space) / Short (5 tokens) / Long (50 tokens) Rej / Rej / Rej Acc / Rej / Rej Acc / Acc / Rej Acc / Acc / Acc
Qwen3-1.7B Empty / Short / Long Rej / Rej / Rej Acc / Acc / Rej Acc / Acc / Acc Acc / Acc / Acc
Qwen2.5-1.5B-Instruct Empty / Short / Long Acc / Rej / Rej Acc / Acc / Rej Acc / Acc / Acc Acc / Acc / Acc
Gemma-3-1B-it Empty / Short / Long Rej / Rej / Rej Rej / Rej / Rej Acc / Rej / Rej Acc / Acc / Rej
Hammer2.1-1.5B Empty / Short / Long Acc / Acc / Rej Acc / Acc / Rej Acc / Acc / Acc Acc / Acc / Acc
Overall False Positive Rate Aggregated over 15 Test Conditions 10/15 (66.7%) 8/15 (53.3%) 3/15 (20.0%) 1/15 (6.7%)

Ablation Study

Table 3: Template Count (\(K\)) vs. Sample Budget (\(N\)) Sweep on Cross-Model Rejection Rate (630 Distinct Pairs)

Number of Templates (\(K\)) \(N = 5\) Samples \(N = 10\) Samples \(N = 25\) Samples \(N = 50\) Samples (Standard)
\(K = 5\) 36.67% 66.67% 92.70% 97.46%
\(K = 10\) 60.48% 91.90% 98.89% 99.84%
\(K = 20\) (Default) 65.56% 94.13% 99.84% 100.00%
\(K = 30\) 64.44% 95.24% 100.00% 100.00%
\(K = 40\) 64.76% 95.71% 100.00% 100.00%
\(K = 60\) 68.57% 96.03% 100.00% 100.00%

Key Findings

  • Unprecedented robustness against deployment-context prompt shifts: Under controlled system prompt injections, MET suffered false rejections on 3 out of 5 models even with a single whitespace token, reaching an overall false positive rate of 66.7%. RUT degraded rapidly to a 53.3% false positive rate under longer prompts. AgentProv maintained a 0.0% false positive rate on both empty and short prompt conditions, with its single false positive occurring on Gemma-3-1B under a 50-token persona prompt where tool-call syntax drifted, yielding an overall false positive rate of only 6.7%.
  • Sample allocation efficiency: Increasing the sample count per template (\(N\)) yields substantially greater statistical power than merely scaling the number of templates (\(K\)). At \(N=5\), increasing \(K\) from 20 to 60 only lifted accuracy from 65.56% to 68.57%. In contrast, fixing \(K=20\) and increasing \(N\) from 5 to 50 drove accuracy from 65.56% to a saturated 100.00%. A lightweight configuration of \(K=10, N=50\) (500 queries, costing well under $0.50) achieves 99.84% detection accuracy.
  • Action space captures lineage and resists adaptive mimicry: Analysis of pairwise MMD distances reveals tight within-family clustering across sizes (within-to-between ratios of 1.8x to 3.0x for Qwen2.5, Qwen3, Llama, and Hammer). In adaptive evasion tests, wrapping substitute models with adversarial system prompts containing reference model profiles or few-shot exemplars failed to evade detection, with AgentProv maintaining 100% rejection.

Highlights & Insights

  • Paradigm shift from fragile text tokens to internalized tool actions: AgentProv circumvents the major vulnerability of black-box LLM auditing by leveraging internalized tool-choice priors, which remain fully accessible in action-only serving architectures while staying invariant to surface prompt modifications.
  • Block-diagonal delta kernel formulation: Modeling categorical tool choices with the delta kernel simplifies continuous RKHS MMD computation into an interpretable squared \(L_2\) distance between multinomial mean vectors, providing instant per-template diagnostic breakdowns.
  • Extreme economic and operational efficiency: A comprehensive audit at \(K=20, N=50\) requires only 1,000 queries per endpoint, executing in minutes at a cost under $1.00 without needing logit access or privileged access to model weights.

Limitations & Future Work

  • Requirement for authoritative reference instances: AgentProv operates as a two-sample equality test and inherently requires access to a trusted instance of the claimed model. It cannot audit completely private, proprietary models that lack any authoritative reference channel.
  • Sensitivity to severe syntactic tool formatting drift: While robust to typical system prompts, extremely long and coercive provider prompts can occasionally alter the tool-call syntax of smaller models (e.g., Gemma-3-1B), leading to parse errors that trigger false positives.
  • Vulnerability to probe-overfitting distillation: Attackers can fine-tune small models using parameter-efficient techniques (LoRA) on known probe templates to mimic reference behavior. Auditors must mitigate this by continually generating and rotating fresh, unreleased private probe pools.
  • vs. MET (Model Equality Testing, Gao et al., 2025): MET applies an MMD two-sample test with a Hamming kernel on natural language completions. It is completely inapplicable to action-only serving stacks and suffers a 66.7% false positive rate under benign system prompt injections, whereas AgentProv achieves 6.7% FPR while operating purely on action payloads.
  • vs. RUT (Rank-based Uniformity Testing, Zhu et al., 2026): RUT is a grey-box method requiring log-probabilities from reference instances. It is fragile to surface token modifications (53.3% FPR under system prompt injections), while AgentProv is strictly black-box and robust.
  • vs. AGENTWM (Wang et al., 2026a): AGENTWM injects active watermarks into action sequences for intellectual property protection via model modification. In contrast, AgentProv performs passive, post-hoc verification on unmodified off-the-shelf deployments.

Rating

  • Novelty: โญโญโญโญโญ Pioneering transition from text-channel completions to internalized action-policy probing for black-box API auditing.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous validation across 36 open-weight models, 9 closed-source models, OpenRouter endpoints, and adaptive attack scenarios.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear framing, mathematically sound derivations, and clean graphical illustrations.
  • Value: โญโญโญโญโญ High industrial relevance for policing commercial LLM aggregators and enforcing model integrity in enterprise deployments.