Here's a fact that should make any rule based on User-Agent uncomfortable: the user-agent string is free text. Anything can claim to be GPTBot. Your "allow the good bots, block the bad ones" config trusts a header that costs nothing to forge. For years that was an acceptable risk. In an agentic web where you're making allow/charge/block decisions — and money — per request, it isn't.
2026 produced a real answer: Web Bot Auth, a way for an automated client to prove its identity cryptographically rather than just assert it in a string. The spec is advancing on the IETF standards track — a working-group draft as of 2026 — and is already enforced in production at the edge across the major platforms. This is the identity layer that sits underneath charging — and it's worth understanding before you price a single request.
The problem: identity by assertion
When a request arrives claiming to be a known crawler, you have two bad options:
- Trust the string. Easy, and wrong — impersonation is a copy-paste away. Anyone wanting your content for free spoofs a friendly user-agent and walks past your rules.
- Verify by IP range. Better, but brittle: IP lists go stale, rotate, and don't exist for every agent. It's a maintenance treadmill that never quite catches up.
Neither gives you a confident answer to the only question that matters before you act on a request: is this agent actually who it says it is?
The fix: identity by signature
Web Bot Auth flips assertion into proof. An agent signs its requests with a private key (using HTTP Message Signatures) and publishes the matching public key in a directory you can check. When the request arrives, the signature either verifies against that agent's known key — or it doesn't. A spoofer can copy the user-agent string all day; it can't produce the signature without the key.
That gives you something you've never reliably had: a cryptographic answer to "which agent is this," at the edge, before the request reaches your origin. The approach has converged into a standard with broad backing — it's used alongside the card networks' agent-payment programs at the edge, and it's supported across the major edge and cloud platforms.
Why identity comes before the price
You might ask: x402 already lets me charge an agent without knowing who it is — the 402 challenge cares about payment proof, not the caller's name. That's true and deliberate. x402 identifies the payment, not the caller, which is exactly right for an open market: anyone with a wallet can pay for one call and disappear.
So why bother verifying identity at all? Because identity unlocks decisions that payment alone can't:
- Differentiated rules. Allow a verified partner agent through free, charge unknown agents, and block the ones that fail verification — all based on proven identity instead of a spoofable string.
- Differentiated pricing. Quote a verified, high-volume agent a different price than an anonymous one-off caller. The price can depend on who's really asking.
- Trust and abuse control. Reserve free or cheap access for agents that have proven who they are, and make impersonators pay the full, unfriendly rate — or nothing.
Identity answers who. x402 answers did they pay. You want both hands on the wheel.
How the two compose
Picture a single request to a priced resource, decided at the edge. Verification and charging are two layers that stack:
- Verify (Web Bot Auth). Your edge platform checks the request signature against the caller's published key — Web Bot Auth support lives in the edge/CDN layer (Cloudflare and the other major platforms), not in the charging layer. Now you know whether this is a verified agent, an unknown one, or an impersonator.
- Decide (your rules). Allow a verified partner through, block a failed impersonator, or send the request on to be charged.
- Charge (x402). Payrelayer returns the 402 with the quote; the agent pays per request and proceeds.
The important part: both happen at the edge, before your origin does any work — the platform proves who's calling, the paywall handles getting paid. Today Payrelayer matches agents by user-agent and path rules (the same place you'd write your crawler rules — see charge AI crawlers for site owners); layering Web Bot Auth on top upgrades "this claims to be GPTBot" into "this is cryptographically GPTBot" before you decide what to charge. The broader case for charging is in don't block AI crawlers — charge them.
Where this fits in the bigger map
Identity is one of the layers in the agent-payment stack — alongside authorization, checkout, and settlement, which we mapped in x402 vs AP2 vs ACP vs MPP. Web Bot Auth is the identity layer; x402 is the settlement layer; they don't compete, they stack. As the agentic web matures, "we charge agents" and "we verify which agent we're charging" stop being separate projects and become one policy.
The takeaway
Stop trusting the user-agent string. Verified identity via Web Bot Auth turns "this request claims to be GPTBot" into "this request is the agent it says it is, cryptographically" — and that's the foundation for charging some agents, trusting others, and blocking impersonators, all on proof instead of a guess. Pair it with per-request charging and you have an edge that knows who's knocking and gets paid when it opens the door.
Next steps
- The case for charging, not blocking: don't block AI crawlers — charge them
- The edge walkthrough for site owners: charge AI crawlers (Cloudflare path)
- Where identity fits the stack: x402 vs AP2 vs ACP vs MPP
- For site owners: charging AI crawlers for your content · create an account