◇Paywall
2026-06-24 · 4 min read

Verify the agent before you charge it: Web Bot Auth, signed agents, and x402

Here's a fact that should make any rule based on User-Agent uncomfortable: the user-agent string is free text. Anything can claim to be GPTBot. Your "allow the good bots, block the bad ones" config trusts a header that costs nothing to forge. For years that was an acceptable risk. In an agentic web where you're making allow/charge/block decisions — and money — per request, it isn't.

2026 produced a real answer: Web Bot Auth, a way for an automated client to prove its identity cryptographically rather than just assert it in a string. The spec was finalized in 2026 and is enforced at the edge across the major platforms. This is the identity layer that sits underneath charging — and it's worth understanding before you price a single request.

The problem: identity by assertion

When a request arrives claiming to be a known crawler, you have two bad options:

  • Trust the string. Easy, and wrong — impersonation is a copy-paste away. Anyone wanting your content for free spoofs a friendly user-agent and walks past your rules.
  • Verify by IP range. Better, but brittle: IP lists go stale, rotate, and don't exist for every agent. It's a maintenance treadmill that never quite catches up.

Neither gives you a confident answer to the only question that matters before you act on a request: is this agent actually who it says it is?

The fix: identity by signature

Web Bot Auth flips assertion into proof. An agent signs its requests with a private key (using HTTP Message Signatures) and publishes the matching public key in a directory you can check. When the request arrives, the signature either verifies against that agent's known key — or it doesn't. A spoofer can copy the user-agent string all day; it can't produce the signature without the key.

That gives you something you've never reliably had: a cryptographic answer to "which agent is this," at the edge, before the request reaches your origin. The approach has converged into a standard with broad backing — it's being used as the identity foundation under the card networks' agent-payment programs, and it's supported across the major edge and cloud platforms.

Why identity comes before the price

You might ask: x402 already lets me charge an agent without knowing who it is — the 402 challenge cares about payment proof, not the caller's name. That's true and deliberate. x402 identifies the payment, not the caller, which is exactly right for an open market: anyone with a wallet can pay for one call and disappear.

So why bother verifying identity at all? Because identity unlocks decisions that payment alone can't:

  • Differentiated rules. Allow a verified partner agent through free, charge unknown agents, and block the ones that fail verification — all based on proven identity instead of a spoofable string.
  • Differentiated pricing. Quote a verified, high-volume agent a different price than an anonymous one-off caller. The price can depend on who's really asking.
  • Trust and abuse control. Reserve free or cheap access for agents that have proven who they are, and make impersonators pay the full, unfriendly rate — or nothing.

Identity answers who. x402 answers did they pay. You want both hands on the wheel.

How the two compose

Picture a single request to a priced resource, decided at the edge:

  1. Verify (Web Bot Auth). Is the signature valid? Now you know whether this is a verified agent, an unknown one, or an impersonator.
  2. Decide (your rules). Based on the verified identity: allow, block, or quote a price — and which price.
  3. Charge (x402). If the decision is "charge," return the 402 with the quote; the agent pays per request and proceeds.

The important part: both checks happen at the edge, before your origin does any work. You're not bolting identity onto your application code — you're making it part of the same turnstile that already decides allow/charge/block. For site owners, that's the same place you'd write your crawler rules today; see charge AI crawlers for site owners and the broader case in don't block AI crawlers — charge them.

Where this fits in the bigger map

Identity is one of the layers in the agent-payment stack — alongside authorization, checkout, and settlement, which we mapped in x402 vs AP2 vs ACP vs MPP. Web Bot Auth is the identity layer; x402 is the settlement layer; they don't compete, they stack. As the agentic web matures, "we charge agents" and "we verify which agent we're charging" stop being separate projects and become one policy.

The takeaway

Stop trusting the user-agent string. Verified identity via Web Bot Auth turns "this request claims to be GPTBot" into "this request is the agent it says it is, cryptographically" — and that's the foundation for charging some agents, trusting others, and blocking impersonators, all on proof instead of a guess. Pair it with per-request charging and you have an edge that knows who's knocking and gets paid when it opens the door.

Next steps

Share:XLinkedInEmail
Related reading
Is x402 worth it yet? An honest look at the skeptics

The loudest take on x402 is that demand isn't there, it's just crypto, and micropayments always fail. Some of that is fair. Here's an honest answer to each objection — including the one reason the old micropayment arguments don't apply when the buyer is software.

x402 vs AP2 vs ACP vs MPP: which agent-payment protocol do you actually need?

2026 gave us an alphabet soup of agent-payment standards — x402, AP2, ACP, MPP, plus L402. They sound like competitors. They're mostly layers. Here's the map: what each one does, where they overlap, and how to pick without guessing.

Don't block AI crawlers — charge them

Blocking GPTBot and ClaudeBot is the default advice in 2026, and it's the option that leaves money on the table. Here's the third choice — charge crawlers per request at the edge — and when it beats both blocking and allowing.

All postsBrowse the SDKs Glossary Use cases