If verification failure just falls back to normal bot detection, what stops an attacker from sending garbage signatures?

Nothing, and that is intentional. A forged signature gains the attacker nothing, because they cannot produce a valid one without the private key behind a published directory. The goal is not to stop invalid traffic - it is to give well-behaved operators a way to distinguish themselves from it.

Does signing my requests mean I will stop getting blocked?

No. It means you get identified, which is not the same thing. A verified identity that ignores rate limits or crawls disallowed paths is easier to block than an anonymous one, not harder, because the origin now has something stable to attach a policy to. Signing is a commitment to behave, and it is enforceable.

Can I use Web Bot Auth with an existing HTTP client?

Yes. It is ordinary request headers, so any client that lets you set them works. The work is in constructing the signature base correctly and re-signing per request, since `created` and `expires` change every time. Do not cache a signature across requests.

How do I detect agentic browsers if they look like real Chrome?

Not by user agent or surface fingerprint, which will match genuine Chrome. The signal is in behavior and timing - navigation that is too direct, form fills without plausible input events, sessions that never idle - plus network-layer artifacts from the control channel. Where the agent signs via Web Bot Auth you get the answer for free, which is the outcome the standard is trying to buy.

The Bot Identity Problem: What Happens When Bots Start Signing Their Requests

I have spent years making automated clients indistinguishable from real browsers. Matching TLS fingerprints, replaying HTTP/2 SETTINGS frames in the right order, patching navigator.webdriver before the detection script reads it. The entire discipline rests on one assumption: the bot wants to look like a human.

That assumption is now optional. A mechanism called Web Bot Auth lets a bot cryptographically prove it is a bot, on every single request, and get better treatment for it. The interesting part is not the crypto, which is boring by design. It is what happens to everyone who keeps hiding.

What Web Bot Auth Actually Is

Web Bot Auth is an IETF Internet-Draft, not a ratified standard. That distinction matters and most write-ups get it wrong - I have seen several claim a "finalized W3C specification," which does not exist. What does exist is a set of drafts profiling RFC 9421 (HTTP Message Signatures, published 2024) for the specific case of bot identity.

The shape is simple. A bot holds an Ed25519 key pair, publishes the public half as a JWK Set at a well-known URL on its own domain, and signs every outbound request. Three headers carry it:

Header	Carries
`Signature-Agent`	URL of the key directory
`Signature-Input`	Which components are signed, plus `created`, `expires`, `keyid`, `tag`
`Signature`	The Ed25519 signature itself

The directory lives at /.well-known/http-message-signature-directory. There is no registration handshake, no shared secret, no API key to leak. Cloudflare announced the mechanism in May 2025 and shipped signed agent classification that August, with Browser Rendering, ChatGPT agent, Browserbase and Anchor Browser in the first cohort. AWS WAF, Akamai, Vercel and Shopify have since implemented support.

The Signature Base Is the Whole Trick

Everyone fixates on the key. The key is the easy part. What actually decides whether your signature validates is the signature base: a canonical serialization of the request components you claim to be signing. Get one byte wrong and the signature is arithmetically fine and completely useless.

RFC 9421 pins this down precisely, and section B.2.6 gives a normative Ed25519 test vector. Here is a signer that reproduces it exactly:

import base64
from cryptography.hazmat.primitives.serialization import load_pem_private_key

# RFC 9421 B.1.4 test key. Never ship a key published in a spec.
PRIVATE_KEY_PEM = b"""-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEIJ+DYvh6SEqVTm50DFtMDoQikTmiCqirVv9mWG9qfSnF
-----END PRIVATE KEY-----"""


def build_signature_base(components, params):
    """Serialize covered components per RFC 9421 section 2.5."""
    lines = ['"{}": {}'.format(name, value) for name, value in components.items()]
    lines.append('"@signature-params": {}'.format(params))
    return "\n".join(lines)


def sign(base, pem):
    key = load_pem_private_key(pem, password=None)
    return base64.b64encode(key.sign(base.encode())).decode()


components = {
    "date": "Tue, 20 Apr 2021 02:07:55 GMT",
    "@method": "POST",
    "@path": "/foo",
    "@authority": "example.com",
    "content-type": "application/json",
    "content-length": "18",
}
params = (
    '("date" "@method" "@path" "@authority" "content-type" "content-length")'
    ';created=1618884473;keyid="test-key-ed25519"'
)

print(sign(build_signature_base(components, params), PRIVATE_KEY_PEM))
# wqcAqbmYJ2ji2glfAMaRy4gruYYnx2nEFN2HN6jrnDnQCK1u02Gb04v9EDgwUPiu4A0w6vuQv5lIp5WPpBKRCw==

Run it. The output matches the signature printed in the RFC byte for byte. That is a correctness check worth having, because a signing bug produces a well-formed signature that simply fails verification, with no useful error on either side.

Three details bite people:

Component order is significant. The base lists components in the order declared in @signature-params, not alphabetically and not in header order.
@signature-params always comes last, and its value is the parameter list verbatim, including the parentheses.
Field values are header values after normalization, not raw wire bytes. Leading and trailing whitespace goes, and obsolete line folding gets collapsed.

For real Web Bot Auth traffic you also set tag="web-bot-auth" and an expires in the near future. Verifiers reject stale signatures, which is what stops someone lifting your Signature header out of a log and replaying it.

What Verification Looks Like From the Other Side

The defender's path is mechanical: confirm the headers exist, check keyid against a known directory, confirm the signature has not expired, confirm tag is web-bot-auth, rebuild the signature base from the request as received, verify against the published key.

The step that matters operationally is the last one: what happens on failure. Cloudflare falls back to conventional bot detection - scoring, fingerprinting, challenges. A failed signature does not get you blocked. It gets you treated exactly as if you had never signed.

That asymmetry is the entire incentive design. Signing is upside-only for the bot operator, which is why adoption is moving faster than the draft status suggests.

The Fork in the Road

If you run scrapers at any scale, this splits your options cleanly, and the split is less comfortable than it first looks.

Sign, and become legible. You get verified-agent classification, predictable rate limits, and a support path when you are blocked by mistake. You also get a stable identity attached to every request you have ever made, which means rate limits you cannot escape by rotating infrastructure, and an auditable record of your behavior. Legibility cuts both ways.

Stay stealth, against a worsening baseline. Cloudflare now blocks AI crawlers by default on many properties and ships pay-per-crawl, which answers unpaid requests with HTTP 402. One analysis of robots.txt and network traffic found 403 responses to AI bots more than doubled year over year, from 3.63% to 8.56% of requests. The open-by-default web is being partitioned into open, blocked, and paid.

I do not think this is a clean win for either side, and I am suspicious of anyone selling it as one. Identity solves attribution for well-behaved operators who were never really the problem. It does nothing about an adversary who simply does not sign, which is the population anti-bot systems were built for in the first place. What it genuinely changes is the default: unsigned traffic stops being the norm and starts being a signal.

The Gray Zone Nobody Has Modeled

Agentic browsers break the taxonomy. When someone asks an AI browser to book a flight, a human initiated the request and a machine executed it. Is that a bot?

robots.txt has no vocabulary for this. Neither do bot scores, which collapse a spectrum onto a single axis. And the browsers are Chromium, so surface fingerprinting sees ordinary Chrome - you are down to behavioral signals and network-layer artifacts. HUMAN Security reported roughly a 6,900% rise in agentic requests since mid-2025.

The category is also consolidating fast enough to make specifics a bad bet. OpenAI announced on July 9, 2026 that it was retiring Atlas, shutting it down a month later and folding browsing into the ChatGPT app. Build detection against a product name and you are maintaining a dead branch within a year. Build it against the behavior - a user-directed, machine-executed session - and it survives the churn. Web Bot Auth's framing is the useful one here: signed agents are defined as user-directed rather than company-directed, which is a property of the traffic, not a brand.

Key Takeaways

Web Bot Auth is an IETF draft profiling RFC 9421 for bot identity. It is not a finalized W3C standard, whatever the SEO blogs tell you.
The mechanism is Ed25519 signatures over a canonical signature base, with the public key at /.well-known/http-message-signature-directory.
The signature base, not the key, is where implementations break. Validate against the RFC 9421 B.2.6 test vector before debugging anything else.
Always set expires. It is what makes a captured signature non-replayable.
Failed verification falls back to normal bot detection rather than blocking, which makes signing upside-only and explains the adoption rate.
Signing buys legibility and pays for it with a permanent identity you cannot rotate away from.
The unsigned baseline is worsening independently: default AI-crawler blocking, HTTP 402 pay-per-crawl, and 403 rates to AI bots that doubled year over year.
Detect agentic traffic by behavior, not product name. The products are consolidating faster than your rules can track.

This is an educational overview of public IETF drafts and published RFCs, written for security research and authorized testing. Always ensure you have permission before testing any system.