AI tokens follow each operator's crawler documentation as of October 2026. Google-Extended and Applebot-Extended are control tokens, not crawlers: blocking them keeps pages in Google and Apple search. User-initiated fetchers such as ChatGPT-User, Perplexity-User, and Meta-ExternalFetcher are left out because their operators say robots.txt may not apply to them.

Crawl-delay is not part of RFC 9309. Bingbot and Anthropic's crawlers read it; Googlebot ignores it.