What is an AI crawler?

An AI crawler is a bot run by an AI company that reads web pages. Some collect pages to train models (training), some build the index AI search draws on (search index), and some fetch a page live while answering a person's question (answer fetch).

How do I block AI crawlers?

Give each crawler you want to stop its own User-agent line in robots.txt, followed by Disallow: /. The snippet on this page blocks the training crawlers while leaving search and answer fetchers alone. robots.txt is honored only by crawlers that choose to follow it.

Which AI crawlers can be verified?

Those whose operators publish IP ranges or a reverse DNS pattern: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, Amazonbot, DuckAssistBot, MistralAI-User. The rest can only be recognized by user agent, which anyone can copy.

What is the difference between training, search index and answer fetch?

Training crawlers download pages to train AI models. Search index crawlers store pages so they can appear in search results. Answer fetchers read a page live because someone just asked a question it can help answer.

Where does this list come from?

From the bot registry HadoSEO's crawl analytics uses to classify and verify every request: each bot's operator, intent, user agent token, verification method and the official IP list it is checked against. The IP range date above is read live from that registry.

[ ai crawler list ]

Every AI crawler, and why it reads your site.

This AI crawler list covers 19 AI and search crawlers from HadoSEO's bot registry: who runs each one, whether it's training a model, building a search index or answering a question right now, the user agent it sends, and the official IP list that proves a request is real.

IP ranges last updated Oct 25, 2026

Verify a bot IP Analyze your server logs

[ training ]

AI training crawlers

Downloads your content to train AI models.

bot	operator	user agent contains	verification
GPTBot	OpenAI	`gptbot`	published IP ranges
ClaudeBot	Anthropic	`claudebot`	published IP ranges
Google-Extended	Google	`google-extended`	user agent only
Applebot-Extended	Apple	`applebot-extended`	user agent only
Bytespider	ByteDance	`bytespider`	user agent only
CCBot	Common Crawl	`ccbot`	user agent only
Amazonbot	Amazon	`amazonbot`	reverse DNS
Meta-ExternalAgent	Meta	`meta-externalagent`	user agent only

[ search index ]

Search index crawlers

Stores your pages so they can show up in search results.

bot	operator	user agent contains	verification
OAI-SearchBot	OpenAI	`oai-searchbot`	published IP ranges
Claude-SearchBot	Anthropic	`claude-searchbot`	published IP ranges
PerplexityBot	Perplexity	`perplexitybot`	published IP ranges
Googlebot	Google	`googlebot`	reverse DNS
Bingbot	Microsoft	`bingbot`	published IP ranges
Applebot	Apple	`applebot`	reverse DNS

[ answer fetch ]

AI answer fetchers

Fetches your page live to answer a user's question.

bot	operator	user agent contains	verification
ChatGPT-User	OpenAI	`chatgpt-user`	published IP ranges
Claude-User	Anthropic	`claude-user`	published IP ranges
Perplexity-User	Perplexity	`perplexity-user`	published IP ranges
DuckAssistBot	DuckDuckGo	`duckassistbot`	published IP ranges
MistralAI-User	Mistral	`mistralai-user`	published IP ranges

[ robots.txt ]

How to block AI crawlers that train models.

This robots.txt group asks the training crawlers above to stay out and leaves search crawlers and answer fetchers alone. Each bot's page has its own allow and block rules.

block AI training crawlers

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
Disallow: /

robots.txt is a request. Scrapers that borrow a crawler's name ignore it, so HadoSEO checks every request's IP against the operator's list and flags the ones that fail. See it on your own site with AI crawler analytics.

[ questions ]

AI crawlers, answered.

[ free scan ]