[ search crawler · Anthropic ]
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's search crawler. It indexes pages so Claude can find them when it searches the web.
In robots.txt it answers to the Claude-SearchBot token — rules below — though robots.txt binds only crawlers that choose to honor it. Its intent is search index: stores your pages so they can show up in search results.
See whether Claude-SearchBot actually visits your site. Upload a week of server logs to the free log file analyzer, or connect crawl analytics for verified visits by page, live.
IP ranges last updated Nov 8, 2026
Updated · HadoSEO
[ technical facts ]
- operator
- Anthropic
- intent
- search index
- user agent contains
claude-searchbot- robots.txt token
Claude-SearchBot- verification
- published IP rangesverified
- official IP list
- claude.com/crawling/bots.json
- ranges in registry
- 14 ranges loaded
- last updated
- Nov 8, 2026
- operator docs
- support.anthropic.com/en/articles/8896518
[ user agent ]
The Claude-SearchBot user agent.
claude-searchbot
The full User-Agent string includes a version number that changes over time; Anthropic's documentation lists the current one. A matching user agent is a claim, not proof — verify the IP below.
[ verify ]
How to verify a real Claude-SearchBot request.
Anthropic publishes the IP ranges Claude-SearchBot uses. A request is real when its IP is inside that list.
- 01Check the User-Agent header contains "claude-searchbot" (case-insensitive). That is only a claim: anyone can send it.
- 02Download Anthropic's published list (https://claude.com/crawling/bots.json) and check the request IP falls inside one of its ranges.
- 03Inside the list: the real Claude-SearchBot. Outside it: a request using Claude-SearchBot's name from someone else's server.
Skip the lookups: the free bot verifier checks one IP against the same registry, and the log file analyzer checks every Claude-SearchBot request in a week of access logs. Requests that fail are flagged as spoofed, not hidden.
[ robots.txt ]
Block or allow Claude-SearchBot in robots.txt.
Add one of these groups to the robots.txt at your domain root. The token matches case-insensitively. Anthropic runs other crawlers with their own tokens, so a rule for Claude-SearchBot leaves ClaudeBot (training) and Claude-User (answer fetch) untouched.
Mind the training vs. search split: blocking ClaudeBot (training) does not block Claude-SearchBot (search index) or Claude-User (answer fetch). A site can opt out of Anthropic's model training and still appear in Claude search results — each crawler needs its own robots.txt rule.
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
robots.txt only binds crawlers that choose to follow it. A scraper sending Claude-SearchBot's user agent from its own servers won't, which is why HadoSEO checks the IP behind every request.
[ questions ]
Claude-SearchBot, answered.
[ free scan ]
