Skip to content
    HadoSEO

    [ search crawler · Anthropic ]

    What is Claude-SearchBot?

    Claude-SearchBot is Anthropic's search crawler. It indexes pages so Claude can find them when it searches the web.

    In robots.txt it answers to the Claude-SearchBot token — rules below — though robots.txt binds only crawlers that choose to honor it. Its intent is search index: stores your pages so they can show up in search results.

    See whether Claude-SearchBot actually visits your site. Upload a week of server logs to the free log file analyzer, or connect crawl analytics for verified visits by page, live.

    search indexverifiable by published IP ranges

    IP ranges last updated Nov 8, 2026

    Updated · HadoSEO

    [ technical facts ]

    operator
    Anthropic
    intent
    search index
    user agent contains
    claude-searchbot
    robots.txt token
    Claude-SearchBot
    verification
    published IP rangesverified
    ranges in registry
    14 ranges loaded
    last updated
    Nov 8, 2026

    [ user agent ]

    The Claude-SearchBot user agent.

    the User-Agent contains (case-insensitive)
    claude-searchbot

    The full User-Agent string includes a version number that changes over time; Anthropic's documentation lists the current one. A matching user agent is a claim, not proof — verify the IP below.

    [ verify ]

    How to verify a real Claude-SearchBot request.

    Anthropic publishes the IP ranges Claude-SearchBot uses. A request is real when its IP is inside that list.

    1. 01Check the User-Agent header contains "claude-searchbot" (case-insensitive). That is only a claim: anyone can send it.
    2. 02Download Anthropic's published list (https://claude.com/crawling/bots.json) and check the request IP falls inside one of its ranges.
    3. 03Inside the list: the real Claude-SearchBot. Outside it: a request using Claude-SearchBot's name from someone else's server.

    Skip the lookups: the free bot verifier checks one IP against the same registry, and the log file analyzer checks every Claude-SearchBot request in a week of access logs. Requests that fail are flagged as spoofed, not hidden.

    [ robots.txt ]

    Block or allow Claude-SearchBot in robots.txt.

    Add one of these groups to the robots.txt at your domain root. The token matches case-insensitively. Anthropic runs other crawlers with their own tokens, so a rule for Claude-SearchBot leaves ClaudeBot (training) and Claude-User (answer fetch) untouched.

    Mind the training vs. search split: blocking ClaudeBot (training) does not block Claude-SearchBot (search index) or Claude-User (answer fetch). A site can opt out of Anthropic's model training and still appear in Claude search results — each crawler needs its own robots.txt rule.

    block Claude-SearchBot
    User-agent: Claude-SearchBot
    Disallow: /
    allow Claude-SearchBot
    User-agent: Claude-SearchBot
    Allow: /

    robots.txt only binds crawlers that choose to follow it. A scraper sending Claude-SearchBot's user agent from its own servers won't, which is why HadoSEO checks the IP behind every request.

    [ questions ]

    Claude-SearchBot, answered.

    [ free scan ]

    How much of your site can AI actually see?

    No signup required · results in ~60 seconds