Common Crawl's crawler, classed as AI training: the pages it collects go into a public web archive. Intent: training. Downloads your content to train AI models.

What is the CCBot user agent?

Requests from CCBot carry "ccbot" in the User-Agent header (matching is case-insensitive). The full string includes a version number that changes over time; Common Crawl's documentation lists the current one. In robots.txt, the token is CCBot.

How do I block CCBot in robots.txt?

Add "User-agent: CCBot" followed by "Disallow: /" to your robots.txt. robots.txt is a request, not a lock: a scraper using CCBot's name ignores it, which is why verification matters.

Can I see when CCBot visits my site?

Yes. HadoSEO logs every CCBot request by page, and labels it declared, since Common Crawl publishes nothing to verify it against. For a one-off look, the free log file analyzer reads a week of your server logs in the browser.

[ AI training crawler · Common Crawl ]

CCBot: Common Crawl's AI training crawler.

Common Crawl's crawler, classed as AI training: the pages it collects go into a public web archive. Its intent is training: downloads your content to train AI models.

traininguser agent only

IP ranges last updated Oct 25, 2026

Check a CCBot IP All AI crawlers

[ technical facts ]

operator: Common Crawl
intent: training
user agent contains: ccbot
robots.txt token: CCBot
verification: user agent onlydeclared
official IP list: not published
operator docs: commoncrawl.org/ccbot

[ verify ]

How to verify a real CCBot request.

Common Crawl publishes no IP list or reverse DNS pattern that HadoSEO checks today, so CCBot is recognized by user agent only.

01Check the User-Agent header contains "ccbot" (case-insensitive). That is only a claim: anyone can send it.
02Common Crawl publishes no IP list or reverse DNS pattern for CCBot that HadoSEO can check, so there is no way to prove a request is real.
03Treat it as declared: it says it is CCBot, and nothing confirms or contradicts that.

Skip the lookups: the free bot verifier checks one IP against the same registry, and the log file analyzer checks every CCBot request in a week of access logs. Requests that fail are flagged as spoofed, not hidden.

[ robots.txt ]

Block or allow CCBot in robots.txt.

Add one of these groups to the robots.txt at your domain root. The token matches case-insensitively.

block CCBot

User-agent: CCBot
Disallow: /

allow CCBot

User-agent: CCBot
Allow: /

robots.txt only binds crawlers that choose to follow it. A scraper sending CCBot's user agent from its own servers won't, which is why HadoSEO checks the IP behind every request.

[ questions ]

CCBot, answered.

[ free scan ]