Skip to content
    See every crawl by page and by botSee every crawl, by page
    HadoSEO

    [ crawl analytics ]

    Every crawl, by page and by bot.

    AI crawler analytics for every page: know which pages AI is reading, and whether it's learning about you or answering questions about you right now. Log file analysis without the log files: every request from GPTBot, ClaudeBot, Googlebot and 40+ crawlers, logged and verified. No exports. No sampling.

    Start free trialRun free scan
    access.log
    [ your access logs, already parsed ]
    timebotpathresult
    • 18:34:25"ClaudeBot"/.env403spoofed
    • 18:34:20ChatGPT-User/pricing200answer
    • 18:34:15YandexBot/200index
    • 18:34:09Amazonbot/docs/api200training
    • 18:34:02Google-Extended/blog/aeo-vs-seo200training
    • 18:33:55Applebot/product/prerendering200index
    • 18:33:50ClaudeBot/security200training

    crawls 30d 12,840 · verified 84.2% · answer fetches 46% · median render 210ms

    [ by page ]

    Open any page. See who read it, and when.

    Bot traffic by page: every path on your site shows exactly which crawlers read it, how often, and how recently.

    Crawl recency per bot, to the hourVerified, declared and unverified, side by sideFilter by path prefix: /blog/, /docs/Scanner probes like /.env show up too
    app.hadoseo.com/acme-app.com/pages/pricing
    acme-app.com▾

    Pages ›

    /pricing

    Last 30 days ▾

    CrawlsTotal crawls

    760

    Last crawlLast crawled

    2h ago

    CachedServed from cache

    100%

    532 verified104 declared124 unverified

    Crawls by bot

    GooglebotGPTBot

    Last crawl by bot

    Googlebotverified2h ago
    GPTBotverified5h ago
    ClaudeBotverified1d ago
    PerplexityBotverified2d ago
    "Slackbot"unverified6d ago

    ✓Every visit logged as it happens. Real bots checked against published IP ranges.

    [ by bot ]

    Open any crawler. See every page it fetched.

    Every page each crawler fetched, from real requests. The gaps matter as much as the hits. Spot the pages AI has never seen.

    app.hadoseo.com/acme-app.com/bots/gptbot
    acme-app.com▾

    Bots › OpenAI

    GPTBottraining

    Last 30 days ▾

    CrawlsTotal crawls

    2,825

    Verified

    2,090

    Pages readPages read

    40/4140 of 41

    Crawls by page

    PageCrawlsLast
    /pricing3125h ago
    /blog/aeo-vs-seo2681d ago
    /docs/api2212d ago
    /security1021w ago
    /changelog0never

    /changelog has never been read by GPTBot.

    AI crawlers
    GPTBotOpenAI · training22%
    GooglebotGoogle · search index19%
    ClaudeBotAnthropic · training12%
    PerplexityBotPerplexity · AI Search Index8%
    Movers · week over week
    PerplexityBot↑ +198%
    ChatGPT-User↑ +64%
    Bytespider↓ -41%

    [ verified bots ]

    The real ChatGPT, or a scraper pretending to be it?

    Scrapers borrow big names to sneak past. Every visit is checked against the operator's published IP ranges and reverse DNS, so a name like Googlebot only counts when it really is Googlebot. The AI crawler list shows every bot we verify and the official IP list each one is checked against.

    Reverse-DNS + published IP ranges confirm the bot is who it claims.

    GPTBot · Googlebot · ClaudeBot

    declared

    A recognized user-agent with no verifiable origin. Logged, not trusted.

    SemrushBot · CCBot

    Claims a verified name from an unverifiable IP. Flagged in your logs.

    "Googlebot" from a datacenter IP

    last 30 days

    • verified84.2%10,811
    • declared11.5%1,477
    • spoofed4.3%552 flagged

    552 unverified requests impersonating known AI crawlers · "ChatGPT-User" · "PerplexityBot" · "GPTBot"

    how impersonation detection works →

    [ intent ]

    Is AI learning about you, or answering questions with your pages right now?

    • answer fetch46%
    • search index38%
    • training16%
    • Answering someone's question about you right nowLive retrieval to build an answer.
    • Filing you for searchBuilding a searchable index for later.
    • Learning about youGathering pages to train or refresh a model.

    Switch the crawl volume chart between operator, intent and tier.

    operatorintenttier

    [ snapshot versions ]

    The exact HTML each bot was served, version by version.DNS + prerender

    Every copy we serve is versioned. See which bot got which build on which day, and open exactly what it read.

    app.hadoseo.com/acme-app.com/pages/pricing/versions
    acme-app.com▾

    Pages › /pricing ›

    Which bot got which version

    Sep 22 – Oct 3 ▾
    GPTBotv11v12v12v13
    ClaudeBotv11v12v12v13
    Googlebotv11v12v12v13
    PerplexityBotv11v12v13
    Bingbotv11v11v12v13
    VersionHashStatusSizeServed
    v13d308deca200134.2 kBOct 2 – Oct 3View source →
    v12a91f04c7200133.8 kBSep 27 – Oct 2View source →
    v114be2d110200129.4 kBSep 22 – Sep 27View source →

    v13d308deca200 · 134.2 kB

    Oct 2 – Oct 3View source →

    v12a91f04c7200 · 133.8 kB

    Sep 27 – Oct 2View source →

    v114be2d110200 · 129.4 kB

    Sep 22 – Sep 27View source →

    ✓See exactly what each bot was served, on the day it visited.

    curl
    per-request record · for developers
    bot=GPTBot intent=answer-fetch url=/pricing served_from=snapshot v13 snapshot_age=2m14s render=210ms status=200 cache=HIT answer_fetch=ChatGPT

    On the Cloudflare Worker in analytics-only mode, requests are logged as served from origin, with origin-captured snapshots for view-source.

    [ the funnel ]

    Crawler reads → answer fetches → likely cited → visits from AI.

    Reads, serves and visits are counted from real requests. Answer fetches are live retrievals by AI assistants while they compose a reply; likely cited is the subset we attribute to an answer.

    [ HadoSEO does the work ]

    12,840crawler reads

    99% served

    12,690served in full

    46% of reads

    [ proof it paid off ]

    5,906answer fetches

    14% likely cited

    824likely cited

    23% clicked

    189visits from AI

    Funnel, last 30 days (illustrative sample)
    stage count conversion
    crawler reads 12,840 —
    served in full 12,690 99% served
    answer fetches 5,906 46% of reads
    likely cited 824 14% likely cited
    visits from AI 189 23% clicked
    Sample site, one month. Bar heights are compressed so small numbers stay visible.→ 189 people from AI answers this month

    12,840

    crawler reads

    12,690

    served in full · 99%

    5,906

    answer fetches · 46% of reads

    824

    likely cited · 14%

    189

    visits from AI · 23%

    [ per engine ]

    ChatGPT2,410 · 386 16%
    Perplexity1,520 · 228 15%
    Claude1,090 · 131 12%
    Gemini640 · 51 8%

    + 3 more engines →

    [ setup ]

    Same analytics on either path.

    path a · dns record

    your domain provider · DNS
    DNS recordsyourdomain.com
    TypeNameValueTTL
    MX@mail.protection.example1h
    A@137.66.32.95Auto
    TXT@v=spf1 include:example …1h

    ✓Saved. Crawlers start getting full pages.

    path b · cloudflare worker

    dash.cloudflare.com
    Workers Routesyourdomain.com
    RouteWorkerStatus
    yourdomain.com/*hadoseo-workerActive

    Added to the zone once. No cutover, no downtime; visitors pass straight through.

    ✓Route active. Every request logged.

    [ + ]compare the two paths
    feature DNS Cloudflare Worker
    best for DNSClient-side apps: Lovable, Base44, React, Angular Cloudflare WorkerServer-rendered sites, or client-side apps you run on Cloudflare
    setup DNSAn A record at your domain provider. No code, no redeploy. About 5 minutes. Cloudflare WorkerA small Worker on your Cloudflare zone, with your API key. About 10 minutes.
    you need DNSAccess to your domain's DNS settings Cloudflare WorkerYour domain on Cloudflare
    traffic flows DNSThrough HadoSEO's edge Cloudflare WorkerThrough your own Cloudflare zone
    crawl analytics DNS✓ every request Cloudflare Worker✓ every request
    prerendering for bots DNS✓ automatic Cloudflare WorkerYour choice: prerendered pages, or analytics-only
    Connect with DNS →Deploy the Worker →

    Features that need prerendering (snapshot versions, served-from cache) carry the DNS + prerender chip.

    [ export + api ]

    Your crawl data, wherever you need it.

    Export scoped JSON for your warehouse, or query the same records over the API with an API key. API docs →

    curl
    GET /v1/crawls?intent=answer-fetch&verified=true
    { "bot": "GPTBot", "tier": "verified", "intent": "answer-fetch", "path": "/pricing", "status": 200, "answer_fetch": "ChatGPT" }

    [ what you'll know ]

    Questions your logs can finally answer.

    Have a week of access logs already? Run them through the free SEO log file analyzer to answer a few of these today, in your browser.

    Which pages has GPTBot never read?

    Every path, every bot, every gap.

    /pricing ✓/blog/new-post · never

    When did Googlebot last see my pricing page?

    Crawl recency per bot, to the hour.

    Googlebot · 3h ago

    Which version of this page did ChatGPT get?

    Versioned snapshots with view-source.

    Who's pretending to be ClaudeBot?

    Unverified requests flagged by IP ownership.

    "ClaudeBot" · unverified

    Which AI fetches my pages but never cites them?

    Indexed → fetched → cited, per engine.

    Which crawlers are ramping up?

    Rising and falling bots, week over week.

    PerplexityBot ↑ 198%

    How much crawl load never hit my server?

    Cache, prerender or origin, per request.

    Who's probing /.env?

    Scanner traffic surfaces in the same logs.

    /.env · blocked 403

    [ questions ]

    Crawl analytics, asked directly.

    [ start logging ]

    Your crawl logs start the moment you connect.

    Start free trialRun free scan