Skip to content
    See every crawl by page and by botSee every crawl, by page
    HadoSEO

    [ log report · free · in-browser ]

    SEO log file analysis: see every real Googlebot and AI crawler visit.

    Log file analysis (also called server log analysis) shows how search engines and AI crawlers actually crawl your site, straight from your server logs. Upload a week of access logs and this free log file analyzer tells you which bots really visited, which were fake, and which pages AI never read.

    Drop your access logs here

    Apache/Nginx combined or common log format, JSON lines, or .gz: up to a week of traffic.

    Up to 10 files · 1 GB total (uncompressed)

    Your logs are parsed in this tab. Only the IPs of requests claiming to be known bots are checked against our registry. Raw logs and visitor IPs never leave your machine, and nothing is stored.

    Where do I find my logs?
    • Nginx: usually /var/log/nginx/access.log (use the combined log format)
    • Apache: usually /var/log/apache2/access.log; set CustomLog … combined
    • Cloudflare: Logpush to any store, then upload the JSON files. Or skip uploads: the HadoSEO Worker sees every request live.
    • cPanel: Metrics, then Raw Access: download the archive for your domain
    • Vercel, Netlify, Lovable, Base44: these hosts don't give you raw access logs. Connect HadoSEO instead; we'll detect the right setup and build the same report from live traffic.

    [ the basics ]

    What is log file analysis in SEO?

    SEO log file analysis means reviewing your web server logs to see what crawlers actually did, instead of what a crawl tool or Search Console estimates. Every request is one line: the IP, the user agent, the URL and the status code. From those lines you can see how crawl budget is spent, which status codes bots hit (404s, redirects, server errors), whether bots ever reach orphan pages that no internal link points to, and, with bot verification, which "Googlebot" requests really came from Google.

    [ the report ]

    What this log file analyzer shows

    Verified vs. fake bots
    Every IP that claims to be a known crawler is checked against the operator's published IP ranges and reverse DNS. Claims that fail are flagged as fake. Operators that publish nothing are marked as can't verify, never as fake.
    Crawls by page
    Which URLs AI crawlers read and how often, split into verified and unverified crawls, with the visits AI sent to each page.
    Pages AI never read
    Pages your visitors load that no AI crawler fetched during the week. These are the pages AI answers can't quote.
    AI vs. search crawlers
    Every named bot, from Googlebot and Bingbot to GPTBot and ClaudeBot, labeled by intent: classic search, AI training, AI search index or answer fetch.

    [ three steps ]

    How to run a server log analysis for SEO

    1. 01

      Get your logs

      Download about a week of raw access logs, in the combined format so each line carries a user agent.

      • Nginx: usually /var/log/nginx/access.log (use the combined log format)
      • Apache: usually /var/log/apache2/access.log; set CustomLog … combined
      • Cloudflare: Logpush to any store, then upload the JSON files. Or skip uploads: the HadoSEO Worker sees every request live.
      • cPanel: Metrics, then Raw Access: download the archive for your domain

    2. 02

      Upload them

      Drop up to 10 files (1 GB in total) into the log file analyzer at the top of this page. Plain and .gz files both work. Parsing happens in your browser, so the raw log lines never leave your machine.

    3. 03

      Read the report

      Start with the fake-bot callout, then the crawlers table, then the pages AI never read. The report covers the newest seven days in your upload, with days in UTC, and you can save it as a PDF.

    [ live traffic ]

    Log file analysis without uploads

    An upload is a snapshot. HadoSEO crawl analytics builds the same report from live traffic, with every crawler request verified as it arrives. Browser-rendered apps connect with one DNS record; server-rendered sites on Cloudflare add a Cloudflare Worker. Connect HadoSEO and stop exporting logs. Just one suspicious IP? Check it with the fake Googlebot checker.

    [ questions ]

    FAQ

    Is my log data uploaded?

    No. Your log files are read and parsed in this browser tab. The only thing sent to HadoSEO is the list of IP addresses whose user agent claimed to be a known crawler, so they can be checked against published IP ranges. Raw log lines and visitor IPs never leave your machine, and nothing is stored.

    Which log formats are supported?

    Apache and Nginx access logs in the combined or common log format, and JSON lines such as Cloudflare Logpush exports, as plain text or .gz. You can add up to 10 files and 1 GB at once. The common log format has no user agent, so use combined to tell bots from people.

    How many days of logs should I analyze?

    About a week. For log file analysis, seven days shows each crawler's normal rhythm without a huge upload, and the report covers the newest seven days in your files; older lines are skipped. Use recent logs, because crawler behavior changes quickly.

    How is this different from Screaming Frog's Log File Analyser?

    Screaming Frog's Log File Analyser is a desktop application you download and install. This log file analyzer runs in your browser with nothing to install and no account, and it verifies AI crawlers such as GPTBot, ClaudeBot and PerplexityBot against their operators' published IP ranges, alongside Googlebot and Bingbot.