Skip to content
    HadoSEO

    ← all articles

    Technical Help

    HadoSEO · · 9 min read

    How to Get Google to Crawl & Index Your Site (2026 Steps)

    [ summarize with ai ]

    You can ask Google to crawl a page, and it takes about a minute. Getting it indexed is the part that takes care. This guide walks through the exact steps in Google Search Console, how long a recrawl usually takes, and what to check when a page still won't show up.

    If your site is built with a JavaScript framework or an AI app builder (Lovable, Replit, Bolt.new, Base44), read the JavaScript section too. It's the most common reason a requested page never makes it into the index.


    Crawling vs indexing: what you're actually asking for

    Crawling is Googlebot fetching your URL. Indexing is Google deciding to store the page so it can appear in results. Search Console's "Request indexing" button asks for a crawl and an indexing decision, but it can't force a yes.

    Google indexes a page when it can fetch it, read real content in it, and decide it's worth keeping. Most "Google won't index my site" problems are one of those three failing, so the steps below check each one before you press the button.


    How to get Google to crawl and index a page, step by step

    You need a Google account and access to your domain's DNS or site files (for verification). Everything else happens inside Google Search Console.

    Step 1: Verify your site in Google Search Console

    Add your site as a property. You'll be offered two types:

    • Domain property (e.g. example.com): covers every subdomain and both http and https. It's verified by adding a DNS TXT record at your domain registrar or DNS host.
    • URL-prefix property (e.g. https://www.example.com/): covers only that exact origin. It can be verified by DNS, an HTML file upload, a meta tag, Google Analytics or Google Tag Manager.

    If you can edit DNS, choose the Domain property. It saves you from missing data when Google picks the www or non-www version.

    Step 2: Inspect the URL

    Paste the full URL, including https://, into the inspection bar at the top of Search Console and press Enter. Google returns the indexed status of that exact URL:

    • URL is on Google: it's indexed. If you changed the page, you can still request a recrawl.
    • URL is not on Google: the report tells you why, under Page indexing. Common reasons are "Discovered – currently not indexed", "Crawled – currently not indexed", "Excluded by 'noindex' tag", "Blocked by robots.txt", "Duplicate without user-selected canonical" and "Page with redirect".

    Note the reason. It decides what you do next. Our guide to Crawled and Discovered, currently not indexed covers the two most confusing ones in detail.

    Step 3: Test the live URL

    Click Test live URL. Google fetches the page right now, as Googlebot, and reports whether it could be indexed. When it finishes, click View tested page. You get three tabs:

    • HTML: the rendered HTML Google ended up with.
    • Screenshot: what the page looked like to Google.
    • More info: page resources that failed to load, and JavaScript console messages.

    Look for your actual headline and paragraph text in the HTML tab. If you see an almost empty <div id="root"></div>, a loading spinner in the screenshot, or a title that's the same on every page, Google isn't getting your content reliably. Skip to the JavaScript section before requesting indexing.

    Step 4: Fix what blocks indexing

    Requesting indexing for a page Google can't use wastes a request from your daily quota. Before you click, check:

    • No noindex. No <meta name="robots" content="noindex"> in the HTML and no X-Robots-Tag: noindex header.
    • Not blocked in robots.txt. The live test says so if it is.
    • A 200 status. Not a 404, a 5xx, or a redirect to somewhere else.
    • The right canonical. The page's <link rel="canonical"> points to itself, not to the homepage or another URL.
    • Real content in the HTML. Covered in step 3.
    • At least one internal link. Pages nothing links to look unimportant, and are harder for Google to rediscover.

    Step 5: Request indexing

    Click Request indexing. Google runs a short live check, then confirms the URL was added to a priority crawl queue. That's it for this URL.

    Two things Google is clear about in its recrawl documentation: there's a quota for individual URL requests, and requesting the same URL several times won't get it crawled any faster. Request once, then wait.

    Step 6: Submit a sitemap for the rest of the site

    Request indexing works one URL at a time. For a whole site, submit a sitemap:

    1. Open Sitemaps in the left menu of Search Console.
    2. Enter your sitemap URL, usually https://yourdomain.com/sitemap.xml.
    3. Click Submit.

    The report shows Success if Google could fetch and read it, or Couldn't fetch if the URL is blocked, missing or invalid. Also add a Sitemap: https://yourdomain.com/sitemap.xml line to your robots.txt so every crawler can find it.

    Keep <lastmod> dates honest. Google uses them when they're consistently accurate, and ignores them when every page claims to have changed today.


    How long until Google recrawls?

    Google's own answer is that crawling can take anywhere from a few days to a few weeks. In practice, a few things move a page toward the short end:

    • Links from pages Google already visits often. A link from your homepage or navigation gets a new page discovered on the next crawl of that page.
    • An accurate sitemap. New URLs and real <lastmod> changes give Google a reason to come back.
    • A healthy server. If your server is slow or returns errors, Googlebot backs off and crawls less.
    • Complete HTML on the first fetch. Pages that need JavaScript to show their content go through an extra rendering step before Google can use what's on them. Pages that arrive as finished HTML skip that wait.
    • Demand. Popular and frequently updated pages are recrawled more often than pages nobody links to.

    To see when Google actually came by, check your crawl records.

    After a recrawl, the index status can lag by a few more days. Re-inspect rather than searching site:yourdomain.com, which is approximate.


    How to check whether Google actually crawled your page

    Search Console gives you two views: Settings → Crawl stats for site-wide totals, and the "Last crawl" date in URL Inspection for one page. Both are useful, but Crawl stats is sampled and both run behind real time. Your server logs are the only record of every request Googlebot made.

    With log file analysis, you can see the full picture. Upload a week of access logs to the free analyzer and it lists each verified Googlebot visit by page, so you can see whether a page you requested was fetched, and when.

    Don't trust the user agent alone. Anyone can send "Googlebot" in a request, and in our analysis of 1.3 million requests, about 40% of "Googlebot" traffic was fake. To check whether a Googlebot IP is real, paste it into the free verifier.

    Many hosts don't give you raw logs at all, including Lovable, Base44, Vercel and Netlify. For those, crawl analytics records every request at the edge instead: live, verified crawl logs by page and by bot, with no log export.

    The same logs show AI crawlers too, such as GPTBot, ClaudeBot and PerplexityBot. Our AI crawler list covers what each one does.


    Still not indexed? It may be a JavaScript problem

    If you followed every step and the page is still "Discovered – currently not indexed" or "Crawled – currently not indexed" weeks later, check how the page is built.

    Apps made with React, Vue or Svelte, and most apps from AI builders like Lovable, Replit, Bolt.new and Base44, build each page in the browser. The HTML the server sends is a nearly empty shell plus a script. A visitor's browser runs the script and the page appears. A crawler that reads the raw HTML sees almost nothing.

    Google can run JavaScript, but rendering is a separate, later step, and the result isn't guaranteed. Titles and descriptions set by JavaScript may be missed, every URL can look identical, and thin-looking pages are easy for Google to skip. Most AI crawlers, such as GPTBot and ClaudeBot, don't run JavaScript at all, so the same pages are blank to ChatGPT and Claude.

    Check it in under a minute:

    • Run a free scan. It fetches your pages the way Google and AI crawlers do and shows what each one receives.
    • Try the Googlebot simulator on a single URL to compare what a person sees with what Googlebot gets.
    • Or from a terminal: curl -s https://yourdomain.com/some-page | grep -i "a sentence from your page". No match means the text isn't in the HTML.

    Fix it by giving crawlers complete HTML. You can switch to server-side rendering, generate static pages at build time, or prerender: serve bots a rendered copy while people keep the live app. Our page on prerendering for client-side apps explains the options without a rebuild, and SSR vs SSG vs CSR explains the trade-offs. If you're on Lovable, our Lovable SEO page covers which projects already get server rendering.

    Once crawlers get real HTML, go back to step 3, confirm the live test shows your content, and request indexing again.


    Other ways to help Google find your pages

    None of these replace the steps above, but they make future crawls faster:

    • Link new pages from existing ones. Navigation, footer, a blog index, or a related post.
    • Earn links from other sites. Google discovers URLs by following links, including from other domains.
    • Avoid thin and near-duplicate pages. Large numbers of low-value URLs make it harder for Google to prioritize the good ones.
    • Keep your sitemap current. Remove URLs that redirect or 404.

    What doesn't help: the Indexing API is only for job posting and livestream pages, and the old sitemap "ping" URL was retired in 2023.


    Frequently Asked Questions

    How do I ask Google to crawl my website?

    Verify your site in Google Search Console, paste the page URL into the URL Inspection bar, run Test live URL, then click Request indexing. For many pages at once, submit a sitemap in the Sitemaps report instead.

    How long does it take Google to crawl a page after I request indexing?

    Google's documentation says crawling can take anywhere from a few days to a few weeks. Requesting indexing again doesn't speed it up. Pages that are linked from your important pages, listed in an accurate sitemap and served as complete HTML tend to be picked up faster.

    Is there a limit on Request indexing?

    Yes. Search Console has a daily quota for individual URL submissions. Use it for your most important new or changed pages, and rely on your sitemap for everything else.

    Why is my page still not indexed after requesting indexing?

    Requesting indexing only puts the URL in the crawl queue. Google can still decide not to index it, for example because of a noindex tag, a duplicate canonical, thin content, or because the HTML it received was nearly empty. JavaScript apps that render in the browser are a common cause of the last one.

    Can I still ping Google with my sitemap?

    No. Google retired the sitemap ping endpoint in 2023. Submit your sitemap in Search Console and reference it in robots.txt with a Sitemap: line.

    How do I check if Google crawled my site?

    For a single page, run URL Inspection in Search Console and read the Last crawl date. For site-wide totals, open Settings → Crawl stats. To see every verified Googlebot visit by page, use server log analysis, for example the free HadoSEO log file analyzer.

    Can I see when Googlebot visited a specific page?

    Yes. Your server logs record every request with its time and URL, and HadoSEO crawl analytics shows the same visits live, by page. Both check each visit against Google's published IP ranges and reverse DNS, so fake Googlebots are not counted.


    Related reading:

    [ free scan ]

    How much of your site can AI actually see?

    No signup required · results in ~60 seconds