How Google Search works for site owners: crawling, indexing and serving
Distinguish crawling, indexing and serving results when investigating why a page is missing from Google Search.
When a page does not show up in Google, the cause sits in one of three stages. This article summarises Google's "In-depth guide to how Google Search works", as read on 15 November 2026. The page says it was last updated 2025-12-18, so it may not reflect later changes. It covers Google only.
Two statements to start with
Google says it does not accept payment to crawl a site more often or to rank it higher, and that anyone who says otherwise is wrong. It also says it does not guarantee that it will crawl, index or serve a page, even if the page follows Google Search Essentials. Not every page makes it through every stage.
Stage 1: crawling
- There is no central list of web pages. Google finds URLs from pages it already knows (a category page linking to a new post, for example), and from sitemaps you submit.
- Googlebot decides algorithmically which sites to crawl, how often and how many pages. Google says its crawlers try not to crawl too fast, using the site's responses; for example HTTP 500 errors mean "slow down".
- Not every discovered page is crawled. Some are disallowed by the owner, or are behind a login.
- During the crawl Google renders the page, running JavaScript with a recent version of Chrome, because without that it might not see content the JavaScript adds.
- Common access problems the page lists: server problems, network issues, and robots.txt rules that stop Googlebot.
Stage 2: indexing
- Google analyses text and key content tags and attributes such as title elements and alt attributes, plus images and video.
- It decides whether a page is a duplicate of another. It groups pages with similar content (clustering) and picks the most representative one as the canonical, the page that may be shown. The others are alternates that can be shown in other contexts, for example on a mobile device or for a very specific search.
- It collects signals about the canonical, such as page language, the country the content is local to, and usability.
- Indexing is not guaranteed. The page lists common problems: low-quality content, robots meta rules that disallow indexing, and site design that makes indexing difficult.
Stage 3: serving results
- Ranking is done programmatically. Google says relevance is based on hundreds of factors, which can include the user's location, language and device. Its example is "bicycle repair shops" returning different results in Paris and Hong Kong.
- The features on the results page also change with the query: local results for "bicycle repair shops", more likely images for "modern bicycle".
- If Search Console says a page is indexed but you do not see it in results, the page lists three possible reasons: the content is irrelevant to the queries, its quality is low, or robots meta rules prevent serving.
Omnistu advice (not from the source)
- When a new page is missing, work through the stages in order: can Google reach it (links, sitemap, robots.txt, login), was it indexed (canonical choice, noindex, content quality), and is it a good match for what people search?
- Make sure important pages are linked from pages that are already crawled, and listed in a sitemap; our sitemaps article covers this.
- If you publish near-identical pages, expect Google to choose one of them as canonical. Decide which one you want and say so; see our canonical, robots and noindex article.
- Do not pay anyone who promises faster crawling or guaranteed ranking, given Google's statement above.
- Check that important content is visible after rendering, not only in the raw HTML; see our JavaScript SEO article.
Limits
One Google page, read in full on 15 November 2026, dated 2025-12-18. We did not read the linked pages on Search Essentials, sitemaps, canonicalization or the Visual Element gallery, and we tested nothing. The page describes the stages in general terms and does not give timings or ranking weights; neither do we. Google only; other search engines work differently.
By Omnistu.