Crawling

Crawling is how search engines read your website: a program visits each page, follows the links, and records what it finds. If a page cannot be crawled, it cannot rank. Digger crawls your site the same way, which is how problems are found before they cost you visitors.

What stops a crawler

  • No link to the page. If nothing links to it and it is not in your sitemap, a crawler has no way to know it exists.
  • A rule that blocks it. A robots.txt line or a noindex tag, often left behind after a site launch.
  • A server that is too slow or erroring. Crawlers give up.
  • Content that only appears after JavaScript runs. Sometimes fine, sometimes not, and never as reliable as content in the HTML.
  • A redirect chain or loop. Each hop costs, and loops end the visit.

Crawling is not ranking

Being crawled means the page was read. Being indexed means it was kept. Ranking means it was chosen for a query. They happen in that order, and problems at the first step make the other two impossible, which is why an audit starts here.

Related terms

Indexing

Indexing is a search engine deciding to keep a page it has crawled, and storing it so it can be returned for a search. A page can be crawled and still not indexed, usually because the engine judged it thin, duplicated, or not worth storing.

Core Web Vitals

Core Web Vitals are three measurements Google uses to describe how a page feels to use: how fast the main content appears, how quickly the page responds to input, and how much the layout moves while loading. They are a ranking signal, and a small one.