The four ways a page gets excluded, and why they are not interchangeable
Every indexability bug comes down to one of four mechanisms, and most of the damage comes from treating them as the same thing. They are not.
| Mechanism | Crawled? | Indexed? | Use it when |
|---|---|---|---|
| meta robots noindex | Yes | No | You want the page gone from results but links on it still followed |
| X-Robots-Tag: noindex | Yes | No | The file has no head to put a tag in — PDFs, images, feeds |
| robots.txt Disallow | No | Maybe — URL only | You want to save crawl budget on worthless paths, not remove pages |
| rel=canonical elsewhere | Yes | Usually not, but it is a hint | Duplicate URLs that should consolidate into one |
The row that causes the most damage is the third one. A disallowed URL is not removed — Google simply stops looking at it. If other sites link to it, the URL can sit in the results as a naked link with no title and no snippet, sometimes for years, because the one instruction that would remove it is inside a page Googlebot has been told not to open.
The conflict that keeps dead pages alive
Someone wants a page gone. They add noindex to the template, and to be thorough they also add a Disallow line to robots.txt. The two changes ship together, and the page never leaves the index.
Robots.txt is evaluated before the fetch. Googlebot reads the disallow, skips the request, and never sees the noindex tag it was supposed to obey. The correct sequence is the boring one:
- Allow crawling of the path in robots.txt.
- Serve noindex via meta tag or X-Robots-Tag header.
- Wait for recrawl and confirm the URL drops out of Search Console's coverage report.
- Only then add the disallow, if you still want to save the crawl budget.
This tool flags that exact pairing on any URL where it finds both, because it is invisible in a normal audit: each file looks correct on its own.
Where stray noindex tags actually come from
In practice it is rarely a deliberate decision. The recurring sources, in rough order of how often they show up in a real audit:
- A staging flag that shipped. The single most expensive one. Staging environments serve a site-wide noindex, the environment variable does not get flipped at launch, and organic traffic falls off a cliff two weeks later.
- CMS defaults. WordPress' "Discourage search engines" checkbox, Shopify's automatic noindex on filtered collections and internal search, Webflow's per-page toggle.
- SEO plugin rules. Yoast and RankMath noindex entire post types, tag archives, and paginated series by default — settings that are usually right, but worth confirming rather than assuming.
- Server and CDN rules. An X-Robots-Tag added at the nginx or Cloudflare layer for one directory that later grew to cover real pages. Nothing in the HTML hints at it.
- Paginated and faceted URLs. Page two onwards noindexed by a rule that also catches category pages you want ranking.
None of these announce themselves. They are found by checking a list of URLs and reading the column.
What to do with the output
Filter to Noindex first and decide, URL by URL, whether the exclusion was intended. Thin tag archives and internal search results usually should be noindexed; money pages, blog posts, and category pages usually should not. Then check Blocked — a robots.txt disallow on a path you are actively trying to rank is a quiet traffic cap. Finally scan Canonicalised for pages pointing at a URL that no longer exists or redirects, which strands the consolidation.
Once a URL is confirmed indexable and still is not ranking, the problem has moved off this page and onto the authority side. That is where inbound links and crawl frequency do the work — a page nobody links to gets crawled rarely, no matter how clean its directives are. The robots.txt generator is the companion tool for fixing the crawl layer once you know which paths are wrongly blocked, and the llms.txt generator handles the AI-crawler side of the same question. The full set lives at /tools.