noindex vs. robots.txt: Which One Actually Stops Indexing?

These two get treated as interchangeable "keep this out of Google" tools, and that mix-up causes more accidental deindexing (and accidental non-deindexing) than almost anything else in technical SEO. They control different stages of the pipeline, and using the wrong one for the job — or stacking both without understanding the order — produces exactly the opposite of what people intend.

Here's the actual mechanism for each, and the rule that tells you which one you need.

The pipeline they each control

Google's process for a URL runs in stages: discover it, crawl it, then decide whether to index it.

• robots.txt acts at the crawl stage. A Disallow rule tells Googlebot: don't fetch this URL's content. Google can still know the URL exists (from links, sitemaps, or history) — it just won't read what's on the page.
• noindex acts at the indexing stage, but only if Google can reach it. It's an instruction inside the page (a meta tag or HTTP header) that says: you can read this, but don't put it in search results. Google has to crawl the page to see this instruction at all.

That dependency is the whole trap: noindex only works if the page is crawlable. Block it in robots.txt first, and the noindex tag becomes invisible — Google can't fetch the page to read it.

The mistake in practice

A page gets both a robots.txt disallow and a noindex tag, usually because someone wanted to be extra thorough about removing it. What actually happens:

1. If the page was never indexed, it likely stays out — robots.txt alone is often enough to prevent new indexing.
2. If the page was already indexed (Google indexed it earlier, or from external links), it can stay indexed indefinitely. Google can't re-crawl to see the noindex, so it has no updated instruction to act on — it just keeps the old index entry, sometimes with a blank or stale snippet.

This exact scenario has its own status in Search Console: Indexed, though blocked by robots.txt. If you're seeing that status, this is almost always the cause.

Which one to use, by goal

1. Goal: remove a page from search results entirely

   Use noindex alone. Leave the page crawlable — no robots.txt block. Google needs to fetch the page repeatedly to keep confirming the noindex instruction is still there. Only after the page is confirmed removed from the index should you consider a robots.txt block, and only if you also want to save crawl budget on it going forward — it's not required.

2. Goal: save crawl budget on low-value URLs that were never indexed

   Use robots.txt alone. Good candidates: faceted/filtered URL combinations, internal search result pages, infinite calendar or pagination parameters — high-volume, low-value URLs you don't want Googlebot spending time on. Just be aware this doesn't guarantee the URL can never appear in search; if something links to it externally, Google can still index it from that signal alone, snippet-less.

3. Goal: keep crawlable but stop specific pages from ranking, permanently and reliably

   noindex, and leave it that way. Don't add a robots.txt block later "for cleanliness" — it breaks Google's ability to reconfirm the noindex on future crawls if the tag is ever accidentally removed, and it adds no real benefit once the page is already out of the index.

Checking which one is actually in effect

1. Check robots.txt directly

   Open https://yourdomain.com/robots.txt and look for a Disallow rule matching the page's path. This is a static file — read it directly rather than guessing from behavior.

2. Check for noindex in the raw response

   curl -I https://yourdomain.com/page for an X-Robots-Tag: noindex header, and view-source:https://yourdomain.com/page (or curl -s URL | grep -i robots) for a <meta name="robots" content="noindex"> tag. Use the raw response, not DevTools' Elements panel — a JS-injected tag can appear there without being in what Google actually fetches.

3. Confirm with URL Inspection

   Search Console's URL Inspection tool shows both mechanisms plainly: whether the page is blocked by robots.txt, and separately, whether indexing is allowed. If both say blocked/no, you have the stacking problem above.

The fix if you're already stuck

If a page is indexed, robots.txt-blocked, and noindexed, and you want it gone:

1. Remove the robots.txt block first. This is the step people skip — they think adding noindex is enough and never realize the block is preventing Google from ever seeing it.
2. Leave the noindex tag in place and the page crawlable.
3. Request indexing in URL Inspection to prompt a recrawl.
4. Once URL Inspection confirms the page is out of the index, you can reapply a robots.txt block if you still want it for crawl-budget reasons — it's optional at that point, not required.

When robots.txt is genuinely the better tool

Not every "don't index this" case needs noindex. If a URL was never indexed and never will be linked to from anywhere Google trusts, robots.txt alone is simpler and saves crawl budget without the complexity of a meta tag on every generated variant — this is common for parameter-heavy e-commerce filters. The risk calculus only changes once external links make accidental indexing plausible.