Robots.txt vs Noindex vs X-Robots-Tag

Robots.txt, a robots meta tag, and the X-Robots-Tag HTTP header can all influence search crawler behavior, but they solve different problems. Confusing them can produce the opposite of the intended result, especially when a site blocks a page from crawling and then expects a noindex directive on that page to be seen.

The simplest distinction is this: robots.txt manages crawling, while noindex manages indexing. X-Robots-Tag is an HTTP header that can deliver indexing directives, including for non HTML files.

What robots.txt controls

Robots.txt tells compliant crawlers which URL paths they may request. A rule such as:

User-agent: Googlebot
Disallow: /internal-search/

asks Googlebot not to crawl matching URLs.

This can help control unnecessary crawling, but it is not a reliable way to guarantee that a URL never appears in search. A blocked URL can still be discovered from links or other signals even when its content cannot be fetched.

Robots.txt is also not security. If a resource must be private, protect it with authentication or appropriate access controls.

What meta robots noindex controls

For an HTML page, you can place a robots meta tag such as:

<meta name="robots" content="noindex">

A crawler that supports the directive and is allowed to fetch the page can read it and exclude the page from its index.

Google states that when it crawls a page and detects a supported noindex, it drops that page from Google Search regardless of whether other sites link to it.

The important requirement is crawl access. If robots.txt prevents Googlebot from requesting the URL, Google cannot fetch the page to discover the meta robots directive.

What X-Robots-Tag controls

X-Robots-Tag carries robots directives in the HTTP response headers instead of the HTML.

Example:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

This is especially valuable for non HTML resources such as PDFs, images, or other files where adding an HTML meta tag is impossible.

For HTML pages, either a robots meta tag or X-Robots-Tag can be used for supported indexing directives. Choose the method that fits your architecture and can be maintained reliably.

The most common mistake: Disallow plus noindex

Consider a page that contains:

<meta name="robots" content="noindex">

but robots.txt also contains:

User-agent: Googlebot
Disallow: /page/

The intention may be "do not crawl and do not index." The problem is that the crawl block can prevent Googlebot from seeing the noindex directive. Google explicitly documents that a page needs to remain accessible to the crawler for noindex to be read.

If the page is public but should disappear from search results, allow crawling and provide noindex. If it must be private, require login or another real access restriction.

Which method should you use?

Use robots.txt when the goal is crawl management

Examples include reducing crawling of internal search results, temporary generated paths, or other areas where crawler requests provide little value.

Do not choose robots.txt simply because you want a public URL removed from an index.

Use a meta robots noindex tag for HTML pages

This is straightforward when you control page templates or CMS SEO settings. It is appropriate for public HTML pages that users may visit but that should not appear in supported search indexes.

Examples can include certain account related public pages, thin utility results, duplicate landing variants, or other pages where indexing is intentionally disabled.

Use X-Robots-Tag when headers are a better control point

Use the header for PDFs and other non HTML resources, or when server level rules are easier to manage than editing markup.

For example, a server could return X-Robots-Tag: noindex for a set of downloadable documents that should remain accessible but not appear in search results.

What about nofollow?

nofollow is separate from noindex. A noindex directive concerns whether the page itself should appear in search results. nofollow relates to following links on the page according to the crawler's supported interpretation.

Do not automatically pair noindex with nofollow. If crawlers can access a nonindexed page, allowing them to discover useful links from it may still be desirable.

What response code should a noindex page return?

A normal accessible HTML page carrying noindex generally needs a successful response so the crawler can retrieve and process the directive.

If the resource truly no longer exists, a proper 404 or 410 may be more semantically correct than serving a fake page with 200 and noindex. Status codes and indexing directives should describe the real state of the resource.

How X-Robots-Tag helps with PDFs

A PDF cannot contain an HTML robots meta tag in the same way a web page can. The HTTP response header solves that problem.

A response can look like:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

The document remains downloadable, while the header communicates the indexing instruction to crawlers that support it.

This is one of the strongest reasons to include header analysis in a technical SEO audit. Important directives can exist outside the visible HTML.

Troubleshooting a page that remains indexed

If you added noindex but the page still appears in search, check the following in order.

First, confirm the live response contains the directive. CMS caching, templates, CDN rules, or environment differences can make the production output different from what you edited.

Second, verify that robots.txt allows the crawler to fetch the URL. A blocked page can prevent the directive from being seen.

Third, check the status code. A redirect means the crawler may be processing the destination instead of the original page.

Fourth, remember that search engines need to recrawl the page before they can discover a newly added directive. The change is not necessarily reflected immediately.

Troubleshooting a page that should be indexed

If an important page is unexpectedly absent, inspect all three layers:

• robots.txt crawl permission
• meta robots tags in the HTML
• X-Robots-Tag response headers

A header level noindex can be easy to miss because it is invisible in the page source. Likewise, a CMS may inject a meta robots tag based on a global "discourage indexing" setting.

Then confirm the page returns the intended successful status and is not redirecting to another URL.

A safe decision framework

Ask what outcome you actually need.

If the question is "Should crawlers request this path?" evaluate robots.txt.

If the question is "Should this accessible resource appear in search results?" evaluate noindex through HTML or X-Robots-Tag.

If the question is "Should anyone without authorization be able to access this resource?" use security controls, not crawler directives.

Keeping these jobs separate prevents many indexing problems.