Robots.txt Explained: Rules, Syntax, and Common Mistakes

A robots.txt file tells compliant web crawlers which parts of a site they may or may not request. It is a crawl management file, not an access control system and not a guaranteed method for removing URLs from search results.

That distinction matters. A single incorrect rule can prevent crawlers from fetching important pages, while an overly permissive file can allow unnecessary crawling. The safest way to work with robots.txt is to understand exactly what each rule does and test the final file against real URLs.

Where robots.txt must be located

The file belongs at the root of the host it controls. For a site at https://example.com, the normal location is:

https://example.com/robots.txt

A file placed at /folder/robots.txt does not control the entire host. Protocols, hosts, and subdomains also matter. Rules served from www.example.com do not automatically become the robots file for a different subdomain.

Before analyzing any directive, confirm that the expected robots.txt URL loads successfully and that you are checking the same host used by the pages in question.

The four main fields you need to understand

Google supports user-agent, allow, disallow, and sitemap in robots.txt.

User-agent

User-agent identifies the crawler or crawler group that the following rules apply to.

User-agent: *

The asterisk is commonly used as a general group for crawlers that do not have a more specific matching group.

You can also target a crawler explicitly:

User-agent: Googlebot
Disallow: /private-section/

Crawler matching rules can differ by search engine, so do not assume every bot supports every extension in exactly the same way.

Disallow

Disallow specifies a path that the selected crawler should not request.

User-agent: *
Disallow: /admin-temp/

This blocks crawling of matching paths for compliant bots. It does not make the directory private. Anyone can read a public robots.txt file, so confidential areas require authentication or other real access controls.

An empty Disallow value means there is no restriction from that rule.

Allow

Allow can create an exception inside a broader blocked path.

User-agent: Googlebot
Disallow: /documents/
Allow: /documents/public/

This pattern says that the broader directory is disallowed while the more specific public path may be crawled.

Google selects the most specific matching path. If equally specific rules conflict, Google uses the less restrictive rule.

Sitemap

A robots.txt file can advertise one or more XML sitemaps using a fully qualified URL:

Sitemap: https://example.com/sitemap.xml

The sitemap line is not a crawl permission rule. It helps crawlers discover the sitemap location.

How path matching works

Robots rules apply to URL paths, and matching can be more precise than many site owners expect.

For Google, path values are case sensitive. A rule for /Folder/ is not automatically the same as /folder/.

Google also supports * as a wildcard and $ to represent the end of a URL pattern. For example:

User-agent: Googlebot
Disallow: /*.pdf$

This pattern blocks matching PDF URLs that end at .pdf. A query string after .pdf can affect whether an end anchored pattern matches, so test representative real URLs instead of relying on intuition.

Avoid clever patterns when a simpler directory rule will do. Complexity increases the chance of accidentally blocking valuable content.

A practical robots.txt example

A small public site might use:

User-agent: *
Disallow: /internal-search/
Disallow: /temporary-preview/

Sitemap: https://example.com/sitemap.xml

The file leaves the rest of the site crawlable while discouraging crawling of two specific areas. Whether those areas should also be excluded from indexing is a separate decision.

Robots.txt does not reliably prevent indexing

One of the most important rules to remember is that crawl blocking and indexing control are different jobs.

If a URL is blocked by robots.txt, Google may be unable to fetch its page content. However, the URL can still be discovered through links and may in some situations appear in search results without content being crawled.

If your goal is to prevent an accessible page from being indexed, use a supported noindex mechanism and allow the crawler to fetch it so the directive can be seen. For private content, require authentication rather than relying on robots.txt.

This is also why putting noindex inside robots.txt is not the correct solution for Google. Google documents that noindex in robots.txt is not supported.

What happens when robots.txt returns an error

The HTTP status of the robots file changes crawler behavior.

Google processes a successful 2xx response as the robots file. For most 4xx responses other than 429, Google behaves as if no valid robots.txt file exists, which means it assumes no crawl restrictions from that file.

Server failures are treated more cautiously. Google documents temporary crawling pauses and reuse of a last known good robots file when appropriate during 5xx failures. That makes accidental server errors on robots.txt more serious than simply having no file.

Common robots.txt mistakes

Blocking the entire site accidentally

This rule blocks all matching crawling:

User-agent: *
Disallow: /

It may be intentional on a staging environment but disastrous if carried into production.

Blocking pages that contain noindex

If a crawler cannot fetch a page because robots.txt blocks it, it may not see the page's noindex meta tag or X-Robots-Tag. If deindexing is the goal, crawl access is usually required for the directive to be processed.

Treating robots.txt as security

Listing private looking paths in robots.txt does not protect them. The file is publicly visible and can even reveal directory names. Use authentication and authorization for sensitive resources.

Copying another site's file

Robots rules depend on URL structure and business requirements. A rule that is safe for another domain may block essential pages on yours.

Forgetting subdomains

A robots file for the main host does not automatically govern every subdomain. Check each host that search engines can crawl.

How to test a robots.txt file properly

First, fetch the live file and confirm its HTTP response. Then examine which user-agent group applies to the crawler you care about.

Next, test representative URLs from important page types, blocked directories, assets, search pages, parameterized URLs, and any exceptions created with Allow.

After deployment, monitor crawling and indexing data for unexpected changes. If important pages suddenly become inaccessible to crawlers, compare the current robots file with the previous version and check whether a deployment or CMS setting regenerated it.

Keep the file as simple as your crawl requirements allow. A short, understandable file is easier to audit than dozens of overlapping wildcard rules.