Understanding Robots.txt Checker
Learn how the checker works, how to interpret its output, and how to use the results in development, troubleshooting, and technical SEO workflows.
What a robots.txt checker does
A robots.txt file gives automated crawlers instructions about which URL paths they may request. It is a plain-text file located at the root of a host, such as https://example.com/robots.txt. This checker finds that location, reports the HTTP status, displays the original text, and organizes User-agent groups, Allow rules, Disallow rules, Crawl-delay values, sitemap declarations, and other recognizable directives into a readable structure.
The optional path test compares a page URL with the rules that best match Googlebot, Bingbot, or a generic crawler. It identifies the longest matching path pattern and gives Allow precedence when rules of equal length conflict. The result is a practical interpretation rather than a promise about a search engine's final behavior, because crawler implementations, invalid syntax, percent encoding, and vendor-specific directives can introduce ambiguity.
How robots.txt groups and rules work
A group begins with one or more User-agent lines and continues with directives for those crawlers. An asterisk represents a generic group. Specific product tokens such as Googlebot can override the generic group when they match the crawler. Disallow identifies paths the selected crawler should not fetch; Allow creates an exception, often inside a broader blocked directory. An empty Disallow value blocks nothing, which is different from Disallow: /, which targets the entire host.
Rules normally match from the beginning of the URL path. Many major crawlers support an asterisk wildcard and a dollar sign that anchors the end of a pattern. When multiple rules match, the most specific—typically the one with the longest path—is selected, with Allow winning an equal-length tie. Robots policies are host and protocol specific: instructions on www.example.com do not automatically govern shop.example.com, and HTTP and HTTPS are separate origins.
Using the page URL test
Enter the website whose root robots file should be fetched. To test a particular path, provide a complete URL on that same origin and choose a crawler. The checker parses the relevant groups and reports the matched rule. Test the actual canonical URL, including path and query string when these affect rules. A result of appears allowed means no selected rule prevents crawling; it does not mean the page is reachable, indexable, canonical, or likely to rank.
If the syntax is malformed or several unusual directives conflict, verify the result with the search engine's own diagnostic tools. Robots matching operates on URL paths, not the visible page title or filesystem permissions. A blocked directory can still have URLs discovered through external links, and a permitted path can still return 404, redirect elsewhere, require authentication, or include a noindex directive after a crawler retrieves it.
Robots.txt controls crawling, not indexing
This distinction is essential in technical SEO. Disallow asks compliant crawlers not to request a URL; it is not a reliable removal mechanism. A search engine may know the URL from links or a sitemap and retain a minimal indexed reference without fetching the blocked content. Because the crawler cannot retrieve the page, it may not see a meta robots noindex tag. To remove an accessible page from search, allow crawling and use an appropriate noindex response or removal workflow.
Robots.txt is also not access control. The file is public, the blocked paths are visible to everyone, and malicious clients can ignore it. Sensitive content requires authentication and authorization at the application or server layer. Avoid listing secret admin paths merely to hide them. Standard CMS admin rules can reduce unwanted crawler load, but confidential reports, private APIs, staging environments, and customer records must be protected independently.
Sitemap declarations and crawl guidance
Sitemap lines can contain absolute XML sitemap URLs and may appear outside User-agent groups. Multiple declarations are allowed. A sitemap helps crawlers discover canonical, indexable URLs and their update signals, while robots rules limit requests. The two mechanisms should agree: routinely submitting blocked, redirected, noncanonical, or error URLs creates noise and makes diagnostics harder. Validate each sitemap separately and keep only URLs you want search engines to process.
Crawl-delay is recognized by some crawlers but not universally, and search engines may provide their own crawl controls. Nonstandard directives such as Host or Clean-param are vendor specific. This checker preserves unknown directives in parsed data instead of pretending they have a universal meaning. When managing a large site, rely on official documentation for the target crawler and monitor server logs to understand actual request behavior.
Common robots.txt mistakes
Frequent errors include uploading the file below a subdirectory, returning an HTML error page with status 200, using smart quotes, omitting colons, blocking CSS or JavaScript needed for rendering, and accidentally deploying Disallow: / from staging. Rules are case-sensitive for path matching on case-sensitive URLs. A UTF-8 text response is safest. Redirects can work, but serving the file directly at the root reduces uncertainty and avoids a failure in the redirect chain.
Another mistake is trying to target individual query parameters with syntax the crawler does not support. Broad wildcard patterns can match far more pages than expected. Before releasing a rule, test representative allowed and disallowed URLs, review the raw file after CDN delivery, confirm status and Content-Type, and compare with search-console tools. Keep comments helpful, remove obsolete sections, and treat production robots changes with the same review discipline as application code.
HTTP status behavior and limitations
A 200 response supplies rules. A genuine 404 generally means no robots file exists and crawlers can proceed without file-based restrictions. Temporary 5xx failures can cause crawlers to pause because they cannot know the intended policy, while a persistent error may eventually be treated differently by each engine. A 401 or 403 makes the instructions inaccessible. Always interpret the status alongside the raw content; a branded not-found page returned as 200 is not a valid robots file.
This checker requests public text without executing JavaScript, enforces a response-size limit, and blocks private network targets. Its path test covers the widely used longest-match model and common wildcard behavior, but it cannot predict every crawler's extensions or error recovery. For a critical launch, combine this result with the crawler vendor's official tester, live URL inspection, server logs, sitemap validation, and the Meta Tag Checker for page-level robots directives.
Related website tools
Continue the investigation with these closely related checks.
Redirect & HTTP Status Checker
Trace redirect chains, status codes, final destinations, and response timing.
Open toolHTTP Headers Checker
Review response, caching, security, server, and access-control headers.
Open toolOpen Graph & Meta Tag Checker
Audit SEO metadata, canonical tags, Open Graph fields, and Twitter cards.
Open toolYou can also return to the Website CMS Detector to identify a site's content management system, framework, WordPress theme, and visible plugins.
Frequently asked questions
No. Disallow primarily controls crawling. A URL can still be discovered or referenced in search. Use an accessible noindex directive or an appropriate removal process when removal is required.