Robots.txt checker

How to check your robots.txt file

A robots.txt file tells crawlers which URLs they may fetch. It sits at the root of a host, for example https://example.com/robots.txt, and applies only to that host, protocol and port. Checking it takes four steps:

1. Open the file. Type your address followed by /robots.txt into a browser. A page of plain text is the file. A 404 means there isn't one, and crawlers treat that as no restrictions.
2. Paste it into the box above. Copy all of it, including the Sitemap: lines.
3. Enter the page you want to test and a crawler. Use the full URL, or just the path. Leave the user-agent as Googlebot for Google, or type another name.
4. Read the verdict. The result says allowed or blocked, the group of rules it used and the exact line that decided. Lines the checker flags are ones Google ignores.

Because the checker works on text you paste, you can try a change before you upload it. That also means it tests the copy you pasted, not what your server serves right now, so paste a fresh copy if the file has changed.

How a robots.txt tester decides

This checker follows the rules Google documents for how it reads robots.txt. In order:

1. Pick the group. Rules are written in groups, each starting with one or more User-agent: lines. The crawler uses the group with the most specific matching name, and falls back to the * group if none matches. Names are not case-sensitive. If several groups name the same crawler, their rules are merged.
2. Find every matching rule. An Allow or Disallow rule matches when the URL's path (with its query string) starts with the rule's path. * stands for any run of characters and $ at the end means the URL must end there. Paths are case-sensitive.
3. The longest rule wins. The most specific rule, meaning the one with the longest path, decides. If an Allow and a Disallow tie on length, Allow wins. If nothing matches, the URL is allowed. An empty Disallow: blocks nothing. Google also reads Allow: /folder/index.html as allowing /folder/ itself.

A worked example, using the file the example button fills in:

User-agent: * Disallow: /private/ Allow: /private/press/

Rules are from Google's documentation of how it interprets robots.txt and its open-source robots.txt parser, checked 23 October 2026. Other crawlers can read the same file a little differently.

Common robots.txt mistakes the checker catches

The checker reads the whole file as it tests, and lists lines that will not do what they look like they do:

Blocking crawling is not the same as blocking indexing. Google says robots.txt is not a mechanism for keeping a page out of Google: a blocked URL can still show up in results if other sites link to it, without a description. To keep a page out of search, use a noindex tag or password protection, and leave the page crawlable so the tag can be read.

Check AI crawlers, Googlebot and Bingbot

Every crawler reads the same file, so you test a different one by changing the user-agent box. Type GPTBot, ClaudeBot, PerplexityBot or Bingbot and the checker applies the group written for that name, or the * group if there isn't one. The checker matches the name you type exactly. It doesn't know which crawlers a vendor lets fall back to another one's group, so test the name the vendor documents.

A robots.txt file states what you want, and well-behaved crawlers follow it. It doesn't prove a crawler did. Our guide to AI crawlers lists each bot's name, who runs it and what it is for, and says which ones fetch pages on a person's request and may not apply robots.txt. To see who actually visited, read your server logs or a bot report: the bot traffic guide covers how to tell bots from people.

After you've checked robots.txt

This tool tells you what your file says. For what Google has actually seen, Google Search Console has a robots.txt report, which shows the files Google found for your site, when it last crawled them and any warnings or errors, and a URL inspection tool that tests a single URL.

Blocking crawling can affect search traffic because Google cannot fetch the page's content, but the URL can still appear in results. When traffic falls after a robots.txt change, why website traffic drops walks through what to look at next. To see how many visits each page and channel brings in once crawling is fixed, that is what Stickybeak is for: live analytics for every site you own, in early access.