robots.txt Tester (RFC 9309)
Test whether URLs are allowed or blocked by a robots.txt for a crawler, using RFC 9309 rules: group choice, longest match, * and $ wildcards, with the deciding rule.
Matching follows RFC 9309: the most specific (longest) matching rule wins, Allow wins a tie, * matches any characters and $ anchors the end. robots.txt only asks crawlers not to fetch pages; it does not hide them from search results or protect them.
| URL | Result | Rule used |
|---|---|---|
| /admin/login | Blocked | line 2: Disallow: /admin/ |
| /search?q=shoes | Blocked | line 3: Disallow: /search |
| /search/about | Allowed | line 4: Allow: /search/about |
| /files/report.pdf | Blocked | line 5: Disallow: /*.pdf$ |
| /files/report.pdf?download=1 | Allowed | no rule matches |
| https://example.com/blog/hello-world | Allowed | no rule matches |
A single misplaced rule in robots.txt can hide a whole site from search engines, or leave an admin area crawlable. This tester applies a robots.txt to a list of URLs exactly as RFC 9309 and Google describe: it picks the group for the crawler you name (merging duplicate groups and falling back to *), finds the longest matching Allow or Disallow rule with * and $ wildcards, and reports Allowed or Blocked with the line that decided it. It also flags lines crawlers ignore.
How to use this tool
- Paste the robots.txt content.
- List the URLs or paths you want to check, one per line.
- Enter the crawler token, e.g. Googlebot, Bingbot or Applebot.
- Read each result and the rule behind it, then fix any warnings.
Formula
Group: the one whose user-agent equals the crawler token (case-insensitive), else *. Rule: among Allow and Disallow paths that match the start of the URL path, the longest wins; on a tie, Allow wins.
* matches any run of characters and a final $ means "end of URL". The path includes the query string. Characters outside ASCII are compared percent-encoded, and escapes of plain letters and digits are decoded first (RFC 9309 §2.2.2). /robots.txt is always allowed.
Example
With User-agent: *, Allow: /p and Disallow: /, the URL /page is Allowed: both rules match, but /p is longer than /. With Disallow: /*.php$, /filename.php is blocked while /filename.php?parameters is allowed, because $ requires the URL to end there.
Frequently asked questions
Does Disallow remove a page from Google?
No. It stops crawling, but a blocked URL can still be indexed from links, without its content. To keep a page out of search results, allow crawling and use a noindex robots meta tag or X-Robots-Tag header.
Why does Googlebot-Image follow the Googlebot group?
Google lists two user-agent tokens for some of its crawlers — Googlebot-Image, for example, obeys a googlebot-image group and otherwise the googlebot group. The tester applies that only to the Google crawlers documented that way; other names fall back to *. Untick the option to test strict RFC 9309 behaviour.
Are paths case-sensitive?
Yes. Disallow: /Folder does not block /folder. The user-agent name, however, is matched without regard to case.
Sources
Last updated:
Page address with these inputs: https://kakao.mobile/tools/robots-txt-tester/
Cite or embed this tool
kakaoCalc. (2026, November 8). robots.txt Tester (RFC 9309). https://kakao.mobile/tools/robots-txt-tester/
“robots.txt Tester (RFC 9309).” kakaoCalc, 8 Nov. 2026, kakao.mobile/tools/robots-txt-tester/.
Paste this code into your page. The widget runs in your visitors' browsers, shows no ads and keeps a link back to this page. Embedding terms