R2Robots line
robots.txt tester
Paste a robots.txt, pick a crawler and list URLs. Each one is marked allowed or blocked the way RFC 9309 describes, with the line that decided it, and the file itself is linted for mistakes.
robots.txt
—Results update as you type. Nothing leaves your browser.
Test
Results
—| URL | Result | Decided by |
|---|---|---|
| Add a robots.txt and some URLs to see results. | ||
Lint
—Nothing to lint yet.Typos, rules outside a group, unsupported lines, full URLs used as paths and the 500 KiB limit are flagged here.
How a crawler reads the file
- Pick the group. The crawler looks for groups whose
User-agentmatches its product token, ignoring case. Several matching groups are merged; with none, the*group applies; with neither, everything is allowed. - Match the path. Each rule is compared with the URL's path and query from the first character.
*matches any characters, a final$means the URL must end there. - Longest rule wins. The matching rule with the most characters decides; if an Allow and a Disallow are equally long, Allow wins.
/robots.txtitself is always allowed.
Before comparing, encoding is evened out: non-ASCII becomes UTF-8 escapes, escaped letters and digits are decoded, so %62 equals b, while %2F stays distinct from /.
FAQ
Is matching case-sensitive?
Paths are: /Admin and /admin are different. User-agent names are not.
How large can robots.txt be?
RFC 9309 asks crawlers to read at least 500 KiB, and Google ignores anything past that point. The lint warns when a file is larger.
Why does Googlebot-Image follow my Googlebot group?
Some crawlers fall back to a more general token when no group names them. Test both tokens to see each case.