robots.txt Tester

How it works

The tester parses your robots.txt into user-agent groups, picks the group matching your test user-agent exactly or falls back to the wildcard "*" group, then converts every Allow/Disallow pattern in that group into a regular expression — "*" becomes "match anything," a trailing "$" anchors the end of the string, and everything else matches as a prefix of the path. It then finds the single longest matching pattern and applies Google's own tie-breaking rule: if an Allow and a Disallow pattern are the same length, Allow wins.

FAQ

Why does a shorter Disallow rule sometimes lose to a longer Allow rule?

Because Google's algorithm picks the single longest matching pattern, not the first one written in the file — order doesn't matter, only pattern length; a longer, more specific rule always overrides a shorter, broader one covering the same path.

What does the trailing $ do in a pattern like /*.pdf$?

It anchors the match to the end of the URL path, so /*.pdf$ only matches paths that literally end in .pdf — without the $, it would also match something like /report.pdf.backup, since these patterns match as a prefix by default.