Web & SEO
Robots.txt Tester
Test a crawler and URL path against pasted Allow and Disallow rules.
Free foreverRuns in your browserNo sign-up
Result
Blocked
Supports straightforward User-agent, Allow and Disallow prefix rules; production crawlers may apply additional matching details.
About the Robots.txt Tester
Robots.txt Tester evaluates straightforward User-agent, Allow and Disallow prefix rules for a chosen crawler and path.
It is a fast local check, not a complete emulation of every crawler's wildcard, encoding and precedence behavior.
Key features
- Crawler-specific user agent
- Path allow/block result
- Longest matching prefix handling
- Local robots.txt input
How to use
- 1Paste the relevant robots.txt.
- 2Enter a crawler name and URL path.
- 3Confirm critical resources and indexable pages are allowed.
Examples
Use the Robots.txt Tester
Input:
Disallow /admin/ and test /admin/pageOutput:
BlockedAllow public assets required to render indexable pages.
Frequently asked questions
- Does blocking prevent indexing?
- Robots rules control crawling, not guaranteed indexing. Use noindex where a crawler can access it.
- Does it support every wildcard rule?
- No. Use the target search engine's official tester for edge cases.
- Is my data uploaded?
- No. Analysis and generation happen entirely in your browser.