Inspect a website robots.txt file
Check a public robots.txt file, parse user-agent, allow, disallow and sitemap directives, and flag obvious crawl blocking issues.
Reads the public /robots.txt file and highlights basic directives. It does not emulate every crawler-specific rule.
How to use the Robots.txt Tester
Check a public robots.txt file, parse user-agent, allow, disallow and sitemap directives, and flag obvious crawl blocking issues.
Formula and method
The tester downloads /robots.txt from the public site and groups recognized crawl directives without simulating every search engine rule.
Example
Enter https://example.com and review User-agent, Disallow, Allow and Sitemap directives returned by the live robots.txt file.
Planning examples
These examples are mathematical scenarios designed to help with planning. They are not claims about typical market performance.
| Use | Recommendation |
|---|---|
| Planning | Use your own recent data whenever possible. |
| Comparisons | Keep the same definition and time period across comparisons. |
| Benchmarks | Treat public benchmarks as context rather than guarantees. |
How to interpret the result
robots.txt controls crawling, not guaranteed indexing removal. Use meta robots or HTTP directives when a page must be noindexed.
Frequently asked questions
Does this tool use an external API?
No. The calculation runs from the values you enter.
Is the result a guarantee?
No. It is a mathematical planning result based on your inputs.
Can I use my own real data?
Yes. Your own recent account or business data is usually the best input.
Can I compare different platforms?
Use caution because metric definitions and monetization models can differ by platform.