robots.txt & Sitemap Maker

Write a robots.txt without guessing at the syntax, then check any page against your own rules to see whether it would really be blocked. Plus a sitemap builder from a plain list of URLs.

Your site address

robots.txt

Overall approach

Block these paths, one per line

Always allow these, one per line

Block particular crawlers

Crawl delay in seconds (0 for none) Google ignores this. Bing and others honour it.

Test a page against these rules

sitemap.xml

Your pages, one per line Paths or full addresses both work. Duplicates are removed.

How often it changes

Priority

robots.txt & Sitemap Maker

robots.txt

sitemap.xml

More developer tools