Robots.txt Generator
Generate a basic robots.txt file with common block rules and sitemap output.
Why use this tool
01Assemble a starter robots.txt file for standard admin, search, and preview exclusions.
Use this when you need a clean robots.txt starting point instead of editing directives from scratch.
It keeps the common allow/disallow structure visible while you build.
How to use
02Quick checks before you copy
03Confirm the input is the format you intended.
Scan the result before using it in a document, URL, config, or message.
Copy only the output you need.
FAQ
04Is this enough for every site?
No. It is a starter file that should be reviewed against your routing and crawl policy.
Can I block specific bots by name?
Yes. Set the User-agent field to a specific bot name, or use * to apply rules to all crawlers.
Can I add a sitemap URL to the output?
Yes. The sitemap field adds a Sitemap: directive at the end of the generated file.
Related tools
03Sitemap URL List Generator→
Clean, trim, and deduplicate page URLs before building a sitemap file.
Crawl & Sitemap Manager→
Generate robots.txt, sitemap inventory, canonical host rules, and hreflang output in one workspace.
Canonical Tag Checker→
Compare a page URL and canonical URL to spot self, cross-domain, and mismatched canonicals.