Robots.txt Generator
Generate a basic robots.txt file with common block rules and sitemap output.
工具代码会在浏览器中处理所选文件和输入内容，不会将其提交给 TOOLGRID 的处理接口。 TOOLGRID 只衡量工具使用情况，不记录你输入的内容。
- 不上传工具输入至 TOOLGRID
- 不登录
- 复制前先检查
Loading tool…
这个工具能做什么
Assemble a starter robots.txt file for standard admin, search, and preview exclusions.
Use this when you need a clean robots.txt starting point instead of editing directives from scratch.
It keeps the common allow/disallow structure visible while you build.
这个工具适合解决什么问题
Draft conservative disallow rules for staging areas, dashboards, previews, and internal search pages before deployment.
Add the canonical sitemap URL to the generated file so compliant crawlers can discover the preferred URL inventory.
Create a readable robots.txt draft that SEO, engineering, and platform teams can inspect before it ships.
依赖结果前需要检查什么
- Performance and maximum practical input size depend on browser memory, device speed, and the structure of the input.
- Review the generated result before replacing or publishing an original file.
如果还需要校验、转换或复用结果，可以继续打开相近的浏览器本地工具。
如何使用
01使用场景
Draft conservative disallow rules for staging areas, dashboards, previews, and internal search pages before deployment.
Add the canonical sitemap URL to the generated file so compliant crawlers can discover the preferred URL inventory.
Create a readable robots.txt draft that SEO, engineering, and platform teams can inspect before it ships.
使用技巧
- 01Robots.txt is not a deindexing tool
Disallow can stop crawling, but already indexed pages may remain visible. Use noindex or removal workflows for deindexing decisions.
- 02Test broad disallow patterns carefully
A short path rule can block more URLs than intended. Review every generated directive against real site paths before publishing.
常见问题
02Is this enough for every site?
No. It is a starter file that should be reviewed against your routing and crawl policy.
Can I block specific bots by name?
Yes. Set the User-agent field to a specific bot name, or use * to apply rules to all crawlers.
Can I add a sitemap URL to the output?
Yes. The sitemap field adds a Sitemap: directive at the end of the generated file.