Free technical SEO tool
Robots.txt generator
Build a simple robots.txt file with crawler rules and a sitemap location, then copy the finished text. The check runs privately and starts instantly in your browser.
Using website origin: https://example.com
The sitemap line is omitted when this field is empty.
Generated robots.txt
Review the rules before publishing them.
User-agent: *
Allow: /
Disallow: /admin
Disallow: /private
Sitemap: https://example.com/sitemap.xmlUse robots.txt to guide compliant crawlers and authentication to protect sensitive pages.
How to use the result
Give crawlers clear access instructions
A robots.txt file tells compliant crawlers which areas of a site they may request. It belongs at the root of the website, such as example.com/robots.txt.
Use robots.txt to guide compliant crawlers around public content. Protect sensitive content with authentication, and review important rules with a developer before publishing them.
Frequently asked questions
Clear answers about this tool
Where should robots.txt be uploaded?
Publish it at the root of the hostname, for example https://www.example.com/robots.txt.
How should private pages be protected?
Protect sensitive pages with authentication. Robots.txt remains public and provides instructions to compliant crawlers.
Should every website include a sitemap line?
It is useful when an XML sitemap exists because it gives crawlers a direct path to the sitemap file.
More free SEO tools
Continue checking your website
Ready to grow your visibility on Google & ChatGPT?
Build one prioritized plan for stronger search rankings and AI visibility across every important page.