ffee
All tools
Robots.txt Generator
Build a valid robots.txt file with user-agent groups, allow and disallow rules and an optional sitemap line.
Build a robots.txt file
Rules added: 0
Current rules: none
How to use the Robots.txt Generator
- Choose the user-agent you want the rules to apply to.
- Add allow and disallow paths with the buttons, then set a sitemap address if you have one.
- Copy the finished robots.txt file to the root of your site.
Features
- User-agent groups with allow and disallow rules
- One tick blocks every crawler
- Optional sitemap line at the end of the file
- Copy-paste ready plain text output
Advertisement
Frequently Asked Questions
Common questions about the free online robots.txt generator.
It is a plain text file at the root of your site that tells search engine crawlers which
pages they may request and which they should skip.
Upload it to the top level of your domain so it is reachable at your domain followed by
/robots.txt, for example https://example.com/robots.txt.
Type the folder path such as /admin/ in the disallow box and click Add disallow. The
generator writes the matching rule line for you.
Allow rules mark paths crawlers may visit, while disallow rules mark paths they should skip.
When both match, the most specific rule wins.
Yes. The asterisk user-agent group applies to every crawler that does not have its own group,
which is what most sites want.
Tick the block all crawlers box. The generator then writes a single disallow line for the
root path, which asks crawlers to skip the whole site.
Paste the full sitemap address in the sitemap box. The generator adds a sitemap line at the
end of the finished file.
No. A robots.txt rule only asks crawlers not to request a page. Use a noindex meta tag or a
password when a page must stay out of results.
Last updated: October 10, 2026