Robots.txt Generator: create a robots.txt file for free

A Robots.txt Generator creates a robots.txt file from your settings, so you don't have to write the syntax yourself. This one has two modes: “Quick Create” for individual Allow/Disallow rules by path and bot, and “Per Bot” for a default rule plus a setting for each crawler. The modes aren't combined; each generates its own robots.txt file. The tool is free, requires no sign-up, and generates the file in your browser.

At a minimum, you need a User-agent line for the bot, followed by Allow or Disallow rules: Disallow blocks a path, while Allow explicitly permits one—for example, to block /wp-admin/ but allow /wp-admin/admin-ajax.php. You can optionally add a Sitemap line with the full URL, including https://. If you don't add any rules in “Quick Create,” the generator outputs User-agent: * and Allow: /.

Upload it to your domain's root directory, not a subfolder, for example via FTP/SFTP or your hosting file manager. You can then access it at your-domain.com/robots.txt. The file downloaded from the generator is already named robots.txt.

In “Per Bot” mode, set GPTBot, Google-Extended, CCBot, ClaudeBot, or PerplexityBot to “Block.” The generator writes a separate User-agent block with Disallow: / for each of these crawlers. You can leave search engines like Google and Bing on Default. Note that robots.txt is a request followed by reputable crawlers, not a technical barrier.

In “Per Bot,” set the default for all bots to “Not allowed.” This produces User-agent: * with Disallow: /. Search engines that follow robots.txt will then stop crawling your entire website. Only use this intentionally, for example for a test environment.

No. Google ignores Crawl-delay, and so has Yandex since 2018. Only some other crawlers, such as Bing, take it into account. In “Per Bot” mode, you can choose 1, 5, 10, or 20 seconds. The value appears in the block for all bots and in each bot block you've individually set to Allow or Block.

Not reliably. A robots.txt file only controls crawling. A blocked URL can still be indexed if other pages link to it. To keep a page out of search results, use a noindex meta tag or password protection. This generator doesn't create noindex rules, and robots.txt isn't suitable for protecting private content.

No. The Robots.txt Generator only creates new files; it doesn't check a robots.txt file that's already online. To view a live file, open deine-domain.de/robots.txt in your browser. The robots.txt report in Google Search Console shows whether Google can retrieve it.