RSRobince Studio

Robots.txt Generator

Build a robots.txt file without remembering the syntax. Choose whether crawlers are allowed by default, add rules for specific bots, list your sitemaps and see the file update live. Copy it or download it, then upload it to the root of your website.

Runs in your browser. Your files are never uploaded.

Loading Robots.txt Generator…

How to use Robots.txt Generator

  1. Choose whether all robots are allowed or blocked by default.
  2. Add a rule for each bot you want to treat differently, and list the Allow and Disallow paths.
  3. Add your sitemap addresses and an optional crawl-delay.
  4. Check the live preview, then copy it or download robots.txt and upload it to your site root.

How robots.txt works

A robots.txt file sits at the root of your site and tells crawlers which paths they may visit. Each block starts with User-agent, which names a bot, followed by Allow and Disallow lines. A star means every bot.

An empty Disallow line means nothing is blocked. A single slash blocks the whole site.

Important warnings

  • robots.txt is public. Never rely on it to hide private pages.
  • It does not remove pages that are already indexed. Use noindex or the search engine removal tool for that.
  • A blocked page cannot be crawled, so a noindex tag on it will not be seen.
  • Bots can ignore the file. It only works for crawlers that choose to follow it.

Writing paths

Paths start with a slash and match from the beginning of the address. Disallow: /admin/ blocks everything under that folder. Use Allow to make an exception inside a blocked folder. The wildcard star and the dollar sign are understood by major search engines.

Limits

This tool writes a plain text file in your browser and uploads nothing. It does not test the file against your live site, and it does not check that paths exist. Always test the finished file in the search engine tools after you publish it.

Why use this robots.txt generator tool

  • Live preview. The file is rebuilt as you type, so you always see exactly what will be saved.
  • Preset bots. Pick common crawlers such as Googlebot, Bingbot, GPTBot and CCBot from a list, or type any user-agent name.
  • Several rules. Add as many user-agent blocks as you need, each with its own Allow and Disallow paths.
  • Sitemaps and crawl-delay. Add one or more sitemap addresses and an optional crawl-delay line in one step.

Frequently asked questions

How do I create a robots.txt file for free?

Choose a default rule, add any bot-specific rules and your sitemap address, then press Download robots.txt. Upload the file to the root folder of your website so it opens at yourdomain.com/robots.txt. It is free and needs no account.

How do I block GPTBot or other AI crawlers?

Add a rule, pick GPTBot as the user-agent and put a single slash in the Disallow box. Repeat for other bots such as CCBot. Well-behaved crawlers follow robots.txt, but it is a request, not a lock, so some bots may ignore it.

Does robots.txt remove a page from Google?

No. It only asks crawlers not to visit a URL. A page that is already indexed can stay in search results, sometimes without a description. To remove a page, use a noindex tag or the removal tools in the search engine console.

Is robots.txt a security feature?

No. The file is public, and anyone can read it. Listing a private folder in it can even point people to that folder. Protect private content with a login or server rules, not with robots.txt.

Does Google use crawl-delay?

Google ignores the Crawl-delay line. Some other crawlers, such as Bing, respect it. Leave the box empty if you do not need it. Google crawl speed is managed through its own tools instead.

More free tools

PX to REM Converter

Convert pixels to rem and em, and back, with a custom root font size. Includes a table of common sizes from 8px to 64px and a copy-ready CSS declaration.

Open tool

Base64 Encode & Decode

Encode text to Base64 or decode Base64 back to text. Supports UTF-8 and URL-safe Base64. Free, instant and runs in your browser.

Open tool

URL Encode & Decode

Encode or decode URLs and query strings with percent-encoding. Choose component or full-URL mode. Free, instant and private.

Open tool