robots.txt generator

The robots.txt file at the root of a website tells search engine crawlers and other bots which paths they may crawl. It is one of the first files Googlebot requests. This generator builds a correctly formatted file from per-bot rules and sitemap URLs.

How to use it

1. Add a group for all crawlers (*) and, if needed, specific bots such as Googlebot or GPTBot.
2. Add Allow and Disallow paths to each group.
3. Add your sitemap URLs.
4. Save the result as robots.txt at the root of your domain, for example https://example.com/robots.txt.

Example

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Syntax rules

• Paths are case-sensitive and match by prefix: Disallow: /admin also blocks /administrator.
• * matches any characters and $ anchors the end: Disallow: /*.pdf$ blocks PDF files.
• When Allow and Disallow conflict, Google applies the most specific (longest) matching rule.
• Googlebot ignores Crawl-delay; Bing and some others respect it.

Critical mistakes to avoid

• robots.txt does not remove pages from Google. A blocked URL can still be indexed if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag.
• Do not block redirecting or canonicalised URLs. Google must crawl them to see the redirect or canonical.
• Do not block CSS and JavaScript. Google needs them to render pages.
• It is not security. Anyone can read robots.txt, so do not list secret paths.
• A leftover Disallow: / from staging can deindex an entire site.