Where does the file go?

At the root of your domain: https://yoursite.com/robots.txt

Can I disallow everything?

Disallow: / blocks all paths for that user-agent — use only when intentional (e.g. pre-launch).

Technical SEO

Robots.txt Generator

Generate a robots.txt file for your site.

Use the tool

General Settings

Default - All Robots are:Crawl-Delay

Sitemap

Search Robots

GoogleGoogle ImageGoogle MobileMSN SearchYahooYahoo MMYahoo BlogsAsk/TeomaGigaBlastDMOZ CheckerNutchAlexa/WaybackBaiduNaverMSN PicSearch

Restricted Directories

The path is relative to root and must contain a trailing slash '/'.

Restricted path 1Restricted path 2Restricted path 3Restricted path 4

robots.txt

User-agent: *
Allow: /
Disallow: /cgi-bin/

Sitemap: http://example.com/sitemap.xml

About & guide

About Robots.txt Generator

robots.txt tells search engine crawlers which paths they may or may not request on your site. A correct file prevents wasted crawl budget and accidental indexing of admin areas.

This generator builds a standard User-agent, Allow, Disallow, and Sitemap block you can place at your domain root.

Use it when launching a new site, blocking staging paths, or adding a sitemap reference for Googlebot.

How To Use Robots.txt Generator

1
Set Allow And Disallow
Use / for allow-all except disallowed paths; block /admin, /cart, or /api as needed.
2
Enter Site URL
Used to build the Sitemap: directive pointing to your XML sitemap.
3
Copy Robots.txt
Save as robots.txt at your site root (https://example.com/robots.txt).
4
Test In Search Console
Use Google Search Console robots.txt tester after upload.

What Robots.txt Generator Offers

Fast & Accurate Results

User-agent: * template.
Custom Allow and Disallow paths.
Automatic Sitemap line from site URL.

Copy-ready Output

Copy-ready robots.txt file.
Plain text output.
Follows common crawler conventions.

No Signup Required

Free for any domain.
Generate multiple versions for staging vs production.
No hosting tie-in.

Works In Your Browser

Generated locally.
You control deployment to your server.
No third-party crawl access.

Built For SEO Best Practices

References sitemap.xml location.
Blocks sensitive folders like /admin.
Pairs with Sitemap Generator.

Pairs With Other SEO Tools

Validate indexing with Canonical Checker.
Plan redirects with Redirect Generator.
Audit meta via Meta Tag Analyser.

Advantages Of Robots.txt Generator

Avoid Crawl Waste

Stop bots from hammering low-value URLs.

Protect Staging Areas

Disallow paths that should not appear in search.

Sitemap Discovery

One line helps crawlers find your XML sitemap.

No Syntax Guesswork

Reduce typos that break crawler interpretation.