Where do I put the robots.txt file?

Save it as robots.txt in the root of your domain, so it is served at https://yourdomain.com/robots.txt.

Does Disallow hide a page from Google?

No. Disallow stops crawling but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep a page out of results.

Should I add my sitemap?

Yes. Listing your sitemap URL helps search engines discover all of your pages more efficiently.

Web & SEO

Robots.txt Generator

Create a valid robots.txt file to control how search engines crawl your site.

Free foreverRuns in your browserNo sign-up

PresetUser-agentUse * to target all crawlers, or a name like Googlebot.Disallow pathsOne path per line, e.g. /admin/Allow pathsOne path per line. Useful to allow a file inside a blocked folder.

Crawl-delay (seconds)Optional. Leave blank for none.

Sitemap URL(s)One full URL per line.

robots.txt

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

About the Robots.txt Generator

A robots.txt file tells search engine crawlers which parts of your site they may or may not request. This generator builds a valid file for you from simple options, so you do not have to remember the exact syntax.

Choose a preset such as Allow all, Block all or WordPress defaults, or add your own user-agent, allow and disallow paths, an optional crawl-delay and your sitemap URL.

The file is generated in your browser and is ready to save as robots.txt in the root of your website.

Key features

Presets for common setups
Allow and disallow rules per path
Optional crawl-delay and sitemap
Copy-ready, valid output

How to use

1Pick a preset or choose Custom.
2Add your disallow and allow paths and your sitemap URL.
3Copy the output and save it as robots.txt at your site root.

Frequently asked questions

Where do I put the robots.txt file?: Save it as robots.txt in the root of your domain, so it is served at https://yourdomain.com/robots.txt.
Does Disallow hide a page from Google?: No. Disallow stops crawling but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep a page out of results.
Should I add my sitemap?: Yes. Listing your sitemap URL helps search engines discover all of your pages more efficiently.