Create valid robots exclusion protocol files for search engine crawlers
Crawl delay is supported by Bing, Yahoo, and some bots. Googlebot ignores it.
The robots exclusion protocol (robots.txt) tells search engine crawlers which pages of your site should and should not be indexed. It lives at your domain root (e.g., example.com/robots.txt) and is one of the first files bots check before crawling.
A robots.txt file sits at the root of your domain and tells well-behaved crawlers which parts of your site they may request. It is the oldest and most misunderstood file in web publishing, because it controls crawling, not indexing, and it is a request rather than an enforcement mechanism. This generator builds a correct file and keeps you clear of the mistakes that quietly cost sites their search traffic.
The file is a list of groups. Each group starts with one or more User-agent lines naming the crawler, followed by Disallow and Allow rules giving path prefixes. A User-agent: * group applies to any crawler without a group of its own. A Sitemap line can appear anywhere in the file and applies globally rather than to a single group.
Matching is by path prefix, so Disallow: /admin blocks /admin, /admin/, and /administrator alike. Major crawlers also support * as a wildcard and $ to anchor the end of a URL. When rules conflict, the most specific matching rule wins rather than the first one listed.
Disallowing a page does not remove it from search results. It stops the crawler from fetching the page, which means it can never see a noindex instruction on it, so a URL blocked in robots.txt can still appear in results based on external links. If your goal is to keep a page out of the index, let it be crawled and serve a noindex robots meta tag or header instead.
The second common error is blocking assets. Disallowing your CSS or JavaScript directories prevents search engines from rendering the page the way a visitor sees it, which can hurt how the page is assessed. Modern practice is to leave assets crawlable.
A typical site wants everything public except an admin area and internal search results, and wants to advertise its sitemap. That is a single group for all crawlers, two disallow rules, and one sitemap line. Note that the sitemap must be an absolute URL including the protocol and domain, since a relative path is not valid there.
robots.txt at your domain root. It only works at the root, not in a subdirectory.Crawl-delay. Use crawl rate settings in Search Console if you genuinely need to slow it down.Does robots.txt keep a page out of Google? No. Use a noindex meta tag or header on a crawlable page for that.
Can I have more than one? One per host, at the root. Subdomains each need their own.
Do all crawlers obey it? Reputable ones do. It is voluntary, so malicious scrapers ignore it entirely.
A robots.txt file sits at the root of your domain and tells well-behaved crawlers which parts of the site they may request. It is the first thing most search engine bots fetch. The generator above builds a valid file from your choices so you do not have to remember the exact syntax.
The file is organised into groups. Each group starts with a User-agent line naming the crawler it applies to, followed by Disallow and Allow rules that list path prefixes. A Disallow with an empty value allows everything, while Disallow with a single slash blocks the whole site. It is also good practice to point crawlers to your XML sitemap with a Sitemap line.
Robots.txt is a request, not a lock. It asks compliant crawlers to stay out, but it does not stop a page from being accessed directly, and a disallowed URL can still appear in search if other pages link to it. For genuinely private content, use authentication or a noindex directive rather than relying on robots.txt for security.