Robots.txt Explained: How to Control Search Engine Crawling
The robots.txt file sits at your site root and tells search engines which pages to crawl and which to ignore. A misconfigured robots.txt can accidentally de-index your entire site.
Basic Syntax
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://yoursite.com/sitemap.xmlUser-agent: * means these rules apply to all crawlers. You can target specific ones (Googlebot, Bingbot) with specific rules.
Disallow: tells the crawler not to visit that path. An empty Disallow (or omitting it) means everything is allowed.
Common Mistakes
Disallow: /blocks the entire site. Only do this intentionally during development.- Blocking CSS/JS: If you block Googlebot from CSS or JS, Google can't render your page properly for ranking.
- Using robots.txt for security: It's a polite request, not a firewall. Anyone can ignore it. Use authentication, not robots.txt, to protect sensitive data.
Best Practices
- Always include a
Sitemap:directive pointing to your sitemap.xml - Disallow admin, login, and dashboard paths
- Disallow duplicate content paths (print versions, sorted listings)
- Never block CSS, JavaScript, or images that are needed for rendering
When Pages Vanish From Search
The most alarming robots.txt failure looks like this: traffic drops, a section of the site disappears from results, and nothing in the code changed. Almost always the cause is a rule that is broader than intended, because matching is by path prefix rather than by folder. A line reading Disallow: /blog does not block a folder called blog. It blocks every URL that begins with those characters, which includes /blog-archive and /blogging-tips. Adding the trailing slash, Disallow: /blog/, limits it to the directory you meant.
The second common cause is a file that was never meant to reach production. A staging site protected with Disallow: / gets promoted to live along with its robots.txt, and a single line quietly removes the entire domain from crawling. It is worth making the production robots.txt a deliberate, reviewed artefact rather than something copied between environments.
Which Tool for Which Job
Three different mechanisms get confused with one another, and picking the wrong one causes most of the problems people blame on robots.txt.
- robots.txt controls whether a crawler fetches a URL. Use it to save crawl budget on infinite parameter URLs, internal search results, and endpoints that are pointless to crawl.
- A noindex robots meta tag or header controls whether a fetched page is listed. Use it when the page must stay out of results. The crawler has to be allowed to fetch the page to see it, so noindex and a disallow rule on the same URL cancel each other out.
- A canonical link controls which of several similar URLs is treated as authoritative. Use it for duplicates you still want crawled, such as tracking-parameter variants.
The practical rule: if you never want it seen in search, allow the crawl and serve noindex. If you simply do not want crawler time spent there, disallow it.
How Conflicting Rules Resolve
When more than one rule matches a URL, the winner is the most specific one, measured by the length of the path pattern, not the one that appears first. That is what makes a pattern like disallowing a whole directory while allowing a single file inside it work correctly, because the longer Allow rule beats the shorter Disallow.
Crawlers also select exactly one group of rules. If a specific user-agent group exists, that crawler uses it and ignores the wildcard group entirely rather than combining the two. A common mistake is putting the sitemap or a shared rule only in the wildcard group and assuming a named crawler will also honour it. The Sitemap directive is the exception, since it is not tied to any group and applies file-wide.
Verifying It Works
- Request the file directly at your domain root and confirm it returns 200 with a plain text content type. A 404 means no restrictions at all; a 500 can cause crawlers to pause crawling the site.
- Check that it is at the root of the exact host you care about. Subdomains and the www and non-www versions each need their own file.
- Use a robots.txt tester to try specific URLs against the live rules rather than reading the file and reasoning about it.
- After any change, watch coverage reports for pages moving into an excluded state, which is the earliest warning that a rule is too broad.
Frequently Asked Questions
Can robots.txt hide private content? No. The file is publicly readable, so it advertises the paths you wanted hidden. Use authentication for anything genuinely private.
Does an empty file cause problems? No. An empty or missing robots.txt simply means everything may be crawled, which is a fine default for most small sites.
Should I block my CSS and JavaScript? No. Search engines render pages, and blocking assets prevents them from seeing what a visitor sees.
How fast do changes take effect? Crawlers cache the file, typically for around a day, so treat changes as taking effect within roughly 24 hours rather than instantly.
Bottom Line
Every site needs a robots.txt. Keep it simple: allow everything by default, block admin/private paths, include your sitemap URL. Use our generator to create it correctly.