robots.txt - What It Is and How to Use It

What is robots.txt?

robots.txt is a plain text file stored at the root of a domain (yoursite.com/robots.txt) that tells search engine crawlers which paths they're allowed to crawl. It is the oldest and most direct mechanism for controlling search engine access to your site, and it's one of the most common places to break SEO with a single misconfigured line.

Why robots.txt Matters

robots.txt errors cause some of the most catastrophic, fastest-acting SEO failures in the industry. A single 'Disallow: /' line accidentally pushed to production can deindex a site within days. Onely's 2024 audit of 1,000 sites found that 8% had robots.txt rules that blocked search engines from at least one important section, usually a staging-config leak or a leftover development rule that was never removed.

How robots.txt Works

Before crawling a site, well-behaved bots like Googlebot, Bingbot, and ClaudeBot fetch /robots.txt and read its rules. The syntax is simple: User-agent specifies which bot the rule applies to (* means all), Allow and Disallow specify path patterns, and Sitemap directs crawlers to your XML sitemap. Rules are read top-down and the most specific match wins.

robots.txt does not enforce indexing rules. A page can be disallowed in robots.txt but still appear in search results if other sites link to it (Google indexes the URL itself but cannot show a description). To remove a page from the index, use a noindex meta tag on the page, not robots.txt.

Common robots.txt Directives

✓User-agent: which crawler the following rules apply to (* = all bots)

✓Disallow: paths the crawler must not access

✓Allow: explicit allow that overrides a broader Disallow

✓Sitemap: location of the XML sitemap

✓Crawl-delay: pause between requests (honoured by Bing, ignored by Google)

✓Wildcards (* and $): pattern matching inside paths

✓Comments (# prefix): notes for humans, ignored by bots

robots.txt Best Practices

Place robots.txt at the domain root, not in a subdirectory

Disallow only what you genuinely don't want crawled (admin, internal search, faceted duplicates)

Never use 'Disallow: /' in production unless you want the entire site removed

Include a Sitemap directive pointing to your XML sitemap

Test rules in Google Search Console's robots.txt Tester before deploying

Allow AI search crawlers (GPTBot, ClaudeBot, PerplexityBot) for AI visibility

Audit robots.txt every quarter and after any staging-to-production deploy

Frequently Asked Questions

Can robots.txt remove a page from Google's index?

No. Disallowing a page in robots.txt only prevents crawling. The URL can still appear in search results if other sites link to it. To remove a page from the index, use a noindex meta tag (or X-Robots-Tag header), and don't block it in robots.txt or Google won't be able to see the noindex.

Should I block AI crawlers like GPTBot in robots.txt?

It's a strategic choice. Blocking GPTBot, ClaudeBot, or PerplexityBot removes your content from AI search results, which is a fast-growing visibility channel. Most sites should allow them. If you want to allow citation but block training, see the specific bot documentation for each.

What happens if robots.txt is missing?

If /robots.txt returns a 404, bots assume the entire site is crawlable. A missing robots.txt is generally fine for small sites, but most sites should still have one with a Sitemap directive and any necessary Disallow rules for admin or internal areas.