Robots.txt is a text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they should or should not crawl.
The robots.txt file uses directives like "User-agent" (which crawler the rule applies to), "Disallow" (which paths to block), "Allow" (which paths to permit within a blocked directory), and "Sitemap" (location of your XML sitemap). Common uses include blocking admin areas, login pages, search result pages, and duplicate content from being crawled. Important: robots.txt blocks crawling, not indexing — if other sites link to a blocked page, Google may still index it (just without crawling its content). For pages you want completely excluded from search results, use the "noindex" meta tag instead. Always test your robots.txt file using Google Search Console's robots.txt tester before making changes, as incorrect rules can accidentally block important pages from being crawled.
Organic Visit delivers real visitors from search engines. Boost your rankings and grow your website.
View Pricing Plans