robots.txt
robots.txt is a plain-text file at a site's root that uses User-agent and Disallow rules to tell crawlers which paths not to fetch. Compliance is voluntary.
robots.txt is a plain-text file at the root of a website that uses User-agent and Disallow rules to tell crawlers which paths they should not fetch. Whether a crawler obeys is up to the crawler.
It began as a community convention in 1994 and was standardized by the IETF in 2022 as RFC 9309, the Robots Exclusion Protocol. The file always lives at /robots.txt. Rules are grouped by user agent, and the file can include a Sitemap: line pointing to your XML sitemap.
A simple example:
User-agent: *
Disallow: /admin/
Disallow: /search?
Sitemap: https://example.com/sitemap.xmlThe most common misunderstanding
robots.txt controls crawling, not indexing. A URL blocked with Disallow can still end up in a search index if other sites link to it; it just appears without its content.
To keep a page out of the index, you use noindex, placed in the page’s meta robots tag or an HTTP header. But a crawler can only see noindex if it is allowed to fetch the page. Block the page in robots.txt and add noindex, and the two cancel each other out: the crawler never reads the instruction that would have removed the page.
| Goal | Use |
|---|---|
| Save crawl effort on low-value paths | robots.txt Disallow |
| Keep a page out of search results | noindex, with crawling allowed |
| Point crawlers to your important URLs | Sitemap: line plus a sitemap file |
robots.txt and AI crawlers
AI companies publish the user-agent names of their crawlers, so you can allow or block each one by name. That turns robots.txt into a direct lever on AI visibility.
The consequence of blocking is simple: if an AI provider’s crawler cannot fetch your pages, that provider’s assistant cannot read your content when it searches the web, so it cannot list you as a citation source. The reverse does not hold. Allowing a crawler does not mean you will be cited; it only removes one barrier.
Some providers separate crawlers used for model training from those used to fetch pages for live answers. Check each provider’s current documentation before writing rules, because names and purposes change. Decide deliberately rather than copying a block list from another site: a company that wants to be recommended by AI assistants usually wants answer-time crawlers allowed.
Common mistakes
Disallow: /carried over from staging. A rule meant for a test environment ships to production and blocks the entire site.- Blocking static assets. Disallowing CSS or JavaScript directories can stop crawlers from rendering the page correctly.
- Overly broad parameter rules. A pattern meant for filtered or session URLs also catches real content pages.
- Forgetting subdomains. Each host needs its own robots.txt; rules on
wwwdo not apply toblogorshop.
How to check
Before launch, and after any change, fetch /robots.txt with curl and compare it against the pages you care about. Confirm that key service and content pages are allowed for the crawlers you want, and that nothing important is caught by a broad pattern. This is the first step in the technical layer of an AI visibility audit.
Limits
robots.txt is a request, not access control. Well-behaved crawlers follow it; others may not. Anything genuinely private needs authentication, not a Disallow line. And since robots.txt itself is public, listing sensitive paths in it can draw attention to them.
robots.txt works alongside two other root-level files: the sitemap, which lists what you want found, and llms.txt, which suggests what AI clients should read first. For a check of how your crawl rules affect AI assistants, see the free GEO audit.
Related
- Crawl budget — Crawl budget is how much a crawler will fetch from a site over time, set by what the server can handle and whether the content is worth re-crawling.
- XML sitemap — An XML sitemap (sitemap.xml) lists a site's indexable URLs and when each last changed, telling crawlers which pages exist and which were recently updated.
- llms.txt — llms.txt is a plain-text Markdown file at a website's root that tells large language models what the site is and which key pages are worth reading first.