How to allow AI crawlers in robots.txt
Open yourbusiness.com/robots.txt in a browser. If you see Disallow: / under User-agent: *, or under a name such as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, that assistant has been told to stay out. Remove the rule, or use the example file below.
An AI assistant can only describe a business it is allowed to read. robots.txt is the small text file where a website tells crawlers what they may visit, and one line in it can lock an assistant out without the owner ever knowing.
Which crawler belongs to which assistant
Each company runs more than one crawler, and they do different jobs. The ones that fetch pages for answers matter most for being recommended.
| Assistant | Name in robots.txt | What it does |
|---|---|---|
| ChatGPT | OAI-SearchBot | Finds pages to show and link in ChatGPT search answers. |
| ChatGPT | ChatGPT-User | Opens a page when a person asks ChatGPT about it. |
| ChatGPT | GPTBot | Collects content that may be used to train OpenAI's models. |
| Claude | Claude-SearchBot | Finds pages for Claude's search answers. |
| Claude | Claude-User | Opens a page when a person asks Claude about it. |
| Claude | ClaudeBot | Collects content that may be used to train Anthropic's models. |
| Perplexity | PerplexityBot | Builds the index Perplexity uses to answer and cite. |
| Perplexity | Perplexity-User | Opens a page when a person asks Perplexity about it. |
| Gemini | Googlebot | Google's main crawler. Google's AI answers draw on the same index as Google Search. |
| Gemini | Google-Extended | Not a separate crawler. A switch for whether your content may be used for Gemini models. |
| Copilot | Bingbot | Builds the Bing index, which Microsoft Copilot draws on. |
How to check your own file
- Type your address followed by
/robots.txtinto a browser, for exampleyourbusiness.com/robots.txt. - Look for any of the names in the table, and for a group that starts
User-agent: *, which applies to every crawler not named elsewhere. - Under each one, look for
Disallow: /. A single slash means "everything". That crawler is blocked from the whole site.
If the file does not exist at all, that is fine. No robots.txt means every crawler is allowed.
A robots.txt that lets every assistant in
For most small businesses the right file is the simplest one. It welcomes all crawlers and points them to your sitemap:
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xml
Replace the address with your own. If you already have rules that keep crawlers out of private areas such as a cart or an admin page, keep those lines.
If you want answers but not training
Some owners are happy to be quoted in answers but do not want their content used to train models. You can block only the training crawlers and leave the answer crawlers open:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xml
This is a reasonable choice and it is yours to make. The trade-off is that an assistant's built-in knowledge of your business then comes only from what others have written about you. Our score gives most of its credit for the answer crawlers and a smaller share for the training ones, for that reason.
Where to edit robots.txt
- Wix: in your dashboard, open the SEO settings and look for the robots.txt editor.
- WordPress: most SEO plugins include a robots.txt editor in their tools. Otherwise it is a plain text file in the top folder of the site.
- Squarespace: look in the site's crawler settings, where AI crawlers have their own switch.
- Shopify: the file is generated from a theme template named
robots.txt.liquid.
Menus move over time. If you cannot find it, search your platform's help pages for "robots.txt".
Two things robots.txt does not control
- Firewalls and bot protection. Some hosts and security services block AI bots at the network level, separately from robots.txt. If your file is open but assistants still cannot read you, check the bot settings in your hosting or security dashboard.
- A "noindex" tag on the page. This tells crawlers they may visit but may not list or quote the page. It lives in the page's code, not in robots.txt.
See which assistants your site lets in
The free check reads your robots.txt and shows the result for each crawler by name.