AI Visibility Scoreby Web Bloom Solutions

How to allow AI crawlers in robots.txt

By Muhammad Atif, Web Bloom Solutions · Updated November 21, 2026 · 6 minute read

The short answer

Open yourbusiness.com/robots.txt in a browser. If you see Disallow: / under User-agent: *, or under a name such as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, that assistant has been told to stay out. Remove the rule, or use the example file below.

An AI assistant can only describe a business it is allowed to read. robots.txt is the small text file where a website tells crawlers what they may visit, and one line in it can lock an assistant out without the owner ever knowing.

Which crawler belongs to which assistant

Each company runs more than one crawler, and they do different jobs. The ones that fetch pages for answers matter most for being recommended.

AssistantName in robots.txtWhat it does
ChatGPTOAI-SearchBotFinds pages to show and link in ChatGPT search answers.
ChatGPTChatGPT-UserOpens a page when a person asks ChatGPT about it.
ChatGPTGPTBotCollects content that may be used to train OpenAI's models.
ClaudeClaude-SearchBotFinds pages for Claude's search answers.
ClaudeClaude-UserOpens a page when a person asks Claude about it.
ClaudeClaudeBotCollects content that may be used to train Anthropic's models.
PerplexityPerplexityBotBuilds the index Perplexity uses to answer and cite.
PerplexityPerplexity-UserOpens a page when a person asks Perplexity about it.
GeminiGooglebotGoogle's main crawler. Google's AI answers draw on the same index as Google Search.
GeminiGoogle-ExtendedNot a separate crawler. A switch for whether your content may be used for Gemini models.
CopilotBingbotBuilds the Bing index, which Microsoft Copilot draws on.

How to check your own file

  1. Type your address followed by /robots.txt into a browser, for example yourbusiness.com/robots.txt.
  2. Look for any of the names in the table, and for a group that starts User-agent: *, which applies to every crawler not named elsewhere.
  3. Under each one, look for Disallow: /. A single slash means "everything". That crawler is blocked from the whole site.

If the file does not exist at all, that is fine. No robots.txt means every crawler is allowed.

A robots.txt that lets every assistant in

For most small businesses the right file is the simplest one. It welcomes all crawlers and points them to your sitemap:

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

Replace the address with your own. If you already have rules that keep crawlers out of private areas such as a cart or an admin page, keep those lines.

If you want answers but not training

Some owners are happy to be quoted in answers but do not want their content used to train models. You can block only the training crawlers and leave the answer crawlers open:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

This is a reasonable choice and it is yours to make. The trade-off is that an assistant's built-in knowledge of your business then comes only from what others have written about you. Our score gives most of its credit for the answer crawlers and a smaller share for the training ones, for that reason.

Where to edit robots.txt

Menus move over time. If you cannot find it, search your platform's help pages for "robots.txt".

Two things robots.txt does not control

See which assistants your site lets in

The free check reads your robots.txt and shows the result for each crawler by name.

Check my website →

Find out what AI says about you.

Enter your website and get a free score from 0 to 100, with the evidence for every check and a list of what to fix first.

Check my score →