AI Crawlers and robots.txt: Current List and How to Allow or Block Them

They're separate crawlers, so you can say yes to search and no to training.

An easy mistake to make is blocking "all AI" with a list copied from the internet and quietly vanishing from ChatGPT search or Perplexity. This guide lists each company's crawlers based on their official documentation, checked in November 2026.

The three kinds of AI crawler

Most companies now split their bots into three roles:

AI crawler list (November 2026)

The "Token" column is what goes in the User-agent: line of robots.txt.

Official sources: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Common Crawl, and Bing^12345678.

A recommended robots.txt: search yes, training no

This is a common setup for brands and agencies: show up in AI engines without handing over content for model training.

# Search and user actions in AI assistants: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
Allow: /

# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot, Bingbot, and Applebot
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Three things to know before you copy it:

1. A crawler with its own group ignores the * group. If you already have rules under User-agent: * (say, Disallow: /cart/), copy them into the search crawlers' group too.
2. Blocking training is a business call, not a technical one. If you want models to know your brand, you may want to allow training as well. There's no public data measuring how much it matters, so decide based on your content.
3. Changes aren't instant. OpenAI says it takes about 24 hours for its systems to adjust after a robots.txt update, and Meta says up to 24 hours because it caches the file.

Google: Googlebot, Google-Extended, and Search Console

Google's setup is the most confusing, so it's worth spelling out:

• AI Overviews and AI Mode use Googlebot and Google's regular index. Block Googlebot and you disappear from Google altogether.
• Google-Extended doesn't remove you from AI Overviews. It controls whether content is used to train Gemini models and for grounding in the Gemini app and Vertex AI. Google says it has no effect on inclusion or ranking in Google Search.
• To opt out of AI features in Search you have two routes: nosnippet, data-nosnippet, or max-snippet on the page, or the Search generative AI control in Search Console, which is on by default and lets you exclude your site⁹.
• Announced change: Google's docs say it plans to support a shared AI training preference signal, User-agent: __usage_ai-training, in early 2027. The signal is being worked on in the IETF's AIPREF group.

Microsoft: Bing and Copilot

Copilot relies on Bing's index, so Bingbot is the one that matters. Until now, Microsoft used the NOARCHIVE and NOCACHE tags to control how content is used in its AI features. In October 2026 it announced a change⁸:

• Starting February 1, 2027, it will honor a training preference in robots.txt using User-agent: __usage_ai-training with Disallow: /.
• Grounding (use as a source in answers) is allowed by default and controlled at the page level.
• After March 1, 2027, NOARCHIVE and NOCACHE will no longer be read as training or grounding preferences.
• None of this changes eligibility for regular Bing search results.

Crawlers that don't respect robots.txt

User-triggered fetchers are a special case. OpenAI says robots.txt rules "may not apply" to ChatGPT-User; Perplexity says Perplexity-User "generally ignores" robots.txt; Meta says meta-externalfetcher may bypass it. If you truly need to block them, do it at the server or firewall level, using the IP ranges each company publishes.

The reverse is also true: some bots pretend to be well-known crawlers. OpenAI, Anthropic, Perplexity, Google, and Common Crawl publish their IP ranges so you can confirm a visit is legitimate before acting on your logs.

How to check your setup

1. Open https://yourdomain.com/robots.txt and look for each token in the table.
2. Look for dangerous catch-all rules: Disallow: / under User-agent: * blocks every crawler without its own group.
3. Check your firewall or CDN: some block AI bots by default even when robots.txt allows them.
4. Look at your server logs to see which crawlers actually visit.
5. Or run your site through our free GEO audit, which checks your robots.txt against AI crawlers, along with llms.txt and structured data.

Keep going

Letting crawlers in is the first step, not the last. Next you need content worth citing, and a way to measure whether it gets cited: see what GEO is and how to measure AI visibility. In the GEO Plus Academy course, the technical module starts right here, with a task to audit and fix the robots.txt of a real site.