Swaha AI

Home / Blog / AEO

AEO

robots.txt for AI crawlers: who to let in, and how

By Shivam Brahmbhatt · 1 Oct 2026 · 2 min read

Before an AI assistant can describe your business, its crawler has to be allowed to read your site. A blanket rule or a security plugin can block them, and you may never know.

On this page
  1. What robots.txt does
  2. AI crawlers you may see
  3. A simple, permissive example
  4. Deciding who gets in
  5. How to check
  6. Common questions

What robots.txt does

robots.txt is a plain-text file at the root of your site that tells automated crawlers which parts they may visit. Well-behaved crawlers follow it voluntarily. It works like a request, not a lock. Never use it to hide private data.

AI crawlers you may see

  • GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI)
  • ClaudeBot, Claude-SearchBot and Claude-User (Anthropic)
  • PerplexityBot and Perplexity-User (Perplexity)
  • Google-Extended (a control Google offers over use of your content for its AI models)
  • Applebot-Extended (Apple)

Names and behaviour change, so check each provider's current documentation before relying on this list.

A simple, permissive example

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

You can see a live example at swahaai.com/robots.txt.

Deciding who gets in

Allowing AI crawlers is a business decision. A local business that wants to be found usually benefits from being readable. A publisher whose content is the product may reasonably choose differently. Some crawlers fetch pages for training, others fetch live to answer a user's question, and providers offer different controls for each.

The risk to avoid is blocking by accident. Common causes are a blanket Disallow: / left over from a staging site, a security plugin with a bot-blocking default, or a CDN rule.

How to check

  • Open yourdomain.com/robots.txt in a browser and read it.
  • Look for Disallow: / under User-agent: * or under a named AI crawler.
  • Check your hosting, CDN and security plugin settings for bot rules.
  • Re-check after any site migration.

For the rest of the AEO checklist see what is AEO.

Common questions

Does allowing GPTBot guarantee I appear in ChatGPT?

No. It only permits reading. Appearing depends on many other factors.

Can I block training but allow live answers?

Some providers offer separate crawlers for each. Check their current documentation.

Tell us what is slow, manual or missing

Free, plain-English conversation. We reply within one business day with a read and a scoped, written quote.

Get a free read

Keep reading

← All articles