Guide
robots.txt for AI search crawlers: what to allow and why
To be found by AI search, allow OAI-SearchBot (ChatGPT), PerplexityBot (Perplexity) and Claude-SearchBot (Claude) in your robots.txt. These are separate from the crawlers that collect training data — GPTBot, ClaudeBot and Google-Extended — so you can block training while still allowing search.
Included in every report
Search crawlers vs. training crawlers
| User agent | Vendor | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Search features in ChatGPT |
| GPTBot | OpenAI | Model training |
| ChatGPT-User | OpenAI | User-requested fetches; robots.txt may not apply |
| PerplexityBot | Perplexity | Surfacing sites in Perplexity search |
| Perplexity-User | Perplexity | User-requested fetches; generally ignores robots.txt |
| Claude-SearchBot | Anthropic | Search quality for Claude users |
| ClaudeBot | Anthropic | Model training |
| Claude-User | Anthropic | User-requested fetches |
| Google-Extended | Gemini training control; separate from Google Search |
A robots.txt that allows AI search
Paste this at https://yourdomain.com/robots.txt, adjusting the Disallow lines to your own private areas. A crawler that has its own group ignores the * group, so repeat any private paths in each group.
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Optional: opt out of model training only
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlCommon mistakes
- “Disallow: /” under User-agent: * with no specific groups — this blocks every AI search crawler.
- Blocking all AI bots with a plugin or CDN setting meant to stop training scrapers.
- Assuming that allowing a crawler means your pages are indexed. Allowing is necessary, not sufficient.
- A firewall or bot-protection service blocking crawlers before they ever read robots.txt.
How to check your current robots.txt
- Open yourdomain.com/robots.txt in a browser.
- Find the groups that apply to each search crawler: its own User-agent group if it has one, otherwise User-agent: *.
- Make sure none of those groups contains Disallow: / without a more specific Allow.
Every AIRank report includes this check for OAI-SearchBot, PerplexityBot and Claude-SearchBot.
Frequently asked questions
If I block GPTBot, will ChatGPT stop recommending me?
OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent. Blocking GPTBot does not block OAI-SearchBot.
Do I need llms.txt?
No major assistant documents using llms.txt to choose local businesses. robots.txt access and clear pages matter more.
See what AI says about your business
$39 CAD per business, per report, plus applicable tax. Enter your business details, complete payment, then receive your report after it’s prepared.
Check my site’s AI crawler accessLast updated . We use each assistant’s business-facing connection (API). Answers can differ from its app. A report records the answers collected on the assessment date; future answers may change.