What are the main AI crawlers to allow in robots.txt?

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended are the key ones as of 2026.

What's the difference between GPTBot and OAI-SearchBot?

GPTBot handles OpenAI's training and retrieval crawling, while OAI-SearchBot is a separate agent specifically for live ChatGPT search results.

Does allowing Googlebot also allow Google's AI features?

No — Google-Extended is a separate directive that specifically governs whether Gemini and AI Overviews can use your content.

← All Quick Wins

AEO & GEO

Which AI Crawlers Should You Allow in robots.txt?

The list worth knowing in 2026: GPTBot (OpenAI training/retrieval), OAI-SearchBot (ChatGPT's live search — a separate agent from GPTBot, easy to miss), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (governs Gemini/AI Overviews use, separate from standard Google indexing).

The fix:

explicitly Allow: / for each of these in your robots.txt, don't just assume a missing Disallow means they're welcome — being explicit avoids ambiguity if a blanket rule exists elsewhere in the file.

Comments

No comments yet — be the first.