AI robots.txt Generator
Choose which AI crawlers can read your site and get robots.txt rules ready to paste.
| User-agent | Operator | Purpose | Allow |
|---|---|---|---|
| GPTBot | OpenAI | Model training | |
| OAI-SearchBot | OpenAI | AI search index | |
| ChatGPT-User | OpenAI | User-requested fetch | |
| ClaudeBot | Anthropic | Model training | |
| anthropic-ai | Anthropic | Model training | |
| Claude-SearchBot | Anthropic | AI search index | |
| Claude-User | Anthropic | User-requested fetch | |
| PerplexityBot | Perplexity | AI search index | |
| Perplexity-User | Perplexity | User-requested fetch | |
| Google-Extended | Model training | ||
| Applebot-Extended | Apple | Model training | |
| MistralAI-User | Mistral | User-requested fetch | |
| DuckAssistBot | DuckDuckGo | User-requested fetch | |
| Meta-ExternalFetcher | Meta | User-requested fetch | |
| CCBot | Common Crawl | Model training | |
| Bytespider | ByteDance | Model training | |
| Meta-ExternalAgent | Meta | Model training |
# AI crawler rules — add to your existing robots.txt
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: MistralAI-User
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /Get the full AI-visibility report for your site
This tool checks one thing. The full scan checks everything ChatGPT, Perplexity, Gemini and Claude look at, scores it, and re-checks your site every month by email. Free, no account.
Only emails about this domain, no spam. Unsubscribe anytime.
Need daily monitoring, more sites or white-label reports? See plansWhich AI bots should you block?
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent, Applebot-Extended) collect pages to train future models. Blocking them keeps your content out of training sets and does not remove you from AI answers.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index that ChatGPT search and Perplexity cite. Blocking them means your pages cannot be quoted or linked in those answers.
User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot, Meta-ExternalFetcher) load a page only when a person asks the assistant about it. Blocking them breaks answers about your own site.
Paste the rules into the robots.txt at the root of your domain, next to your existing rules. Crawlers apply the group that matches their user-agent; changes take effect on their next visit.