AI robots.txt Generator
Choose which AI crawlers can read your site and get robots.txt rules ready to paste.
| User-agent | Operator | Purpose | Allow |
|---|---|---|---|
| GPTBot | OpenAI | Model training | |
| OAI-SearchBot | OpenAI | AI search index | |
| ChatGPT-User | OpenAI | User-requested fetch | |
| ClaudeBot | Anthropic | Model training | |
| anthropic-ai | Anthropic | Model training | |
| Claude-User | Anthropic | User-requested fetch | |
| PerplexityBot | Perplexity | AI search index | |
| Perplexity-User | Perplexity | User-requested fetch | |
| Google-Extended | Model training | ||
| Applebot-Extended | Apple | Model training | |
| CCBot | Common Crawl | Model training | |
| Bytespider | ByteDance | Model training | |
| Meta-ExternalAgent | Meta | Model training |
# AI crawler rules — add to your existing robots.txt
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /Which AI bots should you block?
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent, Applebot-Extended) collect pages to train future models. Blocking them keeps your content out of training sets and does not remove you from AI answers.
Search crawlers (OAI-SearchBot, PerplexityBot) build the index that ChatGPT search and Perplexity cite. Blocking them means your pages cannot be quoted or linked in those answers.
User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page only when a person asks the assistant about it. Blocking them breaks answers about your own site.
Paste the rules into the robots.txt at the root of your domain, next to your existing rules. Crawlers apply the group that matches their user-agent; changes take effect on their next visit.