AI bots: the complete user-agent list
Every AI answer engine uses a crawler with a declared user-agent. This list is generated directly from the same constant used by our scanning engine: it stays in sync automatically as new bots emerge.
| User-agent | Engine / purpose | Category |
|---|---|---|
| GPTBot | OpenAI (ChatGPT) — training | Training |
| OAI-SearchBot | OpenAI — ChatGPT search & citations | Live citation |
| ChatGPT-User | OpenAI — live browsing in ChatGPT | Live citation |
| ClaudeBot | Anthropic (Claude) — training/crawling | Training |
| anthropic-ai | Anthropic — generic crawling (legacy) | Training |
| Claude-SearchBot | Anthropic — Claude search index & citations | Training |
| Claude-User | Anthropic — live browsing in Claude | Live citation |
| PerplexityBot | Perplexity — crawling & citations | Live citation |
| Perplexity-User | Perplexity — live browsing | Live citation |
| Google-Extended | Google — training Gemini/AI Overviews | Training |
| Applebot-Extended | Apple Intelligence | Training |
| MistralAI-User | Mistral — live browsing in Le Chat | Training |
| DuckAssistBot | DuckDuckGo — DuckAssist answers | Training |
| Meta-ExternalFetcher | Meta AI — live fetching | Training |
| CCBot | Common Crawl (used to train many LLMs) | Training |
| Bytespider | ByteDance (also used for AI training) | Training |
| Meta-ExternalAgent | Meta AI | Training |
"Training" gathers content to train future models; "Live citation" answers a specific query in real time, citing sources. Blocking one doesn't automatically block the other: they need to be managed separately in robots.txt.
Want to know which of these bots your site blocks today?
Check your robots.txt for free