AI bots: the complete user-agent list
Every AI answer engine uses a crawler with a declared user-agent. This list is generated directly from the same constant used by our scanning engine: it stays in sync automatically as new bots emerge.
| User-agent | Engine / purpose | Category |
|---|---|---|
| GPTBot | OpenAI (ChatGPT) — training | Training |
| OAI-SearchBot | OpenAI — ricerca/citazioni ChatGPT | Live citation |
| ChatGPT-User | OpenAI — browsing live in ChatGPT | Live citation |
| ClaudeBot | Anthropic (Claude) — training/crawling | Training |
| anthropic-ai | Anthropic — crawling generico | Training |
| Claude-User | Anthropic — browsing live in Claude | Live citation |
| PerplexityBot | Perplexity — crawling/citazioni | Live citation |
| Perplexity-User | Perplexity — browsing live | Live citation |
| Google-Extended | Google — training Gemini/AI Overviews | Training |
| Applebot-Extended | Apple Intelligence | Training |
| CCBot | Common Crawl (used to train many LLMs) | Training |
| Bytespider | ByteDance (also used for AI training) | Training |
| Meta-ExternalAgent | Meta AI | Training |
"Training" gathers content to train future models; "Live citation" answers a specific query in real time, citing sources. Blocking one doesn't automatically block the other: they need to be managed separately in robots.txt.
Want to know which of these bots your site blocks today?
Check your robots.txt for free