AI-friendly robots.txt template

# AI-friendly robots.txt template — generated by AIVIS
# Add your own Disallow rules for private/admin paths as needed.

# Classic crawlers (search indexing)
User-agent: *
Allow: /

# AI bots — explicitly allowed so this site can be read, trained on,
# and cited by AI answer engines. Remove a block below to opt out of
# that specific bot.

# OpenAI (ChatGPT) — training
User-agent: GPTBot
Allow: /

# OpenAI — ChatGPT search & citations
User-agent: OAI-SearchBot
Allow: /

# OpenAI — live browsing in ChatGPT
User-agent: ChatGPT-User
Allow: /

# Anthropic (Claude) — training/crawling
User-agent: ClaudeBot
Allow: /

# Anthropic — generic crawling (legacy)
User-agent: anthropic-ai
Allow: /

# Anthropic — Claude search index & citations
User-agent: Claude-SearchBot
Allow: /

# Anthropic — live browsing in Claude
User-agent: Claude-User
Allow: /

# Perplexity — crawling & citations
User-agent: PerplexityBot
Allow: /

# Perplexity — live browsing
User-agent: Perplexity-User
Allow: /

# Google — training Gemini/AI Overviews
User-agent: Google-Extended
Allow: /

# Apple Intelligence
User-agent: Applebot-Extended
Allow: /

# Mistral — live browsing in Le Chat
User-agent: MistralAI-User
Allow: /

# DuckDuckGo — DuckAssist answers
User-agent: DuckAssistBot
Allow: /

# Meta AI — live fetching
User-agent: Meta-ExternalFetcher
Allow: /

# Common Crawl (used to train many LLMs)
User-agent: CCBot
Allow: /

# ByteDance (also used for AI training)
User-agent: Bytespider
Allow: /

# Meta AI
User-agent: Meta-ExternalAgent
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Every Allow: / line for an AI user-agent is intentional: without an explicit directive, many default configurations (WordPress themes, SEO plugins, CDNs) block unknown crawlers out of caution — including AI bots, even when that wasn't the intent.