KI-freundliche robots.txt-Vorlage

# AI-friendly robots.txt template — generated by AIVIS
# Add your own Disallow rules for private/admin paths as needed.

# Classic crawlers (search indexing)
User-agent: *
Allow: /

# AI bots — explicitly allowed so this site can be read, trained on,
# and cited by AI answer engines. Remove a block below to opt out of
# that specific bot.

# OpenAI (ChatGPT) — training
User-agent: GPTBot
Allow: /

# OpenAI — ricerca/citazioni ChatGPT
User-agent: OAI-SearchBot
Allow: /

# OpenAI — browsing live in ChatGPT
User-agent: ChatGPT-User
Allow: /

# Anthropic (Claude) — training/crawling
User-agent: ClaudeBot
Allow: /

# Anthropic — crawling generico
User-agent: anthropic-ai
Allow: /

# Anthropic — browsing live in Claude
User-agent: Claude-User
Allow: /

# Perplexity — crawling/citazioni
User-agent: PerplexityBot
Allow: /

# Perplexity — browsing live
User-agent: Perplexity-User
Allow: /

# Google — training Gemini/AI Overviews
User-agent: Google-Extended
Allow: /

# Apple Intelligence
User-agent: Applebot-Extended
Allow: /

# Common Crawl (used to train many LLMs)
User-agent: CCBot
Allow: /

# ByteDance (also used for AI training)
User-agent: Bytespider
Allow: /

# Meta AI
User-agent: Meta-ExternalAgent
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Jede Zeile Allow: / für einen KI-User-Agent ist beabsichtigt: ohne explizite Anweisung blockieren viele Standardkonfigurationen (WordPress-Themes, SEO-Plugins, CDNs) aus Vorsicht unbekannte Crawler — einschließlich KI-Bots, auch wenn das nicht die Absicht war.