thetimes.co.uk

Check breakdown

• AI bot access (robots.txt)Issue
  • 4/13 known AI bots are allowed by robots.txt.
  • Bloccato: ClaudeBot (Anthropic (Claude) — training/crawling).
  • Bloccato: anthropic-ai (Anthropic — crawling generico).
  • Bloccato: Claude-User (Anthropic — browsing live in Claude).
  • Bloccato: PerplexityBot (Perplexity — crawling/citazioni).
  • Bloccato: Perplexity-User (Perplexity — browsing live).
  • Bloccato: Google-Extended (Google — training Gemini/AI Overviews).
  • Bloccato: CCBot (Common Crawl (used to train many LLMs)).
  • Bloccato: Bytespider (ByteDance (also used for AI training)).
  • Bloccato: Meta-ExternalAgent (Meta AI).

  How to fix it: Remove the Disallow rules for: ClaudeBot, anthropic-ai, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent.

• llms.txt presence and qualityIssue
  • /llms.txt found (1334 characters).
  • Missing an H1 title at the top of the file.
  • It contains no markdown links to specific pages/resources.

  How to fix it: Improve llms.txt: an H1 with the site name, sections per topic area, markdown links to key pages.

• Structured data (Schema.org JSON-LD)Issue
  • No HTML page available for analysis (homepage unreachable).
• Meta title/description and Open GraphIssue
  • No HTML page available for analysis.
• XML sitemapNeeds work
  • Sitemap reachable with 0 URLs.
  • The sitemap contains no URLs (it may be an unfollowed sitemap index, or empty).
  • It is not declared in robots.txt (crawlers have to guess its location).

  How to fix it: Make sure the sitemap contains valid URLs and is declared in robots.txt.

• Content freshness (sitemap lastmod)Not applicable
  • The sitemap has no <lastmod> tags: freshness cannot be assessed.

  How to fix it: Add <lastmod> to every sitemap URL to signal the update date.

• Content structure (answer-first)Issue
  • No HTML page available for analysis.
• Brand identity consistencyNot applicable
  • At least 2 crawled pages with content are needed to assess cross-page consistency.