# RidingWatch — robots.txt # Last reviewed: 2026-09-08 (removed Disallow /archive/ — path no longer exists; let Google see the 404s and drop ~340 stale URLs) # Allow major AI training and search bots (we want to be cited). # Restrict the API + admin paths. # ── AI training and search bots — explicitly welcomed ─────────────── User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: cohere-ai Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: Bytespider Allow: / # ── Generic search engines ────────────────────────────────────────── User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # Bingbot honours crawl-delay; Googlebot ignores it. Crawl-delay: 1 # ── Default ───────────────────────────────────────────────────────── User-agent: * Allow: / Disallow: /api/ Disallow: /server/ Disallow: /scripts/ Disallow: /openapi.json Disallow: /data/timeline.json # ── Sitemaps (the canonical "what changed when" signal) ───────────── Sitemap: https://ridingwatch.ca/sitemap-index.xml Sitemap: https://ridingwatch.ca/sitemap.xml Sitemap: https://ridingwatch.ca/sitemap-ridings.xml