From fde7bddeb1a9a499fe3797e55b02a4a8c14b1fc0 Mon Sep 17 00:00:00 2001
From: Yuriy Andamasov <yuriy@vyos.io>
Date: Thu, 7 May 2026 10:18:05 +0300
Subject: docs(robots): allow AI crawlers + correct sitemap URL on sagitta
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Two changes (was buried inside the now-closed [#1870](https://github.com/vyos/vyos-documentation/pull/1870), the rest of which
became redundant after [#1876](https://github.com/vyos/vyos-documentation/pull/1876) replaced `_html_extra/llms.txt` with a
build-time render):

1. Allow rules for the AI training/retrieval crawlers we want to index
   the docs (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot,
   ChatGPT-User, Claude-SearchBot, Claude-User, OAI-SearchBot,
   Perplexity-User). These are the same rules already present on
   `current` and `circinus`, just backported to sagitta.

2. Fix the `Sitemap:` URL. Previously `https://docs.vyos.io/sitemap.xml`,
   which doesn't exist on RTD's per-version layout. The actual sitemap
   is at `/en/1.4/sitemap.xml` (matching `html_baseurl` and
   `sphinx-sitemap`'s output, both already configured by [#1876](https://github.com/vyos/vyos-documentation/pull/1876)).

The `atlassian-bot` Allow that was previously here is dropped to match
the same form as `current` and `circinus` — `User-agent: *` covers all
well-behaved crawlers including atlassian-bot.

\xf0\x9f\xa4\x96 Generated by [robots](https://vyos.io)
---
 docs/_html_extra/robots.txt | 37 +++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

(limited to 'docs')

diff --git a/docs/_html_extra/robots.txt b/docs/_html_extra/robots.txt
index e5c4db82..8c2fb298 100644
--- a/docs/_html_extra/robots.txt
+++ b/docs/_html_extra/robots.txt
@@ -1,7 +1,36 @@
-User-agent: atlassian-bot
+User-agent: *
 Allow: /
 
-User-agent: *
-Disallow: # Allow everything
+# AI Training Crawlers
+User-agent: GPTBot
+Allow: /
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: CCBot
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+# AI Search/Retrieval
+User-agent: ChatGPT-User
+Allow: /
+
+User-agent: Claude-SearchBot
+Allow: /
+
+User-agent: Claude-User
+Allow: /
+
+User-agent: OAI-SearchBot
+Allow: /
+
+User-agent: Perplexity-User
+Allow: /
 
-Sitemap: https://docs.vyos.io/sitemap.xml
+Sitemap: https://docs.vyos.io/en/1.4/sitemap.xml
-- 
cgit v1.2.3