diff options
| author | Yuriy Andamasov <yuriy@vyos.io> | 2026-05-07 10:18:05 +0300 |
|---|---|---|
| committer | Yuriy Andamasov <yuriy@vyos.io> | 2026-05-07 10:18:05 +0300 |
| commit | fde7bddeb1a9a499fe3797e55b02a4a8c14b1fc0 (patch) | |
| tree | 9d6adcdb0a292c6a5467b667c02336b8d6b491e6 /docs/_html_extra | |
| parent | eb3945a6b23773a3c4e0ad1f38a672ef919d41e7 (diff) | |
| download | vyos-documentation-fde7bddeb1a9a499fe3797e55b02a4a8c14b1fc0.tar.gz vyos-documentation-fde7bddeb1a9a499fe3797e55b02a4a8c14b1fc0.zip | |
docs(robots): allow AI crawlers + correct sitemap URL on sagitta
Two changes (was buried inside the now-closed [#1870](https://github.com/vyos/vyos-documentation/pull/1870), the rest of which
became redundant after [#1876](https://github.com/vyos/vyos-documentation/pull/1876) replaced `_html_extra/llms.txt` with a
build-time render):
1. Allow rules for the AI training/retrieval crawlers we want to index
the docs (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot,
ChatGPT-User, Claude-SearchBot, Claude-User, OAI-SearchBot,
Perplexity-User). These are the same rules already present on
`current` and `circinus`, just backported to sagitta.
2. Fix the `Sitemap:` URL. Previously `https://docs.vyos.io/sitemap.xml`,
which doesn't exist on RTD's per-version layout. The actual sitemap
is at `/en/1.4/sitemap.xml` (matching `html_baseurl` and
`sphinx-sitemap`'s output, both already configured by [#1876](https://github.com/vyos/vyos-documentation/pull/1876)).
The `atlassian-bot` Allow that was previously here is dropped to match
the same form as `current` and `circinus` — `User-agent: *` covers all
well-behaved crawlers including atlassian-bot.
\xf0\x9f\xa4\x96 Generated by [robots](https://vyos.io)
Diffstat (limited to 'docs/_html_extra')
| -rw-r--r-- | docs/_html_extra/robots.txt | 37 |
1 files changed, 33 insertions, 4 deletions
diff --git a/docs/_html_extra/robots.txt b/docs/_html_extra/robots.txt index e5c4db82..8c2fb298 100644 --- a/docs/_html_extra/robots.txt +++ b/docs/_html_extra/robots.txt @@ -1,7 +1,36 @@ -User-agent: atlassian-bot +User-agent: * Allow: / -User-agent: * -Disallow: # Allow everything +# AI Training Crawlers +User-agent: GPTBot +Allow: / + +User-agent: ClaudeBot +Allow: / + +User-agent: Google-Extended +Allow: / + +User-agent: CCBot +Allow: / + +User-agent: PerplexityBot +Allow: / + +# AI Search/Retrieval +User-agent: ChatGPT-User +Allow: / + +User-agent: Claude-SearchBot +Allow: / + +User-agent: Claude-User +Allow: / + +User-agent: OAI-SearchBot +Allow: / + +User-agent: Perplexity-User +Allow: / -Sitemap: https://docs.vyos.io/sitemap.xml +Sitemap: https://docs.vyos.io/en/1.4/sitemap.xml |