vyos-documentation.git/docs/_html_extra, branch 1.4VyOS readthedocs (mirror of https://github.com/vyos/vyos-documentation.git)
https://git.amelek.net/vyos/vyos-documentation.git/atom?h=1.42026-05-07T07:18:05+00:00docs(robots): allow AI crawlers + correct sitemap URL on sagitta2026-05-07T07:18:05+00:00Yuriy Andamasovyuriy@vyos.io2026-05-07T07:18:05+00:00urn:sha1:fde7bddeb1a9a499fe3797e55b02a4a8c14b1fc0
Two changes (was buried inside the now-closed [#1870](https://github.com/vyos/vyos-documentation/pull/1870), the rest of which
became redundant after [#1876](https://github.com/vyos/vyos-documentation/pull/1876) replaced `_html_extra/llms.txt` with a
build-time render):
1. Allow rules for the AI training/retrieval crawlers we want to index
the docs (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot,
ChatGPT-User, Claude-SearchBot, Claude-User, OAI-SearchBot,
Perplexity-User). These are the same rules already present on
`current` and `circinus`, just backported to sagitta.
2. Fix the `Sitemap:` URL. Previously `https://docs.vyos.io/sitemap.xml`,
which doesn't exist on RTD's per-version layout. The actual sitemap
is at `/en/1.4/sitemap.xml` (matching `html_baseurl` and
`sphinx-sitemap`'s output, both already configured by [#1876](https://github.com/vyos/vyos-documentation/pull/1876)).
The `atlassian-bot` Allow that was previously here is dropped to match
the same form as `current` and `circinus` — `User-agent: *` covers all
well-behaved crawlers including atlassian-bot.
\xf0\x9f\xa4\x96 Generated by [robots](https://vyos.io)
change 'disallow' to 'allow: /' for atlassian-bot in robots.txt2025-02-19T16:49:38+00:00Robert Göhlergithub@ghlr.de2025-02-19T16:49:38+00:00urn:sha1:4eae940b4a673eb509e32f8b1016a338e0666dd8changed bots order2025-02-19T12:19:00+00:00Yevhen Bondarenkoevgeniy.bondarenko@sentrium.io2025-02-19T11:09:44+00:00urn:sha1:2850c0947da5c80565af5e747c488efdb37b4659add sitemap to robots.txt2025-02-18T18:39:06+00:00rebortggithub@ghlr.de2025-02-18T18:39:06+00:00urn:sha1:9a592485a234477bdb8bb43e969218d950b2bd29add atlassian-bot to robots.txt2025-02-18T18:13:11+00:00rebortggithub@ghlr.de2025-02-18T18:13:11+00:00urn:sha1:aa4a2101f82a5be4369b0f7bf44c59ddb533769bremove robots.txt2025-02-17T21:34:58+00:00Robert Göhlergithub@ghlr.de2025-02-17T21:34:58+00:00urn:sha1:b881812b04c63b0c8caced0715cecefdce164bbfDisallow atlassian-bot in robots.txt2025-02-17T20:55:41+00:00rebortggithub@ghlr.de2025-02-17T20:55:41+00:00urn:sha1:0e440465338457b513e8b295f95b112bde75074b