summaryrefslogtreecommitdiff
path: root/docs/_html_extra/robots.txt
AgeCommit message (Collapse)Author
2026-05-07docs(robots): fix stale sitemap URL on currentYuriy Andamasov
Was `https://docs.vyos.io/sitemap.xml` — doesn't exist on RTD's per-version layout. The actual sitemap is at `/en/rolling/sitemap.xml` (matching `html_baseurl` and `sphinx-sitemap`'s output). `circinus`/`1.5` already has the correct `/en/1.5/sitemap.xml`; `sagitta`/`1.4` is being fixed in [#1904](https://github.com/vyos/vyos-documentation/pull/1904). \xf0\x9f\xa4\x96 Generated by [robots](https://vyos.io)
2026-04-14feat: LLM documentation adaptation (llms.txt, sitemap, robots.txt) (#1832)Yuriy Andamasov
* feat: add sphinx-llms-txt and sphinx-sitemap dependencies Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: register sphinx-llms-txt and sphinx-sitemap extensions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add curated llms.txt for LLM discovery Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: update robots.txt with explicit AI bot allowances Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: configure sphinx-llms-txt and sphinx-sitemap options Disable auto-generated llms.txt (curated version served from _html_extra instead). Set sitemap_url_scheme to avoid doubled lang/version path segments since html_baseurl already includes /en/latest/. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2025-02-19changed bots orderallow-atlassian-botYevhen Bondarenko
2025-02-18add atlassian-bot to robots.txt to currentrebortg
2025-02-18add atlassian-bot to robots.txtrebortg
(cherry picked from commit aa4a2101f82a5be4369b0f7bf44c59ddb533769b)
2025-02-17remove robots.txtrebortg
2025-02-17Disallow atlassian-bot in robots.txtrebortg
(cherry picked from commit 0e440465338457b513e8b295f95b112bde75074b)