Merge pull request #2020 from vyos/yuriy/myst-readme-agents-sagitta

docs(sagitta): refresh README + AGENTS source-format claim, inline doc-linter

4 files changed, 364 insertions, 19 deletions

diff --git a/.github/workflows/lint-doc.yml b/.github/workflows/lint-doc.yml
index 7f2f2099..d7f25f38 100644
--- a/.github/workflows/lint-doc.yml
+++ b/.github/workflows/lint-doc.yml
@@ -2,9 +2,33 @@ name: Lint Doc
 on:
   pull_request:
 
+permissions:
+  contents: read
+  pull-requests: read
+
 jobs:
-  lint-doc:
-    uses: vyos/.github/.github/workflows/lint-doc.yml@feature/T6349-reusable-workflows
-    secrets: inherit
-    
-    
+  doc-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Bullfrog Secure Runner
+        continue-on-error: true
+        uses: bullfrogsec/bullfrog@v0.8.4
+        with:
+          egress-policy: audit
+
+      - name: Get File Changes
+        id: file_changes
+        uses: trilom/file-changes-action@v1.2.4
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.x'
+
+      - name: run doc linter
+        env:
+          FILES_MODIFIED: ${{ steps.file_changes.outputs.files_modified }}
+        run: python scripts/doc-linter.py "$FILES_MODIFIED"
diff --git a/AGENTS.md b/AGENTS.md
index 923beae9..bb9da750 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -4,9 +4,16 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project
 
-VyOS user documentation, built with Sphinx and hosted on Read the Docs at https://docs.vyos.io. Sources are MyST Markdown (`.md`) for migrated pages and RST (`.rst`) for pages that haven't been migrated yet. Both formats are first-class to Sphinx.
+VyOS user documentation, built with Sphinx and hosted on Read the Docs at
+https://docs.vyos.io. This branch (`sagitta`, 1.4.x previous LTS) is migrated
+to MyST Markdown (`.md`) except for two pages with known converter issues that
+stayed in RST: `docs/cli.rst` and `docs/installation/cloud/aws.rst`.
+`source_suffix` in `docs/conf.py` is `['.rst', '.md']`. Both formats are
+first-class to Sphinx.
 
-Pre-migration RST shadows of migrated pages are archived under `docs/_rst_legacy/` for reference only — they are excluded from the build and not consulted by Sphinx.
+Pre-migration RST shadows of converted pages are archived under
+`docs/_rst_legacy/` for reference only — they are excluded from the build,
+not consulted by Sphinx, and must not be edited.
 
 ## Build
 
@@ -32,9 +39,9 @@ Output: `docs/_build/html/`.
 ## Lint
 
 The repo doesn't ship a local lint config or pin a linter binary. CI runs
-`vyoslinter` (`doc-linter.py` from the `vyos/.github` repo, via the
-`lint-doc.yml` workflow) on changed files only — see the CI section
-below. For local checks, manually grep for the rules in
+`vyoslinter` (`scripts/doc-linter.py` in this repo, invoked from
+`.github/workflows/lint-doc.yml`) on changed files only — see the CI
+section below. For local checks, manually grep for the rules in
 [Source conventions](#source-conventions) (line length, address space,
 suppression markers).
 
@@ -74,13 +81,22 @@ Mergify is configured at the org level (no `.mergify.yml` in the repo). The PR t
 
 ### Source files
 
-- `docs/<subdir>/<page>.md` — canonical MD source for migrated pages.
-- `docs/<page>.rst` — canonical RST source for pages that have not been migrated yet (no `rst-` prefix, no MD sibling).
-- `docs/_rst_legacy/<subdir>/rst-<page>.rst` — archived pre-migration RST shadows. Excluded from the Sphinx build and from the Context7 index. Reference only.
+- `docs/<subdir>/<page>.md` — canonical MD source for migrated pages
+  (most of the tree).
+- The two RST-only pages on this branch: `docs/cli.rst` and
+  `docs/installation/cloud/aws.rst`. Both stayed RST due to converter issues
+  at MyST-migration time and have not been revisited since.
+- `docs/_include/<name>.txt` — shared RST snippets included into MyST
+  pages via `cmdincludemd`. Their content is parsed as RST so the
+  legacy templates keep working unchanged.
+- `docs/_rst_legacy/<subdir>/rst-<page>.rst` — archived pre-migration RST
+  shadows of converted pages. Excluded from the Sphinx build and from the
+  Context7 index. Reference only.
 
 **Editing rules:**
-- Migrated page (has `<page>.md`): edit the `.md`. Do not touch the archived shadow under `_rst_legacy/`.
-- Non-migrated page (RST-only): edit the `.rst`.
+- Existing migrated page (has `<page>.md`): edit the `.md`. Do not touch the
+  archived shadow under `_rst_legacy/`.
+- One of the two remaining RST-only pages above: edit the `.rst`.
 - New page: write it as `.md` from the start. The `md-` prefix that earlier MyST migration commits used is gone — never add it.
 
 ### Command directives
@@ -207,7 +223,9 @@ serves and crawlers skip the redirect hop.
 
 ## CI
 
-- **vyoslinter** (`doc-linter.py` from the `vyos/.github` repo, run via `lint-doc.yml`) — line length and IP rules, on changed files only.
+- **vyoslinter** (`scripts/doc-linter.py` in this repo, invoked via
+  `.github/workflows/lint-doc.yml`) — line length and IP rules, on
+  changed files only.
 - **Sphinx build** — runs on Read the Docs for every PR; preview URL appears as a check.
 - **CLA check** — contributors must sign the VyOS CLA before merge.
 - **Conflict check** — fails the PR if it doesn't merge cleanly into base.
diff --git a/README.md b/README.md
index 290c6ce1..daed740d 100644
--- a/README.md
+++ b/README.md
@@ -8,6 +8,23 @@ https://docs.vyos.io.
 The earlier wiki for VyOS 1.1.x and pre-1.2.0 docs is preserved on the
 [Wayback Machine](https://web.archive.org/web/20200225171529/https://wiki.vyos.net/wiki/Main_Page).
 
+## Source format
+
+This branch (`sagitta`, 1.4.x previous LTS) is migrated to
+[MyST Markdown](https://myst-parser.readthedocs.io/) (`.md`) except
+for two pages with known converter issues that stayed in RST:
+`docs/cli.rst` and `docs/installation/cloud/aws.rst`. `source_suffix`
+in `docs/conf.py` is `['.rst', '.md']` to cover both. The
+pre-migration RST shadows of converted pages are archived under
+`docs/_rst_legacy/` for reference; they are excluded from the build
+and must not be edited.
+
+VyOS-specific command directives (`cfgcmd`, `opcmd`, `cmdincludemd`)
+are written as MyST fenced blocks in `.md` pages
+(`myst_fence_as_directive` in `conf.py`) and as the RST forms
+`.. cfgcmd::` / `.. opcmd::` / `.. cmdinclude::` in the two
+remaining `.rst` pages and in shared `_include/*.txt` snippets.
+
 ## Branches
 
 The documentation repository tracks the same branch convention as the VyOS
@@ -60,6 +77,6 @@ Output lands in `docs/_build/html/`.
 
 ## Contributing
 
-See [AGENTS.md](AGENTS.md) for the full contributor guide — source format
-conventions (MyST Markdown for migrated pages, RST for the rest), CLI
-directive syntax, IP-address rules, the linter, and the bot review workflow.
+See [AGENTS.md](AGENTS.md) for the full contributor guide — MyST
+conventions, CLI directive syntax, IP-address rules, linter
+suppression markers, and the Copilot / CodeRabbit bot workflow.
diff --git a/scripts/doc-linter.py b/scripts/doc-linter.py
new file mode 100644
index 00000000..8e74600f
--- /dev/null
+++ b/scripts/doc-linter.py
@@ -0,0 +1,286 @@
+import os
+import re
+import ipaddress
+import sys
+import ast
+
+IPV4SEG  = r'(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])'
+IPV4ADDR = r'\b(?:(?:' + IPV4SEG + r'\.){3,3}' + IPV4SEG + r')\b'
+IPV6SEG  = r'(?:(?:[0-9a-fA-F]){1,4})'
+IPV6GROUPS = (
+    r'(?:' + IPV6SEG + r':){7,7}' + IPV6SEG,                  # 1:2:3:4:5:6:7:8
+    r'(?:\s' + IPV6SEG + r':){1,7}:',                           # 1::                                 1:2:3:4:5:6:7::
+    r'(?:' + IPV6SEG + r':){1,6}:' + IPV6SEG,                 # 1::8               1:2:3:4:5:6::8   1:2:3:4:5:6::8
+    r'(?:' + IPV6SEG + r':){1,5}(?::' + IPV6SEG + r'){1,2}',  # 1::7:8             1:2:3:4:5::7:8   1:2:3:4:5::8
+    r'(?:' + IPV6SEG + r':){1,4}(?::' + IPV6SEG + r'){1,3}',  # 1::6:7:8           1:2:3:4::6:7:8   1:2:3:4::8
+    r'(?:' + IPV6SEG + r':){1,3}(?::' + IPV6SEG + r'){1,4}',  # 1::5:6:7:8         1:2:3::5:6:7:8   1:2:3::8
+    r'(?:' + IPV6SEG + r':){1,2}(?::' + IPV6SEG + r'){1,5}',  # 1::4:5:6:7:8       1:2::4:5:6:7:8   1:2::8
+    IPV6SEG + r':(?:(?::' + IPV6SEG + r'){1,6})',             # 1::3:4:5:6:7:8     1::3:4:5:6:7:8   1::8
+    r':(?:(?::' + IPV6SEG + r'){1,7}|:)',                     # ::2:3:4:5:6:7:8    ::2:3:4:5:6:7:8  ::8       ::
+    r'fe80:(?::' + IPV6SEG + r'){0,4}%[0-9a-zA-Z]{1,}',       # fe80::7:8%eth0     fe80::7:8%1  (link-local IPv6 addresses with zone index)
+    r'::(?:ffff(?::0{1,4}){0,1}:){0,1}[^\s:]' + IPV4ADDR,     # ::255.255.255.255  ::ffff:255.255.255.255  ::ffff:0:255.255.255.255 (IPv4-mapped IPv6 addresses and IPv4-translated addresses)
+    r'(?:' + IPV6SEG + r':){1,4}:[^\s:]' + IPV4ADDR,          # 2001:db8:3:4::192.0.2.33  64:ff9b::192.0.2.33 (IPv4-Embedded IPv6 Address)
+)
+IPV6ADDR = '|'.join(['(?:{})'.format(g) for g in IPV6GROUPS[::-1]])  # Reverse rows for greedy match
+
+MAC = r'([0-9A-F]{2}[:-]){5}([0-9A-F]{2})'
+
+NUMBER = r"([\s']\d+[\s'])"
+
+SUPPORTED_EXTS = ('.md', '.rst', '.txt')
+
+# Linter only applies to published documentation sources under docs/. Repo-root
+# files (README.md, AGENTS.md, .github/copilot-instructions.md) are project
+# meta, not docs content, and are out of scope.
+DOCS_ROOT = 'docs'
+
+
+def is_docs_path(path):
+    """Return True iff `path` resolves under the repo's `docs/` tree.
+
+    Accepts both repo-relative and absolute paths. Both `path` and
+    `DOCS_ROOT` are resolved with `os.path.realpath`, so symlinks are
+    followed to their real targets — a symlink under `docs/` that
+    points outside the tree is correctly treated as out-of-scope, and
+    a `docs/` that is itself a symlink (e.g., in some CI checkouts) is
+    correctly treated as the docs root. Paths are normalized against
+    the linter's working directory (CI invokes it from the repo root;
+    the same is expected for local runs).
+    """
+    abs_path = os.path.realpath(path)
+    abs_docs = os.path.realpath(DOCS_ROOT)
+    try:
+        return os.path.commonpath([abs_path, abs_docs]) == abs_docs
+    except ValueError:
+        # commonpath raises on mixed drives (Windows) or empty input.
+        return False
+
+# MyST / Markdown fenced code block: leading whitespace + 3+ backticks or 3+ colons.
+# Same character and length-or-greater closes.
+MD_FENCE_RE = re.compile(r'^(\s*)(`{3,}|:{3,})(.*)$')
+
+SUPPRESSION_MARKER_RE = {
+    'rst': {
+        'stop': re.compile(r'^\s*\.\.\s+stop_vyoslinter\s*$'),
+        'start': re.compile(r'^\s*\.\.\s+start_vyoslinter\s*$'),
+    },
+    'md': {
+        'stop': re.compile(r'^\s*%\s+stop_vyoslinter\s*$'),
+        'start': re.compile(r'^\s*%\s+start_vyoslinter\s*$'),
+    },
+}
+
+
+def is_suppression_marker(line, kind, in_md_fence, in_rst_codeblock,
+                          md_fence_is_eval_rst, file_ext):
+    """Detect valid stop/start markers in the parser context where they apply.
+
+    `% ...` is only valid in MyST Markdown (`.md`) at top level (outside any fence).
+    `.. ...` is valid in `.rst`/`.txt` at top level (outside RST code-block) and
+    in `.md` only when inside an `{eval-rst}` fence.
+    """
+    if SUPPRESSION_MARKER_RE['md'][kind].match(line):
+        return file_ext == '.md' and not in_md_fence
+    if not SUPPRESSION_MARKER_RE['rst'][kind].match(line):
+        return False
+    if in_rst_codeblock:
+        return False
+    if file_ext in ('.rst', '.txt'):
+        return True
+    if in_md_fence:
+        return md_fence_is_eval_rst
+    return False
+
+
+def lint_mac(cnt, line):
+    """Flag MAC addresses outside the RFC 7042 documentation range."""
+    mac = re.search(MAC, line, re.I)
+    if mac is not None:
+        mac = mac.group()
+        u_mac = re.search(r'((00)[:-](53)([:-][0-9A-F]{2}){4})', mac, re.I)
+        m_mac = re.search(r'((90)[:-](10)([:-][0-9A-F]{2}){4})', mac, re.I)
+        if u_mac is None and m_mac is None:
+            return (f"Use MAC reserved for Documentation (RFC7042): {mac}", cnt, 'error')
+
+
+def lint_ipv4(cnt, line):
+    """Flag IPv4 addresses outside RFC 5737 / private / multicast ranges."""
+    ip = re.search(IPV4ADDR, line, re.I)
+    if ip is not None:
+        ip = ipaddress.ip_address(ip.group().strip(' '))
+        # https://docs.python.org/3/library/ipaddress.html#ipaddress.IPv4Address.is_private
+        if ip.is_private:
+            return None
+        if ip.is_multicast:
+            return None
+        if ip.is_global is False:
+            return None
+        return (f"Use IPv4 reserved for Documentation (RFC 5737) or private Space: {ip}", cnt, 'error')
+
+
+def lint_ipv6(cnt, line):
+    """Flag IPv6 addresses outside RFC 3849 / private / multicast ranges."""
+    ip = re.search(IPV6ADDR, line, re.I)
+    if ip is not None:
+        ip = ipaddress.ip_address(ip.group().strip(' '))
+        if ip.is_private:
+            return None
+        if ip.is_multicast:
+            return None
+        if ip.is_global is False:
+            return None
+        return (f"Use IPv6 reserved for Documentation (RFC 3849) or private Space: {ip}", cnt, 'error')
+
+
+def lint_AS(cnt, line):
+    """Placeholder for future AS-number documentation-range checks (RFC 5398)."""
+    number = re.search(NUMBER, line, re.I)
+    if number:
+        pass
+        # find a way to detect AS numbers
+
+
+def lint_linelen(cnt, line):
+    """Warn when a line exceeds the 80-character docs convention."""
+    line = line.rstrip()
+    if len(line) > 80:
+        return (f"Line too long: len={len(line)}", cnt, 'warning')
+
+def handle_file_action(filepath):
+    """Run all lint checks on one file, respecting fence/code-block and suppression context."""
+    errors = []
+    file_ext = os.path.splitext(filepath)[1].lower()
+    # Stack of open MD/MyST fences: (char, min_len). Supports nesting like
+    # `:::{note}` containing `::::{code-block}`, where the inner opener does
+    # not close the outer note (CommonMark closing rule: matching closer must
+    # have no info string after the fence chars).
+    md_fence_stack = []
+    md_fence_is_eval_rst = False
+    in_rst_codeblock = False
+    rst_codeblock_indent = 0
+    start_vyoslinter = True
+    cnt = 0
+
+    with open(filepath) as fp:
+        for cnt, line in enumerate(fp, start=1):
+            # MD/MyST fenced code block tracking (``` or :::).
+            fence_match = MD_FENCE_RE.match(line)
+            if fence_match:
+                fence = fence_match.group(2)
+                fence_char = fence[0]
+                fence_len = len(fence)
+                info = fence_match.group(3).strip()
+                # A closer matches the top of the stack (same char, len >=
+                # opener) and has no info string. Anything else opens a new
+                # (possibly nested) fence.
+                is_closer = (
+                    md_fence_stack
+                    and not info
+                    and fence_char == md_fence_stack[-1][0]
+                    and fence_len >= md_fence_stack[-1][1]
+                )
+                if is_closer:
+                    md_fence_stack.pop()
+                    if not md_fence_stack:
+                        md_fence_is_eval_rst = False
+                else:
+                    if not md_fence_stack:
+                        md_fence_is_eval_rst = info.startswith('{eval-rst}')
+                    md_fence_stack.append((fence_char, fence_len))
+
+            in_md_fence = bool(md_fence_stack)
+
+            # RST `.. code-block::` tracking (existing semantics for .rst/.txt).
+            # Each `.. code-block::` directive resets the tracked indent so a
+            # later dedent past that column exits the block — even when the
+            # directive itself appears inside an already-open outer block.
+            if in_rst_codeblock:
+                if len(line) > rst_codeblock_indent and not line[rst_codeblock_indent].isspace():
+                    in_rst_codeblock = False
+            if ".. code-block::" in line:
+                in_rst_codeblock = True
+                rst_codeblock_indent = 0
+                for ch in line:
+                    if ch.isspace():
+                        rst_codeblock_indent += 1
+                    else:
+                        break
+
+            if is_suppression_marker(
+                line, 'stop', in_md_fence, in_rst_codeblock,
+                md_fence_is_eval_rst, file_ext,
+            ):
+                start_vyoslinter = False
+            if is_suppression_marker(
+                line, 'start', in_md_fence, in_rst_codeblock,
+                md_fence_is_eval_rst, file_ext,
+            ):
+                start_vyoslinter = True
+
+            if not start_vyoslinter:
+                continue
+
+            test_line_length = not (in_md_fence or in_rst_codeblock)
+
+            err_mac = lint_mac(cnt, line.strip())
+            # disable mac detection for the moment, too many false positives
+            err_mac = None
+            err_ip4 = lint_ipv4(cnt, line.strip())
+            err_ip6 = lint_ipv6(cnt, line.strip())
+            err_len = lint_linelen(cnt, line) if test_line_length else None
+            for e in (err_mac, err_ip4, err_ip6, err_len):
+                if e:
+                    errors.append(e)
+
+        if not start_vyoslinter:
+            errors.append(("Don't forget to turn linter back on", cnt, 'error'))
+
+    if len(errors) > 0:
+        '''
+        "::{$type} file={$filename},line={$line},col=$column::{$log}"
+        '''
+        print(f"File: {filepath}")
+        for error in errors:
+            print(f"::{error[2]} file={filepath},line={error[1]}::{error[0]}")
+        print('')
+        return False
+
+
+def main():
+    """Entry point: lint the changed-file list from argv, or fall back to walking `docs/`."""
+    bool_error = True
+    print('start')
+    # Only the argv-parsing step is wrapped in try/except. Errors raised by
+    # handle_file_action() must propagate so CI failures stay visible instead
+    # of silently triggering a full docs/ walk.
+    try:
+        files = ast.literal_eval(sys.argv[1])
+    except (IndexError, SyntaxError, ValueError):
+        # No argv or malformed list -> fall back to walking DOCS_ROOT.
+        files = None
+
+    if files is not None:
+        for file in files:
+            if (
+                file.endswith(SUPPORTED_EXTS)
+                and "_build" not in file
+                and is_docs_path(file)
+            ):
+                if handle_file_action(file) is False:
+                    bool_error = False
+    else:
+        for root, _dirs, files in os.walk(DOCS_ROOT):
+            path = root.split(os.sep)
+            for file in files:
+                if file.endswith(SUPPORTED_EXTS) and "_build" not in path:
+                    fpath = '/'.join(path)
+                    filepath = f"{fpath}/{file}"
+                    if handle_file_action(filepath) is False:
+                        bool_error = False
+
+    return bool_error
+
+
+if __name__ == "__main__":
+    if main() == False:
+        exit(1)