Add incremental RST-to-MyST swap mechanism (sagitta) (#1868)

* feat(swap-sagitta): add incremental RST-to-MyST swap mechanism

Backport of the swap mechanism from feat/incremental-myst-swap onto
the sagitta release branch. Built directly on top of origin/sagitta,
so the underlying RST tree is sagitta's (not current's).

Mechanism:
- scripts/import_myst.py — import md from myst/* with md- prefix
- scripts/swap_sources.py — rename md-{name}.md → {name}.md before
  Sphinx builds, restore after; writes _build/_swap_state.json and
  _build/_swap_exclude.txt
- docs/Makefile — html/dirhtml/pdf/livehtml all run swap → build →
  trap restore; explicit `swap` and `restore` targets too
- docs/conf.py — MyST extensions enabled; swap exclude_patterns
  loader; _prefer_webp builder hook so html prefers webp over png

Content:
- 202 md-prefixed pages from origin/myst/sagitta (md-{name}.md
  alongside each {name}.rst counterpart)
- 1 plain MyST-only page from myst/sagitta where no .rst exists
  (already at canonical name on sagitta: docs/copyright.md)
- 240 .webp images from myst/sagitta (added alongside the existing
  PNG/JPG so RST builds keep their assets)
- docs/_swap.txt populated with all 202 stems → MyST is served by
  default, revert a page by removing its stem from _swap.txt

🤖 Generated by [robots](https://vyos.io)

* feat(conf): copy .md sources into HTML output for plain-text serving

Adds a build-finished hook that mirrors every .md file from the Sphinx
source tree into the HTML output directory verbatim, making unrendered
MyST sources accessible alongside HTML renders at the same URL path.

🤖 Generated by [robots](https://vyos.io)

* docs: address review feedback (backport from PR #1857)

Fix conversion artifacts, typos, and technical inaccuracies applicable
to the sagitta branch: curly quotes, typos (deamonless, cammans,
amdifferent, trough), incorrect firewall command paths, missing closing
brace in zone-policy, peer name inconsistencies, hardcoded passwords
replaced with vault references, and md-*.md exclusion in conf.py.

🤖 Generated by [robots](https://vyos.io)

* docs: port .readthedocs.yml jobs, _ext/vyos.py fallback and swap-script tests from PR #1857

Parity backport from PR #1857 (current) — three pieces were missing on
sagitta.

- .readthedocs.yml: add build.jobs.pre_build / post_build hooks that run
  scripts/swap_sources.py --swap before the Sphinx build and --restore
  after. Without this, the swap mechanism ships but never runs on RTD
  builds for this branch — the swap is a silent no-op.
- docs/_ext/vyos.py: CmdInclude.run() now falls back to nested_parse()
  when self.state._renderer is not present. Required for cfgcmd /
  opcmd / cmdincludemd directives to render correctly when included
  from MyST pages (the swap mechanism's whole point). Sagitta-only
  delta on _ext/vyos.py (the path = str(path) line on 224) is
  intentionally untouched.
- tests/test_import_myst.py, tests/test_swap_sources.py: tests for the
  swap scripts. The scripts on this branch are byte-identical to
  current's, so the same tests apply. Travels with the branch so CI
  catches per-branch regressions if the scripts ever drift.

🤖 Generated by [robots](https://vyos.io)

* fix(conf): skip md-*.md staging files in _copy_md_sources

Agent-Logs-Url: https://github.com/vyos/vyos-documentation/sessions/919695a7-688d-41b9-89f0-540684625dbc

Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>

2 files changed, 597 insertions, 0 deletions

diff --git a/scripts/import_myst.py b/scripts/import_myst.py
new file mode 100644
index 00000000..b9dd7898
--- /dev/null
+++ b/scripts/import_myst.py
@@ -0,0 +1,270 @@
+#!/usr/bin/env python3
+"""import_myst.py — Import converted MD files from myst/* branches.
+
+Usage
+-----
+  python scripts/import_myst.py --source myst/current --all
+  python scripts/import_myst.py --source myst/current --pages quick-start configuration/firewall/zone
+  python scripts/import_myst.py --source myst/current --all --force
+  python scripts/import_myst.py --source myst/current --all --dry-run
+
+Each imported file is written as md-{name}.md alongside the existing RST
+file. Importing does not activate a page — use swap_sources.py for that.
+"""
+
+import argparse
+import subprocess
+import sys
+import warnings
+from pathlib import Path
+from typing import Optional
+
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+DOCS_DIR = REPO_ROOT / "docs"
+
+
+# ---------------------------------------------------------------------------
+# Core helpers
+# ---------------------------------------------------------------------------
+
+def git_show(ref: str, path: str) -> Optional[bytes]:
+    """Run `git show {ref}:{path}` and return raw bytes, or None on error."""
+    result = subprocess.run(
+        ["git", "show", f"{ref}:{path}"],
+        capture_output=True,
+        cwd=REPO_ROOT,
+    )
+    if result.returncode != 0:
+        return None
+    return result.stdout
+
+
+def list_myst_files(source: str) -> list[str]:
+    """Return list of MD stems from `git ls-tree -r --name-only {source} -- docs/`.
+
+    Stems are relative to docs/ with the .md extension stripped.
+    Only .md files are returned.
+    """
+    result = subprocess.run(
+        ["git", "ls-tree", "-r", "--name-only", source, "--", "docs/"],
+        capture_output=True,
+        text=True,
+        cwd=REPO_ROOT,
+    )
+    if result.returncode != 0:
+        raise SystemExit(
+            f"import_myst: git ls-tree {source!r} failed "
+            f"(exit {result.returncode}): {result.stderr.strip()}"
+        )
+
+    stems = []
+    for line in result.stdout.splitlines():
+        line = line.strip()
+        if not line.endswith(".md"):
+            continue
+        # Strip leading "docs/" prefix and .md extension
+        rel = line[len("docs/"):] if line.startswith("docs/") else line
+        stem = rel[:-3]  # strip ".md"
+        stems.append(stem)
+    return stems
+
+
+def list_rst_files(docs_dir: Path) -> set[str]:
+    """Return set of stems (relative to docs_dir, no extension) for all *.rst files."""
+    build_dir = (docs_dir / "_build").resolve()
+    stems = set()
+    for rst in docs_dir.rglob("*.rst"):
+        try:
+            rst.resolve().relative_to(build_dir)
+            continue  # inside _build/, skip
+        except ValueError:
+            pass
+        rel = rst.relative_to(docs_dir)
+        stem = str(rel)[:-4]  # strip ".rst"
+        stems.add(stem)
+    return stems
+
+
+def import_page(stem: str, md_content: bytes, docs_dir: Path, force: bool) -> bool:
+    """Write md-{name}.md for the given stem.
+
+    - Creates parent dirs as needed.
+    - Skips (returns False) if identical content already exists.
+    - Without --force: warns and skips (returns False) if different content exists.
+    - With --force: overwrites (returns True).
+    - Returns True on successful write.
+
+    Refuses (raises ValueError) if *stem* would resolve outside *docs_dir*.
+    """
+    p = Path(stem)
+    if p.is_absolute() or ".." in p.parts:
+        raise ValueError(f"import_myst: refusing unsafe stem {stem!r}")
+    name = p.name
+    parent = p.parent
+    if str(parent) == ".":
+        target_dir = docs_dir
+    else:
+        target_dir = docs_dir / parent
+
+    dest = target_dir / f"md-{name}.md"
+    docs_root = docs_dir.resolve()
+    if not dest.resolve().parent.is_relative_to(docs_root):
+        raise ValueError(f"import_myst: refusing stem {stem!r} — escapes docs/")
+
+    if dest.exists():
+        existing = dest.read_bytes()
+        if existing == md_content:
+            return False  # identical — skip silently
+        if not force:
+            warnings.warn(
+                f"import_myst: {dest} exists with different content; "
+                f"use --force to overwrite",
+                stacklevel=2,
+            )
+            return False
+        # force=True — fall through to write
+
+    target_dir.mkdir(parents=True, exist_ok=True)
+    dest.write_bytes(md_content)
+    return True
+
+
+# ---------------------------------------------------------------------------
+# do_import
+# ---------------------------------------------------------------------------
+
+def do_import(
+    source: str,
+    pages: Optional[list],
+    all_pages: bool,
+    force: bool,
+    dry_run: bool,
+    docs_dir: Path,
+) -> None:
+    """Orchestrate import from source branch.
+
+    If --all: list MD files from source, intersect with RST stems in docs_dir,
+    import only those that have an RST counterpart.
+    If --pages: import only the listed stems.
+    """
+    docs_dir = Path(docs_dir)
+
+    if all_pages:
+        myst_stems = list_myst_files(source)
+        rst_stems = list_rst_files(docs_dir)
+        candidate_stems = [s for s in myst_stems if s in rst_stems]
+    else:
+        candidate_stems = list(pages or [])
+
+    imported = 0
+    skipped = 0
+    would_import = 0
+
+    for stem in candidate_stems:
+        # Path inside the branch: docs/{stem}.md
+        ref_path = f"docs/{stem}.md"
+        content = git_show(source, ref_path)
+        if content is None:
+            print(
+                f"import_myst: WARNING — could not fetch {source}:{ref_path}",
+                file=sys.stderr,
+            )
+            skipped += 1
+            continue
+
+        if dry_run:
+            print(f"  (dry-run) would import {stem}")
+            would_import += 1
+            continue
+
+        wrote = import_page(stem, content, docs_dir, force)
+        if wrote:
+            imported += 1
+        else:
+            skipped += 1
+
+    if dry_run:
+        print(
+            f"import_myst: (dry-run) would import {would_import}, "
+            f"skipped {skipped}.",
+            file=sys.stderr,
+        )
+    else:
+        print(
+            f"import_myst: imported {imported}, skipped {skipped}.",
+            file=sys.stderr,
+        )
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        description="Import MyST MD files from a myst/* branch with md- prefix."
+    )
+    parser.add_argument(
+        "--source",
+        required=True,
+        help="Git ref to import from (e.g. myst/current)",
+    )
+
+    pages_group = parser.add_mutually_exclusive_group(required=True)
+    pages_group.add_argument(
+        "--pages",
+        nargs="+",
+        metavar="STEM",
+        help="Specific page stems to import (e.g. quick-start configuration/firewall/zone)",
+    )
+    pages_group.add_argument(
+        "--all",
+        action="store_true",
+        dest="all_pages",
+        help="Import all MD files that have a matching RST counterpart",
+    )
+
+    parser.add_argument(
+        "--force",
+        action="store_true",
+        help="Overwrite existing md- files even if content differs",
+    )
+    parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        help="Print what would be imported without writing files",
+    )
+    parser.add_argument(
+        "--docs-dir",
+        type=Path,
+        default=None,
+        help="Path to docs/ directory (default: auto-detected from script location)",
+    )
+
+    args = parser.parse_args(argv)
+
+    if args.docs_dir is None:
+        docs_dir = DOCS_DIR
+    else:
+        docs_dir = args.docs_dir.resolve()
+
+    if not docs_dir.is_dir():
+        parser.error(f"docs directory not found: {docs_dir}")
+
+    do_import(
+        source=args.source,
+        pages=args.pages,
+        all_pages=args.all_pages,
+        force=args.force,
+        dry_run=args.dry_run,
+        docs_dir=docs_dir,
+    )
+
+
+if __name__ == "__main__":
+    main()
diff --git a/scripts/swap_sources.py b/scripts/swap_sources.py
new file mode 100644
index 00000000..683d5ef9
--- /dev/null
+++ b/scripts/swap_sources.py
@@ -0,0 +1,327 @@
+#!/usr/bin/env python3
+"""swap_sources.py — Pre-build swap/restore for incremental RST-to-MyST migration.
+
+Usage
+-----
+  python scripts/swap_sources.py --swap          # rename md-{name}.md → {name}.md
+  python scripts/swap_sources.py --restore       # undo renames
+  python scripts/swap_sources.py --dry-run       # show what would be swapped
+  python scripts/swap_sources.py --status        # show current swap state
+  python scripts/swap_sources.py --swap --docs-dir /path/to/docs
+
+Swap list
+---------
+  docs/_swap.txt — one stem per line, relative to docs/, comments (#) skipped.
+  Example:
+      # migrated pages
+      configuration/firewall/zone
+      quick-start
+
+State files (in docs/_build/, gitignored)
+-----------------------------------------
+  _swap_state.json  — records completed renames
+  _swap_exclude.txt — RST paths to pass to Sphinx exclude_patterns
+"""
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+
+STATE_FILE = "_swap_state.json"
+EXCLUDE_FILE = "_swap_exclude.txt"
+SWAP_LIST_FILE = "_swap.txt"
+STATE_VERSION = 1
+
+
+# ---------------------------------------------------------------------------
+# Core helpers
+# ---------------------------------------------------------------------------
+
+def parse_swap_list(path: Path) -> list:
+    """Read *path* and return a list of stems (comments/blanks skipped).
+
+    Returns [] if the file does not exist.
+    """
+    if not path.exists():
+        return []
+    stems = []
+    for line in path.read_text(encoding="utf-8").splitlines():
+        stripped = line.strip()
+        if stripped and not stripped.startswith("#"):
+            stems.append(stripped)
+    return stems
+
+
+def _build_dir(docs_dir: Path) -> Path:
+    build = docs_dir / "_build"
+    build.mkdir(parents=True, exist_ok=True)
+    return build
+
+
+def _state_path(docs_dir: Path) -> Path:
+    return docs_dir / "_build" / STATE_FILE
+
+
+def _exclude_path(docs_dir: Path) -> Path:
+    return docs_dir / "_build" / EXCLUDE_FILE
+
+
+def _resolve_stem_paths(docs_dir: Path, stem: str):
+    """Return (md_from, md_to, rst_path) as absolute Paths for a given stem."""
+    p = Path(stem)
+    name = p.name
+    parent = p.parent  # may be PosixPath('.')
+    if str(parent) == ".":
+        target_dir = docs_dir
+    else:
+        target_dir = docs_dir / parent
+
+    md_from = target_dir / f"md-{name}.md"    # source: md-prefixed
+    md_to = target_dir / f"{name}.md"          # destination: plain .md
+    rst_path = target_dir / f"{name}.rst"      # RST that will be excluded
+    return md_from, md_to, rst_path
+
+
+# ---------------------------------------------------------------------------
+# do_swap
+# ---------------------------------------------------------------------------
+
+def do_swap(docs_dir: Path) -> None:
+    """Rename md-{name}.md → {name}.md for each stem in _swap.txt.
+
+    Validates collisions, records state, writes exclude file.
+    Rolls back all completed renames on any failure.
+    """
+    docs_dir = Path(docs_dir)
+    swap_list = parse_swap_list(docs_dir / SWAP_LIST_FILE)
+    if not swap_list:
+        print("swap_sources: no stems in swap list, nothing to do.", file=sys.stderr)
+        return
+
+    # Stale state: warn and auto-restore first
+    state_path = _state_path(docs_dir)
+    if state_path.exists():
+        print(
+            "swap_sources: WARNING — stale swap state detected. Auto-restoring before new swap.",
+            file=sys.stderr,
+        )
+        do_restore(docs_dir)
+
+    # Validate all stems before touching the filesystem
+    planned = []  # list of (md_from, md_to, rst_path, stem)
+    for stem in swap_list:
+        md_from, md_to, rst_path = _resolve_stem_paths(docs_dir, stem)
+
+        if not md_from.exists():
+            print(
+                f"swap_sources: skipping {stem!r} — md- source file not found: {md_from}",
+                file=sys.stderr,
+            )
+            continue
+
+        if not rst_path.exists():
+            print(
+                f"swap_sources: skipping {stem!r} — RST file not found: {rst_path}",
+                file=sys.stderr,
+            )
+            continue
+
+        if md_to.exists():
+            raise RuntimeError(
+                f"collision: both RST and unprefixed .md exist for stem {stem!r}. "
+                f"Remove {md_to} before swapping."
+            )
+
+        planned.append((md_from, md_to, rst_path, stem))
+
+    # Execute renames with rollback on failure
+    completed = []  # list of (md_from, md_to) that succeeded
+    try:
+        for md_from, md_to, rst_path, stem in planned:
+            os.rename(md_from, md_to)
+            completed.append((md_from, md_to))
+    except Exception as exc:
+        # Rollback completed renames in reverse order
+        for from_path, to_path in reversed(completed):
+            try:
+                os.rename(to_path, from_path)
+            except Exception as rollback_exc:
+                print(
+                    f"swap_sources: ROLLBACK ERROR for {to_path}: {rollback_exc}",
+                    file=sys.stderr,
+                )
+        raise RuntimeError(f"swap_sources: rename failed, rolled back. Cause: {exc}") from exc
+
+    if not completed:
+        return
+
+    # Build state and exclude data using relative paths (relative to docs_dir)
+    _build_dir(docs_dir)
+    swaps = []
+    exclude_lines = []
+    for md_from, md_to, rst_path, stem in planned:
+        rel_from = md_from.relative_to(docs_dir)
+        rel_to = md_to.relative_to(docs_dir)
+        rel_rst = rst_path.relative_to(docs_dir)
+        swaps.append({
+            "stem": stem,
+            "md_from": str(rel_from),
+            "md_to": str(rel_to),
+            "rst_excluded": str(rel_rst),
+        })
+        exclude_lines.append(str(rel_rst))
+
+    state = {"version": STATE_VERSION, "swaps": swaps}
+    state_path.write_text(json.dumps(state, indent=2), encoding="utf-8")
+    _exclude_path(docs_dir).write_text("\n".join(exclude_lines) + "\n", encoding="utf-8")
+
+    print(f"swap_sources: swapped {len(completed)} file(s).", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------------
+# do_restore
+# ---------------------------------------------------------------------------
+
+def do_restore(docs_dir: Path) -> None:
+    """Undo all renames recorded in _swap_state.json (reverse order)."""
+    docs_dir = Path(docs_dir)
+    state_path = _state_path(docs_dir)
+
+    if not state_path.exists():
+        return  # no-op
+
+    try:
+        state = json.loads(state_path.read_text(encoding="utf-8"))
+    except (json.JSONDecodeError, OSError) as exc:
+        raise RuntimeError(
+            f"swap_sources: cannot read state file {state_path}: {exc}. "
+            f"Inspect or delete it manually."
+        ) from exc
+
+    version = state.get("version")
+    if version != STATE_VERSION:
+        raise RuntimeError(
+            f"swap_sources: state file version {version!r} does not match "
+            f"expected {STATE_VERSION!r}. Inspect {state_path} manually."
+        )
+
+    swaps = state.get("swaps", [])
+
+    for entry in reversed(swaps):
+        if not isinstance(entry, dict) or "md_from" not in entry or "md_to" not in entry:
+            raise RuntimeError(
+                f"swap_sources: malformed entry in {state_path}: {entry!r}. "
+                f"Inspect manually."
+            )
+        md_from = docs_dir / entry["md_from"]   # original source (md-prefixed)
+        md_to = docs_dir / entry["md_to"]        # current location (plain .md)
+        if md_to.exists():
+            os.rename(md_to, md_from)
+        else:
+            print(
+                f"swap_sources: WARNING — expected swapped file not found: {md_to}",
+                file=sys.stderr,
+            )
+
+    # Clean up state and exclude files
+    state_path.unlink(missing_ok=True)
+    exclude_path = _exclude_path(docs_dir)
+    exclude_path.unlink(missing_ok=True)
+
+    print(f"swap_sources: restored {len(swaps)} file(s).", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------------
+# do_dry_run / do_status
+# ---------------------------------------------------------------------------
+
+def do_dry_run(docs_dir: Path) -> None:
+    """Print what would be swapped without making any changes."""
+    docs_dir = Path(docs_dir)
+    swap_list = parse_swap_list(docs_dir / SWAP_LIST_FILE)
+    if not swap_list:
+        print("(dry-run) No stems in swap list.")
+        return
+
+    for stem in swap_list:
+        md_from, md_to, rst_path = _resolve_stem_paths(docs_dir, stem)
+        issues = []
+        if not md_from.exists():
+            issues.append("md- source missing")
+        if not rst_path.exists():
+            issues.append("RST missing")
+        if md_to.exists():
+            issues.append("COLLISION: plain .md already exists")
+        if issues:
+            print(f"  SKIP  {stem}: {', '.join(issues)}")
+        else:
+            print(f"  SWAP  {md_from.name} → {md_to.name}  (in {md_from.parent.relative_to(docs_dir) if md_from.parent != docs_dir else '.'})")
+
+
+def do_status(docs_dir: Path) -> None:
+    """Show current swap state."""
+    docs_dir = Path(docs_dir)
+    state_path = _state_path(docs_dir)
+    if not state_path.exists():
+        print("swap_sources: no active swap state.")
+        return
+
+    state = json.loads(state_path.read_text(encoding="utf-8"))
+    swaps = state.get("swaps", [])
+    print(f"swap_sources: {len(swaps)} file(s) currently swapped:")
+    for entry in swaps:
+        print(f"  {entry['md_from']} → {entry['md_to']}  (excludes {entry['rst_excluded']})")
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        description="Swap md-prefixed MyST files in place of RST files before Sphinx builds."
+    )
+    group = parser.add_mutually_exclusive_group(required=True)
+    group.add_argument("--swap", action="store_true", help="Perform swap")
+    group.add_argument("--restore", action="store_true", help="Restore original files")
+    group.add_argument("--dry-run", action="store_true", help="Show what would be swapped")
+    group.add_argument("--status", action="store_true", help="Show current swap state")
+
+    parser.add_argument(
+        "--docs-dir",
+        type=Path,
+        default=None,
+        help="Path to docs/ directory (default: auto-detected from script location)",
+    )
+
+    args = parser.parse_args(argv)
+
+    if args.docs_dir is None:
+        # Default: repo_root/docs where repo_root = parent of scripts/
+        repo_root = Path(__file__).resolve().parent.parent
+        docs_dir = repo_root / "docs"
+    else:
+        docs_dir = args.docs_dir.resolve()
+
+    if not docs_dir.is_dir():
+        parser.error(f"docs directory not found: {docs_dir}")
+
+    if args.swap:
+        do_swap(docs_dir)
+    elif args.restore:
+        do_restore(docs_dir)
+    elif args.dry_run:
+        do_dry_run(docs_dir)
+    elif args.status:
+        do_status(docs_dir)
+
+
+if __name__ == "__main__":
+    main()