Add incremental RST-to-MyST swap mechanism (sagitta) (#1868)

* feat(swap-sagitta): add incremental RST-to-MyST swap mechanism

Backport of the swap mechanism from feat/incremental-myst-swap onto
the sagitta release branch. Built directly on top of origin/sagitta,
so the underlying RST tree is sagitta's (not current's).

Mechanism:
- scripts/import_myst.py — import md from myst/* with md- prefix
- scripts/swap_sources.py — rename md-{name}.md → {name}.md before
  Sphinx builds, restore after; writes _build/_swap_state.json and
  _build/_swap_exclude.txt
- docs/Makefile — html/dirhtml/pdf/livehtml all run swap → build →
  trap restore; explicit `swap` and `restore` targets too
- docs/conf.py — MyST extensions enabled; swap exclude_patterns
  loader; _prefer_webp builder hook so html prefers webp over png

Content:
- 202 md-prefixed pages from origin/myst/sagitta (md-{name}.md
  alongside each {name}.rst counterpart)
- 1 plain MyST-only page from myst/sagitta where no .rst exists
  (already at canonical name on sagitta: docs/copyright.md)
- 240 .webp images from myst/sagitta (added alongside the existing
  PNG/JPG so RST builds keep their assets)
- docs/_swap.txt populated with all 202 stems → MyST is served by
  default, revert a page by removing its stem from _swap.txt

🤖 Generated by [robots](https://vyos.io)

* feat(conf): copy .md sources into HTML output for plain-text serving

Adds a build-finished hook that mirrors every .md file from the Sphinx
source tree into the HTML output directory verbatim, making unrendered
MyST sources accessible alongside HTML renders at the same URL path.

🤖 Generated by [robots](https://vyos.io)

* docs: address review feedback (backport from PR #1857)

Fix conversion artifacts, typos, and technical inaccuracies applicable
to the sagitta branch: curly quotes, typos (deamonless, cammans,
amdifferent, trough), incorrect firewall command paths, missing closing
brace in zone-policy, peer name inconsistencies, hardcoded passwords
replaced with vault references, and md-*.md exclusion in conf.py.

🤖 Generated by [robots](https://vyos.io)

* docs: port .readthedocs.yml jobs, _ext/vyos.py fallback and swap-script tests from PR #1857

Parity backport from PR #1857 (current) — three pieces were missing on
sagitta.

- .readthedocs.yml: add build.jobs.pre_build / post_build hooks that run
  scripts/swap_sources.py --swap before the Sphinx build and --restore
  after. Without this, the swap mechanism ships but never runs on RTD
  builds for this branch — the swap is a silent no-op.
- docs/_ext/vyos.py: CmdInclude.run() now falls back to nested_parse()
  when self.state._renderer is not present. Required for cfgcmd /
  opcmd / cmdincludemd directives to render correctly when included
  from MyST pages (the swap mechanism's whole point). Sagitta-only
  delta on _ext/vyos.py (the path = str(path) line on 224) is
  intentionally untouched.
- tests/test_import_myst.py, tests/test_swap_sources.py: tests for the
  swap scripts. The scripts on this branch are byte-identical to
  current's, so the same tests apply. Travels with the branch so CI
  catches per-branch regressions if the scripts ever drift.

🤖 Generated by [robots](https://vyos.io)

* fix(conf): skip md-*.md staging files in _copy_md_sources

Agent-Logs-Url: https://github.com/vyos/vyos-documentation/sessions/919695a7-688d-41b9-89f0-540684625dbc

Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>

1 files changed, 189 insertions, 0 deletions

diff --git a/docs/contributing/md-debugging.md b/docs/contributing/md-debugging.md
new file mode 100644
index 00000000..b7e66470
--- /dev/null
+++ b/docs/contributing/md-debugging.md
@@ -0,0 +1,189 @@
+# Debugging
+
+There are two flags available to aid in debugging configuration scripts.
+Since configuration loading issues will manifest during boot, the flags are
+passed as kernel boot parameters.
+
+## ISO image build
+
+When having trouble compiling your own ISO image or debugging Jenkins issues
+you can follow the steps at `iso_build_issues`.
+
+## System Startup
+
+The system startup can be debugged (like loading in the configuration
+file from `/config/config.boot`. This can be achieve by extending the
+Kernel command-line in the bootloader.
+
+### Kernel
+
+- `vyos-debug` - Adding the parameter to the linux boot line will produce
+  timing results for the execution of scripts during commit. If one is seeing
+  an unexpected delay during manual or boot commit, this may be useful in
+  identifying bottlenecks. The internal flag is `VYOS_DEBUG`, and is found
+  in [vyatta-cfg](https://github.com/vyos/vyatta-cfg). Output is directed to `/var/log/vyatta/cfg-stdout.log`.
+- `vyos-config-debug` - During development, coding errors can lead to a
+  commit failure on boot, possibly resulting in a failed initialization of the
+  CLI\. In this circumstance, the kernel boot parameter `vyos-config-debug`
+  will ensure access to the system as user `vyos`, and will log a Python
+  stack trace to the file `/tmp/boot-config-trace`.
+  File `boot-config-trace` will generate only if config loaded with a failure
+  status.
+
+## Live System
+
+A number of flags can be set up to change the behaviour of VyOS at runtime.
+These flags can be toggled using either environment variables or creating
+files.
+
+For each feature, a file called `vyos.feature.debug` can be created to
+toggle the feature on. If a parameter is required it can be placed inside
+the file as its first line.
+
+The file can be placed in `/tmp` for one time debugging (as the file
+will be removed on reboot) or placed in '/config' to stay permanently.
+
+For example, `/tmp/vyos.ifconfig.debug` can be created to enable
+interface debugging.
+
+It is also possible to set up the debugging using environment variables.
+In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG.
+
+For example running, `export VYOS_IFCONFIG_DEBUG=""` on your vbash,
+will have the same effect as `touch /tmp/vyos.ifconfig.debug`.
+
+- `ifconfig` - Once set, all commands used, and their responses received
+  from the OS, will be presented on the screen for inspection.
+- `command` - Once set, all commands used, and their responses received
+  from the OS, will be presented on the screen for inspection.
+- `developer` - Should a command fail, instead of printing a message to the
+  user explaining how to report issues, the python interpreter will start a
+  PBD post-mortem session to allow the developer to debug the issue. As the
+  debugger will wait from input from the developer, it has the capacity to
+  prevent a router to boot and therefore should only be permanently set up
+  on production if you are ready to see the OS fail to boot.
+- `log` - In some rare cases, it may be useful to see what the OS is doing,
+  including during boot. This option sends all commands used by VyOS to a
+  file. The default file is `/tmp/full-log` but it can be changed.
+
+<div class="note">
+
+<div class="title">
+
+Note
+
+</div>
+
+In order to retrieve the debug output on the command-line you need to
+disable `vyos-configd` in addition. This can be run either one-time by
+calling `sudo systemctl stop vyos-configd` or make this reboot-safe by
+calling `sudo systemctl disable vyos-configd`.
+
+</div>
+
+### FRR
+
+Recent versions use the `vyos.frr` framework. The Python class is located
+inside our `vyos-1x:python/vyos/frr.py`. It comes with an embedded debugging/
+(print style) debugger as vyos.ifconfig does.
+
+To enable debugging just run: `$ touch /tmp/vyos.frr.debug`
+
+### Debugging Python Code with PDB
+
+Sometimes it might be useful to debug Python code interactively on the live
+system rather than a IDE. This can be achieved using pdb.
+
+Let us assume you want to debug a Python script that is called by an op-mode
+command. After you found the script by looking up the op-mode-defitions you
+can edit the script in the live system using e.g. vi:
+`vi /usr/libexec/vyos/op_mode/show_xyz.py`
+
+Insert the following statement right before the section where you want to
+investigate a problem (e.g. a statement you see in a backtrace):
+`import pdb; pdb.set_trace()`
+Optionally you can surrounded this statement by an `if` which only triggers
+under the condition you are interested in.
+
+Once you run `show xyz` and your condition is triggered you should be dropped
+into the python debugger:
+
+``` none
+> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process()
+-> rule_type = rule.get('type', '')
+(Pdb)
+```
+
+You can type `help` to get an overview of the available commands, and
+`help command` to get more information on each command.
+
+Useful commands are:
+
+- examine variables using `pp(var)`
+- continue execution using `cont`
+- get a backtrace using `bt`
+
+### Config Migration Scripts
+
+When writing a new configuration migrator it may happen that you see an error
+when you try to invoke it manually on a development system. This error will
+look like:
+
+``` none
+vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot
+Traceback (most recent call last):
+  File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module>
+    config = ConfigTree(config_file)
+  File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__
+    raise ValueError("Failed to parse config: {0}".format(msg))
+ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax.
+```
+
+The reason is that the configuration migration backend is rewritten and uses
+a new form of "magic string" which is applied on demand when real config
+migration is run on boot. When running individual migrators for testing,
+you need to convert the "magic string" on your own by:
+
+``` none
+vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot
+```
+
+### Configuration Error on System Boot
+
+Being brave and running the latest rolling releases will sometimes trigger
+bugs due to corner cases we missed in our design. Those bugs should be filed
+via [Phabricator]() but you can help us to narrow down the issue. Login to your
+VyOS system and change into configuration mode by typing `configure`. Now
+re-load your boot configuration by simply typing `load` followed by return.
+
+You should now see a Python backtrace which will help us to handle the issue,
+please attach it to the [Phabricator]() task.
+
+### Boot Timing
+
+During the migration and extensive rewrite of functionality from Perl into
+Python a significant increase in the overall system boottime was noticed. The
+system boot time can be analysed and a graph can be generated in the end which
+shows in detail who called whom during the system startup phase.
+
+This is done by utilizing the `systemd-bootchart` package which is now
+installed by default on the VyOS 1.3 (equuleus) branch. The configuration is
+also versioned so we get comparable results. `systemd-bootchart` is configured
+using this file: [bootchart.conf](https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf)
+
+To enable boot time graphing change the Kernel commandline and add the following
+string: `init=/usr/lib/systemd/systemd-bootchart`
+
+This can also be done permanently by changing `/boot/grub/grub.cfg`.
+
+## Priorities
+
+VyOS CLI is all about priorities. Every CLI node has a corresponding
+`node.def` file and possibly an attached script that is executed when the
+node is present. Nodes can have a priority, and on system bootup - or any
+other `commit` to the config all scripts are executed from lowest to highest
+priority. This is good as this gives a deterministic behavior.
+
+To debug issues in priorities or to see what's going on in the background
+you can use the `/opt/vyatta/sbin/priority.pl` script which lists to you
+the execution order of the scripts.