Add incremental RST-to-MyST swap mechanism (sagitta) (#1868)

* feat(swap-sagitta): add incremental RST-to-MyST swap mechanism

Backport of the swap mechanism from feat/incremental-myst-swap onto
the sagitta release branch. Built directly on top of origin/sagitta,
so the underlying RST tree is sagitta's (not current's).

Mechanism:
- scripts/import_myst.py — import md from myst/* with md- prefix
- scripts/swap_sources.py — rename md-{name}.md → {name}.md before
  Sphinx builds, restore after; writes _build/_swap_state.json and
  _build/_swap_exclude.txt
- docs/Makefile — html/dirhtml/pdf/livehtml all run swap → build →
  trap restore; explicit `swap` and `restore` targets too
- docs/conf.py — MyST extensions enabled; swap exclude_patterns
  loader; _prefer_webp builder hook so html prefers webp over png

Content:
- 202 md-prefixed pages from origin/myst/sagitta (md-{name}.md
  alongside each {name}.rst counterpart)
- 1 plain MyST-only page from myst/sagitta where no .rst exists
  (already at canonical name on sagitta: docs/copyright.md)
- 240 .webp images from myst/sagitta (added alongside the existing
  PNG/JPG so RST builds keep their assets)
- docs/_swap.txt populated with all 202 stems → MyST is served by
  default, revert a page by removing its stem from _swap.txt

🤖 Generated by [robots](https://vyos.io)

* feat(conf): copy .md sources into HTML output for plain-text serving

Adds a build-finished hook that mirrors every .md file from the Sphinx
source tree into the HTML output directory verbatim, making unrendered
MyST sources accessible alongside HTML renders at the same URL path.

🤖 Generated by [robots](https://vyos.io)

* docs: address review feedback (backport from PR #1857)

Fix conversion artifacts, typos, and technical inaccuracies applicable
to the sagitta branch: curly quotes, typos (deamonless, cammans,
amdifferent, trough), incorrect firewall command paths, missing closing
brace in zone-policy, peer name inconsistencies, hardcoded passwords
replaced with vault references, and md-*.md exclusion in conf.py.

🤖 Generated by [robots](https://vyos.io)

* docs: port .readthedocs.yml jobs, _ext/vyos.py fallback and swap-script tests from PR #1857

Parity backport from PR #1857 (current) — three pieces were missing on
sagitta.

- .readthedocs.yml: add build.jobs.pre_build / post_build hooks that run
  scripts/swap_sources.py --swap before the Sphinx build and --restore
  after. Without this, the swap mechanism ships but never runs on RTD
  builds for this branch — the swap is a silent no-op.
- docs/_ext/vyos.py: CmdInclude.run() now falls back to nested_parse()
  when self.state._renderer is not present. Required for cfgcmd /
  opcmd / cmdincludemd directives to render correctly when included
  from MyST pages (the swap mechanism's whole point). Sagitta-only
  delta on _ext/vyos.py (the path = str(path) line on 224) is
  intentionally untouched.
- tests/test_import_myst.py, tests/test_swap_sources.py: tests for the
  swap scripts. The scripts on this branch are byte-identical to
  current's, so the same tests apply. Travels with the branch so CI
  catches per-branch regressions if the scripts ever drift.

🤖 Generated by [robots](https://vyos.io)

* fix(conf): skip md-*.md staging files in _copy_md_sources

Agent-Logs-Url: https://github.com/vyos/vyos-documentation/sessions/919695a7-688d-41b9-89f0-540684625dbc

Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>

1 files changed, 225 insertions, 0 deletions

diff --git a/docs/contributing/md-testing.md b/docs/contributing/md-testing.md
new file mode 100644
index 00000000..4f406e05
--- /dev/null
+++ b/docs/contributing/md-testing.md
@@ -0,0 +1,225 @@
+# Testing
+
+One of the major advantages introduced in VyOS 1.3 is an automated test
+framework. When assembling an ISO image multiple things can go wrong badly and
+publishing a faulty ISO makes no sense. The user is disappointed by the quality
+of the image and the developers get flodded with bug reports over and over
+again.
+
+As the VyOS documentation is not only for users but also for the developers -
+and we keep no secret documentation - this section describes how the automated
+testing works.
+
+## Jenkins CI
+
+Our [VyOS CI]() system is based on Jenkins and builds all our required packages
+for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build
+Job which builds and tests the VyOS ISO image which is published after a
+successful test drive.
+
+We differentiate in two independent tests, which are both run in parallel by
+two separate QEmu instances which are launched via `make test` and `make testc` from within the [vyos-build]() repository.
+
+## Smoketests
+
+Smoketests executes predefined VyOS CLI commands and checks if the desired
+daemon/service configuration is rendert - that is how to put it "short".
+
+When and ISO image is assembled by the [VyOS CI](), the `BUILD_SMOKETEST`
+parameter is enabled by default, which will extend the ISO configuration line
+with the following packages:
+
+``` python
+def CUSTOM_PACKAGES = ''
+  if (params.BUILD_SMOKETESTS)
+    CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest'
+```
+
+So if you plan to build your own custom ISO image and want to make use of our
+smoketests, ensure that you have the <span class="title-ref">vyos-1x-smoketest</span> package installed.
+
+The `make test` command from the [vyos-build]() repository will launch a new
+QEmu instance and the ISO image is first installed to the virtual harddisk.
+
+After its first boot into the newly installed system the main Smoketest script
+is executed, it can be found here: <span class="title-ref">/usr/bin/vyos-smoketest</span>
+
+The script only searches for executable "test-cases" under
+`/usr/libexec/vyos/tests/smoke/cli/` and executes them one by one.
+
+<div class="note">
+
+<div class="title">
+
+Note
+
+</div>
+
+As Smoketests will alter the system configuration and you are logged
+in remote you may loose your connection to the system.
+
+</div>
+
+### Manual Smoketest Run
+
+On the other hand - as each test is contain in its own file - one can always
+execute a single Smoketest by hand by simply running the Python test scripts.
+
+Example:
+
+``` none
+vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
+test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok
+test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok
+test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok
+test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok
+test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok
+test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok
+test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok
+test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok
+test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok
+test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok
+test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok
+test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok
+test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok
+
+----------------------------------------------------------------------
+Ran 13 tests in 348.191s
+
+OK
+```
+
+### Interface based tests
+
+Our smoketests not only test daemons and serives, but also check if what we
+configure for an interface works. Thus there is a common base classed named:
+`base_interfaces_test.py` which holds all the common code that an interface
+supports and is tested.
+
+Those common tests consists out of:
+
+- Add one or more IP addresses
+- DHCP client and DHCPv6 prefix delegation
+- MTU size
+- IP and IPv6 options
+- Port description
+- Port disable
+- VLANs (QinQ and regular 802.1q)
+- ...
+
+<div class="note">
+
+<div class="title">
+
+Note
+
+</div>
+
+When you are working on interface configuration and you also want to
+test if the Smoketests pass you would normally loose the remote SSH connection
+to your `DUT (Device Under Test)`. To handle this issue, some of the
+interface based tests can be called with an environment variable beforehand
+to limit the number of interfaces used in the test. By default all interface
+e.g. all Ethernet interfaces are used.
+
+</div>
+
+``` none
+vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py
+test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok
+test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok
+test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok
+test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok
+test_bonding_min_links (__main__.BondingInterfaceTest) ... ok
+test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok
+test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok
+test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok
+test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok
+test_interface_description (__main__.BondingInterfaceTest) ... ok
+test_interface_disable (__main__.BondingInterfaceTest) ... ok
+test_interface_ip_options (__main__.BondingInterfaceTest) ... ok
+test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok
+test_interface_mtu (__main__.BondingInterfaceTest) ... ok
+test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok
+test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok
+test_span_mirror (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok
+test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok
+test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok
+
+----------------------------------------------------------------------
+Ran 23 tests in 244.694s
+
+OK
+```
+
+This will limit the <span class="title-ref">bond</span> interface test to only make use of <span class="title-ref">eth1</span> and <span class="title-ref">eth2</span>
+as member ports.
+
+## Config Load Tests
+
+The other part of our tests are called "config load tests". The config load tests
+will load - one after another - arbitrary configuration files to test if the
+configuration migration scripts work as designed and that a given set of
+functionality still can be loaded with a fresh VyOS ISO image.
+
+The configurations are all derived from production systems and can not only act
+as a testcase but also as reference if one wants to enable a certain feature.
+The configurations can be found here:
+<https://github.com/vyos/vyos-1x/tree/current/smoketest/configs>
+
+The entire test is controlled by the main wrapper script `/usr/bin/vyos-configtest`
+which behaves in the same way as the main smoketest script. It scans the folder
+for potential configuration files and issues a `load` command one after another.
+
+### Manual config load test
+
+One is not bound to load all configurations one after another but can also load
+individual test configurations on his own.
+
+``` none
+vyos@vyos:~$ configure
+load[edit]
+
+vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small
+Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small'
+Load complete. Use 'commit' to make changes effective.
+[edit]
+vyos@vyos# compare
+[edit interfaces ethernet eth0]
+-hw-id 00:50:56:bf:c5:6d
+[edit interfaces ethernet eth1]
++duplex auto
+-hw-id 00:50:56:b3:38:c5
++speed auto
+[edit interfaces]
+-ethernet eth2 {
+-    hw-id 00:50:56:b3:9c:1d
+-}
+-vti vti1 {
+-    address 192.0.2.1/30
+-}
+...
+
+vyos@vyos# commit
+vyos@vyos#
+```
+
+<div class="note">
+
+<div class="title">
+
+Note
+
+</div>
+
+Some of the configurations have preconditions which need to be met.
+Those most likely include generation of crypographic keys before the config
+can be applied - you will get a commit error otherwise. If you are interested
+how those preconditions are fulfilled check the [vyos-build]() repository and
+the `scripts/check-qemu-install` file.
+
+</div>