Merge pull request #1898 from vyos/fix/myst-import-sagitta-v2

feat: bundle MyST migration + swap mechanism for sagitta (replaces #1886)

6 files changed, 2089 insertions, 0 deletions

diff --git a/docs/contributing/md-build-vyos.md b/docs/contributing/md-build-vyos.md
new file mode 100644
index 00000000..4541e828
--- /dev/null
+++ b/docs/contributing/md-build-vyos.md
@@ -0,0 +1,838 @@
+(build)=
+
+# Build VyOS
+
+## Prerequisites
+
+There are different ways you can build VyOS.
+
+Building using a {ref}`build_docker` container, although not the only way,
+is the easiest way as all dependencies are managed for you. However, you can
+also set up your own build machine and run a {ref}`build_native`.
+
+:::{note}
+Starting with VyOS 1.2 the release model of VyOS has changed. VyOS
+is now **free as in speech, but not as in beer**. This means that while
+VyOS is still an open source project, the release ISOs are no longer free
+and can only be obtained via subscription, or by contributing to the
+community.
+
+The source code remains public and an ISO can be built using the process
+outlined in this chapter.
+:::
+
+(build_native)=
+
+### Native Build
+
+To build VyOS natively you require a properly configured build host with the
+following Debian versions installed:
+
+- Debian Jessie for VyOS 1.2 (crux)
+- Debian Buster for VyOS 1.3 (equuleus)
+- Debian Bookworm for VyOS 1.4 (sagitta)
+- Debian Bookworm or updated for VyOS 1.5 (circinus, current) - aka the
+  rolling release
+
+To start, clone the repository to your local machine:
+
+```none
+# For VyOS 1.2 (crux)
+$ git clone -b crux --single-branch https://github.com/vyos/vyos-build
+
+# For VyOS 1.3 (equuleus)
+$ git clone -b equuleus --single-branch https://github.com/vyos/vyos-build
+
+# For VyOS 1.4 (sagitta)
+$ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build
+
+# For VyOS 1.5 (circinus, current)
+$ git clone -b current --single-branch https://github.com/vyos/vyos-build
+
+$ cd vyos-build
+
+# For VyOS 1.2 (crux) and VyOS 1.3 (equuleus)
+$ ./configure --architecture amd64 --build-by "j.randomhacker@vyos.io"
+$ sudo make iso
+
+# For VyOS 1.4 (sagitta) and VyOS 1.5 (circinus, current)
+$ sudo make clean
+$ sudo ./build-vyos-image iso --architecture amd64 --build-by "j.randomhacker@vyos.io"
+```
+
+For the packages required, you can refer to the `docker/Dockerfile` file
+in the [repository]. The `./build-vyos-image` script will also warn you if any
+dependencies are missing.
+
+(build_docker)=
+
+### Docker
+
+This will guide you through the process of building a VyOS ISO using [Docker].
+This process has been tested on clean installs of Debian Bullseye (11) and
+Bookworm (12).
+
+Installing [Docker] and prerequisites:
+
+:::{hint}
+Due to the updated version of Docker, the following examples may
+become invalid.
+
+Due to differences in version updates and build processes, content related
+to VyOS 1.3 and below is no longer included below.
+:::
+
+[On Debian]
+
+```none
+# Add Docker's official GPG key:
+$ sudo apt-get update
+$ sudo apt-get install ca-certificates curl gnupg
+$ sudo install -m 0755 -d /etc/apt/keyrings
+$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+$ sudo chmod a+r /etc/apt/keyrings/docker.gpg
+
+# Add the repository to Apt sources:
+$ echo \
+    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+    https://download.docker.com/linux/debian \
+    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
+    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+
+$ sudo apt-get update
+$ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+```
+
+To be able to use [Docker] without `sudo`, the current non-root user must be
+added to the `docker` group by calling: `sudo usermod -aG docker
+yourusername`.
+
+:::{hint}
+Doing so grants privileges equivalent to the `root` user! It is
+recommended to remove the non-root user from the `docker` group after
+building the VyOS ISO. See also [Docker as non-root].
+:::
+
+:::{note}
+The build process needs to be built on a local file system, building
+on SMB or NFS shares will result in the container failing to build properly!
+VirtualBox Drive Share is also not an option as block device operations
+are not implemented and the drive is always mounted as "nodev"
+:::
+
+#### Build Container
+
+The container can be built by hand or by fetching the pre-built one from
+DockerHub. Using the pre-built containers from the [VyOS DockerHub
+organisation][vyos dockerhub organisation] will ensure that the container is always up-to-date. A rebuild
+is triggered once the container changes (please note this will take 2-3 hours
+after pushing to the vyos-build repository).
+
+% note: If you are using the pre-built container, it will be automatically
+% downloaded from DockerHub if it is not found on your local machine when
+% you build the ISO.
+
+##### Dockerhub
+
+To manually download the container from DockerHub, run:
+
+```none
+$ docker pull vyos/vyos-build:sagitta  # For VyOS 1.4
+$ docker pull vyos/vyos-build:current  # For rolling release
+```
+
+##### Build from source
+
+The container can also be built directly from source:
+
+```none
+# For VyOS 1.4 (sagitta)
+$ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build
+
+# For VyOS 1.5 (circinus, current)
+$ git clone -b current --single-branch https://github.com/vyos/vyos-build
+
+
+$ cd vyos-build
+$ docker build -t vyos/vyos-build:sagitta docker     # For VyOS 1.4
+$ docker build -t vyos/vyos-build:current docker     # For rolling release
+```
+
+:::{note}
+Since VyOS has switched to Debian (12) Bookworm in its `current`
+branch, It is recommended to use the official Docker Hub container image
+to build `equleus` and `crux`.
+:::
+
+#### Tips and Tricks
+
+You can create yourself some handy Bash aliases to always launch the latest -
+per release train (`current` or `sagitta`) - container. Add the following to
+your `.bash_aliases` file:
+
+```none
+alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \
+    -v "$(pwd)":/vyos \
+    -v "$HOME/.gitconfig":/etc/gitconfig \
+    -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \
+    -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \
+    -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
+    -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \
+    vyos/vyos-build:current bash'
+
+alias vybld_sagitta='docker pull vyos/vyos-build:sagitta && docker run --rm -it \
+    -v "$(pwd)":/vyos \
+    -v "$HOME/.gitconfig":/etc/gitconfig \
+    -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \
+    -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \
+    -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
+    -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \
+    vyos/vyos-build:sagitta bash'
+```
+
+Now you are prepared with two new aliases `vybld` and `vybld_sagitta` to
+spawn your development containers in your current working directory.
+
+:::{note}
+Some VyOS packages (namely vyos-1x) come with build-time tests which
+verify some of the internal library calls that they work as expected. Those
+tests are carried out through the Python Unittest module. If you want to
+build the `vyos-1x` package (which is our main development package) you
+need to start your Docker container using the following argument:
+`--sysctl net.ipv6.conf.lo.disable_ipv6=0`, otherwise those tests will
+fail.
+:::
+
+(build_iso)=
+
+## Build ISO
+
+Now as you are aware of the prerequisites we can continue and build our own
+ISO from source. For this we have to fetch the latest source code from GitHub.
+Please note as this will differ for both `current` and `crux`.
+
+```none
+# For VyOS 1.4 (sagitta)
+$ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build
+
+# For VyOS 1.5 (circinus, current)
+$ git clone -b current --single-branch https://github.com/vyos/vyos-build
+```
+
+Now a fresh build of the VyOS ISO can begin. Change directory to the
+`vyos-build` directory and run:
+
+```none
+$ cd vyos-build
+
+# For VyOS 1.4 (sagitta)
+$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:sagitta bash
+
+# For VyOS 1.5 (circinus, current)
+$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash
+```
+
+```none
+# For MacOS (crux, equuleus, sagitta)
+$ git clone https://github.com/vyos/vyos-utils-misc
+$ cd build-tools/macos-build
+
+# For VyOS 1.2 (crux)
+$ os=jessie64 branch=crux make build
+
+# For VyOS 1.3 (equuleus)
+$ os=buster64 branch=equuleus make build
+
+# For VyOS 1.4 (sagitta)
+$ os=buster64 branch=sagitta make build
+```
+
+Start the build:
+
+```none
+# For VyOS 1.4 (sagitta) and For VyOS 1.5 (circinus, current)
+vyos_bld@8153428c7e1f:/vyos$ sudo make clean
+vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image iso --architecture amd64 --build-by "j.randomhacker@vyos.io"
+```
+
+When the build is successful, the resulting iso can be found inside the
+`build` directory as `live-image-[architecture].hybrid.iso`.
+
+Good luck!
+
+:::{hint}
+Building VyOS on Windows WSL2 with Docker integrated into WSL2 will
+work like a charm. No problems are known so far!
+:::
+
+(build-source)=
+
+(customize)=
+
+### Customize
+
+This ISO can be customized with the following list of configure options.
+The full and current list can be generated with `./build-vyos-image --help`:
+
+```none
+$ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help
+  I: Checking if packages required for VyOS image build are installed
+  usage: build-vyos-image [-h] [--architecture ARCHITECTURE]
+  [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR]
+  [--debian-security-mirror DEBIAN_SECURITY_MIRROR]
+  [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR]
+  [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE]
+  [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run]
+  [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY]
+  [--custom-package CUSTOM_PACKAGE]
+      [build_flavor]
+
+  positional arguments:
+  build_flavor          Build flavor
+
+  optional arguments:
+  -h, --help            show this help message and exit
+  --architecture ARCHITECTURE
+                          Image target architecture (amd64 or arm64)
+  --build-by BUILD_BY   Builder identifier (e.g. jrandomhacker@example.net)
+  --debian-mirror DEBIAN_MIRROR
+                          Debian repository mirror
+  --debian-security-mirror DEBIAN_SECURITY_MIRROR
+                          Debian security updates mirror
+  --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR
+                          Debian repository mirror for pbuilder env bootstrap
+  --vyos-mirror VYOS_MIRROR
+                          VyOS package mirror
+  --build-type BUILD_TYPE
+                          Build type, release or development
+  --version VERSION     Version number (release builds only)
+  --build-comment BUILD_COMMENT
+                          Optional build comment
+  --debug               Enable debug output
+  --dry-run             Check build configuration and exit
+  --custom-apt-entry CUSTOM_APT_ENTRY
+                          Custom APT entry
+  --custom-apt-key CUSTOM_APT_KEY
+                          Custom APT key file
+  --custom-package CUSTOM_PACKAGE
+                          Custom package to install from repositories
+```
+
+(iso_build_issues)=
+
+#### ISO Build Issues
+
+There are (rare) situations where building an ISO image is not possible at all
+due to a broken package feed in the background. APT is not very good at
+reporting the root cause of the issue. Your ISO build will likely fail with a
+more or less similar looking error message:
+
+```none
+The following packages have unmet dependencies:
+ vyos-1x : Depends: accel-ppp but it is not installable
+E: Unable to correct problems, you have held broken packages.
+P: Begin unmounting filesystems...
+P: Saving caches...
+Reading package lists...
+Building dependency tree...
+Reading state information...
+Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB]
+Del accel-ppp 1.12.0-95-g59f8e1b [475 kB]
+Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB]
+Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB]
+Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB]
+make: *** [Makefile:30: iso] Error 1
+(10:13) vyos_bld ece068908a5b:/vyos [current] #
+```
+
+To debug the build process and gain additional information of what could be the
+root cause, you need to use `chroot` to change into the build directory. This is
+explained in the following step by step procedure:
+
+```none
+vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash
+```
+
+We now need to mount some required, volatile filesystems
+
+```none
+(live)root@ece068908a5b:/# mount -t proc none /proc
+(live)root@ece068908a5b:/# mount -t sysfs none /sys
+(live)root@ece068908a5b:/# mount -t devtmpfs none /dev
+```
+
+We now are free to run any command we would like to use for debugging, e.g.
+re-installing the failed package after updating the repository.
+
+```none
+(live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x
+Get:1 file:/root/packages ./ InRelease
+Ign:1 file:/root/packages ./ InRelease
+Get:2 file:/root/packages ./ Release [1235 B]
+Get:2 file:/root/packages ./ Release [1235 B]
+Get:3 file:/root/packages ./ Release.gpg
+Ign:3 file:/root/packages ./ Release.gpg
+Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease
+Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease
+Hit:6 http://deb.debian.org/debian bullseye InRelease
+Hit:7 http://deb.debian.org/debian buster InRelease
+Hit:8 http://deb.debian.org/debian-security buster/updates InRelease
+Hit:9 http://deb.debian.org/debian buster-updates InRelease
+Hit:10 http://deb.debian.org/debian buster-backports InRelease
+Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease
+Reading package lists... Done
+N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
+Reading package lists... Done
+Building dependency tree
+Reading state information... Done
+Some packages could not be installed. This may mean that you have
+requested an impossible situation or if you are using the unstable
+distribution that some required packages have not yet been created
+or been moved out of Incoming.
+The following information may help to resolve the situation:
+
+The following packages have unmet dependencies:
+ vyos-1x : Depends: accel-ppp but it is not installable
+E: Unable to correct problems, you have held broken packages.
+```
+
+Now it's time to fix the package mirror and rerun the last step until the
+package installation succeeds again!
+
+(build_custom_packages)=
+
+### Linux Kernel
+
+The Linux kernel used by VyOS is heavily tied to the ISO build process. The
+file `data/defaults.toml` hosts a TOML definition of the kernel version used
+`kernel_version` and the `kernel_flavor` of the kernel which represents the
+kernel's LOCAL_VERSION. Both together form the kernel version variable in the
+system:
+
+```none
+vyos@vyos:~$ uname -r
+6.1.52-amd64-vyos
+```
+
+- Accel-PPP
+- Intel NIC drivers
+- Inter QAT
+
+Each of those modules holds a dependency on the kernel version and if you are
+lucky enough to receive an ISO build error which sounds like:
+
+```none
+I: Create initramfs if it does not exist.
+Extra argument '6.1.52-amd64-vyos'
+Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory]
+Options:
+ -k version     Specify kernel version or 'all'
+ -c             Create a new initramfs
+ -u             Update an existing initramfs
+ -d             Remove an existing initramfs
+ -b directory   Set alternate boot directory
+ -v             Be verbose
+See update-initramfs(8) for further details.
+E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors.
+```
+
+The most obvious reasons could be:
+
+- `vyos-build` repo is outdated, please `git pull` to update to the latest
+  release kernel version from us.
+- You have your own custom kernel `*.deb` packages in the `packages` folder but
+  neglected to create all required out-of tree modules like Accel-PPP, Intel
+  QAT or Intel NIC drivers
+
+#### Building The Kernel
+
+The kernel build is quite easy, most of the required steps can be found in the
+`vyos-build/packages/linux-kernel/Jenkinsfile` but we will walk you through
+it.
+
+Clone the kernel source to `vyos-build/packages/linux-kernel/`:
+
+```none
+$ cd vyos-build/packages/linux-kernel/
+$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+```
+
+Check out the required kernel version - see `vyos-build/data/defaults.toml`
+file (example uses kernel 4.19.146):
+
+```none
+$ cd vyos-build/packages/linux-kernel/linux
+$ git checkout v4.19.146
+Checking out files: 100% (61536/61536), done.
+Note: checking out 'v4.19.146'.
+
+You are in 'detached HEAD' state. You can look around, make experimental
+changes and commit them, and you can discard any commits you make in this
+state without impacting any branches by performing another checkout.
+
+If you want to create a new branch to retain commits you create, you may
+do so (now or later) by using -b with the checkout command again. Example:
+
+  git checkout -b <new-branch-name>
+
+HEAD is now at 015e94d0e37b Linux 4.19.146
+```
+
+Now we can use the helper script `build-kernel.sh` which does all the
+necessary voodoo by applying required patches from the
+`vyos-build/packages/linux-kernel/patches` folder, copying our kernel
+configuration `x86_64_vyos_defconfig` to the right location, and finally
+building the Debian packages.
+
+:::{note}
+Building the kernel will take some time depending on the speed and
+quantity of your CPU/cores and disk speed. Expect 20 minutes
+(or even longer) on lower end hardware.
+:::
+
+```none
+(18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh
+I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source
+I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch
+patching file Documentation/networking/ip-sysctl.txt
+patching file include/linux/inetdevice.h
+patching file include/linux/ipv6.h
+patching file include/uapi/linux/ip.h
+patching file include/uapi/linux/ipv6.h
+patching file net/ipv4/devinet.c
+Hunk #1 succeeded at 2319 (offset 1 line).
+patching file net/ipv6/addrconf.c
+patching file net/ipv6/route.c
+I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch
+patching file fs/notify/inotify/Kconfig
+patching file fs/notify/inotify/inotify_user.c
+patching file fs/overlayfs/super.c
+Hunk #2 succeeded at 1713 (offset 9 lines).
+Hunk #3 succeeded at 1739 (offset 9 lines).
+Hunk #4 succeeded at 1762 (offset 9 lines).
+patching file include/linux/inotify.h
+I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch
+patching file scripts/package/builddeb
+I: make x86_64_vyos_defconfig
+  HOSTCC  scripts/basic/fixdep
+  HOSTCC  scripts/kconfig/conf.o
+  YACC    scripts/kconfig/zconf.tab.c
+  LEX     scripts/kconfig/zconf.lex.c
+  HOSTCC  scripts/kconfig/zconf.tab.o
+  HOSTLD  scripts/kconfig/conf
+#
+# configuration written to .config
+#
+I: Generate environment file containing Kernel variable
+I: Build Debian Kernel package
+  UPD     include/config/kernel.release
+/bin/sh ./scripts/package/mkdebian
+dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc
+dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos
+dpkg-buildpackage: info: source version 4.19.146-1
+dpkg-buildpackage: info: source distribution buster
+dpkg-buildpackage: info: source changed by vyos_bld <christian@poessinger.com>
+dpkg-buildpackage: info: host architecture amd64
+dpkg-buildpackage: warning: debian/rules is not executable; fixing that
+ dpkg-source --before-build .
+ debian/rules build
+make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86         KBUILD_BUILD_VERSION=1 KBUILD_SRC=
+  SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
+
+...
+
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols)
+dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols)
+dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'.
+ dpkg-genbuildinfo --build=binary
+ dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes
+dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list
+dpkg-genchanges: info: binary-only upload (no source code included)
+ dpkg-source --after-build .
+dpkg-buildpackage: info: binary-only upload (no source included)
+```
+
+In the end you will be presented with the kernel binary packages which you can
+then use in your custom ISO build process, by placing all the `*.deb` files in
+the vyos-build/packages folder where they will be used automatically when
+building VyOS as documented above.
+
+##### Firmware
+
+If you upgrade your kernel or include new drivers you may need new firmware.
+Build a new `vyos-linux-firmware` package with the included helper scripts.
+
+```none
+$ cd vyos-build/packages/linux-kernel
+$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
+$ ./build-linux-firmware.sh
+$ cp vyos-linux-firmware_*.deb ../
+```
+
+This tries to automatically detect which blobs are needed based on which drivers
+were built. If it fails to find the correct files you can add them manually to
+`vyos-build/packages/linux-kernel/build-linux-firmware.sh`:
+
+```bash
+ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin"
+```
+
+#### Building Out-Of-Tree Modules
+
+Building the kernel is one part, but now you also need to build the required
+out-of-tree modules so everything is lined up and the ABIs match. To do so,
+you can again take a look at `vyos-build/packages/linux-kernel/Jenkinsfile`
+to see all of the required modules and their selected versions. We will show
+you how to build all the current required modules.
+
+##### Accel-PPP
+
+First, clone the source code and check out the appropriate version by running:
+
+```none
+$ cd vyos-build/packages/linux-kernel
+$ git clone https://github.com/accel-ppp/accel-ppp.git
+```
+
+We again make use of a helper script and some patches to make the build work.
+Just run the following command:
+
+```none
+$ ./build-accel-ppp.sh
+I: Build Accel-PPP Debian package
+CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy):
+  The OLD behavior for policy CMP0003 will be removed from a future version
+  of CMake.
+
+  The cmake-policies(7) manual explains that the OLD behaviors of all
+  policies are deprecated and that a policy should be set to OLD only under
+  specific short-term circumstances.  Projects should be ported to the NEW
+  behavior and not rely on setting a policy to OLD.
+
+-- The C compiler identification is GNU 8.3.0
+
+...
+
+CPack: Create package using DEB
+CPack: Install projects
+CPack: - Run preinstall target for: accel-ppp
+CPack: - Install project: accel-ppp
+CPack: Create package
+CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated.
+```
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in `vyos-build/packages/linux-kernel` from which you can copy them
+to the `vyos-build/packages` folder for inclusion during the ISO build.
+
+##### Intel NIC
+
+The Intel NIC drivers do not come from a Git repository, instead we just fetch
+the tarballs from our mirror and compile them.
+
+Simply use our wrapper script to build all of the driver modules.
+
+```none
+./build-intel-drivers.sh
+  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                 Dload  Upload   Total   Spent    Left  Speed
+100  490k  100  490k    0     0   648k      0 --:--:-- --:--:-- --:--:--  648k
+I: Compile Kernel module for Intel ixgbe driver
+
+...
+
+I: Building Debian package vyos-intel-iavf
+Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
+Require just the needed backports instead, or 'backports/latest'.
+Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn}
+Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"}
+I: Cleanup iavf source
+```
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in `vyos-build/packages/linux-kernel` from which you can copy them
+to the `vyos-build/packages` folder for inclusion during the ISO build.
+
+##### Intel QAT
+
+The Intel QAT (Quick Assist Technology) drivers do not come from a Git
+repository, instead we just fetch the tarballs from 01.org, Intel's
+open-source website.
+
+Simply use our wrapper script to build all of the driver modules.
+
+```none
+$ ./build-intel-qat.sh
+  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                 Dload  Upload   Total   Spent    Left  Speed
+100 5065k  100 5065k    0     0  1157k      0  0:00:04  0:00:04 --:--:-- 1157k
+I: Compile Kernel module for Intel qat driver
+checking for a BSD-compatible install... /usr/bin/install -c
+checking whether build environment is sane... yes
+checking for a thread-safe mkdir -p... /bin/mkdir -p
+checking for gawk... gawk
+checking whether make sets $(MAKE)... yes
+
+...
+
+I: Building Debian package vyos-intel-qat
+Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
+Require just the needed backports instead, or 'backports/latest'.
+Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn}
+Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"}
+I: Cleanup qat source
+```
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in `vyos-build/packages/linux-kernel` from which you can copy them
+to the `vyos-build/packages` folder for inclusion during the ISO build.
+
+### Packages
+
+If you are brave enough to build yourself an ISO image containing any modified
+package from our GitHub organisation - this is the place to be.
+
+Any "modified" package may refer to an altered version of e.g. vyos-1x package
+that you would like to test before filing a pull request on GitHub.
+
+Building an ISO with any customized package is in no way different than
+building a regular (customized or not) ISO image. Simply place your modified
+`*.deb` package inside the `packages` folder within `vyos-build`. The build
+process will then pickup your custom package and integrate it into your ISO.
+
+### Troubleshooting
+
+Debian APT is not very verbose when it comes to errors. If your ISO build breaks
+for whatever reason and you suspect it's a problem with APT dependencies or
+installation you can add this small patch which increases the APT verbosity
+during ISO build.
+
+
+```diff
+diff --git i/scripts/live-build-config w/scripts/live-build-config
+index 1b3b454..3696e4e 100755
+--- i/scripts/live-build-config
++++ w/scripts/live-build-config
+@@ -57,7 +57,8 @@ lb config noauto \
+         --firmware-binary false \
+         --updates true \
+         --security true \
+-        --apt-options "--yes -oAcquire::Check-Valid-Until=false" \
++        --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \
++                             -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \
+         --apt-indices false
+         "${@}"
+ """
+```
+
+
+### Virtualization Platforms
+
+#### QEMU
+
+Run the following command after building the ISO image.
+
+```none
+$ make qemu
+```
+
+#### VMware
+
+Run the following command after building the QEMU image.
+
+```none
+$ make vmware
+```
+
+(build-packages)=
+
+## Packages
+
+VyOS itself comes with a bunch of packages that are specific to our system and
+thus cannot be found in any Debian mirror. Those packages can be found at the
+[VyOS GitHub project] in their source format can easily be compiled into
+a custom Debian (`*.deb`) package.
+
+The easiest way to compile your package is with the above mentioned
+{ref}`build_docker` container, it includes all required dependencies for
+all VyOS related packages.
+
+Assume we want to build the vyos-1x package on our own and modify it to our
+needs. We first need to clone the repository from GitHub.
+
+```none
+$ git clone https://github.com/vyos/vyos-1x
+```
+
+### Build
+
+Launch Docker container and build package
+
+```none
+# For VyOS 1.3 (equuleus, current)
+$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash
+
+# Change to source directory
+$ cd vyos-1x
+
+# Build DEB
+$ dpkg-buildpackage -uc -us -tc -b
+```
+
+After a minute or two you will find the generated DEB packages next to the
+vyos-1x source directory:
+
+```none
+# ls -al ../vyos-1x*.deb
+-rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug  3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb
+-rw-r--r-- 1 vyos_bld vyos_bld   3808 Aug  3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb
+```
+
+### Install
+
+To take your newly created package on a test drive you can simply SCP it to a
+running VyOS instance and install the new `*.deb` package over the current
+running one.
+
+Just install using the following commands:
+
+```none
+vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb
+(Reading database ... 58209 files and directories currently installed.)
+Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ...
+Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ...
+Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ...
+Processing triggers for rsyslog (8.1901.0-1) ...
+```
+
+You can also place the generated `*.deb` into your ISO build environment to
+include it in a custom iso, see {ref}`build_custom_packages` for more
+information.
+
+:::{warning}
+Any packages in the packages directory will be added to the iso
+during build, replacing the upstream ones. Make sure you delete them (both
+the source directories and built deb packages) if you want to build an iso
+from purely upstream packages.
+:::
+
+
+
+[docker]: https://www.docker.com
+[docker as non-root]: https://docs.docker.com/engine/install/linux-postinstall
+[on debian]: https://docs.docker.com/engine/install/debian/
+[repository]: https://github.com/vyos/vyos-build
+[vyos dockerhub organisation]: https://hub.docker.com/u/vyos
+[vyos github project]: https://github.com/vyos
diff --git a/docs/contributing/md-debugging.md b/docs/contributing/md-debugging.md
new file mode 100644
index 00000000..73b622c6
--- /dev/null
+++ b/docs/contributing/md-debugging.md
@@ -0,0 +1,191 @@
+(debugging)=
+
+# Debugging
+
+There are two flags available to aid in debugging configuration scripts.
+Since configuration loading issues will manifest during boot, the flags are
+passed as kernel boot parameters.
+
+## ISO image build
+
+When having trouble compiling your own ISO image or debugging Jenkins issues
+you can follow the steps at {ref}`iso_build_issues`.
+
+## System Startup
+
+The system startup can be debugged (like loading in the configuration
+file from `/config/config.boot`. This can be achieve by extending the
+Kernel command-line in the bootloader.
+
+### Kernel
+
+- `vyos-debug` - Adding the parameter to the linux boot line will produce
+  timing results for the execution of scripts during commit. If one is seeing
+  an unexpected delay during manual or boot commit, this may be useful in
+  identifying bottlenecks. The internal flag is `VYOS_DEBUG`, and is found
+  in [vyatta-cfg]. Output is directed to `/var/log/vyatta/cfg-stdout.log`.
+- `vyos-config-debug` - During development, coding errors can lead to a
+  commit failure on boot, possibly resulting in a failed initialization of the
+  CLI. In this circumstance, the kernel boot parameter `vyos-config-debug`
+  will ensure access to the system as user `vyos`, and will log a Python
+  stack trace to the file `/tmp/boot-config-trace`.
+  File `boot-config-trace` will generate only if config loaded with a failure
+  status.
+
+## Live System
+
+A number of flags can be set up to change the behaviour of VyOS at runtime.
+These flags can be toggled using either environment variables or creating
+files.
+
+For each feature, a file called `vyos.feature.debug` can be created to
+toggle the feature on. If a parameter is required it can be placed inside
+the file as its first line.
+
+The file can be placed in `/tmp` for one time debugging (as the file
+will be removed on reboot) or placed in '/config' to stay permanently.
+
+For example, `/tmp/vyos.ifconfig.debug` can be created to enable
+interface debugging.
+
+It is also possible to set up the debugging using environment variables.
+In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG.
+
+For example running, `export VYOS_IFCONFIG_DEBUG=""` on your vbash,
+will have the same effect as `touch /tmp/vyos.ifconfig.debug`.
+
+- `ifconfig` - Once set, all commands used, and their responses received
+  from the OS, will be presented on the screen for inspection.
+- `command` - Once set, all commands used, and their responses received
+  from the OS, will be presented on the screen for inspection.
+- `developer` - Should a command fail, instead of printing a message to the
+  user explaining how to report issues, the python interpreter will start a
+  PBD post-mortem session to allow the developer to debug the issue. As the
+  debugger will wait from input from the developer, it has the capacity to
+  prevent a router to boot and therefore should only be permanently set up
+  on production if you are ready to see the OS fail to boot.
+- `log` - In some rare cases, it may be useful to see what the OS is doing,
+  including during boot. This option sends all commands used by VyOS to a
+  file. The default file is `/tmp/full-log` but it can be changed.
+
+:::{note}
+In order to retrieve the debug output on the command-line you need to
+disable `vyos-configd` in addition. This can be run either one-time by
+calling `sudo systemctl stop vyos-configd` or make this reboot-safe by
+calling `sudo systemctl disable vyos-configd`.
+:::
+
+### FRR
+
+Recent versions use the `vyos.frr` framework. The Python class is located
+inside our `vyos-1x:python/vyos/frr.py`. It comes with an embedded debugging/
+(print style) debugger as vyos.ifconfig does.
+
+To enable debugging just run: `$ touch /tmp/vyos.frr.debug`
+
+### Debugging Python Code with PDB
+
+Sometimes it might be useful to debug Python code interactively on the live
+system rather than a IDE. This can be achieved using pdb.
+
+Let us assume you want to debug a Python script that is called by an op-mode
+command. After you found the script by looking up the op-mode-defitions you
+can edit the script in the live system using e.g. vi:
+`vi /usr/libexec/vyos/op_mode/show_xyz.py`
+
+Insert the following statement right before the section where you want to
+investigate a problem (e.g. a statement you see in a backtrace):
+`import pdb; pdb.set_trace()`
+Optionally you can surrounded this statement by an `if` which only triggers
+under the condition you are interested in.
+
+Once you run `show xyz` and your condition is triggered you should be dropped
+into the python debugger:
+
+```none
+> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process()
+-> rule_type = rule.get('type', '')
+(Pdb)
+```
+
+You can type `help` to get an overview of the available commands, and
+`help command` to get more information on each command.
+
+Useful commands are:
+
+- examine variables using `pp(var)`
+- continue execution using `cont`
+- get a backtrace using `bt`
+
+### Config Migration Scripts
+
+When writing a new configuration migrator it may happen that you see an error
+when you try to invoke it manually on a development system. This error will
+look like:
+
+```none
+vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot
+Traceback (most recent call last):
+  File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module>
+    config = ConfigTree(config_file)
+  File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__
+    raise ValueError("Failed to parse config: {0}".format(msg))
+ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax.
+```
+
+The reason is that the configuration migration backend is rewritten and uses
+a new form of "magic string" which is applied on demand when real config
+migration is run on boot. When running individual migrators for testing,
+you need to convert the "magic string" on your own by:
+
+```none
+vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot
+```
+
+### Configuration Error on System Boot
+
+Being brave and running the latest rolling releases will sometimes trigger
+bugs due to corner cases we missed in our design. Those bugs should be filed
+via [Phabricator] but you can help us to narrow down the issue. Login to your
+VyOS system and change into configuration mode by typing `configure`. Now
+re-load your boot configuration by simply typing `load` followed by return.
+
+You should now see a Python backtrace which will help us to handle the issue,
+please attach it to the [Phabricator] task.
+
+### Boot Timing
+
+During the migration and extensive rewrite of functionality from Perl into
+Python a significant increase in the overall system boottime was noticed. The
+system boot time can be analysed and a graph can be generated in the end which
+shows in detail who called whom during the system startup phase.
+
+This is done by utilizing the `systemd-bootchart` package which is now
+installed by default on the VyOS 1.3 (equuleus) branch. The configuration is
+also versioned so we get comparable results. `systemd-bootchart` is configured
+using this file: [bootchart.conf]
+
+To enable boot time graphing change the Kernel commandline and add the following
+string: `init=/usr/lib/systemd/systemd-bootchart`
+
+This can also be done permanently by changing `/boot/grub/grub.cfg`.
+
+## Priorities
+
+VyOS CLI is all about priorities. Every CLI node has a corresponding
+`node.def` file and possibly an attached script that is executed when the
+node is present. Nodes can have a priority, and on system bootup - or any
+other `commit` to the config all scripts are executed from lowest to highest
+priority. This is good as this gives a deterministic behavior.
+
+To debug issues in priorities or to see what's going on in the background
+you can use the `/opt/vyatta/sbin/priority.pl` script which lists to you
+the execution order of the scripts.
+
+
+```{include} /_include/common-references.txt
+```
+
+
+[bootchart.conf]: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf
+[vyatta-cfg]: https://github.com/vyos/vyatta-cfg
diff --git a/docs/contributing/md-development.md b/docs/contributing/md-development.md
new file mode 100644
index 00000000..dcfe257a
--- /dev/null
+++ b/docs/contributing/md-development.md
@@ -0,0 +1,694 @@
+(development)=
+
+# Development
+
+All VyOS source code is hosted on GitHub under the VyOS organization which can
+be found here: <https://github.com/vyos>
+
+Our code is split into several modules. VyOS is composed of multiple individual
+packages, some of them are forks of upstream packages and are periodically
+synced with upstream, so keeping the whole source under a single repository
+would be very inconvenient and slow. There is now an ongoing effort to
+consolidate all VyOS-specific framework/config packages into vyos-1x package,
+but the basic structure is going to stay the same, just with fewer and fewer
+packages while the base code is rewritten from Perl/BASH into Python using and
+XML based interface definition for the CLI.
+
+The repository that contains all the ISO build scripts is:
+<https://github.com/vyos/vyos-build>
+
+The README.md file will guide you to use the this top level repository.
+
+## Submit a Patch
+
+Patches are always more than welcome. To have a clean and easy to maintain
+repository we have some guidelines when working with Git. A clean repository
+eases the automatic generation of a changelog file.
+
+A good approach for writing commit messages is actually to have a look at the
+file(s) history by invoking `git log path/to/file.txt`.
+
+(prepare_commit)=
+
+### Prepare patch/commit
+
+In a big system, such as VyOS, that is comprised of multiple components, it's
+impossible to keep track of all the changes and bugs/feature requests in one's
+head. We use a bugtracker known as [Phabricator] for it ("issue tracker" would
+be a better term, but this one stuck).
+
+The information is used in three ways:
+
+- Keep track of the progress (what we've already done in this branch and what
+  we still need to do).
+- Prepare release notes for upcoming releases
+- Help future maintainers of VyOS (it could be you!) to find out why certain
+  things have been changed in the codebase or why certain features have been
+  added
+
+To make this approach work, every change must be associated with a task number
+(prefixed with **T**) and a component. If there is no bug report/feature request
+for the changes you are going to make, you have to create a [Phabricator] task
+first. Once there is an entry in [Phabricator], you should reference its id in
+your commit message, as shown below:
+
+- `ddclient: T1030: auto create runtime directories`
+- `Jenkins: add current Git commit ID to build description`
+
+If there is no [Phabricator] reference in the commits of your pull request, we
+have to ask you to amend the commit message. Otherwise we will have to reject
+it.
+
+#### Writing good commit messages
+
+The format should be and is inspired by: <https://git-scm.com/book/ch5-2.html>
+It is also worth reading <https://chris.beams.io/posts/git-commit/>
+
+- A single, short, summary of the commit (recommended 50 characters or less,
+  not exceeding 80 characters) containing a prefix of the changed component
+  and the corresponding [Phabricator] reference e.g. `snmp: T1111:` or
+  `ethernet: T2222:` - multiple components could be concatenated as in
+  `snmp: ethernet: T3333`
+
+- In some contexts, the first line is treated as the subject of an email and
+  the rest of the text as the body. The blank line separating the summary from
+  the body is critical (unless you omit the body entirely); tools like rebase
+  can get confused if you run the two together.
+
+- Followed by a message which describes all the details like:
+
+  - What/why/how something has been changed, makes everyone's life easier when
+    working with `git bisect`
+  - All text of the commit message should be wrapped at 72 characters if
+    possible which makes reading commit logs easier with `git log` on a
+    standard terminal (which happens to be 80x25)
+  - If applicable a reference to a previous commit should be made linking
+    those commits nicely when browsing the history: `After commit abcd12ef
+    ("snmp: this is a headline") a Python import statement is missing,
+    throwing the following exception: ABCDEF`
+
+- Always use the `-x` option to the `git cherry-pick` command when back or
+  forward porting an individual commit. This automatically appends the line:
+  `(cherry picked from commit <ID>)` to the original authors commit message
+  making it easier when bisecting problems.
+
+- Every change set must be consistent (self containing)! Do not fix multiple
+  bugs in a single commit. If you already worked on multiple fixes in the same
+  file use `git add --patch` to only add the parts related to the one issue
+  into your upcoming commit.
+
+Limits:
+
+- We only accept bugfixes in packages other than <https://github.com/vyos/vyos-1x>
+  as no new functionality should use the old style templates (`node.def` and
+  Perl/BASH code. Use the new style XML/Python interface instead.
+
+Please submit your patches using the well-known GitHub pull-request against our
+repositories found in the VyOS GitHub organisation at <https://github.com/vyos>
+
+### Determinine source package
+
+Suppose you want to make a change in the webproxy script but yet you do not know
+which of the many VyOS packages ship this file. You can determine the VyOS
+package name in question by using Debian's `dpkg -S` command of your running
+VyOS installation.
+
+```none
+vyos@vyos:~ dpkg -S /opt/vyatta/sbin/vyatta-update-webproxy.pl
+vyatta-webproxy: /opt/vyatta/sbin/vyatta-update-webproxy.pl
+```
+
+This means the file in question (`/opt/vyatta/sbin/vyatta-update-webproxy.pl`)
+is located in the `vyatta-webproxy` package which can be found here:
+<https://github.com/vyos/vyatta-webproxy>
+
+### Fork Repository and submit Patch
+
+Forking the repository and submitting a GitHub pull-request is the preferred
+way of submitting your changes to VyOS. You can fork any VyOS repository to your
+very own GitHub account by just appending `/fork` to any repository's URL on
+GitHub. To e.g. fork the `vyos-1x` repository, open the following URL in your
+favourite browser: <https://github.com/vyos/vyos-1x/fork>
+
+You then can proceed with cloning your fork or add a new remote to your local
+repository:
+
+- Clone: `git clone https://github.com/<user>/vyos-1x.git`
+- Fork: `git remote add myfork https://github.com/<user>/vyos-1x.git`
+
+In order to record you as the author of the fix please identify yourself to Git
+by setting up your name and email. This can be done local for this one and only
+repository `git config` or globally using `git config --global`.
+
+```none
+git config --global user.name "J. Random Hacker"
+git config --global user.email "jrhacker@example.net"
+```
+
+Make your changes and save them. Do the following for all changes files to
+record them in your created Git commit:
+
+- Add file to Git index using `git add myfile`, or for a whole directory:
+  `git add somedir/*`
+- Commit the changes by calling `git commit`. Please use a meaningful commit
+  headline (read above) and don't forget to reference the [Phabricator] ID.
+- Submit the patch `git push` and create the GitHub pull-request.
+
+### Attach patch to Phabricator task
+
+Follow the above steps on how to "Fork repository to submit a Patch". Instead
+of uploading "pushing" your changes to GitHub you can export the patches/
+commits and send it to <mailto:maintainers@vyos.net> or attach it directly to the bug
+(preferred over email)
+
+- Export last commit to patch file: `git format-patch` or export the last two
+  commits into its appropriate patch files: `git format-patch -2`
+
+## Coding Guidelines
+
+Like any other project we have some small guidelines about our source code, too.
+The rules we have are not there to punish you - the rules are in place to help
+us all. By having a consistent coding style it becomes very easy for new
+and also longtime contributors to navigate through the sources and all the
+implied logic of any one source file..
+
+Python 3 **shall** be used. How long can we keep Python 2 alive anyway? No
+considerations for Python 2 compatibility **should** be taken at any time.
+
+### Formatting
+
+- Python: Tabs **shall not** be used. Every indentation level should be 4 spaces
+- XML: Tabs **shall not** be used. Every indentation level should be 2 spaces
+
+:::{note}
+There are extensions to e.g. VIM (xmllint) which will help you to get
+your indention levels correct. Add to following to your .vimrc file:
+`au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\
+2>/dev/null` now you can call the linter using `gg=G` in command mode.
+:::
+
+#### Text generation
+
+Template processor **should** be used for generating config files. Built-in
+string formatting **may** be used for simple line-oriented formats where every
+line is self-contained, such as iptables rules. Template processor **must** be
+used for structured, multi-line formats such as those used by ISC DHCPd.
+
+The default template processor for VyOS code is [Jinja2].
+
+### Summary
+
+When modifying the source code, remember these rules of the legacy elimination
+campaign:
+
+- No new features in Perl
+- No old style command definitions
+- No code incompatible with Python3
+
+## Python
+
+The switch to the Python programming language for new code is not merely a
+change of the language, but a chance to rethink and improve the programming
+approach.
+
+Let's face it: VyOS is full of spaghetti code where logic for reading the VyOS
+config, generating daemon configs, and restarting processes is all mixed up.
+
+Python (or any other language, for that matter) does not provide automatic
+protection from bad design, so we need to also devise design guidelines and
+follow them to keep the system extensible and maintainable.
+
+But we are here to assist you and want to guide you through how you can become
+a good VyOS contributor. The rules we have are not there to punish you - the
+rules are in place to help us all. What does it mean? By having a consistent
+coding style it becomes very easy for new contributors and also longtime
+contributors to navigate through the sources and all the implied logic of
+the spaghetti code.
+
+Please use the following template as good starting point when developing new
+modules or even rewrite a whole bunch of code in the new style XML/Python
+interface.
+
+### Configuration Script Structure and Behaviour
+
+Your configuration script or operation mode script which is also written in
+Python3 should have a line break on 80 characters. This seems to be a bit odd
+nowadays but as some people also work remotely or program using vi(m) this is
+a fair good standard which I hope we can rely on.
+
+In addition this also helps when browsing the GitHub codebase on a mobile
+device if you happen to be a crazy scientist.
+
+```python
+#!/usr/bin/env python3
+#
+# Copyright (C) 2020 VyOS maintainers and contributors
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 or later as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+
+from vyos.config import Config
+from vyos import ConfigError
+
+def get_config():
+    if config:
+        conf = config
+    else:
+        conf = Config()
+
+    # Base path to CLI nodes
+    base = ['...', '...']
+    # Convert the VyOS config to an abstract internal representation
+    config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True)
+    return config_data
+
+def verify(config):
+    # Verify that configuration is valid
+    if invalid:
+        raise ConfigError("Descriptive message")
+    return True
+
+def generate(config):
+    # Generate daemon configs
+    pass
+
+def apply(config):
+    # Apply the generated configs to the live system
+    pass
+
+try:
+    c = get_config()
+    verify(c)
+    generate(c)
+    apply(c)
+except ConfigError as e:
+    print(e)
+    sys.exit(1)
+```
+
+The `get_config()` function must convert the VyOS config to an abstract,
+internal representation. No other function is allowed to call the `vyos.config.
+Config` object method directly. The rationale for it is that when config reads
+are mixed with other logic, it's very hard to change the config syntax since
+you need to weed out every occurrence of the old syntax. If syntax-specific
+code is confined to a single function, the rest of the code can be left
+untouched as long as the internal representation remains compatible.
+
+Another advantage is testability of the code. Mocking the entire config
+subsystem is hard, while constructing an internal representation by hand is
+way simpler.
+
+The `verify()` function takes your internal representation of the config and
+checks if it's valid, otherwise it must raise `ConfigError` with an error
+message that describes the problem and possibly suggests how to fix it. It must
+not make any changes to the system. The rationale for it is again testability
+and, in the future when the config backend is ready and every script is
+rewritten in this fashion, ability to execute commit dry run ("commit test"
+like in JunOS) and abort commit before making any changes to the system if an
+error is found in any component.
+
+The `generate()` function generates config files for system components.
+
+The `apply()` function applies the generated configuration to the live
+system. It should use non-disruptive reload whenever possible. It may execute
+disruptive operations such as daemon process restart if a particular component
+does not support non-disruptive reload, or when the expected service degradation
+is minimal (for example, in case of auxiliary services such as LLDPd). In case
+of high impact services such as VPN daemon and routing protocols, when non-
+disruptive reload is supported for some but not all types of configuration
+changes, scripts authors should make effort to determine if a configuration
+change can be done in a non-disruptive way and only resort to disruptive restart
+if it cannot be avoided.
+
+Unless absolutely necessary, configuration scripts should not modify the active
+configuration of system components directly. Whenever at all possible, scripts
+should generate a configuration file or files that can be applied with a single
+command such as reloading a service through systemd init. Inserting statements
+one by one is particularly discouraged, for example, when configuring netfilter
+rules, saving them to a file and loading it with iptables-restore should always
+be preferred to executing iptables directly.
+
+The `apply()` and `generate()` functions may `raise ConfigError` if, for
+example, the daemon failed to start with the updated config. It shouldn't be a
+substitute for proper config checking in the `verify()` function. All
+reasonable effort should be made to verify that generated configuration is
+valid and will be accepted by the daemon, including, when necessary, cross-
+checks with other VyOS configuration subtrees.
+
+Exceptions, including `VyOSError` (which is raised by `vyos.config.Config`
+on improper config operations, such as trying to use `list_nodes()` on a
+non-tag node) should not be silenced or caught and re-raised as config error.
+Sure this will not look pretty on user's screen, but it will make way better
+bug reports, and help users (and most VyOS users are IT professionals) do their
+own debugging as well.
+
+For easy orientation we suggest you take a look on the `ntp.py` or
+`interfaces-bonding.py` (for tag nodes) implementation. Both files can be
+found in the [vyos-1x] repository.
+
+## XML (used for CLI definitions)
+
+The bash (or better vbash) completion in VyOS is defined in *templates*.
+Templates are text files (called `node.def`) stored in a directory tree. The
+directory names define the command names, and template files define the command
+behaviour. Before VyOS 1.2 (crux) this files were created by hand. After a
+complex redesign [process] the new style template are automatically generated
+from a XML input file.
+
+XML interface definitions for VyOS come with a RelaxNG schema and are located
+in the [vyos-1x] module. This schema is a slightly modified schema from [VyConf]
+alias VyOS 2.0 So VyOS 1.2.x interface definitions will be reusable in Nextgen
+VyOS Versions with very minimal changes.
+
+The great thing about schemas is not only that people can know the complete
+grammar for certain, but also that it can be automatically verified. The
+`scripts/build-command-templates` script that converts the XML definitions to
+old style templates also verifies them against the schema, so a bad definition
+will cause the package build to fail. I do agree that the format is verbose, but
+there is no other format now that would allow this. Besides, a specialized XML
+editor can alleviate the issue with verbosity.
+
+Example:
+
+```xml
+<?xml version="1.0"?>
+<!-- Cron configuration -->
+<interfaceDefinition>
+  <node name="system">
+    <children>
+      <node name="task-scheduler">
+        <properties>
+          <help>Task scheduler settings</help>
+        </properties>
+        <children>
+          <tagNode name="task" owner="${vyos_conf_scripts_dir}/task_scheduler.py">
+            <properties>
+              <help>Scheduled task</help>
+              <valueHelp>
+                <format>&lt;string&gt;</format>
+                <description>Task name</description>
+              </valueHelp>
+              <priority>999</priority>
+            </properties>
+            <children>
+              <leafNode name="crontab-spec">
+                <properties>
+                  <help>UNIX crontab time specification string</help>
+                </properties>
+              </leafNode>
+              <leafNode name="interval">
+                <properties>
+                  <help>Execution interval</help>
+                  <valueHelp>
+                    <format>&lt;minutes&gt;</format>
+                    <description>Execution interval in minutes</description>
+                  </valueHelp>
+                  <valueHelp>
+                    <format>&lt;minutes&gt;m</format>
+                    <description>Execution interval in minutes</description>
+                  </valueHelp>
+                  <valueHelp>
+                    <format>&lt;hours&gt;h</format>
+                    <description>Execution interval in hours</description>
+                  </valueHelp>
+                  <valueHelp>
+                    <format>&lt;days&gt;d</format>
+                    <description>Execution interval in days</description>
+                  </valueHelp>
+                  <constraint>
+                    <regex>[1-9]([0-9]*)([mhd]{0,1})</regex>
+                  </constraint>
+                </properties>
+              </leafNode>
+              <node name="executable">
+                <properties>
+                  <help>Executable path and arguments</help>
+                </properties>
+                <children>
+                  <leafNode name="path">
+                    <properties>
+                      <help>Path to executable</help>
+                    </properties>
+                  </leafNode>
+                  <leafNode name="arguments">
+                    <properties>
+                      <help>Arguments passed to the executable</help>
+                    </properties>
+                  </leafNode>
+                </children>
+              </node>
+            </children>
+          </tagNode>
+        </children>
+      </node>
+    </children>
+  </node>
+</interfaceDefinition>
+```
+
+Command definitions are purely declarative, and cannot contain any logic. All
+logic for generating config files for target applications, restarting services
+and so on is implemented in configuration scripts instead.
+
+### GNU Preprocessor
+
+XML interface definition files use the `xml.in` file extension which was
+implemented in {vytask}`T1843`. XML interface definitions tend to have a lot of
+duplicated code in areas such as:
+
+- VIF (incl. VIF-S/VIF-C)
+- Address
+- Description
+- Enabled/Disabled
+
+Instead of supplying all those XML nodes multiple times there are now include
+files with predefined features. Brief overview:
+
+- [IPv4, IPv6 and DHCP(v6)] address assignment
+- [IPv4, IPv6] address assignment
+- [VLAN (VIF)] definition
+- [MAC address] assignment
+
+All interface definition XML input files (.in suffix) will be sent to the GCC
+preprocess and the output is stored in the `build/interface-definitions`
+folder. The previously mentioned `scripts/build-command-templates` script
+operates on the `build/interface-definitions` folder to generate all required
+CLI nodes.
+
+```none
+$ make interface_definitions
+install -d -m 0755 build/interface-definitions
+install -d -m 0755 build/op-mode-definitions
+Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in
+Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in
+Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in
+Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in
+Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in
+Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in
+[...]
+```
+
+### Guidelines
+
+#### Use of numbers
+
+Use of numbers in command names **should** be avoided unless a number is a
+part of a protocol name or similar. Thus, `protocols ospfv3` is perfectly
+fine, but something like `server-1` is questionable at best.
+
+#### Help String
+
+To ensure uniform look and feel, and improve readability, we should follow a
+set of guidelines consistently.
+
+##### Capitalization and punctuation
+
+The first word of every help string **must** be capitalized. There **must not**
+be a period at the end of help strings.
+
+Rationale: this seems to be the unwritten standard in network device CLIs, and
+a good aesthetic compromise.
+
+Examples:
+
+- Good: "Frobnication algorithm"
+- Bad: "frobnication algorithm"
+- Bad: "Frobnication algorithm."
+- Horrible: "frobnication algorithm."
+
+##### Use of abbreviations and acronyms
+
+Abbreviations and acronyms **must** be capitalized.
+
+Examples:
+
+- Good: "TCP connection timeout"
+- Bad: "tcp connection timeout"
+- Horrible: "Tcp connection timeout"
+
+Acronyms also **must** be capitalized to visually distinguish them from normal
+words:
+
+Examples:
+
+- Good: RADIUS (as in remote authentication for dial-in user services)
+- Bad: radius (unless it's about the distance between a center of a circle and
+  any of its points)
+
+Some abbreviations are traditionally written in mixed case. Generally, if it
+contains words "over" or "version", the letter **should** be lowercase. If
+there's an accepted spelling (especially if defined by an RFC or another
+standard), it **must** be followed.
+
+Examples:
+
+- Good: PPPoE, IPsec
+- Bad: PPPOE, IPSEC
+- Bad: pppoe, ipsec
+
+##### Use of verbs
+
+Verbs **should** be avoided. If a verb can be omitted, omit it.
+
+Examples:
+
+- Good: "TCP connection timeout"
+- Bad: "Set TCP connection timeout"
+
+If a verb is essential, keep it. For example, in the help text of `set system
+ipv6 disable-forwarding`, "Disable IPv6 forwarding on all interfaces" is a
+perfectly justified wording.
+
+##### Prefer infinitives
+
+Verbs, when they are necessary, **should** be in their infinitive form.
+
+Examples:
+
+- Good: "Disable IPv6 forwarding"
+- Bad: "Disables IPv6 forwarding"
+
+### Migrating old CLI
+
+```{eval-rst}
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 1
+
+   * - Old concept/syntax
+     - New syntax
+     - Notes
+   * - mynode/node.def
+     - <node name="mynode"> </node>
+     - Leaf nodes (nodes with values) use <leafNode> tag instead
+   * - mynode/node.tag , tag:
+     - <tagNode name="mynode> </node>
+     -
+   * - help: My node
+     - <properties> <help>My node</help>
+     -
+   * - val_help: <format>; some string
+     - <properties> <valueHelp> <format> format </format> <description> some
+       string </description>
+     - Do not add angle brackets around the format, they will be inserted
+       automatically
+   * - syntax:expression: pattern
+     - <properties> <constraint> <regex> ...
+     - <constraintErrorMessage> will be displayed on failure
+   * - syntax:expression: $VAR(@) in "foo", "bar", "baz"
+     - None
+     - Use regex
+   * - syntax:expression: exec ...
+     - <properties> <constraint> <validator> <name ="foo" argument="bar">
+     - "${vyos_libexecdir}/validators/foo bar $VAR(@)" will be executed,
+       <constraintErrorMessage> will be displayed on failure
+   * - syntax:expression: (arithmetic expression)
+     - None
+     - External arithmetic validator may be added if there's demand, complex
+       validation is better left to commit-time scripts
+   * - priority: 999
+     - <properties> <priority>999</priority>
+     - Please leave a comment explaining why the priority was chosen
+       (e.g. "after interfaces are configured")
+   * - multi:
+     - <properties> <multi/>
+     - Only applicable to leaf nodes
+   * - allowed: echo foo bar
+     - <properties> <completionHelp> <list> foo bar </list>
+     -
+   * - allowed: cli-shell-api listNodes vpn ipsec esp-group
+     - <properties> <completionHelp> <path> vpn ipsec esp-group </path> ...
+     -
+   * - allowed: /path/to/script
+     - <properties> <completionHelp> <script> /path/to/script </script> ...
+     -
+   * - default:
+     - None
+     - Move default values to scripts
+   * - commit:expression:
+     - None
+     - All commit time checks should be in the verify() function of the script
+   * - begin:/create:/delete:
+     - None
+     - All logic should be in the scripts
+```
+
+## C++ Backend Code
+
+The CLI parser used in VyOS is a mix of bash, bash-completion helper and the
+C++ backend library \[vyatta-cfg\](<https://github.com/vyos/vyatta-cfg>). This
+section is a reference of common CLI commands and the respective entry point
+in the C/C++ code.
+
+- `set`
+
+  - <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352>
+  - <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549>
+
+- `commit`
+
+  - <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252>
+
+## Continuous Integration
+
+VyOS makes use of [Jenkins] as our Continuous Integration (CI) service. Our
+[VyOS CI] server is publicly accessible here: <https://ci.vyos.net>. You can get
+a brief overview of all required components shipped in a VyOS ISO.
+
+To build our modules we utilize a CI/CD Pipeline script. Each and every VyOS
+component comes with it's own `Jenkinsfile` which is (more or less) a copy.
+The Pipeline utilizes the Docker container from the {ref}`build_iso` section -
+but instead of building it from source on every run, we rather always fetch a
+fresh copy (if needed) from [Dockerhub].
+
+Each module is build on demand if a new commit on the branch in question is
+found. After a successful run the resulting Debian Package(s) will be deployed
+to our Debian repository which is used during build time. It is located here:
+<http://dev.packages.vyos.net/repositories/>.
+
+
+```{include} /_include/common-references.txt
+```
+
+
+[dockerhub]: https://hub.docker.com/u/vyos/
+[ipv4, ipv6]: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i
+[ipv4, ipv6 and dhcp(v6)]: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i
+[jenkins]: https://jenkins.io/
+[jinja2]: https://jinja.palletsprojects.com/
+[mac address]: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/mac.xml.i
+[process]: https://blog.vyos.io/vyos-development-digest-10
+[vlan (vif)]: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/vif.xml.i
+[vyconf]: https://github.com/vyos/vyconf/tree/master/data/schemata
+[vyos-1x]: https://github.com/vyos/vyos-1x/tree/current/schema
diff --git a/docs/contributing/md-issues-features.md b/docs/contributing/md-issues-features.md
new file mode 100644
index 00000000..6a748f36
--- /dev/null
+++ b/docs/contributing/md-issues-features.md
@@ -0,0 +1,73 @@
+(issues_features)=
+
+# Issues/Feature requests
+
+(bug_report)=
+
+## Bug Report/Issue
+
+Issues or bugs are found in any software project. VyOS is not an exception.
+
+All issues should be reported to the developers. This lets the developers know
+what is not working properly. Without this sort of feedback every developer
+will believe that everything is working correctly.
+
+### I have found a bug, what should I do?
+
+When you believe you have found a bug, it is always a good idea to verify the
+issue prior to opening a bug request.
+
+- Consult the [documentation] to ensure that you have configured your system
+  correctly
+- Get community support via [Slack] or our [Forum]
+
+### Ensure the problem is reproducible
+
+When you are able to verify that it is actually a bug, spend some time to
+document how to reproduce the issue. This documentation can be invaluable.
+
+When you wish to have a developer fix a bug that you found, helping them
+reproduce the issue is beneficial to everyone. Be sure to include information
+about the hardware you are using, commands that you were running, any other
+activities that you may have been doing at the time. This additional
+information can be very useful.
+
+- What were you attempting to achieve?
+- What was the configuration prior to the change?
+- What commands did you use? Use e.g. `run show configuration commands`
+
+### Include output
+
+The output you get when you find a bug can provide lots of information. If you
+get an error message on the screen, copy it exactly. Having the exact message
+can provide detail that the developers can use. Like wise if you have any log
+messages that also are from the time of the issue, include those. They may
+also contain information that is helpful for the development team.
+
+### Report a Bug
+
+In order to open up a bug-report/feature request you need to create yourself
+an account on VyOS [Phabricator]. On the left side of the specific project (VyOS
+1.2 or VyOS 1.3) you will find quick-links for opening a bug-report/feature
+request.
+
+- Provide as much information as you can
+- Which version of VyOS are you using? `run show version`
+- How can we reproduce this Bug?
+
+(feature-request)=
+
+## Feature Request
+
+You have an idea of how to make VyOS better or you are in need of a specific
+feature which all users of VyOS would benefit from? To send a feature request
+please search [Phabricator] if there is already a request pending. You can
+enhance it or if you don't find one, create a new one by use the quick link in
+the left side under the specific project.
+
+```{include} /_include/common-references.txt
+```
+
+[documentation]: https://docs.vyos.io
+[forum]: https://forum.vyos.io
+[slack]: https://slack.vyos.io
diff --git a/docs/contributing/md-testing.md b/docs/contributing/md-testing.md
new file mode 100644
index 00000000..b2929936
--- /dev/null
+++ b/docs/contributing/md-testing.md
@@ -0,0 +1,214 @@
+(testing)=
+
+# Testing
+
+One of the major advantages introduced in VyOS 1.3 is an automated test
+framework. When assembling an ISO image multiple things can go wrong badly and
+publishing a faulty ISO makes no sense. The user is disappointed by the quality
+of the image and the developers get flodded with bug reports over and over
+again.
+
+As the VyOS documentation is not only for users but also for the developers -
+and we keep no secret documentation - this section describes how the automated
+testing works.
+
+## Jenkins CI
+
+Our [VyOS CI] system is based on Jenkins and builds all our required packages
+for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build
+Job which builds and tests the VyOS ISO image which is published after a
+successful test drive.
+
+We differentiate in two independent tests, which are both run in parallel by
+two separate QEmu instances which are launched via `make test` and `make
+testc` from within the [vyos-build] repository.
+
+## Smoketests
+
+Smoketests executes predefined VyOS CLI commands and checks if the desired
+daemon/service configuration is rendert - that is how to put it "short".
+
+When and ISO image is assembled by the [VyOS CI], the `BUILD_SMOKETEST`
+parameter is enabled by default, which will extend the ISO configuration line
+with the following packages:
+
+```python
+def CUSTOM_PACKAGES = ''
+  if (params.BUILD_SMOKETESTS)
+    CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest'
+```
+
+So if you plan to build your own custom ISO image and want to make use of our
+smoketests, ensure that you have the `vyos-1x-smoketest` package installed.
+
+The `make test` command from the [vyos-build] repository will launch a new
+QEmu instance and the ISO image is first installed to the virtual harddisk.
+
+After its first boot into the newly installed system the main Smoketest script
+is executed, it can be found here: `/usr/bin/vyos-smoketest`
+
+The script only searches for executable "test-cases" under
+`/usr/libexec/vyos/tests/smoke/cli/` and executes them one by one.
+
+:::{note}
+As Smoketests will alter the system configuration and you are logged
+in remote you may loose your connection to the system.
+:::
+
+### Manual Smoketest Run
+
+On the other hand - as each test is contain in its own file - one can always
+execute a single Smoketest by hand by simply running the Python test scripts.
+
+Example:
+
+```none
+vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
+test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok
+test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok
+test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok
+test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok
+test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok
+test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok
+test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok
+test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok
+test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok
+test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok
+test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok
+test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok
+test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok
+
+----------------------------------------------------------------------
+Ran 13 tests in 348.191s
+
+OK
+```
+
+### Interface based tests
+
+Our smoketests not only test daemons and serives, but also check if what we
+configure for an interface works. Thus there is a common base classed named:
+`base_interfaces_test.py` which holds all the common code that an interface
+supports and is tested.
+
+Those common tests consists out of:
+
+- Add one or more IP addresses
+
+- DHCP client and DHCPv6 prefix delegation
+
+- MTU size
+
+- IP and IPv6 options
+
+- Port description
+
+- Port disable
+
+- VLANs (QinQ and regular 802.1q)
+
+- ...
+
+:::{note}
+When you are working on interface configuration and you also want to
+test if the Smoketests pass you would normally loose the remote SSH connection
+to your {abbr}`DUT (Device Under Test)`. To handle this issue, some of the
+interface based tests can be called with an environment variable beforehand
+to limit the number of interfaces used in the test. By default all interface
+e.g. all Ethernet interfaces are used.
+:::
+
+```none
+vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py
+test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok
+test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok
+test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok
+test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok
+test_bonding_min_links (__main__.BondingInterfaceTest) ... ok
+test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok
+test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok
+test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok
+test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok
+test_interface_description (__main__.BondingInterfaceTest) ... ok
+test_interface_disable (__main__.BondingInterfaceTest) ... ok
+test_interface_ip_options (__main__.BondingInterfaceTest) ... ok
+test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok
+test_interface_mtu (__main__.BondingInterfaceTest) ... ok
+test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok
+test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok
+test_span_mirror (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok
+test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok
+test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok
+test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok
+
+----------------------------------------------------------------------
+Ran 23 tests in 244.694s
+
+OK
+```
+
+This will limit the `bond` interface test to only make use of `eth1` and `eth2`
+as member ports.
+
+## Config Load Tests
+
+The other part of our tests are called "config load tests". The config load tests
+will load - one after another - arbitrary configuration files to test if the
+configuration migration scripts work as designed and that a given set of
+functionality still can be loaded with a fresh VyOS ISO image.
+
+The configurations are all derived from production systems and can not only act
+as a testcase but also as reference if one wants to enable a certain feature.
+The configurations can be found here:
+<https://github.com/vyos/vyos-1x/tree/current/smoketest/configs>
+
+The entire test is controlled by the main wrapper script `/usr/bin/vyos-configtest`
+which behaves in the same way as the main smoketest script. It scans the folder
+for potential configuration files and issues a `load` command one after another.
+
+### Manual config load test
+
+One is not bound to load all configurations one after another but can also load
+individual test configurations on his own.
+
+```none
+vyos@vyos:~$ configure
+load[edit]
+
+vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small
+Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small'
+Load complete. Use 'commit' to make changes effective.
+[edit]
+vyos@vyos# compare
+[edit interfaces ethernet eth0]
+-hw-id 00:50:56:bf:c5:6d
+[edit interfaces ethernet eth1]
++duplex auto
+-hw-id 00:50:56:b3:38:c5
++speed auto
+[edit interfaces]
+-ethernet eth2 {
+-    hw-id 00:50:56:b3:9c:1d
+-}
+-vti vti1 {
+-    address 192.0.2.1/30
+-}
+...
+
+vyos@vyos# commit
+vyos@vyos#
+```
+
+:::{note}
+Some of the configurations have preconditions which need to be met.
+Those most likely include generation of crypographic keys before the config
+can be applied - you will get a commit error otherwise. If you are interested
+how those preconditions are fulfilled check the [vyos-build] repository and
+the `scripts/check-qemu-install` file.
+:::
+
+```{include} /_include/common-references.txt
+```
diff --git a/docs/contributing/md-upstream-packages.md b/docs/contributing/md-upstream-packages.md
new file mode 100644
index 00000000..6e7a7fb6
--- /dev/null
+++ b/docs/contributing/md-upstream-packages.md
@@ -0,0 +1,79 @@
+(upstream-packages)=
+
+# Upstream packages
+
+Many base system packages are pulled straight from Debian's main and contrib
+repositories, but there are exceptions.
+
+This chapter lists those exceptions and gives you a brief overview what we
+have done on those packages. If you only want to build yourself a fresh ISO
+you can completely skip this chapter. It may become interesting once you have
+a VyOS deep dive.
+
+## vyos-netplug
+
+Due to issues in the upstream version that sometimes set interfaces down, a
+modified version is used.
+
+The source is located at <https://github.com/vyos/vyos-netplug>
+
+In the future, we may switch to using systemd infrastructure instead. Building
+it doesn't require a special procedure.
+
+## keepalived
+
+Keepalived normally isn't updated to newer feature releases between Debian
+versions, so we are building it from source.
+
+Debian does keep their package in git, but it's upstream tarball imported into
+git without its original commit history. To be able to merge new tags in, we
+keep a fork of the upstream repository with packaging files imported from
+Debian at <https://github.com/vyos/keepalived-upstream>
+
+## strongswan
+
+Our StrongSWAN build differs from the upstream:
+
+- strongswan-nm package build is disabled since we don't use NetworkManager
+- Patches for DMVPN are merged in
+
+The source is at <https://github.com/vyos/vyos-strongswan>
+
+DMVPN patches are added by this commit:
+<https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226>
+
+Our op mode scripts use the python-vici module, which is not included in
+Debian's build, and isn't quite easy to integrate in that build. For this
+reason we debianize that module by hand now, using this procedure:
+
+0. Install <https://pypi.org/project/stdeb/>
+1. `cd vyos-strongswan`
+2. `./configure --enable-python-eggs`
+3. `cd src/libcharon/plugins/vici/python`
+4. `make`
+5. `python3 setup.py --command-packages=stdeb.command bdist_deb`
+
+The package ends up in deb_dist dir.
+
+## mdns-repeater
+
+This package doesn't exist in Debian. A debianized fork is kept at
+<https://github.com/vyos/mdns-repeater>
+
+No special build procedure is required.
+
+## udp-broadcast-relay
+
+This package doesn't exist in Debian. A debianized fork is kept at
+<https://github.com/vyos/udp-broadcast-relay>
+
+No special build procedure is required.
+
+## hvinfo
+
+A fork with packaging changes for VyOS is kept at <https://github.com/vyos/hvinfo>
+
+The original repo is at <https://github.com/dmbaturin/hvinfo>
+
+It's an Ada program and requires GNAT and gprbuild for building, dependencies
+are properly specified so just follow debuild's suggestions.