Merge pull request #1 from vyos/master

Update fork

7 files changed, 465 insertions, 452 deletions

diff --git a/docs/contributing/build-vyos.rst b/docs/contributing/build-vyos.rst
index 43c1e608..cb97e418 100644
--- a/docs/contributing/build-vyos.rst
+++ b/docs/contributing/build-vyos.rst
@@ -10,9 +10,9 @@ Prerequisites
 
 There are different ways you can build VyOS.
 
-Building using a :ref:`build_docker` container, although not the only way, is the
-easiest way as all dependencies are managed for you. However, you can also
-set up your own build machine and run a :ref:`build_native`.
+Building using a :ref:`build_docker` container, although not the only way,
+is the easiest way as all dependencies are managed for you. However, you can
+also set up your own build machine and run a :ref:`build_native`.
 
 .. note:: Starting with VyOS 1.2 the release model of VyOS has changed. VyOS
    is now **free as in speech, but not as in beer**. This means that while
@@ -93,10 +93,11 @@ The container can also be built directly from source:
 
   $ cd vyos-build
   $ docker build -t vyos/vyos-build:crux docker # For VyOS 1.2
-  $ docker build -t vyos/vyos-build docker      # For rolling release
+  $ docker build -t vyos/vyos-build:current docker      # For rolling release
 
-.. note:: Since VyOS has switched to Debian (10) Buster in its ``current`` branch,
-   you will require individual container for `current` and `crux` builds.
+.. note:: Since VyOS has switched to Debian (10) Buster in its ``current``
+   branch, you will require individual container for `current` and `crux`
+   builds.
 
 Tips and Tricks
 ---------------
@@ -125,7 +126,7 @@ per release train (`current` or `crux`) - container. Add the following to your
       -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \
       vyos/vyos-build:crux bash'
 
-Now you are prepared with two new aliases ``vybld`` and ``vybld_crux`` to spwan
+Now you are prepared with two new aliases ``vybld`` and ``vybld_crux`` to spawn
 your development containers in your current working directory.
 
 .. _build_native:
@@ -154,7 +155,7 @@ in the repository_. The ``./configure`` script will also warn you if any
 dependencies are missing.
 
 Once you have the required dependencies installed, you may proceed with the
-steps descirbed in :ref:`build_iso`.
+steps described in :ref:`build_iso`.
 
 
 .. _build_iso:
@@ -175,7 +176,8 @@ Please note as this will differ for both `current` and `crux`.
   # For VyOS 1.3 (equuleus, current)
   $ git clone -b current --single-branch https://github.com/vyos/vyos-build
 
-Now a fresh build of the VyOS ISO can begin. Change directory to the ``vyos-build`` directory and run:
+Now a fresh build of the VyOS ISO can begin. Change directory to the
+``vyos-build`` directory and run:
 
 .. code-block:: none
 
@@ -184,7 +186,7 @@ Now a fresh build of the VyOS ISO can begin. Change directory to the ``vyos-buil
   $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:crux bash
 
   # For VyOS 1.3 (equuleus, current)
-  $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build bash
+  $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash
 
 Start the build:
 
@@ -193,8 +195,8 @@ Start the build:
   vyos_bld@d4220bb519a0:/vyos# ./configure --architecture amd64 --build-by "j.randomhacker@vyos.io"
   vyos_bld@d4220bb519a0:/vyos# sudo make iso
 
-When the build is successful, the resulting iso can be found inside the ``build``
-directory as ``live-image-[architecture].hybrid.iso``.
+When the build is successful, the resulting iso can be found inside the
+``build`` directory as ``live-image-[architecture].hybrid.iso``.
 
 Good luck!
 
@@ -202,8 +204,8 @@ Good luck!
    Docker does not expose all the filesystem feature required to the container.
    Building within a VirtualBox server on Mac however possible.
 
-.. hint:: Building VyOS on Windows WSL2 with Docker integrated into WSL2 will work
-   like a charm. No problems are known so far!
+.. hint:: Building VyOS on Windows WSL2 with Docker integrated into WSL2 will
+   work like a charm. No problems are known so far!
 
 .. _build source:
 
@@ -257,6 +259,367 @@ The full and current list can be generated with ``./configure --help``:
 
 .. _build_custom_packages:
 
+Linux Kernel
+============
+
+The Linux kernel used by VyOS is heavily tied to the ISO build process. The
+file ``data/defaults.json`` hosts a JSON definition of the kernel version used
+``kernel_version`` and the ``kernel_flavor`` of the kernel which represents the
+kernel's LOCAL_VERSION. Both together form the kernel version variable in the
+system:
+
+.. code-block:: none
+
+  vyos@vyos:~$ uname -r
+  4.19.146-amd64-vyos
+
+Other packages (e.g. vyos-1x) add dependencies to the ISO build procedure on
+e.g. the wireguard-modules package which itself adds a dependency on the kernel
+version used due to the module it ships. This may change (for WireGuard) in
+future kernel releases but as long as we have out-of-tree modules.
+
+* WireGuard
+* Accel-PPP
+* Intel NIC drivers
+* Inter QAT
+
+Each of those modules holds a dependency on the kernel version and if you are
+lucky enough to receive an ISO build error which sounds like:
+
+.. code-block:: none
+
+  I: Create initramfs if it does not exist.
+  Extra argument '4.19.146-amd64-vyos'
+  Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory]
+  Options:
+   -k version     Specify kernel version or 'all'
+   -c             Create a new initramfs
+   -u             Update an existing initramfs
+   -d             Remove an existing initramfs
+   -b directory   Set alternate boot directory
+   -v             Be verbose
+  See update-initramfs(8) for further details.
+  E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors.
+
+The most obvious reasons could be:
+
+* ``vyos-build`` repo is outdated, please ``git pull`` to update to the latest
+  release kernel version from us.
+
+* You have your own custom kernel `*.deb` packages in the `packages` folder but
+  neglected to create all required out-of tree modules like Accel-PPP,
+  WireGuard, Intel QAT, Intel NIC
+
+Building The Kernel
+-------------------
+
+The kernel build is quite easy, most of the required steps can be found in the
+``vyos-build/packages/linux-kernel/Jenkinsfile`` but we will walk you through
+it.
+
+Clone the kernel source to `vyos-build/packages/linux-kernel/`:
+
+.. code-block:: none
+
+  $ cd vyos-build/packages/linux-kernel/
+  $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+
+Check out the required kernel version - see ``vyos-build/data/defaults.json``
+file (example uses kernel 4.19.146):
+
+.. code-block:: none
+
+  $ cd vyos-build/packages/linux-kernel/linux
+  $ git checkout v4.19.146
+  Checking out files: 100% (61536/61536), done.
+  Note: checking out 'v4.19.146'.
+
+  You are in 'detached HEAD' state. You can look around, make experimental
+  changes and commit them, and you can discard any commits you make in this
+  state without impacting any branches by performing another checkout.
+
+  If you want to create a new branch to retain commits you create, you may
+  do so (now or later) by using -b with the checkout command again. Example:
+
+    git checkout -b <new-branch-name>
+
+  HEAD is now at 015e94d0e37b Linux 4.19.146
+
+Now we can use the helper script ``build-kernel.sh`` which does all the
+necessary voodoo by applying required patches from the
+`vyos-build/packages/linux-kernel/patches` folder, copying our kernel
+configuration ``x86_64_vyos_defconfig`` to the right location, and finally
+building the Debian packages.
+
+.. note:: Building the kernel will take some time depending on the speed and
+   quantity of your CPU/cores and disk speed. Expect 20 minutes
+   (or even longer) on lower end hardware.
+
+.. code-block:: none
+
+  (18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh
+  I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source
+  I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch
+  patching file Documentation/networking/ip-sysctl.txt
+  patching file include/linux/inetdevice.h
+  patching file include/linux/ipv6.h
+  patching file include/uapi/linux/ip.h
+  patching file include/uapi/linux/ipv6.h
+  patching file net/ipv4/devinet.c
+  Hunk #1 succeeded at 2319 (offset 1 line).
+  patching file net/ipv6/addrconf.c
+  patching file net/ipv6/route.c
+  I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch
+  patching file fs/notify/inotify/Kconfig
+  patching file fs/notify/inotify/inotify_user.c
+  patching file fs/overlayfs/super.c
+  Hunk #2 succeeded at 1713 (offset 9 lines).
+  Hunk #3 succeeded at 1739 (offset 9 lines).
+  Hunk #4 succeeded at 1762 (offset 9 lines).
+  patching file include/linux/inotify.h
+  I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch
+  patching file scripts/package/builddeb
+  I: make x86_64_vyos_defconfig
+    HOSTCC  scripts/basic/fixdep
+    HOSTCC  scripts/kconfig/conf.o
+    YACC    scripts/kconfig/zconf.tab.c
+    LEX     scripts/kconfig/zconf.lex.c
+    HOSTCC  scripts/kconfig/zconf.tab.o
+    HOSTLD  scripts/kconfig/conf
+  #
+  # configuration written to .config
+  #
+  I: Generate environment file containing Kernel variable
+  I: Build Debian Kernel package
+    UPD     include/config/kernel.release
+  /bin/sh ./scripts/package/mkdebian
+  dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc
+  dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos
+  dpkg-buildpackage: info: source version 4.19.146-1
+  dpkg-buildpackage: info: source distribution buster
+  dpkg-buildpackage: info: source changed by vyos_bld <christian@poessinger.com>
+  dpkg-buildpackage: info: host architecture amd64
+  dpkg-buildpackage: warning: debian/rules is not executable; fixing that
+   dpkg-source --before-build .
+   debian/rules build
+  make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86         KBUILD_BUILD_VERSION=1 KBUILD_SRC=
+    SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
+
+  ...
+
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
+  dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols)
+  dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols)
+  dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'.
+   dpkg-genbuildinfo --build=binary
+   dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes
+  dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list
+  dpkg-genchanges: info: binary-only upload (no source code included)
+   dpkg-source --after-build .
+  dpkg-buildpackage: info: binary-only upload (no source included)
+
+
+In the end you will be presented with the kernel binary packages which you can
+then use in your custom ISO build process, by placing all the `*.deb` files in
+the vyos-build/packages folder where they will be used automatically when
+building VyOS as documented above.
+
+Firmware
+^^^^^^^^
+
+If you upgrade your kernel or include new drivers you may need new firmware.
+Build a new ``vyos-linux-firmware`` package with the included helper scripts.
+
+.. code-block:: none
+
+  $ cd vyos-build/packages/linux-kernel
+  $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
+  $ ./build-linux-firmware.sh
+  $ cp vyos-linux-firmware_*.deb ../
+
+This tries to automatically detect which blobs are needed based on which drivers
+were built. If it fails to find the correct files you can add them manually to
+``vyos-build/packages/linux-kernel/build-linux-firmware.sh``:
+
+.. code-block:: bash
+
+  ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin"
+
+
+Building Out-Of-Tree Modules
+----------------------------
+
+Building the kernel is one part, but now you also need to build the required
+out-of-tree modules so everything is lined up and the ABIs match. To do so,
+you can again take a look at ``vyos-build/packages/linux-kernel/Jenkinsfile``
+to see all of the required modules and their selected versions. We will show
+you how to build all the current required modules.
+
+WireGuard
+^^^^^^^^^
+
+First, clone the source code and check out the appropriate version by running:
+
+.. code-block:: none
+
+  $ cd vyos-build/packages/linux-kernel
+  $ git clone https://salsa.debian.org/debian/wireguard-linux-compat.git
+  $ cd wireguard-linux-compat
+  $ git checkout debian/1.0.20200712-1_bpo10+1
+
+We again make use of a helper script and some patches to make the build work.
+Just run the following command:
+
+.. code-block:: none
+
+  $ cd vyos-build/packages/linux-kernel
+  $ ./build-wireguard-modules.sh
+  I: Apply WireGuard patch: /vyos/packages/linux-kernel/patches/wireguard-linux-compat/0001-Debian-build-wireguard-modules-package.patch
+  patching file debian/control
+  patching file debian/rules
+  I: Build Debian WireGuard package
+  dpkg-buildpackage: info: source package wireguard-linux-compat
+  dpkg-buildpackage: info: source version 1.0.20200712-1~bpo10+1
+  dpkg-buildpackage: info: source distribution buster-backports
+  dpkg-buildpackage: info: source changed by Unit 193 <unit193@debian.org>
+  dpkg-buildpackage: info: host architecture amd64
+   dpkg-source --before-build .
+  dpkg-source: info: using patch list from debian/patches/series
+  dpkg-source: info: applying 0001-Makefile-do-not-use-git-to-get-version-number.patch
+  dpkg-source: info: applying 0002-Avoid-trying-to-compile-on-debian-5.5-kernels-Closes.patch
+
+  ...
+
+  dpkg-genchanges: info: binary-only upload (no source code included)
+   debian/rules clean
+  dh clean
+     dh_clean
+   dpkg-source --after-build .
+  dpkg-source: info: unapplying 0002-Avoid-trying-to-compile-on-debian-5.5-kernels-Closes.patch
+  dpkg-source: info: unapplying 0001-Makefile-do-not-use-git-to-get-version-number.patch
+  dpkg-buildpackage: info: binary-only upload (no source included)
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them
+to the ``vyos-build/packages`` folder for inclusion during the ISO build.
+
+Accel-PPP
+^^^^^^^^^
+
+First, clone the source code and check out the appropriate version by running:
+
+.. code-block:: none
+
+  $ cd vyos-build/packages/linux-kernel
+  $ git clone https://github.com/accel-ppp/accel-ppp.git
+
+We again make use of a helper script and some patches to make the build work.
+Just run the following command:
+
+.. code-block:: none
+
+  $ ./build-accel-ppp.sh
+  I: Build Accel-PPP Debian package
+  CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy):
+    The OLD behavior for policy CMP0003 will be removed from a future version
+    of CMake.
+
+    The cmake-policies(7) manual explains that the OLD behaviors of all
+    policies are deprecated and that a policy should be set to OLD only under
+    specific short-term circumstances.  Projects should be ported to the NEW
+    behavior and not rely on setting a policy to OLD.
+
+  -- The C compiler identification is GNU 8.3.0
+
+  ...
+
+  CPack: Create package using DEB
+  CPack: Install projects
+  CPack: - Run preinstall target for: accel-ppp
+  CPack: - Install project: accel-ppp
+  CPack: Create package
+  CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated.
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them
+to the ``vyos-build/packages`` folder for inclusion during the ISO build.
+
+Intel NIC
+^^^^^^^^^
+
+The Intel NIC drivers do not come from a Git repository, instead we just fetch
+the tarballs from our mirror and compile them.
+
+Simply use our wrapper script to build all of the driver modules.
+
+.. code-block:: none
+
+  ./build-intel-drivers.sh
+    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                   Dload  Upload   Total   Spent    Left  Speed
+  100  490k  100  490k    0     0   648k      0 --:--:-- --:--:-- --:--:--  648k
+  I: Compile Kernel module for Intel ixgbe driver
+
+  ...
+
+  I: Building Debian package vyos-intel-iavf
+  Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
+  Require just the needed backports instead, or 'backports/latest'.
+  Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn}
+  Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"}
+  I: Cleanup iavf source
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them
+to the ``vyos-build/packages`` folder for inclusion during the ISO build.
+
+Intel QAT
+^^^^^^^^^
+The Intel QAT (Quick Assist Technology) drivers do not come from a Git
+repository, instead we just fetch the tarballs from 01.org, Intel's
+open-source website.
+
+Simply use our wrapper script to build all of the driver modules.
+
+.. code-block:: none
+
+  $ ./build-intel-qat.sh
+    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                   Dload  Upload   Total   Spent    Left  Speed
+  100 5065k  100 5065k    0     0  1157k      0  0:00:04  0:00:04 --:--:-- 1157k
+  I: Compile Kernel module for Intel qat driver
+  checking for a BSD-compatible install... /usr/bin/install -c
+  checking whether build environment is sane... yes
+  checking for a thread-safe mkdir -p... /bin/mkdir -p
+  checking for gawk... gawk
+  checking whether make sets $(MAKE)... yes
+
+  ...
+
+  I: Building Debian package vyos-intel-qat
+  Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
+  Require just the needed backports instead, or 'backports/latest'.
+  Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn}
+  Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"}
+  I: Cleanup qat source
+
+
+After compiling the packages you will find yourself the newly generated `*.deb`
+binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them
+to the ``vyos-build/packages`` folder for inclusion during the ISO build.
+
+
 Packages
 ========
 
@@ -264,7 +627,7 @@ If you are brave enough to build yourself an ISO image containing any modified
 package from our GitHub organisation - this is the place to be.
 
 Any "modified" package may refer to an altered version of e.g. vyos-1x package
-that you would like to test before filing a PullRequest on GitHub.
+that you would like to test before filing a pull request on GitHub.
 
 Building an ISO with any customized package is in no way different then
 building a regular (customized or not) ISO image. Simply place your modified
@@ -275,10 +638,12 @@ Troubleshooting
 ===============
 
 Debian APT is not very verbose when it comes to errors. If your ISO build breaks
-for whatever reason and you supect its a problem with APT dependencies or
+for whatever reason and you suspect it's a problem with APT dependencies or
 installation you can add this small patch which increases the APT verbosity
 during ISO build.
 
+.. stop_vyoslinter
+
 .. code-block:: diff
 
   diff --git i/scripts/live-build-config w/scripts/live-build-config
@@ -296,6 +661,9 @@ during ISO build.
            "${@}"
    """
 
+.. start_vyoslinter
+
+
 
 Virtualization Platforms
 ========================
@@ -324,9 +692,9 @@ Run following command after building the QEMU image.
 Packages
 ********
 
-VyOS itself comes with a bunch of packages which are specific to our system and
-thus can not be found in any Debian mirrror. Those packages can be found at the
-`VyOS GitHub project`_ in their source format can can easily be compiled into
+VyOS itself comes with a bunch of packages that are specific to our system and
+thus cannot be found in any Debian mirror. Those packages can be found at the
+`VyOS GitHub project`_ in their source format can easily be compiled into
 a custom Debian (`*.deb`) package.
 
 The easiest way to compile your package is with the above mentioned
@@ -348,7 +716,7 @@ Launch Docker container and build package
 .. code-block:: none
 
   # For VyOS 1.3 (equuleus, current)
-  $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build bash
+  $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash
 
   # Change to source directory
   $ cd vyos-1x
@@ -356,8 +724,8 @@ Launch Docker container and build package
   # Build DEB
   $ dpkg-buildpackage -uc -us -tc -b
 
-After a minute or two you will find the generated DEB packages next to the vyos-1x
-source directory:
+After a minute or two you will find the generated DEB packages next to the
+vyos-1x source directory:
 
 .. code-block:: none
 
@@ -392,8 +760,14 @@ information.
    the source directories and built deb packages) if you want to build an iso
    from purely upstream packages.
 
+
+.. stop_vyoslinter
+
 .. _Docker: https://www.docker.com
 .. _`Docker as non-root`: https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user
 .. _VyOS DockerHub organisation: https://hub.docker.com/u/vyos
 .. _repository: https://github.com/vyos/vyos-build
 .. _VyOS GitHub project: https://github.com/vyos
+
+.. start_vyoslinter
+
diff --git a/docs/contributing/debugging.rst b/docs/contributing/debugging.rst
deleted file mode 100644
index ac2e0510..00000000
--- a/docs/contributing/debugging.rst
+++ /dev/null
@@ -1,146 +0,0 @@
-.. _debugging:
-
-#########
-Debugging
-#########
-
-There are two flags available to aid in debugging configuration scripts.
-Since configuration loading issues will manifest during boot, the flags are
-passed as kernel boot parameters.
-
-System Startup
-==============
-
-The system startup can be debugged (like loading in the configuration
-file from ``/config/config.boot``. This can be achieve by extending the
-Kernel command-line in the bootloader.
-
-Kernel
-------
-
-* ``vyos-debug`` - Adding the parameter to the linux boot line will produce
-  timing results for the execution of scripts during commit. If one is seeing
-  an unexpected delay during manual or boot commit, this may be useful in
-  identifying bottlenecks. The internal flag is ``VYOS_DEBUG``, and is found
-  in vyatta-cfg_. Output is directed to ``/var/log/vyatta/cfg-stdout.log``.
-
-* ``vyos-config-debug`` - During development, coding errors can lead to a
-  commit failure on boot, possibly resulting in a failed initialization of the
-  CLI. In this circumstance, the kernel boot parameter ``vyos-config-debug``
-  will ensure access to the system as user ``vyos``, and will log a Python
-  stack trace to the file ``/tmp/boot-config-trace``.
-  File ``boot-config-trace`` will generate only if config loaded with a failure status.
-
-Live System
-===========
-
-A number of flags can be set up to change the behaviour of VyOS at runtime.
-These flags can be toggled using either environment variables or creating
-files.
-
-For each feature, a file called ``vyos.feature.debug`` can be created to
-toggle the feature on. If a parameter is required it can be placed inside
-the file as its first line.
-
-The file can be placed in ``/tmp`` for one time debugging (as the file
-will be removed on reboot) or placed in '/config' to stay permanently.
-
-For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable
-interface debugging.
-
-It is also possible to set up the debugging using environment variables.
-In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG.
-
-For example running, ``export VYOS_IFCONFIG_DEBUG=""`` on your vash,
-will have the same effect as ``touch /tmp/vyos.ifconfig.debug``.
-
-* ``ifconfig`` - Once set, all commands used, and their responses received
-  from the OS, will be presented on the screen for inspection.
-
-* ``command`` - Once set, all commands used, and their responses received
-  from the OS, will be presented on the screen for inspection.
-
-* ``developer`` - Should a command fail, instead of printing a message to the
-  user explaining how to report issues, the python interpreter will start a
-  PBD post-mortem session to allow the developer to debug the issue. As the
-  debugger will wait from input from the developer, it has the capacity to
-  prevent a router to boot and therefore should only be permanently set up
-  on production if you are ready to see the OS fail to boot.
-
-* ``log`` - In some rare cases, it may be useful to see what the OS is doing,
-  including during boot. This option sends all commands used by VyOS to a
-  file. The default file is ``/tmp/full-log`` but it can be changed.
-
-Config Migration Scripts
-------------------------
-
-When writing a new configuration migrator it may happen that you see an error
-when you try to invoke it manually on a development system. This error will
-look like:
-
-.. code-block:: none
-
-  vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot
-  Traceback (most recent call last):
-    File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module>
-      config = ConfigTree(config_file)
-    File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__
-      raise ValueError("Failed to parse config: {0}".format(msg))
-  ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax.
-
-The reason is that the configuration migration backend is rewritten and uses
-a new form of "magic string" which is applied on demand when real config
-migration is run on boot. When runnint individual migrators for testing,
-you need to convert the "magic string" on your own by:
-
-.. code-block:: none
-
-  vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot
-
-Configuration Error on System Boot
-----------------------------------
-
-Beeing brave and running the latest rolling releases will sometimes trigger
-bugs due to corner cases we missed in our design. Those bugs should be filed
-via Phabricator_ but you can help us to narrow doen the issue. Login to your
-VyOS system and change into configuration mode by typing ``configure``. Now
-re-load your boot configuration by simply typing ``load`` followed by return.
-
-You shoudl now see a Python backtrace which will help us to handle the issue,
-please attach it to the Phabricator_ task.
-
-Boot Timing
------------
-
-During the migration and extensive rewrite of functionality from Perl into
-Python a significant increase in the overall system boottime was noticed. The
-system boot time can be analysed and a graph can be generated in the end which
-shows in detail who called whom during the system startup phase.
-
-This is done by utilizing the ``systemd-bootchart`` package which is now
-installed by default on the VyOS 1.3 (equuleus) branch. The configuration is
-also versioned so we get comparable results. ``systemd-bootchart`` is configured
-using this file: bootchart.conf_
-
-To enable boot time graphing change the Kernel commandline and add the folowing
-string: ``init=/usr/lib/systemd/systemd-bootchart``
-
-This can also be done permanently by changing ``/boot/grub/grub.cfg``.
-
-Priorities
-==========
-
-VyOS CLI is all about priorities. Every CLI node has a corresponding ``node.def``
-file and possibly an attached script that is executed when the node is present.
-Nodes can have a priority, and on system bootup - or any other ``commit`` to the
-config all scripts are executed from lowest to higest priority. This is good as
-this gives a deterministic behavior.
-
-To debug issues in priorities or to see what's going on in the background you can
-use the ``/opt/vyatta/sbin/priority.pl`` script which lists to you the execution
-order of the scripts.
-
-.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg
-.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf
-
-.. include:: ../common-references.rst
diff --git a/docs/contributing/development.rst b/docs/contributing/development.rst
index 17f5cc48..cf274a33 100644
--- a/docs/contributing/development.rst
+++ b/docs/contributing/development.rst
@@ -271,7 +271,7 @@ device if you happen to be a crazy scientist.
 
   #!/usr/bin/env python3
   #
-  # Copyright (C) 2019 VyOS maintainers and contributors
+  # Copyright (C) 2020 VyOS maintainers and contributors
   #
   # This program is free software; you can redistribute it and/or modify
   # it under the terms of the GNU General Public License version 2 or later as
@@ -291,10 +291,16 @@ device if you happen to be a crazy scientist.
   from vyos import ConfigError
 
   def get_config():
-      vc = Config()
+      if config:
+          conf = config
+      else:
+          conf = Config()
+        
+      # Base path to CLI nodes
+      base = ['...', '...']
       # Convert the VyOS config to an abstract internal representation
-      config = ...
-      return config
+      config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True)
+      return config_data
 
   def verify(config):
       # Verify that configuration is valid
@@ -311,8 +317,10 @@ device if you happen to be a crazy scientist.
       pass
 
   try:
-      config = get_config()
-      verify(config)
+      c = get_config()
+      verify(c)
+      generate(c)
+      apply(c)      
   except ConfigError as e:
       print(e)
       sys.exit(1)
@@ -650,8 +658,8 @@ Migrating old CLI
        validation is better left to commit-time scripts
    * - priority: 999
      - <properties> <priority>999</priority>
-     - Please leave a comment explaining why the priority was chosen (e.g. "after
-       interfaces are configured")
+     - Please leave a comment explaining why the priority was chosen
+       (e.g. "after interfaces are configured")
    * - multi:
      - <properties> <multi/>
      - Only applicable to leaf nodes
@@ -692,6 +700,9 @@ found. After a successful run the resulting Debian Package(s) will be deployed
 to our Debian repository which is used during build time. It is located here:
 http://dev.packages.vyos.net/repositories/.
 
+
+.. stop_vyoslinter
+
 .. _Jenkins: https://jenkins.io/
 .. _Dockerhub: https://hub.docker.com/u/vyos/
 .. _process: https://blog.vyos.io/vyos-development-digest-10
@@ -703,4 +714,6 @@ http://dev.packages.vyos.net/repositories/.
 .. _`VLAN (VIF)`: https://github.com/vyos/vyos-1x/tree/current/interface-definitions/include/vif.xml.i
 .. _`MAC address`: https://github.com/vyos/vyos-1x/tree/current/interface-definitions/include/interface-mac.xml.i
 
-.. include:: ../common-references.rst
+.. include:: /_include/common-references.txt
+
+.. start_vyoslinter
diff --git a/docs/contributing/documentation.rst b/docs/contributing/documentation.rst
index 8102e9a9..eb8db3e1 100644
--- a/docs/contributing/documentation.rst
+++ b/docs/contributing/documentation.rst
@@ -1,218 +1,50 @@
-.. _documentation:
+.. _contrib-documentation:
 
 Documentation
-=============
+-------------
 
-As most software projects we also have a lack in documentation. We encourage
-every VyOS user to help us improve our documentation. This will not only be
-beneficial for you (when reading something up) but also for the whole world.
+VyOS documentation is written in reStructuredText and generated to Read the Docs
+pages with Sphinx, as per the Python tradition, as well as PDF files for offline
+use through LaTeX.
 
-If you are willing to contribute to our documentation this is the definite
-guide how to do so.
+We welcome all sorts of contributions to the documentation. Not just
+new additions but also corrections to existing documentation.
 
-.. note:: In contrast to submitting code patches, there is no requirement that
-   you open up a Phabricator_ task prior to submitting a Pull-Request to the
-   documentation.
+Guidelines
+^^^^^^^^^^
 
-Forking Workflow
-----------------
+There are a few things to keep in mind when contributing to the
+documentation, for the sake of consistency and readability.
 
-The Forking Workflow is fundamentally different than other popular Git
-workflows. Instead of using a single server-side repository to act as the
-"central" codebase, it gives every developer their own server-side repository.
-This means that each contributor has not one, but two Git repositories: a
-private local one and a public server-side one.
+Take a look at the :doc:`/documentation` page for an intricate explanation
+of the documentation process.
 
-The main advantage of the Forking Workflow is that contributions can be
-integrated without the need for everybody to push to a single central
-repository. Developers push to their own server-side repositories, and only the
-project maintainer can push to the official repository. This allows the
-maintainer to accept commits from any developer without giving them write
-access to the official codebase.
+The following is a quick summary of the rules:
 
-.. note:: Updates to our documentation should be delivered by a GitHub
-   pull-request. This requires you already have a GitHub account.
+- Use American English at all times. It's always a good idea to run
+  your text through a grammar and spell checker, such as `Grammarly`_.
+- Don't forget to update ``index.rst`` when adding a new node.
+- Try not to exceed 80 characters per line, but don't break URLs over this.
+- Properly quote commands, filenames and brief code snippets with double backticks.
+- Use literal blocks for longer snippets.
+- Leave a newline before and after a header.
+- Indent with two spaces.
+- When in doubt, follow the style of existing documentation.
 
-* Fork this project on GitHub https://github.com/vyos/vyos-documentation/fork
+And finally, remember that the reStructuredText files aren't
+exclusively for generating HTML and PDF. They should be human-readable
+and easily perused from a console.
 
-* Clone fork to local machine
-
-* Change to your new local directory ``$ cd vyos-documentation``
-
-* Create new branch for your work, use a descriptive name of your work:
-  ``$ git checkout -b fix-vxlan-typo``
-
-* Make all your changes - please keep our commit rules in mind
-  (:ref:`prepare_commit`). This mainly applies to proper commit messages
-  describing your change (how and why). Please check out the documentation of
-  Sphinx-doc_ or reStructuredText_ if you are not familiar with it. This is used
-  for writing our docs.
-
-* Check your changes by locally building the documentation ``$ make html``.
-  Sphinx will build the html files in the ``docs/_build`` folder. We provide
-  you with a Docker container for an easy to use user experience. Check the
-  README.md_ file of this repository.
-
-* View modified files by calling ``$ git status``. You will get an overview of
-  all files modified by you. You can add individual files to the Git Index in
-  the next step.
-
-* Add modified files to Git index ``$ git add path/to/filename`` or add all
-  unstaged files ``$ git add .``. All files added to the Git index will be part
-  of you following Git commit.
-
-* Commit your changes ``$ git commit -v`` - your configured editor will now ne
-  launched where you can type in a commit message. Again please make yourself
-  comfortable with out rules (:ref:`prepare_commit`).
-
-* Push your commits to your GitHub project: ``$ git push -u origin foo-branch``
-
-* Submit pull-request. In GitHub visit the main repository and you should
-  see a banner suggesting to make a pull request. Fill out the form and
-  describe what you do.
-
-* Once pull resquests have been approved, you may want to locally update
-  your forked repository too. First you'll have to add a second remote
-  called `upstream` which points to our main repository. ``$ git remote add
-  upstream https://github.com/vyos/vyos-documentation.git``
-
-  Check your configured remote repositories:
-
-  .. code-block:: none
-
-    $ git remote -v
-    origin    https://github.com/<username>/vyos-documentation.git (fetch)
-    origin    https://github.com/<username>/vyos.documentation.git (push)
-    upstream  https://github.com/vyos/vyos-documentation.git (fetch)
-    upstream  https://github.com/vyos/vyos-documentation.git (push)
-
-  Your remote repo on Github is called ``origin``, while the original repo you
-  have forked is called ``upstream``. Now you can locally update your forked
-  repo.
-
-  .. code-block:: none
-
-    $ git fetch upstream
-    $ git checkout master
-    $ git merge upstream/master
-
-* If you want to update your fork on GitHub, too use the following: ``$ git
-  push origin master``
-
-Style Guide
------------
-
-Sections
+Building
 ^^^^^^^^
 
-We use the following syntax for Headlines.
-
-.. code-block:: none
-
-  #####
-  Parts
-  #####
-
-  ********
-  Chapters
-  ********
-
-  Sections
-  ========
-
-  Subsections
-  -----------
-
-  Subsubsections
-  ^^^^^^^^^^^^^^
-
-  Paragraphs
-  """"""""""
-
-Address space
-^^^^^^^^^^^^^
-
-Note the following RFCs (:rfc:`5737`, :rfc:`3849`, :rfc:`5389` and
-:rfc:`7042`), which describe the reserved public IP addresses and autonomous
-system numbers for the documentation:
-
-  * ``192.0.2.0/24``
-  * ``198.51.100.0/24``
-  * ``203.0.113.0/24``
-  * ``2001:db8::/32``
-  * 16bit ASN: ``64496 - 64511``
-  * 32bit ASN: ``65536 - 65551``
-  * Unicast MAC Addresses: ``00-53-00`` to ``00-53-FF``
-  * Multicast MAC-Addresses: ``90-10-00`` to ``90-10-FF``
-
-Please don't use other public address space.
-
-Custom Sphinx-doc Markup
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-When writing the documentation custom commands have been developed. Please
-make yourself comfortable with those commands as this eases the way we
-render the documentation.
-
-cfgcmd
-""""""
-
-When documenting CLI commands use the ``.. cfgcmd::`` directive for all
-configuration mode commands. An explanation of the described command should be
-added below this statement.
-
-With those custom commands it will be possible to render them in a more
-descriptive way in the resulting HTML/PDF manual.
-
-.. code-block:: none
-
-  .. cfgcmd:: set protocols static arp 192.0.2.100 hwaddr 00:53:27:de:23:aa
-
-     This will configure a static ARP entry always resolving `192.0.2.100` to
-     `00:53:27:de:23:aa`.
-
-For a inline configuration level command use ``:cfgcmd:``
-
-.. code-block:: none
-  
-  :cfgcmd:`set interface ethernet eth0`
-
-opcmd
-"""""
-
-When documenting operational level command use the ``.. opcmd::`` directive.
-An explanation of the described command should be added below this statement.
-
-With those custom commands it will be possible to render them in a more
-descriptive way in the resulting HTML/PDF manual.
-
-.. code-block:: none
-
-  .. opcmd:: show protocols static arp
-
-     Display all known ARP table entries spanning across all interfaces
-
-For a inline operational level command use ``:opcmd:``
-
-.. code-block:: none
-  
-  :opcmd:`add system image`
-
-vytask
-""""""
-
-When referencing to VyOS Phabricator Tasks, there is a custom Sphinx Markup
-command called ``vytask`` which automatically renders to a proper Phabricator
-URL. This is heavily used in the :ref:`release-notes` section.
-
-.. code-block:: none
-
-  * :vytask:`T1605` Fixed regression in L2TP/IPsec server
-  * :vytask:`T1613` Netflow/sFlow captures IPv6 traffic correctly
+The source is kept in the Git repository
+https://github.com/vyos/vyos-documentation
 
+You can follow the instructions in the README to build and test your changes.
 
-.. _Sphinx-doc: https://www.sphinx-doc.org
-.. _reStructuredText: http://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html
-.. _README.md: https://github.com/vyos/vyos-documentation/blob/master/README.md
+You can either install Sphinx (and TeX Live for PDF output) and build the
+documentation locally, or use the `Dockerfile`_ to build it in a container.
 
-.. include:: ../common-references.rst
+.. _Dockerfile: https://github.com/vyos/vyos-documentation/blob/master/docker/Dockerfile
+.. _Grammarly: https://www.grammarly.com/
diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst
new file mode 100644
index 00000000..481c6da8
--- /dev/null
+++ b/docs/contributing/index.rst
@@ -0,0 +1,12 @@
+############
+Contributing
+############
+
+.. toctree::
+   :maxdepth: 2
+
+   build-vyos
+   development
+   documentation
+   issues-features
+   upstream-packages
diff --git a/docs/contributing/issues-features.rst b/docs/contributing/issues-features.rst
index 60e49974..9b6602f9 100644
--- a/docs/contributing/issues-features.rst
+++ b/docs/contributing/issues-features.rst
@@ -77,4 +77,4 @@ the left side under the specific project.
 .. _Slack: https://slack.vyos.io
 .. _Forum: https://forum.vyos.io
 
-.. include:: ../common-references.rst
+.. include:: /_include/common-references.txt
diff --git a/docs/contributing/upstream-packages.rst b/docs/contributing/upstream-packages.rst
index d43e61f3..5c48bbb3 100644
--- a/docs/contributing/upstream-packages.rst
+++ b/docs/contributing/upstream-packages.rst
@@ -59,26 +59,6 @@ reason we debianize that module by hand now, using this procedure:
 
 The package ends up in deb_dist dir.
 
-ppp
-^^^
-
-Properly renaming PPTP and L2TP interfaces to pptpX and l2tpX from generic and
-non-informative pppX requires a patch that is neither in the upstream nor in
-Debian.
-
-We keep a fork of Debian's repo at https://github.com/vyos/ppp-debian
-
-The patches for pre-up renaming are:
-
-* https://github.com/vyos/ppp-debian/commit/e728180026a051d2a96396276e7e4ae
-* https://github.com/vyos/ppp-debian/commit/f29ba8d9ebb043335a096d70bcd07e9
-
-Additionally, there's a patch for reopening the log file to better support
-logging to files, even though it's less essential:
-https://github.com/vyos/ppp-debian/commit/dd2ebd5cdcddb40230dc4cc43d374055f
-
-The patches were written by Stephen Hemminger back in the Vyatta times.
-
 mdns-repeater
 ^^^^^^^^^^^^^
 
@@ -95,58 +75,6 @@ https://github.com/vyos/udp-broadcast-relay
 
 No special build procedure is required.
 
-Linux kernel
-^^^^^^^^^^^^
-
-In the past a fork of the Kernel source code was kept at the well-known
-location of https://github.com/vyos/vyos-kernel - where it is kept for history.
-
-Nowadays the Kernel we use is the upstream source code which is patched
-with two additional patches from the good old Vyatta times which never made it
-into the mainstream Kernel. The patches can be found here:
-https://github.com/vyos/vyos-build-kernel/tree/master/patches/kernel and are
-automatically applied to the Kernel by the Jenkins Pipeline which is used to
-generate the Kernel binaries.
-
-The Pipeline script not only builds the Kernel with the configuration named
-``x86_64_vyos_defconfig`` which is located in the vyos-build-kernel repository,
-too - but in addition also builds some Intel out-of-tree drivers, WireGuard
-(as long it is not upstreamed) and Accel-PPP.
-
-The ``Jenkinsfile`` tries to be as verbose as possible on each individual build
-step.
-
-Linux Firmware
-^^^^^^^^^^^^^^
-
-More and more hardware cards require an additional firmware which is not open
-source. The Kernel community hosts a special linux-firmware Git repository
-with all available binary files which can be loaded by the Kernel.
-
-The ``vyos-build`` repository fetches a specific commit of the linux-firmware
-repository and embeds those binaries into the resulting ISO image. This step is
-done in the ``data/live-build-config/hooks/live/40-linux-firmware.chroot`` file.
-
-If the firmware needs to be updated it is sufficient to just exchange the Git
-commit id we reference in our build.
-
-Intel NIC drivers
-^^^^^^^^^^^^^^^^^
-
-We do not make use of the building Intel NIC drivers except for e1000e. Main
-reason is that the out of tree Intel drivers seem be perform a bit better,
-e.q. have proper receive-side-scaling and multi-queue support.
-
-Drivers are build as part of the Kernel Pipeline - read above.
-
-Accel-PPP
-^^^^^^^^^
-
-Accel-PPP used to be an upstream fork for quite some time but now has been
-converted to make use of the upstream source code and build system.
-
-It is build as part of the Kernel Pipeline - read above.
-
 hvinfo
 ^^^^^^