diff options
| author | Yuriy Andamasov <yuriy@vyos.io> | 2026-05-10 17:27:05 +0300 |
|---|---|---|
| committer | Yuriy Andamasov <yuriy@vyos.io> | 2026-05-10 17:27:05 +0300 |
| commit | e7a0bebdb5dc4b436b8b610bcb4f01afc33152e0 (patch) | |
| tree | d9ad6bc834c8e5240c300dff6d47841c2787ce61 /docs/contributing | |
| parent | 15855844e1fe5b0bd39b020639f4c08c69d24864 (diff) | |
| download | vyos-documentation-e7a0bebdb5dc4b436b8b610bcb4f01afc33152e0.tar.gz vyos-documentation-e7a0bebdb5dc4b436b8b610bcb4f01afc33152e0.zip | |
chore: remove RST swap mechanism, archive rst-*.rst under docs/_rst_legacy/
The swap mechanism (RST-as-fallback for migrated MD pages) is dormant —
docs/_rst_overrides.txt has been empty since the MyST flip trio
landed. The mechanism's surface area is dead weight and the rst-*.rst
shadows scattered across the source tree cause Context7's parser to
misclassify the project as RST.
Sibling PRs:
- yuriy/remove-rst-swap-mechanism (rolling)
- yuriy/remove-rst-swap-mechanism-circinus
Changes:
- Move 210 rst-*.rst shadow files into docs/_rst_legacy/ preserving
subdirectory structure. They remain in the repo for reference; Sphinx
excludes the folder via exclude_patterns.
- Strip swap_sources.py invocation from docs/Makefile.
- Strip rst-*.rst exclude entry and the _md_exclude.txt loader from
docs/conf.py; replace with a single _rst_legacy exclude.
- Delete scripts/swap_sources.py, tests/test_swap_sources.py,
docs/_rst_overrides.txt.
- Update AGENTS.md: drop the "RST override mechanism" section and the
test-runner snippet for the deleted test.
Note: .readthedocs.yml on sagitta has no jobs: block to remove (the
swap was wired only at build-time via the Makefile chain on this branch).
Verified: sphinx-build -b html with --keep-going produces identical
warning set (409 unique — pre-existing cli.rst/aws.rst title-level
warnings on this branch), identical sitemap entry count (215),
identical llms.txt entry count (23), zero rst-* URLs in any artifact.
🤖 Generated by [robots](https://vyos.io)
Diffstat (limited to 'docs/contributing')
| -rw-r--r-- | docs/contributing/rst-build-vyos.rst | 865 | ||||
| -rw-r--r-- | docs/contributing/rst-debugging.rst | 206 | ||||
| -rw-r--r-- | docs/contributing/rst-development.rst | 738 | ||||
| -rw-r--r-- | docs/contributing/rst-issues-features.rst | 80 | ||||
| -rw-r--r-- | docs/contributing/rst-testing.rst | 208 | ||||
| -rw-r--r-- | docs/contributing/rst-upstream-packages.rst | 86 |
6 files changed, 0 insertions, 2183 deletions
diff --git a/docs/contributing/rst-build-vyos.rst b/docs/contributing/rst-build-vyos.rst deleted file mode 100644 index a9dacda4..00000000 --- a/docs/contributing/rst-build-vyos.rst +++ /dev/null @@ -1,865 +0,0 @@ -.. _build: - -########## -Build VyOS -########## - -************* -Prerequisites -************* - -There are different ways you can build VyOS. - -Building using a :ref:`build_docker` container, although not the only way, -is the easiest way as all dependencies are managed for you. However, you can -also set up your own build machine and run a :ref:`build_native`. - -.. note:: Starting with VyOS 1.2 the release model of VyOS has changed. VyOS - is now **free as in speech, but not as in beer**. This means that while - VyOS is still an open source project, the release ISOs are no longer free - and can only be obtained via subscription, or by contributing to the - community. - - The source code remains public and an ISO can be built using the process - outlined in this chapter. - -.. _build_native: - -Native Build -============ - -To build VyOS natively you require a properly configured build host with the -following Debian versions installed: - -- Debian Jessie for VyOS 1.2 (crux) -- Debian Buster for VyOS 1.3 (equuleus) -- Debian Bookworm for VyOS 1.4 (sagitta) -- Debian Bookworm or updated for VyOS 1.5 (circinus, current) - aka the - rolling release - -To start, clone the repository to your local machine: - -.. code-block:: none - - # For VyOS 1.2 (crux) - $ git clone -b crux --single-branch https://github.com/vyos/vyos-build - - # For VyOS 1.3 (equuleus) - $ git clone -b equuleus --single-branch https://github.com/vyos/vyos-build - - # For VyOS 1.4 (sagitta) - $ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build - - # For VyOS 1.5 (circinus, current) - $ git clone -b current --single-branch https://github.com/vyos/vyos-build - - $ cd vyos-build - - # For VyOS 1.2 (crux) and VyOS 1.3 (equuleus) - $ ./configure --architecture amd64 --build-by "j.randomhacker@vyos.io" - $ sudo make iso - - # For VyOS 1.4 (sagitta) and VyOS 1.5 (circinus, current) - $ sudo make clean - $ sudo ./build-vyos-image iso --architecture amd64 --build-by "j.randomhacker@vyos.io" - -For the packages required, you can refer to the ``docker/Dockerfile`` file -in the repository_. The ``./build-vyos-image`` script will also warn you if any -dependencies are missing. - - -.. _build_docker: - -Docker -====== - -This will guide you through the process of building a VyOS ISO using Docker_. -This process has been tested on clean installs of Debian Bullseye (11) and -Bookworm (12). - -Installing Docker_ and prerequisites: - -.. hint:: Due to the updated version of Docker, the following examples may - become invalid. - - Due to differences in version updates and build processes, content related - to VyOS 1.3 and below is no longer included below. - -`On Debian`_ - -.. code-block:: none - - # Add Docker's official GPG key: - $ sudo apt-get update - $ sudo apt-get install ca-certificates curl gnupg - $ sudo install -m 0755 -d /etc/apt/keyrings - $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg - $ sudo chmod a+r /etc/apt/keyrings/docker.gpg - - # Add the repository to Apt sources: - $ echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ - https://download.docker.com/linux/debian \ - $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null - - $ sudo apt-get update - $ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin - -To be able to use Docker_ without ``sudo``, the current non-root user must be -added to the ``docker`` group by calling: ``sudo usermod -aG docker -yourusername``. - -.. hint:: Doing so grants privileges equivalent to the ``root`` user! It is - recommended to remove the non-root user from the ``docker`` group after - building the VyOS ISO. See also `Docker as non-root`_. - -.. note:: The build process needs to be built on a local file system, building - on SMB or NFS shares will result in the container failing to build properly! - VirtualBox Drive Share is also not an option as block device operations - are not implemented and the drive is always mounted as "nodev" - -Build Container ---------------- - -The container can be built by hand or by fetching the pre-built one from -DockerHub. Using the pre-built containers from the `VyOS DockerHub -organisation`_ will ensure that the container is always up-to-date. A rebuild -is triggered once the container changes (please note this will take 2-3 hours -after pushing to the vyos-build repository). - -.. note: If you are using the pre-built container, it will be automatically - downloaded from DockerHub if it is not found on your local machine when - you build the ISO. - -Dockerhub -^^^^^^^^^ - -To manually download the container from DockerHub, run: - -.. code-block:: none - - $ docker pull vyos/vyos-build:sagitta # For VyOS 1.4 - $ docker pull vyos/vyos-build:current # For rolling release - -Build from source -^^^^^^^^^^^^^^^^^ - -The container can also be built directly from source: - -.. code-block:: none - - # For VyOS 1.4 (sagitta) - $ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build - - # For VyOS 1.5 (circinus, current) - $ git clone -b current --single-branch https://github.com/vyos/vyos-build - - - $ cd vyos-build - $ docker build -t vyos/vyos-build:sagitta docker # For VyOS 1.4 - $ docker build -t vyos/vyos-build:current docker # For rolling release - -.. note:: Since VyOS has switched to Debian (12) Bookworm in its ``current`` - branch, It is recommended to use the official Docker Hub container image - to build ``equleus`` and ``crux``. - -Tips and Tricks ---------------- - -You can create yourself some handy Bash aliases to always launch the latest - -per release train (`current` or `sagitta`) - container. Add the following to -your ``.bash_aliases`` file: - -.. code-block:: none - - alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ - -v "$(pwd)":/vyos \ - -v "$HOME/.gitconfig":/etc/gitconfig \ - -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ - -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ - -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ - vyos/vyos-build:current bash' - - alias vybld_sagitta='docker pull vyos/vyos-build:sagitta && docker run --rm -it \ - -v "$(pwd)":/vyos \ - -v "$HOME/.gitconfig":/etc/gitconfig \ - -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ - -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ - -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ - vyos/vyos-build:sagitta bash' - -Now you are prepared with two new aliases ``vybld`` and ``vybld_sagitta`` to -spawn your development containers in your current working directory. - -.. note:: Some VyOS packages (namely vyos-1x) come with build-time tests which - verify some of the internal library calls that they work as expected. Those - tests are carried out through the Python Unittest module. If you want to - build the ``vyos-1x`` package (which is our main development package) you - need to start your Docker container using the following argument: - ``--sysctl net.ipv6.conf.lo.disable_ipv6=0``, otherwise those tests will - fail. - -.. _build_iso: - -********* -Build ISO -********* - -Now as you are aware of the prerequisites we can continue and build our own -ISO from source. For this we have to fetch the latest source code from GitHub. -Please note as this will differ for both `current` and `crux`. - -.. code-block:: none - - # For VyOS 1.4 (sagitta) - $ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build - - # For VyOS 1.5 (circinus, current) - $ git clone -b current --single-branch https://github.com/vyos/vyos-build - -Now a fresh build of the VyOS ISO can begin. Change directory to the -``vyos-build`` directory and run: - -.. code-block:: none - - $ cd vyos-build - - # For VyOS 1.4 (sagitta) - $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:sagitta bash - - # For VyOS 1.5 (circinus, current) - $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash - -.. code-block:: none - - # For MacOS (crux, equuleus, sagitta) - $ git clone https://github.com/vyos/vyos-utils-misc - $ cd build-tools/macos-build - - # For VyOS 1.2 (crux) - $ os=jessie64 branch=crux make build - - # For VyOS 1.3 (equuleus) - $ os=buster64 branch=equuleus make build - - # For VyOS 1.4 (sagitta) - $ os=buster64 branch=sagitta make build - -Start the build: - -.. code-block:: none - - # For VyOS 1.4 (sagitta) and For VyOS 1.5 (circinus, current) - vyos_bld@8153428c7e1f:/vyos$ sudo make clean - vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image iso --architecture amd64 --build-by "j.randomhacker@vyos.io" - -When the build is successful, the resulting iso can be found inside the -``build`` directory as ``live-image-[architecture].hybrid.iso``. - -Good luck! - -.. hint:: Building VyOS on Windows WSL2 with Docker integrated into WSL2 will - work like a charm. No problems are known so far! - -.. _build source: - - -.. _customize: - -Customize -========= - -This ISO can be customized with the following list of configure options. -The full and current list can be generated with ``./build-vyos-image --help``: - -.. code-block:: none - - $ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help - I: Checking if packages required for VyOS image build are installed - usage: build-vyos-image [-h] [--architecture ARCHITECTURE] - [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR] - [--debian-security-mirror DEBIAN_SECURITY_MIRROR] - [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] - [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] - [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run] - [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY] - [--custom-package CUSTOM_PACKAGE] - [build_flavor] - - positional arguments: - build_flavor Build flavor - - optional arguments: - -h, --help show this help message and exit - --architecture ARCHITECTURE - Image target architecture (amd64 or arm64) - --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) - --debian-mirror DEBIAN_MIRROR - Debian repository mirror - --debian-security-mirror DEBIAN_SECURITY_MIRROR - Debian security updates mirror - --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR - Debian repository mirror for pbuilder env bootstrap - --vyos-mirror VYOS_MIRROR - VyOS package mirror - --build-type BUILD_TYPE - Build type, release or development - --version VERSION Version number (release builds only) - --build-comment BUILD_COMMENT - Optional build comment - --debug Enable debug output - --dry-run Check build configuration and exit - --custom-apt-entry CUSTOM_APT_ENTRY - Custom APT entry - --custom-apt-key CUSTOM_APT_KEY - Custom APT key file - --custom-package CUSTOM_PACKAGE - Custom package to install from repositories - - -.. _iso_build_issues: - -ISO Build Issues ----------------- - -There are (rare) situations where building an ISO image is not possible at all -due to a broken package feed in the background. APT is not very good at -reporting the root cause of the issue. Your ISO build will likely fail with a -more or less similar looking error message: - -.. code-block:: none - - The following packages have unmet dependencies: - vyos-1x : Depends: accel-ppp but it is not installable - E: Unable to correct problems, you have held broken packages. - P: Begin unmounting filesystems... - P: Saving caches... - Reading package lists... - Building dependency tree... - Reading state information... - Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB] - Del accel-ppp 1.12.0-95-g59f8e1b [475 kB] - Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB] - Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB] - Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB] - make: *** [Makefile:30: iso] Error 1 - (10:13) vyos_bld ece068908a5b:/vyos [current] # - -To debug the build process and gain additional information of what could be the -root cause, you need to use `chroot` to change into the build directory. This is -explained in the following step by step procedure: - -.. code-block:: none - - vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash - -We now need to mount some required, volatile filesystems - -.. code-block:: none - - (live)root@ece068908a5b:/# mount -t proc none /proc - (live)root@ece068908a5b:/# mount -t sysfs none /sys - (live)root@ece068908a5b:/# mount -t devtmpfs none /dev - -We now are free to run any command we would like to use for debugging, e.g. -re-installing the failed package after updating the repository. - -.. code-block:: none - - (live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x - Get:1 file:/root/packages ./ InRelease - Ign:1 file:/root/packages ./ InRelease - Get:2 file:/root/packages ./ Release [1235 B] - Get:2 file:/root/packages ./ Release [1235 B] - Get:3 file:/root/packages ./ Release.gpg - Ign:3 file:/root/packages ./ Release.gpg - Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease - Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease - Hit:6 http://deb.debian.org/debian bullseye InRelease - Hit:7 http://deb.debian.org/debian buster InRelease - Hit:8 http://deb.debian.org/debian-security buster/updates InRelease - Hit:9 http://deb.debian.org/debian buster-updates InRelease - Hit:10 http://deb.debian.org/debian buster-backports InRelease - Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease - Reading package lists... Done - N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) - Reading package lists... Done - Building dependency tree - Reading state information... Done - Some packages could not be installed. This may mean that you have - requested an impossible situation or if you are using the unstable - distribution that some required packages have not yet been created - or been moved out of Incoming. - The following information may help to resolve the situation: - - The following packages have unmet dependencies: - vyos-1x : Depends: accel-ppp but it is not installable - E: Unable to correct problems, you have held broken packages. - -Now it's time to fix the package mirror and rerun the last step until the -package installation succeeds again! - -.. _build_custom_packages: - -Linux Kernel -============ - -The Linux kernel used by VyOS is heavily tied to the ISO build process. The -file ``data/defaults.toml`` hosts a TOML definition of the kernel version used -``kernel_version`` and the ``kernel_flavor`` of the kernel which represents the -kernel's LOCAL_VERSION. Both together form the kernel version variable in the -system: - -.. code-block:: none - - vyos@vyos:~$ uname -r - 6.1.52-amd64-vyos - -* Accel-PPP -* Intel NIC drivers -* Inter QAT - -Each of those modules holds a dependency on the kernel version and if you are -lucky enough to receive an ISO build error which sounds like: - -.. code-block:: none - - I: Create initramfs if it does not exist. - Extra argument '6.1.52-amd64-vyos' - Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] - Options: - -k version Specify kernel version or 'all' - -c Create a new initramfs - -u Update an existing initramfs - -d Remove an existing initramfs - -b directory Set alternate boot directory - -v Be verbose - See update-initramfs(8) for further details. - E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. - -The most obvious reasons could be: - -* ``vyos-build`` repo is outdated, please ``git pull`` to update to the latest - release kernel version from us. - -* You have your own custom kernel `*.deb` packages in the `packages` folder but - neglected to create all required out-of tree modules like Accel-PPP, Intel - QAT or Intel NIC drivers - -Building The Kernel -------------------- - -The kernel build is quite easy, most of the required steps can be found in the -``vyos-build/packages/linux-kernel/Jenkinsfile`` but we will walk you through -it. - -Clone the kernel source to `vyos-build/packages/linux-kernel/`: - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel/ - $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git - -Check out the required kernel version - see ``vyos-build/data/defaults.toml`` -file (example uses kernel 4.19.146): - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel/linux - $ git checkout v4.19.146 - Checking out files: 100% (61536/61536), done. - Note: checking out 'v4.19.146'. - - You are in 'detached HEAD' state. You can look around, make experimental - changes and commit them, and you can discard any commits you make in this - state without impacting any branches by performing another checkout. - - If you want to create a new branch to retain commits you create, you may - do so (now or later) by using -b with the checkout command again. Example: - - git checkout -b <new-branch-name> - - HEAD is now at 015e94d0e37b Linux 4.19.146 - -Now we can use the helper script ``build-kernel.sh`` which does all the -necessary voodoo by applying required patches from the -`vyos-build/packages/linux-kernel/patches` folder, copying our kernel -configuration ``x86_64_vyos_defconfig`` to the right location, and finally -building the Debian packages. - -.. note:: Building the kernel will take some time depending on the speed and - quantity of your CPU/cores and disk speed. Expect 20 minutes - (or even longer) on lower end hardware. - -.. code-block:: none - - (18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh - I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source - I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch - patching file Documentation/networking/ip-sysctl.txt - patching file include/linux/inetdevice.h - patching file include/linux/ipv6.h - patching file include/uapi/linux/ip.h - patching file include/uapi/linux/ipv6.h - patching file net/ipv4/devinet.c - Hunk #1 succeeded at 2319 (offset 1 line). - patching file net/ipv6/addrconf.c - patching file net/ipv6/route.c - I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch - patching file fs/notify/inotify/Kconfig - patching file fs/notify/inotify/inotify_user.c - patching file fs/overlayfs/super.c - Hunk #2 succeeded at 1713 (offset 9 lines). - Hunk #3 succeeded at 1739 (offset 9 lines). - Hunk #4 succeeded at 1762 (offset 9 lines). - patching file include/linux/inotify.h - I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch - patching file scripts/package/builddeb - I: make x86_64_vyos_defconfig - HOSTCC scripts/basic/fixdep - HOSTCC scripts/kconfig/conf.o - YACC scripts/kconfig/zconf.tab.c - LEX scripts/kconfig/zconf.lex.c - HOSTCC scripts/kconfig/zconf.tab.o - HOSTLD scripts/kconfig/conf - # - # configuration written to .config - # - I: Generate environment file containing Kernel variable - I: Build Debian Kernel package - UPD include/config/kernel.release - /bin/sh ./scripts/package/mkdebian - dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc - dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos - dpkg-buildpackage: info: source version 4.19.146-1 - dpkg-buildpackage: info: source distribution buster - dpkg-buildpackage: info: source changed by vyos_bld <christian@poessinger.com> - dpkg-buildpackage: info: host architecture amd64 - dpkg-buildpackage: warning: debian/rules is not executable; fixing that - dpkg-source --before-build . - debian/rules build - make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= - SYSTBL arch/x86/include/generated/asm/syscalls_32.h - - ... - - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) - dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) - dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. - dpkg-genbuildinfo --build=binary - dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes - dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list - dpkg-genchanges: info: binary-only upload (no source code included) - dpkg-source --after-build . - dpkg-buildpackage: info: binary-only upload (no source included) - - -In the end you will be presented with the kernel binary packages which you can -then use in your custom ISO build process, by placing all the `*.deb` files in -the vyos-build/packages folder where they will be used automatically when -building VyOS as documented above. - -Firmware -^^^^^^^^ - -If you upgrade your kernel or include new drivers you may need new firmware. -Build a new ``vyos-linux-firmware`` package with the included helper scripts. - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel - $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git - $ ./build-linux-firmware.sh - $ cp vyos-linux-firmware_*.deb ../ - -This tries to automatically detect which blobs are needed based on which drivers -were built. If it fails to find the correct files you can add them manually to -``vyos-build/packages/linux-kernel/build-linux-firmware.sh``: - -.. code-block:: bash - - ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin" - - -Building Out-Of-Tree Modules ----------------------------- - -Building the kernel is one part, but now you also need to build the required -out-of-tree modules so everything is lined up and the ABIs match. To do so, -you can again take a look at ``vyos-build/packages/linux-kernel/Jenkinsfile`` -to see all of the required modules and their selected versions. We will show -you how to build all the current required modules. - -Accel-PPP -^^^^^^^^^ - -First, clone the source code and check out the appropriate version by running: - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel - $ git clone https://github.com/accel-ppp/accel-ppp.git - -We again make use of a helper script and some patches to make the build work. -Just run the following command: - -.. code-block:: none - - $ ./build-accel-ppp.sh - I: Build Accel-PPP Debian package - CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): - The OLD behavior for policy CMP0003 will be removed from a future version - of CMake. - - The cmake-policies(7) manual explains that the OLD behaviors of all - policies are deprecated and that a policy should be set to OLD only under - specific short-term circumstances. Projects should be ported to the NEW - behavior and not rely on setting a policy to OLD. - - -- The C compiler identification is GNU 8.3.0 - - ... - - CPack: Create package using DEB - CPack: Install projects - CPack: - Run preinstall target for: accel-ppp - CPack: - Install project: accel-ppp - CPack: Create package - CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them -to the ``vyos-build/packages`` folder for inclusion during the ISO build. - -Intel NIC -^^^^^^^^^ - -The Intel NIC drivers do not come from a Git repository, instead we just fetch -the tarballs from our mirror and compile them. - -Simply use our wrapper script to build all of the driver modules. - -.. code-block:: none - - ./build-intel-drivers.sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k - I: Compile Kernel module for Intel ixgbe driver - - ... - - I: Building Debian package vyos-intel-iavf - Doing `require 'backports'` is deprecated and will not load any backport in the next major release. - Require just the needed backports instead, or 'backports/latest'. - Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} - Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} - I: Cleanup iavf source - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them -to the ``vyos-build/packages`` folder for inclusion during the ISO build. - -Intel QAT -^^^^^^^^^ -The Intel QAT (Quick Assist Technology) drivers do not come from a Git -repository, instead we just fetch the tarballs from 01.org, Intel's -open-source website. - -Simply use our wrapper script to build all of the driver modules. - -.. code-block:: none - - $ ./build-intel-qat.sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k - I: Compile Kernel module for Intel qat driver - checking for a BSD-compatible install... /usr/bin/install -c - checking whether build environment is sane... yes - checking for a thread-safe mkdir -p... /bin/mkdir -p - checking for gawk... gawk - checking whether make sets $(MAKE)... yes - - ... - - I: Building Debian package vyos-intel-qat - Doing `require 'backports'` is deprecated and will not load any backport in the next major release. - Require just the needed backports instead, or 'backports/latest'. - Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} - Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} - I: Cleanup qat source - - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them -to the ``vyos-build/packages`` folder for inclusion during the ISO build. - - -Packages -======== - -If you are brave enough to build yourself an ISO image containing any modified -package from our GitHub organisation - this is the place to be. - -Any "modified" package may refer to an altered version of e.g. vyos-1x package -that you would like to test before filing a pull request on GitHub. - -Building an ISO with any customized package is in no way different than -building a regular (customized or not) ISO image. Simply place your modified -`*.deb` package inside the `packages` folder within `vyos-build`. The build -process will then pickup your custom package and integrate it into your ISO. - -Troubleshooting -=============== - -Debian APT is not very verbose when it comes to errors. If your ISO build breaks -for whatever reason and you suspect it's a problem with APT dependencies or -installation you can add this small patch which increases the APT verbosity -during ISO build. - -.. stop_vyoslinter - -.. code-block:: diff - - diff --git i/scripts/live-build-config w/scripts/live-build-config - index 1b3b454..3696e4e 100755 - --- i/scripts/live-build-config - +++ w/scripts/live-build-config - @@ -57,7 +57,8 @@ lb config noauto \ - --firmware-binary false \ - --updates true \ - --security true \ - - --apt-options "--yes -oAcquire::Check-Valid-Until=false" \ - + --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \ - + -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \ - --apt-indices false - "${@}" - """ - -.. start_vyoslinter - - - -Virtualization Platforms -======================== - -QEMU ----- - -Run the following command after building the ISO image. - -.. code-block:: none - - $ make qemu - -VMware ------- - -Run the following command after building the QEMU image. - -.. code-block:: none - - $ make vmware - -.. _build_packages: - -******** -Packages -******** - -VyOS itself comes with a bunch of packages that are specific to our system and -thus cannot be found in any Debian mirror. Those packages can be found at the -`VyOS GitHub project`_ in their source format can easily be compiled into -a custom Debian (`*.deb`) package. - -The easiest way to compile your package is with the above mentioned -:ref:`build_docker` container, it includes all required dependencies for -all VyOS related packages. - -Assume we want to build the vyos-1x package on our own and modify it to our -needs. We first need to clone the repository from GitHub. - -.. code-block:: none - - $ git clone https://github.com/vyos/vyos-1x - -Build -===== - -Launch Docker container and build package - -.. code-block:: none - - # For VyOS 1.3 (equuleus, current) - $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash - - # Change to source directory - $ cd vyos-1x - - # Build DEB - $ dpkg-buildpackage -uc -us -tc -b - -After a minute or two you will find the generated DEB packages next to the -vyos-1x source directory: - -.. code-block:: none - - # ls -al ../vyos-1x*.deb - -rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb - -rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb - -Install -======= - -To take your newly created package on a test drive you can simply SCP it to a -running VyOS instance and install the new `*.deb` package over the current -running one. - -Just install using the following commands: - -.. code-block:: none - - vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb - (Reading database ... 58209 files and directories currently installed.) - Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... - Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... - Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... - Processing triggers for rsyslog (8.1901.0-1) ... - -You can also place the generated `*.deb` into your ISO build environment to -include it in a custom iso, see :ref:`build_custom_packages` for more -information. - -.. warning:: Any packages in the packages directory will be added to the iso - during build, replacing the upstream ones. Make sure you delete them (both - the source directories and built deb packages) if you want to build an iso - from purely upstream packages. - - -.. stop_vyoslinter - -.. _Docker: https://www.docker.com -.. _`Docker as non-root`: https://docs.docker.com/engine/install/linux-postinstall -.. _VyOS DockerHub organisation: https://hub.docker.com/u/vyos -.. _repository: https://github.com/vyos/vyos-build -.. _VyOS GitHub project: https://github.com/vyos -.. _`On Debian`: https://docs.docker.com/engine/install/debian/ - -.. start_vyoslinter - diff --git a/docs/contributing/rst-debugging.rst b/docs/contributing/rst-debugging.rst deleted file mode 100644 index e03f3f81..00000000 --- a/docs/contributing/rst-debugging.rst +++ /dev/null @@ -1,206 +0,0 @@ -.. _debugging: - -######### -Debugging -######### - -There are two flags available to aid in debugging configuration scripts. -Since configuration loading issues will manifest during boot, the flags are -passed as kernel boot parameters. - -ISO image build -=============== - -When having trouble compiling your own ISO image or debugging Jenkins issues -you can follow the steps at :ref:`iso_build_issues`. - -System Startup -============== - -The system startup can be debugged (like loading in the configuration -file from ``/config/config.boot``. This can be achieve by extending the -Kernel command-line in the bootloader. - -Kernel ------- - -* ``vyos-debug`` - Adding the parameter to the linux boot line will produce - timing results for the execution of scripts during commit. If one is seeing - an unexpected delay during manual or boot commit, this may be useful in - identifying bottlenecks. The internal flag is ``VYOS_DEBUG``, and is found - in vyatta-cfg_. Output is directed to ``/var/log/vyatta/cfg-stdout.log``. - -* ``vyos-config-debug`` - During development, coding errors can lead to a - commit failure on boot, possibly resulting in a failed initialization of the - CLI. In this circumstance, the kernel boot parameter ``vyos-config-debug`` - will ensure access to the system as user ``vyos``, and will log a Python - stack trace to the file ``/tmp/boot-config-trace``. - File ``boot-config-trace`` will generate only if config loaded with a failure - status. - -Live System -=========== - -A number of flags can be set up to change the behaviour of VyOS at runtime. -These flags can be toggled using either environment variables or creating -files. - -For each feature, a file called ``vyos.feature.debug`` can be created to -toggle the feature on. If a parameter is required it can be placed inside -the file as its first line. - -The file can be placed in ``/tmp`` for one time debugging (as the file -will be removed on reboot) or placed in '/config' to stay permanently. - -For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable -interface debugging. - -It is also possible to set up the debugging using environment variables. -In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG. - -For example running, ``export VYOS_IFCONFIG_DEBUG=""`` on your vbash, -will have the same effect as ``touch /tmp/vyos.ifconfig.debug``. - -* ``ifconfig`` - Once set, all commands used, and their responses received - from the OS, will be presented on the screen for inspection. - -* ``command`` - Once set, all commands used, and their responses received - from the OS, will be presented on the screen for inspection. - -* ``developer`` - Should a command fail, instead of printing a message to the - user explaining how to report issues, the python interpreter will start a - PBD post-mortem session to allow the developer to debug the issue. As the - debugger will wait from input from the developer, it has the capacity to - prevent a router to boot and therefore should only be permanently set up - on production if you are ready to see the OS fail to boot. - -* ``log`` - In some rare cases, it may be useful to see what the OS is doing, - including during boot. This option sends all commands used by VyOS to a - file. The default file is ``/tmp/full-log`` but it can be changed. - -.. note:: In order to retrieve the debug output on the command-line you need to - disable ``vyos-configd`` in addition. This can be run either one-time by - calling ``sudo systemctl stop vyos-configd`` or make this reboot-safe by - calling ``sudo systemctl disable vyos-configd``. - -FRR ---- - -Recent versions use the ``vyos.frr`` framework. The Python class is located -inside our ``vyos-1x:python/vyos/frr.py``. It comes with an embedded debugging/ -(print style) debugger as vyos.ifconfig does. - -To enable debugging just run: ``$ touch /tmp/vyos.frr.debug`` - -Debugging Python Code with PDB ------------------------------- - -Sometimes it might be useful to debug Python code interactively on the live -system rather than a IDE. This can be achieved using pdb. - -Let us assume you want to debug a Python script that is called by an op-mode -command. After you found the script by looking up the op-mode-defitions you -can edit the script in the live system using e.g. vi: -``vi /usr/libexec/vyos/op_mode/show_xyz.py`` - -Insert the following statement right before the section where you want to -investigate a problem (e.g. a statement you see in a backtrace): -``import pdb; pdb.set_trace()`` -Optionally you can surrounded this statement by an ``if`` which only triggers -under the condition you are interested in. - -Once you run ``show xyz`` and your condition is triggered you should be dropped -into the python debugger: - - -.. code-block:: none - - > /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() - -> rule_type = rule.get('type', '') - (Pdb) - -You can type ``help`` to get an overview of the available commands, and -``help command`` to get more information on each command. - -Useful commands are: - -* examine variables using ``pp(var)`` -* continue execution using ``cont`` -* get a backtrace using ``bt`` - -Config Migration Scripts ------------------------- - -When writing a new configuration migrator it may happen that you see an error -when you try to invoke it manually on a development system. This error will -look like: - -.. code-block:: none - - vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot - Traceback (most recent call last): - File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module> - config = ConfigTree(config_file) - File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__ - raise ValueError("Failed to parse config: {0}".format(msg)) - ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax. - -The reason is that the configuration migration backend is rewritten and uses -a new form of "magic string" which is applied on demand when real config -migration is run on boot. When running individual migrators for testing, -you need to convert the "magic string" on your own by: - -.. code-block:: none - - vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot - -Configuration Error on System Boot ----------------------------------- - -Being brave and running the latest rolling releases will sometimes trigger -bugs due to corner cases we missed in our design. Those bugs should be filed -via Phabricator_ but you can help us to narrow down the issue. Login to your -VyOS system and change into configuration mode by typing ``configure``. Now -re-load your boot configuration by simply typing ``load`` followed by return. - -You should now see a Python backtrace which will help us to handle the issue, -please attach it to the Phabricator_ task. - -Boot Timing ------------ - -During the migration and extensive rewrite of functionality from Perl into -Python a significant increase in the overall system boottime was noticed. The -system boot time can be analysed and a graph can be generated in the end which -shows in detail who called whom during the system startup phase. - -This is done by utilizing the ``systemd-bootchart`` package which is now -installed by default on the VyOS 1.3 (equuleus) branch. The configuration is -also versioned so we get comparable results. ``systemd-bootchart`` is configured -using this file: bootchart.conf_ - -To enable boot time graphing change the Kernel commandline and add the following -string: ``init=/usr/lib/systemd/systemd-bootchart`` - -This can also be done permanently by changing ``/boot/grub/grub.cfg``. - -Priorities -========== - -VyOS CLI is all about priorities. Every CLI node has a corresponding -``node.def`` file and possibly an attached script that is executed when the -node is present. Nodes can have a priority, and on system bootup - or any -other ``commit`` to the config all scripts are executed from lowest to highest -priority. This is good as this gives a deterministic behavior. - -To debug issues in priorities or to see what's going on in the background -you can use the ``/opt/vyatta/sbin/priority.pl`` script which lists to you -the execution order of the scripts. - -.. stop_vyoslinter - -.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg -.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf -.. include:: /_include/common-references.txt - -.. start_vyoslinter diff --git a/docs/contributing/rst-development.rst b/docs/contributing/rst-development.rst deleted file mode 100644 index e39af3a5..00000000 --- a/docs/contributing/rst-development.rst +++ /dev/null @@ -1,738 +0,0 @@ -.. _development: - -########### -Development -########### - -All VyOS source code is hosted on GitHub under the VyOS organization which can -be found here: https://github.com/vyos - -Our code is split into several modules. VyOS is composed of multiple individual -packages, some of them are forks of upstream packages and are periodically -synced with upstream, so keeping the whole source under a single repository -would be very inconvenient and slow. There is now an ongoing effort to -consolidate all VyOS-specific framework/config packages into vyos-1x package, -but the basic structure is going to stay the same, just with fewer and fewer -packages while the base code is rewritten from Perl/BASH into Python using and -XML based interface definition for the CLI. - -The repository that contains all the ISO build scripts is: -https://github.com/vyos/vyos-build - -The README.md file will guide you to use the this top level repository. - -Submit a Patch -============== - -Patches are always more than welcome. To have a clean and easy to maintain -repository we have some guidelines when working with Git. A clean repository -eases the automatic generation of a changelog file. - -A good approach for writing commit messages is actually to have a look at the -file(s) history by invoking ``git log path/to/file.txt``. - -.. _prepare_commit: - -Prepare patch/commit --------------------- - -In a big system, such as VyOS, that is comprised of multiple components, it's -impossible to keep track of all the changes and bugs/feature requests in one's -head. We use a bugtracker known as Phabricator_ for it ("issue tracker" would -be a better term, but this one stuck). - -The information is used in three ways: - -* Keep track of the progress (what we've already done in this branch and what - we still need to do). - -* Prepare release notes for upcoming releases - -* Help future maintainers of VyOS (it could be you!) to find out why certain - things have been changed in the codebase or why certain features have been - added - -To make this approach work, every change must be associated with a task number -(prefixed with **T**) and a component. If there is no bug report/feature request -for the changes you are going to make, you have to create a Phabricator_ task -first. Once there is an entry in Phabricator_, you should reference its id in -your commit message, as shown below: - -* ``ddclient: T1030: auto create runtime directories`` -* ``Jenkins: add current Git commit ID to build description`` - -If there is no Phabricator_ reference in the commits of your pull request, we -have to ask you to amend the commit message. Otherwise we will have to reject -it. - -Writing good commit messages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The format should be and is inspired by: https://git-scm.com/book/ch5-2.html -It is also worth reading https://chris.beams.io/posts/git-commit/ - -* A single, short, summary of the commit (recommended 50 characters or less, - not exceeding 80 characters) containing a prefix of the changed component - and the corresponding Phabricator_ reference e.g. ``snmp: T1111:`` or - ``ethernet: T2222:`` - multiple components could be concatenated as in - ``snmp: ethernet: T3333`` - -* In some contexts, the first line is treated as the subject of an email and - the rest of the text as the body. The blank line separating the summary from - the body is critical (unless you omit the body entirely); tools like rebase - can get confused if you run the two together. - -* Followed by a message which describes all the details like: - - * What/why/how something has been changed, makes everyone's life easier when - working with `git bisect` - - * All text of the commit message should be wrapped at 72 characters if - possible which makes reading commit logs easier with ``git log`` on a - standard terminal (which happens to be 80x25) - - * If applicable a reference to a previous commit should be made linking - those commits nicely when browsing the history: ``After commit abcd12ef - ("snmp: this is a headline") a Python import statement is missing, - throwing the following exception: ABCDEF`` - -* Always use the ``-x`` option to the ``git cherry-pick`` command when back or - forward porting an individual commit. This automatically appends the line: - ``(cherry picked from commit <ID>)`` to the original authors commit message - making it easier when bisecting problems. - -* Every change set must be consistent (self containing)! Do not fix multiple - bugs in a single commit. If you already worked on multiple fixes in the same - file use `git add --patch` to only add the parts related to the one issue - into your upcoming commit. - -Limits: - -* We only accept bugfixes in packages other than https://github.com/vyos/vyos-1x - as no new functionality should use the old style templates (``node.def`` and - Perl/BASH code. Use the new style XML/Python interface instead. - -Please submit your patches using the well-known GitHub pull-request against our -repositories found in the VyOS GitHub organisation at https://github.com/vyos - - -Determinine source package --------------------------- - -Suppose you want to make a change in the webproxy script but yet you do not know -which of the many VyOS packages ship this file. You can determine the VyOS -package name in question by using Debian's ``dpkg -S`` command of your running -VyOS installation. - -.. code-block:: none - - vyos@vyos:~ dpkg -S /opt/vyatta/sbin/vyatta-update-webproxy.pl - vyatta-webproxy: /opt/vyatta/sbin/vyatta-update-webproxy.pl - -This means the file in question (``/opt/vyatta/sbin/vyatta-update-webproxy.pl``) -is located in the ``vyatta-webproxy`` package which can be found here: -https://github.com/vyos/vyatta-webproxy - - -Fork Repository and submit Patch --------------------------------- - -Forking the repository and submitting a GitHub pull-request is the preferred -way of submitting your changes to VyOS. You can fork any VyOS repository to your -very own GitHub account by just appending ``/fork`` to any repository's URL on -GitHub. To e.g. fork the ``vyos-1x`` repository, open the following URL in your -favourite browser: https://github.com/vyos/vyos-1x/fork - -You then can proceed with cloning your fork or add a new remote to your local -repository: - -* Clone: ``git clone https://github.com/<user>/vyos-1x.git`` - -* Fork: ``git remote add myfork https://github.com/<user>/vyos-1x.git`` - -In order to record you as the author of the fix please identify yourself to Git -by setting up your name and email. This can be done local for this one and only -repository ``git config`` or globally using ``git config --global``. - -.. code-block:: none - - git config --global user.name "J. Random Hacker" - git config --global user.email "jrhacker@example.net" - -Make your changes and save them. Do the following for all changes files to -record them in your created Git commit: - -* Add file to Git index using ``git add myfile``, or for a whole directory: - ``git add somedir/*`` - -* Commit the changes by calling ``git commit``. Please use a meaningful commit - headline (read above) and don't forget to reference the Phabricator_ ID. - -* Submit the patch ``git push`` and create the GitHub pull-request. - - -Attach patch to Phabricator task --------------------------------- - -Follow the above steps on how to "Fork repository to submit a Patch". Instead -of uploading "pushing" your changes to GitHub you can export the patches/ -commits and send it to maintainers@vyos.net or attach it directly to the bug -(preferred over email) - -* Export last commit to patch file: ``git format-patch`` or export the last two - commits into its appropriate patch files: ``git format-patch -2`` - - -Coding Guidelines -================= - -Like any other project we have some small guidelines about our source code, too. -The rules we have are not there to punish you - the rules are in place to help -us all. By having a consistent coding style it becomes very easy for new -and also longtime contributors to navigate through the sources and all the -implied logic of any one source file.. - -Python 3 **shall** be used. How long can we keep Python 2 alive anyway? No -considerations for Python 2 compatibility **should** be taken at any time. - - -Formatting ----------- - -* Python: Tabs **shall not** be used. Every indentation level should be 4 spaces -* XML: Tabs **shall not** be used. Every indentation level should be 2 spaces - -.. note:: There are extensions to e.g. VIM (xmllint) which will help you to get - your indention levels correct. Add to following to your .vimrc file: - ``au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ - 2>/dev/null`` now you can call the linter using ``gg=G`` in command mode. - - -Text generation -^^^^^^^^^^^^^^^ - -Template processor **should** be used for generating config files. Built-in -string formatting **may** be used for simple line-oriented formats where every -line is self-contained, such as iptables rules. Template processor **must** be -used for structured, multi-line formats such as those used by ISC DHCPd. - -The default template processor for VyOS code is Jinja2_. - - -Summary -------- - -When modifying the source code, remember these rules of the legacy elimination -campaign: - -* No new features in Perl -* No old style command definitions -* No code incompatible with Python3 - - -Python -====== - -The switch to the Python programming language for new code is not merely a -change of the language, but a chance to rethink and improve the programming -approach. - -Let's face it: VyOS is full of spaghetti code where logic for reading the VyOS -config, generating daemon configs, and restarting processes is all mixed up. - -Python (or any other language, for that matter) does not provide automatic -protection from bad design, so we need to also devise design guidelines and -follow them to keep the system extensible and maintainable. - -But we are here to assist you and want to guide you through how you can become -a good VyOS contributor. The rules we have are not there to punish you - the -rules are in place to help us all. What does it mean? By having a consistent -coding style it becomes very easy for new contributors and also longtime -contributors to navigate through the sources and all the implied logic of -the spaghetti code. - -Please use the following template as good starting point when developing new -modules or even rewrite a whole bunch of code in the new style XML/Python -interface. - - -Configuration Script Structure and Behaviour --------------------------------------------- - -Your configuration script or operation mode script which is also written in -Python3 should have a line break on 80 characters. This seems to be a bit odd -nowadays but as some people also work remotely or program using vi(m) this is -a fair good standard which I hope we can rely on. - -In addition this also helps when browsing the GitHub codebase on a mobile -device if you happen to be a crazy scientist. - -.. code-block:: python - - #!/usr/bin/env python3 - # - # Copyright (C) 2020 VyOS maintainers and contributors - # - # This program is free software; you can redistribute it and/or modify - # it under the terms of the GNU General Public License version 2 or later as - # published by the Free Software Foundation. - # - # This program is distributed in the hope that it will be useful, - # but WITHOUT ANY WARRANTY; without even the implied warranty of - # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - # GNU General Public License for more details. - # - # You should have received a copy of the GNU General Public License - # along with this program. If not, see <http://www.gnu.org/licenses/>. - - import sys - - from vyos.config import Config - from vyos import ConfigError - - def get_config(): - if config: - conf = config - else: - conf = Config() - - # Base path to CLI nodes - base = ['...', '...'] - # Convert the VyOS config to an abstract internal representation - config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) - return config_data - - def verify(config): - # Verify that configuration is valid - if invalid: - raise ConfigError("Descriptive message") - return True - - def generate(config): - # Generate daemon configs - pass - - def apply(config): - # Apply the generated configs to the live system - pass - - try: - c = get_config() - verify(c) - generate(c) - apply(c) - except ConfigError as e: - print(e) - sys.exit(1) - -The ``get_config()`` function must convert the VyOS config to an abstract, -internal representation. No other function is allowed to call the ``vyos.config. -Config`` object method directly. The rationale for it is that when config reads -are mixed with other logic, it's very hard to change the config syntax since -you need to weed out every occurrence of the old syntax. If syntax-specific -code is confined to a single function, the rest of the code can be left -untouched as long as the internal representation remains compatible. - -Another advantage is testability of the code. Mocking the entire config -subsystem is hard, while constructing an internal representation by hand is -way simpler. - -The ``verify()`` function takes your internal representation of the config and -checks if it's valid, otherwise it must raise ``ConfigError`` with an error -message that describes the problem and possibly suggests how to fix it. It must -not make any changes to the system. The rationale for it is again testability -and, in the future when the config backend is ready and every script is -rewritten in this fashion, ability to execute commit dry run ("commit test" -like in JunOS) and abort commit before making any changes to the system if an -error is found in any component. - -The ``generate()`` function generates config files for system components. - -The ``apply()`` function applies the generated configuration to the live -system. It should use non-disruptive reload whenever possible. It may execute -disruptive operations such as daemon process restart if a particular component -does not support non-disruptive reload, or when the expected service degradation -is minimal (for example, in case of auxiliary services such as LLDPd). In case -of high impact services such as VPN daemon and routing protocols, when non- -disruptive reload is supported for some but not all types of configuration -changes, scripts authors should make effort to determine if a configuration -change can be done in a non-disruptive way and only resort to disruptive restart -if it cannot be avoided. - -Unless absolutely necessary, configuration scripts should not modify the active -configuration of system components directly. Whenever at all possible, scripts -should generate a configuration file or files that can be applied with a single -command such as reloading a service through systemd init. Inserting statements -one by one is particularly discouraged, for example, when configuring netfilter -rules, saving them to a file and loading it with iptables-restore should always -be preferred to executing iptables directly. - -The ``apply()`` and ``generate()`` functions may ``raise ConfigError`` if, for -example, the daemon failed to start with the updated config. It shouldn't be a -substitute for proper config checking in the ``verify()`` function. All -reasonable effort should be made to verify that generated configuration is -valid and will be accepted by the daemon, including, when necessary, cross- -checks with other VyOS configuration subtrees. - -Exceptions, including ``VyOSError`` (which is raised by ``vyos.config.Config`` -on improper config operations, such as trying to use ``list_nodes()`` on a -non-tag node) should not be silenced or caught and re-raised as config error. -Sure this will not look pretty on user's screen, but it will make way better -bug reports, and help users (and most VyOS users are IT professionals) do their -own debugging as well. - -For easy orientation we suggest you take a look on the ``ntp.py`` or -``interfaces-bonding.py`` (for tag nodes) implementation. Both files can be -found in the vyos-1x_ repository. - - -XML (used for CLI definitions) -============================== - -The bash (or better vbash) completion in VyOS is defined in *templates*. -Templates are text files (called ``node.def``) stored in a directory tree. The -directory names define the command names, and template files define the command -behaviour. Before VyOS 1.2 (crux) this files were created by hand. After a -complex redesign process_ the new style template are automatically generated -from a XML input file. - -XML interface definitions for VyOS come with a RelaxNG schema and are located -in the vyos-1x_ module. This schema is a slightly modified schema from VyConf_ -alias VyOS 2.0 So VyOS 1.2.x interface definitions will be reusable in Nextgen -VyOS Versions with very minimal changes. - -The great thing about schemas is not only that people can know the complete -grammar for certain, but also that it can be automatically verified. The -`scripts/build-command-templates` script that converts the XML definitions to -old style templates also verifies them against the schema, so a bad definition -will cause the package build to fail. I do agree that the format is verbose, but -there is no other format now that would allow this. Besides, a specialized XML -editor can alleviate the issue with verbosity. - -Example: - -.. code-block:: xml - - <?xml version="1.0"?> - <!-- Cron configuration --> - <interfaceDefinition> - <node name="system"> - <children> - <node name="task-scheduler"> - <properties> - <help>Task scheduler settings</help> - </properties> - <children> - <tagNode name="task" owner="${vyos_conf_scripts_dir}/task_scheduler.py"> - <properties> - <help>Scheduled task</help> - <valueHelp> - <format><string></format> - <description>Task name</description> - </valueHelp> - <priority>999</priority> - </properties> - <children> - <leafNode name="crontab-spec"> - <properties> - <help>UNIX crontab time specification string</help> - </properties> - </leafNode> - <leafNode name="interval"> - <properties> - <help>Execution interval</help> - <valueHelp> - <format><minutes></format> - <description>Execution interval in minutes</description> - </valueHelp> - <valueHelp> - <format><minutes>m</format> - <description>Execution interval in minutes</description> - </valueHelp> - <valueHelp> - <format><hours>h</format> - <description>Execution interval in hours</description> - </valueHelp> - <valueHelp> - <format><days>d</format> - <description>Execution interval in days</description> - </valueHelp> - <constraint> - <regex>[1-9]([0-9]*)([mhd]{0,1})</regex> - </constraint> - </properties> - </leafNode> - <node name="executable"> - <properties> - <help>Executable path and arguments</help> - </properties> - <children> - <leafNode name="path"> - <properties> - <help>Path to executable</help> - </properties> - </leafNode> - <leafNode name="arguments"> - <properties> - <help>Arguments passed to the executable</help> - </properties> - </leafNode> - </children> - </node> - </children> - </tagNode> - </children> - </node> - </children> - </node> - </interfaceDefinition> - -Command definitions are purely declarative, and cannot contain any logic. All -logic for generating config files for target applications, restarting services -and so on is implemented in configuration scripts instead. - -GNU Preprocessor ----------------- - -XML interface definition files use the `xml.in` file extension which was -implemented in :vytask:`T1843`. XML interface definitions tend to have a lot of -duplicated code in areas such as: - -* VIF (incl. VIF-S/VIF-C) -* Address -* Description -* Enabled/Disabled - -Instead of supplying all those XML nodes multiple times there are now include -files with predefined features. Brief overview: - -* `IPv4, IPv6 and DHCP(v6)`_ address assignment -* `IPv4, IPv6`_ address assignment -* `VLAN (VIF)`_ definition -* `MAC address`_ assignment - -All interface definition XML input files (.in suffix) will be sent to the GCC -preprocess and the output is stored in the `build/interface-definitions` -folder. The previously mentioned `scripts/build-command-templates` script -operates on the `build/interface-definitions` folder to generate all required -CLI nodes. - -.. code-block:: none - - $ make interface_definitions - install -d -m 0755 build/interface-definitions - install -d -m 0755 build/op-mode-definitions - Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in - Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in - Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in - Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in - Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in - Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in - [...] - - -Guidelines ----------- - -Use of numbers -^^^^^^^^^^^^^^^ - -Use of numbers in command names **should** be avoided unless a number is a -part of a protocol name or similar. Thus, ``protocols ospfv3`` is perfectly -fine, but something like ``server-1`` is questionable at best. - -Help String -^^^^^^^^^^^ - -To ensure uniform look and feel, and improve readability, we should follow a -set of guidelines consistently. - -Capitalization and punctuation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The first word of every help string **must** be capitalized. There **must not** -be a period at the end of help strings. - -Rationale: this seems to be the unwritten standard in network device CLIs, and -a good aesthetic compromise. - -Examples: - -* Good: "Frobnication algorithm" -* Bad: "frobnication algorithm" -* Bad: "Frobnication algorithm." -* Horrible: "frobnication algorithm." - -Use of abbreviations and acronyms -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Abbreviations and acronyms **must** be capitalized. - -Examples: - -* Good: "TCP connection timeout" -* Bad: "tcp connection timeout" -* Horrible: "Tcp connection timeout" - -Acronyms also **must** be capitalized to visually distinguish them from normal -words: - -Examples: - -* Good: RADIUS (as in remote authentication for dial-in user services) -* Bad: radius (unless it's about the distance between a center of a circle and - any of its points) - -Some abbreviations are traditionally written in mixed case. Generally, if it -contains words "over" or "version", the letter **should** be lowercase. If -there's an accepted spelling (especially if defined by an RFC or another -standard), it **must** be followed. - -Examples: - -* Good: PPPoE, IPsec -* Bad: PPPOE, IPSEC -* Bad: pppoe, ipsec - -Use of verbs -~~~~~~~~~~~~ - -Verbs **should** be avoided. If a verb can be omitted, omit it. - -Examples: - -* Good: "TCP connection timeout" -* Bad: "Set TCP connection timeout" - -If a verb is essential, keep it. For example, in the help text of ``set system -ipv6 disable-forwarding``, "Disable IPv6 forwarding on all interfaces" is a -perfectly justified wording. - -Prefer infinitives -~~~~~~~~~~~~~~~~~~ - -Verbs, when they are necessary, **should** be in their infinitive form. - -Examples: - -* Good: "Disable IPv6 forwarding" -* Bad: "Disables IPv6 forwarding" - -Migrating old CLI ------------------ - -.. list-table:: - :widths: 25 25 50 - :header-rows: 1 - - * - Old concept/syntax - - New syntax - - Notes - * - mynode/node.def - - <node name="mynode"> </node> - - Leaf nodes (nodes with values) use <leafNode> tag instead - * - mynode/node.tag , tag: - - <tagNode name="mynode> </node> - - - * - help: My node - - <properties> <help>My node</help> - - - * - val_help: <format>; some string - - <properties> <valueHelp> <format> format </format> <description> some - string </description> - - Do not add angle brackets around the format, they will be inserted - automatically - * - syntax:expression: pattern - - <properties> <constraint> <regex> ... - - <constraintErrorMessage> will be displayed on failure - * - syntax:expression: $VAR(@) in "foo", "bar", "baz" - - None - - Use regex - * - syntax:expression: exec ... - - <properties> <constraint> <validator> <name ="foo" argument="bar"> - - "${vyos_libexecdir}/validators/foo bar $VAR(@)" will be executed, - <constraintErrorMessage> will be displayed on failure - * - syntax:expression: (arithmetic expression) - - None - - External arithmetic validator may be added if there's demand, complex - validation is better left to commit-time scripts - * - priority: 999 - - <properties> <priority>999</priority> - - Please leave a comment explaining why the priority was chosen - (e.g. "after interfaces are configured") - * - multi: - - <properties> <multi/> - - Only applicable to leaf nodes - * - allowed: echo foo bar - - <properties> <completionHelp> <list> foo bar </list> - - - * - allowed: cli-shell-api listNodes vpn ipsec esp-group - - <properties> <completionHelp> <path> vpn ipsec esp-group </path> ... - - - * - allowed: /path/to/script - - <properties> <completionHelp> <script> /path/to/script </script> ... - - - * - default: - - None - - Move default values to scripts - * - commit:expression: - - None - - All commit time checks should be in the verify() function of the script - * - begin:/create:/delete: - - None - - All logic should be in the scripts - -C++ Backend Code -================ - -The CLI parser used in VyOS is a mix of bash, bash-completion helper and the -C++ backend library [vyatta-cfg](https://github.com/vyos/vyatta-cfg). This -section is a reference of common CLI commands and the respective entry point -in the C/C++ code. - -* ``set`` - - - https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352 - - https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549 - - -* ``commit`` - - - https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252 - - -Continuous Integration -====================== - -VyOS makes use of Jenkins_ as our Continuous Integration (CI) service. Our -`VyOS CI`_ server is publicly accessible here: https://ci.vyos.net. You can get -a brief overview of all required components shipped in a VyOS ISO. - -To build our modules we utilize a CI/CD Pipeline script. Each and every VyOS -component comes with it's own ``Jenkinsfile`` which is (more or less) a copy. -The Pipeline utilizes the Docker container from the :ref:`build_iso` section - -but instead of building it from source on every run, we rather always fetch a -fresh copy (if needed) from Dockerhub_. - -Each module is build on demand if a new commit on the branch in question is -found. After a successful run the resulting Debian Package(s) will be deployed -to our Debian repository which is used during build time. It is located here: -http://dev.packages.vyos.net/repositories/. - - -.. stop_vyoslinter - -.. _Jenkins: https://jenkins.io/ -.. _Dockerhub: https://hub.docker.com/u/vyos/ -.. _process: https://blog.vyos.io/vyos-development-digest-10 -.. _VyConf: https://github.com/vyos/vyconf/tree/master/data/schemata -.. _vyos-1x: https://github.com/vyos/vyos-1x/tree/current/schema -.. _Jinja2: https://jinja.palletsprojects.com/ -.. _`IPv4, IPv6 and DHCP(v6)`: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i -.. _`IPv4, IPv6`: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i -.. _`VLAN (VIF)`: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/vif.xml.i -.. _`MAC address`: https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/mac.xml.i - -.. include:: /_include/common-references.txt - -.. start_vyoslinter diff --git a/docs/contributing/rst-issues-features.rst b/docs/contributing/rst-issues-features.rst deleted file mode 100644 index 9b6602f9..00000000 --- a/docs/contributing/rst-issues-features.rst +++ /dev/null @@ -1,80 +0,0 @@ -.. _issues_features: - -####################### -Issues/Feature requests -####################### - -.. _bug_report: - -Bug Report/Issue -================ - -Issues or bugs are found in any software project. VyOS is not an exception. - -All issues should be reported to the developers. This lets the developers know -what is not working properly. Without this sort of feedback every developer -will believe that everything is working correctly. - -I have found a bug, what should I do? -------------------------------------- - -When you believe you have found a bug, it is always a good idea to verify the -issue prior to opening a bug request. - -* Consult the documentation_ to ensure that you have configured your system - correctly -* Get community support via Slack_ or our Forum_ - -Ensure the problem is reproducible ----------------------------------- - -When you are able to verify that it is actually a bug, spend some time to -document how to reproduce the issue. This documentation can be invaluable. - -When you wish to have a developer fix a bug that you found, helping them -reproduce the issue is beneficial to everyone. Be sure to include information -about the hardware you are using, commands that you were running, any other -activities that you may have been doing at the time. This additional -information can be very useful. - -* What were you attempting to achieve? -* What was the configuration prior to the change? -* What commands did you use? Use e.g. ``run show configuration commands`` - -Include output --------------- - -The output you get when you find a bug can provide lots of information. If you -get an error message on the screen, copy it exactly. Having the exact message -can provide detail that the developers can use. Like wise if you have any log -messages that also are from the time of the issue, include those. They may -also contain information that is helpful for the development team. - -Report a Bug ------------- - -In order to open up a bug-report/feature request you need to create yourself -an account on VyOS Phabricator_. On the left side of the specific project (VyOS -1.2 or VyOS 1.3) you will find quick-links for opening a bug-report/feature -request. - -* Provide as much information as you can -* Which version of VyOS are you using? ``run show version`` -* How can we reproduce this Bug? - -.. _feature_request: - -Feature Request -=============== - -You have an idea of how to make VyOS better or you are in need of a specific -feature which all users of VyOS would benefit from? To send a feature request -please search Phabricator_ if there is already a request pending. You can -enhance it or if you don't find one, create a new one by use the quick link in -the left side under the specific project. - -.. _documentation: https://docs.vyos.io -.. _Slack: https://slack.vyos.io -.. _Forum: https://forum.vyos.io - -.. include:: /_include/common-references.txt diff --git a/docs/contributing/rst-testing.rst b/docs/contributing/rst-testing.rst deleted file mode 100644 index 78860c06..00000000 --- a/docs/contributing/rst-testing.rst +++ /dev/null @@ -1,208 +0,0 @@ -.. _testing: - -####### -Testing -####### - -One of the major advantages introduced in VyOS 1.3 is an automated test -framework. When assembling an ISO image multiple things can go wrong badly and -publishing a faulty ISO makes no sense. The user is disappointed by the quality -of the image and the developers get flodded with bug reports over and over -again. - -As the VyOS documentation is not only for users but also for the developers - -and we keep no secret documentation - this section describes how the automated -testing works. - -Jenkins CI -========== - -Our `VyOS CI`_ system is based on Jenkins and builds all our required packages -for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build -Job which builds and tests the VyOS ISO image which is published after a -successful test drive. - -We differentiate in two independent tests, which are both run in parallel by -two separate QEmu instances which are launched via ``make test`` and ``make -testc`` from within the vyos-build_ repository. - -Smoketests -========== - -Smoketests executes predefined VyOS CLI commands and checks if the desired -daemon/service configuration is rendert - that is how to put it "short". - -When and ISO image is assembled by the `VyOS CI`_, the ``BUILD_SMOKETEST`` -parameter is enabled by default, which will extend the ISO configuration line -with the following packages: - -.. code-block:: python - - def CUSTOM_PACKAGES = '' - if (params.BUILD_SMOKETESTS) - CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' - -So if you plan to build your own custom ISO image and want to make use of our -smoketests, ensure that you have the `vyos-1x-smoketest` package installed. - -The ``make test`` command from the vyos-build_ repository will launch a new -QEmu instance and the ISO image is first installed to the virtual harddisk. - -After its first boot into the newly installed system the main Smoketest script -is executed, it can be found here: `/usr/bin/vyos-smoketest` - -The script only searches for executable "test-cases" under -``/usr/libexec/vyos/tests/smoke/cli/`` and executes them one by one. - -.. note:: As Smoketests will alter the system configuration and you are logged - in remote you may loose your connection to the system. - -Manual Smoketest Run --------------------- - -On the other hand - as each test is contain in its own file - one can always -execute a single Smoketest by hand by simply running the Python test scripts. - -Example: - -.. code-block:: none - - vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py - test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok - test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok - test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok - test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok - test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok - test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok - test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok - test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok - test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok - test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok - test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok - test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok - test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok - - ---------------------------------------------------------------------- - Ran 13 tests in 348.191s - - OK - -Interface based tests ---------------------- - -Our smoketests not only test daemons and serives, but also check if what we -configure for an interface works. Thus there is a common base classed named: -``base_interfaces_test.py`` which holds all the common code that an interface -supports and is tested. - -Those common tests consists out of: - -* Add one or more IP addresses -* DHCP client and DHCPv6 prefix delegation -* MTU size -* IP and IPv6 options -* Port description -* Port disable -* VLANs (QinQ and regular 802.1q) -* ... - -.. note:: When you are working on interface configuration and you also want to - test if the Smoketests pass you would normally loose the remote SSH connection - to your :abbr:`DUT (Device Under Test)`. To handle this issue, some of the - interface based tests can be called with an environment variable beforehand - to limit the number of interfaces used in the test. By default all interface - e.g. all Ethernet interfaces are used. - -.. code-block:: none - - vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py - test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok - test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok - test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok - test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok - test_bonding_min_links (__main__.BondingInterfaceTest) ... ok - test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok - test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok - test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok - test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok - test_interface_description (__main__.BondingInterfaceTest) ... ok - test_interface_disable (__main__.BondingInterfaceTest) ... ok - test_interface_ip_options (__main__.BondingInterfaceTest) ... ok - test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok - test_interface_mtu (__main__.BondingInterfaceTest) ... ok - test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok - test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok - test_span_mirror (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok - test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok - test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok - - ---------------------------------------------------------------------- - Ran 23 tests in 244.694s - - OK - -This will limit the `bond` interface test to only make use of `eth1` and `eth2` -as member ports. - -Config Load Tests -================= - -The other part of our tests are called "config load tests". The config load tests -will load - one after another - arbitrary configuration files to test if the -configuration migration scripts work as designed and that a given set of -functionality still can be loaded with a fresh VyOS ISO image. - -The configurations are all derived from production systems and can not only act -as a testcase but also as reference if one wants to enable a certain feature. -The configurations can be found here: -https://github.com/vyos/vyos-1x/tree/current/smoketest/configs - -The entire test is controlled by the main wrapper script ``/usr/bin/vyos-configtest`` -which behaves in the same way as the main smoketest script. It scans the folder -for potential configuration files and issues a ``load`` command one after another. - -Manual config load test ------------------------ - -One is not bound to load all configurations one after another but can also load -individual test configurations on his own. - -.. code-block:: none - - vyos@vyos:~$ configure - load[edit] - - vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small - Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' - Load complete. Use 'commit' to make changes effective. - [edit] - vyos@vyos# compare - [edit interfaces ethernet eth0] - -hw-id 00:50:56:bf:c5:6d - [edit interfaces ethernet eth1] - +duplex auto - -hw-id 00:50:56:b3:38:c5 - +speed auto - [edit interfaces] - -ethernet eth2 { - - hw-id 00:50:56:b3:9c:1d - -} - -vti vti1 { - - address 192.0.2.1/30 - -} - ... - - vyos@vyos# commit - vyos@vyos# - -.. note:: Some of the configurations have preconditions which need to be met. - Those most likely include generation of crypographic keys before the config - can be applied - you will get a commit error otherwise. If you are interested - how those preconditions are fulfilled check the vyos-build_ repository and - the ``scripts/check-qemu-install`` file. - -.. include:: /_include/common-references.txt diff --git a/docs/contributing/rst-upstream-packages.rst b/docs/contributing/rst-upstream-packages.rst deleted file mode 100644 index 5c48bbb3..00000000 --- a/docs/contributing/rst-upstream-packages.rst +++ /dev/null @@ -1,86 +0,0 @@ -.. _upstream_packages: - -Upstream packages ------------------ - -Many base system packages are pulled straight from Debian's main and contrib -repositories, but there are exceptions. - -This chapter lists those exceptions and gives you a brief overview what we -have done on those packages. If you only want to build yourself a fresh ISO -you can completely skip this chapter. It may become interesting once you have -a VyOS deep dive. - -vyos-netplug -^^^^^^^^^^^^ - -Due to issues in the upstream version that sometimes set interfaces down, a -modified version is used. - -The source is located at https://github.com/vyos/vyos-netplug - -In the future, we may switch to using systemd infrastructure instead. Building -it doesn't require a special procedure. - -keepalived -^^^^^^^^^^ - -Keepalived normally isn't updated to newer feature releases between Debian -versions, so we are building it from source. - -Debian does keep their package in git, but it's upstream tarball imported into -git without its original commit history. To be able to merge new tags in, we -keep a fork of the upstream repository with packaging files imported from -Debian at https://github.com/vyos/keepalived-upstream - -strongswan -^^^^^^^^^^ - -Our StrongSWAN build differs from the upstream: - -- strongswan-nm package build is disabled since we don't use NetworkManager -- Patches for DMVPN are merged in - -The source is at https://github.com/vyos/vyos-strongswan - -DMVPN patches are added by this commit: -https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226 - -Our op mode scripts use the python-vici module, which is not included in -Debian's build, and isn't quite easy to integrate in that build. For this -reason we debianize that module by hand now, using this procedure: - -0. Install https://pypi.org/project/stdeb/ -1. `cd vyos-strongswan` -2. `./configure --enable-python-eggs` -3. `cd src/libcharon/plugins/vici/python` -4. `make` -5. `python3 setup.py --command-packages=stdeb.command bdist_deb` - -The package ends up in deb_dist dir. - -mdns-repeater -^^^^^^^^^^^^^ - -This package doesn't exist in Debian. A debianized fork is kept at -https://github.com/vyos/mdns-repeater - -No special build procedure is required. - -udp-broadcast-relay -^^^^^^^^^^^^^^^^^^^ - -This package doesn't exist in Debian. A debianized fork is kept at -https://github.com/vyos/udp-broadcast-relay - -No special build procedure is required. - -hvinfo -^^^^^^ - -A fork with packaging changes for VyOS is kept at https://github.com/vyos/hvinfo - -The original repo is at https://github.com/dmbaturin/hvinfo - -It's an Ada program and requires GNAT and gprbuild for building, dependencies -are properly specified so just follow debuild's suggestions. |