diff options
| author | Yuriy Andamasov <yuriy@vyos.io> | 2026-05-06 14:41:08 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2026-05-06 12:41:08 +0100 |
| commit | 22e34ce5aee24d2fd11f8205522ab7ecdb3c4c5e (patch) | |
| tree | 8d97f7766d9bbd2d0e3a55e5643a60b387308675 /docs/contributing | |
| parent | c21b38dbe24088eaca73dbc8030cfebc898d2186 (diff) | |
| download | vyos-documentation-22e34ce5aee24d2fd11f8205522ab7ecdb3c4c5e.tar.gz vyos-documentation-22e34ce5aee24d2fd11f8205522ab7ecdb3c4c5e.zip | |
Add incremental RST-to-MyST swap mechanism (sagitta) (#1868)
* feat(swap-sagitta): add incremental RST-to-MyST swap mechanism
Backport of the swap mechanism from feat/incremental-myst-swap onto
the sagitta release branch. Built directly on top of origin/sagitta,
so the underlying RST tree is sagitta's (not current's).
Mechanism:
- scripts/import_myst.py — import md from myst/* with md- prefix
- scripts/swap_sources.py — rename md-{name}.md → {name}.md before
Sphinx builds, restore after; writes _build/_swap_state.json and
_build/_swap_exclude.txt
- docs/Makefile — html/dirhtml/pdf/livehtml all run swap → build →
trap restore; explicit `swap` and `restore` targets too
- docs/conf.py — MyST extensions enabled; swap exclude_patterns
loader; _prefer_webp builder hook so html prefers webp over png
Content:
- 202 md-prefixed pages from origin/myst/sagitta (md-{name}.md
alongside each {name}.rst counterpart)
- 1 plain MyST-only page from myst/sagitta where no .rst exists
(already at canonical name on sagitta: docs/copyright.md)
- 240 .webp images from myst/sagitta (added alongside the existing
PNG/JPG so RST builds keep their assets)
- docs/_swap.txt populated with all 202 stems → MyST is served by
default, revert a page by removing its stem from _swap.txt
🤖 Generated by [robots](https://vyos.io)
* feat(conf): copy .md sources into HTML output for plain-text serving
Adds a build-finished hook that mirrors every .md file from the Sphinx
source tree into the HTML output directory verbatim, making unrendered
MyST sources accessible alongside HTML renders at the same URL path.
🤖 Generated by [robots](https://vyos.io)
* docs: address review feedback (backport from PR #1857)
Fix conversion artifacts, typos, and technical inaccuracies applicable
to the sagitta branch: curly quotes, typos (deamonless, cammans,
amdifferent, trough), incorrect firewall command paths, missing closing
brace in zone-policy, peer name inconsistencies, hardcoded passwords
replaced with vault references, and md-*.md exclusion in conf.py.
🤖 Generated by [robots](https://vyos.io)
* docs: port .readthedocs.yml jobs, _ext/vyos.py fallback and swap-script tests from PR #1857
Parity backport from PR #1857 (current) — three pieces were missing on
sagitta.
- .readthedocs.yml: add build.jobs.pre_build / post_build hooks that run
scripts/swap_sources.py --swap before the Sphinx build and --restore
after. Without this, the swap mechanism ships but never runs on RTD
builds for this branch — the swap is a silent no-op.
- docs/_ext/vyos.py: CmdInclude.run() now falls back to nested_parse()
when self.state._renderer is not present. Required for cfgcmd /
opcmd / cmdincludemd directives to render correctly when included
from MyST pages (the swap mechanism's whole point). Sagitta-only
delta on _ext/vyos.py (the path = str(path) line on 224) is
intentionally untouched.
- tests/test_import_myst.py, tests/test_swap_sources.py: tests for the
swap scripts. The scripts on this branch are byte-identical to
current's, so the same tests apply. Travels with the branch so CI
catches per-branch regressions if the scripts ever drift.
🤖 Generated by [robots](https://vyos.io)
* fix(conf): skip md-*.md staging files in _copy_md_sources
Agent-Logs-Url: https://github.com/vyos/vyos-documentation/sessions/919695a7-688d-41b9-89f0-540684625dbc
Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: andamasov <12631358+andamasov@users.noreply.github.com>
Diffstat (limited to 'docs/contributing')
| -rw-r--r-- | docs/contributing/md-build-vyos.md | 876 | ||||
| -rw-r--r-- | docs/contributing/md-debugging.md | 189 | ||||
| -rw-r--r-- | docs/contributing/md-development.md | 708 | ||||
| -rw-r--r-- | docs/contributing/md-issues-features.md | 60 | ||||
| -rw-r--r-- | docs/contributing/md-testing.md | 225 | ||||
| -rw-r--r-- | docs/contributing/md-upstream-packages.md | 77 |
6 files changed, 2135 insertions, 0 deletions
diff --git a/docs/contributing/md-build-vyos.md b/docs/contributing/md-build-vyos.md new file mode 100644 index 00000000..c8074bd7 --- /dev/null +++ b/docs/contributing/md-build-vyos.md @@ -0,0 +1,876 @@ +# Build VyOS + +## Prerequisites + +There are different ways you can build VyOS. + +Building using a `build_docker` container, although not the only way, +is the easiest way as all dependencies are managed for you. However, you can +also set up your own build machine and run a `build_native`. + +<div class="note"> + +<div class="title"> + +Note + +</div> + +Starting with VyOS 1.2 the release model of VyOS has changed. VyOS +is now **free as in speech, but not as in beer**. This means that while +VyOS is still an open source project, the release ISOs are no longer free +and can only be obtained via subscription, or by contributing to the +community. + +The source code remains public and an ISO can be built using the process +outlined in this chapter. + +</div> + +### Native Build + +To build VyOS natively you require a properly configured build host with the +following Debian versions installed: + +- Debian Jessie for VyOS 1.2 (crux) +- Debian Buster for VyOS 1.3 (equuleus) +- Debian Bookworm for VyOS 1.4 (sagitta) +- Debian Bookworm or updated for VyOS 1.5 (circinus, current) - aka the + rolling release + +To start, clone the repository to your local machine: + +``` none +# For VyOS 1.2 (crux) +$ git clone -b crux --single-branch https://github.com/vyos/vyos-build + +# For VyOS 1.3 (equuleus) +$ git clone -b equuleus --single-branch https://github.com/vyos/vyos-build + +# For VyOS 1.4 (sagitta) +$ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build + +# For VyOS 1.5 (circinus, current) +$ git clone -b current --single-branch https://github.com/vyos/vyos-build + +$ cd vyos-build + +# For VyOS 1.2 (crux) and VyOS 1.3 (equuleus) +$ ./configure --architecture amd64 --build-by "j.randomhacker@vyos.io" +$ sudo make iso + +# For VyOS 1.4 (sagitta) and VyOS 1.5 (circinus, current) +$ sudo make clean +$ sudo ./build-vyos-image iso --architecture amd64 --build-by "j.randomhacker@vyos.io" +``` + +For the packages required, you can refer to the `docker/Dockerfile` file +in the [repository](https://github.com/vyos/vyos-build). The `./build-vyos-image` script will also warn you if any +dependencies are missing. + +### Docker + +This will guide you through the process of building a VyOS ISO using [Docker](https://www.docker.com). +This process has been tested on clean installs of Debian Bullseye (11) and +Bookworm (12). + +Installing [Docker](https://www.docker.com) and prerequisites: + +<div class="hint"> + +<div class="title"> + +Hint + +</div> + +Due to the updated version of Docker, the following examples may +become invalid. + +Due to differences in version updates and build processes, content related +to VyOS 1.3 and below is no longer included below. + +</div> + +[On Debian](https://docs.docker.com/engine/install/debian/) + +``` none +# Add Docker's official GPG key: +$ sudo apt-get update +$ sudo apt-get install ca-certificates curl gnupg +$ sudo install -m 0755 -d /etc/apt/keyrings +$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg +$ sudo chmod a+r /etc/apt/keyrings/docker.gpg + +# Add the repository to Apt sources: +$ echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ + https://download.docker.com/linux/debian \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + +$ sudo apt-get update +$ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +``` + +To be able to use [Docker](https://www.docker.com) without `sudo`, the current non-root user must be +added to the `docker` group by calling: `sudo usermod -aG docker yourusername`. + +<div class="hint"> + +<div class="title"> + +Hint + +</div> + +Doing so grants privileges equivalent to the `root` user! It is +recommended to remove the non-root user from the `docker` group after +building the VyOS ISO. See also [Docker as non-root](https://docs.docker.com/engine/install/linux-postinstall). + +</div> + +<div class="note"> + +<div class="title"> + +Note + +</div> + +The build process needs to be built on a local file system, building +on SMB or NFS shares will result in the container failing to build properly! +VirtualBox Drive Share is also not an option as block device operations +are not implemented and the drive is always mounted as "nodev" + +</div> + +#### Build Container + +The container can be built by hand or by fetching the pre-built one from +DockerHub. Using the pre-built containers from the [VyOS DockerHub +organisation](https://hub.docker.com/u/vyos) will ensure that the container is always up-to-date. A rebuild +is triggered once the container changes (please note this will take 2-3 hours +after pushing to the vyos-build repository). + +##### Dockerhub + +To manually download the container from DockerHub, run: + +``` none +$ docker pull vyos/vyos-build:sagitta # For VyOS 1.4 +$ docker pull vyos/vyos-build:current # For rolling release +``` + +##### Build from source + +The container can also be built directly from source: + +``` none +# For VyOS 1.4 (sagitta) +$ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build + +# For VyOS 1.5 (circinus, current) +$ git clone -b current --single-branch https://github.com/vyos/vyos-build + + +$ cd vyos-build +$ docker build -t vyos/vyos-build:sagitta docker # For VyOS 1.4 +$ docker build -t vyos/vyos-build:current docker # For rolling release +``` + +<div class="note"> + +<div class="title"> + +Note + +</div> + +Since VyOS has switched to Debian (12) Bookworm in its `current` +branch, It is recommended to use the official Docker Hub container image +to build `equleus` and `crux`. + +</div> + +#### Tips and Tricks + +You can create yourself some handy Bash aliases to always launch the latest - +per release train (<span class="title-ref">current</span> or <span class="title-ref">sagitta</span>) - container. Add the following to +your `.bash_aliases` file: + +``` none +alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "$HOME/.gitconfig":/etc/gitconfig \ + -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ + -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + vyos/vyos-build:current bash' + +alias vybld_sagitta='docker pull vyos/vyos-build:sagitta && docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "$HOME/.gitconfig":/etc/gitconfig \ + -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ + -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + vyos/vyos-build:sagitta bash' +``` + +Now you are prepared with two new aliases `vybld` and `vybld_sagitta` to +spawn your development containers in your current working directory. + +<div class="note"> + +<div class="title"> + +Note + +</div> + +Some VyOS packages (namely vyos-1x) come with build-time tests which +verify some of the internal library calls that they work as expected. Those +tests are carried out through the Python Unittest module. If you want to +build the `vyos-1x` package (which is our main development package) you +need to start your Docker container using the following argument: +`--sysctl net.ipv6.conf.lo.disable_ipv6=0`, otherwise those tests will +fail. + +</div> + +## Build ISO + +Now as you are aware of the prerequisites we can continue and build our own +ISO from source. For this we have to fetch the latest source code from GitHub. +Please note as this will differ for both <span class="title-ref">current</span> and <span class="title-ref">crux</span>. + +``` none +# For VyOS 1.4 (sagitta) +$ git clone -b sagitta --single-branch https://github.com/vyos/vyos-build + +# For VyOS 1.5 (circinus, current) +$ git clone -b current --single-branch https://github.com/vyos/vyos-build +``` + +Now a fresh build of the VyOS ISO can begin. Change directory to the +`vyos-build` directory and run: + +``` none +$ cd vyos-build + +# For VyOS 1.4 (sagitta) +$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:sagitta bash + +# For VyOS 1.5 (circinus, current) +$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash +``` + +``` none +# For MacOS (crux, equuleus, sagitta) +$ git clone https://github.com/vyos/vyos-utils-misc +$ cd build-tools/macos-build + +# For VyOS 1.2 (crux) +$ os=jessie64 branch=crux make build + +# For VyOS 1.3 (equuleus) +$ os=buster64 branch=equuleus make build + +# For VyOS 1.4 (sagitta) +$ os=buster64 branch=sagitta make build +``` + +Start the build: + +``` none +# For VyOS 1.4 (sagitta) and For VyOS 1.5 (circinus, current) +vyos_bld@8153428c7e1f:/vyos$ sudo make clean +vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image iso --architecture amd64 --build-by "j.randomhacker@vyos.io" +``` + +When the build is successful, the resulting iso can be found inside the +`build` directory as `live-image-[architecture].hybrid.iso`. + +Good luck! + +<div class="hint"> + +<div class="title"> + +Hint + +</div> + +Building VyOS on Windows WSL2 with Docker integrated into WSL2 will +work like a charm. No problems are known so far! + +</div> + +### Customize<span id="build source"></span> + +This ISO can be customized with the following list of configure options. +The full and current list can be generated with `./build-vyos-image --help`: + +``` none +$ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help + I: Checking if packages required for VyOS image build are installed + usage: build-vyos-image [-h] [--architecture ARCHITECTURE] + [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR] + [--debian-security-mirror DEBIAN_SECURITY_MIRROR] + [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] + [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] + [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run] + [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY] + [--custom-package CUSTOM_PACKAGE] + [build_flavor] + + positional arguments: + build_flavor Build flavor + + optional arguments: + -h, --help show this help message and exit + --architecture ARCHITECTURE + Image target architecture (amd64 or arm64) + --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) + --debian-mirror DEBIAN_MIRROR + Debian repository mirror + --debian-security-mirror DEBIAN_SECURITY_MIRROR + Debian security updates mirror + --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR + Debian repository mirror for pbuilder env bootstrap + --vyos-mirror VYOS_MIRROR + VyOS package mirror + --build-type BUILD_TYPE + Build type, release or development + --version VERSION Version number (release builds only) + --build-comment BUILD_COMMENT + Optional build comment + --debug Enable debug output + --dry-run Check build configuration and exit + --custom-apt-entry CUSTOM_APT_ENTRY + Custom APT entry + --custom-apt-key CUSTOM_APT_KEY + Custom APT key file + --custom-package CUSTOM_PACKAGE + Custom package to install from repositories +``` + +#### ISO Build Issues + +There are (rare) situations where building an ISO image is not possible at all +due to a broken package feed in the background. APT is not very good at +reporting the root cause of the issue. Your ISO build will likely fail with a +more or less similar looking error message: + +``` none +The following packages have unmet dependencies: + vyos-1x : Depends: accel-ppp but it is not installable +E: Unable to correct problems, you have held broken packages. +P: Begin unmounting filesystems... +P: Saving caches... +Reading package lists... +Building dependency tree... +Reading state information... +Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB] +Del accel-ppp 1.12.0-95-g59f8e1b [475 kB] +Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB] +Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB] +Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB] +make: *** [Makefile:30: iso] Error 1 +(10:13) vyos_bld ece068908a5b:/vyos [current] # +``` + +To debug the build process and gain additional information of what could be the +root cause, you need to use <span class="title-ref">chroot</span> to change into the build directory. This is +explained in the following step by step procedure: + +``` none +vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash +``` + +We now need to mount some required, volatile filesystems + +``` none +(live)root@ece068908a5b:/# mount -t proc none /proc +(live)root@ece068908a5b:/# mount -t sysfs none /sys +(live)root@ece068908a5b:/# mount -t devtmpfs none /dev +``` + +We now are free to run any command we would like to use for debugging, e.g. +re-installing the failed package after updating the repository. + +``` none +(live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x +Get:1 file:/root/packages ./ InRelease +Ign:1 file:/root/packages ./ InRelease +Get:2 file:/root/packages ./ Release [1235 B] +Get:2 file:/root/packages ./ Release [1235 B] +Get:3 file:/root/packages ./ Release.gpg +Ign:3 file:/root/packages ./ Release.gpg +Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease +Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease +Hit:6 http://deb.debian.org/debian bullseye InRelease +Hit:7 http://deb.debian.org/debian buster InRelease +Hit:8 http://deb.debian.org/debian-security buster/updates InRelease +Hit:9 http://deb.debian.org/debian buster-updates InRelease +Hit:10 http://deb.debian.org/debian buster-backports InRelease +Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease +Reading package lists... Done +N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) +Reading package lists... Done +Building dependency tree +Reading state information... Done +Some packages could not be installed. This may mean that you have +requested an impossible situation or if you are using the unstable +distribution that some required packages have not yet been created +or been moved out of Incoming. +The following information may help to resolve the situation: + +The following packages have unmet dependencies: + vyos-1x : Depends: accel-ppp but it is not installable +E: Unable to correct problems, you have held broken packages. +``` + +Now it's time to fix the package mirror and rerun the last step until the +package installation succeeds again! + +### Linux Kernel + +The Linux kernel used by VyOS is heavily tied to the ISO build process. The +file `data/defaults.toml` hosts a TOML definition of the kernel version used +`kernel_version` and the `kernel_flavor` of the kernel which represents the +kernel's LOCAL_VERSION. Both together form the kernel version variable in the +system: + +``` none +vyos@vyos:~$ uname -r +6.1.52-amd64-vyos +``` + +- Accel-PPP +- Intel NIC drivers +- Inter QAT + +Each of those modules holds a dependency on the kernel version and if you are +lucky enough to receive an ISO build error which sounds like: + +``` none +I: Create initramfs if it does not exist. +Extra argument '6.1.52-amd64-vyos' +Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] +Options: + -k version Specify kernel version or 'all' + -c Create a new initramfs + -u Update an existing initramfs + -d Remove an existing initramfs + -b directory Set alternate boot directory + -v Be verbose +See update-initramfs(8) for further details. +E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. +``` + +The most obvious reasons could be: + +- `vyos-build` repo is outdated, please `git pull` to update to the latest + release kernel version from us. +- You have your own custom kernel <span class="title-ref">\*.deb</span> packages in the <span class="title-ref">packages</span> folder but + neglected to create all required out-of tree modules like Accel-PPP, Intel + QAT or Intel NIC drivers + +#### Building The Kernel + +The kernel build is quite easy, most of the required steps can be found in the +`vyos-build/packages/linux-kernel/Jenkinsfile` but we will walk you through +it. + +Clone the kernel source to \`vyos-build/packages/linux-kernel/\`: + +``` none +$ cd vyos-build/packages/linux-kernel/ +$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git +``` + +Check out the required kernel version - see `vyos-build/data/defaults.toml` +file (example uses kernel 4.19.146): + +``` none +$ cd vyos-build/packages/linux-kernel/linux +$ git checkout v4.19.146 +Checking out files: 100% (61536/61536), done. +Note: checking out 'v4.19.146'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by performing another checkout. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -b with the checkout command again. Example: + + git checkout -b <new-branch-name> + +HEAD is now at 015e94d0e37b Linux 4.19.146 +``` + +Now we can use the helper script `build-kernel.sh` which does all the +necessary voodoo by applying required patches from the +<span class="title-ref">vyos-build/packages/linux-kernel/patches</span> folder, copying our kernel +configuration `x86_64_vyos_defconfig` to the right location, and finally +building the Debian packages. + +<div class="note"> + +<div class="title"> + +Note + +</div> + +Building the kernel will take some time depending on the speed and +quantity of your CPU/cores and disk speed. Expect 20 minutes +(or even longer) on lower end hardware. + +</div> + +``` none +(18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh +I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source +I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch +patching file Documentation/networking/ip-sysctl.txt +patching file include/linux/inetdevice.h +patching file include/linux/ipv6.h +patching file include/uapi/linux/ip.h +patching file include/uapi/linux/ipv6.h +patching file net/ipv4/devinet.c +Hunk #1 succeeded at 2319 (offset 1 line). +patching file net/ipv6/addrconf.c +patching file net/ipv6/route.c +I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch +patching file fs/notify/inotify/Kconfig +patching file fs/notify/inotify/inotify_user.c +patching file fs/overlayfs/super.c +Hunk #2 succeeded at 1713 (offset 9 lines). +Hunk #3 succeeded at 1739 (offset 9 lines). +Hunk #4 succeeded at 1762 (offset 9 lines). +patching file include/linux/inotify.h +I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch +patching file scripts/package/builddeb +I: make x86_64_vyos_defconfig + HOSTCC scripts/basic/fixdep + HOSTCC scripts/kconfig/conf.o + YACC scripts/kconfig/zconf.tab.c + LEX scripts/kconfig/zconf.lex.c + HOSTCC scripts/kconfig/zconf.tab.o + HOSTLD scripts/kconfig/conf +# +# configuration written to .config +# +I: Generate environment file containing Kernel variable +I: Build Debian Kernel package + UPD include/config/kernel.release +/bin/sh ./scripts/package/mkdebian +dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc +dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos +dpkg-buildpackage: info: source version 4.19.146-1 +dpkg-buildpackage: info: source distribution buster +dpkg-buildpackage: info: source changed by vyos_bld <christian@poessinger.com> +dpkg-buildpackage: info: host architecture amd64 +dpkg-buildpackage: warning: debian/rules is not executable; fixing that + dpkg-source --before-build . + debian/rules build +make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= + SYSTBL arch/x86/include/generated/asm/syscalls_32.h + +... + +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) +dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) +dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. + dpkg-genbuildinfo --build=binary + dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes +dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list +dpkg-genchanges: info: binary-only upload (no source code included) + dpkg-source --after-build . +dpkg-buildpackage: info: binary-only upload (no source included) +``` + +In the end you will be presented with the kernel binary packages which you can +then use in your custom ISO build process, by placing all the <span class="title-ref">\*.deb</span> files in +the vyos-build/packages folder where they will be used automatically when +building VyOS as documented above. + +##### Firmware + +If you upgrade your kernel or include new drivers you may need new firmware. +Build a new `vyos-linux-firmware` package with the included helper scripts. + +``` none +$ cd vyos-build/packages/linux-kernel +$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git +$ ./build-linux-firmware.sh +$ cp vyos-linux-firmware_*.deb ../ +``` + +This tries to automatically detect which blobs are needed based on which drivers +were built. If it fails to find the correct files you can add them manually to +`vyos-build/packages/linux-kernel/build-linux-firmware.sh`: + +``` bash +ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin" +``` + +#### Building Out-Of-Tree Modules + +Building the kernel is one part, but now you also need to build the required +out-of-tree modules so everything is lined up and the ABIs match. To do so, +you can again take a look at `vyos-build/packages/linux-kernel/Jenkinsfile` +to see all of the required modules and their selected versions. We will show +you how to build all the current required modules. + +##### Accel-PPP + +First, clone the source code and check out the appropriate version by running: + +``` none +$ cd vyos-build/packages/linux-kernel +$ git clone https://github.com/accel-ppp/accel-ppp.git +``` + +We again make use of a helper script and some patches to make the build work. +Just run the following command: + +``` none +$ ./build-accel-ppp.sh +I: Build Accel-PPP Debian package +CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): + The OLD behavior for policy CMP0003 will be removed from a future version + of CMake. + + The cmake-policies(7) manual explains that the OLD behaviors of all + policies are deprecated and that a policy should be set to OLD only under + specific short-term circumstances. Projects should be ported to the NEW + behavior and not rely on setting a policy to OLD. + +-- The C compiler identification is GNU 8.3.0 + +... + +CPack: Create package using DEB +CPack: Install projects +CPack: - Run preinstall target for: accel-ppp +CPack: - Install project: accel-ppp +CPack: Create package +CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. +``` + +After compiling the packages you will find yourself the newly generated <span class="title-ref">\*.deb</span> +binaries in `vyos-build/packages/linux-kernel` from which you can copy them +to the `vyos-build/packages` folder for inclusion during the ISO build. + +##### Intel NIC + +The Intel NIC drivers do not come from a Git repository, instead we just fetch +the tarballs from our mirror and compile them. + +Simply use our wrapper script to build all of the driver modules. + +``` none +./build-intel-drivers.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k +I: Compile Kernel module for Intel ixgbe driver + +... + +I: Building Debian package vyos-intel-iavf +Doing `require 'backports'` is deprecated and will not load any backport in the next major release. +Require just the needed backports instead, or 'backports/latest'. +Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} +Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} +I: Cleanup iavf source +``` + +After compiling the packages you will find yourself the newly generated <span class="title-ref">\*.deb</span> +binaries in `vyos-build/packages/linux-kernel` from which you can copy them +to the `vyos-build/packages` folder for inclusion during the ISO build. + +##### Intel QAT + +The Intel QAT (Quick Assist Technology) drivers do not come from a Git +repository, instead we just fetch the tarballs from 01.org, Intel's +open-source website. + +Simply use our wrapper script to build all of the driver modules. + +``` none +$ ./build-intel-qat.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k +I: Compile Kernel module for Intel qat driver +checking for a BSD-compatible install... /usr/bin/install -c +checking whether build environment is sane... yes +checking for a thread-safe mkdir -p... /bin/mkdir -p +checking for gawk... gawk +checking whether make sets $(MAKE)... yes + +... + +I: Building Debian package vyos-intel-qat +Doing `require 'backports'` is deprecated and will not load any backport in the next major release. +Require just the needed backports instead, or 'backports/latest'. +Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} +Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} +I: Cleanup qat source +``` + +After compiling the packages you will find yourself the newly generated <span class="title-ref">\*.deb</span> +binaries in `vyos-build/packages/linux-kernel` from which you can copy them +to the `vyos-build/packages` folder for inclusion during the ISO build. + +### Packages + +If you are brave enough to build yourself an ISO image containing any modified +package from our GitHub organisation - this is the place to be. + +Any "modified" package may refer to an altered version of e.g. vyos-1x package +that you would like to test before filing a pull request on GitHub. + +Building an ISO with any customized package is in no way different than +building a regular (customized or not) ISO image. Simply place your modified +<span class="title-ref">\*.deb</span> package inside the <span class="title-ref">packages</span> folder within <span class="title-ref">vyos-build</span>. The build +process will then pickup your custom package and integrate it into your ISO. + +### Troubleshooting + +Debian APT is not very verbose when it comes to errors. If your ISO build breaks +for whatever reason and you suspect it's a problem with APT dependencies or +installation you can add this small patch which increases the APT verbosity +during ISO build. + +``` diff +diff --git i/scripts/live-build-config w/scripts/live-build-config +index 1b3b454..3696e4e 100755 +--- i/scripts/live-build-config ++++ w/scripts/live-build-config +@@ -57,7 +57,8 @@ lb config noauto \ + --firmware-binary false \ + --updates true \ + --security true \ +- --apt-options "--yes -oAcquire::Check-Valid-Until=false" \ ++ --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \ ++ -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \ + --apt-indices false + "${@}" + """ +``` + +### Virtualization Platforms + +#### QEMU + +Run the following command after building the ISO image. + +``` none +$ make qemu +``` + +#### VMware + +Run the following command after building the QEMU image. + +``` none +$ make vmware +``` + +## Packages + +VyOS itself comes with a bunch of packages that are specific to our system and +thus cannot be found in any Debian mirror. Those packages can be found at the +[VyOS GitHub project](https://github.com/vyos) in their source format can easily be compiled into +a custom Debian (<span class="title-ref">\*.deb</span>) package. + +The easiest way to compile your package is with the above mentioned +`build_docker` container, it includes all required dependencies for +all VyOS related packages. + +Assume we want to build the vyos-1x package on our own and modify it to our +needs. We first need to clone the repository from GitHub. + +``` none +$ git clone https://github.com/vyos/vyos-1x +``` + +### Build + +Launch Docker container and build package + +``` none +# For VyOS 1.3 (equuleus, current) +$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash + +# Change to source directory +$ cd vyos-1x + +# Build DEB +$ dpkg-buildpackage -uc -us -tc -b +``` + +After a minute or two you will find the generated DEB packages next to the +vyos-1x source directory: + +``` none +# ls -al ../vyos-1x*.deb +-rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb +-rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb +``` + +### Install + +To take your newly created package on a test drive you can simply SCP it to a +running VyOS instance and install the new <span class="title-ref">\*.deb</span> package over the current +running one. + +Just install using the following commands: + +``` none +vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb +(Reading database ... 58209 files and directories currently installed.) +Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... +Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... +Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... +Processing triggers for rsyslog (8.1901.0-1) ... +``` + +You can also place the generated <span class="title-ref">\*.deb</span> into your ISO build environment to +include it in a custom iso, see `build_custom_packages` for more +information. + +<div class="warning"> + +<div class="title"> + +Warning + +</div> + +Any packages in the packages directory will be added to the iso +during build, replacing the upstream ones. Make sure you delete them (both +the source directories and built deb packages) if you want to build an iso +from purely upstream packages. + +</div> diff --git a/docs/contributing/md-debugging.md b/docs/contributing/md-debugging.md new file mode 100644 index 00000000..b7e66470 --- /dev/null +++ b/docs/contributing/md-debugging.md @@ -0,0 +1,189 @@ +# Debugging + +There are two flags available to aid in debugging configuration scripts. +Since configuration loading issues will manifest during boot, the flags are +passed as kernel boot parameters. + +## ISO image build + +When having trouble compiling your own ISO image or debugging Jenkins issues +you can follow the steps at `iso_build_issues`. + +## System Startup + +The system startup can be debugged (like loading in the configuration +file from `/config/config.boot`. This can be achieve by extending the +Kernel command-line in the bootloader. + +### Kernel + +- `vyos-debug` - Adding the parameter to the linux boot line will produce + timing results for the execution of scripts during commit. If one is seeing + an unexpected delay during manual or boot commit, this may be useful in + identifying bottlenecks. The internal flag is `VYOS_DEBUG`, and is found + in [vyatta-cfg](https://github.com/vyos/vyatta-cfg). Output is directed to `/var/log/vyatta/cfg-stdout.log`. +- `vyos-config-debug` - During development, coding errors can lead to a + commit failure on boot, possibly resulting in a failed initialization of the + CLI\. In this circumstance, the kernel boot parameter `vyos-config-debug` + will ensure access to the system as user `vyos`, and will log a Python + stack trace to the file `/tmp/boot-config-trace`. + File `boot-config-trace` will generate only if config loaded with a failure + status. + +## Live System + +A number of flags can be set up to change the behaviour of VyOS at runtime. +These flags can be toggled using either environment variables or creating +files. + +For each feature, a file called `vyos.feature.debug` can be created to +toggle the feature on. If a parameter is required it can be placed inside +the file as its first line. + +The file can be placed in `/tmp` for one time debugging (as the file +will be removed on reboot) or placed in '/config' to stay permanently. + +For example, `/tmp/vyos.ifconfig.debug` can be created to enable +interface debugging. + +It is also possible to set up the debugging using environment variables. +In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG. + +For example running, `export VYOS_IFCONFIG_DEBUG=""` on your vbash, +will have the same effect as `touch /tmp/vyos.ifconfig.debug`. + +- `ifconfig` - Once set, all commands used, and their responses received + from the OS, will be presented on the screen for inspection. +- `command` - Once set, all commands used, and their responses received + from the OS, will be presented on the screen for inspection. +- `developer` - Should a command fail, instead of printing a message to the + user explaining how to report issues, the python interpreter will start a + PBD post-mortem session to allow the developer to debug the issue. As the + debugger will wait from input from the developer, it has the capacity to + prevent a router to boot and therefore should only be permanently set up + on production if you are ready to see the OS fail to boot. +- `log` - In some rare cases, it may be useful to see what the OS is doing, + including during boot. This option sends all commands used by VyOS to a + file. The default file is `/tmp/full-log` but it can be changed. + +<div class="note"> + +<div class="title"> + +Note + +</div> + +In order to retrieve the debug output on the command-line you need to +disable `vyos-configd` in addition. This can be run either one-time by +calling `sudo systemctl stop vyos-configd` or make this reboot-safe by +calling `sudo systemctl disable vyos-configd`. + +</div> + +### FRR + +Recent versions use the `vyos.frr` framework. The Python class is located +inside our `vyos-1x:python/vyos/frr.py`. It comes with an embedded debugging/ +(print style) debugger as vyos.ifconfig does. + +To enable debugging just run: `$ touch /tmp/vyos.frr.debug` + +### Debugging Python Code with PDB + +Sometimes it might be useful to debug Python code interactively on the live +system rather than a IDE. This can be achieved using pdb. + +Let us assume you want to debug a Python script that is called by an op-mode +command. After you found the script by looking up the op-mode-defitions you +can edit the script in the live system using e.g. vi: +`vi /usr/libexec/vyos/op_mode/show_xyz.py` + +Insert the following statement right before the section where you want to +investigate a problem (e.g. a statement you see in a backtrace): +`import pdb; pdb.set_trace()` +Optionally you can surrounded this statement by an `if` which only triggers +under the condition you are interested in. + +Once you run `show xyz` and your condition is triggered you should be dropped +into the python debugger: + +``` none +> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() +-> rule_type = rule.get('type', '') +(Pdb) +``` + +You can type `help` to get an overview of the available commands, and +`help command` to get more information on each command. + +Useful commands are: + +- examine variables using `pp(var)` +- continue execution using `cont` +- get a backtrace using `bt` + +### Config Migration Scripts + +When writing a new configuration migrator it may happen that you see an error +when you try to invoke it manually on a development system. This error will +look like: + +``` none +vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot +Traceback (most recent call last): + File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module> + config = ConfigTree(config_file) + File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__ + raise ValueError("Failed to parse config: {0}".format(msg)) +ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax. +``` + +The reason is that the configuration migration backend is rewritten and uses +a new form of "magic string" which is applied on demand when real config +migration is run on boot. When running individual migrators for testing, +you need to convert the "magic string" on your own by: + +``` none +vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot +``` + +### Configuration Error on System Boot + +Being brave and running the latest rolling releases will sometimes trigger +bugs due to corner cases we missed in our design. Those bugs should be filed +via [Phabricator]() but you can help us to narrow down the issue. Login to your +VyOS system and change into configuration mode by typing `configure`. Now +re-load your boot configuration by simply typing `load` followed by return. + +You should now see a Python backtrace which will help us to handle the issue, +please attach it to the [Phabricator]() task. + +### Boot Timing + +During the migration and extensive rewrite of functionality from Perl into +Python a significant increase in the overall system boottime was noticed. The +system boot time can be analysed and a graph can be generated in the end which +shows in detail who called whom during the system startup phase. + +This is done by utilizing the `systemd-bootchart` package which is now +installed by default on the VyOS 1.3 (equuleus) branch. The configuration is +also versioned so we get comparable results. `systemd-bootchart` is configured +using this file: [bootchart.conf](https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf) + +To enable boot time graphing change the Kernel commandline and add the following +string: `init=/usr/lib/systemd/systemd-bootchart` + +This can also be done permanently by changing `/boot/grub/grub.cfg`. + +## Priorities + +VyOS CLI is all about priorities. Every CLI node has a corresponding +`node.def` file and possibly an attached script that is executed when the +node is present. Nodes can have a priority, and on system bootup - or any +other `commit` to the config all scripts are executed from lowest to highest +priority. This is good as this gives a deterministic behavior. + +To debug issues in priorities or to see what's going on in the background +you can use the `/opt/vyatta/sbin/priority.pl` script which lists to you +the execution order of the scripts. diff --git a/docs/contributing/md-development.md b/docs/contributing/md-development.md new file mode 100644 index 00000000..cc9fae00 --- /dev/null +++ b/docs/contributing/md-development.md @@ -0,0 +1,708 @@ +# Development + +All VyOS source code is hosted on GitHub under the VyOS organization which can +be found here: <https://github.com/vyos> + +Our code is split into several modules. VyOS is composed of multiple individual +packages, some of them are forks of upstream packages and are periodically +synced with upstream, so keeping the whole source under a single repository +would be very inconvenient and slow. There is now an ongoing effort to +consolidate all VyOS-specific framework/config packages into vyos-1x package, +but the basic structure is going to stay the same, just with fewer and fewer +packages while the base code is rewritten from Perl/BASH into Python using and +XML based interface definition for the CLI. + +The repository that contains all the ISO build scripts is: +<https://github.com/vyos/vyos-build> + +The README.md file will guide you to use the this top level repository. + +## Submit a Patch + +Patches are always more than welcome. To have a clean and easy to maintain +repository we have some guidelines when working with Git. A clean repository +eases the automatic generation of a changelog file. + +A good approach for writing commit messages is actually to have a look at the +file(s) history by invoking `git log path/to/file.txt`. + +### Prepare patch/commit + +In a big system, such as VyOS, that is comprised of multiple components, it's +impossible to keep track of all the changes and bugs/feature requests in one's +head. We use a bugtracker known as [Phabricator]() for it ("issue tracker" would +be a better term, but this one stuck). + +The information is used in three ways: + +- Keep track of the progress (what we've already done in this branch and what + we still need to do). +- Prepare release notes for upcoming releases +- Help future maintainers of VyOS (it could be you!) to find out why certain + things have been changed in the codebase or why certain features have been + added + +To make this approach work, every change must be associated with a task number +(prefixed with **T**) and a component. If there is no bug report/feature request +for the changes you are going to make, you have to create a [Phabricator]() task +first. Once there is an entry in [Phabricator](), you should reference its id in +your commit message, as shown below: + +- `ddclient: T1030: auto create runtime directories` +- `Jenkins: add current Git commit ID to build description` + +If there is no [Phabricator]() reference in the commits of your pull request, we +have to ask you to amend the commit message. Otherwise we will have to reject +it. + +#### Writing good commit messages + +The format should be and is inspired by: <https://git-scm.com/book/ch5-2.html> +It is also worth reading <https://chris.beams.io/posts/git-commit/> + +- A single, short, summary of the commit (recommended 50 characters or less, + not exceeding 80 characters) containing a prefix of the changed component + and the corresponding [Phabricator]() reference e.g. `snmp: T1111:` or + `ethernet: T2222:` - multiple components could be concatenated as in + `snmp: ethernet: T3333` +- In some contexts, the first line is treated as the subject of an email and + the rest of the text as the body. The blank line separating the summary from + the body is critical (unless you omit the body entirely); tools like rebase + can get confused if you run the two together. +- Followed by a message which describes all the details like: + - What/why/how something has been changed, makes everyone's life easier when + working with <span class="title-ref">git bisect</span> + - All text of the commit message should be wrapped at 72 characters if + possible which makes reading commit logs easier with `git log` on a + standard terminal (which happens to be 80x25) + - If applicable a reference to a previous commit should be made linking + those commits nicely when browsing the history: `After commit abcd12ef ("snmp: this is a headline") a Python import statement is missing, throwing the following exception: ABCDEF` +- Always use the `-x` option to the `git cherry-pick` command when back or + forward porting an individual commit. This automatically appends the line: + `(cherry picked from commit <ID>)` to the original authors commit message + making it easier when bisecting problems. +- Every change set must be consistent (self containing)! Do not fix multiple + bugs in a single commit. If you already worked on multiple fixes in the same + file use <span class="title-ref">git add --patch</span> to only add the parts related to the one issue + into your upcoming commit. + +Limits: + +- We only accept bugfixes in packages other than <https://github.com/vyos/vyos-1x> + as no new functionality should use the old style templates (`node.def` and + Perl/BASH code. Use the new style XML/Python interface instead. + +Please submit your patches using the well-known GitHub pull-request against our +repositories found in the VyOS GitHub organisation at <https://github.com/vyos> + +### Determinine source package + +Suppose you want to make a change in the webproxy script but yet you do not know +which of the many VyOS packages ship this file. You can determine the VyOS +package name in question by using Debian's `dpkg -S` command of your running +VyOS installation. + +``` none +vyos@vyos:~ dpkg -S /opt/vyatta/sbin/vyatta-update-webproxy.pl +vyatta-webproxy: /opt/vyatta/sbin/vyatta-update-webproxy.pl +``` + +This means the file in question (`/opt/vyatta/sbin/vyatta-update-webproxy.pl`) +is located in the `vyatta-webproxy` package which can be found here: +<https://github.com/vyos/vyatta-webproxy> + +### Fork Repository and submit Patch + +Forking the repository and submitting a GitHub pull-request is the preferred +way of submitting your changes to VyOS. You can fork any VyOS repository to your +very own GitHub account by just appending `/fork` to any repository's URL on +GitHub. To e.g. fork the `vyos-1x` repository, open the following URL in your +favourite browser: <https://github.com/vyos/vyos-1x/fork> + +You then can proceed with cloning your fork or add a new remote to your local +repository: + +- Clone: `git clone https://github.com/<user>/vyos-1x.git` +- Fork: `git remote add myfork https://github.com/<user>/vyos-1x.git` + +In order to record you as the author of the fix please identify yourself to Git +by setting up your name and email. This can be done local for this one and only +repository `git config` or globally using `git config --global`. + +``` none +git config --global user.name "J. Random Hacker" +git config --global user.email "jrhacker@example.net" +``` + +Make your changes and save them. Do the following for all changes files to +record them in your created Git commit: + +- Add file to Git index using `git add myfile`, or for a whole directory: + `git add somedir/*` +- Commit the changes by calling `git commit`. Please use a meaningful commit + headline (read above) and don't forget to reference the [Phabricator]() ID. +- Submit the patch `git push` and create the GitHub pull-request. + +### Attach patch to Phabricator task + +Follow the above steps on how to "Fork repository to submit a Patch". Instead +of uploading "pushing" your changes to GitHub you can export the patches/ +commits and send it to <maintainers@vyos.net> or attach it directly to the bug +(preferred over email) + +- Export last commit to patch file: `git format-patch` or export the last two + commits into its appropriate patch files: `git format-patch -2` + +## Coding Guidelines + +Like any other project we have some small guidelines about our source code, too. +The rules we have are not there to punish you - the rules are in place to help +us all. By having a consistent coding style it becomes very easy for new +and also longtime contributors to navigate through the sources and all the +implied logic of any one source file.. + +Python 3 **shall** be used. How long can we keep Python 2 alive anyway? No +considerations for Python 2 compatibility **should** be taken at any time. + +### Formatting + +- Python: Tabs **shall not** be used. Every indentation level should be 4 spaces +- XML: Tabs **shall not** be used. Every indentation level should be 2 spaces + +<div class="note"> + +<div class="title"> + +Note + +</div> + +There are extensions to e.g. VIM (xmllint) which will help you to get +your indention levels correct. Add to following to your .vimrc file: +`au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null` now you can call the linter using `gg=G` in command mode. + +</div> + +#### Text generation + +Template processor **should** be used for generating config files. Built-in +string formatting **may** be used for simple line-oriented formats where every +line is self-contained, such as iptables rules. Template processor **must** be +used for structured, multi-line formats such as those used by ISC DHCPd. + +The default template processor for VyOS code is [Jinja2](https://jinja.palletsprojects.com/). + +### Summary + +When modifying the source code, remember these rules of the legacy elimination +campaign: + +- No new features in Perl +- No old style command definitions +- No code incompatible with Python3 + +## Python + +The switch to the Python programming language for new code is not merely a +change of the language, but a chance to rethink and improve the programming +approach. + +Let's face it: VyOS is full of spaghetti code where logic for reading the VyOS +config, generating daemon configs, and restarting processes is all mixed up. + +Python (or any other language, for that matter) does not provide automatic +protection from bad design, so we need to also devise design guidelines and +follow them to keep the system extensible and maintainable. + +But we are here to assist you and want to guide you through how you can become +a good VyOS contributor. The rules we have are not there to punish you - the +rules are in place to help us all. What does it mean? By having a consistent +coding style it becomes very easy for new contributors and also longtime +contributors to navigate through the sources and all the implied logic of +the spaghetti code. + +Please use the following template as good starting point when developing new +modules or even rewrite a whole bunch of code in the new style XML/Python +interface. + +### Configuration Script Structure and Behaviour + +Your configuration script or operation mode script which is also written in +Python3 should have a line break on 80 characters. This seems to be a bit odd +nowadays but as some people also work remotely or program using vi(m) this is +a fair good standard which I hope we can rely on. + +In addition this also helps when browsing the GitHub codebase on a mobile +device if you happen to be a crazy scientist. + +``` python +#!/usr/bin/env python3 +# +# Copyright (C) 2020 VyOS maintainers and contributors +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 or later as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +import sys + +from vyos.config import Config +from vyos import ConfigError + +def get_config(): + if config: + conf = config + else: + conf = Config() + + # Base path to CLI nodes + base = ['...', '...'] + # Convert the VyOS config to an abstract internal representation + config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) + return config_data + +def verify(config): + # Verify that configuration is valid + if invalid: + raise ConfigError("Descriptive message") + return True + +def generate(config): + # Generate daemon configs + pass + +def apply(config): + # Apply the generated configs to the live system + pass + +try: + c = get_config() + verify(c) + generate(c) + apply(c) +except ConfigError as e: + print(e) + sys.exit(1) +``` + +The `get_config()` function must convert the VyOS config to an abstract, +internal representation. No other function is allowed to call the `vyos.config. Config` object method directly. The rationale for it is that when config reads +are mixed with other logic, it's very hard to change the config syntax since +you need to weed out every occurrence of the old syntax. If syntax-specific +code is confined to a single function, the rest of the code can be left +untouched as long as the internal representation remains compatible. + +Another advantage is testability of the code. Mocking the entire config +subsystem is hard, while constructing an internal representation by hand is +way simpler. + +The `verify()` function takes your internal representation of the config and +checks if it's valid, otherwise it must raise `ConfigError` with an error +message that describes the problem and possibly suggests how to fix it. It must +not make any changes to the system. The rationale for it is again testability +and, in the future when the config backend is ready and every script is +rewritten in this fashion, ability to execute commit dry run ("commit test" +like in JunOS) and abort commit before making any changes to the system if an +error is found in any component. + +The `generate()` function generates config files for system components. + +The `apply()` function applies the generated configuration to the live +system. It should use non-disruptive reload whenever possible. It may execute +disruptive operations such as daemon process restart if a particular component +does not support non-disruptive reload, or when the expected service degradation +is minimal (for example, in case of auxiliary services such as LLDPd). In case +of high impact services such as VPN daemon and routing protocols, when non- +disruptive reload is supported for some but not all types of configuration +changes, scripts authors should make effort to determine if a configuration +change can be done in a non-disruptive way and only resort to disruptive restart +if it cannot be avoided. + +Unless absolutely necessary, configuration scripts should not modify the active +configuration of system components directly. Whenever at all possible, scripts +should generate a configuration file or files that can be applied with a single +command such as reloading a service through systemd init. Inserting statements +one by one is particularly discouraged, for example, when configuring netfilter +rules, saving them to a file and loading it with iptables-restore should always +be preferred to executing iptables directly. + +The `apply()` and `generate()` functions may `raise ConfigError` if, for +example, the daemon failed to start with the updated config. It shouldn't be a +substitute for proper config checking in the `verify()` function. All +reasonable effort should be made to verify that generated configuration is +valid and will be accepted by the daemon, including, when necessary, cross- +checks with other VyOS configuration subtrees. + +Exceptions, including `VyOSError` (which is raised by `vyos.config.Config` +on improper config operations, such as trying to use `list_nodes()` on a +non-tag node) should not be silenced or caught and re-raised as config error. +Sure this will not look pretty on user's screen, but it will make way better +bug reports, and help users (and most VyOS users are IT professionals) do their +own debugging as well. + +For easy orientation we suggest you take a look on the `ntp.py` or +`interfaces-bonding.py` (for tag nodes) implementation. Both files can be +found in the [vyos-1x](https://github.com/vyos/vyos-1x/tree/current/schema) repository. + +## XML (used for CLI definitions) + +The bash (or better vbash) completion in VyOS is defined in *templates*. +Templates are text files (called `node.def`) stored in a directory tree. The +directory names define the command names, and template files define the command +behaviour. Before VyOS 1.2 (crux) this files were created by hand. After a +complex redesign [process](https://blog.vyos.io/vyos-development-digest-10) the new style template are automatically generated +from a XML input file. + +XML interface definitions for VyOS come with a RelaxNG schema and are located +in the [vyos-1x](https://github.com/vyos/vyos-1x/tree/current/schema) module. This schema is a slightly modified schema from [VyConf](https://github.com/vyos/vyconf/tree/master/data/schemata) +alias VyOS 2.0 So VyOS 1.2.x interface definitions will be reusable in Nextgen +VyOS Versions with very minimal changes. + +The great thing about schemas is not only that people can know the complete +grammar for certain, but also that it can be automatically verified. The +<span class="title-ref">scripts/build-command-templates</span> script that converts the XML definitions to +old style templates also verifies them against the schema, so a bad definition +will cause the package build to fail. I do agree that the format is verbose, but +there is no other format now that would allow this. Besides, a specialized XML +editor can alleviate the issue with verbosity. + +Example: + +``` xml +<?xml version="1.0"?> +<!-- Cron configuration --> +<interfaceDefinition> + <node name="system"> + <children> + <node name="task-scheduler"> + <properties> + <help>Task scheduler settings</help> + </properties> + <children> + <tagNode name="task" owner="${vyos_conf_scripts_dir}/task_scheduler.py"> + <properties> + <help>Scheduled task</help> + <valueHelp> + <format><string></format> + <description>Task name</description> + </valueHelp> + <priority>999</priority> + </properties> + <children> + <leafNode name="crontab-spec"> + <properties> + <help>UNIX crontab time specification string</help> + </properties> + </leafNode> + <leafNode name="interval"> + <properties> + <help>Execution interval</help> + <valueHelp> + <format><minutes></format> + <description>Execution interval in minutes</description> + </valueHelp> + <valueHelp> + <format><minutes>m</format> + <description>Execution interval in minutes</description> + </valueHelp> + <valueHelp> + <format><hours>h</format> + <description>Execution interval in hours</description> + </valueHelp> + <valueHelp> + <format><days>d</format> + <description>Execution interval in days</description> + </valueHelp> + <constraint> + <regex>[1-9]([0-9]*)([mhd]{0,1})</regex> + </constraint> + </properties> + </leafNode> + <node name="executable"> + <properties> + <help>Executable path and arguments</help> + </properties> + <children> + <leafNode name="path"> + <properties> + <help>Path to executable</help> + </properties> + </leafNode> + <leafNode name="arguments"> + <properties> + <help>Arguments passed to the executable</help> + </properties> + </leafNode> + </children> + </node> + </children> + </tagNode> + </children> + </node> + </children> + </node> +</interfaceDefinition> +``` + +Command definitions are purely declarative, and cannot contain any logic. All +logic for generating config files for target applications, restarting services +and so on is implemented in configuration scripts instead. + +### GNU Preprocessor + +XML interface definition files use the <span class="title-ref">xml.in</span> file extension which was +implemented in `T1843`. XML interface definitions tend to have a lot of +duplicated code in areas such as: + +- VIF (incl. VIF-S/VIF-C) +- Address +- Description +- Enabled/Disabled + +Instead of supplying all those XML nodes multiple times there are now include +files with predefined features. Brief overview: + +- [IPv4, IPv6 and DHCP(v6)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i) address assignment +- [IPv4, IPv6](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i) address assignment +- [VLAN (VIF)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/vif.xml.i) definition +- [MAC address](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/mac.xml.i) assignment + +All interface definition XML input files (.in suffix) will be sent to the GCC +preprocess and the output is stored in the <span class="title-ref">build/interface-definitions</span> +folder. The previously mentioned <span class="title-ref">scripts/build-command-templates</span> script +operates on the <span class="title-ref">build/interface-definitions</span> folder to generate all required +CLI nodes. + +``` none +$ make interface_definitions +install -d -m 0755 build/interface-definitions +install -d -m 0755 build/op-mode-definitions +Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in +Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in +Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in +Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in +Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in +Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in +[...] +``` + +### Guidelines + +#### Use of numbers + +Use of numbers in command names **should** be avoided unless a number is a +part of a protocol name or similar. Thus, `protocols ospfv3` is perfectly +fine, but something like `server-1` is questionable at best. + +#### Help String + +To ensure uniform look and feel, and improve readability, we should follow a +set of guidelines consistently. + +##### Capitalization and punctuation + +The first word of every help string **must** be capitalized. There **must not** +be a period at the end of help strings. + +Rationale: this seems to be the unwritten standard in network device CLIs, and +a good aesthetic compromise. + +Examples: + +- Good: "Frobnication algorithm" +- Bad: "frobnication algorithm" +- Bad: "Frobnication algorithm." +- Horrible: "frobnication algorithm." + +##### Use of abbreviations and acronyms + +Abbreviations and acronyms **must** be capitalized. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "tcp connection timeout" +- Horrible: "Tcp connection timeout" + +Acronyms also **must** be capitalized to visually distinguish them from normal +words: + +Examples: + +- Good: RADIUS (as in remote authentication for dial-in user services) +- Bad: radius (unless it's about the distance between a center of a circle and + any of its points) + +Some abbreviations are traditionally written in mixed case. Generally, if it +contains words "over" or "version", the letter **should** be lowercase. If +there's an accepted spelling (especially if defined by an RFC or another +standard), it **must** be followed. + +Examples: + +- Good: PPPoE, IPsec +- Bad: PPPOE, IPSEC +- Bad: pppoe, ipsec + +##### Use of verbs + +Verbs **should** be avoided. If a verb can be omitted, omit it. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "Set TCP connection timeout" + +If a verb is essential, keep it. For example, in the help text of `set system ipv6 disable-forwarding`, "Disable IPv6 forwarding on all interfaces" is a +perfectly justified wording. + +##### Prefer infinitives + +Verbs, when they are necessary, **should** be in their infinitive form. + +Examples: + +- Good: "Disable IPv6 forwarding" +- Bad: "Disables IPv6 forwarding" + +### Migrating old CLI + +<table> +<colgroup> +<col style="width: 25%" /> +<col style="width: 25%" /> +<col style="width: 50%" /> +</colgroup> +<thead> +<tr> +<th>Old concept/syntax</th> +<th>New syntax</th> +<th>Notes</th> +</tr> +</thead> +<tbody> +<tr> +<td>mynode/node.def</td> +<td><node name="mynode"> </node></td> +<td>Leaf nodes (nodes with values) use <leafNode> tag instead</td> +</tr> +<tr> +<td>mynode/node.tag , tag:</td> +<td><tagNode name="mynode> </node></td> +<td></td> +</tr> +<tr> +<td>help: My node</td> +<td><properties> <help>My node</help></td> +<td></td> +</tr> +<tr> +<td>val_help: <format>; some string</td> +<td><properties> <valueHelp> <format> format </format> <description> some +string </description></td> +<td>Do not add angle brackets around the format, they will be inserted +automatically</td> +</tr> +<tr> +<td>syntax:expression: pattern</td> +<td><properties> <constraint> <regex> ...</td> +<td><constraintErrorMessage> will be displayed on failure</td> +</tr> +<tr> +<td>syntax:expression: $VAR(@) in "foo", "bar", "baz"</td> +<td>None</td> +<td>Use regex</td> +</tr> +<tr> +<td>syntax:expression: exec ...</td> +<td><properties> <constraint> <validator> <name ="foo" argument="bar"></td> +<td>"${vyos_libexecdir}/validators/foo bar $VAR(@)" will be executed, +<constraintErrorMessage> will be displayed on failure</td> +</tr> +<tr> +<td>syntax:expression: (arithmetic expression)</td> +<td>None</td> +<td>External arithmetic validator may be added if there's demand, complex +validation is better left to commit-time scripts</td> +</tr> +<tr> +<td>priority: 999</td> +<td><properties> <priority>999</priority></td> +<td>Please leave a comment explaining why the priority was chosen +(e.g. "after interfaces are configured")</td> +</tr> +<tr> +<td>multi:</td> +<td><properties> <multi/></td> +<td>Only applicable to leaf nodes</td> +</tr> +<tr> +<td>allowed: echo foo bar</td> +<td><properties> <completionHelp> <list> foo bar </list></td> +<td></td> +</tr> +<tr> +<td>allowed: cli-shell-api listNodes vpn ipsec esp-group</td> +<td><properties> <completionHelp> <path> vpn ipsec esp-group </path> ...</td> +<td></td> +</tr> +<tr> +<td>allowed: /path/to/script</td> +<td><properties> <completionHelp> <script> /path/to/script </script> ...</td> +<td></td> +</tr> +<tr> +<td>default:</td> +<td>None</td> +<td>Move default values to scripts</td> +</tr> +<tr> +<td>commit:expression:</td> +<td>None</td> +<td>All commit time checks should be in the verify() function of the script</td> +</tr> +<tr> +<td>begin:/create:/delete:</td> +<td>None</td> +<td>All logic should be in the scripts</td> +</tr> +</tbody> +</table> + +## C++ Backend Code + +The CLI parser used in VyOS is a mix of bash, bash-completion helper and the +C++ backend library \[vyatta-cfg\](<https://github.com/vyos/vyatta-cfg>). This +section is a reference of common CLI commands and the respective entry point +in the C/C++ code. + +- `set` + - <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352> + - <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549> +- `commit` + - <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252> + +## Continuous Integration + +VyOS makes use of [Jenkins](https://jenkins.io/) as our Continuous Integration (CI) service. Our +[VyOS CI]() server is publicly accessible here: <https://ci.vyos.net>. You can get +a brief overview of all required components shipped in a VyOS ISO. + +To build our modules we utilize a CI/CD Pipeline script. Each and every VyOS +component comes with it's own `Jenkinsfile` which is (more or less) a copy. +The Pipeline utilizes the Docker container from the `build_iso` section - +but instead of building it from source on every run, we rather always fetch a +fresh copy (if needed) from [Dockerhub](https://hub.docker.com/u/vyos/). + +Each module is build on demand if a new commit on the branch in question is +found. After a successful run the resulting Debian Package(s) will be deployed +to our Debian repository which is used during build time. It is located here: +<http://dev.packages.vyos.net/repositories/>. diff --git a/docs/contributing/md-issues-features.md b/docs/contributing/md-issues-features.md new file mode 100644 index 00000000..c481fec5 --- /dev/null +++ b/docs/contributing/md-issues-features.md @@ -0,0 +1,60 @@ +# Issues/Feature requests + +## Bug Report/Issue + +Issues or bugs are found in any software project. VyOS is not an exception. + +All issues should be reported to the developers. This lets the developers know +what is not working properly. Without this sort of feedback every developer +will believe that everything is working correctly. + +### I have found a bug, what should I do? + +When you believe you have found a bug, it is always a good idea to verify the +issue prior to opening a bug request. + +- Consult the [documentation](https://docs.vyos.io) to ensure that you have configured your system + correctly +- Get community support via [Slack](https://slack.vyos.io) or our [Forum](https://forum.vyos.io) + +### Ensure the problem is reproducible + +When you are able to verify that it is actually a bug, spend some time to +document how to reproduce the issue. This documentation can be invaluable. + +When you wish to have a developer fix a bug that you found, helping them +reproduce the issue is beneficial to everyone. Be sure to include information +about the hardware you are using, commands that you were running, any other +activities that you may have been doing at the time. This additional +information can be very useful. + +- What were you attempting to achieve? +- What was the configuration prior to the change? +- What commands did you use? Use e.g. `run show configuration commands` + +### Include output + +The output you get when you find a bug can provide lots of information. If you +get an error message on the screen, copy it exactly. Having the exact message +can provide detail that the developers can use. Like wise if you have any log +messages that also are from the time of the issue, include those. They may +also contain information that is helpful for the development team. + +### Report a Bug + +In order to open up a bug-report/feature request you need to create yourself +an account on VyOS [Phabricator](). On the left side of the specific project (VyOS +1.2 or VyOS 1.3) you will find quick-links for opening a bug-report/feature +request. + +- Provide as much information as you can +- Which version of VyOS are you using? `run show version` +- How can we reproduce this Bug? + +## Feature Request + +You have an idea of how to make VyOS better or you are in need of a specific +feature which all users of VyOS would benefit from? To send a feature request +please search [Phabricator]() if there is already a request pending. You can +enhance it or if you don't find one, create a new one by use the quick link in +the left side under the specific project. diff --git a/docs/contributing/md-testing.md b/docs/contributing/md-testing.md new file mode 100644 index 00000000..4f406e05 --- /dev/null +++ b/docs/contributing/md-testing.md @@ -0,0 +1,225 @@ +# Testing + +One of the major advantages introduced in VyOS 1.3 is an automated test +framework. When assembling an ISO image multiple things can go wrong badly and +publishing a faulty ISO makes no sense. The user is disappointed by the quality +of the image and the developers get flodded with bug reports over and over +again. + +As the VyOS documentation is not only for users but also for the developers - +and we keep no secret documentation - this section describes how the automated +testing works. + +## Jenkins CI + +Our [VyOS CI]() system is based on Jenkins and builds all our required packages +for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build +Job which builds and tests the VyOS ISO image which is published after a +successful test drive. + +We differentiate in two independent tests, which are both run in parallel by +two separate QEmu instances which are launched via `make test` and `make testc` from within the [vyos-build]() repository. + +## Smoketests + +Smoketests executes predefined VyOS CLI commands and checks if the desired +daemon/service configuration is rendert - that is how to put it "short". + +When and ISO image is assembled by the [VyOS CI](), the `BUILD_SMOKETEST` +parameter is enabled by default, which will extend the ISO configuration line +with the following packages: + +``` python +def CUSTOM_PACKAGES = '' + if (params.BUILD_SMOKETESTS) + CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' +``` + +So if you plan to build your own custom ISO image and want to make use of our +smoketests, ensure that you have the <span class="title-ref">vyos-1x-smoketest</span> package installed. + +The `make test` command from the [vyos-build]() repository will launch a new +QEmu instance and the ISO image is first installed to the virtual harddisk. + +After its first boot into the newly installed system the main Smoketest script +is executed, it can be found here: <span class="title-ref">/usr/bin/vyos-smoketest</span> + +The script only searches for executable "test-cases" under +`/usr/libexec/vyos/tests/smoke/cli/` and executes them one by one. + +<div class="note"> + +<div class="title"> + +Note + +</div> + +As Smoketests will alter the system configuration and you are logged +in remote you may loose your connection to the system. + +</div> + +### Manual Smoketest Run + +On the other hand - as each test is contain in its own file - one can always +execute a single Smoketest by hand by simply running the Python test scripts. + +Example: + +``` none +vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py +test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok +test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok +test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok +test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok +test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok +test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok +test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok +test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok +test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok +test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok +test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok +test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok +test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok + +---------------------------------------------------------------------- +Ran 13 tests in 348.191s + +OK +``` + +### Interface based tests + +Our smoketests not only test daemons and serives, but also check if what we +configure for an interface works. Thus there is a common base classed named: +`base_interfaces_test.py` which holds all the common code that an interface +supports and is tested. + +Those common tests consists out of: + +- Add one or more IP addresses +- DHCP client and DHCPv6 prefix delegation +- MTU size +- IP and IPv6 options +- Port description +- Port disable +- VLANs (QinQ and regular 802.1q) +- ... + +<div class="note"> + +<div class="title"> + +Note + +</div> + +When you are working on interface configuration and you also want to +test if the Smoketests pass you would normally loose the remote SSH connection +to your `DUT (Device Under Test)`. To handle this issue, some of the +interface based tests can be called with an environment variable beforehand +to limit the number of interfaces used in the test. By default all interface +e.g. all Ethernet interfaces are used. + +</div> + +``` none +vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py +test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok +test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok +test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok +test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok +test_bonding_min_links (__main__.BondingInterfaceTest) ... ok +test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok +test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok +test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok +test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok +test_interface_description (__main__.BondingInterfaceTest) ... ok +test_interface_disable (__main__.BondingInterfaceTest) ... ok +test_interface_ip_options (__main__.BondingInterfaceTest) ... ok +test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok +test_interface_mtu (__main__.BondingInterfaceTest) ... ok +test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok +test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok +test_span_mirror (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok +test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok +test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok + +---------------------------------------------------------------------- +Ran 23 tests in 244.694s + +OK +``` + +This will limit the <span class="title-ref">bond</span> interface test to only make use of <span class="title-ref">eth1</span> and <span class="title-ref">eth2</span> +as member ports. + +## Config Load Tests + +The other part of our tests are called "config load tests". The config load tests +will load - one after another - arbitrary configuration files to test if the +configuration migration scripts work as designed and that a given set of +functionality still can be loaded with a fresh VyOS ISO image. + +The configurations are all derived from production systems and can not only act +as a testcase but also as reference if one wants to enable a certain feature. +The configurations can be found here: +<https://github.com/vyos/vyos-1x/tree/current/smoketest/configs> + +The entire test is controlled by the main wrapper script `/usr/bin/vyos-configtest` +which behaves in the same way as the main smoketest script. It scans the folder +for potential configuration files and issues a `load` command one after another. + +### Manual config load test + +One is not bound to load all configurations one after another but can also load +individual test configurations on his own. + +``` none +vyos@vyos:~$ configure +load[edit] + +vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small +Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' +Load complete. Use 'commit' to make changes effective. +[edit] +vyos@vyos# compare +[edit interfaces ethernet eth0] +-hw-id 00:50:56:bf:c5:6d +[edit interfaces ethernet eth1] ++duplex auto +-hw-id 00:50:56:b3:38:c5 ++speed auto +[edit interfaces] +-ethernet eth2 { +- hw-id 00:50:56:b3:9c:1d +-} +-vti vti1 { +- address 192.0.2.1/30 +-} +... + +vyos@vyos# commit +vyos@vyos# +``` + +<div class="note"> + +<div class="title"> + +Note + +</div> + +Some of the configurations have preconditions which need to be met. +Those most likely include generation of crypographic keys before the config +can be applied - you will get a commit error otherwise. If you are interested +how those preconditions are fulfilled check the [vyos-build]() repository and +the `scripts/check-qemu-install` file. + +</div> diff --git a/docs/contributing/md-upstream-packages.md b/docs/contributing/md-upstream-packages.md new file mode 100644 index 00000000..bc0eeed1 --- /dev/null +++ b/docs/contributing/md-upstream-packages.md @@ -0,0 +1,77 @@ +# Upstream packages + +Many base system packages are pulled straight from Debian's main and contrib +repositories, but there are exceptions. + +This chapter lists those exceptions and gives you a brief overview what we +have done on those packages. If you only want to build yourself a fresh ISO +you can completely skip this chapter. It may become interesting once you have +a VyOS deep dive. + +## vyos-netplug + +Due to issues in the upstream version that sometimes set interfaces down, a +modified version is used. + +The source is located at <https://github.com/vyos/vyos-netplug> + +In the future, we may switch to using systemd infrastructure instead. Building +it doesn't require a special procedure. + +## keepalived + +Keepalived normally isn't updated to newer feature releases between Debian +versions, so we are building it from source. + +Debian does keep their package in git, but it's upstream tarball imported into +git without its original commit history. To be able to merge new tags in, we +keep a fork of the upstream repository with packaging files imported from +Debian at <https://github.com/vyos/keepalived-upstream> + +## strongswan + +Our StrongSWAN build differs from the upstream: + +- strongswan-nm package build is disabled since we don't use NetworkManager +- Patches for DMVPN are merged in + +The source is at <https://github.com/vyos/vyos-strongswan> + +DMVPN patches are added by this commit: +<https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226> + +Our op mode scripts use the python-vici module, which is not included in +Debian's build, and isn't quite easy to integrate in that build. For this +reason we debianize that module by hand now, using this procedure: + +0. Install <https://pypi.org/project/stdeb/> +1. <span class="title-ref">cd vyos-strongswan</span> +2. <span class="title-ref">./configure --enable-python-eggs</span> +3. <span class="title-ref">cd src/libcharon/plugins/vici/python</span> +4. <span class="title-ref">make</span> +5. <span class="title-ref">python3 setup.py --command-packages=stdeb.command bdist_deb</span> + +The package ends up in deb_dist dir. + +## mdns-repeater + +This package doesn't exist in Debian. A debianized fork is kept at +<https://github.com/vyos/mdns-repeater> + +No special build procedure is required. + +## udp-broadcast-relay + +This package doesn't exist in Debian. A debianized fork is kept at +<https://github.com/vyos/udp-broadcast-relay> + +No special build procedure is required. + +## hvinfo + +A fork with packaging changes for VyOS is kept at <https://github.com/vyos/hvinfo> + +The original repo is at <https://github.com/dmbaturin/hvinfo> + +It's an Ada program and requires GNAT and gprbuild for building, dependencies +are properly specified so just follow debuild's suggestions. |