From 5d6fa52b8985f8068314aba26878a1d7d5cb84e5 Mon Sep 17 00:00:00 2001 From: Yuriy Andamasov Date: Wed, 6 May 2026 20:42:32 +0300 Subject: feat: flip swap mechanism — MD as primary, RST as override (Phase 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the first of three phases inverting the per-page swap mechanism so MD becomes the canonical primary and RST becomes the rare override. Phase 1 — file renames + conf.py exclude_patterns flip only: - Rename docs/**/md-.md to docs/**/.md (drop md- prefix) for all 254 stems previously listed in docs/_swap.txt - Rename docs/**/.rst to docs/**/rst-.rst (add rst- prefix) for the same 254 stems - Repurpose docs/_swap.txt as docs/_rst_overrides.txt; initially empty comment-only since no pages need the RST fallback right now - conf.py exclude_patterns flipped: rst-*.rst is now excluded by default instead of md-*.md - conf.py runtime-artifact references updated to _rst_override_state.json and _md_exclude.txt (Phase 2 will rewrite swap_sources.py to produce these names; for now no swap script runs because overrides list is empty) Phase 2 (next commit on this branch) will rewrite scripts/swap_sources.py with inverted rename direction, delete scripts/import_myst.py + tests, and update tests/test_swap_sources.py for the new semantics. Phase 3 will be the cleanup pass and ready-for-review flip. Generated by robots https://vyos.io --- docs/contributing/build-vyos.md | 730 ++++++++++++++++++++++++++ docs/contributing/build-vyos.rst | 759 ---------------------------- docs/contributing/cla.md | 45 ++ docs/contributing/cla.rst | 45 -- docs/contributing/debugging.md | 204 ++++++++ docs/contributing/debugging.rst | 223 -------- docs/contributing/development.md | 549 ++++++++++++++++++++ docs/contributing/development.rst | 588 --------------------- docs/contributing/index.md | 13 + docs/contributing/index.rst | 14 - docs/contributing/issues-features.md | 122 +++++ docs/contributing/issues-features.rst | 130 ----- docs/contributing/md-build-vyos.md | 730 -------------------------- docs/contributing/md-cla.md | 45 -- docs/contributing/md-debugging.md | 204 -------- docs/contributing/md-development.md | 549 -------------------- docs/contributing/md-index.md | 13 - docs/contributing/md-issues-features.md | 122 ----- docs/contributing/md-testing.md | 207 -------- docs/contributing/md-upstream-packages.md | 147 ------ docs/contributing/rst-build-vyos.rst | 759 ++++++++++++++++++++++++++++ docs/contributing/rst-cla.rst | 45 ++ docs/contributing/rst-debugging.rst | 223 ++++++++ docs/contributing/rst-development.rst | 588 +++++++++++++++++++++ docs/contributing/rst-index.rst | 14 + docs/contributing/rst-issues-features.rst | 130 +++++ docs/contributing/rst-testing.rst | 197 ++++++++ docs/contributing/rst-upstream-packages.rst | 159 ++++++ docs/contributing/testing.md | 207 ++++++++ docs/contributing/testing.rst | 197 -------- docs/contributing/upstream-packages.md | 147 ++++++ docs/contributing/upstream-packages.rst | 159 ------ 32 files changed, 4132 insertions(+), 4132 deletions(-) create mode 100644 docs/contributing/build-vyos.md delete mode 100644 docs/contributing/build-vyos.rst create mode 100644 docs/contributing/cla.md delete mode 100644 docs/contributing/cla.rst create mode 100644 docs/contributing/debugging.md delete mode 100644 docs/contributing/debugging.rst create mode 100644 docs/contributing/development.md delete mode 100644 docs/contributing/development.rst create mode 100644 docs/contributing/index.md delete mode 100644 docs/contributing/index.rst create mode 100644 docs/contributing/issues-features.md delete mode 100644 docs/contributing/issues-features.rst delete mode 100644 docs/contributing/md-build-vyos.md delete mode 100644 docs/contributing/md-cla.md delete mode 100644 docs/contributing/md-debugging.md delete mode 100644 docs/contributing/md-development.md delete mode 100644 docs/contributing/md-index.md delete mode 100644 docs/contributing/md-issues-features.md delete mode 100644 docs/contributing/md-testing.md delete mode 100644 docs/contributing/md-upstream-packages.md create mode 100644 docs/contributing/rst-build-vyos.rst create mode 100644 docs/contributing/rst-cla.rst create mode 100644 docs/contributing/rst-debugging.rst create mode 100644 docs/contributing/rst-development.rst create mode 100644 docs/contributing/rst-index.rst create mode 100644 docs/contributing/rst-issues-features.rst create mode 100644 docs/contributing/rst-testing.rst create mode 100644 docs/contributing/rst-upstream-packages.rst create mode 100644 docs/contributing/testing.md delete mode 100644 docs/contributing/testing.rst create mode 100644 docs/contributing/upstream-packages.md delete mode 100644 docs/contributing/upstream-packages.rst (limited to 'docs/contributing') diff --git a/docs/contributing/build-vyos.md b/docs/contributing/build-vyos.md new file mode 100644 index 00000000..bfda5298 --- /dev/null +++ b/docs/contributing/build-vyos.md @@ -0,0 +1,730 @@ +--- +lastproofread: '2025-12-05' +--- + +(build)= + +# Build VyOS + +## Prerequisites + +There are different ways you can build VyOS. Building using a +{ref}`build_docker` +container is the easiest way because all dependencies are managed for you. +Alternatively, you can set up your own build machine and run a +{ref}`build_native` build. + +:::{note} +Starting with VyOS 1.4, only source code and Debian package +repositories of the rolling release (the **current** branch) are publicly +available. + +The source code and pre-built Debian package repositories of LTS releases +are only available to subscription holders (customers and active community +members with contributors subscriptions). + +The following includes the build process for VyOS rolling release. +::: + +This will guide you through the process of building a VyOS ISO using [Docker]. +This process has been tested on clean installs of Debian Bookworm. + +(build_native)= + +### Native Build + +To build VyOS natively, you need a properly configured build host with +Debian Bookworm installed. + +To get started, clone the repository to your local machine: + +```none +$ sudo make clean +$ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic +``` + +For required packages, refer to the `docker/Dockerfile` file in the +[repository]. The `./build-vyos-image` script will also warn you if any +dependencies are missing. + +(build_docker)= + +### Docker + +Installing [Docker] and prerequisites: + +:::{hint} +Docker versions are updated frequently. The following examples may +become outdated. +::: + +```none +# Add Docker's official GPG key: +sudo apt-get update +sudo apt-get install ca-certificates curl gnupg +sudo install -m 0755 -d /etc/apt/keyrings +curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg +sudo chmod a+r /etc/apt/keyrings/docker.gpg + +# Add the repository to Apt sources: +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + +sudo apt-get update +sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +``` + +To use Docker without `sudo`, add your current non-root user to the `docker` +group: `sudo usermod -aG docker yourusername`. + +:::{hint} +Adding a user to the `docker` group grants privileges equivalent to +`root`. It is recommended to remove the non-root user from the `docker` +group after building the VyOS ISO. See also [Docker as non-root]. +::: + +:::{note} +The build process must run on a local file system. Building on SMB or +NFS shares will cause the container to fail. VirtualBox shared folders are +also not supported because block device operations are not implemented. +::: + +#### Build Container + +The container can be built by hand or by fetching the pre-built one from +DockerHub. It is recommended to use the pre-built containers from the +[VyOS DockerHub organisation](https://hub.docker.com/u/vyos). The container +is built from Docker packages automatically after every commit to the +``vyos-build`` repository (this process may take 2-3 hours). + +:::{note} +If you use the pre-built container, it will be automatically +downloaded from DockerHub if it is not found on your local machine when +you build the ISO. +::: + +##### Dockerhub + +To manually download the container from DockerHub, run: + +```none +$ docker pull vyos/vyos-build:current # For VyOS rolling release +``` + + +##### Build from source + +The container can also be built directly from source: + +```none +$ git clone -b current --single-branch https://github.com/vyos/vyos-build + +$ cd vyos-build +$ docker build -t vyos/vyos-build:current docker +``` + +:::{note} +VyOS switched to Debian Bookworm (12) in its `current` branch. +Due to software version updates, it is recommended to use the official +Docker Hub image to build VyOS ISO. +::: + +#### Tips and Tricks + +You can create Bash aliases to easily launch the latest container per release +train (`current`). Add the following to your `.bash_aliases` file: + +```none +alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "$HOME/.gitconfig":/etc/gitconfig \ + -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ + -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + vyos/vyos-build:current bash' +``` + +Now you have a new alias `vybld` that launches development containers in +your current working directory. + +:::{note} +Some VyOS packages (namely vyos-1x) come with build-time tests which +verify some of the internal library calls that they work as expected. Those +tests are carried out through the Python Unittest module. If you want to +build the `vyos-1x` package (which is our main development package) you +need to start your Docker container using the following argument: +`--sysctl net.ipv6.conf.lo.disable_ipv6=0`, otherwise those tests will +fail. +::: + +(build-iso)= + +## Build ISO + +Now that you understand the prerequisites, you can build a VyOS ISO from source. +First, fetch the latest source code from GitHub: + +```none +$ git clone -b current --single-branch https://github.com/vyos/vyos-build +``` + +Now you can begin a fresh VyOS ISO build. Change to the `vyos-build` +directory and run: + +```none +$ cd vyos-build +$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash +``` + +Start the build: + +```none +vyos_bld@8153428c7e1f:/vyos$ sudo make clean +vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic +``` + +When the build is successful, find the resulting ISO in the `build` directory +as `live-image-[architecture].hybrid.iso`. + +(build-source)= + +(customize)= + +### Customize + +You can customize the ISO with the following configure options. Generate the +full and current list with `./build-vyos-image --help`: + +```none +$ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help + I: Checking if packages required for VyOS image build are installed + usage: build-vyos-image [-h] [--architecture ARCHITECTURE] + [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR] + [--debian-security-mirror DEBIAN_SECURITY_MIRROR] + [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] + [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] + [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run] + [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY] + [--custom-package CUSTOM_PACKAGE] + [build_flavor] + + positional arguments: + build_flavor Build flavor + + optional arguments: + -h, --help show this help message and exit + --architecture ARCHITECTURE + Image target architecture (amd64 or arm64) + --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) + --debian-mirror DEBIAN_MIRROR + Debian repository mirror + --debian-security-mirror DEBIAN_SECURITY_MIRROR + Debian security updates mirror + --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR + Debian repository mirror for pbuilder env bootstrap + --vyos-mirror VYOS_MIRROR + VyOS package mirror + --build-type BUILD_TYPE + Build type, release or development + --version VERSION Version number (release builds only) + --build-comment BUILD_COMMENT + Optional build comment + --debug Enable debug output + --dry-run Check build configuration and exit + --custom-apt-entry CUSTOM_APT_ENTRY + Custom APT entry + --custom-apt-key CUSTOM_APT_KEY + Custom APT key file + --custom-package CUSTOM_PACKAGE + Custom package to install from repositories +``` + +(iso_build_issues)= + +#### ISO Build Issues + +There are (rare) situations where building an ISO image is not possible at all +due to a broken package feed in the background. APT is not very good at +reporting the root cause of the issue. Your ISO build will likely fail with a +more or less similar looking error message: + +```none +The following packages have unmet dependencies: + vyos-1x : Depends: accel-ppp but it is not installable +E: Unable to correct problems, you have held broken packages. +P: Begin unmounting filesystems... +P: Saving caches... +Reading package lists... +Building dependency tree... +Reading state information... +Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB] +Del accel-ppp 1.12.0-95-g59f8e1b [475 kB] +Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB] +Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB] +Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB] +make: *** [Makefile:30: iso] Error 1 +(10:13) vyos_bld ece068908a5b:/vyos [current] # +``` + +To debug the build process and gain additional information of what could be the +root cause, you need to use `chroot` to change into the build directory. This is +explained in the following step by step procedure: + +```none +vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash +``` + +We now need to mount some required, volatile filesystems + +```none +(live)root@ece068908a5b:/# mount -t proc none /proc +(live)root@ece068908a5b:/# mount -t sysfs none /sys +(live)root@ece068908a5b:/# mount -t devtmpfs none /dev +``` + +We now are free to run any command we would like to use for debugging, e.g. +re-installing the failed package after updating the repository. + +```none +(live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x +Get:1 file:/root/packages ./ InRelease +Ign:1 file:/root/packages ./ InRelease +Get:2 file:/root/packages ./ Release [1235 B] +Get:2 file:/root/packages ./ Release [1235 B] +Get:3 file:/root/packages ./ Release.gpg +Ign:3 file:/root/packages ./ Release.gpg +Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease +Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease +Hit:6 http://deb.debian.org/debian bullseye InRelease +Hit:7 http://deb.debian.org/debian buster InRelease +Hit:8 http://deb.debian.org/debian-security buster/updates InRelease +Hit:9 http://deb.debian.org/debian buster-updates InRelease +Hit:10 http://deb.debian.org/debian buster-backports InRelease +Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease +Reading package lists... Done +N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) +Reading package lists... Done +Building dependency tree +Reading state information... Done +Some packages could not be installed. This may mean that you have +requested an impossible situation or if you are using the unstable +distribution that some required packages have not yet been created +or been moved out of Incoming. +The following information may help to resolve the situation: + +The following packages have unmet dependencies: + vyos-1x : Depends: accel-ppp but it is not installable +E: Unable to correct problems, you have held broken packages. +``` + +Now it's time to fix the package mirror and rerun the last step until the +package installation succeeds again! + +(build_custom_packages)= + +### Linux Kernel + +The Linux kernel used by VyOS is heavily tied to the ISO build process. The +file `data/defaults.json` hosts a JSON definition of the kernel version used +`kernel_version` and the `kernel_flavor` of the kernel which represents the +kernel's LOCAL_VERSION. Both together form the kernel version variable in the +system: + +```none +vyos@vyos:~$ uname -r +6.1.52-amd64-vyos +``` + +- Accel-PPP +- Intel NIC drivers +- Intel QAT + +Each of those modules holds a dependency on the kernel version and if you are +lucky enough to receive an ISO build error which sounds like: + +```none +I: Create initramfs if it does not exist. +Extra argument '6.1.52-amd64-vyos' +Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] +Options: + -k version Specify kernel version or 'all' + -c Create a new initramfs + -u Update an existing initramfs + -d Remove an existing initramfs + -b directory Set alternate boot directory + -v Be verbose +See update-initramfs(8) for further details. +E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. +``` + +The most obvious reasons could be: + +- `vyos-build` repo is outdated, please `git pull` to update to the latest + release kernel version from us. +- You have your own custom kernel `*.deb` packages in the `packages` folder but + neglected to create all required out-of tree modules like Accel-PPP, Intel + QAT or Intel NIC drivers + +#### Building The Kernel + +The kernel build is quite easy, most of the required steps can be found in the +`vyos-build/packages/linux-kernel/Jenkinsfile` but we will walk you through +it. + +Clone the kernel source to `vyos-build/packages/linux-kernel/`: + +```none +$ cd vyos-build/packages/linux-kernel/ +$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git +``` + +Check out the required kernel version - see `vyos-build/data/defaults.json` +file (example uses kernel 4.19.146): + +```none +$ cd vyos-build/packages/linux-kernel/linux +$ git checkout v4.19.146 +Checking out files: 100% (61536/61536), done. +Note: checking out 'v4.19.146'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by performing another checkout. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -b with the checkout command again. Example: + git checkout -b +HEAD is now at 015e94d0e37b Linux 4.19.146 +``` + +Now you can use the helper script `build-kernel.sh`, which completes all +the necessary steps: applying required patches from the +`vyos-build/packages/linux-kernel/patches` folder, copying the kernel +configuration `x86_64_vyos_defconfig` to the correct location, and building +the Debian packages. + +:::{note} +Building the kernel will take some time depending on the speed and +quantity of your CPU/cores and disk speed. Expect 20 minutes +(or even longer) on lower end hardware. +::: + +```none +(18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh +I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source +I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch +patching file Documentation/networking/ip-sysctl.txt +patching file include/linux/inetdevice.h +patching file include/linux/ipv6.h +patching file include/uapi/linux/ip.h +patching file include/uapi/linux/ipv6.h +patching file net/ipv4/devinet.c +Hunk #1 succeeded at 2319 (offset 1 line). +patching file net/ipv6/addrconf.c +patching file net/ipv6/route.c +I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch +patching file fs/notify/inotify/Kconfig +patching file fs/notify/inotify/inotify_user.c +patching file fs/overlayfs/super.c +Hunk #2 succeeded at 1713 (offset 9 lines). +Hunk #3 succeeded at 1739 (offset 9 lines). +Hunk #4 succeeded at 1762 (offset 9 lines). +patching file include/linux/inotify.h +I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch +patching file scripts/package/builddeb +I: make x86_64_vyos_defconfig + HOSTCC scripts/basic/fixdep + HOSTCC scripts/kconfig/conf.o + YACC scripts/kconfig/zconf.tab.c + LEX scripts/kconfig/zconf.lex.c + HOSTCC scripts/kconfig/zconf.tab.o + HOSTLD scripts/kconfig/conf +# + +# configuration written to .config +# +I: Generate environment file containing Kernel variable +I: Build Debian Kernel package + UPD include/config/kernel.release +/bin/sh ./scripts/package/mkdebian +dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc +dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos +dpkg-buildpackage: info: source version 4.19.146-1 +dpkg-buildpackage: info: source distribution buster +dpkg-buildpackage: info: source changed by vyos_bld +dpkg-buildpackage: info: host architecture amd64 +dpkg-buildpackage: warning: debian/rules is not executable; fixing that + dpkg-source --before-build . + debian/rules build +make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= + SYSTBL arch/x86/include/generated/asm/syscalls_32.h +... +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory +dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) +dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) +dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. + dpkg-genbuildinfo --build=binary + dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes +dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list +dpkg-genchanges: info: binary-only upload (no source code included) + dpkg-source --after-build . +dpkg-buildpackage: info: binary-only upload (no source included) +``` + +When complete, you will have kernel binary packages to use in your custom ISO +build. Place all `*.deb` files in the `vyos-build/packages` folder, where +the build process will use them automatically. + +##### Firmware + +If you upgrade your kernel or include new drivers you may need new firmware. +This builds a new `vyos-linux-firmware` package using the included helper +scripts. + +```none +$ cd vyos-build/packages/linux-kernel +$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git +$ ./build-linux-firmware.sh +$ cp vyos-linux-firmware_*.deb ../ +``` + +The script automatically detects which firmware blobs are needed based on the +built drivers. If detection fails, you can manually add files to +`vyos-build/packages/linux-kernel/build-linux-firmware.sh`: + +```bash +ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin" +``` + + +#### Building Out-Of-Tree Modules + +Building the kernel is one step. You must also build required out-of-tree +modules so the ABIs match. +Refer to `vyos-build/packages/linux-kernel/Jenkinsfile` +for all required modules and their versions. We show you how to build the +currently required modules. + +##### Accel-PPP + +First, clone the source code and check out the appropriate version: + +```none +$ cd vyos-build/packages/linux-kernel +$ git clone https://github.com/accel-ppp/accel-ppp.git +``` + +Use the helper script and patches to build the package. Run the following +command: + +```none +$ ./build-accel-ppp.sh +I: Build Accel-PPP Debian package +CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): + The OLD behavior for policy CMP0003 will be removed from a future version + of CMake. + The cmake-policies(7) manual explains that the OLD behaviors of all + policies are deprecated and that a policy should be set to OLD only under + specific short-term circumstances. Projects should be ported to the NEW + behavior and not rely on setting a policy to OLD. +-- The C compiler identification is GNU 8.3.0 +... + +CPack: Create package using DEB +CPack: Install projects +CPack: - Run preinstall target for: accel-ppp +CPack: - Install project: accel-ppp +CPack: Create package +CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. +``` + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in `vyos-build/packages/linux-kernel` from which you can copy them +to the `vyos-build/packages` folder for inclusion during the ISO build. + +##### Intel NIC + +The Intel NIC drivers do not come from a Git repository. VyOS fetches the +tarballs from a mirror and compiles them. Use the following wrapper script +to build all driver modules: + +```none +./build-intel-drivers.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k +I: Compile Kernel module for Intel ixgbe driver +... + +I: Building Debian package vyos-intel-iavf +Doing `require 'backports'` is deprecated and will not load any backport in the next major release. +Require just the needed backports instead, or 'backports/latest'. +Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} +Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} +I: Cleanup iavf source +``` + +After compilation, find the generated `*.deb` binaries in +`vyos-build/packages/linux-kernel`. Copy them to the `vyos-build/packages` +folder for inclusion in the ISO build. + +##### Intel QAT + +The Intel QAT (Quick Assist Technology) drivers do not come from a Git +repository. VyOS fetches the tarballs from `01.org`, Intel's open-source +website. +Use the following wrapper script to build all driver modules: + +```none +$ ./build-intel-qat.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k +I: Compile Kernel module for Intel qat driver +checking for a BSD-compatible install... /usr/bin/install -c +checking whether build environment is sane... yes +checking for a thread-safe mkdir -p... /bin/mkdir -p +checking for gawk... gawk +checking whether make sets $(MAKE)... yes +... + +I: Building Debian package vyos-intel-qat +Doing `require 'backports'` is deprecated and will not load any backport in the next major release. +Require just the needed backports instead, or 'backports/latest'. +Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} +Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} +I: Cleanup qat source +``` + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in `vyos-build/packages/linux-kernel` from which you can copy them +to the `vyos-build/packages` folder for inclusion during the ISO build. + +### Packages + +If you are brave enough to build your own ISO image with any modified package +from VyOS's GitHub organisation, this is the place for you. + +Any modified package may be an altered version (e.g., `vyos-1x`) that you +want to test before filing a pull request on GitHub. + +Building an ISO with a customized package is the same as building a regular +ISO image. Place your modified `*.deb` package inside the `packages` folder +within `vyos-build`. The build process will automatically use your custom +package during the ISO build. + +### Troubleshooting + +Debian APT does not provide verbose error messages. If your ISO build fails and +you suspect an APT dependencies or installation issue, you can apply this patch +to increase APT verbosity during the ISO build. + +```diff +diff --git i/scripts/live-build-config w/scripts/live-build-config +index 1b3b454..3696e4e 100755 +--- i/scripts/live-build-config ++++ w/scripts/live-build-config +@@ -57,7 +57,8 @@ lb config noauto \ + --firmware-binary false \ + --updates true \ + --security true \ +- --apt-options "--yes -oAcquire::Check-Valid-Until=false" \ ++ --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \ ++ -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \ + --apt-indices false + "${@}" + """ +``` + +(build-packages)= + +## Packages + +VyOS comes with specific packages that cannot be found in any +Debian mirror. These packages are located in the [VyOS GitHub project] in +source format and can easily be compiled into custom +Debian (`*.deb`) packages. + +The easiest way to compile your package is with the {ref}`build_docker` +container mentioned earlier, as it includes all required dependencies for all +VyOS related packages. + +Assuming you want to build the `vyos-1x` package and modify it for your needs, +first clone the repository from GitHub: + +```none +$ git clone --recurse-submodules https://github.com/vyos/vyos-1x +``` + + +### Build + +Launch the Docker container and build the package: + +```none +# For VyOS 1.3 (equuleus, current) +$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash + +# Change to source directory +$ cd vyos-1x + +# Build DEB +$ dpkg-buildpackage -uc -us -tc -b +``` + +After a minute or two, the generated DEB packages are located next to the +`vyos-1x` source directory: + +```none +# ls -al ../vyos-1x*.deb +-rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb +-rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb +``` + + +### Install + +To test your newly created package, you can SCP it to a running VyOS instance +and install the new `*.deb` package to replace the current one. + +Install the package using the following commands: + +```none +vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb +(Reading database ... 58209 files and directories currently installed.) +Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... +Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... +Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... +Processing triggers for rsyslog (8.1901.0-1) ... +``` + +You can also place the generated `*.deb` in your ISO build environment to +include it in a custom ISO. See {ref}`build_custom_packages` for more +information. + +:::{warning} +Any packages in the `packages` directory will be added to the +ISO during the build, replacing upstream packages. Delete both the source +directories and built DEB packages if you want to build an ISO from purely +upstream packages. +::: + +[docker]: https://docs.docker.com/engine/install/debian/ +[docker as non-root]: https://docs.docker.com/engine/install/linux-postinstall +[repository]: https://github.com/vyos/vyos-build +[vyos dockerhub organisation]: https://hub.docker.com/u/vyos +[vyos github project]: https://github.com/vyos diff --git a/docs/contributing/build-vyos.rst b/docs/contributing/build-vyos.rst deleted file mode 100644 index 65298a7a..00000000 --- a/docs/contributing/build-vyos.rst +++ /dev/null @@ -1,759 +0,0 @@ -:lastproofread: 2025-12-05 - -.. _build: - -########## -Build VyOS -########## - -************* -Prerequisites -************* - -There are different ways you can build VyOS. Building using a -:ref:`build_docker` -container is the easiest way because all dependencies are managed for you. -Alternatively, you can set up your own build machine and run a -:ref:`build_native` build. - -.. note:: Starting with VyOS 1.4, only source code and Debian package - repositories of the rolling release (the **current** branch) are publicly - available. - - The source code and pre-built Debian package repositories of LTS releases - are only available to subscription holders (customers and active community - members with contributors subscriptions). - - The following includes the build process for VyOS rolling release. - -This will guide you through the process of building a VyOS ISO using Docker_. -This process has been tested on clean installs of Debian Bookworm. - -.. _build_native: - -Native Build -============ - -To build VyOS natively, you need a properly configured build host with -Debian Bookworm installed. - -To get started, clone the repository to your local machine: - -.. code-block:: none - - $ sudo make clean - $ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic - -For required packages, refer to the ``docker/Dockerfile`` file in the -repository_. The ``./build-vyos-image`` script will also warn you if any -dependencies are missing. - -.. _build_docker: - -Docker -====== - -Installing Docker_ and prerequisites: - -.. hint:: Docker versions are updated frequently. The following examples may - become outdated. - -.. code-block:: none - - # Add Docker's official GPG key: - sudo apt-get update - sudo apt-get install ca-certificates curl gnupg - sudo install -m 0755 -d /etc/apt/keyrings - curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg - sudo chmod a+r /etc/apt/keyrings/docker.gpg - - # Add the repository to Apt sources: - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ - $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null - - sudo apt-get update - sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin - -To use Docker without ``sudo``, add your current non-root user to the ``docker`` -group: ``sudo usermod -aG docker yourusername``. - -.. hint:: Adding a user to the ``docker`` group grants privileges equivalent to - ``root``. It is recommended to remove the non-root user from the ``docker`` - group after building the VyOS ISO. See also `Docker as non-root`_. - -.. note:: The build process must run on a local file system. Building on SMB or - NFS shares will cause the container to fail. VirtualBox shared folders are - also not supported because block device operations are not implemented. - -Build Container ---------------- - -The container can be built by hand or by fetching the pre-built one from -DockerHub. It is recommended to use the pre-built containers from the -`VyOS DockerHub`organization_ -The container is built from Docker packages automatically after every commit -to the ``vyos-build`` repository (this process may take 2-3 hours). - -.. note:: If you use the pre-built container, it will be automatically - downloaded from DockerHub if it is not found on your local machine when - you build the ISO. - -Dockerhub -^^^^^^^^^ - -To manually download the container from DockerHub, run: - -.. code-block:: none - - $ docker pull vyos/vyos-build:current # For VyOS rolling release - -Build from source -^^^^^^^^^^^^^^^^^ - -The container can also be built directly from source: - -.. code-block:: none - - $ git clone -b current --single-branch https://github.com/vyos/vyos-build - - $ cd vyos-build - $ docker build -t vyos/vyos-build:current docker - -.. note:: VyOS switched to Debian Bookworm (12) in its ``current`` branch. - Due to software version updates, it is recommended to use the official - Docker Hub image to build VyOS ISO. - -Tips and Tricks ---------------- - -You can create Bash aliases to easily launch the latest container per release -train (``current``). Add the following to your ``.bash_aliases`` file: - -.. code-block:: none - - alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ - -v "$(pwd)":/vyos \ - -v "$HOME/.gitconfig":/etc/gitconfig \ - -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ - -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ - -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ - vyos/vyos-build:current bash' - -Now you have a new alias ``vybld`` that launches development containers in -your current working directory. - -.. note:: Some VyOS packages (namely vyos-1x) come with build-time tests which - verify some of the internal library calls that they work as expected. Those - tests are carried out through the Python Unittest module. If you want to - build the ``vyos-1x`` package (which is our main development package) you - need to start your Docker container using the following argument: - ``--sysctl net.ipv6.conf.lo.disable_ipv6=0``, otherwise those tests will - fail. - - -.. _build_iso: - -********* -Build ISO -********* - -Now that you understand the prerequisites, you can build a VyOS ISO from source. -First, fetch the latest source code from GitHub: - -.. code-block:: none - - $ git clone -b current --single-branch https://github.com/vyos/vyos-build - - -Now you can begin a fresh VyOS ISO build. Change to the ``vyos-build`` -directory and run: - -.. code-block:: none - - $ cd vyos-build - $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash - -Start the build: - -.. code-block:: none - - vyos_bld@8153428c7e1f:/vyos$ sudo make clean - vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic - -When the build is successful, find the resulting ISO in the ``build`` directory -as ``live-image-[architecture].hybrid.iso``. - -.. _build source: - - -.. _customize: - -Customize -========= - -You can customize the ISO with the following configure options. Generate the -full and current list with ``./build-vyos-image --help``: - -.. code-block:: none - - $ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help - I: Checking if packages required for VyOS image build are installed - usage: build-vyos-image [-h] [--architecture ARCHITECTURE] - [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR] - [--debian-security-mirror DEBIAN_SECURITY_MIRROR] - [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] - [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] - [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run] - [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY] - [--custom-package CUSTOM_PACKAGE] - [build_flavor] - - positional arguments: - build_flavor Build flavor - - optional arguments: - -h, --help show this help message and exit - --architecture ARCHITECTURE - Image target architecture (amd64 or arm64) - --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) - --debian-mirror DEBIAN_MIRROR - Debian repository mirror - --debian-security-mirror DEBIAN_SECURITY_MIRROR - Debian security updates mirror - --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR - Debian repository mirror for pbuilder env bootstrap - --vyos-mirror VYOS_MIRROR - VyOS package mirror - --build-type BUILD_TYPE - Build type, release or development - --version VERSION Version number (release builds only) - --build-comment BUILD_COMMENT - Optional build comment - --debug Enable debug output - --dry-run Check build configuration and exit - --custom-apt-entry CUSTOM_APT_ENTRY - Custom APT entry - --custom-apt-key CUSTOM_APT_KEY - Custom APT key file - --custom-package CUSTOM_PACKAGE - Custom package to install from repositories - - -.. _iso_build_issues: - -ISO Build Issues ----------------- - -There are (rare) situations where building an ISO image is not possible at all -due to a broken package feed in the background. APT is not very good at -reporting the root cause of the issue. Your ISO build will likely fail with a -more or less similar looking error message: - -.. code-block:: none - - The following packages have unmet dependencies: - vyos-1x : Depends: accel-ppp but it is not installable - E: Unable to correct problems, you have held broken packages. - P: Begin unmounting filesystems... - P: Saving caches... - Reading package lists... - Building dependency tree... - Reading state information... - Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB] - Del accel-ppp 1.12.0-95-g59f8e1b [475 kB] - Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB] - Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB] - Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB] - make: *** [Makefile:30: iso] Error 1 - (10:13) vyos_bld ece068908a5b:/vyos [current] # - -To debug the build process and gain additional information of what could be the -root cause, you need to use `chroot` to change into the build directory. This is -explained in the following step by step procedure: - -.. code-block:: none - - vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash - -We now need to mount some required, volatile filesystems - -.. code-block:: none - - (live)root@ece068908a5b:/# mount -t proc none /proc - (live)root@ece068908a5b:/# mount -t sysfs none /sys - (live)root@ece068908a5b:/# mount -t devtmpfs none /dev - -We now are free to run any command we would like to use for debugging, e.g. -re-installing the failed package after updating the repository. - -.. code-block:: none - - (live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x - Get:1 file:/root/packages ./ InRelease - Ign:1 file:/root/packages ./ InRelease - Get:2 file:/root/packages ./ Release [1235 B] - Get:2 file:/root/packages ./ Release [1235 B] - Get:3 file:/root/packages ./ Release.gpg - Ign:3 file:/root/packages ./ Release.gpg - Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease - Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease - Hit:6 http://deb.debian.org/debian bullseye InRelease - Hit:7 http://deb.debian.org/debian buster InRelease - Hit:8 http://deb.debian.org/debian-security buster/updates InRelease - Hit:9 http://deb.debian.org/debian buster-updates InRelease - Hit:10 http://deb.debian.org/debian buster-backports InRelease - Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease - Reading package lists... Done - N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) - Reading package lists... Done - Building dependency tree - Reading state information... Done - Some packages could not be installed. This may mean that you have - requested an impossible situation or if you are using the unstable - distribution that some required packages have not yet been created - or been moved out of Incoming. - The following information may help to resolve the situation: - - The following packages have unmet dependencies: - vyos-1x : Depends: accel-ppp but it is not installable - E: Unable to correct problems, you have held broken packages. - -Now it's time to fix the package mirror and rerun the last step until the -package installation succeeds again! - -.. _build_custom_packages: - -Linux Kernel -============ - -The Linux kernel used by VyOS is heavily tied to the ISO build process. The -file ``data/defaults.json`` hosts a JSON definition of the kernel version used -``kernel_version`` and the ``kernel_flavor`` of the kernel which represents the -kernel's LOCAL_VERSION. Both together form the kernel version variable in the -system: - -.. code-block:: none - - vyos@vyos:~$ uname -r - 6.1.52-amd64-vyos - -* Accel-PPP -* Intel NIC drivers -* Intel QAT - -Each of those modules holds a dependency on the kernel version and if you are -lucky enough to receive an ISO build error which sounds like: - -.. code-block:: none - - I: Create initramfs if it does not exist. - Extra argument '6.1.52-amd64-vyos' - Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] - Options: - -k version Specify kernel version or 'all' - -c Create a new initramfs - -u Update an existing initramfs - -d Remove an existing initramfs - -b directory Set alternate boot directory - -v Be verbose - See update-initramfs(8) for further details. - E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. - -The most obvious reasons could be: - -* ``vyos-build`` repo is outdated, please ``git pull`` to update to the latest - release kernel version from us. - -* You have your own custom kernel `*.deb` packages in the `packages` folder but - neglected to create all required out-of tree modules like Accel-PPP, Intel - QAT or Intel NIC drivers - -Building The Kernel -------------------- - -The kernel build is quite easy, most of the required steps can be found in the -``vyos-build/packages/linux-kernel/Jenkinsfile`` but we will walk you through -it. - -Clone the kernel source to `vyos-build/packages/linux-kernel/`: - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel/ - $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git - -Check out the required kernel version - see ``vyos-build/data/defaults.json`` -file (example uses kernel 4.19.146): - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel/linux - $ git checkout v4.19.146 - Checking out files: 100% (61536/61536), done. - Note: checking out 'v4.19.146'. - - You are in 'detached HEAD' state. You can look around, make experimental - changes and commit them, and you can discard any commits you make in this - state without impacting any branches by performing another checkout. - - If you want to create a new branch to retain commits you create, you may - do so (now or later) by using -b with the checkout command again. Example: - - git checkout -b - - HEAD is now at 015e94d0e37b Linux 4.19.146 - -Now you can use the helper script ``build-kernel.sh``, which completes all -the necessary steps: applying required patches from the -``vyos-build/packages/linux-kernel/patches`` folder, copying the kernel -configuration ``x86_64_vyos_defconfig`` to the correct location, and building -the Debian packages. - -.. note:: Building the kernel will take some time depending on the speed and - quantity of your CPU/cores and disk speed. Expect 20 minutes - (or even longer) on lower end hardware. - -.. code-block:: none - - (18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh - I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source - I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch - patching file Documentation/networking/ip-sysctl.txt - patching file include/linux/inetdevice.h - patching file include/linux/ipv6.h - patching file include/uapi/linux/ip.h - patching file include/uapi/linux/ipv6.h - patching file net/ipv4/devinet.c - Hunk #1 succeeded at 2319 (offset 1 line). - patching file net/ipv6/addrconf.c - patching file net/ipv6/route.c - I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch - patching file fs/notify/inotify/Kconfig - patching file fs/notify/inotify/inotify_user.c - patching file fs/overlayfs/super.c - Hunk #2 succeeded at 1713 (offset 9 lines). - Hunk #3 succeeded at 1739 (offset 9 lines). - Hunk #4 succeeded at 1762 (offset 9 lines). - patching file include/linux/inotify.h - I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch - patching file scripts/package/builddeb - I: make x86_64_vyos_defconfig - HOSTCC scripts/basic/fixdep - HOSTCC scripts/kconfig/conf.o - YACC scripts/kconfig/zconf.tab.c - LEX scripts/kconfig/zconf.lex.c - HOSTCC scripts/kconfig/zconf.tab.o - HOSTLD scripts/kconfig/conf - # - # configuration written to .config - # - I: Generate environment file containing Kernel variable - I: Build Debian Kernel package - UPD include/config/kernel.release - /bin/sh ./scripts/package/mkdebian - dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc - dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos - dpkg-buildpackage: info: source version 4.19.146-1 - dpkg-buildpackage: info: source distribution buster - dpkg-buildpackage: info: source changed by vyos_bld - dpkg-buildpackage: info: host architecture amd64 - dpkg-buildpackage: warning: debian/rules is not executable; fixing that - dpkg-source --before-build . - debian/rules build - make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= - SYSTBL arch/x86/include/generated/asm/syscalls_32.h - - ... - - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory - dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) - dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) - dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. - dpkg-genbuildinfo --build=binary - dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes - dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list - dpkg-genchanges: info: binary-only upload (no source code included) - dpkg-source --after-build . - dpkg-buildpackage: info: binary-only upload (no source included) - - -When complete, you will have kernel binary packages to use in your custom ISO -build. Place all ``*.deb`` files in the ``vyos-build/packages`` folder, where -the build process will use them automatically. - -Firmware -^^^^^^^^ - -If you upgrade your kernel or include new drivers you may need new firmware. -This builds a new ``vyos-linux-firmware`` package using the included helper -scripts. - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel - $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git - $ ./build-linux-firmware.sh - $ cp vyos-linux-firmware_*.deb ../ - -The script automatically detects which firmware blobs are needed based on the -built drivers. If detection fails, you can manually add files to -``vyos-build/packages/linux-kernel/build-linux-firmware.sh``: - -.. code-block:: bash - - ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin" - - -Building Out-Of-Tree Modules ----------------------------- - -Building the kernel is one step. You must also build required out-of-tree -modules so the ABIs match. -Refer to ``vyos-build/packages/linux-kernel/Jenkinsfile`` -for all required modules and their versions. We show you how to build the -currently required modules. - -Accel-PPP -^^^^^^^^^ - -First, clone the source code and check out the appropriate version: - -.. code-block:: none - - $ cd vyos-build/packages/linux-kernel - $ git clone https://github.com/accel-ppp/accel-ppp.git - -Use the helper script and patches to build the package. Run the following -command: - -.. code-block:: none - - $ ./build-accel-ppp.sh - I: Build Accel-PPP Debian package - CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): - The OLD behavior for policy CMP0003 will be removed from a future version - of CMake. - - The cmake-policies(7) manual explains that the OLD behaviors of all - policies are deprecated and that a policy should be set to OLD only under - specific short-term circumstances. Projects should be ported to the NEW - behavior and not rely on setting a policy to OLD. - - -- The C compiler identification is GNU 8.3.0 - - ... - - CPack: Create package using DEB - CPack: Install projects - CPack: - Run preinstall target for: accel-ppp - CPack: - Install project: accel-ppp - CPack: Create package - CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them -to the ``vyos-build/packages`` folder for inclusion during the ISO build. - -Intel NIC -^^^^^^^^^ - -The Intel NIC drivers do not come from a Git repository. VyOS fetches the -tarballs from a mirror and compiles them. Use the following wrapper script -to build all driver modules: - -.. code-block:: none - - ./build-intel-drivers.sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k - I: Compile Kernel module for Intel ixgbe driver - - ... - - I: Building Debian package vyos-intel-iavf - Doing `require 'backports'` is deprecated and will not load any backport in the next major release. - Require just the needed backports instead, or 'backports/latest'. - Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} - Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} - I: Cleanup iavf source - -After compilation, find the generated ``*.deb`` binaries in -``vyos-build/packages/linux-kernel``. Copy them to the ``vyos-build/packages`` -folder for inclusion in the ISO build. - -Intel QAT -^^^^^^^^^ - -The Intel QAT (Quick Assist Technology) drivers do not come from a Git -repository. VyOS fetches the tarballs from ``01.org``, Intel's open-source -website. -Use the following wrapper script to build all driver modules: - -.. code-block:: none - - $ ./build-intel-qat.sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k - I: Compile Kernel module for Intel qat driver - checking for a BSD-compatible install... /usr/bin/install -c - checking whether build environment is sane... yes - checking for a thread-safe mkdir -p... /bin/mkdir -p - checking for gawk... gawk - checking whether make sets $(MAKE)... yes - - ... - - I: Building Debian package vyos-intel-qat - Doing `require 'backports'` is deprecated and will not load any backport in the next major release. - Require just the needed backports instead, or 'backports/latest'. - Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} - Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} - I: Cleanup qat source - - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them -to the ``vyos-build/packages`` folder for inclusion during the ISO build. - -Packages -======== - -If you are brave enough to build your own ISO image with any modified package -from VyOS's GitHub organisation, this is the place for you. - -Any modified package may be an altered version (e.g., ``vyos-1x``) that you -want to test before filing a pull request on GitHub. - -Building an ISO with a customized package is the same as building a regular -ISO image. Place your modified ``*.deb`` package inside the ``packages`` folder -within ``vyos-build``. The build process will automatically use your custom -package during the ISO build. - -Troubleshooting -=============== - -Debian APT does not provide verbose error messages. If your ISO build fails and -you suspect an APT dependencies or installation issue, you can apply this patch -to increase APT verbosity during the ISO build. - -.. stop_vyoslinter - -.. code-block:: diff - - diff --git i/scripts/live-build-config w/scripts/live-build-config - index 1b3b454..3696e4e 100755 - --- i/scripts/live-build-config - +++ w/scripts/live-build-config - @@ -57,7 +57,8 @@ lb config noauto \ - --firmware-binary false \ - --updates true \ - --security true \ - - --apt-options "--yes -oAcquire::Check-Valid-Until=false" \ - + --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \ - + -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \ - --apt-indices false - "${@}" - """ - -.. start_vyoslinter - -.. _build_packages: - -******** -Packages -******** - -VyOS comes with specific packages that cannot be found in any -Debian mirror. These packages are located in the `VyOS GitHub project`_ in -source format and can easily be compiled into custom -Debian (``*.deb``) packages. - -The easiest way to compile your package is with the :ref:`build_docker` -container mentioned earlier, as it includes all required dependencies for all -VyOS related packages. - -Assuming you want to build the ``vyos-1x`` package and modify it for your needs, -first clone the repository from GitHub: - -.. code-block:: none - - $ git clone --recurse-submodules https://github.com/vyos/vyos-1x - -Build -===== - -Launch the Docker container and build the package: - -.. code-block:: none - - # For VyOS 1.3 (equuleus, current) - $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash - - # Change to source directory - $ cd vyos-1x - - # Build DEB - $ dpkg-buildpackage -uc -us -tc -b - -After a minute or two, the generated DEB packages are located next to the -``vyos-1x`` source directory: - -.. code-block:: none - - # ls -al ../vyos-1x*.deb - -rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb - -rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb - -Install -======= - -To test your newly created package, you can SCP it to a running VyOS instance -and install the new ``*.deb`` package to replace the current one. - -Install the package using the following commands: - -.. code-block:: none - - vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb - (Reading database ... 58209 files and directories currently installed.) - Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... - Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... - Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... - Processing triggers for rsyslog (8.1901.0-1) ... - -You can also place the generated ``*.deb`` in your ISO build environment to -include it in a custom ISO. See :ref:`build_custom_packages` for more -information. - -.. warning:: Any packages in the ``packages`` directory will be added to the - ISO during the build, replacing upstream packages. Delete both the source - directories and built DEB packages if you want to build an ISO from purely - upstream packages. - - -.. stop_vyoslinter - -.. _Docker: https://docs.docker.com/engine/install/debian/ -.. _`Docker as non-root`: https://docs.docker.com/engine/install/linux-postinstall -.. _VyOS DockerHub organisation: https://hub.docker.com/u/vyos -.. _repository: https://github.com/vyos/vyos-build -.. _VyOS GitHub project: https://github.com/vyos - -.. start_vyoslinter - diff --git a/docs/contributing/cla.md b/docs/contributing/cla.md new file mode 100644 index 00000000..01323111 --- /dev/null +++ b/docs/contributing/cla.md @@ -0,0 +1,45 @@ +--- +lastproofread: '2025-12-05' +--- + +(cla)= + +# Contributor License Agreement + +Before we can accept your contributions to VyOS, you must sign a **Contributor +License Agreement (CLA)**. + +This is a standard open-source practice that protects both you and the project. + +The process is straightforward and fully automated: + +1. **Review the CLA document** + + Find the CLA text in our + [GitHub repository](https://github.com/vyos/vyos-cla-signatures/). + +2. **Submit a pull request** + + When you open a pull request, a CLA bot automatically checks whether all + commit authors have signed the CLA. + +3. **Follow the bot's instructions** + + If the CLA has not been signed, the bot leaves a comment with instructions. + Reply to that comment with the suggested text to sign the CLA. + +4. **Wait for confirmation** + + The CLA bot verifies your response and updates the pull request status. + Once all commit authors have signed, the bot confirms that the CLA + requirement is met and unlocks the pull request for merging. + +:::{note} +Each commit author must sign the CLA. + +If your pull request includes commits from multiple contributors, each one +must sign the CLA before the pull request can be accepted. +::: + +Once you sign the CLA, it remains valid for all your past and future +contributions to VyOS under the same GitHub identity. diff --git a/docs/contributing/cla.rst b/docs/contributing/cla.rst deleted file mode 100644 index c19e266a..00000000 --- a/docs/contributing/cla.rst +++ /dev/null @@ -1,45 +0,0 @@ -:lastproofread: 2025-12-05 - -.. _cla: - -############################# -Contributor License Agreement -############################# - -Before we can accept your contributions to VyOS, you must sign a **Contributor -License Agreement (CLA)**. - -This is a standard open-source practice that protects both you and the project. - -The process is straightforward and fully automated: - -1. **Review the CLA document** - - Find the CLA text in our - `GitHub repository `__. - -2. **Submit a pull request** - - When you open a pull request, a CLA bot automatically checks whether all - commit authors have signed the CLA. - -3. **Follow the bot's instructions** - - If the CLA has not been signed, the bot leaves a comment with instructions. - Reply to that comment with the suggested text to sign the CLA. - -4. **Wait for confirmation** - - The CLA bot verifies your response and updates the pull request status. - Once all commit authors have signed, the bot confirms that the CLA - requirement is met and unlocks the pull request for merging. - -.. note:: - - Each commit author must sign the CLA. - - If your pull request includes commits from multiple contributors, each one - must sign the CLA before the pull request can be accepted. - -Once you sign the CLA, it remains valid for all your past and future -contributions to VyOS under the same GitHub identity. diff --git a/docs/contributing/debugging.md b/docs/contributing/debugging.md new file mode 100644 index 00000000..d3b4b513 --- /dev/null +++ b/docs/contributing/debugging.md @@ -0,0 +1,204 @@ +--- +lastproofread: '2025-12-05' +--- + +(debugging)= + +# Debugging + +Two flags are available to help debug configuration scripts. Configuration +loading issues manifest during boot, so these flags are passed as kernel boot +parameters. + +## ISO image build + +If you have trouble compiling your own ISO image or debugging Jenkins issues, +follow the steps at {ref}`iso_build_issues`. + +## System Startup + +Debug system startup by examining the configuration file loading from +`/config/config.boot`. Extend the kernel command-line in the bootloader to +enable this. + +### Kernel + +- `vyos-debug` - Add this parameter to the Linux boot line to produce + timing results for script execution during commit. If you see an unexpected + delay during manual or boot commit, this parameter helps identify bottlenecks. + The internal flag is `VYOS_DEBUG`, found in [vyatta-cfg]. Output is directed + to `/var/log/vyatta/cfg-stdout.log`. +- `vyos-config-debug` - During development, coding errors can cause commit + failures on boot, potentially preventing CLI initialization. This kernel boot + parameter ensures access to the system as user `vyos` and logs a Python + stack trace to `/tmp/boot-config-trace`. The file is created only if the + configuration load fails. + +## Live System + +Several flags can be set to change VyOS behavior at runtime. Toggle these flags +using environment variables or by creating files. + +For each feature, create a file called `vyos.feature.debug` to enable it. +If a parameter is required, place it as the first line inside the file. + +Place the file in `/tmp` for one-time debugging (the file is removed on +reboot) or in `/config` to persist permanently. + +For example, `/tmp/vyos.ifconfig.debug` can be created to enable +interface debugging. + +You can also enable debugging using environment variables. +The environment variable name follows the convention `VYOS_FEATURE_DEBUG`. + +For example, `export VYOS_IFCONFIG_DEBUG=""` in your vbash has the same effect +as `touch /tmp/vyos.ifconfig.debug`. + +- `ifconfig` - Display all commands and their responses from the OS on + screen for inspection. +- `command` - Display all commands and their responses from the OS on screen + for inspection. +- `developer` - When a command fails, start a PDB post-mortem session instead + of showing a standard error message. This allows developers to debug issues + interactively. Because the debugger waits for input, it can prevent the router + from booting, so only enable this permanently on production systems if you are + ready for potential boot failures. +- `log` - Send all commands used by VyOS to a log file for inspection. This + is useful in rare cases when you need to see what the OS is doing, including + during boot. The default file is `/tmp/full-log`, but you can change it. + +:::{note} +To retrieve debug output on the command line, disable `vyos-configd` +in addition. You can do this one-time with +`sudo systemctl stop vyos-configd` +or permanently with `sudo systemctl disable vyos-configd`. +::: + +### FRR + +Recent versions use the `vyos.frr` framework. The Python class is located in +`vyos-1x:python/vyos/frr.py`. It includes an embedded debugger similar to the +one in `vyos.ifconfig`. + +Enable debugging by running: `touch /tmp/vyos.frr.debug` + +### Debug Python code with PDB + +Sometimes it is useful to debug Python code interactively on the live system +rather than in an IDE. You can do this using pdb. + +Assuming you want to debug a Python script called by an op-mode command, find +the script by looking up the op-mode definitions, then edit it on the live +system using vi: +`vi /usr/libexec/vyos/op_mode/show_xyz.py` + +Insert the following statement right before the section where you want to +investigate a problem (for example, a statement you see in a backtrace): +`import pdb; pdb.set_trace()` + +Optionally, surround this statement with an `if` condition that triggers only +for the conditions you are interested in. + +When you run `show xyz` and your condition triggers, you enter the Python +debugger: + +```none +> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() +-> rule_type = rule.get('type', '') +(Pdb) +``` + +You can type `help` to get an overview of the available commands, and +`help command` to get more information on each command. + +Common useful commands include: + +- examine variables using `pp(var)` +- continue execution using `cont` +- get a backtrace using `bt` + +### Config Migration Scripts + +Starting with VyOS 1.5, a new mechanism is used for config migration that +improves migration performance. New migrators use only the new format with a +`migration()` function. + +```python +from vyos.configtree import ConfigTree +base = ['vpn', 'ipsec'] +def migrate(config: ConfigTree) -> None: + if not config.exists(base): + # Nothing to do + return + # do your stuff here +``` + +New-style migration scripts can no longer run on their own. However, the new +migration subsystem handler includes a test kit: + +```none +vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --help +usage: run-config-migration.py [-h] [--test-script TEST_SCRIPT] [--output-file OUTPUT_FILE] [--force] config_file + +positional arguments: + config_file configuration file to migrate + +options: + -h, --help show this help message and exit + --test-script TEST_SCRIPT + test named script + --output-file OUTPUT_FILE + write to named output file instead of config file + --force force run of all migration scripts +``` + +To test your migration, run: + +```none +vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --test-script /opt/vyatta/etc/config-migrate/migrate/quagga/11-to-12 --output-file /tmp/foo /tmp/static-route-basic +vyos@vyos:~$ cat /tmp/foo +``` + +The file `/tmp/foo` contains the migrated configuration. + +### Configuration Error on System Boot + +Running the latest rolling releases sometimes exposes bugs due to edge cases +missed in design. File these bugs via [Phabricator](https://vyos.dev/), but you can help narrow +down the issue by following these steps: + +1. Log in to your VyOS system. +2. Enter configuration mode: `configure` +3. Reload your boot configuration: `load` + +You should see a Python backtrace that helps identify the issue. Attach it to +the [Phabricator](https://vyos.dev/) task. + +### Boot Timing + +During the migration and rewrite of functionality from Perl to Python, system +boot time increased significantly. You can analyze and graph boot time to see +detailed call sequences during startup. + +This uses the `systemd-bootchart` package, which is installed by default on +VyOS 1.3 (equuleus) and later. Configuration is versioned for comparable +results. Refer to [bootchart.conf] for the configuration file. + +To enable boot time graphing, add the following to the kernel command line: +`init=/usr/lib/systemd/systemd-bootchart` + +You can also make this permanent by editing `/boot/grub/grub.cfg`. + +## Priorities + +VyOS CLI depends heavily on priorities. Every CLI node has a corresponding +`node.def` file and possibly an attached script. Nodes can have priorities, +and on system bootup or any `commit` to the configuration, scripts execute +from lowest to highest priority. This provides deterministic behavior. + +To debug priority issues or see script execution order, use the +`/opt/vyatta/sbin/priority.pl` script, which lists the execution order of +scripts. + +[bootchart.conf]: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf +[vyatta-cfg]: https://github.com/vyos/vyatta-cfg diff --git a/docs/contributing/debugging.rst b/docs/contributing/debugging.rst deleted file mode 100644 index da8186ae..00000000 --- a/docs/contributing/debugging.rst +++ /dev/null @@ -1,223 +0,0 @@ -:lastproofread: 2025-12-05 - -.. _debugging: - -######### -Debugging -######### - -Two flags are available to help debug configuration scripts. Configuration -loading issues manifest during boot, so these flags are passed as kernel boot -parameters. - -ISO image build -=============== - -If you have trouble compiling your own ISO image or debugging Jenkins issues, -follow the steps at :ref:`iso_build_issues`. - -System Startup -============== - -Debug system startup by examining the configuration file loading from -``/config/config.boot``. Extend the kernel command-line in the bootloader to -enable this. - -Kernel ------- - -* ``vyos-debug`` - Add this parameter to the Linux boot line to produce - timing results for script execution during commit. If you see an unexpected - delay during manual or boot commit, this parameter helps identify bottlenecks. - The internal flag is ``VYOS_DEBUG``, found in vyatta-cfg_. Output is directed - to ``/var/log/vyatta/cfg-stdout.log``. - -* ``vyos-config-debug`` - During development, coding errors can cause commit - failures on boot, potentially preventing CLI initialization. This kernel boot - parameter ensures access to the system as user ``vyos`` and logs a Python - stack trace to ``/tmp/boot-config-trace``. The file is created only if the - configuration load fails. - -Live System -=========== - -Several flags can be set to change VyOS behavior at runtime. Toggle these flags -using environment variables or by creating files. - -For each feature, create a file called ``vyos.feature.debug`` to enable it. -If a parameter is required, place it as the first line inside the file. - -Place the file in ``/tmp`` for one-time debugging (the file is removed on -reboot) or in ``/config`` to persist permanently. - -For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable -interface debugging. - -You can also enable debugging using environment variables. -The environment variable name follows the convention ``VYOS_FEATURE_DEBUG``. - -For example, ``export VYOS_IFCONFIG_DEBUG=""`` in your vbash has the same effect -as ``touch /tmp/vyos.ifconfig.debug``. - -* ``ifconfig`` - Display all commands and their responses from the OS on - screen for inspection. - -* ``command`` - Display all commands and their responses from the OS on screen - for inspection. - -* ``developer`` - When a command fails, start a PDB post-mortem session instead - of showing a standard error message. This allows developers to debug issues - interactively. Because the debugger waits for input, it can prevent the router - from booting, so only enable this permanently on production systems if you are - ready for potential boot failures. - -* ``log`` - Send all commands used by VyOS to a log file for inspection. This - is useful in rare cases when you need to see what the OS is doing, including - during boot. The default file is ``/tmp/full-log``, but you can change it. - -.. note:: To retrieve debug output on the command line, disable ``vyos-configd`` - in addition. You can do this one-time with - ``sudo systemctl stop vyos-configd`` - or permanently with ``sudo systemctl disable vyos-configd``. - -FRR ---- - -Recent versions use the ``vyos.frr`` framework. The Python class is located in -``vyos-1x:python/vyos/frr.py``. It includes an embedded debugger similar to the -one in ``vyos.ifconfig``. - -Enable debugging by running: ``touch /tmp/vyos.frr.debug`` - -Debug Python code with PDB ------------------------------- - -Sometimes it is useful to debug Python code interactively on the live system -rather than in an IDE. You can do this using pdb. - -Assuming you want to debug a Python script called by an op-mode command, find -the script by looking up the op-mode definitions, then edit it on the live -system using vi: -``vi /usr/libexec/vyos/op_mode/show_xyz.py`` - -Insert the following statement right before the section where you want to -investigate a problem (for example, a statement you see in a backtrace): -``import pdb; pdb.set_trace()`` - -Optionally, surround this statement with an ``if`` condition that triggers only -for the conditions you are interested in. - -When you run ``show xyz`` and your condition triggers, you enter the Python -debugger: - - -.. code-block:: none - - > /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() - -> rule_type = rule.get('type', '') - (Pdb) - -You can type ``help`` to get an overview of the available commands, and -``help command`` to get more information on each command. - -Common useful commands include: - -* examine variables using ``pp(var)`` -* continue execution using ``cont`` -* get a backtrace using ``bt`` - -Config Migration Scripts ------------------------- - -Starting with VyOS 1.5, a new mechanism is used for config migration that -improves migration performance. New migrators use only the new format with a -``migration()`` function. - -.. code-block:: python - - from vyos.configtree import ConfigTree - base = ['vpn', 'ipsec'] - def migrate(config: ConfigTree) -> None: - if not config.exists(base): - # Nothing to do - return - # do your stuff here - -New-style migration scripts can no longer run on their own. However, the new -migration subsystem handler includes a test kit: - -.. code-block:: none - - vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --help - usage: run-config-migration.py [-h] [--test-script TEST_SCRIPT] [--output-file OUTPUT_FILE] [--force] config_file - - positional arguments: - config_file configuration file to migrate - - options: - -h, --help show this help message and exit - --test-script TEST_SCRIPT - test named script - --output-file OUTPUT_FILE - write to named output file instead of config file - --force force run of all migration scripts - - -To test your migration, run: - -.. code-block:: none - - vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --test-script /opt/vyatta/etc/config-migrate/migrate/quagga/11-to-12 --output-file /tmp/foo /tmp/static-route-basic - vyos@vyos:~$ cat /tmp/foo - -The file ``/tmp/foo`` contains the migrated configuration. - -Configuration Error on System Boot ----------------------------------- - -Running the latest rolling releases sometimes exposes bugs due to edge cases -missed in design. File these bugs via Phabricator_, but you can help narrow -down the issue by following these steps: - -1. Log in to your VyOS system. -2. Enter configuration mode: ``configure`` -3. Reload your boot configuration: ``load`` - -You should see a Python backtrace that helps identify the issue. Attach it to -the Phabricator_ task. - -Boot Timing ------------ - -During the migration and rewrite of functionality from Perl to Python, system -boot time increased significantly. You can analyze and graph boot time to see -detailed call sequences during startup. - -This uses the ``systemd-bootchart`` package, which is installed by default on -VyOS 1.3 (equuleus) and later. Configuration is versioned for comparable -results. Refer to bootchart.conf_ for the configuration file. - -To enable boot time graphing, add the following to the kernel command line: -``init=/usr/lib/systemd/systemd-bootchart`` - -You can also make this permanent by editing ``/boot/grub/grub.cfg``. - -Priorities -========== - -VyOS CLI depends heavily on priorities. Every CLI node has a corresponding -``node.def`` file and possibly an attached script. Nodes can have priorities, -and on system bootup or any ``commit`` to the configuration, scripts execute -from lowest to highest priority. This provides deterministic behavior. - -To debug priority issues or see script execution order, use the -``/opt/vyatta/sbin/priority.pl`` script, which lists the execution order of -scripts. - -.. stop_vyoslinter - -.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg -.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf -.. include:: /_include/common-references.txt - -.. start_vyoslinter diff --git a/docs/contributing/development.md b/docs/contributing/development.md new file mode 100644 index 00000000..8d2cec40 --- /dev/null +++ b/docs/contributing/development.md @@ -0,0 +1,549 @@ +--- +lastproofread: '2025-12-12' +--- + +(development)= + +# Development + +Learn how to contribute to VyOS. + +(architecture-overview)= + +## Architecture overview + +VyOS source code is hosted on GitHub in the VyOS organization: + + +VyOS is composed of multiple modules spread across different +repositories. Some modules contain forks of upstream +packages and are periodically synced. +VyOS consolidates most packages into the +[vyos-1x](https://github.com/vyos/vyos-1x) +repository while maintaining a consistent structure. +The base code is being rewritten +from Perl and Bash to Python using an XML-based CLI interface definition. + +VyOS ISO build scripts are hosted in the +[vyos-build](https://github.com/vyos/vyos-build) repository. See the +`vyos-build` repository +[README.md file](https://github.com/vyos/vyos-build/blob/current/README.md) +for more information on building VyOS ISO images. + +## Contributing code + +:::{warning} +You must sign the {doc}`Contributor License Agreement` +for your contributions to be accepted. +::: + +VyOS is open-source and welcomes patches. +All submissions must adhere to these guidelines: + +- Each commit addresses a single issue or feature. +- Each commit message references a [Phabricator](https://vyos.dev/) task ID + (for example, `T1234`). +- Each commit is associated with a username and email address + to identify the author (see [Configure your Git identity](configure-your-git-identity)). +- Only submit bugfixes in packages other than . +- Commits follow the [coding guidelines](coding-guidelines) outlined below. + +### Determining package ownership + +To determine which VyOS package contains a file you want to modify, use Debian's +`dpkg -S` command on your running VyOS installation. + +### Submitting your code + +Fork the repository and submit a GitHub pull request. This is the preferred way +to contribute changes to VyOS. + +To fork a VyOS repository: + +1. Append `/fork` to the repository URL on GitHub. For example, to fork + `vyos-1x`, use: + +2. Clone your fork or add it as a remote to your local repository: + + - Clone: `git clone https://github.com//vyos-1x.git` + - Add remote: `git remote add myfork https://github.com//vyos-1x.git` + +(configure-your-git-identity)= + +3. Configure your Git identity: + + ```none + git config --global user.name "J. Random Hacker" + git config --global user.email "jrhacker@example.net" + ``` + +4. Make your changes and add files to the Git index: + + - Single file: `git add myfile` + - Directory: `git add somedir/*` + +5. Commit your changes with a meaningful headline and [Phabricator](https://vyos.dev/) reference: + + `git commit` + +6. Push to your fork and create a GitHub pull request: + + `git push` + +Alternatively, you can export commits as patches and send them to +[maintainers@vyos.net](mailto:maintainers@vyos.net) or attach them directly to the [Phabricator](https://vyos.dev/) task: + +- Export last commit: `git format-patch` +- Export last two commits: `git format-patch -2` + +## Commit messages + +For guidance on writing commit messages, review the file history +with `git log path/to/file.txt`. + +Every change must be associated with a task number (prefixed with **T**) and +a component. If no bug report or feature request exists for your changes, +create a [Phabricator](https://vyos.dev/) task first. Reference the task ID in your commit message: + +- `ddclient: T1030: auto create runtime directories` +- `Jenkins: add current Git commit ID to build description` + +If your pull request lacks a [Phabricator](https://vyos.dev/) reference, maintainers will request +that you amend the commit message. + +### Writing good commit messages + +Follow the format described in +the [Git documentation](https://git-scm.com/book/ch5-2.html) +and [Chris Beams' guide](https://chris.beams.io/posts/git-commit/). + +Commit message format: + +1. **Summary line** (50 characters recommended, 80 maximum): Include the + component + prefix and [Phabricator](https://vyos.dev/) reference (for example, `snmp: T1111:` or + `ethernet: T2222:`). Concatenate multiple components with colons + (for example, `snmp: ethernet: T3333`). +2. **Blank line**: Separate the summary from the body. + This blank line is critical. + +4) **Message body** with details: + + - Describe what changed, why, and how. This helps with `git bisect`. + - Wrap text at 72 characters for readability with `git log` on an 80x25 + terminal. + - Reference previous commits when applicable: + `After commit abcd12ef ("snmp: this is a headline") + a Python import statement is missing, throwing the following exception: + ABCDEF` + +5) **Cherry-pick option**: Always use the `-x` option when back-porting or + forward-porting commits: + + `git cherry-pick -x ` + + This appends `(cherry picked from commit )` to the commit message, + making bisecting easier. + +6) **Single responsibility**: Each commit must be self-contained. Do not fix + multiple bugs in a single commit. Use `git add --patch` to stage only + the parts related to one issue. + +Constraints: + +- Bugfixes are only accepted for packages other than + . + New functionality must use the new XML/Python interface, not old-style + templates (`node.def` files and Perl/Bash code). + +## Coding guidelines + +VyOS maintains consistent coding standards to help contributors navigate the +codebase and understand its logic. + +### Formatting + +- **Python**: Use 4 spaces per indentation level. Tabs **must not** be used. +- **XML**: Use 2 spaces per indentation level. Tabs **must not** be used. + +Use tools like VIM extensions (xmllint) to enforce correct indentation. Add this +to your `.vimrc` file: + +```none +au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null +``` + +Then use `gg=G` in command mode to run the linter. + +### Text generation + +Use a template processor for generating config files: + +- **Jinja2** is the default template processor for VyOS code. +- Built-in string formatting **may** be used for simple line-oriented formats + (for example, iptables rules) where every line is self-contained. +- Template processors **must** be used for structured, multi-line formats + (for example, ISC DHCPd configuration). + +### Python code + +Configuration scripts and operation mode scripts written in Python3 should +follow these guidelines: + +- Wrap lines at 80 characters. This improves readability when browsing + GitHub on mobile devices and reads well in side-by-side diffs. + +Structure your scripts with these functions: + +```python +#!/usr/bin/env python3 +# +# Copyright (C) 2020 VyOS maintainers and contributors +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 or later as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +import sys + +from vyos.config import Config +from vyos import ConfigError + +def get_config(config=None): + if config: + conf = config + else: + conf = Config() + + # Base path to CLI nodes + base = ['...', '...'] + # Convert the VyOS config to an abstract internal representation + config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) + return config_data + +def verify(config): + # Verify that configuration is valid + if invalid: + raise ConfigError("Descriptive message") + +def generate(config): + # Generate daemon configs + pass + +def apply(config): + # Apply the generated configs to the live system + pass + +try: + c = get_config() + verify(c) + generate(c) + apply(c) +except ConfigError as e: + print(e) + sys.exit(1) +``` + +`get_config()`: This function converts a VyOS config object to an abstract +internal representation. No other function may call the `vyos.config.Config` +object directly. Limiting config reads to one function makes it easier to +modify the config syntax in the future. Additionally, this design improves +testability since you can construct an internal representation by hand rather +than mocking the entire config subsystem. + +`verify()`: This function validates the internal representation. It must +raise `ConfigError` with a descriptive message if the config is invalid. It +**must not** make any changes to the system. This design enables future features +like commit dry-run ("commit test" as in JunOS) where the system can abort a +commit before making changes. + +`generate()`: This function generates config files for system components. + +`apply()`: This function applies the generated configuration to the live +system. Prefer non-disruptive reload when possible. Disruptive operations like +daemon restarts are acceptable only when: + +- The component does not support non-disruptive reload, or +- The expected service degradation is minimal (for example, auxiliary services + like LLDPd) + +For high-impact services (VPN daemons, routing protocols), make effort to +determine if changes can be applied non-disruptively before resorting to +restarts. + +Never modify active configuration directly unless absolutely necessary. Instead, +generate configuration files and apply them with a single command like service +reload through systemd. For example, save iptables rules to a file and load them +with `iptables-restore` rather than executing iptables commands one by one. + +The `apply()` and `generate()` functions may raise `ConfigError` if the +daemon fails to start with the updated config. However, this is not a substitute +for proper config validation in the `verify()` function. Make reasonable +effort to verify that generated configuration is valid and will be accepted by +the daemon, including cross-checks with other VyOS configuration subtrees when +necessary. + +Exceptions like `VyOSError` (raised by `vyos.config.Config` on improper +operations) should not be silenced or caught. While this may produce less +polished error output for users, it generates better bug reports and helps +maintainers debug issues. + +For reference implementations, see `ntp.py` or `interfaces-bonding.py` (for +tag nodes) in the [vyos-1x](https://github.com/vyos/vyos-1x) repository. + +### Other considerations: `vyos-configd` + +All scripts now run under the config daemon and must conform to these +requirements: + +1. The signature and first four lines of `get_config(...)` **must** be as + specified above. +2. Each of `get_config`, `verify`, `apply`, and `generate` **must** + appear + with the correct signatures, even if they are a no-op. +3. `Config` objects other than those in `get_config` **must not** appear. +4. The legacy function `my_set` **must not** appear. Modifications to active + config **should not** appear in new code (alternative mechanisms may be used + if absolutely necessary). + +## XML for CLI definitions + +XML interface definitions define the VyOS CLI structure. +Before VyOS `1.2` (crux), these +files were created manually. After a redesign, new-style templates are +automatically generated from XML input files. + +VyOS interface definitions come with a RelaxNG schema located in the +[vyos-1x](https://github.com/vyos/vyos-1x/tree/current/schema) +repository. This schema is a modified version from `VyConf` (VyOS `2.0`). +VyOS `1.2.x` +interface definitions are reusable in future VyOS versions with minimal changes. + +Schemas provide two benefits: + +- Complete grammar verification +- Automatic validation against the schema + +The [build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) +script converts XML definitions to +old-style templates and verifies them against the schema. A bad definition +causes the package build to fail. While the XML format is verbose, no other +format provides this level of verification. Specialized XML editors can help +manage verbosity. + +Example XML interface definition: + +```xml + + + + + + + + Task scheduler settings + + + + + Scheduled task + + <string> + Task name + + 999 + + + + + UNIX crontab time specification string + + + + + Execution interval + + <minutes> + Execution interval in minutes + + + <minutes>m + Execution interval in minutes + + + <hours>h + Execution interval in hours + + + <days>d + Execution interval in days + + + [1-9]([0-9]*)([mhd]{0,1}) + + + + + + Executable path and arguments + + + + + Path to executable + + + + + Arguments passed to the executable + + + + + + + + + + + +``` + +XML definitions are purely declarative and contain no logic. All logic for +generating config files, restarting services, and related tasks is implemented +in configuration scripts. + +### Template Processors + +XML interface definition files use the `.xml.in` file extension (implemented +in {vytask}`T1843`). These files use the GCC preprocessor to reduce code +duplication in common areas: + +- VIF (including VIF-S and VIF-C) +- Address configuration +- Description +- Enabled/Disabled state + +Instead of repeating XML nodes, use include files with predefined features: + +- [IPv4, IPv6, and DHCP(v6)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i) + address assignment. +- [IPv4 and IPv6](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i) + address assignment. +- [VLAN (VIF)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/accel-ppp/vlan.xml.i) + definition. +- [MAC address](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/firewall/mac-address.xml.i) + assignment. + +The `.in` files are preprocessed and stored in the [interface-definitions](https://github.com/vyos/vyos-1x/tree/current/interface-definitions) +folder. The [scripts/build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) +script then operates on this folder to generate all required CLI nodes. + +Example preprocessor output: + +```none +$ make interface_definitions +install -d -m 0755 build/interface-definitions +install -d -m 0755 build/op-mode-definitions +Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in +Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in +Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in +Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in +Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in +Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in +[...] +``` + + +### Command Definition Guidelines + +#### Use of Numbers + +Avoid using numbers in command names unless the number is part of a protocol +name or similar. For example, `protocols ospfv3` is appropriate, +but `server-1` is questionable. + +#### Help Strings + +Follow these guidelines for consistent, readable help strings: + +##### Capitalization and Punctuation + +- Capitalize the first word of every help string. +- Do not use a period at the end of help strings. + +This standard mirrors network device CLIs and improves aesthetics. + +Examples: + +- Good: "Frobnication algorithm" +- Bad: "frobnication algorithm" +- Bad: "Frobnication algorithm." +- Incorrect: "frobnication algorithm." + +##### Abbreviations and Acronyms + +- Capitalize all abbreviations and acronyms. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "tcp connection timeout" +- Bad: "Tcp connection timeout" +- Capitalize acronyms to distinguish them from normal words. + +Examples: + +- Good: RADIUS (remote authentication for dial-in user services) +- Bad: radius (unless referring to circular distance) +- Follow accepted spelling conventions for mixed-case abbreviations. If it + contains "over" or "version", use lowercase. Follow RFC or standard spellings + when they exist. + +Examples: + +- Good: PPPoE, IPsec +- Bad: PPPOE, IPSEC +- Bad: pppoe, ipsec + +##### Verbs + +- Avoid verbs. If a verb can be omitted, omit it. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "Set TCP connection timeout" +- When a verb is essential, use it. For example: "Disable IPv6 forwarding on + all interfaces" for `set system ipv6 disable-forwarding`. +- Use infinitive form for necessary verbs. + +Examples: + +- Good: "Disable IPv6 forwarding" +- Bad: "Disables IPv6 forwarding" + +## C++ Backend Code + +The VyOS CLI parser combines bash, bash-completion helpers, and the C++ backend +library [vyatta-cfg](https://github.com/vyos/vyatta-cfg). This section +references common CLI commands and their C/C++ entry points: + +`set`: + +- +- + +`commit`: + +- + + diff --git a/docs/contributing/development.rst b/docs/contributing/development.rst deleted file mode 100644 index 5de950df..00000000 --- a/docs/contributing/development.rst +++ /dev/null @@ -1,588 +0,0 @@ -:lastproofread: 2025-12-12 - -.. _development: - -########### -Development -########### - -Learn how to contribute to VyOS. - -.. _architecture_overview: - -Architecture overview -===================== -VyOS source code is hosted on GitHub in the VyOS organization: -https://github.com/vyos - -VyOS is composed of multiple modules spread across different -repositories. Some modules contain forks of upstream -packages and are periodically synced. -VyOS consolidates most packages into the -`vyos-1x `__ -repository while maintaining a consistent structure. -The base code is being rewritten -from Perl and Bash to Python using an XML-based CLI interface definition. - -VyOS ISO build scripts are hosted in the -`vyos-build `__ repository. See the -``vyos-build`` repository -`README.md file `__ -for more information on building VyOS ISO images. - -Contributing code -================= - -.. warning:: - - You must sign the :doc:`Contributor License Agreement` - for your contributions to be accepted. - -VyOS is open-source and welcomes patches. -All submissions must adhere to these guidelines: - -* Each commit addresses a single issue or feature. -* Each commit message references a Phabricator_ task ID - (for example, ``T1234``). -* Each commit is associated with a username and email address - to identify the author (see `Configure your Git identity`_). -* Only submit bugfixes in packages other than https://github.com/vyos/vyos-1x. -* Commits follow the `coding guidelines`_ outlined below. - - -Determining package ownership ------------------------------ - -To determine which VyOS package contains a file you want to modify, use Debian's -``dpkg -S`` command on your running VyOS installation. - -Submitting your code --------------------- - -Fork the repository and submit a GitHub pull request. This is the preferred way -to contribute changes to VyOS. - -To fork a VyOS repository: - -1. Append ``/fork`` to the repository URL on GitHub. For example, to fork - ``vyos-1x``, use: https://github.com/vyos/vyos-1x/fork - -2. Clone your fork or add it as a remote to your local repository: - - - Clone: ``git clone https://github.com//vyos-1x.git`` - - Add remote: ``git remote add myfork https://github.com//vyos-1x.git`` - -.. _Configure your Git identity: - -3. Configure your Git identity: - - .. code-block:: none - - git config --global user.name "J. Random Hacker" - git config --global user.email "jrhacker@example.net" - -4. Make your changes and add files to the Git index: - - - Single file: ``git add myfile`` - - Directory: ``git add somedir/*`` - -5. Commit your changes with a meaningful headline and Phabricator_ reference: - - ``git commit`` - -6. Push to your fork and create a GitHub pull request: - - ``git push`` - -Alternatively, you can export commits as patches and send them to -maintainers@vyos.net or attach them directly to the Phabricator_ task: - -* Export last commit: ``git format-patch`` -* Export last two commits: ``git format-patch -2`` - -Commit messages -=============== - -For guidance on writing commit messages, review the file history -with ``git log path/to/file.txt``. - -Every change must be associated with a task number (prefixed with **T**) and -a component. If no bug report or feature request exists for your changes, -create a Phabricator_ task first. Reference the task ID in your commit message: - -* ``ddclient: T1030: auto create runtime directories`` -* ``Jenkins: add current Git commit ID to build description`` - -If your pull request lacks a Phabricator_ reference, maintainers will request -that you amend the commit message. - -Writing good commit messages ------------------------------ - -Follow the format described in -the `Git documentation `__ -and `Chris Beams' guide `__. - -Commit message format: - -1. **Summary line** (50 characters recommended, 80 maximum): Include the - component - prefix and Phabricator_ reference (for example, ``snmp: T1111:`` or - ``ethernet: T2222:``). Concatenate multiple components with colons - (for example, ``snmp: ethernet: T3333``). - -2. **Blank line**: Separate the summary from the body. - This blank line is critical. - -4. **Message body** with details: - - * Describe what changed, why, and how. This helps with ``git bisect``. - * Wrap text at 72 characters for readability with ``git log`` on an 80x25 - terminal. - * Reference previous commits when applicable: - ``After commit abcd12ef ("snmp: this is a headline") - a Python import statement is missing, throwing the following exception: - ABCDEF`` - -5. **Cherry-pick option**: Always use the ``-x`` option when back-porting or - forward-porting commits: - - ``git cherry-pick -x `` - - This appends ``(cherry picked from commit )`` to the commit message, - making bisecting easier. - -6. **Single responsibility**: Each commit must be self-contained. Do not fix - multiple bugs in a single commit. Use ``git add --patch`` to stage only - the parts related to one issue. - -Constraints: - -* Bugfixes are only accepted for packages other than - https://github.com/vyos/vyos-1x. - New functionality must use the new XML/Python interface, not old-style - templates (``node.def`` files and Perl/Bash code). - -Coding guidelines -================= - -VyOS maintains consistent coding standards to help contributors navigate the -codebase and understand its logic. - -Formatting ----------- - -* **Python**: Use 4 spaces per indentation level. Tabs **must not** be used. -* **XML**: Use 2 spaces per indentation level. Tabs **must not** be used. - -Use tools like VIM extensions (xmllint) to enforce correct indentation. Add this -to your ``.vimrc`` file: - -.. code-block:: none - - au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null - -Then use ``gg=G`` in command mode to run the linter. - -Text generation ---------------- - -Use a template processor for generating config files: - -* **Jinja2** is the default template processor for VyOS code. -* Built-in string formatting **may** be used for simple line-oriented formats - (for example, iptables rules) where every line is self-contained. -* Template processors **must** be used for structured, multi-line formats - (for example, ISC DHCPd configuration). - -Python code ------------ - -Configuration scripts and operation mode scripts written in Python3 should -follow these guidelines: - -* Wrap lines at 80 characters. This improves readability when browsing - GitHub on mobile devices and reads well in side-by-side diffs. - -Structure your scripts with these functions: - -.. code-block:: python - - #!/usr/bin/env python3 - # - # Copyright (C) 2020 VyOS maintainers and contributors - # - # This program is free software; you can redistribute it and/or modify - # it under the terms of the GNU General Public License version 2 or later as - # published by the Free Software Foundation. - # - # This program is distributed in the hope that it will be useful, - # but WITHOUT ANY WARRANTY; without even the implied warranty of - # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - # GNU General Public License for more details. - # - # You should have received a copy of the GNU General Public License - # along with this program. If not, see . - - import sys - - from vyos.config import Config - from vyos import ConfigError - - def get_config(config=None): - if config: - conf = config - else: - conf = Config() - - # Base path to CLI nodes - base = ['...', '...'] - # Convert the VyOS config to an abstract internal representation - config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) - return config_data - - def verify(config): - # Verify that configuration is valid - if invalid: - raise ConfigError("Descriptive message") - - def generate(config): - # Generate daemon configs - pass - - def apply(config): - # Apply the generated configs to the live system - pass - - try: - c = get_config() - verify(c) - generate(c) - apply(c) - except ConfigError as e: - print(e) - sys.exit(1) - -``get_config()``: This function converts a VyOS config object to an abstract -internal representation. No other function may call the ``vyos.config.Config`` -object directly. Limiting config reads to one function makes it easier to -modify the config syntax in the future. Additionally, this design improves -testability since you can construct an internal representation by hand rather -than mocking the entire config subsystem. - -``verify()``: This function validates the internal representation. It must -raise ``ConfigError`` with a descriptive message if the config is invalid. It -**must not** make any changes to the system. This design enables future features -like commit dry-run ("commit test" as in JunOS) where the system can abort a -commit before making changes. - -``generate()``: This function generates config files for system components. - -``apply()``: This function applies the generated configuration to the live -system. Prefer non-disruptive reload when possible. Disruptive operations like -daemon restarts are acceptable only when: - -* The component does not support non-disruptive reload, or -* The expected service degradation is minimal (for example, auxiliary services - like LLDPd) - -For high-impact services (VPN daemons, routing protocols), make effort to -determine if changes can be applied non-disruptively before resorting to -restarts. - -Never modify active configuration directly unless absolutely necessary. Instead, -generate configuration files and apply them with a single command like service -reload through systemd. For example, save iptables rules to a file and load them -with ``iptables-restore`` rather than executing iptables commands one by one. - -The ``apply()`` and ``generate()`` functions may raise ``ConfigError`` if the -daemon fails to start with the updated config. However, this is not a substitute -for proper config validation in the ``verify()`` function. Make reasonable -effort to verify that generated configuration is valid and will be accepted by -the daemon, including cross-checks with other VyOS configuration subtrees when -necessary. - -Exceptions like ``VyOSError`` (raised by ``vyos.config.Config`` on improper -operations) should not be silenced or caught. While this may produce less -polished error output for users, it generates better bug reports and helps -maintainers debug issues. - -For reference implementations, see ``ntp.py`` or ``interfaces-bonding.py`` (for -tag nodes) in the `vyos-1x `__ repository. - -Other considerations: ``vyos-configd`` --------------------------------------- - -All scripts now run under the config daemon and must conform to these -requirements: - -1. The signature and first four lines of ``get_config(...)`` **must** be as - specified above. - -2. Each of ``get_config``, ``verify``, ``apply``, and ``generate`` **must** - appear - with the correct signatures, even if they are a no-op. - -3. ``Config`` objects other than those in ``get_config`` **must not** appear. - -4. The legacy function ``my_set`` **must not** appear. Modifications to active - config **should not** appear in new code (alternative mechanisms may be used - if absolutely necessary). - -XML for CLI definitions -======================= - -XML interface definitions define the VyOS CLI structure. -Before VyOS ``1.2`` (crux), these -files were created manually. After a redesign, new-style templates are -automatically generated from XML input files. - -VyOS interface definitions come with a RelaxNG schema located in the -`vyos-1x `__ -repository. This schema is a modified version from ``VyConf`` (VyOS ``2.0``). -VyOS ``1.2.x`` -interface definitions are reusable in future VyOS versions with minimal changes. - -Schemas provide two benefits: - -* Complete grammar verification -* Automatic validation against the schema - -.. stop_vyoslinter -The `build-command-templates `__ -script converts XML definitions to -old-style templates and verifies them against the schema. A bad definition -causes the package build to fail. While the XML format is verbose, no other -format provides this level of verification. Specialized XML editors can help -manage verbosity. -.. start_vyoslinter - -Example XML interface definition: - -.. code-block:: xml - - - - - - - - - Task scheduler settings - - - - - Scheduled task - - <string> - Task name - - 999 - - - - - UNIX crontab time specification string - - - - - Execution interval - - <minutes> - Execution interval in minutes - - - <minutes>m - Execution interval in minutes - - - <hours>h - Execution interval in hours - - - <days>d - Execution interval in days - - - [1-9]([0-9]*)([mhd]{0,1}) - - - - - - Executable path and arguments - - - - - Path to executable - - - - - Arguments passed to the executable - - - - - - - - - - - - -XML definitions are purely declarative and contain no logic. All logic for -generating config files, restarting services, and related tasks is implemented -in configuration scripts. - -Template Processors -------------------- - -XML interface definition files use the ``.xml.in`` file extension (implemented -in :vytask:`T1843`). These files use the GCC preprocessor to reduce code -duplication in common areas: - -* VIF (including VIF-S and VIF-C) -* Address configuration -* Description -* Enabled/Disabled state - -Instead of repeating XML nodes, use include files with predefined features: - -.. stop_vyoslinter - -* `IPv4, IPv6, and DHCP(v6) `__ - address assignment. -* `IPv4 and IPv6 `__ - address assignment. -* `VLAN (VIF) `__ - definition. -* `MAC address `__ - assignment. - - -The ``.in`` files are preprocessed and stored in the `interface-definitions `__ -folder. The `scripts/build-command-templates `__ -script then operates on this folder to generate all required CLI nodes. - -.. start_vyoslinter - -Example preprocessor output: - -.. code-block:: none - - $ make interface_definitions - install -d -m 0755 build/interface-definitions - install -d -m 0755 build/op-mode-definitions - Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in - Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in - Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in - Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in - Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in - Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in - [...] - -Command Definition Guidelines ------------------------------- - -Use of Numbers -^^^^^^^^^^^^^^ - -Avoid using numbers in command names unless the number is part of a protocol -name or similar. For example, ``protocols ospfv3`` is appropriate, -but ``server-1`` is questionable. - -Help Strings -^^^^^^^^^^^^ - -Follow these guidelines for consistent, readable help strings: - -Capitalization and Punctuation -"""""""""""""""""""""""""""""" - -* Capitalize the first word of every help string. -* Do not use a period at the end of help strings. - -This standard mirrors network device CLIs and improves aesthetics. - -Examples: - -* Good: "Frobnication algorithm" -* Bad: "frobnication algorithm" -* Bad: "Frobnication algorithm." -* Incorrect: "frobnication algorithm." - -Abbreviations and Acronyms -"""""""""""""""""""""""""" - -* Capitalize all abbreviations and acronyms. - -Examples: - -* Good: "TCP connection timeout" -* Bad: "tcp connection timeout" -* Bad: "Tcp connection timeout" - -* Capitalize acronyms to distinguish them from normal words. - -Examples: - -* Good: RADIUS (remote authentication for dial-in user services) -* Bad: radius (unless referring to circular distance) - -* Follow accepted spelling conventions for mixed-case abbreviations. If it - contains "over" or "version", use lowercase. Follow RFC or standard spellings - when they exist. - -Examples: - -* Good: PPPoE, IPsec -* Bad: PPPOE, IPSEC -* Bad: pppoe, ipsec - -Verbs -""""" - -* Avoid verbs. If a verb can be omitted, omit it. - -Examples: - -* Good: "TCP connection timeout" -* Bad: "Set TCP connection timeout" - -* When a verb is essential, use it. For example: "Disable IPv6 forwarding on - all interfaces" for ``set system ipv6 disable-forwarding``. - -* Use infinitive form for necessary verbs. - -Examples: - -* Good: "Disable IPv6 forwarding" -* Bad: "Disables IPv6 forwarding" - - -C++ Backend Code -================ - -The VyOS CLI parser combines bash, bash-completion helpers, and the C++ backend -library `vyatta-cfg `__. This section -references common CLI commands and their C/C++ entry points: - -``set``: - -.. stop_vyoslinter - -* https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352 -* https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549 - -``commit``: - -* https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252 - -.. include:: /_include/common-references.txt - -.. start_vyoslinter diff --git a/docs/contributing/index.md b/docs/contributing/index.md new file mode 100644 index 00000000..f26a6b70 --- /dev/null +++ b/docs/contributing/index.md @@ -0,0 +1,13 @@ +# Contributing + +```{toctree} +:maxdepth: 1 + +build-vyos +development +cla +issues-features +upstream-packages +debugging +testing +``` diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst deleted file mode 100644 index 18b925ec..00000000 --- a/docs/contributing/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -############ -Contributing -############ - -.. toctree:: - :maxdepth: 1 - - build-vyos - development - cla - issues-features - upstream-packages - debugging - testing diff --git a/docs/contributing/issues-features.md b/docs/contributing/issues-features.md new file mode 100644 index 00000000..ab235326 --- /dev/null +++ b/docs/contributing/issues-features.md @@ -0,0 +1,122 @@ +--- +lastproofread: '2025-12-08' +--- + +(issues_features)= + +# Issues/Feature requests + +(bug_report)= + +## Bug Report/Issue + +Issues and bugs occur in every software project, and VyOS is no exception. + +### I found a bug, what should I do? + +When you find a potential bug, first: + +- Consult the [documentation] to ensure you configured your system + correctly. +- Check if the VyOS community has identified a workaround for the bug through + [Slack] or the VyOS [Forum]. + +### Ensure the bug is reproducible + +Include the following information when reporting a bug: + +- A sequence of configuration commands or a complete configuration file needed + to recreate the bug. Avoid partial configurations: a sequence of commands is + easy to paste and a complete configuration is easy to load, but a partial + config is hard to reconstruct. +- Describe the expected behavior and how it differs from what you observe. + Include command outputs or traffic dumps. Explain briefly why these outputs + are incorrect and what the correct behavior should be. +- A sequence of actions that trigger the bug. While not always possible, this + helps developers and community members confirm the issue and verify fixes. +- If the bug is a regression, specify the VyOS version where the feature worked + correctly (any working version is acceptable). Identify the exact version + that the feature stopped working, if possible. + +If you are uncertain whether the behavior is a bug or what the correct behavior +is, or if you lack a reliable reproducing procedure, post on the forum or ask in +chat first. If you have a subscription, create a support ticket. The team and +community can help identify the issue, work around it, and create an actionable +bug report. + +### Report a Bug + +To open a bug report or feature request, create an account on +[vyos.dev](https://vyos.dev), the public issue tracker for VyOS. + +When creating a new issue, select the appropriate project and: + +- Provide as much information as you can. +- Specify which VyOS version you are using: `run show version`. +- Explain how to reproduce the bug. + +(feature-request)= + +## Feature Requests + +Have an idea to improve VyOS or need a feature that would benefit all users? +Before submitting a feature request, search the public issue tracker +[vyos.dev](https://vyos.dev) to check if a request already exists. You can +also enhance an existing request by providing additional information. + +Create a task before starting work on a feature, +even if it is a trivial feature. +The task tracker generates release notes, so all work must be reflected +in the tracker. + +Include at least the following information: + +- Provide a detailed description of the feature: what it is, how it works, and + how you would use it. Maintainers may not have experience with every feature, + protocol, and tool in VyOS. Detailed information helps VyOS contributors and + maintainers test new features they are unfamiliar with. +- Include proposed CLI syntax if the feature requires new commands. Provide both + configuration and operational mode commands if both are needed. + +Consider including the following information: + +- Is the feature already supported by the underlying component + (FreeRangeRouting, nftables, Kea, etc.)? +- How would you configure the feature manually within that component? +- Are there any limitations to using the feature + (hardware support, resource usage)? +- Are there any adverse or non-obvious interactions with other features? Should + the feature be mutually exclusive? +- Any relevant documentation or references about the feature. + +You do not need to provide all this information, but if you can, it simplifies +developers' work considerably. Research these questions when possible. + +## Task auto-closing + +A special task status exists for when all work by maintainers and contributors +is complete: **Needs reporter action**. + +VyOS assigns this status to: + +- Feature requests that do not include required information and need + clarification. +- Bug reports that lack reproducing procedures. +- Tasks that are implemented and tested by the implementation author, + but require testing in the real-world environment that only the reporter + can replicate (for example, hardware VyOS does not support or specific + network conditions). + +When a task is set to **Needs reporter action**: + +- If the reporter does not respond within two weeks, the task bot adds a comment + ("Any news?") to remind the reporter. +- If there is still no response after another two weeks, + the task is closed automatically. + +We do not auto-close tasks with any other status and do not close tasks due to +lack of maintainer activity. + +[documentation]: https://docs.vyos.io +[forum]: https://forum.vyos.io +[slack]: https://slack.vyos.io diff --git a/docs/contributing/issues-features.rst b/docs/contributing/issues-features.rst deleted file mode 100644 index b1e1d732..00000000 --- a/docs/contributing/issues-features.rst +++ /dev/null @@ -1,130 +0,0 @@ -:lastproofread: 2025-12-08 - -.. _issues_features: - -####################### -Issues/Feature requests -####################### - -.. _bug_report: - -Bug Report/Issue -================ - -Issues and bugs occur in every software project, and VyOS is no exception. - -I found a bug, what should I do? --------------------------------- - -When you find a potential bug, first: - -* Consult the documentation_ to ensure you configured your system - correctly. -* Check if the VyOS community has identified a workaround for the bug through - Slack_ or the VyOS Forum_. - -Ensure the bug is reproducible ------------------------------- - -Include the following information when reporting a bug: - -* A sequence of configuration commands or a complete configuration file needed - to recreate the bug. Avoid partial configurations: a sequence of commands is - easy to paste and a complete configuration is easy to load, but a partial - config is hard to reconstruct. -* Describe the expected behavior and how it differs from what you observe. - Include command outputs or traffic dumps. Explain briefly why these outputs - are incorrect and what the correct behavior should be. -* A sequence of actions that trigger the bug. While not always possible, this - helps developers and community members confirm the issue and verify fixes. -* If the bug is a regression, specify the VyOS version where the feature worked - correctly (any working version is acceptable). Identify the exact version - that the feature stopped working, if possible. - -If you are uncertain whether the behavior is a bug or what the correct behavior -is, or if you lack a reliable reproducing procedure, post on the forum or ask in -chat first. If you have a subscription, create a support ticket. The team and -community can help identify the issue, work around it, and create an actionable -bug report. - -Report a Bug ------------- - -To open a bug report or feature request, create an account on -`vyos.dev `__, the public issue tracker for VyOS. - -When creating a new issue, select the appropriate project and: - -* Provide as much information as you can. -* Specify which VyOS version you are using: ``run show version``. -* Explain how to reproduce the bug. - -.. _feature_request: - -Feature Requests -================ - -Have an idea to improve VyOS or need a feature that would benefit all users? -Before submitting a feature request, search the public issue tracker -`vyos.dev `__ to check if a request already exists. You can -also enhance an existing request by providing additional information. - -Create a task before starting work on a feature, -even if it is a trivial feature. -The task tracker generates release notes, so all work must be reflected -in the tracker. - -Include at least the following information: - -* Provide a detailed description of the feature: what it is, how it works, and - how you would use it. Maintainers may not have experience with every feature, - protocol, and tool in VyOS. Detailed information helps VyOS contributors and - maintainers test new features they are unfamiliar with. -* Include proposed CLI syntax if the feature requires new commands. Provide both - configuration and operational mode commands if both are needed. - -Consider including the following information: - -* Is the feature already supported by the underlying component - (FreeRangeRouting, nftables, Kea, etc.)? -* How would you configure the feature manually within that component? -* Are there any limitations to using the feature - (hardware support, resource usage)? -* Are there any adverse or non-obvious interactions with other features? Should - the feature be mutually exclusive? -* Any relevant documentation or references about the feature. - -You do not need to provide all this information, but if you can, it simplifies -developers' work considerably. Research these questions when possible. - -Task auto-closing -================= - -A special task status exists for when all work by maintainers and contributors -is complete: **Needs reporter action**. - -VyOS assigns this status to: - -* Feature requests that do not include required information and need - clarification. -* Bug reports that lack reproducing procedures. -* Tasks that are implemented and tested by the implementation author, - but require testing in the real-world environment that only the reporter - can replicate (for example, hardware VyOS does not support or specific - network conditions). - -When a task is set to **Needs reporter action**: - -* If the reporter does not respond within two weeks, the task bot adds a comment - ("Any news?") to remind the reporter. -* If there is still no response after another two weeks, - the task is closed automatically. - -We do not auto-close tasks with any other status and do not close tasks due to -lack of maintainer activity. - -.. _documentation: https://docs.vyos.io -.. _Slack: https://slack.vyos.io -.. _Forum: https://forum.vyos.io - -.. include:: /_include/common-references.txt diff --git a/docs/contributing/md-build-vyos.md b/docs/contributing/md-build-vyos.md deleted file mode 100644 index bfda5298..00000000 --- a/docs/contributing/md-build-vyos.md +++ /dev/null @@ -1,730 +0,0 @@ ---- -lastproofread: '2025-12-05' ---- - -(build)= - -# Build VyOS - -## Prerequisites - -There are different ways you can build VyOS. Building using a -{ref}`build_docker` -container is the easiest way because all dependencies are managed for you. -Alternatively, you can set up your own build machine and run a -{ref}`build_native` build. - -:::{note} -Starting with VyOS 1.4, only source code and Debian package -repositories of the rolling release (the **current** branch) are publicly -available. - -The source code and pre-built Debian package repositories of LTS releases -are only available to subscription holders (customers and active community -members with contributors subscriptions). - -The following includes the build process for VyOS rolling release. -::: - -This will guide you through the process of building a VyOS ISO using [Docker]. -This process has been tested on clean installs of Debian Bookworm. - -(build_native)= - -### Native Build - -To build VyOS natively, you need a properly configured build host with -Debian Bookworm installed. - -To get started, clone the repository to your local machine: - -```none -$ sudo make clean -$ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic -``` - -For required packages, refer to the `docker/Dockerfile` file in the -[repository]. The `./build-vyos-image` script will also warn you if any -dependencies are missing. - -(build_docker)= - -### Docker - -Installing [Docker] and prerequisites: - -:::{hint} -Docker versions are updated frequently. The following examples may -become outdated. -::: - -```none -# Add Docker's official GPG key: -sudo apt-get update -sudo apt-get install ca-certificates curl gnupg -sudo install -m 0755 -d /etc/apt/keyrings -curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg -sudo chmod a+r /etc/apt/keyrings/docker.gpg - -# Add the repository to Apt sources: -echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ - $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null - -sudo apt-get update -sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -``` - -To use Docker without `sudo`, add your current non-root user to the `docker` -group: `sudo usermod -aG docker yourusername`. - -:::{hint} -Adding a user to the `docker` group grants privileges equivalent to -`root`. It is recommended to remove the non-root user from the `docker` -group after building the VyOS ISO. See also [Docker as non-root]. -::: - -:::{note} -The build process must run on a local file system. Building on SMB or -NFS shares will cause the container to fail. VirtualBox shared folders are -also not supported because block device operations are not implemented. -::: - -#### Build Container - -The container can be built by hand or by fetching the pre-built one from -DockerHub. It is recommended to use the pre-built containers from the -[VyOS DockerHub organisation](https://hub.docker.com/u/vyos). The container -is built from Docker packages automatically after every commit to the -``vyos-build`` repository (this process may take 2-3 hours). - -:::{note} -If you use the pre-built container, it will be automatically -downloaded from DockerHub if it is not found on your local machine when -you build the ISO. -::: - -##### Dockerhub - -To manually download the container from DockerHub, run: - -```none -$ docker pull vyos/vyos-build:current # For VyOS rolling release -``` - - -##### Build from source - -The container can also be built directly from source: - -```none -$ git clone -b current --single-branch https://github.com/vyos/vyos-build - -$ cd vyos-build -$ docker build -t vyos/vyos-build:current docker -``` - -:::{note} -VyOS switched to Debian Bookworm (12) in its `current` branch. -Due to software version updates, it is recommended to use the official -Docker Hub image to build VyOS ISO. -::: - -#### Tips and Tricks - -You can create Bash aliases to easily launch the latest container per release -train (`current`). Add the following to your `.bash_aliases` file: - -```none -alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ - -v "$(pwd)":/vyos \ - -v "$HOME/.gitconfig":/etc/gitconfig \ - -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ - -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ - -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ - vyos/vyos-build:current bash' -``` - -Now you have a new alias `vybld` that launches development containers in -your current working directory. - -:::{note} -Some VyOS packages (namely vyos-1x) come with build-time tests which -verify some of the internal library calls that they work as expected. Those -tests are carried out through the Python Unittest module. If you want to -build the `vyos-1x` package (which is our main development package) you -need to start your Docker container using the following argument: -`--sysctl net.ipv6.conf.lo.disable_ipv6=0`, otherwise those tests will -fail. -::: - -(build-iso)= - -## Build ISO - -Now that you understand the prerequisites, you can build a VyOS ISO from source. -First, fetch the latest source code from GitHub: - -```none -$ git clone -b current --single-branch https://github.com/vyos/vyos-build -``` - -Now you can begin a fresh VyOS ISO build. Change to the `vyos-build` -directory and run: - -```none -$ cd vyos-build -$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash -``` - -Start the build: - -```none -vyos_bld@8153428c7e1f:/vyos$ sudo make clean -vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic -``` - -When the build is successful, find the resulting ISO in the `build` directory -as `live-image-[architecture].hybrid.iso`. - -(build-source)= - -(customize)= - -### Customize - -You can customize the ISO with the following configure options. Generate the -full and current list with `./build-vyos-image --help`: - -```none -$ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help - I: Checking if packages required for VyOS image build are installed - usage: build-vyos-image [-h] [--architecture ARCHITECTURE] - [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR] - [--debian-security-mirror DEBIAN_SECURITY_MIRROR] - [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] - [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] - [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run] - [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY] - [--custom-package CUSTOM_PACKAGE] - [build_flavor] - - positional arguments: - build_flavor Build flavor - - optional arguments: - -h, --help show this help message and exit - --architecture ARCHITECTURE - Image target architecture (amd64 or arm64) - --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) - --debian-mirror DEBIAN_MIRROR - Debian repository mirror - --debian-security-mirror DEBIAN_SECURITY_MIRROR - Debian security updates mirror - --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR - Debian repository mirror for pbuilder env bootstrap - --vyos-mirror VYOS_MIRROR - VyOS package mirror - --build-type BUILD_TYPE - Build type, release or development - --version VERSION Version number (release builds only) - --build-comment BUILD_COMMENT - Optional build comment - --debug Enable debug output - --dry-run Check build configuration and exit - --custom-apt-entry CUSTOM_APT_ENTRY - Custom APT entry - --custom-apt-key CUSTOM_APT_KEY - Custom APT key file - --custom-package CUSTOM_PACKAGE - Custom package to install from repositories -``` - -(iso_build_issues)= - -#### ISO Build Issues - -There are (rare) situations where building an ISO image is not possible at all -due to a broken package feed in the background. APT is not very good at -reporting the root cause of the issue. Your ISO build will likely fail with a -more or less similar looking error message: - -```none -The following packages have unmet dependencies: - vyos-1x : Depends: accel-ppp but it is not installable -E: Unable to correct problems, you have held broken packages. -P: Begin unmounting filesystems... -P: Saving caches... -Reading package lists... -Building dependency tree... -Reading state information... -Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB] -Del accel-ppp 1.12.0-95-g59f8e1b [475 kB] -Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB] -Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB] -Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB] -make: *** [Makefile:30: iso] Error 1 -(10:13) vyos_bld ece068908a5b:/vyos [current] # -``` - -To debug the build process and gain additional information of what could be the -root cause, you need to use `chroot` to change into the build directory. This is -explained in the following step by step procedure: - -```none -vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash -``` - -We now need to mount some required, volatile filesystems - -```none -(live)root@ece068908a5b:/# mount -t proc none /proc -(live)root@ece068908a5b:/# mount -t sysfs none /sys -(live)root@ece068908a5b:/# mount -t devtmpfs none /dev -``` - -We now are free to run any command we would like to use for debugging, e.g. -re-installing the failed package after updating the repository. - -```none -(live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x -Get:1 file:/root/packages ./ InRelease -Ign:1 file:/root/packages ./ InRelease -Get:2 file:/root/packages ./ Release [1235 B] -Get:2 file:/root/packages ./ Release [1235 B] -Get:3 file:/root/packages ./ Release.gpg -Ign:3 file:/root/packages ./ Release.gpg -Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease -Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease -Hit:6 http://deb.debian.org/debian bullseye InRelease -Hit:7 http://deb.debian.org/debian buster InRelease -Hit:8 http://deb.debian.org/debian-security buster/updates InRelease -Hit:9 http://deb.debian.org/debian buster-updates InRelease -Hit:10 http://deb.debian.org/debian buster-backports InRelease -Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease -Reading package lists... Done -N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) -Reading package lists... Done -Building dependency tree -Reading state information... Done -Some packages could not be installed. This may mean that you have -requested an impossible situation or if you are using the unstable -distribution that some required packages have not yet been created -or been moved out of Incoming. -The following information may help to resolve the situation: - -The following packages have unmet dependencies: - vyos-1x : Depends: accel-ppp but it is not installable -E: Unable to correct problems, you have held broken packages. -``` - -Now it's time to fix the package mirror and rerun the last step until the -package installation succeeds again! - -(build_custom_packages)= - -### Linux Kernel - -The Linux kernel used by VyOS is heavily tied to the ISO build process. The -file `data/defaults.json` hosts a JSON definition of the kernel version used -`kernel_version` and the `kernel_flavor` of the kernel which represents the -kernel's LOCAL_VERSION. Both together form the kernel version variable in the -system: - -```none -vyos@vyos:~$ uname -r -6.1.52-amd64-vyos -``` - -- Accel-PPP -- Intel NIC drivers -- Intel QAT - -Each of those modules holds a dependency on the kernel version and if you are -lucky enough to receive an ISO build error which sounds like: - -```none -I: Create initramfs if it does not exist. -Extra argument '6.1.52-amd64-vyos' -Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] -Options: - -k version Specify kernel version or 'all' - -c Create a new initramfs - -u Update an existing initramfs - -d Remove an existing initramfs - -b directory Set alternate boot directory - -v Be verbose -See update-initramfs(8) for further details. -E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. -``` - -The most obvious reasons could be: - -- `vyos-build` repo is outdated, please `git pull` to update to the latest - release kernel version from us. -- You have your own custom kernel `*.deb` packages in the `packages` folder but - neglected to create all required out-of tree modules like Accel-PPP, Intel - QAT or Intel NIC drivers - -#### Building The Kernel - -The kernel build is quite easy, most of the required steps can be found in the -`vyos-build/packages/linux-kernel/Jenkinsfile` but we will walk you through -it. - -Clone the kernel source to `vyos-build/packages/linux-kernel/`: - -```none -$ cd vyos-build/packages/linux-kernel/ -$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git -``` - -Check out the required kernel version - see `vyos-build/data/defaults.json` -file (example uses kernel 4.19.146): - -```none -$ cd vyos-build/packages/linux-kernel/linux -$ git checkout v4.19.146 -Checking out files: 100% (61536/61536), done. -Note: checking out 'v4.19.146'. - -You are in 'detached HEAD' state. You can look around, make experimental -changes and commit them, and you can discard any commits you make in this -state without impacting any branches by performing another checkout. - -If you want to create a new branch to retain commits you create, you may -do so (now or later) by using -b with the checkout command again. Example: - git checkout -b -HEAD is now at 015e94d0e37b Linux 4.19.146 -``` - -Now you can use the helper script `build-kernel.sh`, which completes all -the necessary steps: applying required patches from the -`vyos-build/packages/linux-kernel/patches` folder, copying the kernel -configuration `x86_64_vyos_defconfig` to the correct location, and building -the Debian packages. - -:::{note} -Building the kernel will take some time depending on the speed and -quantity of your CPU/cores and disk speed. Expect 20 minutes -(or even longer) on lower end hardware. -::: - -```none -(18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh -I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source -I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch -patching file Documentation/networking/ip-sysctl.txt -patching file include/linux/inetdevice.h -patching file include/linux/ipv6.h -patching file include/uapi/linux/ip.h -patching file include/uapi/linux/ipv6.h -patching file net/ipv4/devinet.c -Hunk #1 succeeded at 2319 (offset 1 line). -patching file net/ipv6/addrconf.c -patching file net/ipv6/route.c -I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch -patching file fs/notify/inotify/Kconfig -patching file fs/notify/inotify/inotify_user.c -patching file fs/overlayfs/super.c -Hunk #2 succeeded at 1713 (offset 9 lines). -Hunk #3 succeeded at 1739 (offset 9 lines). -Hunk #4 succeeded at 1762 (offset 9 lines). -patching file include/linux/inotify.h -I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch -patching file scripts/package/builddeb -I: make x86_64_vyos_defconfig - HOSTCC scripts/basic/fixdep - HOSTCC scripts/kconfig/conf.o - YACC scripts/kconfig/zconf.tab.c - LEX scripts/kconfig/zconf.lex.c - HOSTCC scripts/kconfig/zconf.tab.o - HOSTLD scripts/kconfig/conf -# - -# configuration written to .config -# -I: Generate environment file containing Kernel variable -I: Build Debian Kernel package - UPD include/config/kernel.release -/bin/sh ./scripts/package/mkdebian -dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc -dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos -dpkg-buildpackage: info: source version 4.19.146-1 -dpkg-buildpackage: info: source distribution buster -dpkg-buildpackage: info: source changed by vyos_bld -dpkg-buildpackage: info: host architecture amd64 -dpkg-buildpackage: warning: debian/rules is not executable; fixing that - dpkg-source --before-build . - debian/rules build -make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= - SYSTBL arch/x86/include/generated/asm/syscalls_32.h -... -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory -dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) -dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) -dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. - dpkg-genbuildinfo --build=binary - dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes -dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list -dpkg-genchanges: info: binary-only upload (no source code included) - dpkg-source --after-build . -dpkg-buildpackage: info: binary-only upload (no source included) -``` - -When complete, you will have kernel binary packages to use in your custom ISO -build. Place all `*.deb` files in the `vyos-build/packages` folder, where -the build process will use them automatically. - -##### Firmware - -If you upgrade your kernel or include new drivers you may need new firmware. -This builds a new `vyos-linux-firmware` package using the included helper -scripts. - -```none -$ cd vyos-build/packages/linux-kernel -$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git -$ ./build-linux-firmware.sh -$ cp vyos-linux-firmware_*.deb ../ -``` - -The script automatically detects which firmware blobs are needed based on the -built drivers. If detection fails, you can manually add files to -`vyos-build/packages/linux-kernel/build-linux-firmware.sh`: - -```bash -ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin" -``` - - -#### Building Out-Of-Tree Modules - -Building the kernel is one step. You must also build required out-of-tree -modules so the ABIs match. -Refer to `vyos-build/packages/linux-kernel/Jenkinsfile` -for all required modules and their versions. We show you how to build the -currently required modules. - -##### Accel-PPP - -First, clone the source code and check out the appropriate version: - -```none -$ cd vyos-build/packages/linux-kernel -$ git clone https://github.com/accel-ppp/accel-ppp.git -``` - -Use the helper script and patches to build the package. Run the following -command: - -```none -$ ./build-accel-ppp.sh -I: Build Accel-PPP Debian package -CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): - The OLD behavior for policy CMP0003 will be removed from a future version - of CMake. - The cmake-policies(7) manual explains that the OLD behaviors of all - policies are deprecated and that a policy should be set to OLD only under - specific short-term circumstances. Projects should be ported to the NEW - behavior and not rely on setting a policy to OLD. --- The C compiler identification is GNU 8.3.0 -... - -CPack: Create package using DEB -CPack: Install projects -CPack: - Run preinstall target for: accel-ppp -CPack: - Install project: accel-ppp -CPack: Create package -CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. -``` - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in `vyos-build/packages/linux-kernel` from which you can copy them -to the `vyos-build/packages` folder for inclusion during the ISO build. - -##### Intel NIC - -The Intel NIC drivers do not come from a Git repository. VyOS fetches the -tarballs from a mirror and compiles them. Use the following wrapper script -to build all driver modules: - -```none -./build-intel-drivers.sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k -I: Compile Kernel module for Intel ixgbe driver -... - -I: Building Debian package vyos-intel-iavf -Doing `require 'backports'` is deprecated and will not load any backport in the next major release. -Require just the needed backports instead, or 'backports/latest'. -Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} -Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} -I: Cleanup iavf source -``` - -After compilation, find the generated `*.deb` binaries in -`vyos-build/packages/linux-kernel`. Copy them to the `vyos-build/packages` -folder for inclusion in the ISO build. - -##### Intel QAT - -The Intel QAT (Quick Assist Technology) drivers do not come from a Git -repository. VyOS fetches the tarballs from `01.org`, Intel's open-source -website. -Use the following wrapper script to build all driver modules: - -```none -$ ./build-intel-qat.sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed -100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k -I: Compile Kernel module for Intel qat driver -checking for a BSD-compatible install... /usr/bin/install -c -checking whether build environment is sane... yes -checking for a thread-safe mkdir -p... /bin/mkdir -p -checking for gawk... gawk -checking whether make sets $(MAKE)... yes -... - -I: Building Debian package vyos-intel-qat -Doing `require 'backports'` is deprecated and will not load any backport in the next major release. -Require just the needed backports instead, or 'backports/latest'. -Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} -Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} -I: Cleanup qat source -``` - -After compiling the packages you will find yourself the newly generated `*.deb` -binaries in `vyos-build/packages/linux-kernel` from which you can copy them -to the `vyos-build/packages` folder for inclusion during the ISO build. - -### Packages - -If you are brave enough to build your own ISO image with any modified package -from VyOS's GitHub organisation, this is the place for you. - -Any modified package may be an altered version (e.g., `vyos-1x`) that you -want to test before filing a pull request on GitHub. - -Building an ISO with a customized package is the same as building a regular -ISO image. Place your modified `*.deb` package inside the `packages` folder -within `vyos-build`. The build process will automatically use your custom -package during the ISO build. - -### Troubleshooting - -Debian APT does not provide verbose error messages. If your ISO build fails and -you suspect an APT dependencies or installation issue, you can apply this patch -to increase APT verbosity during the ISO build. - -```diff -diff --git i/scripts/live-build-config w/scripts/live-build-config -index 1b3b454..3696e4e 100755 ---- i/scripts/live-build-config -+++ w/scripts/live-build-config -@@ -57,7 +57,8 @@ lb config noauto \ - --firmware-binary false \ - --updates true \ - --security true \ -- --apt-options "--yes -oAcquire::Check-Valid-Until=false" \ -+ --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \ -+ -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \ - --apt-indices false - "${@}" - """ -``` - -(build-packages)= - -## Packages - -VyOS comes with specific packages that cannot be found in any -Debian mirror. These packages are located in the [VyOS GitHub project] in -source format and can easily be compiled into custom -Debian (`*.deb`) packages. - -The easiest way to compile your package is with the {ref}`build_docker` -container mentioned earlier, as it includes all required dependencies for all -VyOS related packages. - -Assuming you want to build the `vyos-1x` package and modify it for your needs, -first clone the repository from GitHub: - -```none -$ git clone --recurse-submodules https://github.com/vyos/vyos-1x -``` - - -### Build - -Launch the Docker container and build the package: - -```none -# For VyOS 1.3 (equuleus, current) -$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash - -# Change to source directory -$ cd vyos-1x - -# Build DEB -$ dpkg-buildpackage -uc -us -tc -b -``` - -After a minute or two, the generated DEB packages are located next to the -`vyos-1x` source directory: - -```none -# ls -al ../vyos-1x*.deb --rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb --rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb -``` - - -### Install - -To test your newly created package, you can SCP it to a running VyOS instance -and install the new `*.deb` package to replace the current one. - -Install the package using the following commands: - -```none -vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb -(Reading database ... 58209 files and directories currently installed.) -Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... -Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... -Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... -Processing triggers for rsyslog (8.1901.0-1) ... -``` - -You can also place the generated `*.deb` in your ISO build environment to -include it in a custom ISO. See {ref}`build_custom_packages` for more -information. - -:::{warning} -Any packages in the `packages` directory will be added to the -ISO during the build, replacing upstream packages. Delete both the source -directories and built DEB packages if you want to build an ISO from purely -upstream packages. -::: - -[docker]: https://docs.docker.com/engine/install/debian/ -[docker as non-root]: https://docs.docker.com/engine/install/linux-postinstall -[repository]: https://github.com/vyos/vyos-build -[vyos dockerhub organisation]: https://hub.docker.com/u/vyos -[vyos github project]: https://github.com/vyos diff --git a/docs/contributing/md-cla.md b/docs/contributing/md-cla.md deleted file mode 100644 index 01323111..00000000 --- a/docs/contributing/md-cla.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -lastproofread: '2025-12-05' ---- - -(cla)= - -# Contributor License Agreement - -Before we can accept your contributions to VyOS, you must sign a **Contributor -License Agreement (CLA)**. - -This is a standard open-source practice that protects both you and the project. - -The process is straightforward and fully automated: - -1. **Review the CLA document** - - Find the CLA text in our - [GitHub repository](https://github.com/vyos/vyos-cla-signatures/). - -2. **Submit a pull request** - - When you open a pull request, a CLA bot automatically checks whether all - commit authors have signed the CLA. - -3. **Follow the bot's instructions** - - If the CLA has not been signed, the bot leaves a comment with instructions. - Reply to that comment with the suggested text to sign the CLA. - -4. **Wait for confirmation** - - The CLA bot verifies your response and updates the pull request status. - Once all commit authors have signed, the bot confirms that the CLA - requirement is met and unlocks the pull request for merging. - -:::{note} -Each commit author must sign the CLA. - -If your pull request includes commits from multiple contributors, each one -must sign the CLA before the pull request can be accepted. -::: - -Once you sign the CLA, it remains valid for all your past and future -contributions to VyOS under the same GitHub identity. diff --git a/docs/contributing/md-debugging.md b/docs/contributing/md-debugging.md deleted file mode 100644 index d3b4b513..00000000 --- a/docs/contributing/md-debugging.md +++ /dev/null @@ -1,204 +0,0 @@ ---- -lastproofread: '2025-12-05' ---- - -(debugging)= - -# Debugging - -Two flags are available to help debug configuration scripts. Configuration -loading issues manifest during boot, so these flags are passed as kernel boot -parameters. - -## ISO image build - -If you have trouble compiling your own ISO image or debugging Jenkins issues, -follow the steps at {ref}`iso_build_issues`. - -## System Startup - -Debug system startup by examining the configuration file loading from -`/config/config.boot`. Extend the kernel command-line in the bootloader to -enable this. - -### Kernel - -- `vyos-debug` - Add this parameter to the Linux boot line to produce - timing results for script execution during commit. If you see an unexpected - delay during manual or boot commit, this parameter helps identify bottlenecks. - The internal flag is `VYOS_DEBUG`, found in [vyatta-cfg]. Output is directed - to `/var/log/vyatta/cfg-stdout.log`. -- `vyos-config-debug` - During development, coding errors can cause commit - failures on boot, potentially preventing CLI initialization. This kernel boot - parameter ensures access to the system as user `vyos` and logs a Python - stack trace to `/tmp/boot-config-trace`. The file is created only if the - configuration load fails. - -## Live System - -Several flags can be set to change VyOS behavior at runtime. Toggle these flags -using environment variables or by creating files. - -For each feature, create a file called `vyos.feature.debug` to enable it. -If a parameter is required, place it as the first line inside the file. - -Place the file in `/tmp` for one-time debugging (the file is removed on -reboot) or in `/config` to persist permanently. - -For example, `/tmp/vyos.ifconfig.debug` can be created to enable -interface debugging. - -You can also enable debugging using environment variables. -The environment variable name follows the convention `VYOS_FEATURE_DEBUG`. - -For example, `export VYOS_IFCONFIG_DEBUG=""` in your vbash has the same effect -as `touch /tmp/vyos.ifconfig.debug`. - -- `ifconfig` - Display all commands and their responses from the OS on - screen for inspection. -- `command` - Display all commands and their responses from the OS on screen - for inspection. -- `developer` - When a command fails, start a PDB post-mortem session instead - of showing a standard error message. This allows developers to debug issues - interactively. Because the debugger waits for input, it can prevent the router - from booting, so only enable this permanently on production systems if you are - ready for potential boot failures. -- `log` - Send all commands used by VyOS to a log file for inspection. This - is useful in rare cases when you need to see what the OS is doing, including - during boot. The default file is `/tmp/full-log`, but you can change it. - -:::{note} -To retrieve debug output on the command line, disable `vyos-configd` -in addition. You can do this one-time with -`sudo systemctl stop vyos-configd` -or permanently with `sudo systemctl disable vyos-configd`. -::: - -### FRR - -Recent versions use the `vyos.frr` framework. The Python class is located in -`vyos-1x:python/vyos/frr.py`. It includes an embedded debugger similar to the -one in `vyos.ifconfig`. - -Enable debugging by running: `touch /tmp/vyos.frr.debug` - -### Debug Python code with PDB - -Sometimes it is useful to debug Python code interactively on the live system -rather than in an IDE. You can do this using pdb. - -Assuming you want to debug a Python script called by an op-mode command, find -the script by looking up the op-mode definitions, then edit it on the live -system using vi: -`vi /usr/libexec/vyos/op_mode/show_xyz.py` - -Insert the following statement right before the section where you want to -investigate a problem (for example, a statement you see in a backtrace): -`import pdb; pdb.set_trace()` - -Optionally, surround this statement with an `if` condition that triggers only -for the conditions you are interested in. - -When you run `show xyz` and your condition triggers, you enter the Python -debugger: - -```none -> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() --> rule_type = rule.get('type', '') -(Pdb) -``` - -You can type `help` to get an overview of the available commands, and -`help command` to get more information on each command. - -Common useful commands include: - -- examine variables using `pp(var)` -- continue execution using `cont` -- get a backtrace using `bt` - -### Config Migration Scripts - -Starting with VyOS 1.5, a new mechanism is used for config migration that -improves migration performance. New migrators use only the new format with a -`migration()` function. - -```python -from vyos.configtree import ConfigTree -base = ['vpn', 'ipsec'] -def migrate(config: ConfigTree) -> None: - if not config.exists(base): - # Nothing to do - return - # do your stuff here -``` - -New-style migration scripts can no longer run on their own. However, the new -migration subsystem handler includes a test kit: - -```none -vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --help -usage: run-config-migration.py [-h] [--test-script TEST_SCRIPT] [--output-file OUTPUT_FILE] [--force] config_file - -positional arguments: - config_file configuration file to migrate - -options: - -h, --help show this help message and exit - --test-script TEST_SCRIPT - test named script - --output-file OUTPUT_FILE - write to named output file instead of config file - --force force run of all migration scripts -``` - -To test your migration, run: - -```none -vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --test-script /opt/vyatta/etc/config-migrate/migrate/quagga/11-to-12 --output-file /tmp/foo /tmp/static-route-basic -vyos@vyos:~$ cat /tmp/foo -``` - -The file `/tmp/foo` contains the migrated configuration. - -### Configuration Error on System Boot - -Running the latest rolling releases sometimes exposes bugs due to edge cases -missed in design. File these bugs via [Phabricator](https://vyos.dev/), but you can help narrow -down the issue by following these steps: - -1. Log in to your VyOS system. -2. Enter configuration mode: `configure` -3. Reload your boot configuration: `load` - -You should see a Python backtrace that helps identify the issue. Attach it to -the [Phabricator](https://vyos.dev/) task. - -### Boot Timing - -During the migration and rewrite of functionality from Perl to Python, system -boot time increased significantly. You can analyze and graph boot time to see -detailed call sequences during startup. - -This uses the `systemd-bootchart` package, which is installed by default on -VyOS 1.3 (equuleus) and later. Configuration is versioned for comparable -results. Refer to [bootchart.conf] for the configuration file. - -To enable boot time graphing, add the following to the kernel command line: -`init=/usr/lib/systemd/systemd-bootchart` - -You can also make this permanent by editing `/boot/grub/grub.cfg`. - -## Priorities - -VyOS CLI depends heavily on priorities. Every CLI node has a corresponding -`node.def` file and possibly an attached script. Nodes can have priorities, -and on system bootup or any `commit` to the configuration, scripts execute -from lowest to highest priority. This provides deterministic behavior. - -To debug priority issues or see script execution order, use the -`/opt/vyatta/sbin/priority.pl` script, which lists the execution order of -scripts. - -[bootchart.conf]: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf -[vyatta-cfg]: https://github.com/vyos/vyatta-cfg diff --git a/docs/contributing/md-development.md b/docs/contributing/md-development.md deleted file mode 100644 index 8d2cec40..00000000 --- a/docs/contributing/md-development.md +++ /dev/null @@ -1,549 +0,0 @@ ---- -lastproofread: '2025-12-12' ---- - -(development)= - -# Development - -Learn how to contribute to VyOS. - -(architecture-overview)= - -## Architecture overview - -VyOS source code is hosted on GitHub in the VyOS organization: - - -VyOS is composed of multiple modules spread across different -repositories. Some modules contain forks of upstream -packages and are periodically synced. -VyOS consolidates most packages into the -[vyos-1x](https://github.com/vyos/vyos-1x) -repository while maintaining a consistent structure. -The base code is being rewritten -from Perl and Bash to Python using an XML-based CLI interface definition. - -VyOS ISO build scripts are hosted in the -[vyos-build](https://github.com/vyos/vyos-build) repository. See the -`vyos-build` repository -[README.md file](https://github.com/vyos/vyos-build/blob/current/README.md) -for more information on building VyOS ISO images. - -## Contributing code - -:::{warning} -You must sign the {doc}`Contributor License Agreement` -for your contributions to be accepted. -::: - -VyOS is open-source and welcomes patches. -All submissions must adhere to these guidelines: - -- Each commit addresses a single issue or feature. -- Each commit message references a [Phabricator](https://vyos.dev/) task ID - (for example, `T1234`). -- Each commit is associated with a username and email address - to identify the author (see [Configure your Git identity](configure-your-git-identity)). -- Only submit bugfixes in packages other than . -- Commits follow the [coding guidelines](coding-guidelines) outlined below. - -### Determining package ownership - -To determine which VyOS package contains a file you want to modify, use Debian's -`dpkg -S` command on your running VyOS installation. - -### Submitting your code - -Fork the repository and submit a GitHub pull request. This is the preferred way -to contribute changes to VyOS. - -To fork a VyOS repository: - -1. Append `/fork` to the repository URL on GitHub. For example, to fork - `vyos-1x`, use: - -2. Clone your fork or add it as a remote to your local repository: - - - Clone: `git clone https://github.com//vyos-1x.git` - - Add remote: `git remote add myfork https://github.com//vyos-1x.git` - -(configure-your-git-identity)= - -3. Configure your Git identity: - - ```none - git config --global user.name "J. Random Hacker" - git config --global user.email "jrhacker@example.net" - ``` - -4. Make your changes and add files to the Git index: - - - Single file: `git add myfile` - - Directory: `git add somedir/*` - -5. Commit your changes with a meaningful headline and [Phabricator](https://vyos.dev/) reference: - - `git commit` - -6. Push to your fork and create a GitHub pull request: - - `git push` - -Alternatively, you can export commits as patches and send them to -[maintainers@vyos.net](mailto:maintainers@vyos.net) or attach them directly to the [Phabricator](https://vyos.dev/) task: - -- Export last commit: `git format-patch` -- Export last two commits: `git format-patch -2` - -## Commit messages - -For guidance on writing commit messages, review the file history -with `git log path/to/file.txt`. - -Every change must be associated with a task number (prefixed with **T**) and -a component. If no bug report or feature request exists for your changes, -create a [Phabricator](https://vyos.dev/) task first. Reference the task ID in your commit message: - -- `ddclient: T1030: auto create runtime directories` -- `Jenkins: add current Git commit ID to build description` - -If your pull request lacks a [Phabricator](https://vyos.dev/) reference, maintainers will request -that you amend the commit message. - -### Writing good commit messages - -Follow the format described in -the [Git documentation](https://git-scm.com/book/ch5-2.html) -and [Chris Beams' guide](https://chris.beams.io/posts/git-commit/). - -Commit message format: - -1. **Summary line** (50 characters recommended, 80 maximum): Include the - component - prefix and [Phabricator](https://vyos.dev/) reference (for example, `snmp: T1111:` or - `ethernet: T2222:`). Concatenate multiple components with colons - (for example, `snmp: ethernet: T3333`). -2. **Blank line**: Separate the summary from the body. - This blank line is critical. - -4) **Message body** with details: - - - Describe what changed, why, and how. This helps with `git bisect`. - - Wrap text at 72 characters for readability with `git log` on an 80x25 - terminal. - - Reference previous commits when applicable: - `After commit abcd12ef ("snmp: this is a headline") - a Python import statement is missing, throwing the following exception: - ABCDEF` - -5) **Cherry-pick option**: Always use the `-x` option when back-porting or - forward-porting commits: - - `git cherry-pick -x ` - - This appends `(cherry picked from commit )` to the commit message, - making bisecting easier. - -6) **Single responsibility**: Each commit must be self-contained. Do not fix - multiple bugs in a single commit. Use `git add --patch` to stage only - the parts related to one issue. - -Constraints: - -- Bugfixes are only accepted for packages other than - . - New functionality must use the new XML/Python interface, not old-style - templates (`node.def` files and Perl/Bash code). - -## Coding guidelines - -VyOS maintains consistent coding standards to help contributors navigate the -codebase and understand its logic. - -### Formatting - -- **Python**: Use 4 spaces per indentation level. Tabs **must not** be used. -- **XML**: Use 2 spaces per indentation level. Tabs **must not** be used. - -Use tools like VIM extensions (xmllint) to enforce correct indentation. Add this -to your `.vimrc` file: - -```none -au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null -``` - -Then use `gg=G` in command mode to run the linter. - -### Text generation - -Use a template processor for generating config files: - -- **Jinja2** is the default template processor for VyOS code. -- Built-in string formatting **may** be used for simple line-oriented formats - (for example, iptables rules) where every line is self-contained. -- Template processors **must** be used for structured, multi-line formats - (for example, ISC DHCPd configuration). - -### Python code - -Configuration scripts and operation mode scripts written in Python3 should -follow these guidelines: - -- Wrap lines at 80 characters. This improves readability when browsing - GitHub on mobile devices and reads well in side-by-side diffs. - -Structure your scripts with these functions: - -```python -#!/usr/bin/env python3 -# -# Copyright (C) 2020 VyOS maintainers and contributors -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 or later as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -import sys - -from vyos.config import Config -from vyos import ConfigError - -def get_config(config=None): - if config: - conf = config - else: - conf = Config() - - # Base path to CLI nodes - base = ['...', '...'] - # Convert the VyOS config to an abstract internal representation - config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) - return config_data - -def verify(config): - # Verify that configuration is valid - if invalid: - raise ConfigError("Descriptive message") - -def generate(config): - # Generate daemon configs - pass - -def apply(config): - # Apply the generated configs to the live system - pass - -try: - c = get_config() - verify(c) - generate(c) - apply(c) -except ConfigError as e: - print(e) - sys.exit(1) -``` - -`get_config()`: This function converts a VyOS config object to an abstract -internal representation. No other function may call the `vyos.config.Config` -object directly. Limiting config reads to one function makes it easier to -modify the config syntax in the future. Additionally, this design improves -testability since you can construct an internal representation by hand rather -than mocking the entire config subsystem. - -`verify()`: This function validates the internal representation. It must -raise `ConfigError` with a descriptive message if the config is invalid. It -**must not** make any changes to the system. This design enables future features -like commit dry-run ("commit test" as in JunOS) where the system can abort a -commit before making changes. - -`generate()`: This function generates config files for system components. - -`apply()`: This function applies the generated configuration to the live -system. Prefer non-disruptive reload when possible. Disruptive operations like -daemon restarts are acceptable only when: - -- The component does not support non-disruptive reload, or -- The expected service degradation is minimal (for example, auxiliary services - like LLDPd) - -For high-impact services (VPN daemons, routing protocols), make effort to -determine if changes can be applied non-disruptively before resorting to -restarts. - -Never modify active configuration directly unless absolutely necessary. Instead, -generate configuration files and apply them with a single command like service -reload through systemd. For example, save iptables rules to a file and load them -with `iptables-restore` rather than executing iptables commands one by one. - -The `apply()` and `generate()` functions may raise `ConfigError` if the -daemon fails to start with the updated config. However, this is not a substitute -for proper config validation in the `verify()` function. Make reasonable -effort to verify that generated configuration is valid and will be accepted by -the daemon, including cross-checks with other VyOS configuration subtrees when -necessary. - -Exceptions like `VyOSError` (raised by `vyos.config.Config` on improper -operations) should not be silenced or caught. While this may produce less -polished error output for users, it generates better bug reports and helps -maintainers debug issues. - -For reference implementations, see `ntp.py` or `interfaces-bonding.py` (for -tag nodes) in the [vyos-1x](https://github.com/vyos/vyos-1x) repository. - -### Other considerations: `vyos-configd` - -All scripts now run under the config daemon and must conform to these -requirements: - -1. The signature and first four lines of `get_config(...)` **must** be as - specified above. -2. Each of `get_config`, `verify`, `apply`, and `generate` **must** - appear - with the correct signatures, even if they are a no-op. -3. `Config` objects other than those in `get_config` **must not** appear. -4. The legacy function `my_set` **must not** appear. Modifications to active - config **should not** appear in new code (alternative mechanisms may be used - if absolutely necessary). - -## XML for CLI definitions - -XML interface definitions define the VyOS CLI structure. -Before VyOS `1.2` (crux), these -files were created manually. After a redesign, new-style templates are -automatically generated from XML input files. - -VyOS interface definitions come with a RelaxNG schema located in the -[vyos-1x](https://github.com/vyos/vyos-1x/tree/current/schema) -repository. This schema is a modified version from `VyConf` (VyOS `2.0`). -VyOS `1.2.x` -interface definitions are reusable in future VyOS versions with minimal changes. - -Schemas provide two benefits: - -- Complete grammar verification -- Automatic validation against the schema - -The [build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) -script converts XML definitions to -old-style templates and verifies them against the schema. A bad definition -causes the package build to fail. While the XML format is verbose, no other -format provides this level of verification. Specialized XML editors can help -manage verbosity. - -Example XML interface definition: - -```xml - - - - - - - - Task scheduler settings - - - - - Scheduled task - - <string> - Task name - - 999 - - - - - UNIX crontab time specification string - - - - - Execution interval - - <minutes> - Execution interval in minutes - - - <minutes>m - Execution interval in minutes - - - <hours>h - Execution interval in hours - - - <days>d - Execution interval in days - - - [1-9]([0-9]*)([mhd]{0,1}) - - - - - - Executable path and arguments - - - - - Path to executable - - - - - Arguments passed to the executable - - - - - - - - - - - -``` - -XML definitions are purely declarative and contain no logic. All logic for -generating config files, restarting services, and related tasks is implemented -in configuration scripts. - -### Template Processors - -XML interface definition files use the `.xml.in` file extension (implemented -in {vytask}`T1843`). These files use the GCC preprocessor to reduce code -duplication in common areas: - -- VIF (including VIF-S and VIF-C) -- Address configuration -- Description -- Enabled/Disabled state - -Instead of repeating XML nodes, use include files with predefined features: - -- [IPv4, IPv6, and DHCP(v6)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i) - address assignment. -- [IPv4 and IPv6](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i) - address assignment. -- [VLAN (VIF)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/accel-ppp/vlan.xml.i) - definition. -- [MAC address](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/firewall/mac-address.xml.i) - assignment. - -The `.in` files are preprocessed and stored in the [interface-definitions](https://github.com/vyos/vyos-1x/tree/current/interface-definitions) -folder. The [scripts/build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) -script then operates on this folder to generate all required CLI nodes. - -Example preprocessor output: - -```none -$ make interface_definitions -install -d -m 0755 build/interface-definitions -install -d -m 0755 build/op-mode-definitions -Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in -Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in -Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in -Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in -Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in -Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in -[...] -``` - - -### Command Definition Guidelines - -#### Use of Numbers - -Avoid using numbers in command names unless the number is part of a protocol -name or similar. For example, `protocols ospfv3` is appropriate, -but `server-1` is questionable. - -#### Help Strings - -Follow these guidelines for consistent, readable help strings: - -##### Capitalization and Punctuation - -- Capitalize the first word of every help string. -- Do not use a period at the end of help strings. - -This standard mirrors network device CLIs and improves aesthetics. - -Examples: - -- Good: "Frobnication algorithm" -- Bad: "frobnication algorithm" -- Bad: "Frobnication algorithm." -- Incorrect: "frobnication algorithm." - -##### Abbreviations and Acronyms - -- Capitalize all abbreviations and acronyms. - -Examples: - -- Good: "TCP connection timeout" -- Bad: "tcp connection timeout" -- Bad: "Tcp connection timeout" -- Capitalize acronyms to distinguish them from normal words. - -Examples: - -- Good: RADIUS (remote authentication for dial-in user services) -- Bad: radius (unless referring to circular distance) -- Follow accepted spelling conventions for mixed-case abbreviations. If it - contains "over" or "version", use lowercase. Follow RFC or standard spellings - when they exist. - -Examples: - -- Good: PPPoE, IPsec -- Bad: PPPOE, IPSEC -- Bad: pppoe, ipsec - -##### Verbs - -- Avoid verbs. If a verb can be omitted, omit it. - -Examples: - -- Good: "TCP connection timeout" -- Bad: "Set TCP connection timeout" -- When a verb is essential, use it. For example: "Disable IPv6 forwarding on - all interfaces" for `set system ipv6 disable-forwarding`. -- Use infinitive form for necessary verbs. - -Examples: - -- Good: "Disable IPv6 forwarding" -- Bad: "Disables IPv6 forwarding" - -## C++ Backend Code - -The VyOS CLI parser combines bash, bash-completion helpers, and the C++ backend -library [vyatta-cfg](https://github.com/vyos/vyatta-cfg). This section -references common CLI commands and their C/C++ entry points: - -`set`: - -- -- - -`commit`: - -- - - diff --git a/docs/contributing/md-index.md b/docs/contributing/md-index.md deleted file mode 100644 index f26a6b70..00000000 --- a/docs/contributing/md-index.md +++ /dev/null @@ -1,13 +0,0 @@ -# Contributing - -```{toctree} -:maxdepth: 1 - -build-vyos -development -cla -issues-features -upstream-packages -debugging -testing -``` diff --git a/docs/contributing/md-issues-features.md b/docs/contributing/md-issues-features.md deleted file mode 100644 index ab235326..00000000 --- a/docs/contributing/md-issues-features.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -lastproofread: '2025-12-08' ---- - -(issues_features)= - -# Issues/Feature requests - -(bug_report)= - -## Bug Report/Issue - -Issues and bugs occur in every software project, and VyOS is no exception. - -### I found a bug, what should I do? - -When you find a potential bug, first: - -- Consult the [documentation] to ensure you configured your system - correctly. -- Check if the VyOS community has identified a workaround for the bug through - [Slack] or the VyOS [Forum]. - -### Ensure the bug is reproducible - -Include the following information when reporting a bug: - -- A sequence of configuration commands or a complete configuration file needed - to recreate the bug. Avoid partial configurations: a sequence of commands is - easy to paste and a complete configuration is easy to load, but a partial - config is hard to reconstruct. -- Describe the expected behavior and how it differs from what you observe. - Include command outputs or traffic dumps. Explain briefly why these outputs - are incorrect and what the correct behavior should be. -- A sequence of actions that trigger the bug. While not always possible, this - helps developers and community members confirm the issue and verify fixes. -- If the bug is a regression, specify the VyOS version where the feature worked - correctly (any working version is acceptable). Identify the exact version - that the feature stopped working, if possible. - -If you are uncertain whether the behavior is a bug or what the correct behavior -is, or if you lack a reliable reproducing procedure, post on the forum or ask in -chat first. If you have a subscription, create a support ticket. The team and -community can help identify the issue, work around it, and create an actionable -bug report. - -### Report a Bug - -To open a bug report or feature request, create an account on -[vyos.dev](https://vyos.dev), the public issue tracker for VyOS. - -When creating a new issue, select the appropriate project and: - -- Provide as much information as you can. -- Specify which VyOS version you are using: `run show version`. -- Explain how to reproduce the bug. - -(feature-request)= - -## Feature Requests - -Have an idea to improve VyOS or need a feature that would benefit all users? -Before submitting a feature request, search the public issue tracker -[vyos.dev](https://vyos.dev) to check if a request already exists. You can -also enhance an existing request by providing additional information. - -Create a task before starting work on a feature, -even if it is a trivial feature. -The task tracker generates release notes, so all work must be reflected -in the tracker. - -Include at least the following information: - -- Provide a detailed description of the feature: what it is, how it works, and - how you would use it. Maintainers may not have experience with every feature, - protocol, and tool in VyOS. Detailed information helps VyOS contributors and - maintainers test new features they are unfamiliar with. -- Include proposed CLI syntax if the feature requires new commands. Provide both - configuration and operational mode commands if both are needed. - -Consider including the following information: - -- Is the feature already supported by the underlying component - (FreeRangeRouting, nftables, Kea, etc.)? -- How would you configure the feature manually within that component? -- Are there any limitations to using the feature - (hardware support, resource usage)? -- Are there any adverse or non-obvious interactions with other features? Should - the feature be mutually exclusive? -- Any relevant documentation or references about the feature. - -You do not need to provide all this information, but if you can, it simplifies -developers' work considerably. Research these questions when possible. - -## Task auto-closing - -A special task status exists for when all work by maintainers and contributors -is complete: **Needs reporter action**. - -VyOS assigns this status to: - -- Feature requests that do not include required information and need - clarification. -- Bug reports that lack reproducing procedures. -- Tasks that are implemented and tested by the implementation author, - but require testing in the real-world environment that only the reporter - can replicate (for example, hardware VyOS does not support or specific - network conditions). - -When a task is set to **Needs reporter action**: - -- If the reporter does not respond within two weeks, the task bot adds a comment - ("Any news?") to remind the reporter. -- If there is still no response after another two weeks, - the task is closed automatically. - -We do not auto-close tasks with any other status and do not close tasks due to -lack of maintainer activity. - -[documentation]: https://docs.vyos.io -[forum]: https://forum.vyos.io -[slack]: https://slack.vyos.io diff --git a/docs/contributing/md-testing.md b/docs/contributing/md-testing.md deleted file mode 100644 index b68cad1b..00000000 --- a/docs/contributing/md-testing.md +++ /dev/null @@ -1,207 +0,0 @@ ---- -lastproofread: '2025-12-02' ---- - -(testing)= - -# Testing - -One of the major features introduced in VyOS 1.3 is an automated test -framework. When you assemble an ISO image, several things can go wrong. -VyOS uses this framework to detect issues before they cause downstream problems. - -This section describes how the automated testing process at VyOS works. - -## Smoketests - -Smoketests execute predefined VyOS CLI commands and check if the desired -daemon or service configuration is rendered. - -When an ISO image is assembled by the [VyOS CI](https://ci.vyos.net), the `BUILD_SMOKETEST` -parameter is enabled by default. This extends the ISO configuration line -with the following packages: - -```python -def CUSTOM_PACKAGES = '' - if (params.BUILD_SMOKETESTS) - CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' -``` - -If you plan to build your own custom ISO image and want to use VyOS's -smoketests, ensure that you have the `vyos-1x-smoketest` package installed. - -The `make test` command from the [vyos-build](https://github.com/vyos/vyos-build) repository launches a new -QEMU instance, and the ISO image is first installed to the virtual hard disk. - -After the first boot into the newly installed system, the main Smoketest script -is executed. It can be found at `/usr/bin/vyos-smoketest`. - -The script searches for executable test cases under -`/usr/libexec/vyos/tests/smoke/cli/` and executes them one by one. - -:::{note} -Smoketests will alter the system configuration. If you are logged -in remotely, you may lose your connection to the system. -::: - -:::{note} -To enable smoketest debugging (print the CLI set commands used), -run: `touch /tmp/vyos.smoketest.debug`. -::: - -### Manual Smoketest Run - -Each test is contained in its own file, so you can execute a single Smoketest -manually by running the Python test script. - -Example: - -```none -vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py -test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok -test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok -test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok -test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok -test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok -test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok -test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok -test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok -test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok -test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok -test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok -test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok -test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok - ----------------------------------------------------------------------- -Ran 13 tests in 348.191s - -OK -``` - - -### Interface-based tests - -Our smoketests not only test daemons and services, but also check if interface -configuration works as expected. There is a common base class named -`base_interfaces_test.py` that holds all the common code for interface tests. - -These common tests consist of: - -- Add one or more IP addresses - -- DHCP client and DHCPv6 prefix delegation - -- MTU size - -- IP and IPv6 options - -- Port description - -- Port disable - -- VLANs (QinQ and regular 802.1q) - -- ... - -:::{note} -When you are working on interface configuration and want to test -if the Smoketests pass, you would normally lose the remote SSH connection -to your {abbr}`DUT (Device Under Test)`. To handle this, some interface-based -tests can be called with an environment variable beforehand to limit the -number of interfaces used in the test. By default, all interfaces (e.g., all -Ethernet interfaces) are used. -::: - -```none -vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py -test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok -test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok -test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok -test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok -test_bonding_min_links (__main__.BondingInterfaceTest) ... ok -test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok -test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok -test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok -test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok -test_interface_description (__main__.BondingInterfaceTest) ... ok -test_interface_disable (__main__.BondingInterfaceTest) ... ok -test_interface_ip_options (__main__.BondingInterfaceTest) ... ok -test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok -test_interface_mtu (__main__.BondingInterfaceTest) ... ok -test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok -test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok -test_span_mirror (__main__.BondingInterfaceTest) ... ok -test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok -test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok -test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok -test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok -test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok -test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok - ----------------------------------------------------------------------- -Ran 23 tests in 244.694s - -OK -``` - -This will limit the `bond` interface test to use only `eth1` and `eth2` -as member ports. - -## Config Load Tests - -The other part of our tests are called "config load tests." Config load tests -sequentially load arbitrary configuration files to verify that configuration -migration scripts work as designed and that a given set of functionality can -still be loaded with a fresh VyOS ISO image. - -The configurations are all derived from production systems and can act as -test cases or as references for enabling certain features. The configurations -can be found here: - - -The entire test is controlled by the main wrapper script -`/usr/bin/vyos-configtest`. -It behaves in the same way as the main smoketest script. It scans the folder -for potential configuration files and issues a `load` command for each file. - -### Manual config load test - -You do not have to load all configurations sequentially; you can also load -individual test configurations manually. - -```none -vyos@vyos:~$ configure -load[edit] - -vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small -Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' -Load complete. Use 'commit' to make changes effective. -[edit] -vyos@vyos# compare -[edit interfaces ethernet eth0] --hw-id 00:50:56:bf:c5:6d -[edit interfaces ethernet eth1] -+duplex auto --hw-id 00:50:56:b3:38:c5 -+speed auto -[edit interfaces] --ethernet eth2 { -- hw-id 00:50:56:b3:9c:1d --} --vti vti1 { -- address 192.0.2.1/30 --} -... - -vyos@vyos# commit -vyos@vyos# -``` - -:::{note} -Some configurations have preconditions that must be met. These most -likely include generation of cryptographic keys before the config can be -applied; otherwise, you will get a commit error. If you are interested in -how those preconditions are fulfilled, check the [vyos-build](https://github.com/vyos/vyos-build) repository and -the `scripts/check-qemu-install` file. -::: - diff --git a/docs/contributing/md-upstream-packages.md b/docs/contributing/md-upstream-packages.md deleted file mode 100644 index fe01df82..00000000 --- a/docs/contributing/md-upstream-packages.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -lastproofread: '2026-01-30' ---- - -(upstream-packages)= - -# Upstream Packages - -Many base system packages are pulled straight from Debian's `main` and -`contrib` repositories, but there are exceptions. If you only want to build -a fresh ISO image, you can skip -this section. This information may be useful for a deeper dive into VyOS. - -System packages that are not directly pulled from Debian are built through a -separate build system, `build.py` in the [vyos-build](https://github.com/vyos/vyos-build/tree/current/scripts/package-build) repository. - -## Overview - -Previously, VyOS used Jenkins for building upstream packages. With the move away -from Jenkins, the build system was replaced with a Python-based solution using -`build.py` and `package.toml` configuration files. - -Each package directory contains: - -- A `package.toml` configuration file that defines how the package is built. -- A symlink to the common `build.py` script in the build system. - -## Building Packages - -To build a package, navigate to the package directory and execute the -build script: - -```console -cd package-build/ -./build.py -``` - -The script will: - -1. Check out the source code from the configured repository. -2. Apply any patches defined in the configuration. -3. Execute pre-build hooks (if configured). -4. Build the package using the specified build command. -5. Generate both binary (`.deb`) packages and source tarballs. - -## Package Configuration (package.toml) - -Each package directory contains a `package.toml` file that defines the build -parameters. The key configuration fields are: - -```{eval-rst} -**name** - The package name (e.g., ``frr``) - -**commit_id** - The specific commit, tag, or branch to check out from the source repository - (e.g., ``stable/10.5``) - -**scm_url** - The Git URL of the upstream source repository - (e.g., ``https://github.com/FRRouting/frr.git``) - -.. stop_vyoslinter -**build_cmd** - The command to execute for building the package. This replaces what was - previously defined in the Jenkins ``Jenkinsfile``. - - Default if not specified: ``dpkg-buildpackage -uc -us -tc -F --source-option=--tar-ignore=.git --source-option=--tar-ignore=.github`` - - Example with custom build command: - - .. code-block:: toml - - build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" -.. start_vyoslinter - -**pre_build_hook** (Optional) - A shell command or script that executes after the repository is checked out - and before the build process begins. This allows you to perform preparatory - tasks such as: - - - Creating directories - - Copying files - - Running custom setup scripts - - Installing dependencies - - Single command example: - - .. code-block:: toml - - pre_build_hook = "echo 'Preparing build environment'" - - Multi-line commands example: - - .. code-block:: toml - - pre_build_hook = """ - mkdir -p ../hello/vyos - mkdir -p ../vyos - cp example.txt ../vyos - """ - - Combined commands and scripts: - - .. code-block:: toml - - pre_build_hook = "ls -l; ./script.sh" - -**apply_patches** (Optional) - Boolean flag to control whether patches should be applied. Defaults to - ``True``. - - .. code-block:: toml - - apply_patches = false - -**prepare_package** (Optional) - Boolean flag to enable package preparation. When set to ``True``, the - ``install_data`` configuration is used. - -**install_data** (Optional) - Data used for package preparation when ``prepare_package`` is enabled. -``` - -## Example package.toml file - -Here's an example configuration for the FRRouting (FRR) package: - -```toml -name = "frr" -commit_id = "stable/10.5" -scm_url = "https://github.com/FRRouting/frr.git" -build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" -``` - - -## Build Output - -After running `./build.py`, the following artifacts are generated in the -package directory: - -- `.deb` files - Binary Debian packages ready for installation -- `.tar.gz` files - Source tarballs of the checked-out repositories -- Additional build artifacts as produced by the Debian build system - -The build script also creates build dependency packages (`*build-deps*.deb`), -which are automatically cleaned up after the build completes. diff --git a/docs/contributing/rst-build-vyos.rst b/docs/contributing/rst-build-vyos.rst new file mode 100644 index 00000000..65298a7a --- /dev/null +++ b/docs/contributing/rst-build-vyos.rst @@ -0,0 +1,759 @@ +:lastproofread: 2025-12-05 + +.. _build: + +########## +Build VyOS +########## + +************* +Prerequisites +************* + +There are different ways you can build VyOS. Building using a +:ref:`build_docker` +container is the easiest way because all dependencies are managed for you. +Alternatively, you can set up your own build machine and run a +:ref:`build_native` build. + +.. note:: Starting with VyOS 1.4, only source code and Debian package + repositories of the rolling release (the **current** branch) are publicly + available. + + The source code and pre-built Debian package repositories of LTS releases + are only available to subscription holders (customers and active community + members with contributors subscriptions). + + The following includes the build process for VyOS rolling release. + +This will guide you through the process of building a VyOS ISO using Docker_. +This process has been tested on clean installs of Debian Bookworm. + +.. _build_native: + +Native Build +============ + +To build VyOS natively, you need a properly configured build host with +Debian Bookworm installed. + +To get started, clone the repository to your local machine: + +.. code-block:: none + + $ sudo make clean + $ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic + +For required packages, refer to the ``docker/Dockerfile`` file in the +repository_. The ``./build-vyos-image`` script will also warn you if any +dependencies are missing. + +.. _build_docker: + +Docker +====== + +Installing Docker_ and prerequisites: + +.. hint:: Docker versions are updated frequently. The following examples may + become outdated. + +.. code-block:: none + + # Add Docker's official GPG key: + sudo apt-get update + sudo apt-get install ca-certificates curl gnupg + sudo install -m 0755 -d /etc/apt/keyrings + curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg + sudo chmod a+r /etc/apt/keyrings/docker.gpg + + # Add the repository to Apt sources: + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + + sudo apt-get update + sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin + +To use Docker without ``sudo``, add your current non-root user to the ``docker`` +group: ``sudo usermod -aG docker yourusername``. + +.. hint:: Adding a user to the ``docker`` group grants privileges equivalent to + ``root``. It is recommended to remove the non-root user from the ``docker`` + group after building the VyOS ISO. See also `Docker as non-root`_. + +.. note:: The build process must run on a local file system. Building on SMB or + NFS shares will cause the container to fail. VirtualBox shared folders are + also not supported because block device operations are not implemented. + +Build Container +--------------- + +The container can be built by hand or by fetching the pre-built one from +DockerHub. It is recommended to use the pre-built containers from the +`VyOS DockerHub`organization_ +The container is built from Docker packages automatically after every commit +to the ``vyos-build`` repository (this process may take 2-3 hours). + +.. note:: If you use the pre-built container, it will be automatically + downloaded from DockerHub if it is not found on your local machine when + you build the ISO. + +Dockerhub +^^^^^^^^^ + +To manually download the container from DockerHub, run: + +.. code-block:: none + + $ docker pull vyos/vyos-build:current # For VyOS rolling release + +Build from source +^^^^^^^^^^^^^^^^^ + +The container can also be built directly from source: + +.. code-block:: none + + $ git clone -b current --single-branch https://github.com/vyos/vyos-build + + $ cd vyos-build + $ docker build -t vyos/vyos-build:current docker + +.. note:: VyOS switched to Debian Bookworm (12) in its ``current`` branch. + Due to software version updates, it is recommended to use the official + Docker Hub image to build VyOS ISO. + +Tips and Tricks +--------------- + +You can create Bash aliases to easily launch the latest container per release +train (``current``). Add the following to your ``.bash_aliases`` file: + +.. code-block:: none + + alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "$HOME/.gitconfig":/etc/gitconfig \ + -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ + -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + vyos/vyos-build:current bash' + +Now you have a new alias ``vybld`` that launches development containers in +your current working directory. + +.. note:: Some VyOS packages (namely vyos-1x) come with build-time tests which + verify some of the internal library calls that they work as expected. Those + tests are carried out through the Python Unittest module. If you want to + build the ``vyos-1x`` package (which is our main development package) you + need to start your Docker container using the following argument: + ``--sysctl net.ipv6.conf.lo.disable_ipv6=0``, otherwise those tests will + fail. + + +.. _build_iso: + +********* +Build ISO +********* + +Now that you understand the prerequisites, you can build a VyOS ISO from source. +First, fetch the latest source code from GitHub: + +.. code-block:: none + + $ git clone -b current --single-branch https://github.com/vyos/vyos-build + + +Now you can begin a fresh VyOS ISO build. Change to the ``vyos-build`` +directory and run: + +.. code-block:: none + + $ cd vyos-build + $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash + +Start the build: + +.. code-block:: none + + vyos_bld@8153428c7e1f:/vyos$ sudo make clean + vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --architecture amd64 --build-by "j.randomhacker@vyos.io" generic + +When the build is successful, find the resulting ISO in the ``build`` directory +as ``live-image-[architecture].hybrid.iso``. + +.. _build source: + + +.. _customize: + +Customize +========= + +You can customize the ISO with the following configure options. Generate the +full and current list with ``./build-vyos-image --help``: + +.. code-block:: none + + $ vyos_bld@8153428c7e1f:/vyos$ sudo ./build-vyos-image --help + I: Checking if packages required for VyOS image build are installed + usage: build-vyos-image [-h] [--architecture ARCHITECTURE] + [--build-by BUILD_BY] [--debian-mirror DEBIAN_MIRROR] + [--debian-security-mirror DEBIAN_SECURITY_MIRROR] + [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] + [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] + [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] [--dry-run] + [--custom-apt-entry CUSTOM_APT_ENTRY] [--custom-apt-key CUSTOM_APT_KEY] + [--custom-package CUSTOM_PACKAGE] + [build_flavor] + + positional arguments: + build_flavor Build flavor + + optional arguments: + -h, --help show this help message and exit + --architecture ARCHITECTURE + Image target architecture (amd64 or arm64) + --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) + --debian-mirror DEBIAN_MIRROR + Debian repository mirror + --debian-security-mirror DEBIAN_SECURITY_MIRROR + Debian security updates mirror + --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR + Debian repository mirror for pbuilder env bootstrap + --vyos-mirror VYOS_MIRROR + VyOS package mirror + --build-type BUILD_TYPE + Build type, release or development + --version VERSION Version number (release builds only) + --build-comment BUILD_COMMENT + Optional build comment + --debug Enable debug output + --dry-run Check build configuration and exit + --custom-apt-entry CUSTOM_APT_ENTRY + Custom APT entry + --custom-apt-key CUSTOM_APT_KEY + Custom APT key file + --custom-package CUSTOM_PACKAGE + Custom package to install from repositories + + +.. _iso_build_issues: + +ISO Build Issues +---------------- + +There are (rare) situations where building an ISO image is not possible at all +due to a broken package feed in the background. APT is not very good at +reporting the root cause of the issue. Your ISO build will likely fail with a +more or less similar looking error message: + +.. code-block:: none + + The following packages have unmet dependencies: + vyos-1x : Depends: accel-ppp but it is not installable + E: Unable to correct problems, you have held broken packages. + P: Begin unmounting filesystems... + P: Saving caches... + Reading package lists... + Building dependency tree... + Reading state information... + Del frr-pythontools 7.5-20210215-00-g8a5d3b7cd-0 [38.9 kB] + Del accel-ppp 1.12.0-95-g59f8e1b [475 kB] + Del frr 7.5-20210215-00-g8a5d3b7cd-0 [2671 kB] + Del frr-snmp 7.5-20210215-00-g8a5d3b7cd-0 [55.1 kB] + Del frr-rpki-rtrlib 7.5-20210215-00-g8a5d3b7cd-0 [37.3 kB] + make: *** [Makefile:30: iso] Error 1 + (10:13) vyos_bld ece068908a5b:/vyos [current] # + +To debug the build process and gain additional information of what could be the +root cause, you need to use `chroot` to change into the build directory. This is +explained in the following step by step procedure: + +.. code-block:: none + + vyos_bld ece068908a5b:/vyos [current] # sudo chroot build/chroot /bin/bash + +We now need to mount some required, volatile filesystems + +.. code-block:: none + + (live)root@ece068908a5b:/# mount -t proc none /proc + (live)root@ece068908a5b:/# mount -t sysfs none /sys + (live)root@ece068908a5b:/# mount -t devtmpfs none /dev + +We now are free to run any command we would like to use for debugging, e.g. +re-installing the failed package after updating the repository. + +.. code-block:: none + + (live)root@ece068908a5b:/# apt-get update; apt-get install vyos-1x + Get:1 file:/root/packages ./ InRelease + Ign:1 file:/root/packages ./ InRelease + Get:2 file:/root/packages ./ Release [1235 B] + Get:2 file:/root/packages ./ Release [1235 B] + Get:3 file:/root/packages ./ Release.gpg + Ign:3 file:/root/packages ./ Release.gpg + Hit:4 http://repo.powerdns.com/debian buster-rec-43 InRelease + Hit:5 http://repo.saltstack.com/py3/debian/10/amd64/archive/3002.2 buster InRelease + Hit:6 http://deb.debian.org/debian bullseye InRelease + Hit:7 http://deb.debian.org/debian buster InRelease + Hit:8 http://deb.debian.org/debian-security buster/updates InRelease + Hit:9 http://deb.debian.org/debian buster-updates InRelease + Hit:10 http://deb.debian.org/debian buster-backports InRelease + Hit:11 http://dev.packages.vyos.net/repositories/current current InRelease + Reading package lists... Done + N: Download is performed unsandboxed as root as file '/root/packages/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) + Reading package lists... Done + Building dependency tree + Reading state information... Done + Some packages could not be installed. This may mean that you have + requested an impossible situation or if you are using the unstable + distribution that some required packages have not yet been created + or been moved out of Incoming. + The following information may help to resolve the situation: + + The following packages have unmet dependencies: + vyos-1x : Depends: accel-ppp but it is not installable + E: Unable to correct problems, you have held broken packages. + +Now it's time to fix the package mirror and rerun the last step until the +package installation succeeds again! + +.. _build_custom_packages: + +Linux Kernel +============ + +The Linux kernel used by VyOS is heavily tied to the ISO build process. The +file ``data/defaults.json`` hosts a JSON definition of the kernel version used +``kernel_version`` and the ``kernel_flavor`` of the kernel which represents the +kernel's LOCAL_VERSION. Both together form the kernel version variable in the +system: + +.. code-block:: none + + vyos@vyos:~$ uname -r + 6.1.52-amd64-vyos + +* Accel-PPP +* Intel NIC drivers +* Intel QAT + +Each of those modules holds a dependency on the kernel version and if you are +lucky enough to receive an ISO build error which sounds like: + +.. code-block:: none + + I: Create initramfs if it does not exist. + Extra argument '6.1.52-amd64-vyos' + Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] + Options: + -k version Specify kernel version or 'all' + -c Create a new initramfs + -u Update an existing initramfs + -d Remove an existing initramfs + -b directory Set alternate boot directory + -v Be verbose + See update-initramfs(8) for further details. + E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. + +The most obvious reasons could be: + +* ``vyos-build`` repo is outdated, please ``git pull`` to update to the latest + release kernel version from us. + +* You have your own custom kernel `*.deb` packages in the `packages` folder but + neglected to create all required out-of tree modules like Accel-PPP, Intel + QAT or Intel NIC drivers + +Building The Kernel +------------------- + +The kernel build is quite easy, most of the required steps can be found in the +``vyos-build/packages/linux-kernel/Jenkinsfile`` but we will walk you through +it. + +Clone the kernel source to `vyos-build/packages/linux-kernel/`: + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel/ + $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + +Check out the required kernel version - see ``vyos-build/data/defaults.json`` +file (example uses kernel 4.19.146): + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel/linux + $ git checkout v4.19.146 + Checking out files: 100% (61536/61536), done. + Note: checking out 'v4.19.146'. + + You are in 'detached HEAD' state. You can look around, make experimental + changes and commit them, and you can discard any commits you make in this + state without impacting any branches by performing another checkout. + + If you want to create a new branch to retain commits you create, you may + do so (now or later) by using -b with the checkout command again. Example: + + git checkout -b + + HEAD is now at 015e94d0e37b Linux 4.19.146 + +Now you can use the helper script ``build-kernel.sh``, which completes all +the necessary steps: applying required patches from the +``vyos-build/packages/linux-kernel/patches`` folder, copying the kernel +configuration ``x86_64_vyos_defconfig`` to the correct location, and building +the Debian packages. + +.. note:: Building the kernel will take some time depending on the speed and + quantity of your CPU/cores and disk speed. Expect 20 minutes + (or even longer) on lower end hardware. + +.. code-block:: none + + (18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh + I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source + I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch + patching file Documentation/networking/ip-sysctl.txt + patching file include/linux/inetdevice.h + patching file include/linux/ipv6.h + patching file include/uapi/linux/ip.h + patching file include/uapi/linux/ipv6.h + patching file net/ipv4/devinet.c + Hunk #1 succeeded at 2319 (offset 1 line). + patching file net/ipv6/addrconf.c + patching file net/ipv6/route.c + I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch + patching file fs/notify/inotify/Kconfig + patching file fs/notify/inotify/inotify_user.c + patching file fs/overlayfs/super.c + Hunk #2 succeeded at 1713 (offset 9 lines). + Hunk #3 succeeded at 1739 (offset 9 lines). + Hunk #4 succeeded at 1762 (offset 9 lines). + patching file include/linux/inotify.h + I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch + patching file scripts/package/builddeb + I: make x86_64_vyos_defconfig + HOSTCC scripts/basic/fixdep + HOSTCC scripts/kconfig/conf.o + YACC scripts/kconfig/zconf.tab.c + LEX scripts/kconfig/zconf.lex.c + HOSTCC scripts/kconfig/zconf.tab.o + HOSTLD scripts/kconfig/conf + # + # configuration written to .config + # + I: Generate environment file containing Kernel variable + I: Build Debian Kernel package + UPD include/config/kernel.release + /bin/sh ./scripts/package/mkdebian + dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc + dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos + dpkg-buildpackage: info: source version 4.19.146-1 + dpkg-buildpackage: info: source distribution buster + dpkg-buildpackage: info: source changed by vyos_bld + dpkg-buildpackage: info: host architecture amd64 + dpkg-buildpackage: warning: debian/rules is not executable; fixing that + dpkg-source --before-build . + debian/rules build + make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= + SYSTBL arch/x86/include/generated/asm/syscalls_32.h + + ... + + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) + dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) + dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. + dpkg-genbuildinfo --build=binary + dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes + dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list + dpkg-genchanges: info: binary-only upload (no source code included) + dpkg-source --after-build . + dpkg-buildpackage: info: binary-only upload (no source included) + + +When complete, you will have kernel binary packages to use in your custom ISO +build. Place all ``*.deb`` files in the ``vyos-build/packages`` folder, where +the build process will use them automatically. + +Firmware +^^^^^^^^ + +If you upgrade your kernel or include new drivers you may need new firmware. +This builds a new ``vyos-linux-firmware`` package using the included helper +scripts. + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel + $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git + $ ./build-linux-firmware.sh + $ cp vyos-linux-firmware_*.deb ../ + +The script automatically detects which firmware blobs are needed based on the +built drivers. If detection fails, you can manually add files to +``vyos-build/packages/linux-kernel/build-linux-firmware.sh``: + +.. code-block:: bash + + ADD_FW_FILES="iwlwifi* ath11k/QCA6390/*/*.bin" + + +Building Out-Of-Tree Modules +---------------------------- + +Building the kernel is one step. You must also build required out-of-tree +modules so the ABIs match. +Refer to ``vyos-build/packages/linux-kernel/Jenkinsfile`` +for all required modules and their versions. We show you how to build the +currently required modules. + +Accel-PPP +^^^^^^^^^ + +First, clone the source code and check out the appropriate version: + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel + $ git clone https://github.com/accel-ppp/accel-ppp.git + +Use the helper script and patches to build the package. Run the following +command: + +.. code-block:: none + + $ ./build-accel-ppp.sh + I: Build Accel-PPP Debian package + CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): + The OLD behavior for policy CMP0003 will be removed from a future version + of CMake. + + The cmake-policies(7) manual explains that the OLD behaviors of all + policies are deprecated and that a policy should be set to OLD only under + specific short-term circumstances. Projects should be ported to the NEW + behavior and not rely on setting a policy to OLD. + + -- The C compiler identification is GNU 8.3.0 + + ... + + CPack: Create package using DEB + CPack: Install projects + CPack: - Run preinstall target for: accel-ppp + CPack: - Install project: accel-ppp + CPack: Create package + CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them +to the ``vyos-build/packages`` folder for inclusion during the ISO build. + +Intel NIC +^^^^^^^^^ + +The Intel NIC drivers do not come from a Git repository. VyOS fetches the +tarballs from a mirror and compiles them. Use the following wrapper script +to build all driver modules: + +.. code-block:: none + + ./build-intel-drivers.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k + I: Compile Kernel module for Intel ixgbe driver + + ... + + I: Building Debian package vyos-intel-iavf + Doing `require 'backports'` is deprecated and will not load any backport in the next major release. + Require just the needed backports instead, or 'backports/latest'. + Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} + Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} + I: Cleanup iavf source + +After compilation, find the generated ``*.deb`` binaries in +``vyos-build/packages/linux-kernel``. Copy them to the ``vyos-build/packages`` +folder for inclusion in the ISO build. + +Intel QAT +^^^^^^^^^ + +The Intel QAT (Quick Assist Technology) drivers do not come from a Git +repository. VyOS fetches the tarballs from ``01.org``, Intel's open-source +website. +Use the following wrapper script to build all driver modules: + +.. code-block:: none + + $ ./build-intel-qat.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k + I: Compile Kernel module for Intel qat driver + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + checking for a thread-safe mkdir -p... /bin/mkdir -p + checking for gawk... gawk + checking whether make sets $(MAKE)... yes + + ... + + I: Building Debian package vyos-intel-qat + Doing `require 'backports'` is deprecated and will not load any backport in the next major release. + Require just the needed backports instead, or 'backports/latest'. + Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} + Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} + I: Cleanup qat source + + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them +to the ``vyos-build/packages`` folder for inclusion during the ISO build. + +Packages +======== + +If you are brave enough to build your own ISO image with any modified package +from VyOS's GitHub organisation, this is the place for you. + +Any modified package may be an altered version (e.g., ``vyos-1x``) that you +want to test before filing a pull request on GitHub. + +Building an ISO with a customized package is the same as building a regular +ISO image. Place your modified ``*.deb`` package inside the ``packages`` folder +within ``vyos-build``. The build process will automatically use your custom +package during the ISO build. + +Troubleshooting +=============== + +Debian APT does not provide verbose error messages. If your ISO build fails and +you suspect an APT dependencies or installation issue, you can apply this patch +to increase APT verbosity during the ISO build. + +.. stop_vyoslinter + +.. code-block:: diff + + diff --git i/scripts/live-build-config w/scripts/live-build-config + index 1b3b454..3696e4e 100755 + --- i/scripts/live-build-config + +++ w/scripts/live-build-config + @@ -57,7 +57,8 @@ lb config noauto \ + --firmware-binary false \ + --updates true \ + --security true \ + - --apt-options "--yes -oAcquire::Check-Valid-Until=false" \ + + --apt-options "--yes -oAcquire::Check-Valid-Until=false -oDebug::BuildDeps=true -oDebug::pkgDepCache::AutoInstall=true \ + + -oDebug::pkgDepCache::Marker=true -oDebug::pkgProblemResolver=true -oDebug::Acquire::gpgv=true" \ + --apt-indices false + "${@}" + """ + +.. start_vyoslinter + +.. _build_packages: + +******** +Packages +******** + +VyOS comes with specific packages that cannot be found in any +Debian mirror. These packages are located in the `VyOS GitHub project`_ in +source format and can easily be compiled into custom +Debian (``*.deb``) packages. + +The easiest way to compile your package is with the :ref:`build_docker` +container mentioned earlier, as it includes all required dependencies for all +VyOS related packages. + +Assuming you want to build the ``vyos-1x`` package and modify it for your needs, +first clone the repository from GitHub: + +.. code-block:: none + + $ git clone --recurse-submodules https://github.com/vyos/vyos-1x + +Build +===== + +Launch the Docker container and build the package: + +.. code-block:: none + + # For VyOS 1.3 (equuleus, current) + $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash + + # Change to source directory + $ cd vyos-1x + + # Build DEB + $ dpkg-buildpackage -uc -us -tc -b + +After a minute or two, the generated DEB packages are located next to the +``vyos-1x`` source directory: + +.. code-block:: none + + # ls -al ../vyos-1x*.deb + -rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb + -rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb + +Install +======= + +To test your newly created package, you can SCP it to a running VyOS instance +and install the new ``*.deb`` package to replace the current one. + +Install the package using the following commands: + +.. code-block:: none + + vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb + (Reading database ... 58209 files and directories currently installed.) + Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... + Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... + Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... + Processing triggers for rsyslog (8.1901.0-1) ... + +You can also place the generated ``*.deb`` in your ISO build environment to +include it in a custom ISO. See :ref:`build_custom_packages` for more +information. + +.. warning:: Any packages in the ``packages`` directory will be added to the + ISO during the build, replacing upstream packages. Delete both the source + directories and built DEB packages if you want to build an ISO from purely + upstream packages. + + +.. stop_vyoslinter + +.. _Docker: https://docs.docker.com/engine/install/debian/ +.. _`Docker as non-root`: https://docs.docker.com/engine/install/linux-postinstall +.. _VyOS DockerHub organisation: https://hub.docker.com/u/vyos +.. _repository: https://github.com/vyos/vyos-build +.. _VyOS GitHub project: https://github.com/vyos + +.. start_vyoslinter + diff --git a/docs/contributing/rst-cla.rst b/docs/contributing/rst-cla.rst new file mode 100644 index 00000000..c19e266a --- /dev/null +++ b/docs/contributing/rst-cla.rst @@ -0,0 +1,45 @@ +:lastproofread: 2025-12-05 + +.. _cla: + +############################# +Contributor License Agreement +############################# + +Before we can accept your contributions to VyOS, you must sign a **Contributor +License Agreement (CLA)**. + +This is a standard open-source practice that protects both you and the project. + +The process is straightforward and fully automated: + +1. **Review the CLA document** + + Find the CLA text in our + `GitHub repository `__. + +2. **Submit a pull request** + + When you open a pull request, a CLA bot automatically checks whether all + commit authors have signed the CLA. + +3. **Follow the bot's instructions** + + If the CLA has not been signed, the bot leaves a comment with instructions. + Reply to that comment with the suggested text to sign the CLA. + +4. **Wait for confirmation** + + The CLA bot verifies your response and updates the pull request status. + Once all commit authors have signed, the bot confirms that the CLA + requirement is met and unlocks the pull request for merging. + +.. note:: + + Each commit author must sign the CLA. + + If your pull request includes commits from multiple contributors, each one + must sign the CLA before the pull request can be accepted. + +Once you sign the CLA, it remains valid for all your past and future +contributions to VyOS under the same GitHub identity. diff --git a/docs/contributing/rst-debugging.rst b/docs/contributing/rst-debugging.rst new file mode 100644 index 00000000..da8186ae --- /dev/null +++ b/docs/contributing/rst-debugging.rst @@ -0,0 +1,223 @@ +:lastproofread: 2025-12-05 + +.. _debugging: + +######### +Debugging +######### + +Two flags are available to help debug configuration scripts. Configuration +loading issues manifest during boot, so these flags are passed as kernel boot +parameters. + +ISO image build +=============== + +If you have trouble compiling your own ISO image or debugging Jenkins issues, +follow the steps at :ref:`iso_build_issues`. + +System Startup +============== + +Debug system startup by examining the configuration file loading from +``/config/config.boot``. Extend the kernel command-line in the bootloader to +enable this. + +Kernel +------ + +* ``vyos-debug`` - Add this parameter to the Linux boot line to produce + timing results for script execution during commit. If you see an unexpected + delay during manual or boot commit, this parameter helps identify bottlenecks. + The internal flag is ``VYOS_DEBUG``, found in vyatta-cfg_. Output is directed + to ``/var/log/vyatta/cfg-stdout.log``. + +* ``vyos-config-debug`` - During development, coding errors can cause commit + failures on boot, potentially preventing CLI initialization. This kernel boot + parameter ensures access to the system as user ``vyos`` and logs a Python + stack trace to ``/tmp/boot-config-trace``. The file is created only if the + configuration load fails. + +Live System +=========== + +Several flags can be set to change VyOS behavior at runtime. Toggle these flags +using environment variables or by creating files. + +For each feature, create a file called ``vyos.feature.debug`` to enable it. +If a parameter is required, place it as the first line inside the file. + +Place the file in ``/tmp`` for one-time debugging (the file is removed on +reboot) or in ``/config`` to persist permanently. + +For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable +interface debugging. + +You can also enable debugging using environment variables. +The environment variable name follows the convention ``VYOS_FEATURE_DEBUG``. + +For example, ``export VYOS_IFCONFIG_DEBUG=""`` in your vbash has the same effect +as ``touch /tmp/vyos.ifconfig.debug``. + +* ``ifconfig`` - Display all commands and their responses from the OS on + screen for inspection. + +* ``command`` - Display all commands and their responses from the OS on screen + for inspection. + +* ``developer`` - When a command fails, start a PDB post-mortem session instead + of showing a standard error message. This allows developers to debug issues + interactively. Because the debugger waits for input, it can prevent the router + from booting, so only enable this permanently on production systems if you are + ready for potential boot failures. + +* ``log`` - Send all commands used by VyOS to a log file for inspection. This + is useful in rare cases when you need to see what the OS is doing, including + during boot. The default file is ``/tmp/full-log``, but you can change it. + +.. note:: To retrieve debug output on the command line, disable ``vyos-configd`` + in addition. You can do this one-time with + ``sudo systemctl stop vyos-configd`` + or permanently with ``sudo systemctl disable vyos-configd``. + +FRR +--- + +Recent versions use the ``vyos.frr`` framework. The Python class is located in +``vyos-1x:python/vyos/frr.py``. It includes an embedded debugger similar to the +one in ``vyos.ifconfig``. + +Enable debugging by running: ``touch /tmp/vyos.frr.debug`` + +Debug Python code with PDB +------------------------------ + +Sometimes it is useful to debug Python code interactively on the live system +rather than in an IDE. You can do this using pdb. + +Assuming you want to debug a Python script called by an op-mode command, find +the script by looking up the op-mode definitions, then edit it on the live +system using vi: +``vi /usr/libexec/vyos/op_mode/show_xyz.py`` + +Insert the following statement right before the section where you want to +investigate a problem (for example, a statement you see in a backtrace): +``import pdb; pdb.set_trace()`` + +Optionally, surround this statement with an ``if`` condition that triggers only +for the conditions you are interested in. + +When you run ``show xyz`` and your condition triggers, you enter the Python +debugger: + + +.. code-block:: none + + > /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() + -> rule_type = rule.get('type', '') + (Pdb) + +You can type ``help`` to get an overview of the available commands, and +``help command`` to get more information on each command. + +Common useful commands include: + +* examine variables using ``pp(var)`` +* continue execution using ``cont`` +* get a backtrace using ``bt`` + +Config Migration Scripts +------------------------ + +Starting with VyOS 1.5, a new mechanism is used for config migration that +improves migration performance. New migrators use only the new format with a +``migration()`` function. + +.. code-block:: python + + from vyos.configtree import ConfigTree + base = ['vpn', 'ipsec'] + def migrate(config: ConfigTree) -> None: + if not config.exists(base): + # Nothing to do + return + # do your stuff here + +New-style migration scripts can no longer run on their own. However, the new +migration subsystem handler includes a test kit: + +.. code-block:: none + + vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --help + usage: run-config-migration.py [-h] [--test-script TEST_SCRIPT] [--output-file OUTPUT_FILE] [--force] config_file + + positional arguments: + config_file configuration file to migrate + + options: + -h, --help show this help message and exit + --test-script TEST_SCRIPT + test named script + --output-file OUTPUT_FILE + write to named output file instead of config file + --force force run of all migration scripts + + +To test your migration, run: + +.. code-block:: none + + vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --test-script /opt/vyatta/etc/config-migrate/migrate/quagga/11-to-12 --output-file /tmp/foo /tmp/static-route-basic + vyos@vyos:~$ cat /tmp/foo + +The file ``/tmp/foo`` contains the migrated configuration. + +Configuration Error on System Boot +---------------------------------- + +Running the latest rolling releases sometimes exposes bugs due to edge cases +missed in design. File these bugs via Phabricator_, but you can help narrow +down the issue by following these steps: + +1. Log in to your VyOS system. +2. Enter configuration mode: ``configure`` +3. Reload your boot configuration: ``load`` + +You should see a Python backtrace that helps identify the issue. Attach it to +the Phabricator_ task. + +Boot Timing +----------- + +During the migration and rewrite of functionality from Perl to Python, system +boot time increased significantly. You can analyze and graph boot time to see +detailed call sequences during startup. + +This uses the ``systemd-bootchart`` package, which is installed by default on +VyOS 1.3 (equuleus) and later. Configuration is versioned for comparable +results. Refer to bootchart.conf_ for the configuration file. + +To enable boot time graphing, add the following to the kernel command line: +``init=/usr/lib/systemd/systemd-bootchart`` + +You can also make this permanent by editing ``/boot/grub/grub.cfg``. + +Priorities +========== + +VyOS CLI depends heavily on priorities. Every CLI node has a corresponding +``node.def`` file and possibly an attached script. Nodes can have priorities, +and on system bootup or any ``commit`` to the configuration, scripts execute +from lowest to highest priority. This provides deterministic behavior. + +To debug priority issues or see script execution order, use the +``/opt/vyatta/sbin/priority.pl`` script, which lists the execution order of +scripts. + +.. stop_vyoslinter + +.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg +.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf +.. include:: /_include/common-references.txt + +.. start_vyoslinter diff --git a/docs/contributing/rst-development.rst b/docs/contributing/rst-development.rst new file mode 100644 index 00000000..5de950df --- /dev/null +++ b/docs/contributing/rst-development.rst @@ -0,0 +1,588 @@ +:lastproofread: 2025-12-12 + +.. _development: + +########### +Development +########### + +Learn how to contribute to VyOS. + +.. _architecture_overview: + +Architecture overview +===================== +VyOS source code is hosted on GitHub in the VyOS organization: +https://github.com/vyos + +VyOS is composed of multiple modules spread across different +repositories. Some modules contain forks of upstream +packages and are periodically synced. +VyOS consolidates most packages into the +`vyos-1x `__ +repository while maintaining a consistent structure. +The base code is being rewritten +from Perl and Bash to Python using an XML-based CLI interface definition. + +VyOS ISO build scripts are hosted in the +`vyos-build `__ repository. See the +``vyos-build`` repository +`README.md file `__ +for more information on building VyOS ISO images. + +Contributing code +================= + +.. warning:: + + You must sign the :doc:`Contributor License Agreement` + for your contributions to be accepted. + +VyOS is open-source and welcomes patches. +All submissions must adhere to these guidelines: + +* Each commit addresses a single issue or feature. +* Each commit message references a Phabricator_ task ID + (for example, ``T1234``). +* Each commit is associated with a username and email address + to identify the author (see `Configure your Git identity`_). +* Only submit bugfixes in packages other than https://github.com/vyos/vyos-1x. +* Commits follow the `coding guidelines`_ outlined below. + + +Determining package ownership +----------------------------- + +To determine which VyOS package contains a file you want to modify, use Debian's +``dpkg -S`` command on your running VyOS installation. + +Submitting your code +-------------------- + +Fork the repository and submit a GitHub pull request. This is the preferred way +to contribute changes to VyOS. + +To fork a VyOS repository: + +1. Append ``/fork`` to the repository URL on GitHub. For example, to fork + ``vyos-1x``, use: https://github.com/vyos/vyos-1x/fork + +2. Clone your fork or add it as a remote to your local repository: + + - Clone: ``git clone https://github.com//vyos-1x.git`` + - Add remote: ``git remote add myfork https://github.com//vyos-1x.git`` + +.. _Configure your Git identity: + +3. Configure your Git identity: + + .. code-block:: none + + git config --global user.name "J. Random Hacker" + git config --global user.email "jrhacker@example.net" + +4. Make your changes and add files to the Git index: + + - Single file: ``git add myfile`` + - Directory: ``git add somedir/*`` + +5. Commit your changes with a meaningful headline and Phabricator_ reference: + + ``git commit`` + +6. Push to your fork and create a GitHub pull request: + + ``git push`` + +Alternatively, you can export commits as patches and send them to +maintainers@vyos.net or attach them directly to the Phabricator_ task: + +* Export last commit: ``git format-patch`` +* Export last two commits: ``git format-patch -2`` + +Commit messages +=============== + +For guidance on writing commit messages, review the file history +with ``git log path/to/file.txt``. + +Every change must be associated with a task number (prefixed with **T**) and +a component. If no bug report or feature request exists for your changes, +create a Phabricator_ task first. Reference the task ID in your commit message: + +* ``ddclient: T1030: auto create runtime directories`` +* ``Jenkins: add current Git commit ID to build description`` + +If your pull request lacks a Phabricator_ reference, maintainers will request +that you amend the commit message. + +Writing good commit messages +----------------------------- + +Follow the format described in +the `Git documentation `__ +and `Chris Beams' guide `__. + +Commit message format: + +1. **Summary line** (50 characters recommended, 80 maximum): Include the + component + prefix and Phabricator_ reference (for example, ``snmp: T1111:`` or + ``ethernet: T2222:``). Concatenate multiple components with colons + (for example, ``snmp: ethernet: T3333``). + +2. **Blank line**: Separate the summary from the body. + This blank line is critical. + +4. **Message body** with details: + + * Describe what changed, why, and how. This helps with ``git bisect``. + * Wrap text at 72 characters for readability with ``git log`` on an 80x25 + terminal. + * Reference previous commits when applicable: + ``After commit abcd12ef ("snmp: this is a headline") + a Python import statement is missing, throwing the following exception: + ABCDEF`` + +5. **Cherry-pick option**: Always use the ``-x`` option when back-porting or + forward-porting commits: + + ``git cherry-pick -x `` + + This appends ``(cherry picked from commit )`` to the commit message, + making bisecting easier. + +6. **Single responsibility**: Each commit must be self-contained. Do not fix + multiple bugs in a single commit. Use ``git add --patch`` to stage only + the parts related to one issue. + +Constraints: + +* Bugfixes are only accepted for packages other than + https://github.com/vyos/vyos-1x. + New functionality must use the new XML/Python interface, not old-style + templates (``node.def`` files and Perl/Bash code). + +Coding guidelines +================= + +VyOS maintains consistent coding standards to help contributors navigate the +codebase and understand its logic. + +Formatting +---------- + +* **Python**: Use 4 spaces per indentation level. Tabs **must not** be used. +* **XML**: Use 2 spaces per indentation level. Tabs **must not** be used. + +Use tools like VIM extensions (xmllint) to enforce correct indentation. Add this +to your ``.vimrc`` file: + +.. code-block:: none + + au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null + +Then use ``gg=G`` in command mode to run the linter. + +Text generation +--------------- + +Use a template processor for generating config files: + +* **Jinja2** is the default template processor for VyOS code. +* Built-in string formatting **may** be used for simple line-oriented formats + (for example, iptables rules) where every line is self-contained. +* Template processors **must** be used for structured, multi-line formats + (for example, ISC DHCPd configuration). + +Python code +----------- + +Configuration scripts and operation mode scripts written in Python3 should +follow these guidelines: + +* Wrap lines at 80 characters. This improves readability when browsing + GitHub on mobile devices and reads well in side-by-side diffs. + +Structure your scripts with these functions: + +.. code-block:: python + + #!/usr/bin/env python3 + # + # Copyright (C) 2020 VyOS maintainers and contributors + # + # This program is free software; you can redistribute it and/or modify + # it under the terms of the GNU General Public License version 2 or later as + # published by the Free Software Foundation. + # + # This program is distributed in the hope that it will be useful, + # but WITHOUT ANY WARRANTY; without even the implied warranty of + # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + # GNU General Public License for more details. + # + # You should have received a copy of the GNU General Public License + # along with this program. If not, see . + + import sys + + from vyos.config import Config + from vyos import ConfigError + + def get_config(config=None): + if config: + conf = config + else: + conf = Config() + + # Base path to CLI nodes + base = ['...', '...'] + # Convert the VyOS config to an abstract internal representation + config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) + return config_data + + def verify(config): + # Verify that configuration is valid + if invalid: + raise ConfigError("Descriptive message") + + def generate(config): + # Generate daemon configs + pass + + def apply(config): + # Apply the generated configs to the live system + pass + + try: + c = get_config() + verify(c) + generate(c) + apply(c) + except ConfigError as e: + print(e) + sys.exit(1) + +``get_config()``: This function converts a VyOS config object to an abstract +internal representation. No other function may call the ``vyos.config.Config`` +object directly. Limiting config reads to one function makes it easier to +modify the config syntax in the future. Additionally, this design improves +testability since you can construct an internal representation by hand rather +than mocking the entire config subsystem. + +``verify()``: This function validates the internal representation. It must +raise ``ConfigError`` with a descriptive message if the config is invalid. It +**must not** make any changes to the system. This design enables future features +like commit dry-run ("commit test" as in JunOS) where the system can abort a +commit before making changes. + +``generate()``: This function generates config files for system components. + +``apply()``: This function applies the generated configuration to the live +system. Prefer non-disruptive reload when possible. Disruptive operations like +daemon restarts are acceptable only when: + +* The component does not support non-disruptive reload, or +* The expected service degradation is minimal (for example, auxiliary services + like LLDPd) + +For high-impact services (VPN daemons, routing protocols), make effort to +determine if changes can be applied non-disruptively before resorting to +restarts. + +Never modify active configuration directly unless absolutely necessary. Instead, +generate configuration files and apply them with a single command like service +reload through systemd. For example, save iptables rules to a file and load them +with ``iptables-restore`` rather than executing iptables commands one by one. + +The ``apply()`` and ``generate()`` functions may raise ``ConfigError`` if the +daemon fails to start with the updated config. However, this is not a substitute +for proper config validation in the ``verify()`` function. Make reasonable +effort to verify that generated configuration is valid and will be accepted by +the daemon, including cross-checks with other VyOS configuration subtrees when +necessary. + +Exceptions like ``VyOSError`` (raised by ``vyos.config.Config`` on improper +operations) should not be silenced or caught. While this may produce less +polished error output for users, it generates better bug reports and helps +maintainers debug issues. + +For reference implementations, see ``ntp.py`` or ``interfaces-bonding.py`` (for +tag nodes) in the `vyos-1x `__ repository. + +Other considerations: ``vyos-configd`` +-------------------------------------- + +All scripts now run under the config daemon and must conform to these +requirements: + +1. The signature and first four lines of ``get_config(...)`` **must** be as + specified above. + +2. Each of ``get_config``, ``verify``, ``apply``, and ``generate`` **must** + appear + with the correct signatures, even if they are a no-op. + +3. ``Config`` objects other than those in ``get_config`` **must not** appear. + +4. The legacy function ``my_set`` **must not** appear. Modifications to active + config **should not** appear in new code (alternative mechanisms may be used + if absolutely necessary). + +XML for CLI definitions +======================= + +XML interface definitions define the VyOS CLI structure. +Before VyOS ``1.2`` (crux), these +files were created manually. After a redesign, new-style templates are +automatically generated from XML input files. + +VyOS interface definitions come with a RelaxNG schema located in the +`vyos-1x `__ +repository. This schema is a modified version from ``VyConf`` (VyOS ``2.0``). +VyOS ``1.2.x`` +interface definitions are reusable in future VyOS versions with minimal changes. + +Schemas provide two benefits: + +* Complete grammar verification +* Automatic validation against the schema + +.. stop_vyoslinter +The `build-command-templates `__ +script converts XML definitions to +old-style templates and verifies them against the schema. A bad definition +causes the package build to fail. While the XML format is verbose, no other +format provides this level of verification. Specialized XML editors can help +manage verbosity. +.. start_vyoslinter + +Example XML interface definition: + +.. code-block:: xml + + + + + + + + + Task scheduler settings + + + + + Scheduled task + + <string> + Task name + + 999 + + + + + UNIX crontab time specification string + + + + + Execution interval + + <minutes> + Execution interval in minutes + + + <minutes>m + Execution interval in minutes + + + <hours>h + Execution interval in hours + + + <days>d + Execution interval in days + + + [1-9]([0-9]*)([mhd]{0,1}) + + + + + + Executable path and arguments + + + + + Path to executable + + + + + Arguments passed to the executable + + + + + + + + + + + + +XML definitions are purely declarative and contain no logic. All logic for +generating config files, restarting services, and related tasks is implemented +in configuration scripts. + +Template Processors +------------------- + +XML interface definition files use the ``.xml.in`` file extension (implemented +in :vytask:`T1843`). These files use the GCC preprocessor to reduce code +duplication in common areas: + +* VIF (including VIF-S and VIF-C) +* Address configuration +* Description +* Enabled/Disabled state + +Instead of repeating XML nodes, use include files with predefined features: + +.. stop_vyoslinter + +* `IPv4, IPv6, and DHCP(v6) `__ + address assignment. +* `IPv4 and IPv6 `__ + address assignment. +* `VLAN (VIF) `__ + definition. +* `MAC address `__ + assignment. + + +The ``.in`` files are preprocessed and stored in the `interface-definitions `__ +folder. The `scripts/build-command-templates `__ +script then operates on this folder to generate all required CLI nodes. + +.. start_vyoslinter + +Example preprocessor output: + +.. code-block:: none + + $ make interface_definitions + install -d -m 0755 build/interface-definitions + install -d -m 0755 build/op-mode-definitions + Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in + Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in + Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in + Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in + Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in + Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in + [...] + +Command Definition Guidelines +------------------------------ + +Use of Numbers +^^^^^^^^^^^^^^ + +Avoid using numbers in command names unless the number is part of a protocol +name or similar. For example, ``protocols ospfv3`` is appropriate, +but ``server-1`` is questionable. + +Help Strings +^^^^^^^^^^^^ + +Follow these guidelines for consistent, readable help strings: + +Capitalization and Punctuation +"""""""""""""""""""""""""""""" + +* Capitalize the first word of every help string. +* Do not use a period at the end of help strings. + +This standard mirrors network device CLIs and improves aesthetics. + +Examples: + +* Good: "Frobnication algorithm" +* Bad: "frobnication algorithm" +* Bad: "Frobnication algorithm." +* Incorrect: "frobnication algorithm." + +Abbreviations and Acronyms +"""""""""""""""""""""""""" + +* Capitalize all abbreviations and acronyms. + +Examples: + +* Good: "TCP connection timeout" +* Bad: "tcp connection timeout" +* Bad: "Tcp connection timeout" + +* Capitalize acronyms to distinguish them from normal words. + +Examples: + +* Good: RADIUS (remote authentication for dial-in user services) +* Bad: radius (unless referring to circular distance) + +* Follow accepted spelling conventions for mixed-case abbreviations. If it + contains "over" or "version", use lowercase. Follow RFC or standard spellings + when they exist. + +Examples: + +* Good: PPPoE, IPsec +* Bad: PPPOE, IPSEC +* Bad: pppoe, ipsec + +Verbs +""""" + +* Avoid verbs. If a verb can be omitted, omit it. + +Examples: + +* Good: "TCP connection timeout" +* Bad: "Set TCP connection timeout" + +* When a verb is essential, use it. For example: "Disable IPv6 forwarding on + all interfaces" for ``set system ipv6 disable-forwarding``. + +* Use infinitive form for necessary verbs. + +Examples: + +* Good: "Disable IPv6 forwarding" +* Bad: "Disables IPv6 forwarding" + + +C++ Backend Code +================ + +The VyOS CLI parser combines bash, bash-completion helpers, and the C++ backend +library `vyatta-cfg `__. This section +references common CLI commands and their C/C++ entry points: + +``set``: + +.. stop_vyoslinter + +* https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352 +* https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549 + +``commit``: + +* https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252 + +.. include:: /_include/common-references.txt + +.. start_vyoslinter diff --git a/docs/contributing/rst-index.rst b/docs/contributing/rst-index.rst new file mode 100644 index 00000000..18b925ec --- /dev/null +++ b/docs/contributing/rst-index.rst @@ -0,0 +1,14 @@ +############ +Contributing +############ + +.. toctree:: + :maxdepth: 1 + + build-vyos + development + cla + issues-features + upstream-packages + debugging + testing diff --git a/docs/contributing/rst-issues-features.rst b/docs/contributing/rst-issues-features.rst new file mode 100644 index 00000000..b1e1d732 --- /dev/null +++ b/docs/contributing/rst-issues-features.rst @@ -0,0 +1,130 @@ +:lastproofread: 2025-12-08 + +.. _issues_features: + +####################### +Issues/Feature requests +####################### + +.. _bug_report: + +Bug Report/Issue +================ + +Issues and bugs occur in every software project, and VyOS is no exception. + +I found a bug, what should I do? +-------------------------------- + +When you find a potential bug, first: + +* Consult the documentation_ to ensure you configured your system + correctly. +* Check if the VyOS community has identified a workaround for the bug through + Slack_ or the VyOS Forum_. + +Ensure the bug is reproducible +------------------------------ + +Include the following information when reporting a bug: + +* A sequence of configuration commands or a complete configuration file needed + to recreate the bug. Avoid partial configurations: a sequence of commands is + easy to paste and a complete configuration is easy to load, but a partial + config is hard to reconstruct. +* Describe the expected behavior and how it differs from what you observe. + Include command outputs or traffic dumps. Explain briefly why these outputs + are incorrect and what the correct behavior should be. +* A sequence of actions that trigger the bug. While not always possible, this + helps developers and community members confirm the issue and verify fixes. +* If the bug is a regression, specify the VyOS version where the feature worked + correctly (any working version is acceptable). Identify the exact version + that the feature stopped working, if possible. + +If you are uncertain whether the behavior is a bug or what the correct behavior +is, or if you lack a reliable reproducing procedure, post on the forum or ask in +chat first. If you have a subscription, create a support ticket. The team and +community can help identify the issue, work around it, and create an actionable +bug report. + +Report a Bug +------------ + +To open a bug report or feature request, create an account on +`vyos.dev `__, the public issue tracker for VyOS. + +When creating a new issue, select the appropriate project and: + +* Provide as much information as you can. +* Specify which VyOS version you are using: ``run show version``. +* Explain how to reproduce the bug. + +.. _feature_request: + +Feature Requests +================ + +Have an idea to improve VyOS or need a feature that would benefit all users? +Before submitting a feature request, search the public issue tracker +`vyos.dev `__ to check if a request already exists. You can +also enhance an existing request by providing additional information. + +Create a task before starting work on a feature, +even if it is a trivial feature. +The task tracker generates release notes, so all work must be reflected +in the tracker. + +Include at least the following information: + +* Provide a detailed description of the feature: what it is, how it works, and + how you would use it. Maintainers may not have experience with every feature, + protocol, and tool in VyOS. Detailed information helps VyOS contributors and + maintainers test new features they are unfamiliar with. +* Include proposed CLI syntax if the feature requires new commands. Provide both + configuration and operational mode commands if both are needed. + +Consider including the following information: + +* Is the feature already supported by the underlying component + (FreeRangeRouting, nftables, Kea, etc.)? +* How would you configure the feature manually within that component? +* Are there any limitations to using the feature + (hardware support, resource usage)? +* Are there any adverse or non-obvious interactions with other features? Should + the feature be mutually exclusive? +* Any relevant documentation or references about the feature. + +You do not need to provide all this information, but if you can, it simplifies +developers' work considerably. Research these questions when possible. + +Task auto-closing +================= + +A special task status exists for when all work by maintainers and contributors +is complete: **Needs reporter action**. + +VyOS assigns this status to: + +* Feature requests that do not include required information and need + clarification. +* Bug reports that lack reproducing procedures. +* Tasks that are implemented and tested by the implementation author, + but require testing in the real-world environment that only the reporter + can replicate (for example, hardware VyOS does not support or specific + network conditions). + +When a task is set to **Needs reporter action**: + +* If the reporter does not respond within two weeks, the task bot adds a comment + ("Any news?") to remind the reporter. +* If there is still no response after another two weeks, + the task is closed automatically. + +We do not auto-close tasks with any other status and do not close tasks due to +lack of maintainer activity. + +.. _documentation: https://docs.vyos.io +.. _Slack: https://slack.vyos.io +.. _Forum: https://forum.vyos.io + +.. include:: /_include/common-references.txt diff --git a/docs/contributing/rst-testing.rst b/docs/contributing/rst-testing.rst new file mode 100644 index 00000000..d7455103 --- /dev/null +++ b/docs/contributing/rst-testing.rst @@ -0,0 +1,197 @@ +:lastproofread: 2025-12-02 + +.. _testing: + +####### +Testing +####### + +One of the major features introduced in VyOS 1.3 is an automated test +framework. When you assemble an ISO image, several things can go wrong. +VyOS uses this framework to detect issues before they cause downstream problems. + +This section describes how the automated testing process at VyOS works. + +Smoketests +========== + +Smoketests execute predefined VyOS CLI commands and check if the desired +daemon or service configuration is rendered. + +When an ISO image is assembled by the `VyOS CI`_, the ``BUILD_SMOKETEST`` +parameter is enabled by default. This extends the ISO configuration line +with the following packages: + +.. code-block:: python + + def CUSTOM_PACKAGES = '' + if (params.BUILD_SMOKETESTS) + CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' + +If you plan to build your own custom ISO image and want to use VyOS's +smoketests, ensure that you have the `vyos-1x-smoketest` package installed. + +The ``make test`` command from the vyos-build_ repository launches a new +QEMU instance, and the ISO image is first installed to the virtual hard disk. + +After the first boot into the newly installed system, the main Smoketest script +is executed. It can be found at `/usr/bin/vyos-smoketest`. + +The script searches for executable test cases under +``/usr/libexec/vyos/tests/smoke/cli/`` and executes them one by one. + +.. note:: Smoketests will alter the system configuration. If you are logged + in remotely, you may lose your connection to the system. + +.. note:: To enable smoketest debugging (print the CLI set commands used), + run: ``touch /tmp/vyos.smoketest.debug``. + +Manual Smoketest Run +-------------------- + +Each test is contained in its own file, so you can execute a single Smoketest +manually by running the Python test script. + +Example: + +.. code-block:: none + + vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py + test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok + test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok + test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok + test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok + test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok + test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok + test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok + test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok + test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok + test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok + test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok + test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok + test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok + + ---------------------------------------------------------------------- + Ran 13 tests in 348.191s + + OK + +Interface-based tests +--------------------- + +Our smoketests not only test daemons and services, but also check if interface +configuration works as expected. There is a common base class named +``base_interfaces_test.py`` that holds all the common code for interface tests. + +These common tests consist of: + +* Add one or more IP addresses +* DHCP client and DHCPv6 prefix delegation +* MTU size +* IP and IPv6 options +* Port description +* Port disable +* VLANs (QinQ and regular 802.1q) +* ... + +.. note:: When you are working on interface configuration and want to test + if the Smoketests pass, you would normally lose the remote SSH connection + to your :abbr:`DUT (Device Under Test)`. To handle this, some interface-based + tests can be called with an environment variable beforehand to limit the + number of interfaces used in the test. By default, all interfaces (e.g., all + Ethernet interfaces) are used. + +.. code-block:: none + + vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py + test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok + test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok + test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok + test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok + test_bonding_min_links (__main__.BondingInterfaceTest) ... ok + test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok + test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok + test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok + test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok + test_interface_description (__main__.BondingInterfaceTest) ... ok + test_interface_disable (__main__.BondingInterfaceTest) ... ok + test_interface_ip_options (__main__.BondingInterfaceTest) ... ok + test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok + test_interface_mtu (__main__.BondingInterfaceTest) ... ok + test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok + test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok + test_span_mirror (__main__.BondingInterfaceTest) ... ok + test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok + test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok + test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok + test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok + test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok + test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok + + ---------------------------------------------------------------------- + Ran 23 tests in 244.694s + + OK + +This will limit the `bond` interface test to use only `eth1` and `eth2` +as member ports. + +Config Load Tests +================= + +The other part of our tests are called "config load tests." Config load tests +sequentially load arbitrary configuration files to verify that configuration +migration scripts work as designed and that a given set of functionality can +still be loaded with a fresh VyOS ISO image. + +The configurations are all derived from production systems and can act as +test cases or as references for enabling certain features. The configurations +can be found here: +https://github.com/vyos/vyos-1x/tree/current/smoketest/configs + +The entire test is controlled by the main wrapper script +``/usr/bin/vyos-configtest``. +It behaves in the same way as the main smoketest script. It scans the folder +for potential configuration files and issues a ``load`` command for each file. + +Manual config load test +----------------------- + +You do not have to load all configurations sequentially; you can also load +individual test configurations manually. + +.. code-block:: none + + vyos@vyos:~$ configure + load[edit] + + vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small + Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' + Load complete. Use 'commit' to make changes effective. + [edit] + vyos@vyos# compare + [edit interfaces ethernet eth0] + -hw-id 00:50:56:bf:c5:6d + [edit interfaces ethernet eth1] + +duplex auto + -hw-id 00:50:56:b3:38:c5 + +speed auto + [edit interfaces] + -ethernet eth2 { + - hw-id 00:50:56:b3:9c:1d + -} + -vti vti1 { + - address 192.0.2.1/30 + -} + ... + + vyos@vyos# commit + vyos@vyos# + +.. note:: Some configurations have preconditions that must be met. These most + likely include generation of cryptographic keys before the config can be + applied; otherwise, you will get a commit error. If you are interested in + how those preconditions are fulfilled, check the vyos-build_ repository and + the ``scripts/check-qemu-install`` file. + +.. include:: /_include/common-references.txt diff --git a/docs/contributing/rst-upstream-packages.rst b/docs/contributing/rst-upstream-packages.rst new file mode 100644 index 00000000..c740b190 --- /dev/null +++ b/docs/contributing/rst-upstream-packages.rst @@ -0,0 +1,159 @@ +:lastproofread: 2026-01-30 + +.. _upstream_packages: + +################# +Upstream Packages +################# + +Many base system packages are pulled straight from Debian's ``main`` and +``contrib`` repositories, but there are exceptions. If you only want to build +a fresh ISO image, you can skip +this section. This information may be useful for a deeper dive into VyOS. + + +.. stop_vyoslinter + +System packages that are not directly pulled from Debian are built through a +separate build system, ``build.py`` in the `vyos-build `__ repository. + +.. start_vyoslinter + + +Overview +======== + +Previously, VyOS used Jenkins for building upstream packages. With the move away +from Jenkins, the build system was replaced with a Python-based solution using +``build.py`` and ``package.toml`` configuration files. + +Each package directory contains: + +- A ``package.toml`` configuration file that defines how the package is built. +- A symlink to the common ``build.py`` script in the build system. + + +Building Packages +================= + +To build a package, navigate to the package directory and execute the +build script: + +.. code-block:: console + + cd package-build/ + ./build.py + +The script will: + +1. Check out the source code from the configured repository. +2. Apply any patches defined in the configuration. +3. Execute pre-build hooks (if configured). +4. Build the package using the specified build command. +5. Generate both binary (``.deb``) packages and source tarballs. + + +Package Configuration (package.toml) +==================================== + +Each package directory contains a ``package.toml`` file that defines the build +parameters. The key configuration fields are: + +**name** + The package name (e.g., ``frr``) + +**commit_id** + The specific commit, tag, or branch to check out from the source repository + (e.g., ``stable/10.5``) + +**scm_url** + The Git URL of the upstream source repository + (e.g., ``https://github.com/FRRouting/frr.git``) + +.. stop_vyoslinter +**build_cmd** + The command to execute for building the package. This replaces what was + previously defined in the Jenkins ``Jenkinsfile``. + + Default if not specified: ``dpkg-buildpackage -uc -us -tc -F --source-option=--tar-ignore=.git --source-option=--tar-ignore=.github`` + + Example with custom build command: + + .. code-block:: toml + + build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" +.. start_vyoslinter + +**pre_build_hook** (Optional) + A shell command or script that executes after the repository is checked out + and before the build process begins. This allows you to perform preparatory + tasks such as: + + - Creating directories + - Copying files + - Running custom setup scripts + - Installing dependencies + + Single command example: + + .. code-block:: toml + + pre_build_hook = "echo 'Preparing build environment'" + + Multi-line commands example: + + .. code-block:: toml + + pre_build_hook = """ + mkdir -p ../hello/vyos + mkdir -p ../vyos + cp example.txt ../vyos + """ + + Combined commands and scripts: + + .. code-block:: toml + + pre_build_hook = "ls -l; ./script.sh" + +**apply_patches** (Optional) + Boolean flag to control whether patches should be applied. Defaults to + ``True``. + + .. code-block:: toml + + apply_patches = false + +**prepare_package** (Optional) + Boolean flag to enable package preparation. When set to ``True``, the + ``install_data`` configuration is used. + +**install_data** (Optional) + Data used for package preparation when ``prepare_package`` is enabled. + + +Example package.toml file +=========================== + +Here's an example configuration for the FRRouting (FRR) package: + +.. code-block:: toml + + name = "frr" + commit_id = "stable/10.5" + scm_url = "https://github.com/FRRouting/frr.git" + build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" + + +Build Output +============ + +After running ``./build.py``, the following artifacts are generated in the +package directory: + +- ``.deb`` files - Binary Debian packages ready for installation +- ``.tar.gz`` files - Source tarballs of the checked-out repositories +- Additional build artifacts as produced by the Debian build system + +The build script also creates build dependency packages (`*build-deps*.deb`), +which are automatically cleaned up after the build completes. diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md new file mode 100644 index 00000000..b68cad1b --- /dev/null +++ b/docs/contributing/testing.md @@ -0,0 +1,207 @@ +--- +lastproofread: '2025-12-02' +--- + +(testing)= + +# Testing + +One of the major features introduced in VyOS 1.3 is an automated test +framework. When you assemble an ISO image, several things can go wrong. +VyOS uses this framework to detect issues before they cause downstream problems. + +This section describes how the automated testing process at VyOS works. + +## Smoketests + +Smoketests execute predefined VyOS CLI commands and check if the desired +daemon or service configuration is rendered. + +When an ISO image is assembled by the [VyOS CI](https://ci.vyos.net), the `BUILD_SMOKETEST` +parameter is enabled by default. This extends the ISO configuration line +with the following packages: + +```python +def CUSTOM_PACKAGES = '' + if (params.BUILD_SMOKETESTS) + CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' +``` + +If you plan to build your own custom ISO image and want to use VyOS's +smoketests, ensure that you have the `vyos-1x-smoketest` package installed. + +The `make test` command from the [vyos-build](https://github.com/vyos/vyos-build) repository launches a new +QEMU instance, and the ISO image is first installed to the virtual hard disk. + +After the first boot into the newly installed system, the main Smoketest script +is executed. It can be found at `/usr/bin/vyos-smoketest`. + +The script searches for executable test cases under +`/usr/libexec/vyos/tests/smoke/cli/` and executes them one by one. + +:::{note} +Smoketests will alter the system configuration. If you are logged +in remotely, you may lose your connection to the system. +::: + +:::{note} +To enable smoketest debugging (print the CLI set commands used), +run: `touch /tmp/vyos.smoketest.debug`. +::: + +### Manual Smoketest Run + +Each test is contained in its own file, so you can execute a single Smoketest +manually by running the Python test script. + +Example: + +```none +vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py +test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok +test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok +test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok +test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok +test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok +test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok +test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok +test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok +test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok +test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok +test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok +test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok +test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok + +---------------------------------------------------------------------- +Ran 13 tests in 348.191s + +OK +``` + + +### Interface-based tests + +Our smoketests not only test daemons and services, but also check if interface +configuration works as expected. There is a common base class named +`base_interfaces_test.py` that holds all the common code for interface tests. + +These common tests consist of: + +- Add one or more IP addresses + +- DHCP client and DHCPv6 prefix delegation + +- MTU size + +- IP and IPv6 options + +- Port description + +- Port disable + +- VLANs (QinQ and regular 802.1q) + +- ... + +:::{note} +When you are working on interface configuration and want to test +if the Smoketests pass, you would normally lose the remote SSH connection +to your {abbr}`DUT (Device Under Test)`. To handle this, some interface-based +tests can be called with an environment variable beforehand to limit the +number of interfaces used in the test. By default, all interfaces (e.g., all +Ethernet interfaces) are used. +::: + +```none +vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py +test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok +test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok +test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok +test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok +test_bonding_min_links (__main__.BondingInterfaceTest) ... ok +test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok +test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok +test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok +test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok +test_interface_description (__main__.BondingInterfaceTest) ... ok +test_interface_disable (__main__.BondingInterfaceTest) ... ok +test_interface_ip_options (__main__.BondingInterfaceTest) ... ok +test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok +test_interface_mtu (__main__.BondingInterfaceTest) ... ok +test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok +test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok +test_span_mirror (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok +test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok +test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok + +---------------------------------------------------------------------- +Ran 23 tests in 244.694s + +OK +``` + +This will limit the `bond` interface test to use only `eth1` and `eth2` +as member ports. + +## Config Load Tests + +The other part of our tests are called "config load tests." Config load tests +sequentially load arbitrary configuration files to verify that configuration +migration scripts work as designed and that a given set of functionality can +still be loaded with a fresh VyOS ISO image. + +The configurations are all derived from production systems and can act as +test cases or as references for enabling certain features. The configurations +can be found here: + + +The entire test is controlled by the main wrapper script +`/usr/bin/vyos-configtest`. +It behaves in the same way as the main smoketest script. It scans the folder +for potential configuration files and issues a `load` command for each file. + +### Manual config load test + +You do not have to load all configurations sequentially; you can also load +individual test configurations manually. + +```none +vyos@vyos:~$ configure +load[edit] + +vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small +Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' +Load complete. Use 'commit' to make changes effective. +[edit] +vyos@vyos# compare +[edit interfaces ethernet eth0] +-hw-id 00:50:56:bf:c5:6d +[edit interfaces ethernet eth1] ++duplex auto +-hw-id 00:50:56:b3:38:c5 ++speed auto +[edit interfaces] +-ethernet eth2 { +- hw-id 00:50:56:b3:9c:1d +-} +-vti vti1 { +- address 192.0.2.1/30 +-} +... + +vyos@vyos# commit +vyos@vyos# +``` + +:::{note} +Some configurations have preconditions that must be met. These most +likely include generation of cryptographic keys before the config can be +applied; otherwise, you will get a commit error. If you are interested in +how those preconditions are fulfilled, check the [vyos-build](https://github.com/vyos/vyos-build) repository and +the `scripts/check-qemu-install` file. +::: + diff --git a/docs/contributing/testing.rst b/docs/contributing/testing.rst deleted file mode 100644 index d7455103..00000000 --- a/docs/contributing/testing.rst +++ /dev/null @@ -1,197 +0,0 @@ -:lastproofread: 2025-12-02 - -.. _testing: - -####### -Testing -####### - -One of the major features introduced in VyOS 1.3 is an automated test -framework. When you assemble an ISO image, several things can go wrong. -VyOS uses this framework to detect issues before they cause downstream problems. - -This section describes how the automated testing process at VyOS works. - -Smoketests -========== - -Smoketests execute predefined VyOS CLI commands and check if the desired -daemon or service configuration is rendered. - -When an ISO image is assembled by the `VyOS CI`_, the ``BUILD_SMOKETEST`` -parameter is enabled by default. This extends the ISO configuration line -with the following packages: - -.. code-block:: python - - def CUSTOM_PACKAGES = '' - if (params.BUILD_SMOKETESTS) - CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' - -If you plan to build your own custom ISO image and want to use VyOS's -smoketests, ensure that you have the `vyos-1x-smoketest` package installed. - -The ``make test`` command from the vyos-build_ repository launches a new -QEMU instance, and the ISO image is first installed to the virtual hard disk. - -After the first boot into the newly installed system, the main Smoketest script -is executed. It can be found at `/usr/bin/vyos-smoketest`. - -The script searches for executable test cases under -``/usr/libexec/vyos/tests/smoke/cli/`` and executes them one by one. - -.. note:: Smoketests will alter the system configuration. If you are logged - in remotely, you may lose your connection to the system. - -.. note:: To enable smoketest debugging (print the CLI set commands used), - run: ``touch /tmp/vyos.smoketest.debug``. - -Manual Smoketest Run --------------------- - -Each test is contained in its own file, so you can execute a single Smoketest -manually by running the Python test script. - -Example: - -.. code-block:: none - - vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py - test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok - test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok - test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok - test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok - test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok - test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok - test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok - test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok - test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok - test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok - test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok - test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok - test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok - - ---------------------------------------------------------------------- - Ran 13 tests in 348.191s - - OK - -Interface-based tests ---------------------- - -Our smoketests not only test daemons and services, but also check if interface -configuration works as expected. There is a common base class named -``base_interfaces_test.py`` that holds all the common code for interface tests. - -These common tests consist of: - -* Add one or more IP addresses -* DHCP client and DHCPv6 prefix delegation -* MTU size -* IP and IPv6 options -* Port description -* Port disable -* VLANs (QinQ and regular 802.1q) -* ... - -.. note:: When you are working on interface configuration and want to test - if the Smoketests pass, you would normally lose the remote SSH connection - to your :abbr:`DUT (Device Under Test)`. To handle this, some interface-based - tests can be called with an environment variable beforehand to limit the - number of interfaces used in the test. By default, all interfaces (e.g., all - Ethernet interfaces) are used. - -.. code-block:: none - - vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py - test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok - test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok - test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok - test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok - test_bonding_min_links (__main__.BondingInterfaceTest) ... ok - test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok - test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok - test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok - test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok - test_interface_description (__main__.BondingInterfaceTest) ... ok - test_interface_disable (__main__.BondingInterfaceTest) ... ok - test_interface_ip_options (__main__.BondingInterfaceTest) ... ok - test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok - test_interface_mtu (__main__.BondingInterfaceTest) ... ok - test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok - test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok - test_span_mirror (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok - test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok - test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok - test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok - - ---------------------------------------------------------------------- - Ran 23 tests in 244.694s - - OK - -This will limit the `bond` interface test to use only `eth1` and `eth2` -as member ports. - -Config Load Tests -================= - -The other part of our tests are called "config load tests." Config load tests -sequentially load arbitrary configuration files to verify that configuration -migration scripts work as designed and that a given set of functionality can -still be loaded with a fresh VyOS ISO image. - -The configurations are all derived from production systems and can act as -test cases or as references for enabling certain features. The configurations -can be found here: -https://github.com/vyos/vyos-1x/tree/current/smoketest/configs - -The entire test is controlled by the main wrapper script -``/usr/bin/vyos-configtest``. -It behaves in the same way as the main smoketest script. It scans the folder -for potential configuration files and issues a ``load`` command for each file. - -Manual config load test ------------------------ - -You do not have to load all configurations sequentially; you can also load -individual test configurations manually. - -.. code-block:: none - - vyos@vyos:~$ configure - load[edit] - - vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small - Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' - Load complete. Use 'commit' to make changes effective. - [edit] - vyos@vyos# compare - [edit interfaces ethernet eth0] - -hw-id 00:50:56:bf:c5:6d - [edit interfaces ethernet eth1] - +duplex auto - -hw-id 00:50:56:b3:38:c5 - +speed auto - [edit interfaces] - -ethernet eth2 { - - hw-id 00:50:56:b3:9c:1d - -} - -vti vti1 { - - address 192.0.2.1/30 - -} - ... - - vyos@vyos# commit - vyos@vyos# - -.. note:: Some configurations have preconditions that must be met. These most - likely include generation of cryptographic keys before the config can be - applied; otherwise, you will get a commit error. If you are interested in - how those preconditions are fulfilled, check the vyos-build_ repository and - the ``scripts/check-qemu-install`` file. - -.. include:: /_include/common-references.txt diff --git a/docs/contributing/upstream-packages.md b/docs/contributing/upstream-packages.md new file mode 100644 index 00000000..fe01df82 --- /dev/null +++ b/docs/contributing/upstream-packages.md @@ -0,0 +1,147 @@ +--- +lastproofread: '2026-01-30' +--- + +(upstream-packages)= + +# Upstream Packages + +Many base system packages are pulled straight from Debian's `main` and +`contrib` repositories, but there are exceptions. If you only want to build +a fresh ISO image, you can skip +this section. This information may be useful for a deeper dive into VyOS. + +System packages that are not directly pulled from Debian are built through a +separate build system, `build.py` in the [vyos-build](https://github.com/vyos/vyos-build/tree/current/scripts/package-build) repository. + +## Overview + +Previously, VyOS used Jenkins for building upstream packages. With the move away +from Jenkins, the build system was replaced with a Python-based solution using +`build.py` and `package.toml` configuration files. + +Each package directory contains: + +- A `package.toml` configuration file that defines how the package is built. +- A symlink to the common `build.py` script in the build system. + +## Building Packages + +To build a package, navigate to the package directory and execute the +build script: + +```console +cd package-build/ +./build.py +``` + +The script will: + +1. Check out the source code from the configured repository. +2. Apply any patches defined in the configuration. +3. Execute pre-build hooks (if configured). +4. Build the package using the specified build command. +5. Generate both binary (`.deb`) packages and source tarballs. + +## Package Configuration (package.toml) + +Each package directory contains a `package.toml` file that defines the build +parameters. The key configuration fields are: + +```{eval-rst} +**name** + The package name (e.g., ``frr``) + +**commit_id** + The specific commit, tag, or branch to check out from the source repository + (e.g., ``stable/10.5``) + +**scm_url** + The Git URL of the upstream source repository + (e.g., ``https://github.com/FRRouting/frr.git``) + +.. stop_vyoslinter +**build_cmd** + The command to execute for building the package. This replaces what was + previously defined in the Jenkins ``Jenkinsfile``. + + Default if not specified: ``dpkg-buildpackage -uc -us -tc -F --source-option=--tar-ignore=.git --source-option=--tar-ignore=.github`` + + Example with custom build command: + + .. code-block:: toml + + build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" +.. start_vyoslinter + +**pre_build_hook** (Optional) + A shell command or script that executes after the repository is checked out + and before the build process begins. This allows you to perform preparatory + tasks such as: + + - Creating directories + - Copying files + - Running custom setup scripts + - Installing dependencies + + Single command example: + + .. code-block:: toml + + pre_build_hook = "echo 'Preparing build environment'" + + Multi-line commands example: + + .. code-block:: toml + + pre_build_hook = """ + mkdir -p ../hello/vyos + mkdir -p ../vyos + cp example.txt ../vyos + """ + + Combined commands and scripts: + + .. code-block:: toml + + pre_build_hook = "ls -l; ./script.sh" + +**apply_patches** (Optional) + Boolean flag to control whether patches should be applied. Defaults to + ``True``. + + .. code-block:: toml + + apply_patches = false + +**prepare_package** (Optional) + Boolean flag to enable package preparation. When set to ``True``, the + ``install_data`` configuration is used. + +**install_data** (Optional) + Data used for package preparation when ``prepare_package`` is enabled. +``` + +## Example package.toml file + +Here's an example configuration for the FRRouting (FRR) package: + +```toml +name = "frr" +commit_id = "stable/10.5" +scm_url = "https://github.com/FRRouting/frr.git" +build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" +``` + + +## Build Output + +After running `./build.py`, the following artifacts are generated in the +package directory: + +- `.deb` files - Binary Debian packages ready for installation +- `.tar.gz` files - Source tarballs of the checked-out repositories +- Additional build artifacts as produced by the Debian build system + +The build script also creates build dependency packages (`*build-deps*.deb`), +which are automatically cleaned up after the build completes. diff --git a/docs/contributing/upstream-packages.rst b/docs/contributing/upstream-packages.rst deleted file mode 100644 index c740b190..00000000 --- a/docs/contributing/upstream-packages.rst +++ /dev/null @@ -1,159 +0,0 @@ -:lastproofread: 2026-01-30 - -.. _upstream_packages: - -################# -Upstream Packages -################# - -Many base system packages are pulled straight from Debian's ``main`` and -``contrib`` repositories, but there are exceptions. If you only want to build -a fresh ISO image, you can skip -this section. This information may be useful for a deeper dive into VyOS. - - -.. stop_vyoslinter - -System packages that are not directly pulled from Debian are built through a -separate build system, ``build.py`` in the `vyos-build `__ repository. - -.. start_vyoslinter - - -Overview -======== - -Previously, VyOS used Jenkins for building upstream packages. With the move away -from Jenkins, the build system was replaced with a Python-based solution using -``build.py`` and ``package.toml`` configuration files. - -Each package directory contains: - -- A ``package.toml`` configuration file that defines how the package is built. -- A symlink to the common ``build.py`` script in the build system. - - -Building Packages -================= - -To build a package, navigate to the package directory and execute the -build script: - -.. code-block:: console - - cd package-build/ - ./build.py - -The script will: - -1. Check out the source code from the configured repository. -2. Apply any patches defined in the configuration. -3. Execute pre-build hooks (if configured). -4. Build the package using the specified build command. -5. Generate both binary (``.deb``) packages and source tarballs. - - -Package Configuration (package.toml) -==================================== - -Each package directory contains a ``package.toml`` file that defines the build -parameters. The key configuration fields are: - -**name** - The package name (e.g., ``frr``) - -**commit_id** - The specific commit, tag, or branch to check out from the source repository - (e.g., ``stable/10.5``) - -**scm_url** - The Git URL of the upstream source repository - (e.g., ``https://github.com/FRRouting/frr.git``) - -.. stop_vyoslinter -**build_cmd** - The command to execute for building the package. This replaces what was - previously defined in the Jenkins ``Jenkinsfile``. - - Default if not specified: ``dpkg-buildpackage -uc -us -tc -F --source-option=--tar-ignore=.git --source-option=--tar-ignore=.github`` - - Example with custom build command: - - .. code-block:: toml - - build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" -.. start_vyoslinter - -**pre_build_hook** (Optional) - A shell command or script that executes after the repository is checked out - and before the build process begins. This allows you to perform preparatory - tasks such as: - - - Creating directories - - Copying files - - Running custom setup scripts - - Installing dependencies - - Single command example: - - .. code-block:: toml - - pre_build_hook = "echo 'Preparing build environment'" - - Multi-line commands example: - - .. code-block:: toml - - pre_build_hook = """ - mkdir -p ../hello/vyos - mkdir -p ../vyos - cp example.txt ../vyos - """ - - Combined commands and scripts: - - .. code-block:: toml - - pre_build_hook = "ls -l; ./script.sh" - -**apply_patches** (Optional) - Boolean flag to control whether patches should be applied. Defaults to - ``True``. - - .. code-block:: toml - - apply_patches = false - -**prepare_package** (Optional) - Boolean flag to enable package preparation. When set to ``True``, the - ``install_data`` configuration is used. - -**install_data** (Optional) - Data used for package preparation when ``prepare_package`` is enabled. - - -Example package.toml file -=========================== - -Here's an example configuration for the FRRouting (FRR) package: - -.. code-block:: toml - - name = "frr" - commit_id = "stable/10.5" - scm_url = "https://github.com/FRRouting/frr.git" - build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" - - -Build Output -============ - -After running ``./build.py``, the following artifacts are generated in the -package directory: - -- ``.deb`` files - Binary Debian packages ready for installation -- ``.tar.gz`` files - Source tarballs of the checked-out repositories -- Additional build artifacts as produced by the Debian build system - -The build script also creates build dependency packages (`*build-deps*.deb`), -which are automatically cleaned up after the build completes. -- cgit v1.2.3