diff options
Diffstat (limited to 'docs/contributing')
| -rw-r--r-- | docs/contributing/build-vyos.rst | 335 | ||||
| -rw-r--r-- | docs/contributing/development.rst | 20 | ||||
| -rw-r--r-- | docs/contributing/upstream-packages.rst | 72 |
3 files changed, 349 insertions, 78 deletions
diff --git a/docs/contributing/build-vyos.rst b/docs/contributing/build-vyos.rst index e8bb9f37..c6098763 100644 --- a/docs/contributing/build-vyos.rst +++ b/docs/contributing/build-vyos.rst @@ -257,6 +257,341 @@ The full and current list can be generated with ``./configure --help``: .. _build_custom_packages: +Linux Kernel +============ + +The Linux Kernel used by VyOS is heavily tied to the ISO build process. The +file ``data/defaults.json`` hosts a JSON definition if the Kernel version used +``kernel_version`` and the ``kernel_flavor`` of the Kernel which represents the +Kernels LOCAL_VERSION. Both together form the Kernel Version variable in the +system: + +.. code-block:: none + + vyos@vyos:~$ uname -r + 4.19.146-amd64-vyos + +Other packages (e.g. vyos-1x) add dependencies to the ISO build procedure on +e.g. the wireguard-modules package which itself adds a dependency on the Kernel +version used due to the module it ships. This may change (for WireGuard) in +future Kernel releases but as long as we have out-of-tree modules. + +* WireGuard +* Accel-PPP +* Intel NIC drivers +* Inter QAT + +Each of those modules holds a dependency on the Kernel Version and if you are +lucky enough to receive an ISO build error which sounds like: + +.. code-block:: none + + I: Create initramfs if it does not exist. + Extra argument '4.19.146-amd64-vyos' + Usage: update-initramfs {-c|-d|-u} [-k version] [-v] [-b directory] + Options: + -k version Specify kernel version or 'all' + -c Create a new initramfs + -u Update an existing initramfs + -d Remove an existing initramfs + -b directory Set alternate boot directory + -v Be verbose + See update-initramfs(8) for further details. + E: config/hooks/live/17-gen_initramfs.chroot failed (exit non-zero). You should check for errors. + +The most obvious reasons could be: + +* ``vyos-build`` repo is outdate, please ``git pull`` to update to the latest + release Kernel version from us. + +* You have your own custom Kernel `*.deb` packages in the `packages` folder but + missed to create all required out-of tree modules like Accel-PPP, WireGuard, + Intel QAT, Intel NIC + +Building The Kernel +------------------- + +The Kernel build is quiet easy, most of the required steps can be found in the +``vyos-build/packages/linux-kernel/Jenkinsfile`` but we will walk you through +it. + +Clone the Kernel source to `vyos-build/packages/linux-kernel/`: + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel/ + $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git + +Checkout the required Kernel version - see ``vyos-build/data/defaults.json`` +file (example uses kernel 4.19.146): + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel/linux + $ git checkout v4.19.146 + Checking out files: 100% (61536/61536), done. + Note: checking out 'v4.19.146'. + + You are in 'detached HEAD' state. You can look around, make experimental + changes and commit them, and you can discard any commits you make in this + state without impacting any branches by performing another checkout. + + If you want to create a new branch to retain commits you create, you may + do so (now or later) by using -b with the checkout command again. Example: + + git checkout -b <new-branch-name> + + HEAD is now at 015e94d0e37b Linux 4.19.146 + +Now we can use the helper script ``build-kernel.sh`` which does all the necessary +Voodoo by applying required patches from the `vyos-build/packages/linux-kernel/ +patches` folder, copying our Kernel configuration ``x86_64_vyos_defconfig`` to +the right location, and finally building the Debian packages. + +.. note:: Building the kernel will take some time depending on the speed and quantity of your CPU/cores and disk speed. Plan on 20 minutes or even longer on lower end harder + +.. code-block:: none + + (18:59) vyos_bld 412374ca36b8:/vyos/vyos-build/packages/linux-kernel [current] # ./build-kernel.sh + I: Copy Kernel config (x86_64_vyos_defconfig) to Kernel Source + I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0001-VyOS-Add-linkstate-IP-device-attribute.patch + patching file Documentation/networking/ip-sysctl.txt + patching file include/linux/inetdevice.h + patching file include/linux/ipv6.h + patching file include/uapi/linux/ip.h + patching file include/uapi/linux/ipv6.h + patching file net/ipv4/devinet.c + Hunk #1 succeeded at 2319 (offset 1 line). + patching file net/ipv6/addrconf.c + patching file net/ipv6/route.c + I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0002-VyOS-add-inotify-support-for-stackable-filesystems-o.patch + patching file fs/notify/inotify/Kconfig + patching file fs/notify/inotify/inotify_user.c + patching file fs/overlayfs/super.c + Hunk #2 succeeded at 1713 (offset 9 lines). + Hunk #3 succeeded at 1739 (offset 9 lines). + Hunk #4 succeeded at 1762 (offset 9 lines). + patching file include/linux/inotify.h + I: Apply Kernel patch: /vyos/vyos-build/packages/linux-kernel/patches/kernel/0003-RFC-builddeb-add-linux-tools-package-with-perf.patch + patching file scripts/package/builddeb + I: make x86_64_vyos_defconfig + HOSTCC scripts/basic/fixdep + HOSTCC scripts/kconfig/conf.o + YACC scripts/kconfig/zconf.tab.c + LEX scripts/kconfig/zconf.lex.c + HOSTCC scripts/kconfig/zconf.tab.o + HOSTLD scripts/kconfig/conf + # + # configuration written to .config + # + I: Generate environment file containing Kernel variable + I: Build Debian Kernel package + UPD include/config/kernel.release + /bin/sh ./scripts/package/mkdebian + dpkg-buildpackage -r"fakeroot -u" -a$(cat debian/arch) -b -nc -uc + dpkg-buildpackage: info: source package linux-4.19.146-amd64-vyos + dpkg-buildpackage: info: source version 4.19.146-1 + dpkg-buildpackage: info: source distribution buster + dpkg-buildpackage: info: source changed by vyos_bld <christian@poessinger.com> + dpkg-buildpackage: info: host architecture amd64 + dpkg-buildpackage: warning: debian/rules is not executable; fixing that + dpkg-source --before-build . + debian/rules build + make KERNELRELEASE=4.19.146-amd64-vyos ARCH=x86 KBUILD_BUILD_VERSION=1 KBUILD_SRC= + SYSTBL arch/x86/include/generated/asm/syscalls_32.h + + ... + + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory + dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypto.so.1.1 (they use none of the library's symbols) + dpkg-shlibdeps: warning: package could avoid a useless dependency if /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/trace /vyos/vyos-build/packages/linux-kernel/linux/debian/toolstmp/usr/bin/perf were not linked against libcrypt.so.1 (they use none of the library's symbols) + dpkg-deb: building package 'linux-tools-4.19.146-amd64-vyos' in '../linux-tools-4.19.146-amd64-vyos_4.19.146-1_amd64.deb'. + dpkg-genbuildinfo --build=binary + dpkg-genchanges --build=binary >../linux-4.19.146-amd64-vyos_4.19.146-1_amd64.changes + dpkg-genchanges: warning: package linux-image-4.19.146-amd64-vyos-dbg in control file but not in files list + dpkg-genchanges: info: binary-only upload (no source code included) + dpkg-source --after-build . + dpkg-buildpackage: info: binary-only upload (no source included) + + +In the end you will be presented with the Kernel binary packages which you can +then use in your custom ISO build process, by placing all the `*.deb` files in +the vyos-build/packages folder where they will be used automatically when building VyOS as documented above. + +Building Out-Of-Tree Modules +---------------------------- + +Building the Kernel is one part, but now you also need to build the required +out-of-tree modules so everything is lined up and the ABIs match. To do so, +you can again take a look at ``vyos-build/packages/linux-kernel/Jenkinsfile`` +to see all of the required modules and their selected version. We will show you +once how to build all the current required modules. + +WireGuard +^^^^^^^^^ + +First clone the source code and checkout the appropriate version by running: + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel + $ git clone https://salsa.debian.org/debian/wireguard-linux-compat.git + $ cd wireguard-linux-compat + $ git checkout debian/1.0.20200712-1_bpo10+1 + +We again make use of a helper script and some patches to make the build work. +Just run the following command: + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel + $ ./build-wireguard-modules.sh + I: Apply WireGuard patch: /vyos/packages/linux-kernel/patches/wireguard-linux-compat/0001-Debian-build-wireguard-modules-package.patch + patching file debian/control + patching file debian/rules + I: Build Debian WireGuard package + dpkg-buildpackage: info: source package wireguard-linux-compat + dpkg-buildpackage: info: source version 1.0.20200712-1~bpo10+1 + dpkg-buildpackage: info: source distribution buster-backports + dpkg-buildpackage: info: source changed by Unit 193 <unit193@debian.org> + dpkg-buildpackage: info: host architecture amd64 + dpkg-source --before-build . + dpkg-source: info: using patch list from debian/patches/series + dpkg-source: info: applying 0001-Makefile-do-not-use-git-to-get-version-number.patch + dpkg-source: info: applying 0002-Avoid-trying-to-compile-on-debian-5.5-kernels-Closes.patch + + ... + + dpkg-genchanges: info: binary-only upload (no source code included) + debian/rules clean + dh clean + dh_clean + dpkg-source --after-build . + dpkg-source: info: unapplying 0002-Avoid-trying-to-compile-on-debian-5.5-kernels-Closes.patch + dpkg-source: info: unapplying 0001-Makefile-do-not-use-git-to-get-version-number.patch + dpkg-buildpackage: info: binary-only upload (no source included) + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them +to the ``vyos-build/packages`` folder for inclusion during the ISO build. + +Accel-PPP +^^^^^^^^^ + +First clone the source code and checkout the appropriate version by running: + +.. code-block:: none + + $ cd vyos-build/packages/linux-kernel + $ git clone https://github.com/accel-ppp/accel-ppp.git + +We again make use of a helper script and some patches to make the build work. +Just run the following command: + +.. code-block:: none + + $ ./build-accel-ppp.sh + I: Build Accel-PPP Debian package + CMake Deprecation Warning at CMakeLists.txt:3 (cmake_policy): + The OLD behavior for policy CMP0003 will be removed from a future version + of CMake. + + The cmake-policies(7) manual explains that the OLD behaviors of all + policies are deprecated and that a policy should be set to OLD only under + specific short-term circumstances. Projects should be ported to the NEW + behavior and not rely on setting a policy to OLD. + + -- The C compiler identification is GNU 8.3.0 + + ... + + CPack: Create package using DEB + CPack: Install projects + CPack: - Run preinstall target for: accel-ppp + CPack: - Install project: accel-ppp + CPack: Create package + CPack: - package: /vyos/vyos-build/packages/linux-kernel/accel-ppp/build/accel-ppp.deb generated. + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them +to the ``vyos-build/packages`` folder for inclusion during the ISO build. + +Intel NIC +^^^^^^^^^ + +The Intel NIC drivers do not come from a Git repository, instead we just fetch +the tarballs from our mirror and compile them. + +Simply use our wrapper script to build all of the driver modules. + +.. code-block:: none + + ./build-intel-drivers.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 490k 100 490k 0 0 648k 0 --:--:-- --:--:-- --:--:-- 648k + I: Compile Kernel module for Intel ixgbe driver + + ... + + I: Building Debian package vyos-intel-iavf + Doing `require 'backports'` is deprecated and will not load any backport in the next major release. + Require just the needed backports instead, or 'backports/latest'. + Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} + Created package {:path=>"vyos-intel-iavf_4.0.1-0_amd64.deb"} + I: Cleanup iavf source + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them +to the ``vyos-build/packages`` folder for inclusion during the ISO build. + +Intel QAT +^^^^^^^^^ +The Intel QAT (Quick Assist Technology) drivers do not come from a Git +repository, instead we just fetch the tarballs from 01.org, Intels Open-Source +website. + +Simply use our wrapper script to build all of the driver modules. + +.. code-block:: none + + $ ./build-intel-qat.sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 5065k 100 5065k 0 0 1157k 0 0:00:04 0:00:04 --:--:-- 1157k + I: Compile Kernel module for Intel qat driver + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + checking for a thread-safe mkdir -p... /bin/mkdir -p + checking for gawk... gawk + checking whether make sets $(MAKE)... yes + + ... + + I: Building Debian package vyos-intel-qat + Doing `require 'backports'` is deprecated and will not load any backport in the next major release. + Require just the needed backports instead, or 'backports/latest'. + Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} + Created package {:path=>"vyos-intel-qat_1.7.l.4.9.0-00008-0_amd64.deb"} + I: Cleanup qat source + + +After compiling the packages you will find yourself the newly generated `*.deb` +binaries in ``vyos-build/packages/linux-kernel`` from which you can copy them +to the ``vyos-build/packages`` folder for inclusion during the ISO build. + + Packages ======== diff --git a/docs/contributing/development.rst b/docs/contributing/development.rst index 17f5cc48..86371845 100644 --- a/docs/contributing/development.rst +++ b/docs/contributing/development.rst @@ -271,7 +271,7 @@ device if you happen to be a crazy scientist. #!/usr/bin/env python3 # - # Copyright (C) 2019 VyOS maintainers and contributors + # Copyright (C) 2020 VyOS maintainers and contributors # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 or later as @@ -291,10 +291,16 @@ device if you happen to be a crazy scientist. from vyos import ConfigError def get_config(): - vc = Config() + if config: + conf = config + else: + conf = Config() + + # Base path to CLI nodes + base = ['...', '...'] # Convert the VyOS config to an abstract internal representation - config = ... - return config + config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) + return config_data def verify(config): # Verify that configuration is valid @@ -311,8 +317,10 @@ device if you happen to be a crazy scientist. pass try: - config = get_config() - verify(config) + c = get_config() + verify(c) + generate(c) + apply(c) except ConfigError as e: print(e) sys.exit(1) diff --git a/docs/contributing/upstream-packages.rst b/docs/contributing/upstream-packages.rst index d43e61f3..5c48bbb3 100644 --- a/docs/contributing/upstream-packages.rst +++ b/docs/contributing/upstream-packages.rst @@ -59,26 +59,6 @@ reason we debianize that module by hand now, using this procedure: The package ends up in deb_dist dir. -ppp -^^^ - -Properly renaming PPTP and L2TP interfaces to pptpX and l2tpX from generic and -non-informative pppX requires a patch that is neither in the upstream nor in -Debian. - -We keep a fork of Debian's repo at https://github.com/vyos/ppp-debian - -The patches for pre-up renaming are: - -* https://github.com/vyos/ppp-debian/commit/e728180026a051d2a96396276e7e4ae -* https://github.com/vyos/ppp-debian/commit/f29ba8d9ebb043335a096d70bcd07e9 - -Additionally, there's a patch for reopening the log file to better support -logging to files, even though it's less essential: -https://github.com/vyos/ppp-debian/commit/dd2ebd5cdcddb40230dc4cc43d374055f - -The patches were written by Stephen Hemminger back in the Vyatta times. - mdns-repeater ^^^^^^^^^^^^^ @@ -95,58 +75,6 @@ https://github.com/vyos/udp-broadcast-relay No special build procedure is required. -Linux kernel -^^^^^^^^^^^^ - -In the past a fork of the Kernel source code was kept at the well-known -location of https://github.com/vyos/vyos-kernel - where it is kept for history. - -Nowadays the Kernel we use is the upstream source code which is patched -with two additional patches from the good old Vyatta times which never made it -into the mainstream Kernel. The patches can be found here: -https://github.com/vyos/vyos-build-kernel/tree/master/patches/kernel and are -automatically applied to the Kernel by the Jenkins Pipeline which is used to -generate the Kernel binaries. - -The Pipeline script not only builds the Kernel with the configuration named -``x86_64_vyos_defconfig`` which is located in the vyos-build-kernel repository, -too - but in addition also builds some Intel out-of-tree drivers, WireGuard -(as long it is not upstreamed) and Accel-PPP. - -The ``Jenkinsfile`` tries to be as verbose as possible on each individual build -step. - -Linux Firmware -^^^^^^^^^^^^^^ - -More and more hardware cards require an additional firmware which is not open -source. The Kernel community hosts a special linux-firmware Git repository -with all available binary files which can be loaded by the Kernel. - -The ``vyos-build`` repository fetches a specific commit of the linux-firmware -repository and embeds those binaries into the resulting ISO image. This step is -done in the ``data/live-build-config/hooks/live/40-linux-firmware.chroot`` file. - -If the firmware needs to be updated it is sufficient to just exchange the Git -commit id we reference in our build. - -Intel NIC drivers -^^^^^^^^^^^^^^^^^ - -We do not make use of the building Intel NIC drivers except for e1000e. Main -reason is that the out of tree Intel drivers seem be perform a bit better, -e.q. have proper receive-side-scaling and multi-queue support. - -Drivers are build as part of the Kernel Pipeline - read above. - -Accel-PPP -^^^^^^^^^ - -Accel-PPP used to be an upstream fork for quite some time but now has been -converted to make use of the upstream source code and build system. - -It is build as part of the Kernel Pipeline - read above. - hvinfo ^^^^^^ |