diff options
| -rw-r--r-- | CONTRIBUTING.md | 12 | ||||
| -rw-r--r-- | README.md | 215 | ||||
| -rwxr-xr-x | scripts/build-packages | 370 |
3 files changed, 17 insertions, 580 deletions
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..d8177a5f --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,12 @@ +# Contributing to VyOS + +You wan't to help us improve VyOS? This is awesome. We accept any kind of Pull +Requests on GitHub. To make the life of the maintainers and you as future +contributor (or maybe maintainer) much easier we have come up with some basic +rules. Instead of copy/pasting or maintaining two instances of how to contribute +to VyOS you can find the entire process documented in our online documentation: +https://docs.vyos.io/en/latest/contributing/development.html + +Also this guide might not be complete so any PR is much appreciated. + +It might also worth browsing our blog: https://blog.vyos.io @@ -47,218 +47,13 @@ There are several directories with their own purpose: purposes, but not during ISO build process * `vars/` Jenkins Pipeline library for reusable functions -# Building images using Docker +# Building VyOS -Using our [Dockerfile](docker/Dockerfile) you create your own Docker container -that is used to build a VyOS ISO image or other required VyOS packages. The -[Dockerfile](docker/Dockerfile) contains some of the most used packages needed -to build a VyOS ISO, a qemu image, and several of the submodules. +In order to have a single manual and not maintining multiple copies the +instructions on how to build VyOS either in a Docker container or natively can +be found in our [Documentation - Build VyOS](https://docs.vyos.io/en/latest/contributing/build-vyos.html). -To build the docker image ensure you have a working [Docker](https://www.docker.com) -environment and then run the following commands: - -```bash -$ docker build -t vyos-builder docker -``` - -Run newly built container: -```bash -$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos-builder bash -``` - -This will drop you into a bash shell with this vyos-build repo mounted at -`/vyos`. Then follow the instructions bellow to build the VyOS ISO and QEMU -image. - -```bash -vyos_bld@948a2be7c52c:/vyos$ uname -a -Linux 948a2be7c52c 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03) x86_64 GNU/Linux -``` - -**NOTE:** - -* Docker container must be run with `--privileged` flag -* We recommended to run the container with a volume mapped in order to easy - export built VyOS ISO images to the "external" world -* UNIX ownership is automatically inherited from your host directory but can be - altered by specifying the following environment variables when running the - container: `-e GOSU_UID=$(id -u)` and/or `-e GOSU_GID=$(id -g)` - -After the Docker container is running you can follow up the instructions below in -order to build the VyOS ISO image. - -## Building the ISO image - -The `./configure` script will warn you if any dependencies are missing. Individual -packages may have other build dependencies. If some dependencies are missing, -package build scripts will tell you. - -```bash -$ ./configure --help -usage: configure [-h] [--build-by BUILD_BY] [--version VERSION] - [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] - [--debian-security-mirror DEBIAN_SECURITY_MIRROR] - [--architecture ARCHITECTURE] [--vyos-mirror VYOS_MIRROR] - [--build-type BUILD_TYPE] [--debian-mirror DEBIAN_MIRROR] - [--debug] [--custom-apt-entry CUSTOM_APT_ENTRY] - [--custom-apt-key CUSTOM_APT_KEY] - [--custom-package CUSTOM_PACKAGE] - -optional arguments: - -h, --help show this help message and exit - --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) - --version VERSION Version number (release builds only) - --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR - Debian repository mirror for pbuilder env bootstrap - --debian-security-mirror DEBIAN_SECURITY_MIRROR - Debian security updated mirror - --architecture ARCHITECTURE - Image target architecture (amd64 or i386 or armhf) - --vyos-mirror VYOS_MIRROR - VyOS package mirror - --build-type BUILD_TYPE - Build type, release or development - --debian-mirror DEBIAN_MIRROR - Debian repository mirror for ISO build - --debug Enable debug output - --custom-apt-entry CUSTOM_APT_ENTRY - Custom APT entry - --custom-apt-key CUSTOM_APT_KEY - Custom APT key file - --custom-package CUSTOM_PACKAGE - Custom package to install from repositories -``` - -Before you can build an image, you need to configure your build. - -Each build needs to run the `./configure` step first where you can extend your -ISO by additional packages (`--custom-package`) or specify who build this nice -ISO image (`--build-by`). If you have local Debian mirrors, you can select them -by `--debian-mirror` or `--debian-security-mirror`. - -```bash -$ ./configure --custom-package vim --build-by jrandom@hacker.com -$ sudo make iso -``` - -After some time you will find the resulting ISO image in the `build` folder. - -### Building images for virtualization platforms - -#### QEMU - -Run following command after building the ISO image. - -```bash -$ make qemu -``` - -#### VMware - -Run following command after building the QEMU image. - -```bash -$ make vmware -``` - -## Building subpackages inside Docker - -VyOS requires a bunch of packages which are VyOS specific and thus can not be -found in any Debian Upstream mirrror. Those packages can be found at the VyOS -GitHub project (https://github.com/vyos) and there is a nice helper script -available to build and list those individual packages. - -[scripts/build-packages](scripts/build-packages) provides an easy interface -to automate the process of building all VyOS related packages that are not part -of the upstream Debian version. Execute it in the root of the `vyos-build` -directory to start compilation. - -```bash -$ scripts/build-packages -h -usage: build-packages [-h] [-c | -k | -f] [-v] [-l] [-b BUILD [BUILD ...]] - [-p] [--blacklist BLACKLIST [BLACKLIST ...]] - -optional arguments: - -h, --help show this help message and exit - -c, --clean Re-clone required Git repositories - -k, --keep Keep modified Git repositories - -f, --fetch Fetch sources only, no build - -v, --verbose Increase logging verbosity for each occurance - -l, --list-packages List all packages to build - -b BUILD [BUILD ...], --build BUILD [BUILD ...] - Whitespace separated list of packages to build - -p, --parallel Build on all CPUs - --blacklist BLACKLIST [BLACKLIST ...] - Do not build/report packages when calling --list -``` - -Git repositories are automatically fetched and build on demand. If you want to -work offline you can fetch all source code first with the `-f` option. - -The easiest way to compile is with the above mentioned [Docker](docker/Dockerfile) -container, it includes all dependencies for compiling supported packages. - -```bash -$ docker run --rm -it -v $(pwd):/vyos -w /vyos \ - --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - vyos-builder scripts/build-packages -``` - -**NOTE:** `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is required to build the -`vyos-strongswan` package - -**NOTE:** Prior to executing this script you need to create or build the Docker -container and checkout all packages you want to compile. - -### Building single package(s) - -To build a single package use the same script as above but specify packages with `-b`: - -Executed from the root of `vyos-build` - -```bash -$ docker run --rm -it -v $(pwd):/vyos -w /vyos \ - --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - vyos-builder scripts/build-packages -b <package> -``` - -**NOTE:** `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is only needed when -building `vyos-strongswan` and can be ignored on other packages. - -**NOTE:** `vyos-strongswan` will only compile on a Linux system, running on macOS -or Windows might result in a unittest deadlock (it never exits). - -#### Building single packages from your own repositories - -You can also build packages that are not from the default git repositories, -for example from your own forks of the official vyos repositories. - -First create a directory "packages" at the top level of the vyos-build repository -and clone your package into it (creating a subdirectory with the package contents). -Then checkout the correct branch or commit you want to build before building the package. - -Example using `git@github.com:myname/vyos-1x.git` repository to build vyos-1x: - -```bash -$ mkdir packages -$ cd packages -$ git clone git@github.com:myname/vyos-1x.git -$ cd .. -$ docker run --rm -it -v $(pwd):/vyos -w /vyos \ - --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - vyos-builder scripts/build-packages -b vyos-1x -``` - -**NOTE:** You need to git pull manually after you commit to the remote and before rebuilding, -the local repository won't be updated automatically. - -**WARNING:** Any packages in the packages directory will be added to the iso during build, -replacing the upstream ones. Make sure you delete them (both the source directories and built -deb packages) if you want to build an iso from purely upstream packages. - -# Development process - -## Git branches +# Development Branches The default branch that contains the most recent VyOS code is called `current` rather than `master`. We know it's confusing, but it's not easy to fix. In a diff --git a/scripts/build-packages b/scripts/build-packages deleted file mode 100755 index df7b8ed1..00000000 --- a/scripts/build-packages +++ /dev/null @@ -1,370 +0,0 @@ -#!/usr/bin/env python3 - -import os -import subprocess -import sys -import shutil -import argparse -import logging - -current_working_directory = os.getcwd() -repo_root = subprocess.check_output('git rev-parse --show-toplevel', shell=True, universal_newlines=True).rstrip('\n') -repo_sha = subprocess.check_output('git rev-parse --short=12 HEAD', shell=True, universal_newlines=True).rstrip('\n') - -def add_package(name, url=None, commit=None, branch='current', tag=None, shallow=True, custombuild_cmd=None): - """ - Build up source package with URL and build commands executed during the later - called build_package step. - - If no additional information is passed we will use the latest commit from current - branch - - If no URL is passed we assume it's a regular VyOS package from the VyOS Github - namespace at https://github.com/vyos - """ - - if not url: - url = 'https://github.com/vyos/' + name + '.git' - - package = { - 'name': name, - 'url': url, - 'commit': commit, - 'tag': tag, - 'branch': branch, - 'shallow': shallow, - 'path': repo_root + '/packages/' + name, - 'custombuild_cmd': custombuild_cmd - } - - return package - - - -def call(bashCommand, log, **kwargs): - """ - Run command with arguments. Wait for command to complete. - - Send output to logging module passed as 'log'. - """ - - from subprocess import Popen, PIPE, STDOUT, check_output, CalledProcessError - from tempfile import TemporaryFile - from time import sleep - - log.debug("Executing '{}'".format(bashCommand)) - - # code borrowsed from: - # https://stackoverflow.com/questions/38374063/python-can-we-use-tempfile-with-subprocess-to-get-non-buffering-live-output-in-p - # the temp file will be automatically cleaned up - output = TemporaryFile() - error = TemporaryFile() - - kwargs['stdout'] = output - kwargs['stderr'] = error - kwargs['shell'] = True - kwargs['universal_newlines'] = True - - sub = Popen(bashCommand, **kwargs) - while sub.poll() is None: - where = output.tell() - lines = output.readline() - if not lines: - sleep(0.3) - output.seek(where) - else: - log.debug(lines.decode().rstrip('\n')) - - where = error.tell() - lines = error.readline() - if not lines: - sleep(0.3) - error.seek(where) - else: - log.info(lines.decode().rstrip('\n')) - - - # A last write needed after subprocess ends - log.debug(output.read().decode().rstrip('\n')) - log.info(error.read().decode().rstrip('\n')) - - error.close() - output.close() - return sub.returncode - -def clone_package(pkg, log): - """ - Clone Git repository from URL embedded in pkg to local disk - - First cleanup any possible leftovers from previous builds - """ - - if args.keep and os.path.isdir(pkg['path']): - log.debug("Keep possibly modified package '{}'".format(pkg['path'])) - return False - elif args.clean: - # delete repository from disk - if os.path.isdir(pkg['path']): - log.debug("Cleaning '{}'".format(pkg['path'])) - shutil.rmtree(pkg['path']) - else: - if os.path.isdir(pkg['path']): - # Change current directory into Git repo for this package - os.chdir(pkg['path']) - - bashCommand = 'git clean -d -x --force && git reset --hard ' + pkg['commit'] - return call(bashCommand, log) - - if pkg['commit'] and pkg['shallow']: - log.debug("Package '{}' has both commit and shallow set, unsetting shallow to do a full clone.".format(pkg['name'])) - pkg['shallow']=False - - bashCommand = 'git clone ' + pkg['url'] - - if pkg['shallow']: - bashCommand += ' --depth 1' - - if pkg['tag']: - bashCommand += ' --branch ' + pkg['tag'] - elif pkg['branch']: - bashCommand += ' --branch ' + pkg['branch'] - - bashCommand += ' ' + pkg['path'] - - if pkg['commit']: - bashCommand += ' && cd ' + pkg['path'] + ' && git reset --hard ' + pkg['commit'] - - return call(bashCommand, log) - - -def build_package(pkg, log=None): - """ - Generate Debian package from passed 'pkg' - """ - - # Change current directory into Git repo for this package - os.chdir(pkg['path']) - - # Overwrite custom build command if required, e.g. libyang - bashCommand = '' - if pkg['custombuild_cmd']: - bashCommand = pkg['custombuild_cmd'] - else: - # Build package - bashCommand = 'dpkg-buildpackage -uc -us -tc -b' - if args.parallel: - bashCommand += ' -j' + str(os.cpu_count()) - - return call(bashCommand, log) - -# a List of all Vyatta/VyOS based packages -vyos_packages = ['vyatta-bash', - 'vyatta-cfg', - 'vyatta-op', - 'vyatta-cfg-system', - 'vyatta-cfg-firewall', - 'vyatta-op-firewall', - 'vyatta-cfg-vpn', - 'vyatta-op-vpn', - 'vyatta-cfg-qos', - 'vyatta-op-qos', - 'vyatta-cfg-op-pppoe', - 'vyatta-nat', - 'vyatta-config-mgmt', - 'vyatta-config-migrate', - 'vyatta-zone', - 'vyatta-cluster', - 'vyatta-eventwatch', - 'vyatta-webproxy', - 'vyatta-cfg-quagga', - 'vyatta-op-quagga', - 'vyatta-wanloadbalance', - 'vyatta-ravpn', - 'vyos-nhrp', - 'vyos-world', - 'vyatta-iproute', - 'vyatta-biosdevname', - 'vyos-opennhrp', - 'udp-broadcast-relay', - 'vyos-1x', - 'vyatta-conntrack', - 'vyatta-conntrack-sync', - 'vyos-xe-guest-utilities', - 'vyos-netplug', - 'ddclient', - 'live-boot'] - -# Special packages mean packages which are located no in the VyOS namespace -# or require fancy build instructions -pkg_special = [] - -# libvyosconfig/ipaddrcheck uses a different default branch -libvyosconfig_build_cmd = "eval $(opam env --root=/opt/opam --set-root) && " \ - "dpkg-buildpackage -b -us -uc -tc" -pkg_special.append( add_package('libvyosconfig', branch='master', custombuild_cmd=libvyosconfig_build_cmd)) -pkg_special.append( add_package('ipaddrcheck', branch='master')) - -# Packages where we directly build the upstream source -pkg_special.append( add_package('hvinfo', url='https://github.com/dmbaturin/hvinfo.git', branch='master') ) - -# VyOS strongswan ships additional python3-vici packages required by vyos-1x and this is not build by default -vyos_strongswan_build_cmd = "dpkg-buildpackage -b -us -uc -tc && " \ - "autoreconf -i && ./configure --enable-python-eggs && " \ - "cd src/libcharon/plugins/vici/python && make && " \ - "python3 setup.py --command-packages=stdeb.command bdist_deb && " \ - "mv ./deb_dist/*.deb " + repo_root + "/packages" -pkg_special.append( add_package('vyos-strongswan', custombuild_cmd=vyos_strongswan_build_cmd) ) - -# -# FreeRangeRouting (FRR) packages -# -pkg_special.append( add_package('rtrlib', url='https://github.com/rtrlib/rtrlib.git', branch='master', tag='v0.6.3') ) - -frr_build_cmd = './tools/tarsource.sh -V && dpkg-buildpackage -us -uc -Ppkg.frr.rtrlib -d' -pkg_special.append( add_package('frr', url='https://github.com/FRRouting/frr.git', branch='master', tag='frr-7.0', custombuild_cmd=frr_build_cmd) ) -#libyang_build_cmd = 'mkdir build && cd build && cmake .. && make build-deb && mv debs/* ' + repo_root + '/packages' -#pkg_special.append( add_package('libyang', url='https://github.com/opensourcerouting/libyang.git', commit='179da47', branch='master', custombuild_cmd=libyang_build_cmd) ) - - -# -# We use keepalived from Debian Buster -# -pkg_special.append( add_package('keepalived', url='https://salsa.debian.org/ipvs-team/pkg-keepalived.git', branch='master', commit='eae91c81')) - -# -# Linux (VyOS) Kernel -# -kernel_build_cmd = "make x86_64_vyos_defconfig && " \ - "sed -i 's/\"kernel_version\": \"[0-9].[0-9][0-9].[0-9]*\"/\"kernel_version\": \"'$(make kernelversion)'\"/' " + repo_root + "/data/defaults.json && " \ - "make bindeb-pkg LOCALVERSION='-amd64-vyos' KDEB_PKGVERSION=$(make kernelversion)-1 -j $(getconf _NPROCESSORS_ONLN)" -pkg_special.append( add_package('vyos-kernel', branch='linux-vyos-4.19.y', custombuild_cmd=kernel_build_cmd) ) - - - -# -# Accell-PPP Package and Kernel Module -# -accel_ppp_build_cmd = "echo 'lib/modules/'$(cat " + repo_root + "/data/defaults.json | jq '.kernel_version' | tr -d \\\")-amd64-vyos/extra/*.ko > debian/vyos-accel-ppp-ipoe-kmod.install && " \ - "sed -i 's#[0-9].[0-9][0-9].[0-9]*-amd64-vyos#'$(cat " + repo_root + "/data/defaults.json | jq '.kernel_version' | tr -d \\\")'-amd64-vyos#g' debian/rules && " \ - "KERNELDIR=" + repo_root + "/packages/vyos-kernel dpkg-buildpackage -b -us -uc -tc -j" + str(os.cpu_count()) -pkg_special.append( add_package('vyos-accel-ppp', custombuild_cmd=accel_ppp_build_cmd) ) - -# -# mdns-repeater branch for current is master -# - -pkg_special.append( add_package('mdns-repeater', branch='master') ) - - -# A list of all packages we will build in the end -pkg_build = [] - - - - -if __name__ == '__main__': - parser = argparse.ArgumentParser() - exclusive = parser.add_mutually_exclusive_group(required=False) - exclusive.add_argument('-c', '--clean', action='store_true', help='Re-clone required Git repositories') - exclusive.add_argument('-k', '--keep', action='store_true', help='Keep modified Git repositories') - exclusive.add_argument('-f', '--fetch', action='store_true', help='Fetch sources only, no build') - - parser.add_argument('-v', '--verbose', action='count', default=0, help='Increase logging verbosity for each occurance') - parser.add_argument('-l', '--list-packages', action='store_true', help='List all packages to build') - parser.add_argument('-b', '--build', nargs='+', help='Whitespace separated list of packages to build') - parser.add_argument('-p', '--parallel', action='store_true', help='Build on all CPUs') - parser.add_argument('--blacklist', nargs='+', help='Do not build/report packages when calling --list') - - - args = parser.parse_args() - - levels = [ logging.INFO, logging.WARNING, logging.DEBUG ] - level = levels[min(len(levels)-1,args.verbose)] # capped to number of levels - logging.basicConfig(level=level, format="%(asctime)s %(name)s %(message)s") - - print("Using vyos-build repository ('{}') commit '{}'\n".format(repo_root, repo_sha)) - - # - # Exclude packages from build process, - # also do not list them on --list - # - if args.blacklist: - for exclude in args.blacklist: - if exclude in vyos_packages: - vyos_packages.remove(exclude) - continue - - found = False - for pkg in pkg_special: - if exclude == pkg['name']: - found = True - # package already formed - pkg_special.remove(pkg) - break - - if not found: - print("Invalid choice '" + exclude + "', --list-packages for complete list!") - sys.exit(1) - - # - # List all available (to be build) packages - # - if args.list_packages: - print("Individual packages available for build:") - for pkg in vyos_packages: - print(' * ' + pkg) - for pkg in pkg_special: - print(' * ' + pkg['name']) - - sys.exit(0) - - # - # Only add selective packages to the build list - # - if args.build: - # NOTE: remove double added packages from list - for target in args.build: - if target in vyos_packages: - pkg_build.append(add_package( target )) - continue - - found = False - for pkg in pkg_special: - if target == pkg['name']: - found = True - # package already formed - pkg_build.append( pkg ) - break - - if not found: - print("Invalid choice '" + target + "', for -b/--build use --list-packages for complete list!") - sys.exit(1) - - else: - # Add all VyOS packages to the package list - for pkg in vyos_packages: - pkg_build.append(add_package( pkg )) - - # We also wan't to build all of our special packages - for pkg in pkg_special: - pkg_build.append( pkg ) - - # Build all VyOS packages (packages found on https://github.com/vyos - # and referenced in vyos_packages) - for pkg in pkg_build: - # Create a logging instance per package - log = logging.getLogger(pkg['name']) - - ret = clone_package(pkg, log) - if ret: - log.error("ERROR cloning source") - sys.exit(1) - else: - # only build packages if fetch flag is not set - if not args.fetch: - ret = build_package(pkg, log) - if ret: - log.error("ERROR building source") - sys.exit(1) - - sys.exit(0) |