build-packages: remove obsolete script and link to VyOS documentation

The referenced scripts/build-packages script was a proof of concept from me
some time ago to build VyOS packages through a more or less tiny wrapper. The
script was not maintained at any time and randomly caused more harm then good.

The script has been abandoned and the documentation has been updated on how to
build individual VyOS packages.

3 files changed, 17 insertions, 580 deletions

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 00000000..d8177a5f
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,12 @@
+# Contributing to VyOS
+
+You wan't to help us improve VyOS? This is awesome. We accept any kind of Pull
+Requests on GitHub. To make the life of the maintainers and you as future
+contributor (or maybe maintainer) much easier we have come up with some basic
+rules. Instead of copy/pasting or maintaining two instances of how to contribute
+to VyOS you can find the entire process documented in our online documentation:
+https://docs.vyos.io/en/latest/contributing/development.html
+
+Also this guide might not be complete so any PR is much appreciated.
+
+It might also worth browsing our blog: https://blog.vyos.io
diff --git a/README.md b/README.md
index 88767431..c41847de 100644
--- a/README.md
+++ b/README.md
@@ -47,218 +47,13 @@ There are several directories with their own purpose:
                purposes, but not during ISO build process
  * `vars/`     Jenkins Pipeline library for reusable functions
 
-# Building images using Docker
+# Building VyOS
 
-Using our [Dockerfile](docker/Dockerfile) you create your own Docker container
-that is used to build a VyOS ISO image or other required VyOS packages. The
-[Dockerfile](docker/Dockerfile) contains some of the most used packages needed
-to build a VyOS ISO, a qemu image, and several of the submodules.
+In order to have a single manual and not maintining multiple copies the
+instructions on how to build VyOS either in a Docker container or natively can
+be found in our [Documentation - Build VyOS](https://docs.vyos.io/en/latest/contributing/build-vyos.html).
 
-To build the docker image ensure you have a working [Docker](https://www.docker.com)
-environment and then run the following commands:
-
-```bash
-$ docker build -t vyos-builder docker
-```
-
-Run newly built container:
-```bash
-$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos-builder bash
-```
-
-This will drop you into a bash shell with this vyos-build repo mounted at
-`/vyos`. Then follow the instructions bellow to build the VyOS ISO and QEMU
-image.
-
-```bash
-vyos_bld@948a2be7c52c:/vyos$ uname -a
-Linux 948a2be7c52c 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03) x86_64 GNU/Linux
-```
-
-**NOTE:**
-
-* Docker container must be run with `--privileged` flag
-* We recommended to run the container with a volume mapped in order to easy
-  export built VyOS ISO images to the "external" world
-* UNIX ownership is automatically inherited from your host directory but can be
-  altered by specifying the following environment variables when running the
-  container: `-e GOSU_UID=$(id -u)` and/or `-e GOSU_GID=$(id -g)`
-
-After the Docker container is running you can follow up the instructions below in
-order to build the VyOS ISO image.
-
-## Building the ISO image
-
-The `./configure` script will warn you if any dependencies are missing. Individual
-packages may have other build dependencies. If some dependencies are missing,
-package build scripts will tell you.
-
-```bash
-$ ./configure --help
-usage: configure [-h] [--build-by BUILD_BY] [--version VERSION]
-                 [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR]
-                 [--debian-security-mirror DEBIAN_SECURITY_MIRROR]
-                 [--architecture ARCHITECTURE] [--vyos-mirror VYOS_MIRROR]
-                 [--build-type BUILD_TYPE] [--debian-mirror DEBIAN_MIRROR]
-                 [--debug] [--custom-apt-entry CUSTOM_APT_ENTRY]
-                 [--custom-apt-key CUSTOM_APT_KEY]
-                 [--custom-package CUSTOM_PACKAGE]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --build-by BUILD_BY   Builder identifier (e.g. jrandomhacker@example.net)
-  --version VERSION     Version number (release builds only)
-  --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR
-                        Debian repository mirror for pbuilder env bootstrap
-  --debian-security-mirror DEBIAN_SECURITY_MIRROR
-                        Debian security updated mirror
-  --architecture ARCHITECTURE
-                        Image target architecture (amd64 or i386 or armhf)
-  --vyos-mirror VYOS_MIRROR
-                        VyOS package mirror
-  --build-type BUILD_TYPE
-                        Build type, release or development
-  --debian-mirror DEBIAN_MIRROR
-                        Debian repository mirror for ISO build
-  --debug               Enable debug output
-  --custom-apt-entry CUSTOM_APT_ENTRY
-                        Custom APT entry
-  --custom-apt-key CUSTOM_APT_KEY
-                        Custom APT key file
-  --custom-package CUSTOM_PACKAGE
-                        Custom package to install from repositories
-```
-
-Before you can build an image, you need to configure your build.
-
-Each build needs to run the `./configure` step first where you can extend your
-ISO by additional packages (`--custom-package`) or specify who build this nice
-ISO image (`--build-by`). If you have local Debian mirrors, you can select them
-by `--debian-mirror` or `--debian-security-mirror`.
-
-```bash
-$ ./configure --custom-package vim --build-by jrandom@hacker.com
-$ sudo make iso
-```
-
-After some time you will find the resulting ISO image in the `build` folder.
-
-### Building images for virtualization platforms
-
-#### QEMU
-
-Run following command after building the ISO image.
-
-```bash
-$ make qemu
-```
-
-#### VMware
-
-Run following command after building the QEMU image.
-
-```bash
-$ make vmware
-```
-
-## Building subpackages inside Docker
-
-VyOS requires a bunch of packages which are VyOS specific and thus can not be
-found in any Debian Upstream mirrror. Those packages can be found at the VyOS
-GitHub project (https://github.com/vyos) and there is a nice helper script
-available to build and list those individual packages.
-
-[scripts/build-packages](scripts/build-packages) provides an easy interface
-to automate the process of building all VyOS related packages that are not part
-of the upstream Debian version. Execute it in the root of the `vyos-build`
-directory to start compilation.
-
-```bash
-$  scripts/build-packages -h
-usage: build-packages [-h] [-c | -k | -f] [-v] [-l] [-b BUILD [BUILD ...]]
-                      [-p] [--blacklist BLACKLIST [BLACKLIST ...]]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  -c, --clean           Re-clone required Git repositories
-  -k, --keep            Keep modified Git repositories
-  -f, --fetch           Fetch sources only, no build
-  -v, --verbose         Increase logging verbosity for each occurance
-  -l, --list-packages   List all packages to build
-  -b BUILD [BUILD ...], --build BUILD [BUILD ...]
-                        Whitespace separated list of packages to build
-  -p, --parallel        Build on all CPUs
-  --blacklist BLACKLIST [BLACKLIST ...]
-                        Do not build/report packages when calling --list
-```
-
-Git repositories are automatically fetched and build on demand. If you want to
-work offline you can fetch all source code first with the `-f` option.
-
-The easiest way to compile is with the above mentioned [Docker](docker/Dockerfile)
-container, it includes all dependencies for compiling supported packages.
-
-```bash
-$ docker run --rm -it -v $(pwd):/vyos -w /vyos \
-             --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
-             vyos-builder scripts/build-packages
-```
-
-**NOTE:** `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is required to build the
-`vyos-strongswan` package
-
-**NOTE:** Prior to executing this script you need to create or build the Docker
-container and checkout all packages you want to compile.
-
-### Building single package(s)
-
-To build a single package use the same script as above but specify packages with `-b`:
-
-Executed from the root of `vyos-build`
-
-```bash
-$ docker run --rm -it -v $(pwd):/vyos -w /vyos \
-             --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
-             vyos-builder scripts/build-packages -b <package>
-```
-
-**NOTE:** `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is only needed when
-building `vyos-strongswan` and can be ignored on other packages.
-
-**NOTE:** `vyos-strongswan` will only compile on a Linux system, running on macOS
-or Windows might result in a unittest deadlock (it never exits).
-
-#### Building single packages from your own repositories
-
-You can also build packages that are not from the default git repositories,
-for example from your own forks of the official vyos repositories.
-
-First create a directory "packages" at the top level of the vyos-build repository
-and clone your package into it (creating a subdirectory with the package contents).
-Then checkout the correct branch or commit you want to build before building the package.
-
-Example using `git@github.com:myname/vyos-1x.git` repository to build vyos-1x:
-
-```bash
-$ mkdir packages
-$ cd packages
-$ git clone git@github.com:myname/vyos-1x.git
-$ cd ..
-$ docker run --rm -it -v $(pwd):/vyos -w /vyos \
-             --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
-             vyos-builder scripts/build-packages -b vyos-1x
-```
-
-**NOTE:** You need to git pull manually after you commit to the remote and before rebuilding,
-the local repository won't be updated automatically.
-
-**WARNING:** Any packages in the packages directory will be added to the iso during build,
-replacing the upstream ones. Make sure you delete them (both the source directories and built
-deb packages) if you want to build an iso from purely upstream packages.
-
-# Development process
-
-## Git branches
+# Development Branches
 
 The default branch that contains the most recent VyOS code is called `current`
 rather than `master`. We know it's confusing, but it's not easy to fix. In a
diff --git a/scripts/build-packages b/scripts/build-packages
deleted file mode 100755
index df7b8ed1..00000000
--- a/scripts/build-packages
+++ /dev/null
@@ -1,370 +0,0 @@
-#!/usr/bin/env python3
-
-import os
-import subprocess
-import sys
-import shutil
-import argparse
-import logging
-
-current_working_directory = os.getcwd()
-repo_root = subprocess.check_output('git rev-parse --show-toplevel', shell=True, universal_newlines=True).rstrip('\n')
-repo_sha  = subprocess.check_output('git rev-parse --short=12 HEAD', shell=True, universal_newlines=True).rstrip('\n')
-
-def add_package(name, url=None, commit=None, branch='current', tag=None, shallow=True, custombuild_cmd=None):
-    """
-    Build up source package with URL and build commands executed during the later
-    called build_package step.
-
-    If no additional information is passed we will use the latest commit from current
-    branch
-
-    If no URL is passed we assume it's a regular VyOS package from the VyOS Github
-    namespace at https://github.com/vyos
-    """
-
-    if not url:
-        url = 'https://github.com/vyos/' + name + '.git'
-
-    package = {
-        'name': name,
-        'url': url,
-        'commit': commit,
-        'tag': tag,
-        'branch': branch,
-        'shallow': shallow,
-        'path': repo_root + '/packages/' + name,
-        'custombuild_cmd': custombuild_cmd
-    }
-
-    return package
-
-
-
-def call(bashCommand, log, **kwargs):
-    """
-    Run command with arguments. Wait for command to complete.
-
-    Send output to logging module passed as 'log'.
-    """
-
-    from subprocess import Popen, PIPE, STDOUT, check_output, CalledProcessError
-    from tempfile import TemporaryFile
-    from time import sleep
-
-    log.debug("Executing '{}'".format(bashCommand))
-
-    # code borrowsed from:
-    # https://stackoverflow.com/questions/38374063/python-can-we-use-tempfile-with-subprocess-to-get-non-buffering-live-output-in-p
-    # the temp file will be automatically cleaned up
-    output = TemporaryFile()
-    error = TemporaryFile()
-
-    kwargs['stdout'] = output
-    kwargs['stderr'] = error
-    kwargs['shell'] = True
-    kwargs['universal_newlines'] = True
-
-    sub = Popen(bashCommand, **kwargs)
-    while sub.poll() is None:
-        where = output.tell()
-        lines = output.readline()
-        if not lines:
-            sleep(0.3)
-            output.seek(where)
-        else:
-            log.debug(lines.decode().rstrip('\n'))
-
-        where = error.tell()
-        lines = error.readline()
-        if not lines:
-            sleep(0.3)
-            error.seek(where)
-        else:
-           log.info(lines.decode().rstrip('\n'))
-
-
-    # A last write needed after subprocess ends
-    log.debug(output.read().decode().rstrip('\n'))
-    log.info(error.read().decode().rstrip('\n'))
-
-    error.close()
-    output.close()
-    return sub.returncode
-
-def clone_package(pkg, log):
-    """
-    Clone Git repository from URL embedded in pkg to local disk
-
-    First cleanup any possible leftovers from previous builds
-    """
-
-    if args.keep and os.path.isdir(pkg['path']):
-        log.debug("Keep possibly modified package '{}'".format(pkg['path']))
-        return False
-    elif args.clean:
-        # delete repository from disk
-        if os.path.isdir(pkg['path']):
-            log.debug("Cleaning '{}'".format(pkg['path']))
-            shutil.rmtree(pkg['path'])
-    else:
-        if os.path.isdir(pkg['path']):
-            # Change current directory into Git repo for this package
-            os.chdir(pkg['path'])
-
-            bashCommand = 'git clean -d -x --force && git reset --hard ' + pkg['commit']
-            return call(bashCommand, log)
-
-    if pkg['commit'] and pkg['shallow']:
-        log.debug("Package '{}' has both commit and shallow set, unsetting shallow to do a full clone.".format(pkg['name']))
-        pkg['shallow']=False
-
-    bashCommand = 'git clone ' + pkg['url']
-
-    if pkg['shallow']:
-        bashCommand += ' --depth 1'
-
-    if pkg['tag']:
-        bashCommand += ' --branch ' + pkg['tag']
-    elif pkg['branch']:
-        bashCommand += ' --branch ' + pkg['branch']
-
-    bashCommand += ' ' + pkg['path']
-
-    if pkg['commit']:
-        bashCommand += ' && cd ' + pkg['path'] + ' && git reset --hard ' + pkg['commit']
-
-    return call(bashCommand, log)
-
-
-def build_package(pkg, log=None):
-    """
-    Generate Debian package from passed 'pkg'
-    """
-
-    # Change current directory into Git repo for this package
-    os.chdir(pkg['path'])
-
-    # Overwrite custom build command if required, e.g. libyang
-    bashCommand = ''
-    if pkg['custombuild_cmd']:
-        bashCommand = pkg['custombuild_cmd']
-    else:
-        # Build package
-        bashCommand = 'dpkg-buildpackage -uc -us -tc -b'
-        if args.parallel:
-            bashCommand += ' -j' + str(os.cpu_count())
-
-    return call(bashCommand, log)
-
-# a List of all Vyatta/VyOS based packages
-vyos_packages = ['vyatta-bash',
-                 'vyatta-cfg',
-                 'vyatta-op',
-                 'vyatta-cfg-system',
-                 'vyatta-cfg-firewall',
-                 'vyatta-op-firewall',
-                 'vyatta-cfg-vpn',
-                 'vyatta-op-vpn',
-                 'vyatta-cfg-qos',
-                 'vyatta-op-qos',
-                 'vyatta-cfg-op-pppoe',
-                 'vyatta-nat',
-                 'vyatta-config-mgmt',
-                 'vyatta-config-migrate',
-                 'vyatta-zone',
-                 'vyatta-cluster',
-                 'vyatta-eventwatch',
-                 'vyatta-webproxy',
-                 'vyatta-cfg-quagga',
-                 'vyatta-op-quagga',
-                 'vyatta-wanloadbalance',
-                 'vyatta-ravpn',
-                 'vyos-nhrp',
-                 'vyos-world',
-                 'vyatta-iproute',
-                 'vyatta-biosdevname',
-                 'vyos-opennhrp',
-                 'udp-broadcast-relay',
-                 'vyos-1x',
-                 'vyatta-conntrack',
-                 'vyatta-conntrack-sync',
-                 'vyos-xe-guest-utilities',
-                 'vyos-netplug',
-                 'ddclient',
-                 'live-boot']
-
-# Special packages mean packages which are located no in the VyOS namespace
-# or require fancy build instructions
-pkg_special = []
-
-# libvyosconfig/ipaddrcheck uses a different default branch
-libvyosconfig_build_cmd = "eval $(opam env --root=/opt/opam --set-root) && " \
-                          "dpkg-buildpackage -b -us -uc -tc"
-pkg_special.append( add_package('libvyosconfig', branch='master', custombuild_cmd=libvyosconfig_build_cmd))
-pkg_special.append( add_package('ipaddrcheck', branch='master'))
-
-# Packages where we directly build the upstream source
-pkg_special.append( add_package('hvinfo',  url='https://github.com/dmbaturin/hvinfo.git', branch='master') )
-
-# VyOS strongswan ships additional python3-vici packages required by vyos-1x and this is not build by default
-vyos_strongswan_build_cmd = "dpkg-buildpackage -b -us -uc -tc && " \
-                            "autoreconf -i && ./configure --enable-python-eggs && " \
-                            "cd src/libcharon/plugins/vici/python && make && " \
-                            "python3 setup.py --command-packages=stdeb.command bdist_deb && " \
-                            "mv ./deb_dist/*.deb " +  repo_root + "/packages"
-pkg_special.append( add_package('vyos-strongswan', custombuild_cmd=vyos_strongswan_build_cmd) )
-
-#
-# FreeRangeRouting (FRR) packages
-#
-pkg_special.append( add_package('rtrlib', url='https://github.com/rtrlib/rtrlib.git', branch='master', tag='v0.6.3') )
-
-frr_build_cmd  = './tools/tarsource.sh -V && dpkg-buildpackage -us -uc -Ppkg.frr.rtrlib -d'
-pkg_special.append( add_package('frr', url='https://github.com/FRRouting/frr.git', branch='master', tag='frr-7.0', custombuild_cmd=frr_build_cmd) )
-#libyang_build_cmd = 'mkdir build && cd build && cmake .. && make build-deb && mv debs/* ' + repo_root + '/packages'
-#pkg_special.append( add_package('libyang', url='https://github.com/opensourcerouting/libyang.git', commit='179da47', branch='master', custombuild_cmd=libyang_build_cmd) )
-
-
-#
-# We use keepalived from Debian Buster
-#
-pkg_special.append( add_package('keepalived', url='https://salsa.debian.org/ipvs-team/pkg-keepalived.git', branch='master', commit='eae91c81'))
-
-#
-# Linux (VyOS) Kernel
-#
-kernel_build_cmd = "make x86_64_vyos_defconfig && " \
-                   "sed -i 's/\"kernel_version\": \"[0-9].[0-9][0-9].[0-9]*\"/\"kernel_version\": \"'$(make kernelversion)'\"/' " + repo_root + "/data/defaults.json && " \
-                   "make bindeb-pkg LOCALVERSION='-amd64-vyos' KDEB_PKGVERSION=$(make kernelversion)-1 -j $(getconf _NPROCESSORS_ONLN)"
-pkg_special.append( add_package('vyos-kernel', branch='linux-vyos-4.19.y', custombuild_cmd=kernel_build_cmd) )
-
-
-
-#
-# Accell-PPP Package and Kernel Module
-#
-accel_ppp_build_cmd = "echo 'lib/modules/'$(cat " + repo_root + "/data/defaults.json | jq '.kernel_version' | tr -d \\\")-amd64-vyos/extra/*.ko > debian/vyos-accel-ppp-ipoe-kmod.install && " \
-                      "sed -i 's#[0-9].[0-9][0-9].[0-9]*-amd64-vyos#'$(cat " + repo_root + "/data/defaults.json | jq '.kernel_version' | tr -d \\\")'-amd64-vyos#g' debian/rules && " \
-                      "KERNELDIR=" + repo_root + "/packages/vyos-kernel dpkg-buildpackage -b -us -uc -tc -j" + str(os.cpu_count())
-pkg_special.append( add_package('vyos-accel-ppp', custombuild_cmd=accel_ppp_build_cmd) )
-
-#
-# mdns-repeater branch for current is master
-#
-
-pkg_special.append( add_package('mdns-repeater', branch='master') )
-
-
-# A list of all packages we will build in the end
-pkg_build = []
-
-
-
-
-if __name__ == '__main__':
-    parser = argparse.ArgumentParser()
-    exclusive = parser.add_mutually_exclusive_group(required=False)
-    exclusive.add_argument('-c', '--clean', action='store_true', help='Re-clone required Git repositories')
-    exclusive.add_argument('-k', '--keep', action='store_true', help='Keep modified Git repositories')
-    exclusive.add_argument('-f', '--fetch', action='store_true', help='Fetch sources only, no build')
-
-    parser.add_argument('-v', '--verbose', action='count', default=0, help='Increase logging verbosity for each occurance')
-    parser.add_argument('-l', '--list-packages', action='store_true', help='List all packages to build')
-    parser.add_argument('-b', '--build', nargs='+', help='Whitespace separated list of packages to build')
-    parser.add_argument('-p', '--parallel', action='store_true', help='Build on all CPUs')
-    parser.add_argument('--blacklist', nargs='+', help='Do not build/report packages when calling --list')
-
-
-    args = parser.parse_args()
-
-    levels = [ logging.INFO, logging.WARNING, logging.DEBUG ]
-    level = levels[min(len(levels)-1,args.verbose)]  # capped to number of levels
-    logging.basicConfig(level=level, format="%(asctime)s %(name)s %(message)s")
-
-    print("Using vyos-build repository ('{}') commit '{}'\n".format(repo_root, repo_sha))
-
-    #
-    # Exclude packages from build process,
-    # also do not list them on --list
-    #
-    if args.blacklist:
-        for exclude in args.blacklist:
-            if exclude in vyos_packages:
-                vyos_packages.remove(exclude)
-                continue
-
-            found = False
-            for pkg in pkg_special:
-                if exclude == pkg['name']:
-                    found = True
-                    # package already formed
-                    pkg_special.remove(pkg)
-                    break
-
-            if not found:
-                print("Invalid choice '" + exclude + "', --list-packages for complete list!")
-                sys.exit(1)
-
-    #
-    # List all available (to be build) packages
-    #
-    if args.list_packages:
-        print("Individual packages available for build:")
-        for pkg in vyos_packages:
-            print(' * ' + pkg)
-        for pkg in pkg_special:
-            print(' * ' + pkg['name'])
-
-        sys.exit(0)
-
-    #
-    # Only add selective packages to the build list
-    #
-    if args.build:
-        # NOTE: remove double added packages from list
-        for target in args.build:
-            if target in vyos_packages:
-                pkg_build.append(add_package( target ))
-                continue
-
-            found = False
-            for pkg in pkg_special:
-                if target == pkg['name']:
-                    found = True
-                    # package already formed
-                    pkg_build.append( pkg )
-                    break
-
-            if not found:
-                print("Invalid choice '" + target + "', for -b/--build use --list-packages for complete list!")
-                sys.exit(1)
-
-    else:
-        # Add all VyOS packages to the package list
-        for pkg in vyos_packages:
-            pkg_build.append(add_package( pkg ))
-
-        # We also wan't to build all of our special packages
-        for pkg in pkg_special:
-            pkg_build.append( pkg )
-
-    # Build all VyOS packages (packages found on https://github.com/vyos
-    # and referenced in vyos_packages)
-    for pkg in pkg_build:
-        # Create a logging instance per package
-        log = logging.getLogger(pkg['name'])
-
-        ret = clone_package(pkg, log)
-        if ret:
-            log.error("ERROR cloning source")
-            sys.exit(1)
-        else:
-            # only build packages if fetch flag is not set
-            if not args.fetch:
-                ret = build_package(pkg, log)
-                if ret:
-                    log.error("ERROR building source")
-                    sys.exit(1)
-
-    sys.exit(0)