diff options
| -rw-r--r-- | manual/readme.md | 784 | ||||
| -rw-r--r-- | readme.md | 819 |
2 files changed, 797 insertions, 806 deletions
diff --git a/manual/readme.md b/manual/readme.md new file mode 100644 index 0000000..74e3989 --- /dev/null +++ b/manual/readme.md @@ -0,0 +1,784 @@ +Option 2: Manual guide +== + +All following sections describe manual guide - if you use the automated scripts, then you +don't need this information unless you face some issue, then this information is helpful to understand +how things work and how to debug them. + +General expectations +-- + +Unless specified otherwise all commands/scripts in the instructions should run as `root`. +If you don't use root account then use `sudo -i` from your user to switch to root. +Where other user is expected we provide note and `su` command. + +The current working directory doesn't matter unless specified with `cd`. + +The build system was designed to use 3 or more machines that's why some steps may seem a bit unusual. +This guide merges everything to single host under single user to make it simpler and faster to get started. +You may use another machine as build node for Jenkins (or multiple nodes), you may also use another machine +for reprepro but here it's assumed everything is one host under one user. + +Before you install Jenkins, create its user and group +-- + +Most of Jenkinfiles do respect your UID/GID but not all, for +example [linux-kernel/Jenkinsfile](https://github.com/vyos/vyos-build/blob/equuleus/packages/linux-kernel/Jenkinsfile) +has hardcoded UID and GID to 1006 and this will fail build if you don't have 1006:1006 user. + +That's why we want to create jenkins user and group with ID 1006 before installing Jenkins from apt. + +```bash +groupadd --system --gid 1006 jenkins +useradd --system --comment Jenkins --shell /bin/bash --uid 1006 --gid 1006 --home-dir /var/lib/jenkins jenkins +``` + +If you have already existing user then please [change its UID/GID](extras/legacy-uid-gid.md). + +Install Jenkins, and its java +-- + +Install java, then Jenkins. Let setup guide to install recommended plugins. + +Just follow the usual guide via APT https://www.jenkins.io/doc/book/installing/linux/#debianubuntu + +Install docker +-- + +Just follow the usual guide via APT https://docs.docker.com/engine/install/debian/ + +Allow Jenkins to use docker: + +```bash +usermod -a -G docker jenkins +``` + +Setup local IP +-- + +This guide will simplify the unknown by using static IP on dummy interface, this is hopefully outside your subnet if +not please change all references of this IP with your own. + +```bash +cat << EOT >> /etc/network/interfaces + +auto dummy0 +iface dummy0 inet static + address 172.17.17.17/32 + pre-up ip link add dummy0 type dummy +EOT +``` + +```bash +ifup dummy0 +``` + +Now we can locally point to known IP `172.17.17.17` as it was the host itself. + +After adding docker group and/or after UID/GID change restart Jenkins +-- + +```bash +systemctl restart jenkins.service +``` + +Launch local registry and set it, so it always runs when Docker runs +-- + +```bash +docker run -d -p 5000:5000 --restart always --name registry registry:2.7 +``` + +**Allow insecure docker access to local registry:** + +Add your local IP with 5000 port to `insecure-registries` section in `/etc/docker/daemon.json`, something like this: + +```bash +cat << EOF > /etc/docker/daemon.json +{ + "insecure-registries": [ + "172.17.17.17:5000" + ] +} +EOF +``` + +Then restart docker: + +```bash +systemctl restart docker.service +``` + +Install apt-cacher-ng for ELTS mirror +-- + +This is currently used only by equuleus. + +```bash +apt install apt-cacher-ng +``` + +This will allow us to use `http://172.17.17.17:3142/deb.freexian.com/extended-lts` as ELTS mirror. + +Build patched vyos-build docker images +-- + +The vyos/vyos-build docker image from dockerhub doesn't work for all packages as of now, thus we made some +patches to make it work. If this changed in future then this step can be skipped. + +The below script clones the (patched) vyos-build, then builds and pushes the images to your custom Docker repository. + +```bash +#!/usr/bin/env bash +set -e + +CUSTOM_DOCKER_REPO="172.17.17.17:5000" +ELTS_MIRROR="http://172.17.17.17:3142/deb.freexian.com/extended-lts" + +# +# Clone (patched) vyos-build + +git clone https://github.com/dd010101/vyos-build.git +cd vyos-build/docker + +# +# Build and Push equuleus + +git checkout equuleus +docker build --build-arg "ELTS_MIRROR=$ELTS_MIRROR" \ + --no-cache -t vyos/vyos-build:equuleus . + +docker tag vyos/vyos-build:equuleus ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:equuleus +docker push ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:equuleus + +# +# Build and Push sagitta + +git checkout sagitta +docker build --no-cache -t vyos/vyos-build:sagitta . + +docker tag vyos/vyos-build:sagitta ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:sagitta +docker push ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:sagitta + +# +# Build and Push current -- (current is required for some sagitta packages) + +git checkout current +docker build --no-cache -t vyos/vyos-build:current . + +docker tag vyos/vyos-build:current ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:current +docker push ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:current +``` + +Install Jenkins plugins +-- +**Manage Jenkins -> Plugins -> Available plugins** + +- Docker +- Docker Pipeline +- Copy Artifact +- SSH Agent +- Pipeline Utility Steps +- Job DSL + +Configure Built-In node +-- +**Manage Jenkins -> Nodes -> Built-In Node** + +**Add labels (tags)** + +- Docker +- docker +- ec2_amd64 + +Separated by space thus "Docker docker ec2_amd64" as result + +Configure Jenkins System +-- +**Manage Jenkins -> System** + +**Global properties -> Environmental Variables -> Add** + +> **Name:** DEV_PACKAGES_VYOS_NET_HOST\ +> **Value:** jenkins@172.17.17.17 + +This user+IP/host will be used for SSH access to reprepro, it can be another host, we use the host itself, +this IP needs to be accessible from docker container thus this should be LAN IP not localhost. + +**Global properties -> Environmental Variables -> Add** + + +> **Name:** ARM64_BUILD_DISABLED\ +> **Value:** true + +This is used to disable ARM64 support. The vyos-build expects that you have ARM64 build node and that's not +something that is easy to obtain or emulate on x86. If you have ARM64 build node then skip this step and make sure +your ARM64 node has tag `ec2_arm64`. If you try to build ARM64 without ARM node then most sagitta builds will wait +and eventually fail. + +**Global properties -> Environmental Variables -> Add** + +> **Name:** CUSTOM_BUILD_CHECK_DISABLED\ +> **Value:** true + +This is used to disable custom build check. Custom build check would normally skip upload to reprepro repository +if package is built from non-vyos repository. Unfortunately currently it's impossible to build all packages +from VyOS repositories, and thus we need to use custom repositories. Because some packages don't have +functional build scripts or don't exist at all. This check doesn't make sense anyway since we are using our +reprepro repository. + +**Global properties -> Environmental Variables -> Add** + +> **Name:** CUSTOM_DOCKER_REPO\ +> **Value:** 172.17.17.17:5000 + +This variable is used to specify local docker registry for automatic `vyos-build` docker image rebuild. + +**Global Pipeline Libraries -> Add** + +> **Name:** vyos-build\ +> **Project** repository: https://github.com/dd010101/vyos-build.git + +Currently patched version of vyos-build is required, in the future the official +`https://github.com/vyos/vyos-build.git` may work but doesn't currently. + +Note for developers: equuleus is using only equuleus branch of vyos-build but sagitta is using both sagitta and +current, thus if you fix something aimed at sagitta, you need to backports these changes to current as well, +since some packages will use current and some sagitta branch. + +**Declarative Pipeline (Docker)** + +> **Docker registry URL:** http://172.17.17.17:5000 + +This is required to tell Jenkins to use your own (patched) vyos-build docker image and not the DockerHub version. + +Credentials for ssh-agent +-- +You need to set up SSH key authentication for the host specified in `DEV_PACKAGES_VYOS_NET_HOST` variable. +Basically we want to allow Jenkins to SSH into itself with its own SSH key. + +Login as target user: + +```bash +su - jenkins +``` + +Generate regular SSH key: + +```bash +ssh-keygen -t ed25519 -C "jenkins" +``` + +Update authenticated_keys to allow Jenkins to log in to itself, something like this: + +```bash +cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys +``` + +Accept signature and verify SSH works: + +```bash +ssh 172.17.17.17 +``` + +Then you can add this private key to Jenkins: + +**Manage Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add Credentials** + +> **Kind:** SSH Username with private key\ +> **ID:** SSH-dev.packages.vyos.net\ +> **Username:** jenkins + +**Private Key -> Enter directly -> Add** + +> <paste private key of the generated ssh key like the contents of cat ~/.ssh/id_ed25519> + +Preparation for reprepro SSH host +-- + +Install some packages: + +```bash +apt install reprepro gpg +``` + +Generate GPG singing key (without passphrase): + +```bash +sudo -u jenkins gpg --pinentry-mode loopback --full-gen-key +``` + +This key **needs to be without passphrase**. The reprepro uses this key in background thus there is no way to enter +passphrase. + +Remember your pub key, it's random string like "934824D5C6A72DA964B3AFBD27A7E25D86BB7E2A". + +Create expected folder structure, prepare reprepro config and give Jenkins access, this is done for each release +codename. + +Set SIGN_PUB_KEY: + +```bash +export SIGN_PUB_KEY="<pub key idenitifier from step above>" +``` + +Set RELEASE name: + +```bash +export RELEASE=equuleus +``` + +or + +```bash +export RELEASE=sagitta +``` + +Then create reprepro repository for each RELEASE: + +```bash +export REPOSITORY=/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/$RELEASE +mkdir -p $REPOSITORY +mkdir $REPOSITORY/conf + +cat << EOF > $REPOSITORY/conf/distributions +Origin: $RELEASE +Label: $RELEASE +Codename: $RELEASE +Architectures: source amd64 +Components: main +Description: $RELEASE +SignWith: $SIGN_PUB_KEY +EOF + +cat << EOF > $REPOSITORY/conf/options +verbose +EOF + +chown -R jenkins:jenkins /home/sentrium +``` + +uncron +-- +This is required addition for the reprepro. + +**Install dependencies** + +```bash +apt install opam ocaml socat +``` + +**Login as reprepro user and build uncon, then exit (if asked - confirm defaults)** + +You may have default opem switch already, then you will +see `[ERROR] There already is an installed switch named default` - +if you do then ignore this message and continue. + +```bash +su - jenkins + +git clone https://github.com/vyos/uncron.git +cd uncron + +opam init +opam switch create default 4.13.1 +eval $(opam env --switch=default) +opam install lwt lwt_ppx logs containers +eval $(opam env) + +dune build +exit +``` + +**Setup uncron service** + +```bash +cp /var/lib/jenkins/uncron/_build/install/default/bin/uncron /usr/local/sbin/ + +cat <<'EHLO' > /etc/systemd/system/uncron.service +[Unit] +Description=Command Queue Service +After=auditd.service systemd-user-sessions.service time-sync.target + +[Service] +EnvironmentFile=/etc/uncron.conf +ExecStart=/usr/local/sbin/uncron +ExecReload=/bin/kill -HUP $MAINPID +KillMode=process +User=jenkins +Group=jenkins +Restart=on-failure +RestartSec=5s + +[Install] +WantedBy=multi-user.target +EHLO + +touch /etc/uncron.conf + +systemctl daemon-reload +systemctl enable --now uncron.service + +chmod +x /var/lib/jenkins/uncron/src/uncron-add +``` + +**Create uncron-add script** + +We also use this as hack to fix some of VyOS packaging issues. + +```bash +cat << 'EOF' > /usr/local/bin/uncron-add +#!/usr/bin/env bash +set -e + +COMMAND="$1" + +# this is hack to workaround a issue where vyos didn't create sagitta branch +# like the case of vyos-xe-guest-utilities, thus we need to build current +# branch and then redirect from here to sagitta repository +if [ ! -L ~/VyOS/current ]; then + rm -rf ~/VyOS/current + mkdir -p ~/VyOS/sagitta + ln -s ~/VyOS/sagitta ~/VyOS/current +fi +if [[ "$COMMAND" == *"repositories/current"* ]]; then + COMMAND=${COMMAND//current/sagitta} +fi +if [[ "$COMMAND" == *"vyos-xe-guest-utilities"* ]] && [[ "$COMMAND" == *"current"* ]]; then + COMMAND=${COMMAND//current/sagitta} +fi + +/var/lib/jenkins/uncron/src/uncron-add "$COMMAND" +EOF + +chmod +x /usr/local/bin/uncron-add +``` + +Multibranch Pipelines (by script) +-- + +Script exists to automate pipeline/job creation. + +**Generate Jenkins API token:** + +**User account (click on your username - top-right corner) -> Configure**\ +**API Token -> Current token(s) -> Add new Token** + +> **Name:** CLI\ +> -> [Generate] -> [copy generated key and save it for later use] + +**Get the script** + +```bash +git clone https://github.com/dd010101/vyos-jenkins.git +cd vyos-jenkins/manual +``` + +**Install dependencies** + +```bash +apt install -y xmlstarlet jq +``` + +**Adjust settings to suit your Jenkins** + +```bash +export JENKINS_USER=<your-username> +export JENKINS_TOKEN=<your-token> +``` + +**If you want to build only specific branch** + +Configure `BRANCH` environment variable to desired branch before you run the script. +Not defined or empty value means all branches (the default). + +```bash +export BRANCH="sagitta" +``` + +**Create jobs** + +Then wait for branch indexing to complete. + +```bash +./seed-jobs.sh create +``` + +**After branch indexing you can trigger build for everything** + +Make sure you have >=16GB RAM or 8GB RAM + 8GB swap, since running build for everything like this eats more memory +than building one by one this is also dependent on how many Number of executors you have. + +```bash +./seed-jobs.sh build +``` + +Now wait for build to complete and check Build History and Dashboard for failed builds. If you find any failed +builds then read Console Output to see why it did failed. + +This process is required only once. After you create Jenkins jobs, and you do first build then Jenkins will +periodically check if GIT repository for job changed and will do automatically build given job/packages. + +You can also create Multibranch Pipelines manually, [see bellow](#multibranch-pipelines-manual). + +Mirror preparation +-- + +Use the default procedure to build ISO (via docker) but you need to specify your `--vyos-mirror` and your gpg +singing key `--custom-apt-key`. + +To make `--vyos-mirror` is easy, you just install your favorite webserver and point the webroot +to `/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/`. For example nginx vhost looks +something like this: + +``` +server { + listen 80; + listen [::]:80; + + root /home/sentrium/web/dev.packages.vyos.net/public_html/repositories; + autoindex on; + + server_name _; + + location / { + try_files $uri $uri/ =404; + } + + location ~ /(.*)/conf { + deny all; + } + + location ~ /(.*)/db { + deny all; + } +} +``` + +This will give you HTTP APT repository, like this `http://172.17.17.17/equuleus`. + +To create `--custom-apt-key` you need to export your gpg singing public key, for example: + +```bash +sudo -u jenkins gpg --armor --output /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/apt.gpg.key \ + --export-options export-minimal --export vyos +``` + +This will give you `/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/apt.gpg.key` or +`http://172.17.17.17/apt.gpg.key`. + +If you have the `--vyos-mirror` URL of your own repository and your own singing key `--custom-apt-key` then you +can use these two pieces to build ISO from your own package mirror. + +How to build ISO +-- + +Complete instructions to illustrate how to build ISO via Docker method included with the extra options outline above. + +We assume you already have Docker running if not follow +the [official instructions](https://docs.docker.com/engine/install/debian/) for your OS. + +**Obtain the sources:** + +```bash +git clone https://github.com/dd010101/vyos-build +cd vyos-build +``` + +**Select branch of your choice** + +For all following steps will use BRANCH environment variable since the branch repeats a lot. + +```bash +export BRANCH=equuleus +``` + +or + +```bash +export BRANCH=sagitta +``` + +**Switch to branch** + +```bash +git checkout "$BRANCH" +``` + +**Clear previous build resources (if any)** + +There is `make clean` but that doesn't always clean everything and may produce corrupted build environment. +The `make clean` is trying to remove specific parts of `build` directory, but it doesn't always do so correctly. +This happens mainly if you switch branches - that's why it's better to always delete the whole `build` directory. + +```bash +rm -rf build/ +``` + +**Obtain vyos-build docker container** + +You can reuse your docker container image if you already +have [custom registry](#launch-local-registry-and-set-it-so-it-always-runs-when-docker-runs) +with [patched vyos-build docker container](#build-patched-vyos-build-docker-images). + +Change the registry URL if you build on other machine. + +```bash +docker pull "172.17.17.17:5000/vyos/vyos-build:$BRANCH" +docker tag "172.17.17.17:5000/vyos/vyos-build:$BRANCH" "vyos/vyos-build:$BRANCH" +``` + +If you don't have custom registry then build the container - this will take a while: + +```bash +docker build -t "vyos/vyos-build:$BRANCH" docker +``` + +You should rebuild the container from time to time - not very frequently but sometimes the build will break +if you have too old container. + +**Obtain apt singing key for your custom mirror** + +```bash +wget http://172.17.17.17/apt.gpg.key -O /tmp/apt.gpg.key +``` + +**Launch the vyos-build docker container** + +This is the usual run command from official documentation, we need to add extra mount for our apt singing key +for later use via `-v "/tmp/apt.gpg.key:/opt/apt.gpg.key"`. + +The docker run command will mount current working directory for use inside the container that's why you need to +execute this command inside the `vyos-build` directory (that is the GIT repository you cloned above). +You can also replace the `-v "$(pwd)":/vyos` with static path if you like not to depend on current directory +(for example `-v /opt/vyos-build:/vyos`). + +```bash +docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "/tmp/apt.gpg.key:/opt/apt.gpg.key" \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + "vyos/vyos-build:$BRANCH" bash +``` + +Now we should be inside the container. + +**Configure and build the ISO** + +Command for configuring changed over time, equuleus has `./configure`, sagitta has `./build-vyos-image iso` instead. + +You may want to customize the configuration options, see what is available: + +For equuleus: + +```bash +sudo ./configure --help +``` + +For sagitta: + +```bash +sudo ./build-vyos-image --help +``` + +We need to add extra two options to configure `--vyos-mirror` and `--custom-apt-key`. We also add smoketest +via `--custom-package vyos-1x-smoketest` for good measure. + +Here are examples - please adjust options to your liking: + +For equuleus: + +```bash +sudo ./configure --architecture amd64 --build-by "myself@localhost" \ + --build-type release --version "1.3.x" \ + --vyos-mirror http://172.17.17.17/equuleus --custom-apt-key /opt/apt.gpg.key \ + --debian-elts-mirror http://172.17.17.17:3142/deb.freexian.com/extended-lts \ + --custom-package vyos-1x-smoketest \ + && sudo make iso +``` + +For sagitta: + +```bash +sudo ./build-vyos-image iso --architecture amd64 --build-by "myself@localhost" \ + --build-type release --version "1.4.x" \ + --vyos-mirror http://172.17.17.17/sagitta --custom-apt-key /opt/apt.gpg.key \ + --custom-package vyos-1x-smoketest +``` + +This will take a while - after all is done then you can `exit` the container and you should have +`build/live-image-amd64.hybrid.iso`. + +Multibranch Pipelines (manual) +-- + +Use + button on Jenkins dashboard to add Multibranch Pipeline. Each Jenkinsfile needs its own Multibranch Pipeline, +the setup is the same for all packages, and you just adjust location of Jenkinsfile and/or GIT repository to whatever +you want to build. See packages info bellow for list of all GIT repository and location their Jenkinsfile. + +It makes sense to configure one pipeline and then use "Copy from" and just change the Jenkinsfile location and/or GIT +repository. Start with something small like dropbear, and after you verify your setup works then try the bigger stuff +like linux-kernel. + +There are two types of configurations - first you have single shared GIT repository with many Jenkinsfile, like +the https://github.com/vyos/vyos-build.git, thus each pipeline points to different Jenkinsfile, but they share same GIT +url. Rest of the packages have their own GIT repository with single root Jenkinsfile, thus the Jenkinsfile stays +"Jenkinsfile" and you change the GIT url only. The https://github.com/vyos/vyos-build.git repository has also its own +root Jenkinsfile - ignore it since that one is trying to build ISO with default (blocked) apt mirror. + +You can see all packages and information about them in the [packages.md](extras/packages.md) file. + +**Branch Sources -> Add source -> Git** + +> **Project Repository:** https://github.com/vyos/vyos-build.git + +(or any other repository, like https://github.com/vyos/vyos-1x.git) + +You may want to restrict to only branches you care about, thus: + +**Behaviours -> Add -> Filter by name (with regular expression)** + +> *Regular expression:** (equuleus|sagitta) + +**Behaviours -> Add -> Advanced clone behaviours** + +> **Fetch tags:** [✓] + +(leave defaults) + +Advanced clone is required for some packages to obtain all tags (specifically `vyos-cloud-init` and `vyos-1x`). +It doesn't hurt to have advanced clone for everything, thus you can set-it and copy for everything without +worrying about what to use it for. + +**Build Configuration -> Mode (by Jenkinsfile)** + +> **Script Path:** packages/dropbear/Jenkinsfile + +(if you want to build package from vyos/vyos-build repository) + +> **Script Path:** Jenkinsfile + +(or leave just Jenkinsfile if you want to build repository like vyos/vyos-1x where there is just one package) + +**Scan Multibranch Pipeline Triggers** + +> [✓] **Periodically if not otherwise run** +> +> **Interval:** 1 hour + +Jenkins will check the source GIT repository if changes were made and execute automatic build if needed. This +will keep packages up to date. + +Try to build +-- + +Now it's possible to select some **Multibranch Pipeline**, select your **branch**, and you can press build button +to see what happens! If all is well you should see .deb appearing in +`/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/`: + +```bash +find /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/ -name '*.deb' -print +``` + +If build fails then click the specific build number and check **Console Output** for hints why it does so. @@ -54,18 +54,12 @@ and you will still get good performance this way since the above 8GB threshold i Multiple options -- -Historically this project documented the process how to build all packages and create package repository -as manual guide with light usage of scripts. This is legacy method. +Thanks to work of [@GurliGebis](https://github.com/GurliGebis) we now can use automated scripts instead of the legacy +manual guide. The manual guide is deprecated and available for historical purposes here +[manual/readme.md](./manual/readme.md). The automated scripts are now preferred method. -Now we have possibility to do all steps from manual guide via automated scripts thanks to work of -[@GurliGebis](https://github.com/GurliGebis). This is preferred method. - -We keep both methods in sync with changes. - ---- - -Option 1: Automated scripts -== +Automated scripts +-- The automated scripts execute all manual steps with minimal interaction. The process is divided into 8 steps/scripts. These 8 scripts configure Jenkins, prepare the package repositories, build the packages and there is also one @@ -125,15 +119,15 @@ If all went well, then all steps should complete successfully and then you can: Now you should have the ISO(s) in current directory (`/opt/vyos-jenkins`). -You could be also interested in the [Smoketest](#smoketest). -If something isn't right, then see [Something is wrong](#something-is-wrong). +**If something isn't right, then see [Something is wrong](#something-is-wrong).** You can rerun the scripts as you wish and the scripts should do only required/changed steps. It's also good idea to do this from time to time. If your scripts are somewhat old, and you face some error, then please first try to download and run latest scripts. Beware - like with any custom ISO you shall test every ISO you build with your configuration and traffic flowing. -The Smoketest isn't substitute for real world testing. +If are interested in the Smoketest see the [Smoketest](#smoketest) action. The Smoketest isn't substitute for +real world testing. Jenkins will automatically detect changes and build new packages, thus if you keep build server running then it should keep the repositories up to date by itself. This way you can just use build-iso.sh again and again. @@ -146,720 +140,6 @@ and start the Jenkins, then in worse case you would need to wait up to 15 minute before rebuild of package would start. Then you shall wait before the **Build Queue** and **Build Executor Status** is empty, then make sure no build failed in the **Build History**, after this you can use build-iso.sh again. ---- - -Option 2: Manual guide -== - -All following sections describe manual guide - if you use the automated scripts, then you -don't need this information unless you face some issue, then this information is helpful to understand -how things work and how to debug them. - -General expectations --- - -Unless specified otherwise all commands/scripts in the instructions should run as `root`. -If you don't use root account then use `sudo -i` from your user to switch to root. -Where other user is expected we provide note and `su` command. - -The current working directory doesn't matter unless specified with `cd`. - -The build system was designed to use 3 or more machines that's why some steps may seem a bit unusual. -This guide merges everything to single host under single user to make it simpler and faster to get started. -You may use another machine as build node for Jenkins (or multiple nodes), you may also use another machine -for reprepro but here it's assumed everything is one host under one user. - -Before you install Jenkins, create its user and group --- - -Most of Jenkinfiles do respect your UID/GID but not all, for -example [linux-kernel/Jenkinsfile](https://github.com/vyos/vyos-build/blob/equuleus/packages/linux-kernel/Jenkinsfile) -has hardcoded UID and GID to 1006 and this will fail build if you don't have 1006:1006 user. - -That's why we want to create jenkins user and group with ID 1006 before installing Jenkins from apt. - -```bash -groupadd --system --gid 1006 jenkins -useradd --system --comment Jenkins --shell /bin/bash --uid 1006 --gid 1006 --home-dir /var/lib/jenkins jenkins -``` - -If you have already existing user then please [change its UID/GID](extras/legacy-uid-gid.md). - -Install Jenkins, and its java --- - -Install java, then Jenkins. Let setup guide to install recommended plugins. - -Just follow the usual guide via APT https://www.jenkins.io/doc/book/installing/linux/#debianubuntu - -Install docker --- - -Just follow the usual guide via APT https://docs.docker.com/engine/install/debian/ - -Allow Jenkins to use docker: - -```bash -usermod -a -G docker jenkins -``` - -Setup local IP --- - -This guide will simplify the unknown by using static IP on dummy interface, this is hopefully outside your subnet if -not please change all references of this IP with your own. - -```bash -cat << EOT >> /etc/network/interfaces - -auto dummy0 -iface dummy0 inet static - address 172.17.17.17/32 - pre-up ip link add dummy0 type dummy -EOT -``` - -```bash -ifup dummy0 -``` - -Now we can locally point to known IP `172.17.17.17` as it was the host itself. - -After adding docker group and/or after UID/GID change restart Jenkins --- - -```bash -systemctl restart jenkins.service -``` - -Launch local registry and set it, so it always runs when Docker runs --- - -```bash -docker run -d -p 5000:5000 --restart always --name registry registry:2.7 -``` - -**Allow insecure docker access to local registry:** - -Add your local IP with 5000 port to `insecure-registries` section in `/etc/docker/daemon.json`, something like this: - -```bash -cat << EOF > /etc/docker/daemon.json -{ - "insecure-registries": [ - "172.17.17.17:5000" - ] -} -EOF -``` - -Then restart docker: - -```bash -systemctl restart docker.service -``` - -Install apt-cacher-ng for ELTS mirror --- - -This is currently used only by equuleus. - -```bash -apt install apt-cacher-ng -``` - -This will allow us to use `http://172.17.17.17:3142/deb.freexian.com/extended-lts` as ELTS mirror. - -Build patched vyos-build docker images --- - -The vyos/vyos-build docker image from dockerhub doesn't work for all packages as of now, thus we made some -patches to make it work. If this changed in future then this step can be skipped. - -The below script clones the (patched) vyos-build, then builds and pushes the images to your custom Docker repository. - -```bash -#!/usr/bin/env bash -set -e - -CUSTOM_DOCKER_REPO="172.17.17.17:5000" -ELTS_MIRROR="http://172.17.17.17:3142/deb.freexian.com/extended-lts" - -# -# Clone (patched) vyos-build - -git clone https://github.com/dd010101/vyos-build.git -cd vyos-build/docker - -# -# Build and Push equuleus - -git checkout equuleus -docker build --build-arg "ELTS_MIRROR=$ELTS_MIRROR" \ - --no-cache -t vyos/vyos-build:equuleus . - -docker tag vyos/vyos-build:equuleus ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:equuleus -docker push ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:equuleus - -# -# Build and Push sagitta - -git checkout sagitta -docker build --no-cache -t vyos/vyos-build:sagitta . - -docker tag vyos/vyos-build:sagitta ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:sagitta -docker push ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:sagitta - -# -# Build and Push current -- (current is required for some sagitta packages) - -git checkout current -docker build --no-cache -t vyos/vyos-build:current . - -docker tag vyos/vyos-build:current ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:current -docker push ${CUSTOM_DOCKER_REPO}/vyos/vyos-build:current -``` - -Install Jenkins plugins --- -**Manage Jenkins -> Plugins -> Available plugins** - -- Docker -- Docker Pipeline -- Copy Artifact -- SSH Agent -- Pipeline Utility Steps -- Job DSL - -Configure Built-In node --- -**Manage Jenkins -> Nodes -> Built-In Node** - -**Add labels (tags)** - -- Docker -- docker -- ec2_amd64 - -Separated by space thus "Docker docker ec2_amd64" as result - -Configure Jenkins System --- -**Manage Jenkins -> System** - -**Global properties -> Environmental Variables -> Add** - -> **Name:** DEV_PACKAGES_VYOS_NET_HOST\ -> **Value:** jenkins@172.17.17.17 - -This user+IP/host will be used for SSH access to reprepro, it can be another host, we use the host itself, -this IP needs to be accessible from docker container thus this should be LAN IP not localhost. - -**Global properties -> Environmental Variables -> Add** - - -> **Name:** ARM64_BUILD_DISABLED\ -> **Value:** true - -This is used to disable ARM64 support. The vyos-build expects that you have ARM64 build node and that's not -something that is easy to obtain or emulate on x86. If you have ARM64 build node then skip this step and make sure -your ARM64 node has tag `ec2_arm64`. If you try to build ARM64 without ARM node then most sagitta builds will wait -and eventually fail. - -**Global properties -> Environmental Variables -> Add** - -> **Name:** CUSTOM_BUILD_CHECK_DISABLED\ -> **Value:** true - -This is used to disable custom build check. Custom build check would normally skip upload to reprepro repository -if package is built from non-vyos repository. Unfortunately currently it's impossible to build all packages -from VyOS repositories, and thus we need to use custom repositories. Because some packages don't have -functional build scripts or don't exist at all. This check doesn't make sense anyway since we are using our -reprepro repository. - -**Global properties -> Environmental Variables -> Add** - -> **Name:** CUSTOM_DOCKER_REPO\ -> **Value:** 172.17.17.17:5000 - -This variable is used to specify local docker registry for automatic `vyos-build` docker image rebuild. - -**Global Pipeline Libraries -> Add** - -> **Name:** vyos-build\ -> **Project** repository: https://github.com/dd010101/vyos-build.git - -Currently patched version of vyos-build is required, in the future the official -`https://github.com/vyos/vyos-build.git` may work but doesn't currently. - -Note for developers: equuleus is using only equuleus branch of vyos-build but sagitta is using both sagitta and -current, thus if you fix something aimed at sagitta, you need to backports these changes to current as well, -since some packages will use current and some sagitta branch. - -**Declarative Pipeline (Docker)** - -> **Docker registry URL:** http://172.17.17.17:5000 - -This is required to tell Jenkins to use your own (patched) vyos-build docker image and not the DockerHub version. - -Credentials for ssh-agent --- -You need to set up SSH key authentication for the host specified in `DEV_PACKAGES_VYOS_NET_HOST` variable. -Basically we want to allow Jenkins to SSH into itself with its own SSH key. - -Login as target user: - -```bash -su - jenkins -``` - -Generate regular SSH key: - -```bash -ssh-keygen -t ed25519 -C "jenkins" -``` - -Update authenticated_keys to allow Jenkins to log in to itself, something like this: - -```bash -cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys -``` - -Accept signature and verify SSH works: - -```bash -ssh 172.17.17.17 -``` - -Then you can add this private key to Jenkins: - -**Manage Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add Credentials** - -> **Kind:** SSH Username with private key\ -> **ID:** SSH-dev.packages.vyos.net\ -> **Username:** jenkins - -**Private Key -> Enter directly -> Add** - -> <paste private key of the generated ssh key like the contents of cat ~/.ssh/id_ed25519> - -Preparation for reprepro SSH host --- - -Install some packages: - -```bash -apt install reprepro gpg -``` - -Generate GPG singing key (without passphrase): - -```bash -sudo -u jenkins gpg --pinentry-mode loopback --full-gen-key -``` - -This key **needs to be without passphrase**. The reprepro uses this key in background thus there is no way to enter -passphrase. - -Remember your pub key, it's random string like "934824D5C6A72DA964B3AFBD27A7E25D86BB7E2A". - -Create expected folder structure, prepare reprepro config and give Jenkins access, this is done for each release -codename. - -Set SIGN_PUB_KEY: - -```bash -export SIGN_PUB_KEY="<pub key idenitifier from step above>" -``` - -Set RELEASE name: - -```bash -export RELEASE=equuleus -``` - -or - -```bash -export RELEASE=sagitta -``` - -Then create reprepro repository for each RELEASE: - -```bash -export REPOSITORY=/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/$RELEASE -mkdir -p $REPOSITORY -mkdir $REPOSITORY/conf - -cat << EOF > $REPOSITORY/conf/distributions -Origin: $RELEASE -Label: $RELEASE -Codename: $RELEASE -Architectures: source amd64 -Components: main -Description: $RELEASE -SignWith: $SIGN_PUB_KEY -EOF - -cat << EOF > $REPOSITORY/conf/options -verbose -EOF - -chown -R jenkins:jenkins /home/sentrium -``` - -uncron --- -This is required addition for the reprepro. - -**Install dependencies** - -```bash -apt install opam ocaml socat -``` - -**Login as reprepro user and build uncon, then exit (if asked - confirm defaults)** - -You may have default opem switch already, then you will -see `[ERROR] There already is an installed switch named default` - -if you do then ignore this message and continue. - -```bash -su - jenkins - -git clone https://github.com/vyos/uncron.git -cd uncron - -opam init -opam switch create default 4.13.1 -eval $(opam env --switch=default) -opam install lwt lwt_ppx logs containers -eval $(opam env) - -dune build -exit -``` - -**Setup uncron service** - -```bash -cp /var/lib/jenkins/uncron/_build/install/default/bin/uncron /usr/local/sbin/ - -cat <<'EHLO' > /etc/systemd/system/uncron.service -[Unit] -Description=Command Queue Service -After=auditd.service systemd-user-sessions.service time-sync.target - -[Service] -EnvironmentFile=/etc/uncron.conf -ExecStart=/usr/local/sbin/uncron -ExecReload=/bin/kill -HUP $MAINPID -KillMode=process -User=jenkins -Group=jenkins -Restart=on-failure -RestartSec=5s - -[Install] -WantedBy=multi-user.target -EHLO - -touch /etc/uncron.conf - -systemctl daemon-reload -systemctl enable --now uncron.service - -chmod +x /var/lib/jenkins/uncron/src/uncron-add -``` - -**Create uncron-add script** - -We also use this as hack to fix some of VyOS packaging issues. - -```bash -cat << 'EOF' > /usr/local/bin/uncron-add -#!/usr/bin/env bash -set -e - -COMMAND="$1" - -# this is hack to workaround a issue where vyos didn't create sagitta branch -# like the case of vyos-xe-guest-utilities, thus we need to build current -# branch and then redirect from here to sagitta repository -if [ ! -L ~/VyOS/current ]; then - rm -rf ~/VyOS/current - mkdir -p ~/VyOS/sagitta - ln -s ~/VyOS/sagitta ~/VyOS/current -fi -if [[ "$COMMAND" == *"repositories/current"* ]]; then - COMMAND=${COMMAND//current/sagitta} -fi -if [[ "$COMMAND" == *"vyos-xe-guest-utilities"* ]] && [[ "$COMMAND" == *"current"* ]]; then - COMMAND=${COMMAND//current/sagitta} -fi - -/var/lib/jenkins/uncron/src/uncron-add "$COMMAND" -EOF - -chmod +x /usr/local/bin/uncron-add -``` - -Multibranch Pipelines (by script) --- - -Script exists to automate pipeline/job creation. - -**Generate Jenkins API token:** - -**User account (click on your username - top-right corner) -> Configure**\ -**API Token -> Current token(s) -> Add new Token** - -> **Name:** CLI\ -> -> [Generate] -> [copy generated key and save it for later use] - -**Get the script** - -```bash -git clone https://github.com/dd010101/vyos-jenkins.git -cd vyos-jenkins/manual -``` - -**Install dependencies** - -```bash -apt install -y xmlstarlet jq -``` - -**Adjust settings to suit your Jenkins** - -```bash -export JENKINS_USER=<your-username> -export JENKINS_TOKEN=<your-token> -``` - -**If you want to build only specific branch** - -Configure `BRANCH` environment variable to desired branch before you run the script. -Not defined or empty value means all branches (the default). - -```bash -export BRANCH="sagitta" -``` - -**Create jobs** - -Then wait for branch indexing to complete. - -```bash -./seed-jobs.sh create -``` - -**After branch indexing you can trigger build for everything** - -Make sure you have >=16GB RAM or 8GB RAM + 8GB swap, since running build for everything like this eats more memory -than building one by one this is also dependent on how many Number of executors you have. - -```bash -./seed-jobs.sh build -``` - -Now wait for build to complete and check Build History and Dashboard for failed builds. If you find any failed -builds then read Console Output to see why it did failed. - -This process is required only once. After you create Jenkins jobs, and you do first build then Jenkins will -periodically check if GIT repository for job changed and will do automatically build given job/packages. - -You can also create Multibranch Pipelines manually, [see bellow](#multibranch-pipelines-manual). - -Mirror preparation --- - -Use the default procedure to build ISO (via docker) but you need to specify your `--vyos-mirror` and your gpg -singing key `--custom-apt-key`. - -To make `--vyos-mirror` is easy, you just install your favorite webserver and point the webroot -to `/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/`. For example nginx vhost looks -something like this: - -``` -server { - listen 80; - listen [::]:80; - - root /home/sentrium/web/dev.packages.vyos.net/public_html/repositories; - autoindex on; - - server_name _; - - location / { - try_files $uri $uri/ =404; - } - - location ~ /(.*)/conf { - deny all; - } - - location ~ /(.*)/db { - deny all; - } -} -``` - -This will give you HTTP APT repository, like this `http://172.17.17.17/equuleus`. - -To create `--custom-apt-key` you need to export your gpg singing public key, for example: - -```bash -sudo -u jenkins gpg --armor --output /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/apt.gpg.key \ - --export-options export-minimal --export vyos -``` - -This will give you `/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/apt.gpg.key` or -`http://172.17.17.17/apt.gpg.key`. - -If you have the `--vyos-mirror` URL of your own repository and your own singing key `--custom-apt-key` then you -can use these two pieces to build ISO from your own package mirror. - -How to build ISO --- - -Complete instructions to illustrate how to build ISO via Docker method included with the extra options outline above. - -We assume you already have Docker running if not follow -the [official instructions](https://docs.docker.com/engine/install/debian/) for your OS. - -**Obtain the sources:** - -```bash -git clone https://github.com/dd010101/vyos-build -cd vyos-build -``` - -**Select branch of your choice** - -For all following steps will use BRANCH environment variable since the branch repeats a lot. - -```bash -export BRANCH=equuleus -``` - -or - -```bash -export BRANCH=sagitta -``` - -**Switch to branch** - -```bash -git checkout "$BRANCH" -``` - -**Clear previous build resources (if any)** - -There is `make clean` but that doesn't always clean everything and may produce corrupted build environment. -The `make clean` is trying to remove specific parts of `build` directory, but it doesn't always do so correctly. -This happens mainly if you switch branches - that's why it's better to always delete the whole `build` directory. - -```bash -rm -rf build/ -``` - -**Obtain vyos-build docker container** - -You can reuse your docker container image if you already -have [custom registry](#launch-local-registry-and-set-it-so-it-always-runs-when-docker-runs) -with [patched vyos-build docker container](#build-patched-vyos-build-docker-images). - -Change the registry URL if you build on other machine. - -```bash -docker pull "172.17.17.17:5000/vyos/vyos-build:$BRANCH" -docker tag "172.17.17.17:5000/vyos/vyos-build:$BRANCH" "vyos/vyos-build:$BRANCH" -``` - -If you don't have custom registry then build the container - this will take a while: - -```bash -docker build -t "vyos/vyos-build:$BRANCH" docker -``` - -You should rebuild the container from time to time - not very frequently but sometimes the build will break -if you have too old container. - -**Obtain apt singing key for your custom mirror** - -```bash -wget http://172.17.17.17/apt.gpg.key -O /tmp/apt.gpg.key -``` - -**Launch the vyos-build docker container** - -This is the usual run command from official documentation, we need to add extra mount for our apt singing key -for later use via `-v "/tmp/apt.gpg.key:/opt/apt.gpg.key"`. - -The docker run command will mount current working directory for use inside the container that's why you need to -execute this command inside the `vyos-build` directory (that is the GIT repository you cloned above). -You can also replace the `-v "$(pwd)":/vyos` with static path if you like not to depend on current directory -(for example `-v /opt/vyos-build:/vyos`). - -```bash -docker run --rm -it \ - -v "$(pwd)":/vyos \ - -v "/tmp/apt.gpg.key:/opt/apt.gpg.key" \ - -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ - "vyos/vyos-build:$BRANCH" bash -``` - -Now we should be inside the container. - -**Configure and build the ISO** - -Command for configuring changed over time, equuleus has `./configure`, sagitta has `./build-vyos-image iso` instead. - -You may want to customize the configuration options, see what is available: - -For equuleus: - -```bash -sudo ./configure --help -``` - -For sagitta: - -```bash -sudo ./build-vyos-image --help -``` - -We need to add extra two options to configure `--vyos-mirror` and `--custom-apt-key`. We also add smoketest -via `--custom-package vyos-1x-smoketest` for good measure. - -Here are examples - please adjust options to your liking: - -For equuleus: - -```bash -sudo ./configure --architecture amd64 --build-by "myself@localhost" \ - --build-type release --version "1.3.x" \ - --vyos-mirror http://172.17.17.17/equuleus --custom-apt-key /opt/apt.gpg.key \ - --debian-elts-mirror http://172.17.17.17:3142/deb.freexian.com/extended-lts \ - --custom-package vyos-1x-smoketest \ - && sudo make iso -``` - -For sagitta: - -```bash -sudo ./build-vyos-image iso --architecture amd64 --build-by "myself@localhost" \ - --build-type release --version "1.4.x" \ - --vyos-mirror http://172.17.17.17/sagitta --custom-apt-key /opt/apt.gpg.key \ - --custom-package vyos-1x-smoketest -``` - -This will take a while - after all is done then you can `exit` the container and you should have -`build/live-image-amd64.hybrid.iso`. - Something is wrong -- @@ -867,7 +147,7 @@ You may face situation when Jenkins build may fail or doesn't produce .deb packa with unmet dependencies. Sometimes the Jenkins build fails for temporary reason like network/server issue, thus find the package/job/branch in question (like linux-kernel/sagitta) and retry the build with **Build now**. -Sometimes the build steps/instructions change over time (mainly due to changes on VyOS side), and thus for example +Sometimes the build steps/instructions change over time (mainly due to changes on VyOS side), and thus for example you need to change or add additional package. If you are using automated scripts then make sure you have the latest version, if not then download scripts again and rerun all them. @@ -888,10 +168,10 @@ configuration is in its place. Smoketest -- -It's not bad idea to sanity check your ISO image. This is why VyOS has testing procedure called smoketest. -This test checks mainly configuration, it's not complete test, it will only tell you if something -is very wrong, passed test doesn't mean image will be necessarily fully functional - with that said it's still useful, -just don't put too much trust in it. +VyOS has testing procedure called smoketest. This test checks mainly configuration, it's not complete test, +it will only tell you if something is very wrong, passed test doesn't mean image will be necessarily fully +functional - with that said it's still useful somewhat, just don't put too much trust in it. It also has tendencies +to generate a lot of false positives, beware. You will need host that supports virtualization and thus can run KVM. Main test (`make testd`) takes around two hours to complete, additional tests are significantly faster. @@ -974,76 +254,3 @@ some tests will fail. The test is expecting 8 or more network ports. You do it b I'm not sure if all 8 are required but with few the test will fail for sure. Also make sure you have <= 4 cores/threads and 4GB or more of RAM. If you have more cores/threads then you need more RAM as well, try to keep it 1:1. Smoketest eats a lot of RAM and more so if you have more cores/threads. - -Multibranch Pipelines (manual) --- - -Use + button on Jenkins dashboard to add Multibranch Pipeline. Each Jenkinsfile needs its own Multibranch Pipeline, -the setup is the same for all packages, and you just adjust location of Jenkinsfile and/or GIT repository to whatever -you want to build. See packages info bellow for list of all GIT repository and location their Jenkinsfile. - -It makes sense to configure one pipeline and then use "Copy from" and just change the Jenkinsfile location and/or GIT -repository. Start with something small like dropbear, and after you verify your setup works then try the bigger stuff -like linux-kernel. - -There are two types of configurations - first you have single shared GIT repository with many Jenkinsfile, like -the https://github.com/vyos/vyos-build.git, thus each pipeline points to different Jenkinsfile, but they share same GIT -url. Rest of the packages have their own GIT repository with single root Jenkinsfile, thus the Jenkinsfile stays -"Jenkinsfile" and you change the GIT url only. The https://github.com/vyos/vyos-build.git repository has also its own -root Jenkinsfile - ignore it since that one is trying to build ISO with default (blocked) apt mirror. - -You can see all packages and information about them in the [packages.md](extras/packages.md) file. - -**Branch Sources -> Add source -> Git** - -> **Project Repository:** https://github.com/vyos/vyos-build.git - -(or any other repository, like https://github.com/vyos/vyos-1x.git) - -You may want to restrict to only branches you care about, thus: - -**Behaviours -> Add -> Filter by name (with regular expression)** - -> *Regular expression:** (equuleus|sagitta) - -**Behaviours -> Add -> Advanced clone behaviours** - -> **Fetch tags:** [✓] - -(leave defaults) - -Advanced clone is required for some packages to obtain all tags (specifically `vyos-cloud-init` and `vyos-1x`). -It doesn't hurt to have advanced clone for everything, thus you can set-it and copy for everything without -worrying about what to use it for. - -**Build Configuration -> Mode (by Jenkinsfile)** - -> **Script Path:** packages/dropbear/Jenkinsfile - -(if you want to build package from vyos/vyos-build repository) - -> **Script Path:** Jenkinsfile - -(or leave just Jenkinsfile if you want to build repository like vyos/vyos-1x where there is just one package) - -**Scan Multibranch Pipeline Triggers** - -> [✓] **Periodically if not otherwise run** -> -> **Interval:** 1 hour - -Jenkins will check the source GIT repository if changes were made and execute automatic build if needed. This -will keep packages up to date. - -Try to build --- - -Now it's possible to select some **Multibranch Pipeline**, select your **branch**, and you can press build button -to see what happens! If all is well you should see .deb appearing in -`/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/`: - -```bash -find /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/ -name '*.deb' -print -``` - -If build fails then click the specific build number and check **Console Output** for hints why it does so. |