diff options
36 files changed, 1439 insertions, 815 deletions
@@ -26,7 +26,7 @@ Required Debian Packages: Debian, requires some extra steps for installing `sphinx`, `sphinx-autobuild` and `sphinx-rtd-theme` packages: -First ensure that phython3 is the default: +First ensure that phython2 & phython3 are installed and phython3 is the default: ```bash python --version ``` @@ -35,19 +35,35 @@ Alternatively, to make python3 the default, revise the following line to point to the relevant 3.x version of the binary on your system: ```bash -sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.... +sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 0 ``` Then follow these steps to install sphinx group of packages: ```bash sudo apt-get install python3-sphinx -sudo apt-get install python3-sphinx-rtd-theme ``` -To build the html, start a webeserver, and view the output: +Although mostly everything uses phython3, But to install this specific +package, make sure that pip points to the python2 version of the package manager: + +```bash +python --version +``` + +Then run: + +```bash +sudo pip install sphinx-rtd-theme +``` + + +Do the following to build the html and start a webeserver: * Run `make livehtml` inside the `docs` folder -* Browse to http://localhost:8000 +Then, to view the live output: +* Browse to http://localhost:8000 +Note: The changes you save to the sources are represented in the live HTML outout +automatically (and almost instantly) without the need to rebuild or refresh manually. ## Docker diff --git a/docker/Dockerfile b/docker/Dockerfile index 5e3095d7..02b0fc26 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -32,6 +32,8 @@ RUN apt-get update && apt-get install -y \ RUN pip3 install Sphinx RUN pip3 install sphinx-rtd-theme RUN pip3 install sphinx-autobuild +RUN pip3 install sphinx-notfound-page + # Cleanup RUN rm -rf /var/lib/apt/lists/* diff --git a/docs/404.rst b/docs/404.rst new file mode 100644 index 00000000..85444615 --- /dev/null +++ b/docs/404.rst @@ -0,0 +1,10 @@ +:orphan: + +Page Not Found +============== + +Sorry, We could not find a page. +Try using the search box or go to the release homepage: + + * `1.2.x (crux) <https://docs.vyos.io/en/crux/>`_ + * `rolling release (equuleus) <https://docs.vyos.io/en/latest/>`_
\ No newline at end of file diff --git a/docs/_static/images/multicast-basic.png b/docs/_static/images/multicast-basic.png Binary files differnew file mode 100644 index 00000000..3d4a3de3 --- /dev/null +++ b/docs/_static/images/multicast-basic.png diff --git a/docs/appendix/cmd-index.rst b/docs/appendix/cmd-index.rst deleted file mode 100644 index 71b556cb..00000000 --- a/docs/appendix/cmd-index.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _cmd-index: - -################# -Command Reference -################# - -Configuration Commands -====================== - -.. _cfg_cmd_list: - -.. cfgcmdlist:: - -Operational Commands -==================== - -.. _op_cmd_list: - -.. opcmdlist:: diff --git a/docs/appendix/examples/index.rst b/docs/appendix/examples/index.rst index 427606de..ca3a6251 100644 --- a/docs/appendix/examples/index.rst +++ b/docs/appendix/examples/index.rst @@ -1,7 +1,7 @@ .. _examples: -Configuration Examples -====================== +Configuration Blueprints +======================== This chapter contains various configuration Examples diff --git a/docs/appendix/examples/tunnelbroker-ipv6.rst b/docs/appendix/examples/tunnelbroker-ipv6.rst index 72828292..868b225f 100644 --- a/docs/appendix/examples/tunnelbroker-ipv6.rst +++ b/docs/appendix/examples/tunnelbroker-ipv6.rst @@ -108,9 +108,9 @@ should be replaced with the information from your `Routed /64` tunnel): set interfaces ethernet eth1 address '2001:470:xxxx:xxxx::1/64' set service router-advert interface eth1 name-server '2001:4860:4860::8888' set service router-advert interface eth1 name-server '2001:4860:4860::8844' - set service router-advert interface eth1 prefix 2001:470:xxxx:xxxx::/64 autonomous-flag - set service router-advert interface eth1 prefix 2001:470:xxxx:xxxx::/64 on-link-flag - set service router-advert interface eth1 prefix 2001:470:xxxx:xxxx::/64 valid-lifetime '2592000' + set service router-advert interface eth1 prefix 2001:470:xxxx:xxxx::/64 + +Please note, 'autonomous-flag' and 'on-link-flag' are enabled by default, 'valid-lifetime' and 'preferred-lifetime' are set to default values of 30 days and 4 hours respectively. This accomplishes a few things: @@ -143,23 +143,19 @@ So, when your LAN is eth1, your DMZ is eth2, your cameras live on eth3, etc: set interfaces ethernet eth1 address '2001:470:xxxx:1::1/64' set service router-advert interface eth1 name-server '2001:4860:4860::8888' set service router-advert interface eth1 name-server '2001:4860:4860::8844' - set service router-advert interface eth1 prefix 2001:470:xxxx:1::/64 autonomous-flag 'true' - set service router-advert interface eth1 prefix 2001:470:xxxx:1::/64 on-link-flag 'true' - set service router-advert interface eth1 prefix 2001:470:xxxx:1::/64 valid-lifetime '2592000' - + set service router-advert interface eth1 prefix 2001:470:xxxx:1::/64 + set interfaces ethernet eth2 address '2001:470:xxxx:2::1/64' set service router-advert interface eth2 name-server '2001:4860:4860::8888' set service router-advert interface eth2 name-server '2001:4860:4860::8844' - set service router-advert interface eth2 prefix 2001:470:xxxx:2::/64 autonomous-flag 'true' - set service router-advert interface eth2 prefix 2001:470:xxxx:2::/64 on-link-flag 'true' - set service router-advert interface eth2 prefix 2001:470:xxxx:2::/64 valid-lifetime '2592000' + set service router-advert interface eth2 prefix 2001:470:xxxx:2::/64 set interfaces ethernet eth3 address '2001:470:xxxx:3::1/64' set service router-advert interface eth3 name-server '2001:4860:4860::8888' set service router-advert interface eth3 name-server '2001:4860:4860::8844' - set service router-advert interface eth3 prefix 2001:470:xxxx:3::/64 autonomous-flag 'true' - set service router-advert interface eth3 prefix 2001:470:xxxx:3::/64 on-link-flag 'true' - set service router-advert interface eth3 prefix 2001:470:xxxx:3::/64 valid-lifetime '2592000' + set service router-advert interface eth3 prefix 2001:470:xxxx:3::/64 + +Please note, 'autonomous-flag' and 'on-link-flag' are enabled by default, 'valid-lifetime' and 'preferred-lifetime' are set to default values of 30 days and 4 hours respectively. Firewall ======== diff --git a/docs/appendix/vyos-on-baremetal.rst b/docs/appendix/vyos-on-baremetal.rst index 0116ddc1..db618431 100644 --- a/docs/appendix/vyos-on-baremetal.rst +++ b/docs/appendix/vyos-on-baremetal.rst @@ -106,11 +106,25 @@ dual front cover. Extension Modules ^^^^^^^^^^^^^^^^^ +WiFi +"""" + +Refer to :ref:`wireless-interface` for additional information, below listed modules +have been tested successfully on this Hardware platform: + * Compex WLE900VX mini-PCIe WiFi module, only supported in mPCIe slot 1. - Refer to :ref:`wireless-interface` for additional information. -* Sierra Wireless MC7710 LZE miniPCIe card is supported by VyOS 1.3 (equuleus). - Refer to :ref:`wwan-interface` for additional information. +WWAN +"""" + +Refer to :ref:`wwan-interface` for additional information, below listed modules +have been tested successfully on this Hardware platform using VyOS 1.3 (equuleus): + +* Sierra Wireless AirPrime MC7304 miniPCIe card (LTE) +* Sierra Wireless AirPrime MC7430 miniPCIe card (LTE) +* Sierra Wireless AirPrime MC7455 miniPCIe card (LTE) +* Sierra Wireless AirPrime MC7710 miniPCIe card (LTE) +* Huawei ME909u-521 miniPCIe card (LTE) VyOS 1.2 (crux) --------------- diff --git a/docs/appendix/vyos-on-clouds.rst b/docs/appendix/vyos-on-clouds.rst index d69b2e1b..33b7011e 100644 --- a/docs/appendix/vyos-on-clouds.rst +++ b/docs/appendix/vyos-on-clouds.rst @@ -98,6 +98,13 @@ Deploy VyOS on Azure. ssh -i ~/.ssh/vyos_azure vyos@203.0.113.3 vyos@vyos-doc-r1:~$ +Add interface +------------- + +If instance was deployed with one **eth0** ``WAN`` interface and want to add new one. +To add new interface an example **eth1** ``LAN`` you need shutdown the instance. Attach the interface in the Azure portal and then start the instance. + +.. NOTE:: Azure does not allow you attach interface when the instance in the **Running** state. References ---------- @@ -148,6 +155,7 @@ Click **Add item** and paste your public ssh key. Click ``Save``. .. figure:: /_static/images/cloud-gcp-05.png 6. Connect to the instance. SSH key was generated in the first step. + .. code-block:: none ssh -i ~/.ssh/vyos_gcp vyos@203.0.113.3 diff --git a/docs/appendix/vyos-on-virtual-environments.rst b/docs/appendix/vyos-on-virtual-environments.rst index 59edcfda..eed82390 100644 --- a/docs/appendix/vyos-on-virtual-environments.rst +++ b/docs/appendix/vyos-on-virtual-environments.rst @@ -1,10 +1,12 @@ .. _vyos-on-virtual-environments: +############################### Running in Virtual Environments -###################### +############################### +**************** Libvirt Qemu/KVM -***************** +**************** Libvirt is an open-source API, daemon and management tool for managing platform virtualization. There are several ways to deploy VyOS on libvirt kvm. Use Virt-manager and native CLI. @@ -159,10 +161,12 @@ Download predefined VyOS.qcow2 image for ``KVM`` .. figure:: /_static/images/virt-libvirt-qc-03.png +******* Proxmox -***************** +******* References ----------- +========== + https://www.proxmox.com/en/proxmox-ve diff --git a/docs/command-list-configuration.rst b/docs/command-list-configuration.rst new file mode 100644 index 00000000..a1c370e6 --- /dev/null +++ b/docs/command-list-configuration.rst @@ -0,0 +1,9 @@ +############ +Command List +############ + +Configuration level commands: + +.. _cfg_cmd_list: + +.. cfgcmdlist:: diff --git a/docs/command-list-operation.rst b/docs/command-list-operation.rst new file mode 100644 index 00000000..3dabd653 --- /dev/null +++ b/docs/command-list-operation.rst @@ -0,0 +1,9 @@ +############# +Commands List +############# + +Operational level commands: + +.. _op_cmd_list: + +.. opcmdlist:: diff --git a/docs/conf.py b/docs/conf.py index b63d4a03..bb32aa33 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -44,6 +44,7 @@ extensions = ['sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.ifconfig', 'sphinx.ext.graphviz', + 'notfound.extension', 'vyos' ] diff --git a/docs/contributing/build-vyos.rst b/docs/contributing/build-vyos.rst index d72f4b07..09f8150d 100644 --- a/docs/contributing/build-vyos.rst +++ b/docs/contributing/build-vyos.rst @@ -1,168 +1,278 @@ .. _build: -Building VyOS -============= +########## +Build VyOS +########## + +************* +Prerequisites +************* + +There are different ways you can build VyOS. + +Building using a :ref:`build_docker` container, although not the only way, is the +easiest way as all dependencies are managed for you. However, you can also +set up your own build machine and run a :ref:`build_native`. + +.. note:: Starting with VyOS 1.2 the release model of VyOS has changed. VyOS + is now **free as in speech, but not as in beer**. This means that while + VyOS is still an open source project, the release ISOs are no longer free + and can only be obtained via subscription, or by contributing to the + community. + + The source code remains public and an ISO can be built using the process + outlined in this chapter. This will guide you though the process of building a VyOS ISO using Docker_. This process has been tested on clean installs of Debian Jessie, Stretch, and Buster. -.. note:: Starting with VyOS 1.2 the release model of VyOS has changed. - VyOS is now **free as in speech, but not as in beer**. This means - that while VyOS is still an open source project, the release ISOs are no - longer free and can only be obtained via subscription, or by contributing to - the community. +.. _build_docker: - The source code remains public and an ISO can be built - using the process outlined here. +Docker +====== Installing Docker_ and prerequisites: .. code-block:: none - $ apt-get update - $ apt-get install -y apt-transport-https ca-certificates curl \ - gnupg2 software-properties-common + $ sudo apt-get update + $ sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - - $ add-apt-repository "deb [arch=amd64] \ - https://download.docker.com/linux/debian $(lsb_release -cs) stable" - $ apt-get update - $ apt-get install -y docker-ce + $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" + $ sudo apt-get update + $ sudo apt-get install -y docker-ce -To be able to use Docker_ without ``sudo``, the current non-root user can be added to the -``docker`` group by calling: ``usermod -aG docker yourusername`` +To be able to use Docker_ without ``sudo``, the current non-root user must be +added to the ``docker`` group by calling: ``sudo usermod -aG docker +yourusername``. -.. note:: Doing so grants privileges equivalent to the ``root`` user! It is recommended to remove the non-root user from the ``docker`` group after building the VyOS ISO. See also https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user +.. hint:: Doing so grants privileges equivalent to the ``root`` user! It is + recommended to remove the non-root user from the ``docker`` group after + building the VyOS ISO. See also `Docker as non-root`_. .. note:: The build process needs to be built on a local file system, building on SMB or NFS shares will result in the container failing to build properly! VirtualBox Drive Share is also not an option as block device operations are not implemented and the drive is always mounted as "nodev" -Build Docker Container ----------------------- +Build Container +--------------- The container can built by hand or by fetching the pre-built one from DockerHub. -Using the pre-built VyOS DockerHub organisation (https://hub.docker.com/u/vyos) -will ensure that the container is always up-to-date. A rebuild is triggered once -the container changes (please note this will take 2-3 hours after pushing to -the vyos-build repository). +Using the pre-built containers from the `VyOS DockerHub organisation`_ will +ensure that the container is always up-to-date. A rebuild is triggered once the +container changes (please note this will take 2-3 hours after pushing to the +vyos-build repository). + +.. note: If you are using the pre-built container, it will be automatically + downloaded from DockerHub if it is not found on your local machine when + you build the ISO. + +Dockerhub +^^^^^^^^^ -To download the container from DockerHub run: +To manually download the container from DockerHub, run: .. code-block:: none - $ docker pull vyos/vyos-build:crux # for the LTS version - $ docker pull vyos/vyos-build:current # for the current version + $ docker pull vyos/vyos-build:crux # For VyOS 1.2 + $ docker pull vyos/vyos-build:current # For rolling release + +Build from source +^^^^^^^^^^^^^^^^^ The container can also be built directly from source: .. code-block:: none + # For VyOS 1.2 (crux) + $ git clone -b crux --single-branch https://github.com/vyos/vyos-build + # For VyOS 1.3 (equuleus, current) $ git clone -b current --single-branch https://github.com/vyos/vyos-build + $ cd vyos-build - $ docker build -t vyos/vyos-build docker + $ docker build -t vyos/vyos-build:crux docker # For VyOS 1.2 + $ docker build -t vyos/vyos-build docker # For rollign release + +.. note:: Since VyOS has switched to Debian (10) Buster in its ``current`` branch, + you will require individual container for `current` and `crux` builds. + +Tips and Tricks +--------------- + +You can create yourself some handy Bash aliases to always launch the latest - +per release train (`current` or `crux`) - container. Add the following to your +``.bash_aliases`` file: + +.. code-block:: none + + alias vybld='docker pull vyos/vyos-build:current && docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "$HOME/.gitconfig":/etc/gitconfig \ + -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ + -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + vyos/vyos-build:current bash' + + alias vybld_crux='docker pull vyos/vyos-build:crux && docker run --rm -it \ + -v "$(pwd)":/vyos \ + -v "$HOME/.gitconfig":/etc/gitconfig \ + -v "$HOME/.bash_aliases":/home/vyos_bld/.bash_aliases \ + -v "$HOME/.bashrc":/home/vyos_bld/.bashrc \ + -w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ + -e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \ + vyos/vyos-build:crux bash' + +Now you are prepared with two new aliases ``vybld`` and ``vybld_crux`` to spwan +your development containers in your current working directory. + +.. _build_native: + +Native Build +============ + +To build VyOS natively you require a properly configured build host with the +following Debian versions installed: + +- Debian Jessie for VyOS 1.2 (crux) +- Debian Buster for VyOS 1.3 (equuleus, current) - aka the rolling release + +To start, clone the repository to your local machine: + +.. code-block:: none + + # For VyOS 1.2 (crux) + $ git clone -b crux --single-branch https://github.com/vyos/vyos-build + + # For VyOS 1.3 (equuleus, current) + $ git clone -b current --single-branch https://github.com/vyos/vyos-build -.. note:: The container is automatically downloaded from Dockerhub if it is not - found on your local machine when the below command is executed. +For the packages required, you can refer to the ``docker/Dockerfile`` file +in the repository_. The ``./configure`` script will also warn you if any +dependencies are missing. + +Once you have the required dependencies installed, you may proceed with the +steps descirbed in :ref:`build_iso`. -.. note:: We require one container per build branch, this means that the used - container in ``crux`` and ``current`` can and will differ once VyOS makes - the move towards Debian (10) Buster. .. _build_iso: +********* Build ISO ---------- - -After the container is generated either manually or fetched from DockerHub, -a fresh build of the VyOS ISO can begin. +********* -If you pulled the image from DockerHub, you need to clone the repository to -your local machine: +Now as you are aware of the prerequisites we can continue and build our own +ISO from source. For this we have to fetch the latest source code from GitHub. +Please note as this will differ for both `current` and `crux`. .. code-block:: none + # For VyOS 1.2 (crux) + $ git clone -b crux --single-branch https://github.com/vyos/vyos-build + + # For VyOS 1.3 (equuleus, current) $ git clone -b current --single-branch https://github.com/vyos/vyos-build -After cloning, change directory to the ``vyos-build`` directory and run: +Now a fresh build of the VyOS ISO can begin. Change directory to the ``vyos-build`` directory and run: .. code-block:: none $ cd vyos-build + # For VyOS 1.2 (crux) + $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:crux bash + + # For VyOS 1.3 (equuleus, current) $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build bash - vyos_bld@d4220bb519a0:/vyos# ./configure --architecture amd64 \ - --build-by "your@email.tld" \ - --build-type release --version 1.2.0 + +Start the build: + +.. code-block:: none + + vyos_bld@d4220bb519a0:/vyos# ./configure --architecture amd64 --build-by "j.randomhacker@vyos.io" vyos_bld@d4220bb519a0:/vyos# sudo make iso -When the build is successful, the resulting iso can be found inside the ``build`` -directory. +When the build is successful, the resulting iso can be found inside the ``build`` +directory as ``live-image-[architecture].hybrid.iso``. + +Good luck! + +.. hint:: Attempting to use the Docker build image on MacOS will fail as + Docker does not expose all the filesystem feature required to the container. + Building within a VirtualBox server on Mac however possible. -.. note:: Attempting to use the docker build image on MacOS or Windows will fail - as docker does not expose all the filesystem feature required to the container. - Building within a VirtualBox server on Mac or Windows is however possible. +.. hint:: Building VyOS on Windows WSL2 with Docker integrated into WSL2 will work + like a charm. No problems are known so far! -To select the container you want to run, you need to specify the branch you are -interested in, this can be easily done by selecting the appropriate container -image: +.. _build source: -* VyOS 1.2 (crux) use ``vyos/vyos-build:crux`` -* VyOS rolling release you should use ``vyos/vyos-build`` which will always - refer to the latest image. -Customisation -^^^^^^^^^^^^^ +.. _customize: + +Customize +========= This ISO can be customized with the following list of configure options. The full and current list can be generated with ``./configure --help``: .. code-block:: none - -h, --help show this help message and exit - --architecture ARCHITECTURE - Image target architecture (amd64 or i586 or armhf) - --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) - --custom-package CUSTOM_PACKAGES - Custom packages to install from repositories - --build-type BUILD_TYPE - Build type, release or development - --debian-security-mirror DEBIAN_SECURITY_MIRROR - Debian security updated mirror - --version VERSION Version number (release builds only) - --debian-mirror DEBIAN_MIRROR - Debian repository mirror for ISO build - --vyos-mirror VYOS_MIRROR - VyOS package mirror - --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR - Debian repository mirror for pbuilder env bootstrap - --debug Enable debug output - --custom-apt-entry CUSTOM_APT_ENTRY - Custom APT entry - --custom-apt-key CUSTOM_APT_KEY - Custom APT key file - -The successfully built ISO should now be in the `build/` directory. - -Good luck! - -.. note:: The build process does not differentiate when building a ``crux`` ISO - or ``rolling`` image. Make sure to choose the matching container for the - version of VyOS that is being built. + $ ./configure --help + usage: configure [-h] [--architecture ARCHITECTURE] [--build-by BUILD_BY] + [--debian-mirror DEBIAN_MIRROR] + [--debian-security-mirror DEBIAN_SECURITY_MIRROR] + [--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR] + [--vyos-mirror VYOS_MIRROR] [--build-type BUILD_TYPE] + [--version VERSION] [--build-comment BUILD_COMMENT] [--debug] + [--custom-apt-entry CUSTOM_APT_ENTRY] + [--custom-apt-key CUSTOM_APT_KEY] + [--custom-package CUSTOM_PACKAGE] -Development -^^^^^^^^^^^ + optional arguments: + -h, --help show this help message and exit + --architecture ARCHITECTURE + Image target architecture (amd64 or i386 or armhf) + --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) + --debian-mirror DEBIAN_MIRROR + Debian repository mirror for ISO build + --debian-security-mirror DEBIAN_SECURITY_MIRROR + Debian security updates mirror + --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR + Debian repository mirror for pbuilder env bootstrap + --vyos-mirror VYOS_MIRROR + VyOS package mirror + --build-type BUILD_TYPE + Build type, release or development + --version VERSION Version number (release builds only) + --build-comment BUILD_COMMENT + Optional build comment + --debug Enable debug output + --custom-apt-entry CUSTOM_APT_ENTRY + Custom APT entry + --custom-apt-key CUSTOM_APT_KEY + Custom APT key file + --custom-package CUSTOM_PACKAGE + Custom package to install from repositories + +.. _build_custom_packages: + +Packages +======== If you are brave enough to build yourself an ISO image containing any modified package from our GitHub organisation - this is the place to be. -Building an ISO with a customized package is in no way different then building -a regular (customized or not) ISO image. Simply place your modified `*.deb` -package inside the `packages` folder within `vyos-build`. You may need to create -the folder in advance. +Any "modified" package may refer to an altered version of e.g. vyos-1x package +that you would like to test before filing a PullRequest on GitHub. + +Building an ISO with any customized package is in no way different then +building a regular (customized or not) ISO image. Simply place your modified +`*.deb` package inside the `packages` folder within `vyos-build`. The build +process will then pickup your custom package and integrate it into your ISO. Troubleshooting -^^^^^^^^^^^^^^^ +=============== Debian APT is not very verbose when it comes to errors. If your ISO build breaks for whatever reason and you supect its a problem with APT dependencies or @@ -187,273 +297,103 @@ during ISO build. """ -.. _build_packages: - -Build packages --------------- - -VyOS requires a bunch of packages which are VyOS specific and thus can not be -found in any Debian Upstream mirrror. Those packages can be found at the VyOS -GitHub project (https://github.com/vyos) and there is a nice helper script -available to build and list those individual packages. +Virtualization Platforms +======================== -`scripts/build-packages` provides an easy interface to automate the process -of building all VyOS related packages that are not part of the upstream Debian -version. Execute it in the root of the `vyos-build` directory to start -compilation. +QEMU +---- -.. code-block:: none - - $ scripts/build-packages -h - usage: build-packages [-h] [-c | -k | -f] [-v] [-l] [-b BUILD [BUILD ...]] - [-p] [--blacklist BLACKLIST [BLACKLIST ...]] - - optional arguments: - -h, --help show this help message and exit - -c, --clean Re-clone required Git repositories - -k, --keep Keep modified Git repositories - -f, --fetch Fetch sources only, no build - -v, --verbose Increase logging verbosity for each occurance - -l, --list-packages List all packages to build - -b BUILD [BUILD ...], --build BUILD [BUILD ...] - Whitespace separated list of packages to build - -p, --parallel Build on all CPUs - --blacklist BLACKLIST [BLACKLIST ...] - Do not build/report packages when calling --list - -Git repositoriers are automatically fetched and build on demand. If you want to -work offline you can fetch all source code first with the `-f` option. - -The easiest way to compile is with the above mentioned Docker -container, it includes all dependencies for compiling supported packages. +Run following command after building the ISO image. .. code-block:: none - $ docker run --rm -it -v $(pwd):/vyos -w /vyos \ - --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - vyos-build scripts/build-packages - -.. note:: `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is required to build the - `vyos-strongswan` package - -.. note:: Prior to executing this script you need to create or build the Docker - container and checkout all packages you want to compile. + $ make qemu -Building single package(s) -^^^^^^^^^^^^^^^^^^^^^^^^^^ +VMware +------ -To build a single package use the same script as above but specify packages with -`-b`: - -Executed from the root of `vyos-build` +Run following command after building the QEMU image. .. code-block:: none - $ docker run --rm -it -v $(pwd):/vyos -w /vyos \ - --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - vyos-build scripts/build-packages -b <package> - -.. note:: `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is only needed when - building `vyos-strongswan` and can be ignored on other packages. + $ make vmware -.. note:: `vyos-strongswan` will only compile on a Linux system, running on - macOS or Windows might result in a unittest deadlock (it never exits). +.. _build_packages: -Building single packages from your own repositories -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +******** +Packages +******** -You can also build packages that are not from the default git repositories, -for example from your own forks of the official vyos repositories. +VyOS itself comes with a bunch of packages which are specific to our system and +thus can not be found in any Debian mirrror. Those packages can be found at the +`VyOS GitHub project`_ in their source format can can easily be compiled into +a custom Debian (`*.deb`) package. -First create a directory "packages" at the top level of the vyos-build -repository and clone your package into it (creating a subdirectory with the -package contents). Then checkout the correct branch or commit you want to build -before building the package. +The easiest way to compile your package is with the above mentioned +:ref:`build_docker` container, it includes all required dependencies for +all VyOS related packages. -Example using `git@github.com:myname/vyos-1x.git` repository to build vyos-1x: +Assume we want to build the vyos-1x package on our own and modify it to our +needs. We first need to clone the repository from GitHub. .. code-block:: none - $ mkdir packages - $ cd packages - $ git clone git@github.com:myname/vyos-1x.git - $ cd .. - $ docker run --rm -it -v $(pwd):/vyos -w /vyos \ - --sysctl net.ipv6.conf.lo.disable_ipv6=0 \ - vyos-build scripts/build-packages -b vyos-1x - -.. note:: You need to git pull manually after you commit to the remote and - before rebuilding, the local repository won't be updated automatically. - -.. warning:: Any packages in the packages directory will be added to the iso - during build, replacing the upstream ones. Make sure you delete them (both - the source directories and built deb packages) if you want to build an iso - from purely upstream packages. - - -.. _upstream_packages: - -Upstream packages ------------------ - -Many base system packages are pulled straight from Debian's main and contrib -repositories, but there are exceptions. - -This chapter lists those exceptions and gives you a brief overview what we -have done on those packages. If you only want to build yourself a fresh ISO -you can completely skip this chapter. It may become interesting once you have -a VyOS deep dive. - -vyos-netplug -^^^^^^^^^^^^ - -Due to issues in the upstream version that sometimes set interfaces down, a -modified version is used. - -The source is located at https://github.com/vyos/vyos-netplug - -In the future, we may switch to using systemd infrastructure instead. Building -it doesn't require a special procedure. - -keepalived -^^^^^^^^^^ - -Keepalived normally isn't updated to newer feature releases between Debian -versions, so we are building it from source. - -Debian does keep their package in git, but it's upstream tarball imported into -git without its original commit history. To be able to merge new tags in, we -keep a fork of the upstream repository with packaging files imported from -Debian at https://github.com/vyos/keepalived-upstream - -strongswan -^^^^^^^^^^ - -Our StrongSWAN build differs from the upstream: - -- strongswan-nm package build is disabled since we don't use NetworkManager -- Patches for DMVPN are merged in - -The source is at https://github.com/vyos/vyos-strongswan - -DMVPN patches are added by this commit: -https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226 - -Our op mode scripts use the python-vici module, which is not included in -Debian's build, and isn't quite easy to integrate in that build. For this -reason we debianize that module by hand now, using this procedure: - -0. Install https://pypi.org/project/stdeb/ -1. `cd vyos-strongswan` -2. `./configure --enable-python-eggs` -3. `cd src/libcharon/plugins/vici/python` -4. `make` -5. `python3 setup.py --command-packages=stdeb.command bdist_deb` - -The package ends up in deb_dist dir. + $ git clone https://github.com/vyos/vyos-1x -ppp -^^^ +Build +===== -Properly renaming PPTP and L2TP interfaces to pptpX and l2tpX from generic and -non-informative pppX requires a patch that is neither in the upstream nor in -Debian. +Launch Docker container and build package -We keep a fork of Debian's repo at https://github.com/vyos/ppp-debian - -The patches for pre-up renaming are: - -* https://github.com/vyos/ppp-debian/commit/e728180026a051d2a96396276e7e4ae -* https://github.com/vyos/ppp-debian/commit/f29ba8d9ebb043335a096d70bcd07e9 - -Additionally, there's a patch for reopening the log file to better support -logging to files, even though it's less essential: -https://github.com/vyos/ppp-debian/commit/dd2ebd5cdcddb40230dc4cc43d374055f - -The patches were written by Stephen Hemminger back in the Vyatta times. - -mdns-repeater -^^^^^^^^^^^^^ - -This package doesn't exist in Debian. A debianized fork is kept at -https://github.com/vyos/mdns-repeater - -No special build procedure is required. - -udp-broadcast-relay -^^^^^^^^^^^^^^^^^^^ - -This package doesn't exist in Debian. A debianized fork is kept at -https://github.com/vyos/udp-broadcast-relay - -No special build procedure is required. - -Linux kernel -^^^^^^^^^^^^ - -In the past a fork of the Kernel source code was kept at the well-known -location of https://github.com/vyos/vyos-kernel - where it is kept for history. - -Nowadays the Kernel we use is the upstream source code which is patched -with two additional patches from the good old Vyatta times which never made it -into the mainstream Kernel. The patches can be found here: -https://github.com/vyos/vyos-build-kernel/tree/master/patches/kernel and are -automatically applied to the Kernel by the Jenkins Pipeline which is used to -generate the Kernel binaries. - -The Pipeline script not only builds the Kernel with the configuration named -``x86_64_vyos_defconfig`` which is located in the vyos-build-kernel repository, -too - but in addition also builds some Intel out-of-tree drivers, WireGuard -(as long it is not upstreamed) and Accel-PPP. - -The ``Jenkinsfile`` tries to be as verbose as possible on each individual build -step. - -Linux Firmware -^^^^^^^^^^^^^^ - -More and more hardware cards require an additional firmware which is not open -source. The Kernel community hosts a special linux-firmware Git repository -with all available binary files which can be loaded by the Kernel. - -The ``vyos-build`` repository fetches a specific commit of the linux-firmware -repository and embeds those binaries into the resulting ISO image. This step is -done in the ``data/live-build-config/hooks/live/40-linux-firmware.chroot`` file. +.. code-block:: none -If the firmware needs to be updated it is sufficient to just exchange the Git -commit id we reference in our build. + # For VyOS 1.3 (equuleus, current) + $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build bash -Intel NIC drivers -^^^^^^^^^^^^^^^^^ + # Change to source directory + $ cd vyos-1x -We do not make use of the building Intel NIC drivers except for e1000e. Main -reason is that the out of tree Intel drivers seem be perform a bit better, -e.q. have proper receive-side-scaling and multi-queue support. + # Build DEB + $ dpkg-buildpackage -uc -us -tc -b -Drivers are build as part of the Kernel Pipeline - read above. +After a minute or two you will find the generated DEB packages next to the vyos-1x +source directory: -Accel-PPP -^^^^^^^^^ +.. code-block:: none -Accel-PPP used to be an upstream fork for quite some time but now has been -converted to make use of the upstream source code and build system. + # ls -al ../vyos-1x*.deb + -rw-r--r-- 1 vyos_bld vyos_bld 567420 Aug 3 12:01 ../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb + -rw-r--r-- 1 vyos_bld vyos_bld 3808 Aug 3 12:01 ../vyos-1x-vmware_1.3dev0-1847-gb6dcb0a8_amd64.deb -It is build as part of the Kernel Pipeline - read above. +Install +======= -hvinfo -^^^^^^ +To take your newly created package on a test drive you can simply SCP it to a +running VyOS instance and install the new `*.deb` package over the current +running one. -A fork with packaging changes for VyOS is kept at https://github.com/vyos/hvinfo +Just install using the following commands: -The original repo is at https://github.com/dmbaturin/hvinfo +.. code-block:: none -It's an Ada program and requires GNAT and gprbuild for building, dependencies -are properly specified so just follow debuild's suggestions. + vyos@vyos:~$ dpkg --install /tmp/vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb + (Reading database ... 58209 files and directories currently installed.) + Preparing to unpack .../vyos-1x_1.3dev0-1847-gb6dcb0a8_all.deb ... + Unpacking vyos-1x (1.3dev0-1847-gb6dcb0a8) over (1.3dev0-1847-gb6dcb0a8) ... + Setting up vyos-1x (1.3dev0-1847-gb6dcb0a8) ... + Processing triggers for rsyslog (8.1901.0-1) ... -Per-file modifications -^^^^^^^^^^^^^^^^^^^^^^ +You can also place the generated `*.deb` into your ISO build environment to +include it in a custom iso, see :ref:`build_custom_packages` for more +information. -vyos-replace package replaces the upstream dhclient-script with a modified -version that is aware of the VyOS config. +.. warning:: Any packages in the packages directory will be added to the iso + during build, replacing the upstream ones. Make sure you delete them (both + the source directories and built deb packages) if you want to build an iso + from purely upstream packages. .. _Docker: https://www.docker.com +.. _`Docker as non-root`: https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user +.. _VyOS DockerHub organisation: https://hub.docker.com/u/vyos +.. _repository: https://github.com/vyos/vyos-build +.. _VyOS GitHub project: https://github.com/vyos diff --git a/docs/contributing/debugging.rst b/docs/contributing/debugging.rst new file mode 100644 index 00000000..ac2e0510 --- /dev/null +++ b/docs/contributing/debugging.rst @@ -0,0 +1,146 @@ +.. _debugging: + +######### +Debugging +######### + +There are two flags available to aid in debugging configuration scripts. +Since configuration loading issues will manifest during boot, the flags are +passed as kernel boot parameters. + +System Startup +============== + +The system startup can be debugged (like loading in the configuration +file from ``/config/config.boot``. This can be achieve by extending the +Kernel command-line in the bootloader. + +Kernel +------ + +* ``vyos-debug`` - Adding the parameter to the linux boot line will produce + timing results for the execution of scripts during commit. If one is seeing + an unexpected delay during manual or boot commit, this may be useful in + identifying bottlenecks. The internal flag is ``VYOS_DEBUG``, and is found + in vyatta-cfg_. Output is directed to ``/var/log/vyatta/cfg-stdout.log``. + +* ``vyos-config-debug`` - During development, coding errors can lead to a + commit failure on boot, possibly resulting in a failed initialization of the + CLI. In this circumstance, the kernel boot parameter ``vyos-config-debug`` + will ensure access to the system as user ``vyos``, and will log a Python + stack trace to the file ``/tmp/boot-config-trace``. + File ``boot-config-trace`` will generate only if config loaded with a failure status. + +Live System +=========== + +A number of flags can be set up to change the behaviour of VyOS at runtime. +These flags can be toggled using either environment variables or creating +files. + +For each feature, a file called ``vyos.feature.debug`` can be created to +toggle the feature on. If a parameter is required it can be placed inside +the file as its first line. + +The file can be placed in ``/tmp`` for one time debugging (as the file +will be removed on reboot) or placed in '/config' to stay permanently. + +For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable +interface debugging. + +It is also possible to set up the debugging using environment variables. +In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG. + +For example running, ``export VYOS_IFCONFIG_DEBUG=""`` on your vash, +will have the same effect as ``touch /tmp/vyos.ifconfig.debug``. + +* ``ifconfig`` - Once set, all commands used, and their responses received + from the OS, will be presented on the screen for inspection. + +* ``command`` - Once set, all commands used, and their responses received + from the OS, will be presented on the screen for inspection. + +* ``developer`` - Should a command fail, instead of printing a message to the + user explaining how to report issues, the python interpreter will start a + PBD post-mortem session to allow the developer to debug the issue. As the + debugger will wait from input from the developer, it has the capacity to + prevent a router to boot and therefore should only be permanently set up + on production if you are ready to see the OS fail to boot. + +* ``log`` - In some rare cases, it may be useful to see what the OS is doing, + including during boot. This option sends all commands used by VyOS to a + file. The default file is ``/tmp/full-log`` but it can be changed. + +Config Migration Scripts +------------------------ + +When writing a new configuration migrator it may happen that you see an error +when you try to invoke it manually on a development system. This error will +look like: + +.. code-block:: none + + vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot + Traceback (most recent call last): + File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module> + config = ConfigTree(config_file) + File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__ + raise ValueError("Failed to parse config: {0}".format(msg)) + ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax. + +The reason is that the configuration migration backend is rewritten and uses +a new form of "magic string" which is applied on demand when real config +migration is run on boot. When runnint individual migrators for testing, +you need to convert the "magic string" on your own by: + +.. code-block:: none + + vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot + +Configuration Error on System Boot +---------------------------------- + +Beeing brave and running the latest rolling releases will sometimes trigger +bugs due to corner cases we missed in our design. Those bugs should be filed +via Phabricator_ but you can help us to narrow doen the issue. Login to your +VyOS system and change into configuration mode by typing ``configure``. Now +re-load your boot configuration by simply typing ``load`` followed by return. + +You shoudl now see a Python backtrace which will help us to handle the issue, +please attach it to the Phabricator_ task. + +Boot Timing +----------- + +During the migration and extensive rewrite of functionality from Perl into +Python a significant increase in the overall system boottime was noticed. The +system boot time can be analysed and a graph can be generated in the end which +shows in detail who called whom during the system startup phase. + +This is done by utilizing the ``systemd-bootchart`` package which is now +installed by default on the VyOS 1.3 (equuleus) branch. The configuration is +also versioned so we get comparable results. ``systemd-bootchart`` is configured +using this file: bootchart.conf_ + +To enable boot time graphing change the Kernel commandline and add the folowing +string: ``init=/usr/lib/systemd/systemd-bootchart`` + +This can also be done permanently by changing ``/boot/grub/grub.cfg``. + +Priorities +========== + +VyOS CLI is all about priorities. Every CLI node has a corresponding ``node.def`` +file and possibly an attached script that is executed when the node is present. +Nodes can have a priority, and on system bootup - or any other ``commit`` to the +config all scripts are executed from lowest to higest priority. This is good as +this gives a deterministic behavior. + +To debug issues in priorities or to see what's going on in the background you can +use the ``/opt/vyatta/sbin/priority.pl`` script which lists to you the execution +order of the scripts. + +.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg +.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf + +.. include:: ../common-references.rst diff --git a/docs/contributing/development.rst b/docs/contributing/development.rst index 4a181499..17f5cc48 100644 --- a/docs/contributing/development.rst +++ b/docs/contributing/development.rst @@ -69,6 +69,7 @@ Writing good commit messages ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The format should be and is inspired by: https://git-scm.com/book/ch5-2.html +It is also worth reading https://chris.beams.io/posts/git-commit/ * A single, short, summary of the commit (recommended 50 characters or less, not exceeding 80 characters) containing a prefix of the changed component @@ -673,124 +674,6 @@ Migrating old CLI - None - All logic should be in the scripts -Debugging -========= - -There are two flags available to aid in debugging configuration scripts. -Since configuration loading issues will manifest during boot, the flags are -passed as kernel boot parameters. - -Kernel boot parameters ----------------------- - -* ``vyos-debug`` - Adding the parameter to the linux boot line will produce - timing results for the execution of scripts during commit. If one is seeing - an unexpected delay during manual or boot commit, this may be useful in - identifying bottlenecks. The internal flag is ``VYOS_DEBUG``, and is found - in vyatta-cfg_. Output is directed to ``/var/log/vyatta/cfg-stdout.log``. - -* ``vyos-config-debug`` - During development, coding errors can lead to a - commit failure on boot, possibly resulting in a failed initialization of the - CLI. In this circumstance, the kernel boot parameter ``vyos-config-debug`` - will ensure access to the system as user ``vyos``, and will log a Python - stack trace to the file ``/tmp/boot-config-trace``. - File ``boot-config-trace`` will generate only if config loaded with a failure status. - -Debugging features ------------------- - -A number of flags can be set up to change the behaviour of VyOS at runtime. -These flags can be toggled using either environment variables or creating files. - -For each feature, a file called ``vyos.feature.debug`` can be created to toggle -the feature on. If a parameter is required it can be placed inside the file as -its first line. - -The file can be placed in ``/tmp`` for one time debugging (as the file will be -removed on reboot) or placed in '/config' to stay permanently. - -For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable interface -debugging. - -It is also possible to set up the debugging using environment variables. -In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG. - -for example running, ``export VYOS_IFCONFIG_DEBUG=""`` on your vash, will have -the same effect as ``touch /tmp/vyos.ifconfig.debug``. - -* ``ifconfig`` - Once set, all commands used, and their responses received from - the OS, will be presented on the screen for inspection. - -* ``command`` - Once set, all commands used, and their responses received from - the OS, will be presented on the screen for inspection. - -* ``developer`` - Should a command fail, instead of printing a message to the - user explaining how to report issues, the python interpreter will start a PBD - post-mortem session to allow the developer to debug the issue. As the debugger - will wait from input from the developer, it has the capacity to prevent a - router to boot and therefore should only be permanently set up on production - if you are ready to see the OS fail to boot. - -* ``log`` - In some rare cases, it may be useful to see what the OS is doing, - including during boot. This option sends all commands used by VyOS to a file. - The default file is ``/tmp/full-log`` but it can be changed. - -Config Migration ----------------- - -When writing a new configuration migrator it may happen that you see an error when -you try to invoke it manually on a development system. This error will look like: - -.. code-block:: none - - vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot - Traceback (most recent call last): - File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module> - config = ConfigTree(config_file) - File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__ - raise ValueError("Failed to parse config: {0}".format(msg)) - ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax. - -The reason is that the configuration migration backend is rewritten and uses a new form of -"magic string" which is applied on demand when real config migration is run on boot. When -runnint individual migrators for testing, you need to convert the "magic string" on your -own by: - -.. code-block:: none - - vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot - -Boot Timing ------------ - -During the migration and extensive rewrite of functionality from Perl into -Python a significant increase in the overall system boottime was noticed. The -system boot time can be analysed and a graph can be generated in the end which -shows in detail who called whom during the system startup phase. - -This is done by utilizing the ``systemd-bootchart`` package which is now -installed by default on the VyOS 1.3 (equuleus) branch. The configuration is -also versioned so we get comparable results. ``systemd-bootchart`` is configured -using this file: bootchart.conf_ - -To enable boot time graphing change the Kernel commandline and add the folowing -string: ``init=/usr/lib/systemd/systemd-bootchart`` - -This can also be done permanently by changing ``/boot/grub/grub.cfg``. - -Priorities -========== - -VyOS CLI is all about priorities. Every CLI node has a corresponding ``node.def`` -file and possibly an attached script that is executed when the node is present. -Nodes can have a priority, and on system bootup - or any other ``commit`` to the -config all scripts are executed from lowest to higest priority. This is good as -this gives a deterministic behavior. - -To debug issues in priorities or to see what's going on in the background you can -use the ``/opt/vyatta/sbin/priority.pl`` script which lists to you the execution -order of the scripts. - Continuous Integration ====================== @@ -809,17 +692,15 @@ found. After a successful run the resulting Debian Package(s) will be deployed to our Debian repository which is used during build time. It is located here: http://dev.packages.vyos.net/repositories/. +.. _Jenkins: https://jenkins.io/ +.. _Dockerhub: https://hub.docker.com/u/vyos/ .. _process: https://blog.vyos.io/vyos-development-digest-10 .. _VyConf: https://github.com/vyos/vyconf/tree/master/data/schemata .. _vyos-1x: https://github.com/vyos/vyos-1x/tree/current/schema .. _Jinja2: https://jinja.palletsprojects.com/ -.. _Jenkins: https://jenkins.io/ -.. _Dockerhub: https://hub.docker.com/u/vyos/ .. _`IPv4, IPv6 and DHCP(v6)`: https://github.com/vyos/vyos-1x/tree/current/interface-definitions/include/address-ipv4-ipv6-dhcp.xml.i .. _`IPv4, IPv6`: https://github.com/vyos/vyos-1x/tree/current/interface-definitions/include/address-ipv4-ipv6.xml.i .. _`VLAN (VIF)`: https://github.com/vyos/vyos-1x/tree/current/interface-definitions/include/vif.xml.i .. _`MAC address`: https://github.com/vyos/vyos-1x/tree/current/interface-definitions/include/interface-mac.xml.i -.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg -.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf .. include:: ../common-references.rst diff --git a/docs/contributing/upstream-packages.rst b/docs/contributing/upstream-packages.rst new file mode 100644 index 00000000..d43e61f3 --- /dev/null +++ b/docs/contributing/upstream-packages.rst @@ -0,0 +1,158 @@ +.. _upstream_packages: + +Upstream packages +----------------- + +Many base system packages are pulled straight from Debian's main and contrib +repositories, but there are exceptions. + +This chapter lists those exceptions and gives you a brief overview what we +have done on those packages. If you only want to build yourself a fresh ISO +you can completely skip this chapter. It may become interesting once you have +a VyOS deep dive. + +vyos-netplug +^^^^^^^^^^^^ + +Due to issues in the upstream version that sometimes set interfaces down, a +modified version is used. + +The source is located at https://github.com/vyos/vyos-netplug + +In the future, we may switch to using systemd infrastructure instead. Building +it doesn't require a special procedure. + +keepalived +^^^^^^^^^^ + +Keepalived normally isn't updated to newer feature releases between Debian +versions, so we are building it from source. + +Debian does keep their package in git, but it's upstream tarball imported into +git without its original commit history. To be able to merge new tags in, we +keep a fork of the upstream repository with packaging files imported from +Debian at https://github.com/vyos/keepalived-upstream + +strongswan +^^^^^^^^^^ + +Our StrongSWAN build differs from the upstream: + +- strongswan-nm package build is disabled since we don't use NetworkManager +- Patches for DMVPN are merged in + +The source is at https://github.com/vyos/vyos-strongswan + +DMVPN patches are added by this commit: +https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226 + +Our op mode scripts use the python-vici module, which is not included in +Debian's build, and isn't quite easy to integrate in that build. For this +reason we debianize that module by hand now, using this procedure: + +0. Install https://pypi.org/project/stdeb/ +1. `cd vyos-strongswan` +2. `./configure --enable-python-eggs` +3. `cd src/libcharon/plugins/vici/python` +4. `make` +5. `python3 setup.py --command-packages=stdeb.command bdist_deb` + +The package ends up in deb_dist dir. + +ppp +^^^ + +Properly renaming PPTP and L2TP interfaces to pptpX and l2tpX from generic and +non-informative pppX requires a patch that is neither in the upstream nor in +Debian. + +We keep a fork of Debian's repo at https://github.com/vyos/ppp-debian + +The patches for pre-up renaming are: + +* https://github.com/vyos/ppp-debian/commit/e728180026a051d2a96396276e7e4ae +* https://github.com/vyos/ppp-debian/commit/f29ba8d9ebb043335a096d70bcd07e9 + +Additionally, there's a patch for reopening the log file to better support +logging to files, even though it's less essential: +https://github.com/vyos/ppp-debian/commit/dd2ebd5cdcddb40230dc4cc43d374055f + +The patches were written by Stephen Hemminger back in the Vyatta times. + +mdns-repeater +^^^^^^^^^^^^^ + +This package doesn't exist in Debian. A debianized fork is kept at +https://github.com/vyos/mdns-repeater + +No special build procedure is required. + +udp-broadcast-relay +^^^^^^^^^^^^^^^^^^^ + +This package doesn't exist in Debian. A debianized fork is kept at +https://github.com/vyos/udp-broadcast-relay + +No special build procedure is required. + +Linux kernel +^^^^^^^^^^^^ + +In the past a fork of the Kernel source code was kept at the well-known +location of https://github.com/vyos/vyos-kernel - where it is kept for history. + +Nowadays the Kernel we use is the upstream source code which is patched +with two additional patches from the good old Vyatta times which never made it +into the mainstream Kernel. The patches can be found here: +https://github.com/vyos/vyos-build-kernel/tree/master/patches/kernel and are +automatically applied to the Kernel by the Jenkins Pipeline which is used to +generate the Kernel binaries. + +The Pipeline script not only builds the Kernel with the configuration named +``x86_64_vyos_defconfig`` which is located in the vyos-build-kernel repository, +too - but in addition also builds some Intel out-of-tree drivers, WireGuard +(as long it is not upstreamed) and Accel-PPP. + +The ``Jenkinsfile`` tries to be as verbose as possible on each individual build +step. + +Linux Firmware +^^^^^^^^^^^^^^ + +More and more hardware cards require an additional firmware which is not open +source. The Kernel community hosts a special linux-firmware Git repository +with all available binary files which can be loaded by the Kernel. + +The ``vyos-build`` repository fetches a specific commit of the linux-firmware +repository and embeds those binaries into the resulting ISO image. This step is +done in the ``data/live-build-config/hooks/live/40-linux-firmware.chroot`` file. + +If the firmware needs to be updated it is sufficient to just exchange the Git +commit id we reference in our build. + +Intel NIC drivers +^^^^^^^^^^^^^^^^^ + +We do not make use of the building Intel NIC drivers except for e1000e. Main +reason is that the out of tree Intel drivers seem be perform a bit better, +e.q. have proper receive-side-scaling and multi-queue support. + +Drivers are build as part of the Kernel Pipeline - read above. + +Accel-PPP +^^^^^^^^^ + +Accel-PPP used to be an upstream fork for quite some time but now has been +converted to make use of the upstream source code and build system. + +It is build as part of the Kernel Pipeline - read above. + +hvinfo +^^^^^^ + +A fork with packaging changes for VyOS is kept at https://github.com/vyos/hvinfo + +The original repo is at https://github.com/dmbaturin/hvinfo + +It's an Ada program and requires GNAT and gprbuild for building, dependencies +are properly specified so just follow debuild's suggestions. diff --git a/docs/firewall.rst b/docs/firewall.rst index 7b6be614..2d9001a6 100644 --- a/docs/firewall.rst +++ b/docs/firewall.rst @@ -26,7 +26,7 @@ or zone based firewall policy. Global settings --------------- -Some firewall settings are global and have a affect on the hole system. +Some firewall settings are global and have a affect on the whole system. .. cfgcmd:: set firewall all-ping [enable | disable] @@ -690,7 +690,7 @@ Show Firewall log Show the logs of a specific Rule-Set .. note:: - At the moment it not possible to look at the hole Firewall log with vyos + At the moment it not possible to look at the whole firewall log with VyOS operational commands. All logs will save to ``/var/logs/messages``. For example: ``grep '10.10.0.10' /var/log/messages`` diff --git a/docs/image-mgmt.rst b/docs/image-mgmt.rst index 2c92f7b4..6c1b2587 100644 --- a/docs/image-mgmt.rst +++ b/docs/image-mgmt.rst @@ -108,12 +108,17 @@ for the new image to boot using the current configuration. .. note:: Only LTS releases are PGP-signed. -.. opcmd:: add system image <url | path> +.. opcmd:: add system image <url | path> [vrf name] [username user [password pass]] Use this command to install a new system image. You can reach the image from the web (http://, https://) or from your local system, e.g. /tmp/vyos-1.2.3-amd64.iso. + The `add system image` command also supports installing new versions + of VyOS through an optional given VRF. Also if URL in question requires + authentication, you can specify an optional username and password via + the commandline which will be passed as "Basic-Auth" to the server. + If there is not enough **free disk space available**, the installation will be canceled. To delete images use the :opcmd:`delete system image` command. @@ -122,9 +127,10 @@ VyOS configuration is associated to each image, and **each image has a unique copy of its configuration**. This is different than a traditional network router where the configuration is shared across all images. -.. note:: If you have any personal file, like some scripts you created, - and you don't want them to be deleted during the upgrade, make sure - those files are into the ``/configure`` directory. +.. note:: If you have any personal files, like some scripts you created, + and you don't want them to be lost during the upgrade, make sure + those files are stored in ``/config`` as this directory is always copied + to newer installed images. You can access files from a previous installation and copy them to your current image if they were located in the ``/config`` directory. This diff --git a/docs/index.rst b/docs/index.rst index 8b5c4e5d..e32b90c3 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,7 +18,7 @@ VyOS User Guide .. toctree:: - :caption: Basics + :caption: Basic Configuration :name: basics :maxdepth: 2 @@ -29,7 +29,7 @@ VyOS User Guide .. toctree:: - :caption: Advanced + :caption: Advanced Configuration :name: advanced :maxdepth: 2 @@ -45,7 +45,16 @@ VyOS User Guide high-availability vpn/index load-balancing + command-list-configuration + + +.. toctree:: + :caption: System Operation + :name: system-operation + :maxdepth: 2 + troubleshooting + command-list-operation .. toctree:: @@ -55,7 +64,6 @@ VyOS User Guide appendix/release-notes appendix/examples/index - appendix/cmd-index appendix/vyos-on-vmware appendix/vyos-on-baremetal appendix/vyos-on-clouds @@ -72,8 +80,10 @@ VyOS User Guide :maxdepth: 2 contributing/build-vyos + contributing/upstream-packages contributing/issues-features contributing/development + contributing/debugging contributing/documentation @@ -96,4 +106,3 @@ Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the VyOS maintainers. - diff --git a/docs/interfaces/pseudo-ethernet.rst b/docs/interfaces/pseudo-ethernet.rst index bdd06b7e..a2066555 100644 --- a/docs/interfaces/pseudo-ethernet.rst +++ b/docs/interfaces/pseudo-ethernet.rst @@ -30,8 +30,7 @@ Ethernet interfaces: means that you can not try to ping a Pseudo-Ethernet interface from the host system on which it is defined. The ping will be lost. * Loopbacks occurs at the IP level the same way as for other interfaces, - ethernet packets are not forwarded between Pseudo-Ethernet interfaces. -* Pseudo-Ethernet interfaces can not participate in Link Bonding. + ethernet frames are not forwarded between Pseudo-Ethernet interfaces. * Pseudo-Ethernet interfaces may not work in environments which expect a :abbr:`NIC (Network Interface Card)` to only have a single address. This applies to: diff --git a/docs/interfaces/wirelessmodem.rst b/docs/interfaces/wirelessmodem.rst index 71f8df28..80497359 100644 --- a/docs/interfaces/wirelessmodem.rst +++ b/docs/interfaces/wirelessmodem.rst @@ -114,3 +114,16 @@ Operation .. opcmd:: show interfaces wirelessmodem <interface> log Displays log information for a WWAN interface. + +Supported Modules +################# + +The following hardware modules have been tested successfully in an +:ref:`pc-engines-apu4` board: + +* Sierra Wireless AirPrime MC7304 miniPCIe card (LTE) +* Sierra Wireless AirPrime MC7430 miniPCIe card (LTE) +* Sierra Wireless AirPrime MC7455 miniPCIe card (LTE) +* Sierra Wireless AirPrime MC7710 miniPCIe card (LTE) +* Huawei ME909u-521 miniPCIe card (LTE) + diff --git a/docs/qos.rst b/docs/qos.rst index 51ed0893..a4e56665 100644 --- a/docs/qos.rst +++ b/docs/qos.rst @@ -22,9 +22,9 @@ How to make it work In order to have VyOS Traffic Control working you need to follow 2 steps: - 1. Create a traffic policy. + 1. **Create a traffic policy**. - 2. Apply the traffic policy to an interface ingress or egress. + 2. **Apply the traffic policy to an interface ingress or egress**. But before learning to configure your policy, we will warn you @@ -300,7 +300,7 @@ classes of a Round-Robin policy you have configured. A common example is the case of some policies which, in order to be effective, they need to be applied to an interface that is directly -connected to the link where the bottleneck is. If your router is not +connected where the bottleneck is. If your router is not directly connected to the bottleneck, but some hop before it, you can emulate the bottleneck by embedding your non-shaping policy into a classful shaping one so that it takes effect. @@ -310,15 +310,17 @@ setting. .. code-block:: none - set traffic-policy shaper FQ-SHAPER bandwidth 1gbit + set traffic-policy shaper FQ-SHAPER bandwidth 4gbit set traffic-policy shaper FQ-SHAPER default bandwidth 100% - set traffic-policy shaper FQ-SHAPER default queue-type fair-queue + set traffic-policy shaper FQ-SHAPER default queue-type fq-codel As shown in the last command of the example above, the `queue-type` setting allows these combinations. You will be able to use it in many policies. - +.. note:: Some policies already include other embedded policies inside. + That is the case of Shaper_: each of its classes use fair-queue + unless you change it. .. _creating_a_traffic_policy: @@ -361,8 +363,9 @@ are to be sent, they could get dropped when trying to get enqueued at the tail. This can happen if the queue has still not been able to release enough packets from its head. -**Very likely you do not need this simple policy as you cannot get much -from it. Sometimes it is used just to enable logging.** +This is the policy that requieres the lowest resources for the same +amount of traffic. But **very likely you do not need it as you cannot +get much from it. Sometimes it is used just to enable logging.** .. cfgcmd:: set traffic-policy drop-tail <policy-name> queue-limit <number-of-packets> @@ -378,9 +381,11 @@ Fair Queue | **Applies to:** Outbound traffic. Fair Queue is a work-conserving scheduler which schedules the -transmission of packets based on flows, trying to ensure fairness so -that each flow is able to send data in turn, preventing any single one -from drowning out the rest. +transmission of packets based on flows, that is, it balances traffic +distributing it through different sub-queues in order to ensure +fairness so that each flow is able to send data in turn, preventing any +single one from drowning out the rest. + .. cfgcmd:: set traffic-policy fair-queue <policy-name> @@ -388,13 +393,14 @@ from drowning out the rest. It is based on the Stochastic Fairness Queueing and can be applied to outbound traffic. -The algorithm enqueues packets to hash buckets based on source address, -destination address and source port. Each of these buckets should -represent a unique flow. Because multiple flows may get hashed to the -same bucket, the hashing algorithm is perturbed at configurable -intervals so that the unfairness lasts only for a short while. -Perturbation may however cause some inadvertent packet reordering to -occur. An advisable value could be 10 seconds. +In order to separate traffic, Fair Queue uses a classifier based on +source address, destination address and source port. The algorithm +enqueues packets to hash buckets based on those tree parameters. +Each of these buckets should represent a unique flow. Because multiple +flows may get hashed to the same bucket, the hashing algorithm is +perturbed at configurable intervals so that the unfairness lasts only +for a short while. Perturbation may however cause some inadvertent +packet reordering to occur. An advisable value could be 10 seconds. One of the uses of Fair Queue might be the mitigation of Denial of Service attacks. @@ -423,7 +429,7 @@ fashion. You can configure the length of the queue. -.. _FQ-CoDel +.. _FQ-CoDel: FQ-CoDel -------- @@ -534,7 +540,7 @@ Limiter | **Applies to:** Inbound traffic. Limiter is one of those policies that uses classes_ (Ingress qdisc is -actually classless policy but filters do work in it). +actually a classless policy but filters do work in it). The limiter performs basic ingress policing of traffic flows. Multiple classes of traffic can be defined and traffic limits can be applied to @@ -598,7 +604,7 @@ how you want matching traffic to behave. -Network emulator +Network Emulator ---------------- | **Queueing discipline:** netem (Network Emulator) + TBF (Token Bucket Filter). @@ -622,14 +628,18 @@ under certain network conditions. Use this command to configure the burst size of the traffic in a Network Emulator policy. Define the name of the Network Emulator - policy and its traffic burst size. It will only take effect if you - have configured its bandwidth too. + policy and its traffic burst size (it will be configured through the + Token Bucket Filter qdisc). Default:15kb. It will only take effect if + you have configured its bandwidth too. .. cfgcmd:: set traffic-policy network-emulator <policy-name> network-delay <delay> Use this command to configure a Network Emulator policy defining its name and the fixed amount of time you want to add to all packet going - out of the interface. You can use secs, ms and us. + out of the interface. The latency will be added through the + Token Bucket Filter qdisc. It will only take effect if you have + configured its bandwidth too. You can use secs, ms and us. Default: + 50ms. .. cfgcmd:: set traffic-policy network-emulator <policy-name> packet-corruption <percent> @@ -658,7 +668,7 @@ under certain network conditions. -Priority queue +Priority Queue -------------- | **Queueing discipline:** PRIO. @@ -666,7 +676,8 @@ Priority queue The Priority Queue is a classful scheduling policy. It does not delay -packets, it simply dequeues packets according to their priority. +packets (Priority Queue is not a shaping policy), it simply dequeues +packets according to their priority. .. note:: Priority Queue, as other non-shaping policies, is only useful if your outgoing interface is really full. If it is not, VyOS will @@ -744,8 +755,8 @@ avoiding congestion. That is good for TCP connections as the gradual dropping of packets acts as a signal for the sender to decrease its transmission rate. -In contrast to simple RED, VyOS' Random-Detect uses a Weighted Random -Early Detect policy that prvides different virtual queues based on the +In contrast to simple RED, VyOS' Random-Detect uses a Generalized Random +Early Detect policy that provides different virtual queues based on the IP Precedence value so that some virtual queues can drop more packets than others. @@ -776,10 +787,18 @@ IP precedence as defined in :rfc:`791`: +------------+----------------------+ +Random-Detect could be useful for heavy traffic. One use of this +algorithm might be to prevent a backbone overload. But only for TCP +(because dropped packets could be retransmitted), not for UDP. + + .. cfgcmd:: set traffic-policy random-detect <policy-name> bandwidth <bandwidth> Use this command to configure a Random-Detect policy, set its name - and set the available bandwidth for this policy. + and set the available bandwidth for this policy. It is used for + calculating the average queue size after some idle time. It should be + set to the bandwidth of your interface. Random Detect is not a + shaping policy, this command will not shape. .. cfgcmd:: set traffic-policy random-detect <policy-name> precedence <IP-precedence-value> average-packet <bytes> @@ -848,22 +867,27 @@ The default values for the minimum-threshold depend on IP precedence: length reaches this value. -If the average queue size is lower than the :code:`min-threshold`, an -arriving packet will be placed in the queue. In the case the average -queue size is between :code:`min-threshold` and :code:`max-threshold`, -then an arriving packet would be either dropped or placed in the queue, -it will depend on the defined :code:`mark-probability`. If the current -queue size is larger than :code:`queue-limit`, then packets will be -dropped. The average queue size depends on its former average size and -its current one. If :code:`max-threshold` is set but -:code:`min-threshold` is not, then :code:`min-threshold` is scaled to -50% of :code:`max-threshold`. In principle, values must be +If the average queue size is lower than the **min-threshold**, an +arriving packet will be placed in the queue. + +In the case the average queue size is between **min-threshold** and +**max-threshold**, then an arriving packet would be either dropped or +placed in the queue, it will depend on the defined **mark-probability**. + +If the current queue size is larger than **queue-limit**, +then packets will be dropped. The average queue size depends on its +former average size and its current one. + +If **max-threshold** is set but **min-threshold is not, then +**min-threshold** is scaled to 50% of **max-threshold**. + +In principle, values must be :code:`min-threshold` < :code:`max-threshold` < :code:`queue-limit`. -One use of this algorithm might be to prevent a backbone overload. -Rate control + +Rate Control ------------ | **Queueing discipline:** Tocken Bucket Filter. @@ -908,8 +932,8 @@ you just simply want to slow traffic down. .. _DRR: -Round robin (DRR) ------------------ +Round Robin +----------- | **Queueing discipline:** Deficit Round Robin. | **Applies to:** Outbound traffic. @@ -1005,7 +1029,7 @@ the higher the priority. Use this command to configure a Shaper policy, set its name, define a class and set the size of the `tocken bucket`_ in bytes, which will - be available to be sent at maximum speed (default: 15Kb). + be available to be sent at ceiling speed (default: 15Kb). .. cfgcmd:: set traffic-policy shaper <policy-name> class <class-ID> ceiling <bandwidth> @@ -1059,6 +1083,12 @@ parameters. +.. note:: If you configure a class for **VoIP traffic**, don't give it any + *ceiling*, otherwise new VoIP calls could start when there is available + bandwidth and get suddenly dropped when other classes start using + their bandwidth. + + Example ^^^^^^^ @@ -1068,14 +1098,12 @@ A simple example of Shaper using priorities. .. code-block:: none set traffic-policy shaper MY-HTB bandwidth '50mbit' - set traffic-policy shaper MY-HTB class 10 bandwidth '10%' - set traffic-policy shaper MY-HTB class 10 ceiling '15%' - set traffic-policy shaper MY-HTB class 10 match ADDRESS10 ip source address '192.168.10.0/24' - set traffic-policy shaper MY-HTB class 10 priority '0' - set traffic-policy shaper MY-HTB class 10 queue-type 'fair-queue' + set traffic-policy shaper MY-HTB class 10 bandwidth '20%' + set traffic-policy shaper MY-HTB class 10 match DSCP ip dscp 'EF' + set traffic-policy shaper MY-HTB class 10 queue-type 'fq-codel' set traffic-policy shaper MY-HTB class 20 bandwidth '10%' set traffic-policy shaper MY-HTB class 20 ceiling '50%' - set traffic-policy shaper MY-HTB class 20 match ADDRESS20 ip source address '192.168.20.0/24' + set traffic-policy shaper MY-HTB class 20 match PORT666 ip destination port '666' set traffic-policy shaper MY-HTB class 20 priority '3' set traffic-policy shaper MY-HTB class 20 queue-type 'fair-queue' set traffic-policy shaper MY-HTB class 30 bandwidth '10%' @@ -1087,57 +1115,73 @@ A simple example of Shaper using priorities. set traffic-policy shaper MY-HTB default ceiling '100%' set traffic-policy shaper MY-HTB default priority '7' set traffic-policy shaper MY-HTB default queue-type 'fair-queue' - -.. _ingress-shaping: +Applying a traffic policy +========================= -The case of ingress shaping -=========================== +Once a traffic-policy is created, you can apply it to an interface: -| **Applies to:** Inbound traffic. +.. code-block:: none -For the ingress traffic of an interface, there is only one policy you -can directly apply, a **Limiter** policy. This workaround lets you -redirect every incoming traffic to an in-between virtual interface to -which you will be able to apply there an outbound policy. That -in-between virtual interface" is possible because of the configuration -of an Intermediate Functional Block IFB_. That is how it is possible to -do an "ingress shaping". + set interfaces etherhet eth0 traffic-policy out WAN-OUT +You can only apply one policy per interface and direction, but you could +reuse a policy on different interfaces and directions: .. code-block:: none - set traffic-policy shaper MY-INGRESS-SHAPING bandwidth 1000kbit - set traffic-policy shaper MY-INGRESS-SHAPING default bandwidth 1000kbit - set traffic-policy shaper MY-INGRESS-SHAPING default queue-type fair-queue - - set interfaces input ifb0 traffic-policy out MY-INGRESS-SHAPING - set interfaces ethernet eth0 redirect ifb0 + set interfaces ethernet eth0 traffic-policy in WAN-IN + set interfaces etherhet eth0 traffic-policy out WAN-OUT + set interfaces etherhet eth1 traffic-policy in LAN-IN + set interfaces etherhet eth1 traffic-policy out LAN-OUT + set interfaces ethernet eth2 traffic-policy in LAN-IN + set interfaces ethernet eth2 traffic-policy out LAN-OUT + set interfaces etherhet eth3 traffic-policy in TWO-WAY-POLICY + set interfaces etherhet eth3 traffic-policy out TWO-WAY-POLICY + set interfaces etherhet eth4 traffic-policy in TWO-WAY-POLICY + set interfaces etherhet eth4 traffic-policy out TWO-WAY-POLICY +Getting queueing information +---------------------------- +.. opcmd:: show queueing <interface-type> <interface-name> -Applying a traffic policy -========================= + Use this command to see the queueing information for an interface. + You will be able to see a packet counter (Sent, Dropped, Overlimit + and Backlog) per policy and class configured. -Once a traffic-policy is created, you can apply it to an interface: -.. code-block:: none - set interfaces etherhet eth0 traffic-policy out WAN-OUT +.. _ingress-shaping: -You can only apply one policy per interface and direction, but you can -have several policies working at the same time: +The case of ingress shaping +=========================== -.. code-block:: none +| **Applies to:** Inbound traffic. - set interfaces ethernet eth0 traffic-policy in WAN-IN - set interfaces etherhet eth0 traffic-policy out WAN-OUT - set interfaces etherhet eth1 traffic-policy out WAN-OUT - set interfaces ethernet eth2 traffic-policy out LAN-IN - set interfaces ethernet eth2 traffic-policy out LAN-OUT +For the ingress traffic of an interface, there is only one policy you +can directly apply, a **Limiter** policy. You cannot apply a shaping +policy directly to the ingress traffic of any interface because shaping +only works for outbound traffic. + +This workaround lets you apply a shaping policy to the ingress traffic +by first redirecting it to an in-between virtual interface +(`Intermediate Functional Block`_). There, in that virtual interface, +you will be able to apply any of the policies that work for outbound +traffic, for instance, a shaping one. +That is how it is possible to do the so-called "ingress shaping". + + +.. code-block:: none + set traffic-policy shaper MY-INGRESS-SHAPING bandwidth 1000kbit + set traffic-policy shaper MY-INGRESS-SHAPING default bandwidth 1000kbit + set traffic-policy shaper MY-INGRESS-SHAPING default queue-type fair-queue + + set interfaces input ifb0 traffic-policy out MY-INGRESS-SHAPING + set interfaces ethernet eth0 redirect ifb0 @@ -1146,4 +1190,4 @@ have several policies working at the same time: .. _tc: https://en.wikipedia.org/wiki/Tc_(Linux) .. _tocken bucket: https://en.wikipedia.org/wiki/Token_bucket .. _HFSC: https://en.wikipedia.org/wiki/Hierarchical_fair-service_curve -.. _IFB: https://www.linuxfoundation.org/collaborate/workgroups/networking/ifb +.. _Intermediate Functional Block: https://www.linuxfoundation.org/collaborate/workgroups/networking/ifb diff --git a/docs/routing/igmp-proxy.rst b/docs/routing/igmp-proxy.rst deleted file mode 100644 index 3e69dcc5..00000000 --- a/docs/routing/igmp-proxy.rst +++ /dev/null @@ -1,73 +0,0 @@ -.. _igmp_proxy: - -########## -IGMP Proxy -########## - -:abbr:`IGMP (Internet Group Management Protocol)` proxy sends IGMP host messages -on behalf of a connected client. The configuration must define one, and only one -upstream interface, and one or more downstream interfaces. - -Configuration -============= - -.. cfgcmd:: set protocols igmp-proxy interface <interface> role <upstream | downstream> - - * **upstream:** The upstream network interface is the outgoing interface - which is responsible for communicating to available multicast data sources. - There can only be one upstream interface. - - * **downstream:** Downstream network interfaces are the distribution - interfaces to the destination networks, where multicast clients can join - groups and receive multicast data. One or more downstream interfaces must - be configured. - -.. cfgcmd:: set protocols igmp-proxy interface <interface> alt-subnet <network> - - Defines alternate sources for multicasting and IGMP data. The network address - must be on the following format 'a.b.c.d/n'. By default the router will - accept data from sources on the same network as configured on an interface. - If the multicast source lies on a remote network, one must define from where - traffic should be accepted. - - This is especially useful for the upstream interface, since the source for - multicast traffic is often from a remote location. - - This option can be supplied multiple times. - -.. cfgcmd:: set protocols igmp-proxy disable-quickleave - - Disables quickleave mode. In this mode the daemon will not send a Leave IGMP - message upstream as soon as it receives a Leave message for any downstream - interface. The daemon will not ask for Membership reports on the downstream - interfaces, and if a report is received the group is not joined again - upstream. - - If it's vital that the daemon should act exactly as a real multicast client - on the upstream interface, this function should be enabled. - - Enabling this function increases the risk of bandwidth saturation. - -.. cfgcmd:: set protocols igmp-proxy disable - - Disable this service. - -Example -------- - -Interface `eth1` LAN is behind NAT. In order to subscribe `10.0.0.0/23` subnet -multicast which is in `eth0` WAN we need to configure igmp-proxy. - -.. code-block:: none - - set protocols igmp-proxy interface eth0 role upstream - set protocols igmp-proxy interface eth0 alt-subnet 10.0.0.0/23 - set protocols igmp-proxy interface eth1 role downstream - -Operation -========= - -.. opcmd:: restart igmp-proxy - - Restart the IGMP proxy process. - diff --git a/docs/routing/index.rst b/docs/routing/index.rst index 398d6265..f0214ac5 100644 --- a/docs/routing/index.rst +++ b/docs/routing/index.rst @@ -17,4 +17,4 @@ Routing bfd mss-clamp routing-policy - igmp-proxy + multicast diff --git a/docs/routing/multicast.rst b/docs/routing/multicast.rst new file mode 100644 index 00000000..d20d8e31 --- /dev/null +++ b/docs/routing/multicast.rst @@ -0,0 +1,245 @@ +.. _multicast: + +######### +Multicast +######### + +VyOS facilitates IP Multicast by supporting **PIM Sparse Mode**, +**IGMP** and **IGMP-Proxy**. + + +************ +PIM and IGMP +************ + +PIM (Protocol Independent Multicast) must be configured in every +interface of every participating router. Every router must also have the +location of the Rendevouz Point manually configured. Then, +unidirectional shared trees rooted at the Rendevouz Point will +automatically be built for multicast distribution. + +Traffic from multicast sources will go to the Rendezvous Point, and +receivers will pull it from a shared tree using IGMP (Internet Group +Management Protocol). + +Multicast receivers will talk IGMP to their local router, so, besides +having PIM configured in every router, IGMP must also be configured in +any router where there could be a multicast receiver locally connected. + +VyOS supports both IGMP version 2 and version 3 (which allows +source-specific multicast). + + +Example +======= + +In the following example we can see a basic multicast setup: + +.. image:: /_static/images/multicast-basic.png + :width: 90% + :align: center + :alt: Network Topology Diagram + + + +**Router 1** + +.. code-block:: none + + set interfaces ethernet eth2 address '172.16.0.2/24' + set interfaces ethernet eth1 address '100.64.0.1/24' + set protocols ospf area 0 network '172.16.0.0/24' + set protocols ospf area 0 network '100.64.0.0/24' + set protocols igmp interface eth1 + set protocols pim interface eth1 + set protocols pim interface eth2 + set protocols pim rp address 172.16.255.1 group '224.0.0.0/4' + +**Router 3** + +.. code-block:: none + + set interfaces dummy dum0 address '172.16.255.1/24' + set interfaces ethernet eth0 address '172.16.0.1/24' + set interfaces ethernet eth1 address '172.16.1.1/24' + set protocols ospf area 0 network '172.16.0.0/24' + set protocols ospf area 0 network '172.16.255.0/24' + set protocols ospf area 0 network '172.16.1.0/24' + set protocols pim interface dum0 + set protocols pim interface eth0 + set protocols pim interface eth1 + set protocols pim rp address 172.16.255.1 group '224.0.0.0/4' + +**Router 2** + +.. code-block:: none + + set interfaces ethernet eth1 address '10.0.0.1/24' + set interfaces ethernet eth2 address '172.16.1.2/24' + set protocols ospf area 0 network '10.0.0.0/24' + set protocols ospf area 0 network '172.16.1.0/24' + set protocols pim interface eth1 + set protocols pim interface eth2 + set protocols pim rp address 172.16.255.1 group '224.0.0.0/4' + + + + + +Basic commands +============== + +These are the commands for a basic setup. + +.. cfgcmd:: set protocols pim interface <interface-name> + + Use this command to enable PIM in the selected interface so that it + can communicate with PIM neighbors. + + +.. cfgcmd:: set protocols pim rp address <address> group <multicast-address/mask-bits> + + Use this comand to manually configure a Rendevouz Point for PIM so + that join messages can be sent there. Set the Rendevouz Point address + and the matching prefix of group ranges covered. These values must + be shared with every router participating in the PIM network. + + +.. cfgcmd:: set protocols igmp interface eth1 + + Use this command to configure an interface with IGMP so that PIM can + receive IGMP reports and query on the selected interface. By defaul + IGMP version 3 will be used. + + + +Tuning commands +=============== + +You can also tune multicast with the following commands. + +.. cfgcmd:: set protocols pim interface <interface> dr-priority <value> + + Use this PIM command in the selected interface to set the priority + (1-4294967295) you want to influence in the election of a node to + become the Designated Router for a LAN segment. The default priority + is 1, set a higher value to give the router more preference in the + DR election process. + + +.. cfgcmd:: set protocols pim int <interface> hello <seconds> + + Use this command to configure the PIM hello interval in seconds + (1-180) for the selected interface. + + +.. cfgcmd:: set protocols pim rp keep-alive-timer <seconds> + + Use this PIM command to modify the the time out value (31-60000 + seconds) for an `(S,G) <https://tools.ietf.org/html/rfc7761#section-4.1>`_ + flow. 31 seconds is chosen for a lower bound as some hardware + platforms cannot see data flowing in better than 30 second chunks. + + +.. cfgcmd:: set protocols igmp interface <interface> join <multicast-address> source <IP-address> + + Use this command to allow the selected interface join a multicast + group defining the multicast address you want to join and the source + IP address too. + + +.. cfgcmd:: set protocols igmp interface <interface query-interval <seconds> + + Use this command to configure in the selected interface the IGMP + host query interval (1-1800) in seconds that PIM will use. + + +.. cfgcmd:: set protocols igmp interface <interface query-max-response-time <deciseconds> + + Use this command to configure in the selected interface the IGMP + query response timeout value (10-250) in deciseconds. If a report is + not returned in the specified time, it will be asumed the `(S,G) or + (*,G) state <https://tools.ietf.org/html/rfc7761#section-4.1>`_ has + timed out. + + +.. cfgcmd:: set protocols igmp interface <interface> version <version-number> + + Use this command to define in the selected interface whether you + choose IGMP version 2 or 3. The default value is 3. + + + +********** +IGMP Proxy +********** + +:abbr:`IGMP (Internet Group Management Protocol)` proxy sends IGMP host messages +on behalf of a connected client. The configuration must define one, and only one +upstream interface, and one or more downstream interfaces. + +Configuration +============= + +.. cfgcmd:: set protocols igmp-proxy interface <interface> role <upstream | downstream> + + * **upstream:** The upstream network interface is the outgoing interface + which is responsible for communicating to available multicast data sources. + There can only be one upstream interface. + + * **downstream:** Downstream network interfaces are the distribution + interfaces to the destination networks, where multicast clients can join + groups and receive multicast data. One or more downstream interfaces must + be configured. + +.. cfgcmd:: set protocols igmp-proxy interface <interface> alt-subnet <network> + + Defines alternate sources for multicasting and IGMP data. The network address + must be on the following format 'a.b.c.d/n'. By default the router will + accept data from sources on the same network as configured on an interface. + If the multicast source lies on a remote network, one must define from where + traffic should be accepted. + + This is especially useful for the upstream interface, since the source for + multicast traffic is often from a remote location. + + This option can be supplied multiple times. + +.. cfgcmd:: set protocols igmp-proxy disable-quickleave + + Disables quickleave mode. In this mode the daemon will not send a Leave IGMP + message upstream as soon as it receives a Leave message for any downstream + interface. The daemon will not ask for Membership reports on the downstream + interfaces, and if a report is received the group is not joined again + upstream. + + If it's vital that the daemon should act exactly as a real multicast client + on the upstream interface, this function should be enabled. + + Enabling this function increases the risk of bandwidth saturation. + +.. cfgcmd:: set protocols igmp-proxy disable + + Disable this service. + +Example +------- + +Interface `eth1` LAN is behind NAT. In order to subscribe `10.0.0.0/23` subnet +multicast which is in `eth0` WAN we need to configure igmp-proxy. + +.. code-block:: none + + set protocols igmp-proxy interface eth0 role upstream + set protocols igmp-proxy interface eth0 alt-subnet 10.0.0.0/23 + set protocols igmp-proxy interface eth1 role downstream + +Operation +========= + +.. opcmd:: restart igmp-proxy + + Restart the IGMP proxy process. + + + diff --git a/docs/services/console-server.rst b/docs/services/console-server.rst index 17d1d4d5..7fc43f95 100644 --- a/docs/services/console-server.rst +++ b/docs/services/console-server.rst @@ -33,7 +33,7 @@ distributions. For additional details you can refer to https://phabricator.vyos.net/T2490. -.. opcmd:: show system usb +.. opcmd:: show hardware usb Retrieve a tree like representation of all connected USB devices. @@ -42,7 +42,7 @@ For additional details you can refer to https://phabricator.vyos.net/T2490. .. code-block:: none - vyos@vyos:~$ show system usb + vyos@vyos:~$ show hardware usb /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M |__ Port 3: Dev 4, If 0, Class=Vendor Specific Class, Driver=qcserial, 480M @@ -69,7 +69,7 @@ For additional details you can refer to https://phabricator.vyos.net/T2490. |__ Port 4: Dev 7, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 480M -.. opcmd:: show system usb +.. opcmd:: show hardware usb serial Retrieve a list and description of all connected USB serial devices. The device name displayed, e.g. `usb0b2.4p1.0` can be directly used when accessing the serial console @@ -77,7 +77,7 @@ For additional details you can refer to https://phabricator.vyos.net/T2490. .. code-block:: none - vyos@vyos$ show system usb serial + vyos@vyos$ show hardware usb serial Device Model Vendor ------ ------ ------ usb0b1.3p1.0 MC7710 Sierra Wireless, Inc. diff --git a/docs/services/dns-forwarding.rst b/docs/services/dns-forwarding.rst index a529f6a7..5c154fdf 100644 --- a/docs/services/dns-forwarding.rst +++ b/docs/services/dns-forwarding.rst @@ -11,7 +11,7 @@ VyOS provides DNS infrastructure for small networks. It is designed to be lightweight and have a small footprint, suitable for resource constrained routers and firewalls, for this we utilize PowerDNS recursor. -VyOS DNS forwarder does not require an upstream DNS server. It can serve as a +The VyOS DNS forwarder does not require an upstream DNS server. It can serve as a full recursive DNS server - but it can also forward queries to configurable upstream DNS servers. By not configuring any upstream DNS servers you also avoid to be tracked by the provider of your upstream DNS server. @@ -103,23 +103,23 @@ avoid to be tracked by the provider of your upstream DNS server. .. cfgcmd:: set service dns forwarding listen-address - Local IPv4 or IPv6 addresses to bind to - waiting on this address for + The local IPv4 or IPv6 addresses to bind the DNS forwarder to. The forwarder will listen on this address for incoming connections. Example ======= -Router with two interfaces eth0 (WAN link) and eth1 (LAN) does want to make -use of DNS split-horizon for example.com. +A VyOS router with two interfaces - eth0 (WAN) and eth1 (LAN) - is required to implement a split-horizon DNS configuration for example.com. -* DNS request for example.com need to get forwarded to IPv4 address 192.0.2.254 - and IPv6 address 2001:db8:cafe::1 -* All other DNS requests are forwarded to DNS server listening on 192.0.2.1, +In this scenario: + +* All DNS requests for example.com must be forwarded to a DNS server at 192.0.2.254 + and 2001:db8:cafe::1 +* All other DNS requests will be forwarded to a different set of DNS servers at 192.0.2.1, 192.0.2.2, 2001:db8::1:ffff and 2001:db8::2:ffff -* DNS server is listening on the LAN interface addresses only, 192.168.1.254 +* The VyOS DNS forwarder will only listen for requests on the eth1 (LAN) interface addresses - 192.168.1.254 for IPv4 and 2001:db8::ffff for IPv6 -* Only clients from the LAN segment (192.168.1.0/24) are allowed to use this - server +* The VyOS DNS forwarder will only accept lookup requests from the LAN subnets - 192.168.1.0/24 and 2001:db8::/64 .. code-block:: none @@ -139,9 +139,9 @@ Operation .. opcmd:: reset dns forwarding <all | domain> - Reset local DNS forwarding cache database. You can reset the cache for all + Resets the local DNS forwarding cache database. You can reset the cache for all entries or only for entries to a specific domain. .. opcmd:: restart dns forwarding - Restart DNS recursor process which also invalidates the cache. + Restarts the DNS recursor process. This also invalidates the local DNS forwarding cache. diff --git a/docs/services/snmp.rst b/docs/services/snmp.rst index c27cf02a..3f445ea8 100644 --- a/docs/services/snmp.rst +++ b/docs/services/snmp.rst @@ -1,11 +1,14 @@ +.. _snmp: + +#### SNMP ----- +#### -Simple Network Management Protocol (SNMP_) is an Internet Standard protocol -for collecting and organizing information about managed devices on IP networks -and for modifying that information to change device behavior. Devices that -typically support SNMP include cable modems, routers, switches, servers, -workstations, printers, and more. +:abbr:`SNMP (Simple Network Management Protocol)` is an Internet Standard +protocol for collecting and organizing information about managed devices on +IP networks and for modifying that information to change device behavior. +Devices that typically support SNMP include cable modems, routers, switches, +servers, workstations, printers, and more. SNMP is widely used in network management for network monitoring. SNMP exposes management data in the form of variables on the managed systems organized in @@ -23,7 +26,7 @@ management, including an application layer protocol, a database schema, and a set of data objects. Overview and basic concepts -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +=========================== In typical uses of SNMP, one or more administrative computers called managers have the task of monitoring or managing a group of hosts or devices on a @@ -63,15 +66,15 @@ network. .. note:: VyOS SNMP supports both IPv4 and IPv6. -SNMP protocol versions -^^^^^^^^^^^^^^^^^^^^^^ +SNMP Protocol Versions +====================== VyOS itself supports SNMPv2_ (version 2) and SNMPv3_ (version 3) where the later is recommended because of improved security (optional authentication and encryption). SNMPv2 -^^^^^^ +------ SNMPv2 is the original and most commonly used version. For authorizing clients, SNMP uses the concept of communities. Communities may have authorization set @@ -88,7 +91,7 @@ router. Note that SNMPv2 also supports no encryption and always sends data in plain text. Example -******* +^^^^^^^ .. code-block:: none @@ -116,7 +119,7 @@ Example SNMPv3 -^^^^^^ +------ SNMPv3 (version 3 of the SNMP protocol) introduced a whole slew of new security related features that have been missing from the previous versions. Security @@ -137,60 +140,64 @@ The securityapproach in v3 targets: * Authentication – to verify that the message is from a valid source. Example -******* - -.. code-block:: none +^^^^^^^ - set service snmp v3 engineid '0x0aa0d6c6f450' - set service snmp v3 group defaultgroup mode 'ro' - set service snmp v3 group defaultgroup seclevel 'priv' - set service snmp v3 group defaultgroup view 'defaultview' - set service snmp v3 view defaultview oid '1' +* Let SNMP daemon listen only on IP address 192.0.2.1 +* Configure new SNMP user named "vyos" with password "vyos12345678" +* New user will use SHA/AES for authentication and privacy - set service snmp v3 user testUser1 auth plaintext-key testUserKey1 - set service snmp v3 user testUser1 auth type 'md5' - set service snmp v3 user testUser1 engineid '0x0aa0d6c6f450' - set service snmp v3 user testUser1 group 'defaultgroup' - set service snmp v3 user testUser1 mode 'ro' - set service snmp v3 user testUser1 privacy type aes - set service snmp v3 user testUser1 privacy plaintext-key testUserKey1 - -After commit the resulting configuration will look like: +.. code-block:: none -.. note:: SNMPv3 keys won't we stored in plaintext. On ``commit`` the keys - will be encrypted and the encrypted key is based on the engineid! + set service snmp listen-address 192.0.2.1 + set service snmp location 'VyOS Datacenter' + set service snmp v3 engineid '000000000000000000000002' + set service snmp v3 group default mode 'ro' + set service snmp v3 group default view 'default' + set service snmp v3 user vyos auth plaintext-password 'vyos12345678' + set service snmp v3 user vyos auth type 'sha' + set service snmp v3 user vyos group 'default' + set service snmp v3 user vyos privacy plaintext-password 'vyos12345678' + set service snmp v3 user vyos privacy type 'aes' + set service snmp v3 view default oid 1 + +After commit the plaintext passwords will be hashed and stored in your +configuration. The resulting LCI config will look like: .. code-block:: none vyos@vyos# show service snmp + listen-address 172.18.254.201 { + } + location "Wuerzburg, Dr.-Georg-Fuchs-Str. 8" v3 { - engineid 0x0aa0d6c6f450 - group defaultgroup { + engineid 000000000000000000000002 + group default { mode ro - seclevel priv - view defaultview + view default } - user testUser1 { + user vyos { auth { - encrypted-key 0x3b68d4162c2c817b8e9dfb6f08583e5d - type md5 + encrypted-password 4e52fe55fd011c9c51ae2c65f4b78ca93dcafdfe + type sha } - engineid 0x0aa0d6c6f450 - group defaultgroup - mode ro + group default privacy { - encrypted-key 0x3b68d4162c2c817b8e9dfb6f08583e5d + encrypted-password 4e52fe55fd011c9c51ae2c65f4b78ca93dcafdfe type aes } } - view defaultview { + view default { oid 1 { } } } +You can test the SNMPv3 functionality from any linux based system, just run the +following command: ``snmpwalk -v 3 -u vyos -a SHA -A vyos12345678 -x AES +-X vyos12345678 -l authPriv 192.0.2.1 .1`` + VyOS MIBs -^^^^^^^^^ +========= All SNMP MIBs are located in each image of VyOS here: ``/usr/share/snmp/mibs/`` @@ -200,9 +207,8 @@ you are be able to download the files with the a activate ssh service like this scp -r vyos@your_router:/usr/share/snmp/mibs /your_folder/mibs - SNMP Extensions -^^^^^^^^^^^^^^^ +=============== To extend SNMP agent functionality, custom scripts can be executed every time the agent is being called. This can be achieved by using @@ -230,7 +236,7 @@ contain the output of the extension. NET-SNMP-EXTEND-MIB::nsExtendResult."my-extension" = INTEGER: 0 SolarWinds -^^^^^^^^^^ +========== If you happen to use SolarWinds Orion as NMS you can also use the Device Templates Management. A template for VyOS can be easily imported. @@ -255,7 +261,6 @@ following content: </Configuration-Management> .. _MIB: https://en.wikipedia.org/wiki/Management_information_base -.. _SNMP: https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol .. _SNMPv2: https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol#Version_2 .. _SNMPv3: https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol#Version_3 diff --git a/docs/system/advanced-index.rst b/docs/system/advanced-index.rst index 4557421a..de6cd3e1 100644 --- a/docs/system/advanced-index.rst +++ b/docs/system/advanced-index.rst @@ -10,6 +10,7 @@ Advanced System Tweaks eventhandler flow-accounting ntp + options proxy serial-console syslog diff --git a/docs/system/flow-accounting.rst b/docs/system/flow-accounting.rst index 64c20dcf..f09c1c9a 100644 --- a/docs/system/flow-accounting.rst +++ b/docs/system/flow-accounting.rst @@ -39,8 +39,8 @@ NetFlow is usually enabled on a per-interface basis to limit load on the router components involved in NetFlow, or to limit the amount of NetFlow records exported. -Configururation -=============== +Configuration +============= In order for flow accounting information to be collected and displayed for an interface, the interface must be configured for flow accounting. diff --git a/docs/system/options.rst b/docs/system/options.rst new file mode 100644 index 00000000..82ad4801 --- /dev/null +++ b/docs/system/options.rst @@ -0,0 +1,39 @@ +.. _system_options: + +####### +Options +####### + +This chapter describe the possibilities of advanced system behavior. + +General +####### + +.. cfgcmd:: set system options beep-if-fully-booted + + Send an audible beep to the system speaker when system is ready. + +.. cfgcmd:: set system options ctrl-alt-del-action [ ignore | reboot | poweroff ] + + Action which will be run once the ctrl-alt-del keystroke is received. + +.. cfgcmd:: set system options reboot-on-panic + + Automatically teboot system on kernel panic after 60 seconds. + +HTTP client +########### + +.. cfgcmd:: set system options http-client source-address <address> + + Several commands utilize curl to initiate transfers. Configure the local + source IPv4/IPv6 address used for all CURL operations. + +.. cfgcmd:: set system options http-client source-interface <interface> + + Several commands utilize curl to initiate transfers. Configure the local + source interface used for all CURL operations. + +.. note:: `source-address` and `source-interface` can not be used at the same time. + + diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index e965a8c9..23248507 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -1,98 +1,169 @@ .. _troubleshooting: +############### Troubleshooting -=============== +############### Sometimes things break or don't work as expected. This section describes several troubleshooting tools provided by VyOS that can help when something goes wrong. -Basic Connectivity Verification -------------------------------- +****************** +Connectivity Tests +****************** + +Basic Connectivity Tests +======================== Verifying connectivity can be done with the familiar `ping` and `traceroute` commands. The options for each are shown (the options for each command were displayed using the built-in help as described in the :ref:`cli` section and are omitted from the output here): -.. code-block:: none +.. opcmd:: ping <destination> - vyos@vyos:~$ ping - Possible completions: - <hostname> Send Internet Control Message Protocol (ICMP) echo request - <x.x.x.x> - <h:h:h:h:h:h:h:h> + Send ICMP echo requests to destination host. There are multiple options to + ping, inkl. VRF support. -Several options are available when more extensive troubleshooting is needed: + .. code-block:: none -.. code-block:: none + vyos@vyos:~$ ping 10.1.1.1 + Possible completions: + <Enter> Execute the current command + adaptive Ping options + allow-broadcast + audible + bypass-route + count + deadline + flood + interface + interval + mark + no-loopback + numeric + pattern + quiet + record-route + size + timestamp + tos + ttl + verbose + vrf - vyos@vyos:~$ ping 10.1.1.1 - Possible completions: - <Enter> Execute the current command - adaptive Ping options - allow-broadcast - audible - bypass-route - count - deadline - flood - interface - interval - mark - no-loopback - numeric - pattern - quiet - record-route - size - timestamp - tos - ttl - verbose -.. code-block:: none +.. opcmd:: traceroute <destination> - vyos@vyos:~$ traceroute - Possible completions: - <hostname> Track network path to specified node - <x.x.x.x> - <h:h:h:h:h:h:h:h> - ipv4 Track network path to <hostname|IPv4 address> - ipv6 Track network path to <hostname|IPv6 address> + Trace path to target. -However, another tool, mtr_, is available which combines ping and traceroute -into a single tool. An example of its output is shown: + .. code-block:: none -.. code-block:: none + vyos@vyos:~$ traceroute + Possible completions: + <hostname> Track network path to specified node + <x.x.x.x> + <h:h:h:h:h:h:h:h> + ipv4 Track network path to <hostname|IPv4 address> + ipv6 Track network path to <hostname|IPv6 address> + + +Advanced Connectivity Tests +=========================== + +.. opcmd:: monitor traceroute <destination> + + However, another helper is available which combines ping and traceroute + into a single tool. An example of its output is shown: + + .. code-block:: none - vyos@vyos:~$ mtr 10.62.212.12 + vyos@vyos:~$ mtr 10.62.212.12 - My traceroute [v0.85] - vyos (0.0.0.0) - Keys: Help Display mode Restart statistics Order of fields quit - Packets Pings - Host Loss% Snt Last Avg Best Wrst StDev - 1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1 - 2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1 - 3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1 - 4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0 + My traceroute [v0.85] + vyos (0.0.0.0) + Keys: Help Display mode Restart statistics Order of fields quit + Packets Pings + Host Loss% Snt Last Avg Best Wrst StDev + 1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1 + 2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1 + 3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1 + 4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0 -.. note:: The output of ``mtr`` consumes the screen and will replace your - command prompt. + .. note:: The output consumes the screen and will replace your command + prompt. -Several options are available for changing the display output. Press `h` to -invoke the built in help system. To quit, just press `q` and you'll be returned -to the VyOS command prompt. + Several options are available for changing the display output. Press `h` to + invoke the built in help system. To quit, just press `q` and you'll be + returned to the VyOS command prompt. +IPv6 Topology Discovery +======================= + +IPv6 uses different techniques to discover its Neighbors/topology. + +Router Discovery +---------------- + +.. opcmd:: force ipv6-rd interface <interface> [address <ipv6-address>] + + Discover routers via eth0. + + Example: + + .. code-block:: none + + vyos@vyos:~$ force ipv6-rd interface eth0 + Soliciting ff02::2 (ff02::2) on eth0... + + Hop limit : 60 ( 0x3c) + Stateful address conf. : No + Stateful other conf. : No + Mobile home agent : No + Router preference : high + Neighbor discovery proxy : No + Router lifetime : 1800 (0x00000708) seconds + Reachable time : unspecified (0x00000000) + Retransmit time : unspecified (0x00000000) + Prefix : 240e:fe:8ca7:ea01::/64 + On-link : Yes + Autonomous address conf.: Yes + Valid time : 2592000 (0x00278d00) seconds + Pref. time : 14400 (0x00003840) seconds + Prefix : fc00:470:f1cd:101::/64 + On-link : Yes + Autonomous address conf.: Yes + Valid time : 2592000 (0x00278d00) seconds + Pref. time : 14400 (0x00003840) seconds + Recursive DNS server : fc00:470:f1cd::ff00 + DNS server lifetime : 600 (0x00000258) seconds + Source link-layer address: 00:98:2B:F8:3F:11 + from fe80::298:2bff:fef8:3f11 + +Neighbor Discovery +------------------ + +.. opcmd:: force ipv6-nd interface <interface> address <ipv6-address> + + + Example: + + .. code-block:: none + + vyos@vyos:~$ force ipv6-nd interface eth0 address fc00:470:f1cd:101::1 + + Soliciting fc00:470:f1cd:101::1 (fc00:470:f1cd:101::1) on eth0... + Target link-layer address: 00:98:2B:F8:3F:11 from fc00:470:f1cd:101::1 + +********** Monitoring ----------- +********** VyOS features several monitoring tools. .. code-block:: none - vyos@vyos:~$ monitor + vyos@vyos:~$ monitor Possible completions: bandwidth Monitor interface bandwidth in real time bandwidth-test @@ -110,6 +181,7 @@ VyOS features several monitoring tools. lldp Monitor Link Layer Discovery Protocol (LLDP) daemon log Monitor last lines of messages file nat Monitor network address translation (NAT) + ndp Monitor the NDP information received by the router through the device openvpn Monitor OpenVPN protocol Monitor routing protocols snmp Monitor Simple Network Management Protocol (SNMP) daemon @@ -119,17 +191,17 @@ VyOS features several monitoring tools. vpn Monitor VPN vrrp Monitor Virtual Router Redundancy Protocol (VRRP) webproxy Monitor Webproxy service - + Traffic Dumps -^^^^^^^^^^^^^ +============= To monitor interface traffic, issue the :code:`monitor traffic interface <name>` command, replacing `<name>` with your chosen interface. .. code-block:: none - vyos@vyos:~$ monitor traffic interface eth0 + vyos@vyos:~$ monitor traffic interface eth0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:54:28.581601 IP 192.168.0.1 > vyos: ICMP echo request, id 1870, seq 3848, length 64 @@ -149,15 +221,15 @@ Traffic can be filtered and saved. .. code-block:: none - vyos@vyos:~$ monitor traffic interface eth0 + vyos@vyos:~$ monitor traffic interface eth0 Possible completions: <Enter> Execute the current command filter Monitor traffic matching filter conditions save Save traffic dump from an interface to a file -Interface Bandwidth -^^^^^^^^^^^^^^^^^^^ +Interface Bandwidth Usage +========================= to take a quick view on the used bandwidth of an interface use the ``monitor bandwidth`` command @@ -188,8 +260,8 @@ show the following: 0.61 :::::||..................................................... 1 5 10 15 20 25 30 35 40 45 50 55 60 -Interface performance -^^^^^^^^^^^^^^^^^^^^^ +Interface Performance +===================== To take a look on the network bandwidth between two nodes, the ``monitor bandwidth-test`` command is used to run iperf. @@ -214,7 +286,7 @@ bandwidth-test`` command is used to run iperf. Monitor command -^^^^^^^^^^^^^^^ +=============== The ``monitor command`` command allows you to repeatedly run a command to view a continuously refreshed output. The command is run and output every 2 seconds, @@ -242,8 +314,9 @@ Will clear the screen and show you the output of ``show interfaces`` every vti0 172.25.254.2/30 u/u vti1 172.25.254.9/30 u/u -Clear Command -------------- +**************** +Terminal/Console +**************** Sometimes you need to clear counters or statistics to troubleshoot better. @@ -285,11 +358,12 @@ to clear counters on firewall rulesets or single rules vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> rule <rule#> counters -Basic System Information ------------------------- +****************** +System Information +****************** -Boot steps -^^^^^^^^^^ +Boot Steps +========== VyOS 1.2 uses `Debian Jessie`_ as the base Linux operating system. Jessie was the first version of Debian that uses systemd_ as the default init system. @@ -333,6 +407,5 @@ These are the boot steps for VyOS 1.2 .. _vyatta-cfg: https://github.com/vyos/vyatta-cfg .. _systemd: https://freedesktop.org/wiki/Software/systemd/ .. _`Debian Jessie`: https://www.debian.org/releases/jessie/ -.. _mtr: http://www.bitwizard.nl/mtr/ .. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html .. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html diff --git a/docs/vpn/openvpn.rst b/docs/vpn/openvpn.rst index cbb89fbe..159366dc 100644 --- a/docs/vpn/openvpn.rst +++ b/docs/vpn/openvpn.rst @@ -183,10 +183,10 @@ Server ====== Multi-client server is the most popular OpenVPN mode on routers. It always uses -x.509 authentication and therefore requires a PKI setup. This guide assumes -`you have already setup a PKI`_ and have a CA certificate, a server certificate and -key, a certificate revocation list, a Diffie-Hellman key exchange parameters -file. You do not need client certificates and keys for the server setup. +x.509 authentication and therefore requires a PKI setup. Refer this section +**Generate X.509 Certificate and Keys** to generate a CA certificate, +a server certificate and key, a certificate revocation list, a Diffie-Hellman +key exchange parameters file. You do not need client certificates and keys for the server setup. In this example we will use the most complicated case: a setup where each client is a router that has its own subnet (think HQ and branch offices), since @@ -254,7 +254,88 @@ internally, so we need to create a route to the 10.23.0.0/20 network ourselves: set protocols static interface-route 10.23.0.0/20 next-hop-interface vtun10 -.. _`you have already setup a PKI`: https://support.vyos.io/en/kb/articles/using-easy-rsa-to-generate-x-509-certificates-and-keys-2 +Generate X.509 Certificate and Keys +*********************************** + +OpenVPN ships with a set of scripts called Easy-RSA that can generate the +appropriate files needed for an OpenVPN setup using X.509 certificates. +Easy-RSA comes installed by default on VyOS routers. + +Copy the Easy-RSA scripts to a new directory to modify the values. + +.. code-block:: none + + cp -r /usr/share/easy-rsa/ /config/my-easy-rsa-config + cd /config/my-easy-rsa-config + +To ensure the consistent use of values when generating the PKI, set default +values to be used by the PKI generating scripts. Rename the vars.example filename +to vars + +.. code-block:: none + + mv vars.example vars + +Following is the instance of the file after editing. You may also change other values in +the file at your discretion/need, though for most cases the defaults should be just fine. +(do not leave any of these parameters blank) + +.. code-block:: none + + set_var EASYRSA_DN "org" + set_var EASYRSA_REQ_COUNTRY "US" + set_var EASYRSA_REQ_PROVINCE "California" + set_var EASYRSA_REQ_CITY "San Francisco" + set_var EASYRSA_REQ_ORG "Copyleft Certificate Co" + set_var EASYRSA_REQ_EMAIL "me@example.net" + set_var EASYRSA_REQ_OU "My Organizational Unit" + set_var EASYRSA_KEY_SIZE 2048 + + +init-pki option will create a new pki directory or will delete any previously generated +certificates stored in that folder. The term 'central' is used to refer server and +'branch' for client + +.. note:: Remember the “CA Key Passphrase” prompted in build-ca command, + as it will be asked in signing the server/client certificate. + +.. code-block:: none + + vyos@vyos:/config/my-easy-rsa-config$./easyrsa init-pki + vyos@vyos:/config/my-easy-rsa-config$./easyrsa build-ca + vyos@vyos:/config/my-easy-rsa-config$./easyrsa gen-req central nopass + vyos@vyos:/config/my-easy-rsa-config$./easyrsa sign-req server central + vyos@vyos:/config/my-easy-rsa-config$./easyrsa gen-dh + vyos@vyos:/config/my-easy-rsa-config$./easyrsa build-client-full branch1 nopass + +To generate a certificate revocation list for any client, execute these commands: + +.. code-block:: none + + vyos@vyos:/config/my-easy-rsa-config$./easyrsa revoke client1 + vyos@vyos:/config/my-easy-rsa-config$ ./easyrsa gen-crl + +Copy the files to /config/auth/ovpn/ to use in OpenVPN tunnel creation + +.. code-block:: none + + vyos@vyos:/config/my-easy-rsa-config$ sudo mkdir /config/auth/ovpn + vyos@vyos:/config/my-easy-rsa-config$ sudo cp pki/ca.crt /config/auth/ovpn + vyos@vyos:/config/my-easy-rsa-config$ sudo cp pki/dh.pem /config/auth/ovpn + vyos@vyos:/config/my-easy-rsa-config$ sudo cp pki/private/central.key /config/auth/ovpn + vyos@vyos:/config/my-easy-rsa-config$ sudo cp pki/issued/central.crt /config/auth/ovpn + vyos@vyos:/config/my-easy-rsa-config$ sudo cp pki/crl.pem /config/auth/ovpn + +Additionally, each client needs a copy of ca.crt and its own client key and cert files. +The files are plaintext so they may be copied either manually, +or through a remote file transfer tool like scp. Whichever method you use, +the files need to end up in the proper location on each router. +For example, Branch 1's router might have the following files: + +.. code-block:: none + + vyos@branch1-rtr:$ ls /config/auth/ovpn + ca.crt branch1.crt branch1.key Client Authentication --------------------- diff --git a/docs/vpn/wireguard.rst b/docs/vpn/wireguard.rst index afd9abfd..3580fac3 100644 --- a/docs/vpn/wireguard.rst +++ b/docs/vpn/wireguard.rst @@ -74,7 +74,7 @@ one. set interfaces wireguard wg01 address '10.1.0.1/24' set interfaces wireguard wg01 description 'VPN-to-wg02' set interfaces wireguard wg01 peer to-wg02 allowed-ips '10.2.0.0/24' - set interfaces wireguard wg01 peer to-wg02 address '192.168.0.142:12345' + set interfaces wireguard wg01 peer to-wg02 address '192.168.0.142' set interfaces wireguard wg01 peer to-wg02 port '12345' set interfaces wireguard wg01 peer to-wg02 pubkey 'XMrlPykaxhdAAiSjhtPlvi30NVkvLQliQuKP7AI7CyI=' set interfaces wireguard wg01 port '12345' @@ -85,6 +85,10 @@ through the WireGuard interface `wg01`. Multiple IPs or networks can be defined and routed, the last check is allowed-ips which either prevents or allows the traffic. +.. note:: You can not assign the same allowed-ips statement to multiple + WireGuard peers. This a a design decission. For more information please + check the `WireGuard mailing list`_. + To use a named key on an interface, the option private-key needs to be set. @@ -257,3 +261,5 @@ Operational commands vyos@wg01# wireguard keypair default + +.. _`WireGuard mailing list`: https://lists.zx2c4.com/pipermail/wireguard/2018-December/003704.html diff --git a/requirements.txt b/requirements.txt index 6672039c..fc8e190e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,5 @@ Sphinx>=1.4.3 sphinx-rtd-theme setuptools -docutils
\ No newline at end of file +docutils +sphinx-notfound-page |
