diff options
| -rw-r--r-- | docs/build-vyos.rst | 81 | ||||
| -rw-r--r-- | docs/contributing/build-vyos.rst | 121 | ||||
| -rw-r--r-- | docs/contributing/coding_guidelines.rst | 164 | ||||
| -rw-r--r-- | docs/contributing/development.rst | 256 | ||||
| -rw-r--r-- | docs/contributing/index.rst | 1 | ||||
| -rw-r--r-- | docs/contributing/upstream-packages.rst | 99 | ||||
| -rw-r--r-- | docs/contributing/vyos_cli.rst | 71 | ||||
| -rw-r--r-- | docs/index.rst | 1 |
8 files changed, 449 insertions, 345 deletions
diff --git a/docs/build-vyos.rst b/docs/build-vyos.rst deleted file mode 100644 index bcd32ecb..00000000 --- a/docs/build-vyos.rst +++ /dev/null @@ -1,81 +0,0 @@ -.. _build: - -Building VyOS using Docker -========================== - -This will guide you though the process of building a VyOS ISO yourself using -Docker and works best on a fresh installation of Debain 8 (Jessie). - -.. note:: Starting with VyOS 1.2 the developers have decided to change their - release model. VyOS is now **free as in speech, but not as in beer**, meaning - that while VyOS is still an open source project, the release ISO's are no - longer free and can only be obtained via subscription, or by contributing to - the community. Since the source code is still public you can build your own - ISO using the process described here. - -Installing Docker and it's prerequisites - -.. code-block:: sh - - root@build:~$ apt update - root@build:~$ apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common - root@build:~$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - - root@build:~$ add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" - root@build:~$ apt update - root@build:~$ apt install docker-ce - -Adding you user to the docker group to be able to execute the ``docker`` command -without sudo. - -.. code-block:: sh - - root@build:~$ usermod -aG docker user - -.. note:: It is recommended to use a non-root user from here on out. - -.. note:: The build process needs to be built on a local file system, building - on SMB or NFS shares is not supported! - -Cloning the vyos-build crux branch and creating the docker container - -.. code-block:: sh - - user@build:~$ git clone -b crux --single-branch https://github.com/vyos/vyos-build.git - user@build:~$ cd vyos-build - user@build:~/vyos-build$ docker build -t vyos-builder docker - -Running the container and building the ISO - -.. code-block:: sh - - user@build:~$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos-builder bash - vyos_bld@d4220bb519a0:/vyos# ./configure --architecture amd64 --build-by "your@email.tld" --build-type release --version 1.2.0 - vyos_bld@d4220bb519a0:/vyos# sudo make iso - -You may use these options to customize you ISO -code-block:: sh - - -h, --help show this help message and exit - --architecture ARCHITECTURE - Image target architecture (amd64 or i586 or armhf) - --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) - --custom-package CUSTOM_PACKAGES - Custom packages to install from repositories - --build-type BUILD_TYPE - Build type, release or development - --debian-security-mirror DEBIAN_SECURITY_MIRROR - Debian security updated mirror - --version VERSION Version number (release builds only) - --debian-mirror DEBIAN_MIRROR - Debian repository mirror for ISO build - --vyos-mirror VYOS_MIRROR - VyOS package mirror - --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR - Debian repository mirror for pbuilder env bootstrap - --debug Enable debug output - --custom-apt-entry CUSTOM_APT_ENTRY - Custom APT entry - --custom-apt-key CUSTOM_APT_KEY - Custom APT key file - -*Your freshly built ISO should now be in the build directory. Good luck!* diff --git a/docs/contributing/build-vyos.rst b/docs/contributing/build-vyos.rst new file mode 100644 index 00000000..34e1dfba --- /dev/null +++ b/docs/contributing/build-vyos.rst @@ -0,0 +1,121 @@ +.. _build: + +Building VyOS +============= + +This will guide you though the process of building a VyOS ISO yourself using +Docker_ and works best on a fresh installation of Debain 8 (Jessie). + +.. note:: Starting with VyOS 1.2 the developers have decided to change their + release model. VyOS is now **free as in speech, but not as in beer**, meaning + that while VyOS is still an open source project, the release ISO's are no + longer free and can only be obtained via subscription, or by contributing to + the community. Since the source code is still public you can build your own + ISO using the process described here. + +Installing Docker_ and it's prerequisites + +.. code-block:: sh + + $ apt-get update + $ apt-get install -y apt-transport-https ca-certificates curl \ + gnupg2 software-properties-common + $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - + $ add-apt-repository "deb [arch=amd64] \ + https://download.docker.com/linux/debian $(lsb_release -cs) stable" + $ apt-get update + $ apt-get install -y docker-ce + +To be able to use Docker_ you need to add your current system user to the +``docker`` group by calling: ``usermod -aG docker yourusername`` + +.. note:: It is recommended to use a non-root user from here on out. + +.. note:: The build process needs to be built on a local file system, building + on SMB or NFS shares is not supported! + + +Generate the container +---------------------- + +You can either build the container yourself or fetch a pre-build one from +Dockerhub. Our Dockerhub organisation (https://hub.docker.com/u/vyos) will +ensure that the container is always up2date. A rebuild is triggered once the +container changes (please note this will take 2-3 hours after pushing to +the vyos-build repository). + +If you waer fancy pants you can - of course - always build the container +yourself directly from the source: + +.. code-block:: sh + + $ git clone -b crux --single-branch https://github.com/vyos/vyos-build + $ docker build -t vyos/vyos-build docker + +.. note: The container is automatically downloaded from Dockerhub if it is not + found on your local machine when the below command is executed - so no + worries. + +.. note: We require one container per build branch, this means that the used + container in ``crux`` and ``current`` can and will differ once we make the + move towards Debian (10) Buster. + + +Build ISO inside container +-------------------------- + +After generating the container - or fetching it pre-build from DockerHub you +are all set to invoke yourself a fresh build of a VyOS ISO. + +.. code-block:: sh + + $ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build bash + vyos_bld@d4220bb519a0:/vyos# ./configure --architecture amd64 \ + --build-by "your@email.tld" \ + --build-type release --version 1.2.0 + vyos_bld@d4220bb519a0:/vyos# sudo make iso + +To select the container you want to run you need to specify the branch you are +interested in, this can be easily done by selecting the appropriate container +image: + +* For VyOS 1.2 (crux) use ``vyos/vyos-build:crux`` +* For VyOS 1.3 (equuleus) use ``vyos/vyos-build:crux`` +* For our VyOS rolling release you should use ``vyos/vyos-build`` which will + always refer to the latest image. + +You may use these options to customize you ISO. You can list yourself all +options by calling ``./configure --help``: + +.. code-block:: sh + + -h, --help show this help message and exit + --architecture ARCHITECTURE + Image target architecture (amd64 or i586 or armhf) + --build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net) + --custom-package CUSTOM_PACKAGES + Custom packages to install from repositories + --build-type BUILD_TYPE + Build type, release or development + --debian-security-mirror DEBIAN_SECURITY_MIRROR + Debian security updated mirror + --version VERSION Version number (release builds only) + --debian-mirror DEBIAN_MIRROR + Debian repository mirror for ISO build + --vyos-mirror VYOS_MIRROR + VyOS package mirror + --pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR + Debian repository mirror for pbuilder env bootstrap + --debug Enable debug output + --custom-apt-entry CUSTOM_APT_ENTRY + Custom APT entry + --custom-apt-key CUSTOM_APT_KEY + Custom APT key file + +Your freshly built ISO should now be in the build directory. Good luck!* + +.. note: The process does not differ when building a ``crux`` ISO or ``rolling`` + one. Only make sure you are using the proper Docker container from the branch + you are trying to build. + +.. _Docker: https://www.docker.com diff --git a/docs/contributing/coding_guidelines.rst b/docs/contributing/coding_guidelines.rst index 9c996518..634c3e61 100644 --- a/docs/contributing/coding_guidelines.rst +++ b/docs/contributing/coding_guidelines.rst @@ -3,21 +3,57 @@ Python Coding Guidelines ======================== -The switch to the Python programming language for new code is not merely a change -of the language, but a chance to rethink and improve the programming approach. +The switch to the Python programming language for new code is not merely a +change of the language, but a chance to rethink and improve the programming +approach. Let's face it: VyOS is full of spaghetti code where logic for reading the VyOS config, generating daemon configs, and restarting processes is all mixed up. -Python (or any other language, for that matter) does not provide automatic protection -from bad design, so we need to also devise design guidelines and follow them to -keep the system extensible and maintainable. +Python (or any other language, for that matter) does not provide automatic +protection from bad design, so we need to also devise design guidelines and +follow them to keep the system extensible and maintainable. + +But we are here to assist you and want to guide you through how you can become +a good VyOS contributor. The rules we have are not there to punish you - the +rules are in place to help us all. What does it mean? By having a consistent +coding style it becomes very easy for new contributors and also longtime +contributors to navigate through the sources and all the implied logic of +the spaghetti code. + +Please use the following template as good starting point when developing new +modules or even rewrite a whole bunch of code in the new style XML/Pyhon +interface. Configuration script structure and behaviour -------------------------------------------- +Your configuration script or operation mode script which is also written in +Python3 should have a line break on 80 characters. This seems to be a bit odd +nowadays but as some people also work remotly or programm using vi(m) this is +a fair good standard which I hope we can rely on. + +In addition this also helps when browsing the GitHub codebase on a mobile +device if you happen to be a crazy scientist. + .. code-block:: python + #!/usr/bin/env python3 + # + # Copyright (C) 2019 VyOS maintainers and contributors + # + # This program is free software; you can redistribute it and/or modify + # it under the terms of the GNU General Public License version 2 or later as + # published by the Free Software Foundation. + # + # This program is distributed in the hope that it will be useful, + # but WITHOUT ANY WARRANTY; without even the implied warranty of + # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + # GNU General Public License for more details. + # + # You should have received a copy of the GNU General Public License + # along with this program. If not, see <http://www.gnu.org/licenses/>. + import sys from vyos.config import Config @@ -50,36 +86,39 @@ Configuration script structure and behaviour print(e) sys.exit(1) -The **get_config()** function must convert the VyOS config to an abstract internal -representation. No other function is allowed to call ``vyos.config.Config`` object -methods directly. The rationale for it is that when config reads are mixed with -other logic, it's very hard to change the config syntax since you need to weed -out every occurrence of the old syntax. If syntax-specific code is confined to a -single function, the rest of the code can be left untouched as long as the -internal representation remains compatible. - -Another advantage is testability of the code. Mocking the entire config subsystem -is hard, while constructing an internal representation by hand is way simpler. - -The **verify()** function takes an internal representation of the config and checks -if it's valid, otherwise it must raise ``VyOSError`` with an error message that -describes the problem and possibly suggests how to fix it. It must not make any -changes to the system. The rationale for it is again testability and, in the -future when the config backend is ready and every script is rewritten in this -fashion, ability to execute commit dry run ("commit test" like in JunOS) and abort -commit before making any changes to the system if an error is found in any component. - -The **generate()** function generates config files for system components. - -The **apply()** function applies the generated configuration to the live system. -It should use non-disruptive reload whenever possible. It may execute disruptive -operations such as daemon process restart if a particular component does not -support non-disruptive reload, or when the expected service degradation is minimal -(for example, in case of auxiliary services such as LLDPd). In case of high impact -services such as VPN daemon and routing protocols, when non-disruptive reload is -supported for some but not all types of configuration changes, scripts authors -should make effort to determine if a configuration change can be done in a -non-disruptive way and only resort to disruptive restart if it cannot be avoided. +The ``get_config()`` function must convert the VyOS config to an abstract, +internal representation. No other function is allowed to call the ``vyos.config. +Config`` object method directly. The rationale for it is that when config reads +are mixed with other logic, it's very hard to change the config syntax since +you need to weed out every occurrence of the old syntax. If syntax-specific +code is confined to a single function, the rest of the code can be left +untouched as long as the internal representation remains compatible. + +Another advantage is testability of the code. Mocking the entire config +subsystem is hard, while constructing an internal representation by hand is +way simpler. + +The ``verify()`` function takes your internal representation of the config and +checks if it's valid, otherwise it must raise ``ConfigError`` with an error +message that describes the problem and possibly suggests how to fix it. It must +not make any changes to the system. The rationale for it is again testability +and, in the future when the config backend is ready and every script is +rewritten in this fashion, ability to execute commit dry run ("commit test" +like in JunOS) and abort commit before making any changes to the system if an +error is found in any component. + +The ``generate()`` function generates config files for system components. + +The ``apply()`` function applies the generated configuration to the live +system. It should use non-disruptive reload whenever possible. It may execute +disruptive operations such as daemon process restart if a particular component +does not support non-disruptive reload, or when the expected service degradation +is minimal (for example, in case of auxiliary services such as LLDPd). In case +of high impact services such as VPN daemon and routing protocols, when non- +disruptive reload is supported for some but not all types of configuration +changes, scripts authors should make effort to determine if a configuration +change can be done in a non-disruptive way and only resort to disruptive restart +if it cannot be avoided. Unless absolutely necessary, configuration scripts should not modify the active configuration of system components directly. Whenever at all possible, scripts @@ -89,34 +128,50 @@ one by one is particularly discouraged, for example, when configuring netfilter rules, saving them to a file and loading it with iptables-restore should always be preferred to executing iptables directly. -The **apply()** and **generate()** functions may ``raise ConfigError`` if, for +The ``apply()`` and ``generate()`` functions may ``raise ConfigError`` if, for example, the daemon failed to start with the updated config. It shouldn't be a -substitute for proper config checking in the **verify()** function. All reasonable -effort should be made to verify that generated configuration is valid and will -be accepted by the daemon, including, when necessary, cross-checks with other -VyOS configuration subtrees. - -Exceptions, including ``VyOSError`` (which is raised by ``vyos.config.Config`` on -improper config operations, such as trying to use ``list_nodes()`` on a non-tag -node) should not be silenced or caught and re-raised as config error. Sure this -will not look pretty on user's screen, but it will make way better bug reports, -and help users (and most VyOS users are IT professionals) do their own debugging -as well. +substitute for proper config checking in the ``verify()`` function. All +reasonable effort should be made to verify that generated configuration is +valid and will be accepted by the daemon, including, when necessary, cross- +checks with other VyOS configuration subtrees. + +Exceptions, including ``VyOSError`` (which is raised by ``vyos.config.Config`` +on improper config operations, such as trying to use ``list_nodes()`` on a +non-tag node) should not be silenced or caught and re-raised as config error. +Sure this will not look pretty on user's screen, but it will make way better +bug reports, and help users (and most VyOS users are IT professionals) do their +own debugging as well. + +For easy orientation we suggest you take a look on the ``ntp.py`` or +``interfaces-bonding.py`` (for tag nodes) implementation. Both files can be +found in the vyos-1x_ repository. + Coding guidelines ----------------- +Like any other project we have some small guidelines about our source code, too. +The rules we have are not there to punish you - the rules are in place to help +us all. By having a consistent coding style it becomes very easy for new +and also longtime contributors to navigate through the sources and all the +implied logic of any one source file.. + Language ******** -Python 3 **shall** be used. How long can we keep Python 2 alive anyway? - -No considerations for Python 2 compatibility **should** be taken. +Python 3 **shall** be used. How long can we keep Python 2 alive anyway? No +considerations for Python 2 compatibility **should** be taken at any time. Formatting ********** -Tabs **shall not** be used. Every indentation level should be 4 spaces. +* Python: Tabs **shall not** be used. Every indentation level should be 4 spaces +* XML: Tabs **shall not** be used. Every indentation level should be 2 spaces + +.. note: There are extensions to e.g. VIM (xmllint) which will help you to get + your indention levels correct. Add to following to your .vimrc file: + ``au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ + 2>/dev/null`` now you can call the linter using ``gg=G`` in command mode. Text generation *************** @@ -134,11 +189,10 @@ Code policy When modifying the source code, remember these rules of the legacy elimination campaign: - * No new features in Perl - * No old style command definitions - * No code incompatible with Python3 +* No new features in Perl +* No old style command definitions +* No code incompatible with Python3 .. _process: https://blog.vyos.io/vyos-development-digest-10 .. _vyos-1x: https://github.com/vyos/vyos-1x/blob/current/schema/ .. _VyConf: https://github.com/vyos/vyconf/blob/master/data/schemata - diff --git a/docs/contributing/development.rst b/docs/contributing/development.rst index 9e99ece8..3ac9cf6f 100644 --- a/docs/contributing/development.rst +++ b/docs/contributing/development.rst @@ -3,199 +3,175 @@ Development =========== -The source code is hosted on GitHub under VyOS organization `github.com/vyos`_ +All VyOS source code is hosted on GitHub under the VyOS organization which can +be found here: https://github.com/vyos -The code is split into modules. VyOS is composed of multiple individual packages, -some of them are periodically synced with upstream, so keeping the whole source -under a single repository would be very inconvenient and slow. There is now an -ongoing effort to consolidate all VyOS-specific framework/config packages into vyos-1x package, -but the basic structure is going to stay the same, just with fewer packages. +Our code is split into several modules. VyOS is composed of multiple individual +packages, some of them are forks of upstream packages and are periodically +synced with upstream, so keeping the whole source under a single repository +would be very inconvenient and slow. There is now an ongoing effort to +consolidate all VyOS-specific framework/config packages into vyos-1x package, +but the basic structure is going to stay the same, just with fewer and fewer +packages while the base code is rewritten from Perl/BASH into Python using and +XML based interface definition for the CLI. -The repository that contains the ISO build script is `vyos-build`_. The README will -guide you to use the this top level repository. +The repository that contains all the ISO build scripts is: +https://github.com/vyos/vyos-build + +The README.md file will guide you to use the this top level repository. -.. _github.com/vyos: https://github.com/vyos -.. _vyos-build: https://github.com/vyos/vyos-build Submit a patch -------------- -Patches are always welcome. -You should follow some procedures and guidelines though. - -Before you make a patch -^^^^^^^^^^^^^^^^^^^^^^^ -In a big system, such as VyOS, that is comprised of multiple components, it's impossible to keep track of all the changes and bugs/feature requests in one's head. We use a `bugtracker`_ for it ("issue tracker" would be a better term, but this one stuck). - -This information is used in two ways: - * Keep track of the progress (what we've already done in this branch and what we still need to do). - * Prepare release notes. - -To make this approach work, every change must be associated with a bug number and component. -If there is no bug/enhancement request for the changes you are going to make, you must create a `bugtracker`_ entry first. -Once there is a `bugtracker`_ entry about it, you should reference in your commit message, as in: - -.. code-block:: sh - - [vyos build] T1327: add serial console (115200,8n1) to ISO kernel command-line - [vyos config] T1397: Rewrite the config merge script - -If there is no reference to an item in our `bugtracker`_ the pull request will be rejected. - -Patch limits: - * If there is a bug that has multiple tasks than it is ok to reference multiple items in a commit and pull request. - * Multiple bugs can not be fixed in one patch if they have no reference to each other. - * We only accept bugfixes in packages other than vyos-1x. - * No new functionality should use old style templates and perl/shell code, use python. - -.. _bugtracker: https://phabricator.vyos.net - -How to make a patch -^^^^^^^^^^^^^^^^^^^ -We only accept patches in git format, because traditional UNIX patch is only good if the code it's going to be applied to doesn't change. If the code changes, merge will fail and the patch will need manual editing, even if there are no real conflicting changes. - -Git keeps more information and uses more sophisticated merge algorithms, which makes a fake conflict a rare occurrence. For the same reason you should always make a patch against the latest current branch. - -You can make a pull request on `github.com/vyos`_. - -.. _github.com/vyos: https://github.com/vyos - -Find the package -^^^^^^^^^^^^^^^^ -Suppose you want to make a change in the webproxy script. - -You can find its package with "dpkg -S": - -.. code-block:: sh - - # dpkg -S /opt/vyatta/sbin/vyatta-update-webproxy.pl - vyatta-webproxy: /opt/vyatta/sbin/vyatta-update-webproxy.pl - -This means it's in `vyatta-webproxy`_ package. +Patches are always more then welcome. To have a clean and easy to maintain +repository we have some guidelines when working with Git. A clean repository +eases the automatic generation of a changelog file. -.. _vyatta-webproxy: https://github.com/vyos/vyatta-webproxy +A good approach for writing commit messages is actually to have a look at the +file(s) history by invoking ``git log path/to/file.txt``. -Make a patch using a fork -^^^^^^^^^^^^^^^^^^^^^^^^^ -Fork the repository in github and make a clone. -.. code-block:: sh +Preparding patch/commit +^^^^^^^^^^^^^^^^^^^^^^^ - git clone https://github.com/<user>/vyatta-webproxy.git +In a big system, such as VyOS, that is comprised of multiple components, it's +impossible to keep track of all the changes and bugs/feature requests in one's +head. We use a bugtracker known as Phabricator_ for it ("issue tracker" would +be a better term, but this one stuck). -Set your name and email in the git config: +The information is used in two ways: -.. code-block:: sh +* Keep track of the progress (what we've already done in this branch and what + we still need to do). - git config user.name "J. Random Hacker" - git config user.email "jrhacker@example.net" +* Prepare release notes for upcoming releases -Make your changes and save them. Then do the following for all changes files: +To make this approach work, every change must be associated with a bug number +(prefixed with **T**) and a component. If there is no bug report/feature request +for the changes you are going to make, you have to create a Phabricator_ task +first. Once there is an entry in Phabricator_, you should reference its id in +your commit message, as shown below: -.. code-block:: sh +* ``ddclient: T1030: auto create runtime directories`` +* ``Jenkins: add current Git commit ID to build description`` - git add myfile +If there is no Phabricator_ reference in the commits of your pull request, we +have to ask you to ammend the commit message. Otherwise we will have to reject +it. - # Or, for a whole dir: - git add somedir/* +In general, use an editor to create your commit messages rather than passing +them on the command line. The format should be and is inspired by this blog +post: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html -Commit the changes: +* A single, short, summary of the commit (recommended 70 characters or less, + but not exceeding 80 characters) -.. code-block:: sh + * Add a prefix of the changed component to your commit headline, e.g. ``snmp: + T1111:`` or ``ethernet: T2222:``. If multiple components are touched by this + commit, you can use multiple prefixes, e.g.: ``snmp: ethernet:`` - git commit +* Followed by a blank line (this is mandatory - else Git will treat the whole + commit message as the headline only) -Please use meaningful commit descriptions and don't forget to reference the bug number there! -Now submit the patch, push and make a pull request. +* Followed by a message which describes all the details like: -Make a patch to mail or attach to an item -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -clone the repository. + * What/why/how something has been changed, makes everyones life easier when + working with `git bisect` -.. code-block:: sh + * All text of the commit message should be wrapped at 72 characters if + possible which makes reading commit logs easier with ``git log`` on a + standard terminal (which happens to be 80x25) - git clone https://github.com/vyos/vyatta-webproxy.git + * If applicable a reference to a previous commit should be made linking + those commits nicely when browsing the history: ``After commit abcd12ef + ("snmp: this is a headline") a Python import statement is missing, + throwing the follwoing exception: ABCDEF`` -Set your name and email in the git config: +* Always use the ``-x`` option to the ``git cherry-pick`` command when back or + forward porting an individual commit. This automatically appends the line: + ``(cherry picked from commit <ID>)`` to the original authors commit message + making it easier when bisecting problems. -.. code-block:: sh +* Every change set must be consistent (self containing)! Do not fix multiple + bugs in a single commit. If you already worked on multiple fixes in the same + file use `git add --patch` to only add the parts related to the one issue + into your upcoming commit. - git config user.name "J. Random Hacker" - git config user.email "jrhacker@example.net" +Limits: -Make your changes and save them. Then do the following for all changes files: +* We only accept bugfixes in packages other than https://github.com/vyos/vyos-1x + as no new functionality should use the old style templates (``node.def`` and + Perl/BASH code. Use the new stlye XML/Python interface instead. -.. code-block:: sh +Please submit your patches using the well-known GitHub pull-request against our +repositories found in the VyOS GitHub organisation at https://github.com/vyos - git add myfile - # Or, for a whole dir: - git add somedir/* +Determining package for a fix +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Commit the changes: +Suppose you want to make a change in the webproxy script but yet you do not know +which of the many VyOS packages ship this file. You can determine the VyOS +package name in question by using Debians ``dpkg -S`` command of your running +VyOS installation. .. code-block:: sh - git commit - -Please use meaningful commit descriptions and don't forget to reference the bug number there! -Export the patch and send it to maintainers@vyos.net or attach to the bug. - -.. code-block:: sh - - git format-patch - - # Or, for multiple commits, suppose you made two: - git format-patch -2 - -Make a patch using a feature branch (maintainers only) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -checkout the current branch and make sure it is up to date. - -.. code-block:: sh + vyos@vyos:~ dpkg -S /opt/vyatta/sbin/vyatta-update-webproxy.pl + vyatta-webproxy: /opt/vyatta/sbin/vyatta-update-webproxy.pl - git clone https://github.com/vyos/vyatta-webproxy.git - git checkout current - git pull origin current +This means the file in question (``/opt/vyatta/sbin/vyatta-update-webproxy.pl``) +is located in the ``vyatta-webproxy`` package which can be found here: +https://github.com/vyos/vyatta-webproxy -Set your name and email in the git config: -.. code-block:: sh +Fork repository to submit a Patch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - git config user.name "J. Random Hacker" - git config user.email "jrhacker@example.net" +Forking the repository and submitting a GitHub pull-request is the preferred +way of submitting your changes to VyOS. You can fork any VyOS repository to your +very own GitHub account by just appending ``/fork`` to any repositories URL on +GitHub. To e.g. fork the ``vyos-1x`` repository, open the following URL in your +favourite browser: https://github.com/vyos/vyos-1x/fork -Create the feature branch: +You then can proceed with cloning your fork or add a new remote to your local +repository: -.. code-block:: sh +* Clone: ``git clone https://github.com/<user>/vyos-1x.git`` - git checkout -b <feature branch name> +* Fork: ``git remote add myfork https://github.com/<user>/vyos-1x.git`` -Make your changes and save them. Then do the following for all changes files: +In order to record you as the author of the fix please indentify yourself to Git +by setting up your name and email. This can be done local for this one and only +repository ``git config`` or globally using ``git config --global``. .. code-block:: sh - git add myfile + git config --global user.name "J. Random Hacker" + git config --global user.email "jrhacker@example.net" - # Or, for a whole dir: - git add somedir/* +Make your changes and save them. Do the following for all changes files to +record them in your created Git commit: -Commit the changes: +* Add file to Git index using ``git add myfile``, or for a whole directory: + ``git add somedir/*`` -.. code-block:: sh +* Commit the changes by calling ``git commit``. Please use a meaningful commit + headline (read above) and don't forget to reference the Phabricator_ ID. - git commit +* Submit the patch ``git push`` and create the GitHub pull-request. -Please use meaningful commit descriptions and don't forget to reference the bug number there! -Rebase on the current repo if needed and push your branch +Attach patch to Phabricator task +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. code-block:: sh +Follow the above steps on how to "Fork repository to submit a Patch". Instead +of uploading "pushing" your changes to GitHub you can export the patches/ +commits and send it to maintainers@vyos.net or attach it directly to the bug +(preferred over email) - git checkout current - git pull origin current - git checkout <feature branch name> - git rebase current - git push -u origin <feature branch name> +* Export last commit to patch file: ``git format-patch`` or export the last two + commits into its appropriate patch files: ``git format-patch -2`` -Now you create a pull request. +.. _Phabricator: https://phabricator.vyos.net
\ No newline at end of file diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst index a7dd55ce..c90d533e 100644 --- a/docs/contributing/index.rst +++ b/docs/contributing/index.rst @@ -9,3 +9,4 @@ vyos_cli coding_guidelines upstream-packages + build-vyos diff --git a/docs/contributing/upstream-packages.rst b/docs/contributing/upstream-packages.rst index 4e602866..161cc627 100644 --- a/docs/contributing/upstream-packages.rst +++ b/docs/contributing/upstream-packages.rst @@ -3,27 +3,30 @@ Upstream packages ================= -Many base system packages are pulled straight from Debian's main and contrib repositories, but there are exceptions. +Many base system packages are pulled straight from Debian's main and contrib +repositories, but there are exceptions. vyos-netplug ------------ -Due to issues in the upstream version that sometimes set interfaces down, a modified version is used. +Due to issues in the upstream version that sometimes set interfaces down, a +modified version is used. -The source is at https://github.com/vyos/vyos-netplug +The source is located at https://github.com/vyos/vyos-netplug -In the future, we may switch to using systemd infrastructure instead. - -Building it doesn't require a special procedure. +In the future, we may switch to using systemd infrastructure instead. Building +it doesn't require a special procedure. keepalived ---------- -Keepalived normally isn't updated to newer feature releases between Debian versions, so we are building it from source. +Keepalived normally isn't updated to newer feature releases between Debian +versions, so we are building it from source. -Debian does keep their package in git, but it's upstream tarball imported into git without its original commit history. -To be able to merge new tags in, we keep a fork of the upstream repository with packaging files imported from Debian -at http://github.com/vyos/keepalived-upstream +Debian does keep their package in git, but it's upstream tarball imported into +git without its original commit history. To be able to merge new tags in, we +keep a fork of the upstream repository with packaging files imported from +Debian at http://github.com/vyos/keepalived-upstream strongswan ---------- @@ -35,11 +38,12 @@ Our StrongSWAN build differs from the upstream: The source is at https://github.com/vyos/vyos-strongswan -DMVPN patches are added by this commit: https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226d4a3e9138e7 +DMVPN patches are added by this commit: +https://github.com/vyos/vyos-strongswan/commit/1cf12b0f2f921bfc51affa3b81226 -Our op mode scripts use the python-vici module, which is not included in Debian's build, -and isn't quite easy to integrate in that build. For this reason we debianize that module by hand now, -using this procedure: +Our op mode scripts use the python-vici module, which is not included in +Debian's build, and isn't quite easy to integrate in that build. For this +reason we debianize that module by hand now, using this procedure: 0. Install https://pypi.org/project/stdeb/ 1. `cd vyos-strongswan` @@ -53,58 +57,86 @@ The package ends up in deb_dist dir. ppp --- -Properly renaming PPTP and L2TP interfaces to pptpX and l2tpX from generic and non-informative pppX requires a patch -that is neither in the upstream nor in Debian. +Properly renaming PPTP and L2TP interfaces to pptpX and l2tpX from generic and +non-informative pppX requires a patch that is neither in the upstream nor in +Debian. We keep a fork of Debian's repo at https://github.com/vyos/ppp-debian The patches for pre-up renaming are: -* https://github.com/vyos/ppp-debian/commit/e728180026a051d2a96396276e7e4ae022899e2d -* https://github.com/vyos/ppp-debian/commit/f29ba8d9ebb043335a096d70bcd07e9635bba2e3 +* https://github.com/vyos/ppp-debian/commit/e728180026a051d2a96396276e7e4ae +* https://github.com/vyos/ppp-debian/commit/f29ba8d9ebb043335a096d70bcd07e9 -Additionally, there's a patch for reopening the log file to better support logging to files, even though it's less essential: -https://github.com/vyos/ppp-debian/commit/dd2ebd5cdcddb40230dc4cc43d374055ff374711 +Additionally, there's a patch for reopening the log file to better support +logging to files, even though it's less essential: +https://github.com/vyos/ppp-debian/commit/dd2ebd5cdcddb40230dc4cc43d374055f The patches were written by Stephen Hemminger back in the Vyatta times. mdns-repeater ------------- -This package doesn't exist in Debian. A debianized fork is kept at https://github.com/vyos/mdns-repeater +This package doesn't exist in Debian. A debianized fork is kept at +https://github.com/vyos/mdns-repeater No special build procedure is required. udp-broadcast-relay ------------------- -This package doesn't exist in Debian. A debianized fork is kept at https://github.com/vyos/udp-broadcast-relay +This package doesn't exist in Debian. A debianized fork is kept at +https://github.com/vyos/udp-broadcast-relay No special build procedure is required. Linux kernel ------------ -TBD +In the past a fork of the Kernel source code was kept at the well-known +location of https://github.com/vyos/vyos-kernel - where it is kept for history. + +Nowadays the Kernel we use is the upstream source code which is patched +with two additional patches from the good old Vyatta times which never made it +into the mainstream Kernel. The patches can be found here: +https://github.com/vyos/vyos-build-kernel/tree/current/patches/kernel and are +automatically applied to the Kernel by the Jenkins Pipeline which is used to +generate the Kernel binaries. + +The Pipeline script not only builds the Kernel with the configuration named +``x86_64_vyos_defconfig`` which is located in the vyos-build-kernel repository, +too - but in addition also builds some Intel out-of-tree drivers, WireGuard +(as long it is not upstreamed) and Accel-PPP. + +The ``Jenkinsfile`` tries to be as verbose as possible on each individual build +step. Linux firmware -------------- -TBD +More and more hardware cards require an additional firmware which is not open +source. The Kernel community hosts a special linux-firmware Git repository +with all available binary files which can be loaded by the Kernel. + +The ``vyos-build`` repository fetches a specific commit of the linux-firmware +repository and embeds those binaries into the resulting ISO image. This step is +done in the ``data/live-build-config/hooks/live/40-linux-firmware.chroot`` file. + +If the firmware needs to be updated it is sufficient to just exchange the Git +commit id we reference in our build. Intel drivers ------------- -TBD +Are build as part of the Kernel Pipeline - read above. -accel-ppp +Accel-PPP --------- -accel-ppp has been packaged for the use with vyos, due to the kernel dependencies for its modules. - -* https://github.com/vyos/vyos-accel-ppp +Accel-PPP used to be an upstream fork for quiet some time but now has been +converted to make use of the upstream source code and build system. -Build instructions are being kept up to date on the repos Readme. +It is build as part of the Kernel Pipeline - read above. hvinfo ------ @@ -113,10 +145,11 @@ A fork with packaging changes for VyOS is kept at https://github.com/vyos/hvinfo The original repo is at https://github.com/dmbaturin/hvinfo -It's an Ada program and requires GNAT and gprbuild for building, dependencies are properly specified -so just follow debuild's suggestions. +It's an Ada program and requires GNAT and gprbuild for building, dependencies +are properly specified so just follow debuild's suggestions. Per-file modifications ------------------------ -vyos-replace package replaces the upstream dhclient-script with a modified version that is aware of the VyOS config. +vyos-replace package replaces the upstream dhclient-script with a modified +version that is aware of the VyOS config. diff --git a/docs/contributing/vyos_cli.rst b/docs/contributing/vyos_cli.rst index 49f9a230..322b0be6 100644 --- a/docs/contributing/vyos_cli.rst +++ b/docs/contributing/vyos_cli.rst @@ -1,12 +1,14 @@ .. _vyos_cli: -VyOS CLI -======== +The VyOS CLI +============ -The bash completion in VyOS is defined in *templates*. Templates are text files -stored in a directory tree, where directory names define command names, and -template files define command behaviour. Before VyOS 1.2.x this files were created -by hand. After a complex redesign process_ the new style template are in XML. +The bash (or better vbash) completion in VyOS is defined in *templates*. +Templates are text files (called ``node.def``) stored in a directory tree. The +directory names define the command names, and template files define the command +behaviour. Before VyOS 1.2 (crux) this files were created by hand. After a +complex redesign process_ the new style template are automatically generated +from a XML input file. XML interface definitions for VyOS come with a RelaxNG schema and are located in the vyos-1x_ module. This schema is a slightly modified schema from VyConf_ @@ -113,15 +115,15 @@ Command syntax guidelines Use of numbers ^^^^^^^^^^^^^^ -Use of numbers in command names **should** be avoided unless a number is a part -of a protocol name or similar. Thus, ``protocols ospfv3`` is perfectly fine, but -something like ``server-1`` is questionable at best. +Use of numbers in command names **should** be avoided unless a number is a +part of a protocol name or similar. Thus, ``protocols ospfv3`` is perfectly +fine, but something like ``server-1`` is questionable at best. Help string guidelines ********************** -To ensure uniform look and feel, and improve readability, we should follow a set -of guidelines consistently. +To ensure uniform look and feel, and improve readability, we should follow a +set of guidelines consistently. Capitalization and punctuation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -134,10 +136,10 @@ a good aesthetic compromise. Examples: - * Good: "Frobnication algorithm" - * Bad: "frobnication algorithm" - * Bad: "Frobnication algorithm." - * Horrible: "frobnication algorithm." +* Good: "Frobnication algorithm" +* Bad: "frobnication algorithm" +* Bad: "Frobnication algorithm." +* Horrible: "frobnication algorithm." Use of abbreviations and acronyms ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -146,29 +148,29 @@ Abbreviations and acronyms **must** be capitalized. Examples: - * Good: "TCP connection timeout" - * Bad: "tcp connection timeout" - * Horrible: "Tcp connectin timeout" +* Good: "TCP connection timeout" +* Bad: "tcp connection timeout" +* Horrible: "Tcp connectin timeout" Acronyms also **must** be capitalized to visually distinguish them from normal words: Examples: - * Good: RADIUS (as in remote authentication for dial-in user services) - * Bad: radius (unless it's about the distance between a center of a circle and - any of its points) +* Good: RADIUS (as in remote authentication for dial-in user services) +* Bad: radius (unless it's about the distance between a center of a circle and + any of its points) Some abbreviations are traditionally written in mixed case. Generally, if it -contains words "over" or "version", the letter **should** be lowercase. If there's -an accepted spelling (especially if defined by an RFC or another standard), it -**must** be followed. +contains words "over" or "version", the letter **should** be lowercase. If +there's an accepted spelling (especially if defined by an RFC or another +standard), it **must** be followed. Examples: - * Good: PPPoE, IPsec - * Bad: PPPOE, IPSEC - * Bad: pppoe, ipsec +* Good: PPPoE, IPsec +* Bad: PPPOE, IPSEC +* Bad: pppoe, ipsec Use of verbs ^^^^^^^^^^^^ @@ -177,12 +179,12 @@ Verbs **should** be avoided. If a verb can be omitted, omit it. Examples: - * Good: "TCP connection timeout" - * Bad: "Set TCP connection timeout" +* Good: "TCP connection timeout" +* Bad: "Set TCP connection timeout" -If a verb is essential, keep it. For example, in the help text of `set system ipv6 -disable-forwarding`, "Disable IPv6 forwarding on all interfaces" is a perfectly -justified wording. +If a verb is essential, keep it. For example, in the help text of ``set system +ipv6 disable-forwarding`, "Disable IPv6 forwarding on all interfaces" is a +perfectly justified wording. Prefer infinitives ^^^^^^^^^^^^^^^^^^ @@ -191,8 +193,8 @@ Verbs, when they are necessary, **should** be in their infinitive form. Examples: - * Good: "Disable IPv6 forwarding" - * Bad: "Disables IPv6 forwarding" +* Good: "Disable IPv6 forwarding" +* Bad: "Disables IPv6 forwarding" Mapping old node.def style to new XML definitions ------------------------------------------------- @@ -261,4 +263,3 @@ Mapping old node.def style to new XML definitions .. _process: https://blog.vyos.io/vyos-development-digest-10 .. _vyos-1x: https://github.com/vyos/vyos-1x/blob/current/schema/ .. _VyConf: https://github.com/vyos/vyconf/blob/master/data/schemata - diff --git a/docs/index.rst b/docs/index.rst index 59b74b38..2daf6299 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -52,7 +52,6 @@ as a router and firewall platform for cloud deployments. :caption: Contributing: :includehidden: - build-vyos.rst contributing/index.rst |