contributing: clear structure

3 files changed, 413 insertions, 11 deletions

diff --git a/docs/contributing/debugging.rst b/docs/contributing/debugging.rst
new file mode 100644
index 00000000..fec73257
--- /dev/null
+++ b/docs/contributing/debugging.rst
@@ -0,0 +1,206 @@
+.. _debugging:
+
+#########
+Debugging
+#########
+
+There are two flags available to aid in debugging configuration scripts.
+Since configuration loading issues will manifest during boot, the flags are
+passed as kernel boot parameters.
+
+ISO image build
+===============
+
+When having trouble compiling your own ISO image or debugging Jenkins issues
+you can follow the steps at :ref:`iso_build_issues`.
+
+System Startup
+==============
+
+The system startup can be debugged (like loading in the configuration
+file from ``/config/config.boot``. This can be achieve by extending the
+Kernel command-line in the bootloader.
+
+Kernel
+------
+
+* ``vyos-debug`` - Adding the parameter to the linux boot line will produce
+  timing results for the execution of scripts during commit. If one is seeing
+  an unexpected delay during manual or boot commit, this may be useful in
+  identifying bottlenecks. The internal flag is ``VYOS_DEBUG``, and is found
+  in vyatta-cfg_. Output is directed to ``/var/log/vyatta/cfg-stdout.log``.
+
+* ``vyos-config-debug`` - During development, coding errors can lead to a
+  commit failure on boot, possibly resulting in a failed initialization of the
+  CLI. In this circumstance, the kernel boot parameter ``vyos-config-debug``
+  will ensure access to the system as user ``vyos``, and will log a Python
+  stack trace to the file ``/tmp/boot-config-trace``.
+  File ``boot-config-trace`` will generate only if config loaded with a failure
+  status.
+
+Live System
+===========
+
+A number of flags can be set up to change the behaviour of VyOS at runtime.
+These flags can be toggled using either environment variables or creating
+files.
+
+For each feature, a file called ``vyos.feature.debug`` can be created to
+toggle the feature on. If a parameter is required it can be placed inside
+the file as its first line.
+
+The file can be placed in ``/tmp`` for one time debugging (as the file
+will be removed on reboot) or placed in '/config' to stay permanently.
+
+For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable
+interface debugging.
+
+It is also possible to set up the debugging using environment variables.
+In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG.
+
+For example running, ``export VYOS_IFCONFIG_DEBUG=""`` on your vbash,
+will have the same effect as ``touch /tmp/vyos.ifconfig.debug``.
+
+* ``ifconfig`` - Once set, all commands used, and their responses received
+  from the OS, will be presented on the screen for inspection.
+
+* ``command`` - Once set, all commands used, and their responses received
+  from the OS, will be presented on the screen for inspection.
+
+* ``developer`` - Should a command fail, instead of printing a message to the
+  user explaining how to report issues, the python interpreter will start a
+  PBD post-mortem session to allow the developer to debug the issue. As the
+  debugger will wait from input from the developer, it has the capacity to
+  prevent a router to boot and therefore should only be permanently set up
+  on production if you are ready to see the OS fail to boot.
+
+* ``log`` - In some rare cases, it may be useful to see what the OS is doing,
+  including during boot. This option sends all commands used by VyOS to a
+  file. The default file is ``/tmp/full-log`` but it can be changed.
+
+.. note:: In order to retrieve the debug output on the command-line you need to
+  disable ``vyos-configd`` in addition. This can be run either one-time by
+  calling ``sudo systemctl stop vyos-configd`` or make this reboot-safe by
+  calling ``sudo systemctl disable vyos-configd``.
+
+FRR
+---
+
+Recent versions use the ``vyos.frr`` framework. The Python class is located
+inside our ``vyos-1x:python/vyos/frr.py``. It comes with an embedded debugging/
+(print style) debugger as vyos.ifconfig does.
+
+To enable debugging just run: ``$ touch /tmp/vyos.frr.debug``
+
+Debugging Python Code with PDB
+------------------------------
+
+Sometimes it might be useful to debug Python code interactively on the live
+system rather than a IDE. This can be achieved using pdb.
+
+Let us assume you want to debug a Python script that is called by an op-mode
+command. After you found the script by looking up the op-mode-defitions you
+can edit the script in the live system using e.g. vi:
+``vi /usr/libexec/vyos/op_mode/show_xyz.py``
+
+Insert the following statement right before the section where you want to
+investigate a problem (e.g. a statement you see in a backtrace):
+``import pdb; pdb.set_trace()``
+Optionally you can surrounded this statement by an ``if`` which only triggers
+under the condition you are interested in.
+
+Once you run ``show xyz`` and your condition is triggered you should be dropped
+into the python debugger:
+
+
+.. code-block:: none
+
+   > /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process()
+   -> rule_type = rule.get('type', '')
+   (Pdb)
+
+You can type ``help`` to get an overview of the available commands, and
+``help command`` to get more information on each command.
+
+Useful commands are:
+
+* examine variables using ``pp(var)``
+* contine execution using ``cont``
+* get a backtrace using ``bt``
+
+Config Migration Scripts
+------------------------
+
+When writing a new configuration migrator it may happen that you see an error
+when you try to invoke it manually on a development system. This error will
+look like:
+
+.. code-block:: none
+
+  vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot
+  Traceback (most recent call last):
+    File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module>
+      config = ConfigTree(config_file)
+    File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__
+      raise ValueError("Failed to parse config: {0}".format(msg))
+  ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax.
+
+The reason is that the configuration migration backend is rewritten and uses
+a new form of "magic string" which is applied on demand when real config
+migration is run on boot. When runnint individual migrators for testing,
+you need to convert the "magic string" on your own by:
+
+.. code-block:: none
+
+  vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot
+
+Configuration Error on System Boot
+----------------------------------
+
+Beeing brave and running the latest rolling releases will sometimes trigger
+bugs due to corner cases we missed in our design. Those bugs should be filed
+via Phabricator_ but you can help us to narrow doen the issue. Login to your
+VyOS system and change into configuration mode by typing ``configure``. Now
+re-load your boot configuration by simply typing ``load`` followed by return.
+
+You shoudl now see a Python backtrace which will help us to handle the issue,
+please attach it to the Phabricator_ task.
+
+Boot Timing
+-----------
+
+During the migration and extensive rewrite of functionality from Perl into
+Python a significant increase in the overall system boottime was noticed. The
+system boot time can be analysed and a graph can be generated in the end which
+shows in detail who called whom during the system startup phase.
+
+This is done by utilizing the ``systemd-bootchart`` package which is now
+installed by default on the VyOS 1.3 (equuleus) branch. The configuration is
+also versioned so we get comparable results. ``systemd-bootchart`` is configured
+using this file: bootchart.conf_
+
+To enable boot time graphing change the Kernel commandline and add the folowing
+string: ``init=/usr/lib/systemd/systemd-bootchart``
+
+This can also be done permanently by changing ``/boot/grub/grub.cfg``.
+
+Priorities
+==========
+
+VyOS CLI is all about priorities. Every CLI node has a corresponding
+``node.def`` file and possibly an attached script that is executed when the
+node is present. Nodes can have a priority, and on system bootup - or any
+other ``commit`` to the config all scripts are executed from lowest to higest
+priority. This is good as this gives a deterministic behavior.
+
+To debug issues in priorities or to see what's going on in the background
+you can use the ``/opt/vyatta/sbin/priority.pl`` script which lists to you
+the execution order of the scripts.
+
+.. stop_vyoslinter
+
+.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg
+.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf
+.. include:: /_include/common-references.txt
+
+.. start_vyoslinter
diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst
deleted file mode 100644
index a5942f96..00000000
--- a/docs/contributing/index.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-############
-Contributing
-############
-
-.. toctree::
-   :maxdepth: 2
-
-   build-vyos
-   development
-   issues-features
-   upstream-packages
diff --git a/docs/contributing/testing.rst b/docs/contributing/testing.rst
new file mode 100644
index 00000000..d5df9d59
--- /dev/null
+++ b/docs/contributing/testing.rst
@@ -0,0 +1,207 @@
+.. _testing:
+
+#######
+Testing
+#######
+
+One of the major advantages introduced in VyOS 1.3 is an autmated test framework.
+When assembling an ISO image multiple things can go wrong badly and publishing
+a faulty ISO makes no sense. The user is disappointed by the quality of the image
+and the developers get flodded with bug reports over and over again.
+
+As the VyOS documentation is not only for users but also for the developers -
+and we keep no secret documentation - this section describes how the automated
+testing works.
+
+Jenkins CI
+==========
+
+Our `VyOS CI`_ system is based on Jenkins and builds all our required packages
+for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build
+Job which builds and tests the VyOS ISO image which is published after a
+successfull test drive.
+
+We differentiate in two independent tests, which are both run in parallel by
+two separate QEmu instances which are launched via ``make test`` and ``make
+testc`` from within the vyos-build_ repository.
+
+Smoketests
+==========
+
+Smoketests executes predefined VyOS CLI commands and checks if the desired
+daemon/service configuration is rendert - that is how to put it "short".
+
+When and ISO image is assembled by the `VyOS CI`_, the ``BUILD_SMOKETEST``
+parameter is enabled by default, which will extend the ISO configuration line
+with the following packages:
+
+.. code-block:: python
+
+  def CUSTOM_PACKAGES = ''
+    if (params.BUILD_SMOKETESTS)
+      CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest'
+
+So if you plan to build your own custom ISO image and wan't to make use of our
+smoketests, ensure that you have the `vyos-1x-smoketest` package installed.
+
+The ``make test`` command from the vyos-build_ repository will launch a new
+QEmu instance and the ISO image is first installed to the virtual harddisk.
+
+After its first boot into the newly installed system the main Smoketest script
+is executed, it can be found here: `/usr/bin/vyos-smoketest`
+
+The script only searches for executable "test-cases" under
+``/usr/libexec/vyos/tests/smoke/cli/`` and executes them one by one.
+
+.. note:: As Smoketests will alter the system configuration and you are logged
+   in remote you may loose your connection to the system.
+
+Manual Smoketest Run
+--------------------
+
+On the other hand - as each test is contain in its own file - one can always
+execute a single Smoketest by hand by simply running the Python test scripts.
+
+Example:
+
+.. code-block:: none
+
+  vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
+  test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok
+  test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok
+  test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok
+  test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok
+  test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok
+  test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok
+  test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok
+  test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok
+  test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok
+  test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok
+  test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok
+  test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok
+  test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok
+
+  ----------------------------------------------------------------------
+  Ran 13 tests in 348.191s
+
+  OK
+
+Interface based tests
+---------------------
+
+Our smoketests not only test daemons and serives, but also check if what we
+configure for an interface works. Thus there is a common base classed named:
+``base_interfaces_test.py`` which holds all the common code that an interface
+supports and is tested.
+
+Those common tests consists out of:
+
+* Add one or more IP addresses
+* DHCP client and DHCPv6 prefix delegation
+* MTU size
+* IP and IPv6 options
+* Port description
+* Port disable
+* VLANs (QinQ and regular 802.1q)
+* ...
+
+.. note:: When you are working on interface configuration and you also wan't to
+   test if the Smoketests pass you would normally loose the remote SSH connection
+   to your :abbr:`DUT (Device Under Test)`. To handle this issue, some of the
+   interface based tests can be called with an environment variable beforehand
+   to limit the number of interfaces used in the test. By default all interface
+   e.g. all Ethernet interfaces are used.
+
+.. code-block:: none
+
+  vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py
+  test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok
+  test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok
+  test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok
+  test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok
+  test_bonding_min_links (__main__.BondingInterfaceTest) ... ok
+  test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok
+  test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok
+  test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok
+  test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok
+  test_interface_description (__main__.BondingInterfaceTest) ... ok
+  test_interface_disable (__main__.BondingInterfaceTest) ... ok
+  test_interface_ip_options (__main__.BondingInterfaceTest) ... ok
+  test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok
+  test_interface_mtu (__main__.BondingInterfaceTest) ... ok
+  test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok
+  test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok
+  test_span_mirror (__main__.BondingInterfaceTest) ... ok
+  test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok
+  test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok
+  test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok
+  test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok
+  test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok
+  test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok
+
+  ----------------------------------------------------------------------
+  Ran 23 tests in 244.694s
+
+  OK
+
+This will limit the `bond` interface test to only make use of `eth1` and `eth2`
+as member ports.
+
+Config Load Tests
+=================
+
+The other part of our tests are called "config load tests". The config load tests
+will load - one after another - arbitrary configuration files to test if the
+configuration migration scripts work as designed and that a given set of
+functionality still can be loaded with a fresh VyOS ISO image.
+
+The configurations are all derived from production systems and can not only act
+as a testcase but also as reference if one wants to enable a certain feature.
+The configurations can be found here:
+https://github.com/vyos/vyos-1x/tree/current/smoketest/configs
+
+The entire test is controlled by the main wrapper script ``/usr/bin/vyos-configtest``
+which behaves in the same way as the main smoketest script. It scans the folder
+for potential configuration files and issues a ``load`` command one after another.
+
+Manual config load test
+-----------------------
+
+One is not bound to load all configurations one after another but can also load
+individual test configurations on his own.
+
+.. code-block:: none
+
+  vyos@vyos:~$ configure
+  load[edit]
+
+  vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small
+  Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small'
+  Load complete. Use 'commit' to make changes effective.
+  [edit]
+  vyos@vyos# compare
+  [edit interfaces ethernet eth0]
+  -hw-id 00:50:56:bf:c5:6d
+  [edit interfaces ethernet eth1]
+  +duplex auto
+  -hw-id 00:50:56:b3:38:c5
+  +speed auto
+  [edit interfaces]
+  -ethernet eth2 {
+  -    hw-id 00:50:56:b3:9c:1d
+  -}
+  -vti vti1 {
+  -    address 192.0.2.1/30
+  -}
+  ...
+
+  vyos@vyos# commit
+  vyos@vyos#
+
+.. note:: Some of the configurations have preconditions which need to be met.
+   Those most likely include generation of crypographic keys before the config
+   can be applied - you will get a commit error otherwise. If you are interested
+   how those preconditions are fulfilled check the vyos-build_ repository and
+   the ``scripts/check-qemu-install`` file.
+
+.. include:: /_include/common-references.txt