troubleshooting: move out of appendix

1 files changed, 346 insertions, 0 deletions

diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
new file mode 100644
index 00000000..6448d031
--- /dev/null
+++ b/docs/troubleshooting.rst
@@ -0,0 +1,346 @@
+.. _troubleshooting:
+
+Troubleshooting
+===============
+
+Sometimes things break or don't work as expected. This section describes
+several troubleshooting tools provided by VyOS that can help when something
+goes wrong.
+
+Basic Connectivity Verification
+-------------------------------
+
+Verifying connectivity can be done with the familiar `ping` and `traceroute`
+commands. The options for each are shown (the options for each command were
+displayed using the built-in help as described in the :ref:`cli`
+section and are omitted from the output here):
+
+.. code-block:: sh
+
+  vyos@vyos:~$ ping
+  Possible completions:
+    <hostname>    Send Internet Control Message Protocol (ICMP) echo request
+    <x.x.x.x>
+    <h:h:h:h:h:h:h:h>
+
+Several options are available when more extensive troubleshooting is needed:
+
+.. code-block:: sh
+
+  vyos@vyos:~$ ping 8.8.8.8
+  Possible completions:
+    <Enter>       Execute the current command
+    adaptive      Ping options
+    allow-broadcast
+    audible
+    bypass-route
+    count
+    deadline
+    flood
+    interface
+    interval
+    mark
+    no-loopback
+    numeric
+    pattern
+    quiet
+    record-route
+    size
+    timestamp
+    tos
+    ttl
+    verbose
+
+.. code-block:: sh
+
+  vyos@vyos:~$ traceroute
+  Possible completions:
+    <hostname>    Track network path to specified node
+    <x.x.x.x>
+    <h:h:h:h:h:h:h:h>
+    ipv4          Track network path to <hostname|IPv4 address>
+    ipv6          Track network path to <hostname|IPv6 address>
+
+However, another tool, mtr_, is available which combines ping and traceroute
+into a single tool. An example of its output is shown:
+
+.. code-block:: sh
+
+  vyos@vyos:~$ mtr 10.62.212.12
+
+                             My traceroute  [v0.85]
+  vyos (0.0.0.0)
+  Keys:  Help   Display mode   Restart statistics   Order of fields   quit
+                                    Packets               Pings
+  Host                            Loss%   Snt   Last   Avg  Best  Wrst StDev
+  1. 10.11.110.4                   0.0%    34    0.5   0.5   0.4   0.8   0.1
+  2. 10.62.255.184                 0.0%    34    1.1   1.0   0.9   1.4   0.1
+  3. 10.62.255.71                  0.0%    34    1.4   1.4   1.3   2.0   0.1
+  4. 10.62.212.12                  0.0%    34    1.6   1.6   1.6   1.7   0.0
+
+.. note:: The output of ``mtr`` consumes the screen and will replace your
+   command prompt.
+
+Several options are available for changing the display output. Press `h` to
+invoke the built in help system. To quit, just press `q` and you'll be returned
+to the VyOS command prompt.
+
+Monitoring
+----------
+
+Network Interfaces
+^^^^^^^^^^^^^^^^^^
+
+It's possible to monitor network traffic, either at the flow level or protocol
+level. This can be useful when troubleshooting a variety of protocols and
+configurations. The following interface types can be monitored:
+
+.. code-block:: sh
+
+  vyos@vyos:~$ monitor interfaces
+  Possible completions:
+    <Enter>       Execute the current command
+    bonding       Monitor a bonding interface
+    bridge        Monitor a bridge interface
+    ethernet      Monitor a ethernet interface
+    loopback      Monitor a loopback interface
+    openvpn       Monitor an openvpn interface
+    pppoe         Monitor pppoe interface
+    pseudo-ethernet
+                  Monitor a pseudo-ethernet interface
+    tunnel        Monitor a tunnel interface
+    vrrp          Monitor a vrrp interface
+    vti           Monitor a vti interface
+    wireless      Monitor wireless interface
+
+To monitor traffic flows, issue the :code:`monitor interfaces <type> <name> flow`
+command, replacing `<type>` and `<name>` with your desired interface type and
+name, respectively. Output looks like the following:
+
+.. code-block:: sh
+
+                     12.5Kb              25.0Kb              37.5Kb              50.0Kb        62.5Kb
+  ????????????????????????????????????????????????????????????????????????????????????????????????????
+  10.11.111.255                        => 10.11.110.37                            0b      0b      0b
+                                      <=                                       624b    749b    749b
+  10.11.110.29                         => 10.62.200.11                            0b    198b    198b
+                                      <=                                         0b    356b    356b
+  255.255.255.255                      => 10.11.110.47                            0b      0b      0b
+                                      <=                                       724b    145b    145b
+  10.11.111.255                        => 10.11.110.47                            0b      0b      0b
+                                      <=                                       724b    145b    145b
+  10.11.111.255                        => 10.11.110.255                           0b      0b      0b
+                                      <=                                       680b    136b    136b
+  ????????????????????????????????????????????????????????????????????????????????????????????????????
+  TX:             cumm:  26.7KB   peak:   40.6Kb                      rates:   23.2Kb  21.4Kb  21.4Kb
+  RX:                    67.5KB           63.6Kb                               54.6Kb  54.0Kb  54.0Kb
+  TOTAL:                 94.2KB            104Kb                               77.8Kb  75.4Kb  75.4Kb
+
+Several options are available for changing the display output. Press `h` to
+invoke the built in help system. To quit, just press `q` and you'll be returned
+to the VyOS command prompt.
+
+To monitor interface traffic, issue the :code:`monitor interfaces <type> <name>
+traffic` command, replacing `<type>` and `<name>` with your desired interface
+type and name, respectively. This command invokes the familiar tshark_ utility
+and the following options are available:
+
+.. code-block:: sh
+
+  vyos@vyos:~$ monitor interfaces ethernet eth0 traffic
+  Possible completions:
+    <Enter>       Execute the current command
+    detail        Monitor detailed traffic for the specified ethernet interface
+    filter        Monitor filtered traffic for the specified ethernet interface
+    save          Save monitored traffic to a file
+    unlimited     Monitor traffic for the specified ethernet interface
+
+To quit monitoring, press `Ctrl-c` and you'll be returned to the VyOS command
+prompt. The `detail` keyword provides verbose output of the traffic seen on
+the monitored interface. The `filter` keyword accepts valid `PCAP filter
+expressions`_, enclosed in single or double quotes (e.g. "port 25" or "port 161
+and udp"). The `save` keyword allows you to save the traffic dump to a file.
+The `unlimited` keyword is used to specify that an unlimited number of packets
+can be captured (by default, 1,000 packets are captured and you're returned to
+the VyOS command prompt).
+
+Interface Bandwith
+^^^^^^^^^^^^^^^^^^
+
+to take a quick view on the used bandwith of an interface use the ``monitor
+bandwith`` command
+
+.. code-block:: sh
+
+  vyos@vyos:~$ monitor bandwidth interface eth0
+
+show the following:
+
+.. code-block:: sh
+
+         B                      (RX Bytes/second)
+    198.00 .|....|.....................................................
+    165.00 .|....|.....................................................
+    132.00 ||..|.|.....................................................
+     99.00 ||..|.|.....................................................
+     66.00 |||||||.....................................................
+     33.00 |||||||.....................................................
+           1   5   10   15   20   25   30   35   40   45   50   55   60
+
+       KiB                      (TX Bytes/second)
+      3.67 ......|.....................................................
+      3.06 ......|.....................................................
+      2.45 ......|.....................................................
+      1.84 ......|.....................................................
+      1.22 ......|.....................................................
+      0.61 :::::||.....................................................
+           1   5   10   15   20   25   30   35   40   45   50   55   60
+
+Interface performance
+^^^^^^^^^^^^^^^^^^^^^
+
+To take a look on the network bandwith between two nodes, the ``monitor
+bandwidth-test`` command is used to run iperf.
+
+.. code-block:: sh
+
+  vyos@vyos:~$ monitor bandwidth-test
+  Possible completions:
+    accept        Wait for bandwidth test connections (port TCP/5001)
+    initiate      Initiate a bandwidth test
+
+* The ``accept`` command open a listen iperf server on TCP Port 5001
+* The ``initiate`` command conncet to this server.
+
+.. code-block:: sh
+
+  vyos@vyos:~$ monitor bandwidth-test initiate
+  Possible completions:
+    <hostname>    Initiate a bandwidth test to specified host (port TCP/5001)
+    <x.x.x.x>
+    <h:h:h:h:h:h:h:h>
+
+
+Monitor command
+^^^^^^^^^^^^^^^
+
+The ``monitor command`` command allows you to repeatedly run a command to view
+a continuously refreshed output. The command is run and output every 2 seconds,
+allowing you to monitor the output continuously without having to re-run the
+command. This can be useful to follow routing adjacency formation.
+
+.. code-block:: sh
+
+  vyos@router:~$ monitor command "show interfaces"
+
+Will clear the screen and show you the output of ``show interfaces`` every
+2 seconds.
+
+.. code-block:: sh
+
+  Every 2.0s: /opt/vyatta/bin/vyatta-op-cmd-wrapper    Sun Mar 26 02:49:46 2019
+
+  Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
+  Interface        IP Address                        S/L  Description
+  ---------        ----------                        ---  -----------
+  eth0             192.168.1.1/24                    u/u
+  eth0.5           198.51.100.4/24                   u/u  WAN
+  lo               127.0.0.1/8                       u/u
+                   ::1/128
+  vti0             172.32.254.2/30                   u/u
+  vti1             172.32.254.9/30                   u/u
+
+Clear Command
+-------------
+
+Sometimes you need to clear counters or statistics to troubleshoot better.
+
+To do this use the ``clear`` command in Operational mode.
+
+to clear the console output
+
+.. code-block:: sh
+
+  vyos@vyos:~$ clear console
+
+to clear interface counters
+
+.. code-block:: sh
+
+  # clear all interfaces
+  vyos@vyos:~$ clear interface ethernet counters
+  # clear specific interface
+  vyos@vyos:~$ clear interface ehternet eth0 counters
+
+The command follow the same logic as the ``set`` command in configuration mode.
+
+.. code-block:: sh
+
+  # clear all counters of a interface type
+  vyos@vyos:~$ clear interface <interface_type> counters
+  # clear counter of a interface in interface_type
+  vyos@vyos:~$ clear interface <interface_type> <interace_name> counters
+
+
+to clear counters on firewall rulesets or single rules
+
+.. code-block:: sh
+
+  vyos@vyos:~$ clear firewall name <ipv4 ruleset name> counters
+  vyos@vyos:~$ clear firewall name <ipv4 ruleset name> rule <rule#> counters
+
+  vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> counters
+  vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> rule <rule#> counters
+
+
+Basic System Information
+------------------------
+
+Boot steps
+^^^^^^^^^^
+
+VyOS 1.2 uses `Debian Jessie`_ as the base Linux operating system. Jessie was
+the first version of Debian that uses systemd_ as the default init system.
+
+These are the boot steps for VyOS 1.2
+
+1. The BIOS loads Grub (or isolinux for the Live CD)
+2. Grub then starts the Linux boot and loads the Linux Kernel ``/boot/vmlinuz``
+3. Kernel Launches Systemd ``/lib/systemd/systemd``
+4. Systemd loads the VyOS service file
+   ``/lib/systemd/system/vyos-router.service``
+5. The service file launches the VyOS router init script
+   ``/usr/libexec/vyos/init/vyos-router`` - this is part of the vyatta-cfg_
+   Debian package
+
+  1. Starts FRR_ - successor to `GNU Zebra`_ and Quagga_
+
+  2. Initialises the boot configuration file - copies over
+     ``config.boot.default`` if there is no configuration
+  3. Runs the configuration migration, if the configuration is for an older
+     version of VyOS
+  4. Runs The pre-config script, if there is one
+     ``/config/scripts/vyos-preconfig-bootup.script``
+  5. If the config file was upgraded, runs any post upgrade scripts
+     ``/config/scripts/post-upgrade.d``
+  6. Starts ``rl-system`` and ``firewall``
+  7. Mounts the ``/boot`` partition
+  8. The boot configuration file is then applied by ``/opt/vyatta/sbin/
+     vyatta-boot-config-loader/opt/vyatta/etc/config/config.boot``
+
+    1. The config loader script writes log entries to
+       ``/var/log/vyatta-config-loader.log``
+
+  10. Runs ``telinit q`` to tell the init system to reload ``/etc/inittab``
+  11. Finally it runs the post-config script
+      ``/config/scripts/vyos-postconfig-bootup.script``
+
+.. _Quagga: http://www.quagga.net/
+.. _`GNU Zebra`: https://www.gnu.org/software/zebra/
+.. _FRR: https://frrouting.org/
+.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg
+.. _systemd: https://freedesktop.org/wiki/Software/systemd/
+.. _`Debian Jessie`: https://www.debian.org/releases/jessie/
+.. _mtr: http://www.bitwizard.nl/mtr/
+.. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html
+.. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html