From 2bdc13ab4805eb674900d1d5a16dbcb648be6667 Mon Sep 17 00:00:00 2001 From: Christian Poessinger Date: Sat, 23 Nov 2019 19:10:29 +0100 Subject: troubleshooting: move out of appendix --- docs/appendix/troubleshooting.rst | 346 -------------------------------------- docs/index.rst | 2 +- docs/troubleshooting.rst | 346 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 347 insertions(+), 347 deletions(-) delete mode 100644 docs/appendix/troubleshooting.rst create mode 100644 docs/troubleshooting.rst diff --git a/docs/appendix/troubleshooting.rst b/docs/appendix/troubleshooting.rst deleted file mode 100644 index 6448d031..00000000 --- a/docs/appendix/troubleshooting.rst +++ /dev/null @@ -1,346 +0,0 @@ -.. _troubleshooting: - -Troubleshooting -=============== - -Sometimes things break or don't work as expected. This section describes -several troubleshooting tools provided by VyOS that can help when something -goes wrong. - -Basic Connectivity Verification -------------------------------- - -Verifying connectivity can be done with the familiar `ping` and `traceroute` -commands. The options for each are shown (the options for each command were -displayed using the built-in help as described in the :ref:`cli` -section and are omitted from the output here): - -.. code-block:: sh - - vyos@vyos:~$ ping - Possible completions: - Send Internet Control Message Protocol (ICMP) echo request - - - -Several options are available when more extensive troubleshooting is needed: - -.. code-block:: sh - - vyos@vyos:~$ ping 8.8.8.8 - Possible completions: - Execute the current command - adaptive Ping options - allow-broadcast - audible - bypass-route - count - deadline - flood - interface - interval - mark - no-loopback - numeric - pattern - quiet - record-route - size - timestamp - tos - ttl - verbose - -.. code-block:: sh - - vyos@vyos:~$ traceroute - Possible completions: - Track network path to specified node - - - ipv4 Track network path to - ipv6 Track network path to - -However, another tool, mtr_, is available which combines ping and traceroute -into a single tool. An example of its output is shown: - -.. code-block:: sh - - vyos@vyos:~$ mtr 10.62.212.12 - - My traceroute [v0.85] - vyos (0.0.0.0) - Keys: Help Display mode Restart statistics Order of fields quit - Packets Pings - Host Loss% Snt Last Avg Best Wrst StDev - 1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1 - 2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1 - 3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1 - 4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0 - -.. note:: The output of ``mtr`` consumes the screen and will replace your - command prompt. - -Several options are available for changing the display output. Press `h` to -invoke the built in help system. To quit, just press `q` and you'll be returned -to the VyOS command prompt. - -Monitoring ----------- - -Network Interfaces -^^^^^^^^^^^^^^^^^^ - -It's possible to monitor network traffic, either at the flow level or protocol -level. This can be useful when troubleshooting a variety of protocols and -configurations. The following interface types can be monitored: - -.. code-block:: sh - - vyos@vyos:~$ monitor interfaces - Possible completions: - Execute the current command - bonding Monitor a bonding interface - bridge Monitor a bridge interface - ethernet Monitor a ethernet interface - loopback Monitor a loopback interface - openvpn Monitor an openvpn interface - pppoe Monitor pppoe interface - pseudo-ethernet - Monitor a pseudo-ethernet interface - tunnel Monitor a tunnel interface - vrrp Monitor a vrrp interface - vti Monitor a vti interface - wireless Monitor wireless interface - -To monitor traffic flows, issue the :code:`monitor interfaces flow` -command, replacing `` and `` with your desired interface type and -name, respectively. Output looks like the following: - -.. code-block:: sh - - 12.5Kb 25.0Kb 37.5Kb 50.0Kb 62.5Kb - ???????????????????????????????????????????????????????????????????????????????????????????????????? - 10.11.111.255 => 10.11.110.37 0b 0b 0b - <= 624b 749b 749b - 10.11.110.29 => 10.62.200.11 0b 198b 198b - <= 0b 356b 356b - 255.255.255.255 => 10.11.110.47 0b 0b 0b - <= 724b 145b 145b - 10.11.111.255 => 10.11.110.47 0b 0b 0b - <= 724b 145b 145b - 10.11.111.255 => 10.11.110.255 0b 0b 0b - <= 680b 136b 136b - ???????????????????????????????????????????????????????????????????????????????????????????????????? - TX: cumm: 26.7KB peak: 40.6Kb rates: 23.2Kb 21.4Kb 21.4Kb - RX: 67.5KB 63.6Kb 54.6Kb 54.0Kb 54.0Kb - TOTAL: 94.2KB 104Kb 77.8Kb 75.4Kb 75.4Kb - -Several options are available for changing the display output. Press `h` to -invoke the built in help system. To quit, just press `q` and you'll be returned -to the VyOS command prompt. - -To monitor interface traffic, issue the :code:`monitor interfaces -traffic` command, replacing `` and `` with your desired interface -type and name, respectively. This command invokes the familiar tshark_ utility -and the following options are available: - -.. code-block:: sh - - vyos@vyos:~$ monitor interfaces ethernet eth0 traffic - Possible completions: - Execute the current command - detail Monitor detailed traffic for the specified ethernet interface - filter Monitor filtered traffic for the specified ethernet interface - save Save monitored traffic to a file - unlimited Monitor traffic for the specified ethernet interface - -To quit monitoring, press `Ctrl-c` and you'll be returned to the VyOS command -prompt. The `detail` keyword provides verbose output of the traffic seen on -the monitored interface. The `filter` keyword accepts valid `PCAP filter -expressions`_, enclosed in single or double quotes (e.g. "port 25" or "port 161 -and udp"). The `save` keyword allows you to save the traffic dump to a file. -The `unlimited` keyword is used to specify that an unlimited number of packets -can be captured (by default, 1,000 packets are captured and you're returned to -the VyOS command prompt). - -Interface Bandwith -^^^^^^^^^^^^^^^^^^ - -to take a quick view on the used bandwith of an interface use the ``monitor -bandwith`` command - -.. code-block:: sh - - vyos@vyos:~$ monitor bandwidth interface eth0 - -show the following: - -.. code-block:: sh - - B (RX Bytes/second) - 198.00 .|....|..................................................... - 165.00 .|....|..................................................... - 132.00 ||..|.|..................................................... - 99.00 ||..|.|..................................................... - 66.00 |||||||..................................................... - 33.00 |||||||..................................................... - 1 5 10 15 20 25 30 35 40 45 50 55 60 - - KiB (TX Bytes/second) - 3.67 ......|..................................................... - 3.06 ......|..................................................... - 2.45 ......|..................................................... - 1.84 ......|..................................................... - 1.22 ......|..................................................... - 0.61 :::::||..................................................... - 1 5 10 15 20 25 30 35 40 45 50 55 60 - -Interface performance -^^^^^^^^^^^^^^^^^^^^^ - -To take a look on the network bandwith between two nodes, the ``monitor -bandwidth-test`` command is used to run iperf. - -.. code-block:: sh - - vyos@vyos:~$ monitor bandwidth-test - Possible completions: - accept Wait for bandwidth test connections (port TCP/5001) - initiate Initiate a bandwidth test - -* The ``accept`` command open a listen iperf server on TCP Port 5001 -* The ``initiate`` command conncet to this server. - -.. code-block:: sh - - vyos@vyos:~$ monitor bandwidth-test initiate - Possible completions: - Initiate a bandwidth test to specified host (port TCP/5001) - - - - -Monitor command -^^^^^^^^^^^^^^^ - -The ``monitor command`` command allows you to repeatedly run a command to view -a continuously refreshed output. The command is run and output every 2 seconds, -allowing you to monitor the output continuously without having to re-run the -command. This can be useful to follow routing adjacency formation. - -.. code-block:: sh - - vyos@router:~$ monitor command "show interfaces" - -Will clear the screen and show you the output of ``show interfaces`` every -2 seconds. - -.. code-block:: sh - - Every 2.0s: /opt/vyatta/bin/vyatta-op-cmd-wrapper Sun Mar 26 02:49:46 2019 - - Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down - Interface IP Address S/L Description - --------- ---------- --- ----------- - eth0 192.168.1.1/24 u/u - eth0.5 198.51.100.4/24 u/u WAN - lo 127.0.0.1/8 u/u - ::1/128 - vti0 172.32.254.2/30 u/u - vti1 172.32.254.9/30 u/u - -Clear Command -------------- - -Sometimes you need to clear counters or statistics to troubleshoot better. - -To do this use the ``clear`` command in Operational mode. - -to clear the console output - -.. code-block:: sh - - vyos@vyos:~$ clear console - -to clear interface counters - -.. code-block:: sh - - # clear all interfaces - vyos@vyos:~$ clear interface ethernet counters - # clear specific interface - vyos@vyos:~$ clear interface ehternet eth0 counters - -The command follow the same logic as the ``set`` command in configuration mode. - -.. code-block:: sh - - # clear all counters of a interface type - vyos@vyos:~$ clear interface counters - # clear counter of a interface in interface_type - vyos@vyos:~$ clear interface counters - - -to clear counters on firewall rulesets or single rules - -.. code-block:: sh - - vyos@vyos:~$ clear firewall name counters - vyos@vyos:~$ clear firewall name rule counters - - vyos@vyos:~$ clear firewall ipv6-name counters - vyos@vyos:~$ clear firewall ipv6-name rule counters - - -Basic System Information ------------------------- - -Boot steps -^^^^^^^^^^ - -VyOS 1.2 uses `Debian Jessie`_ as the base Linux operating system. Jessie was -the first version of Debian that uses systemd_ as the default init system. - -These are the boot steps for VyOS 1.2 - -1. The BIOS loads Grub (or isolinux for the Live CD) -2. Grub then starts the Linux boot and loads the Linux Kernel ``/boot/vmlinuz`` -3. Kernel Launches Systemd ``/lib/systemd/systemd`` -4. Systemd loads the VyOS service file - ``/lib/systemd/system/vyos-router.service`` -5. The service file launches the VyOS router init script - ``/usr/libexec/vyos/init/vyos-router`` - this is part of the vyatta-cfg_ - Debian package - - 1. Starts FRR_ - successor to `GNU Zebra`_ and Quagga_ - - 2. Initialises the boot configuration file - copies over - ``config.boot.default`` if there is no configuration - 3. Runs the configuration migration, if the configuration is for an older - version of VyOS - 4. Runs The pre-config script, if there is one - ``/config/scripts/vyos-preconfig-bootup.script`` - 5. If the config file was upgraded, runs any post upgrade scripts - ``/config/scripts/post-upgrade.d`` - 6. Starts ``rl-system`` and ``firewall`` - 7. Mounts the ``/boot`` partition - 8. The boot configuration file is then applied by ``/opt/vyatta/sbin/ - vyatta-boot-config-loader/opt/vyatta/etc/config/config.boot`` - - 1. The config loader script writes log entries to - ``/var/log/vyatta-config-loader.log`` - - 10. Runs ``telinit q`` to tell the init system to reload ``/etc/inittab`` - 11. Finally it runs the post-config script - ``/config/scripts/vyos-postconfig-bootup.script`` - -.. _Quagga: http://www.quagga.net/ -.. _`GNU Zebra`: https://www.gnu.org/software/zebra/ -.. _FRR: https://frrouting.org/ -.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg -.. _systemd: https://freedesktop.org/wiki/Software/systemd/ -.. _`Debian Jessie`: https://www.debian.org/releases/jessie/ -.. _mtr: http://www.bitwizard.nl/mtr/ -.. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html -.. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html diff --git a/docs/index.rst b/docs/index.rst index 742b8eec..19bec74e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -39,6 +39,7 @@ Basics load-balancing image-mgmt commandscripting + troubleshooting ######## Appendix @@ -49,7 +50,6 @@ Appendix :maxdepth: 2 appendix/releasenotes - appendix/troubleshooting appendix/examples/index appendix/commandtree/index appendix/vyos-on-vmware diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst new file mode 100644 index 00000000..6448d031 --- /dev/null +++ b/docs/troubleshooting.rst @@ -0,0 +1,346 @@ +.. _troubleshooting: + +Troubleshooting +=============== + +Sometimes things break or don't work as expected. This section describes +several troubleshooting tools provided by VyOS that can help when something +goes wrong. + +Basic Connectivity Verification +------------------------------- + +Verifying connectivity can be done with the familiar `ping` and `traceroute` +commands. The options for each are shown (the options for each command were +displayed using the built-in help as described in the :ref:`cli` +section and are omitted from the output here): + +.. code-block:: sh + + vyos@vyos:~$ ping + Possible completions: + Send Internet Control Message Protocol (ICMP) echo request + + + +Several options are available when more extensive troubleshooting is needed: + +.. code-block:: sh + + vyos@vyos:~$ ping 8.8.8.8 + Possible completions: + Execute the current command + adaptive Ping options + allow-broadcast + audible + bypass-route + count + deadline + flood + interface + interval + mark + no-loopback + numeric + pattern + quiet + record-route + size + timestamp + tos + ttl + verbose + +.. code-block:: sh + + vyos@vyos:~$ traceroute + Possible completions: + Track network path to specified node + + + ipv4 Track network path to + ipv6 Track network path to + +However, another tool, mtr_, is available which combines ping and traceroute +into a single tool. An example of its output is shown: + +.. code-block:: sh + + vyos@vyos:~$ mtr 10.62.212.12 + + My traceroute [v0.85] + vyos (0.0.0.0) + Keys: Help Display mode Restart statistics Order of fields quit + Packets Pings + Host Loss% Snt Last Avg Best Wrst StDev + 1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1 + 2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1 + 3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1 + 4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0 + +.. note:: The output of ``mtr`` consumes the screen and will replace your + command prompt. + +Several options are available for changing the display output. Press `h` to +invoke the built in help system. To quit, just press `q` and you'll be returned +to the VyOS command prompt. + +Monitoring +---------- + +Network Interfaces +^^^^^^^^^^^^^^^^^^ + +It's possible to monitor network traffic, either at the flow level or protocol +level. This can be useful when troubleshooting a variety of protocols and +configurations. The following interface types can be monitored: + +.. code-block:: sh + + vyos@vyos:~$ monitor interfaces + Possible completions: + Execute the current command + bonding Monitor a bonding interface + bridge Monitor a bridge interface + ethernet Monitor a ethernet interface + loopback Monitor a loopback interface + openvpn Monitor an openvpn interface + pppoe Monitor pppoe interface + pseudo-ethernet + Monitor a pseudo-ethernet interface + tunnel Monitor a tunnel interface + vrrp Monitor a vrrp interface + vti Monitor a vti interface + wireless Monitor wireless interface + +To monitor traffic flows, issue the :code:`monitor interfaces flow` +command, replacing `` and `` with your desired interface type and +name, respectively. Output looks like the following: + +.. code-block:: sh + + 12.5Kb 25.0Kb 37.5Kb 50.0Kb 62.5Kb + ???????????????????????????????????????????????????????????????????????????????????????????????????? + 10.11.111.255 => 10.11.110.37 0b 0b 0b + <= 624b 749b 749b + 10.11.110.29 => 10.62.200.11 0b 198b 198b + <= 0b 356b 356b + 255.255.255.255 => 10.11.110.47 0b 0b 0b + <= 724b 145b 145b + 10.11.111.255 => 10.11.110.47 0b 0b 0b + <= 724b 145b 145b + 10.11.111.255 => 10.11.110.255 0b 0b 0b + <= 680b 136b 136b + ???????????????????????????????????????????????????????????????????????????????????????????????????? + TX: cumm: 26.7KB peak: 40.6Kb rates: 23.2Kb 21.4Kb 21.4Kb + RX: 67.5KB 63.6Kb 54.6Kb 54.0Kb 54.0Kb + TOTAL: 94.2KB 104Kb 77.8Kb 75.4Kb 75.4Kb + +Several options are available for changing the display output. Press `h` to +invoke the built in help system. To quit, just press `q` and you'll be returned +to the VyOS command prompt. + +To monitor interface traffic, issue the :code:`monitor interfaces +traffic` command, replacing `` and `` with your desired interface +type and name, respectively. This command invokes the familiar tshark_ utility +and the following options are available: + +.. code-block:: sh + + vyos@vyos:~$ monitor interfaces ethernet eth0 traffic + Possible completions: + Execute the current command + detail Monitor detailed traffic for the specified ethernet interface + filter Monitor filtered traffic for the specified ethernet interface + save Save monitored traffic to a file + unlimited Monitor traffic for the specified ethernet interface + +To quit monitoring, press `Ctrl-c` and you'll be returned to the VyOS command +prompt. The `detail` keyword provides verbose output of the traffic seen on +the monitored interface. The `filter` keyword accepts valid `PCAP filter +expressions`_, enclosed in single or double quotes (e.g. "port 25" or "port 161 +and udp"). The `save` keyword allows you to save the traffic dump to a file. +The `unlimited` keyword is used to specify that an unlimited number of packets +can be captured (by default, 1,000 packets are captured and you're returned to +the VyOS command prompt). + +Interface Bandwith +^^^^^^^^^^^^^^^^^^ + +to take a quick view on the used bandwith of an interface use the ``monitor +bandwith`` command + +.. code-block:: sh + + vyos@vyos:~$ monitor bandwidth interface eth0 + +show the following: + +.. code-block:: sh + + B (RX Bytes/second) + 198.00 .|....|..................................................... + 165.00 .|....|..................................................... + 132.00 ||..|.|..................................................... + 99.00 ||..|.|..................................................... + 66.00 |||||||..................................................... + 33.00 |||||||..................................................... + 1 5 10 15 20 25 30 35 40 45 50 55 60 + + KiB (TX Bytes/second) + 3.67 ......|..................................................... + 3.06 ......|..................................................... + 2.45 ......|..................................................... + 1.84 ......|..................................................... + 1.22 ......|..................................................... + 0.61 :::::||..................................................... + 1 5 10 15 20 25 30 35 40 45 50 55 60 + +Interface performance +^^^^^^^^^^^^^^^^^^^^^ + +To take a look on the network bandwith between two nodes, the ``monitor +bandwidth-test`` command is used to run iperf. + +.. code-block:: sh + + vyos@vyos:~$ monitor bandwidth-test + Possible completions: + accept Wait for bandwidth test connections (port TCP/5001) + initiate Initiate a bandwidth test + +* The ``accept`` command open a listen iperf server on TCP Port 5001 +* The ``initiate`` command conncet to this server. + +.. code-block:: sh + + vyos@vyos:~$ monitor bandwidth-test initiate + Possible completions: + Initiate a bandwidth test to specified host (port TCP/5001) + + + + +Monitor command +^^^^^^^^^^^^^^^ + +The ``monitor command`` command allows you to repeatedly run a command to view +a continuously refreshed output. The command is run and output every 2 seconds, +allowing you to monitor the output continuously without having to re-run the +command. This can be useful to follow routing adjacency formation. + +.. code-block:: sh + + vyos@router:~$ monitor command "show interfaces" + +Will clear the screen and show you the output of ``show interfaces`` every +2 seconds. + +.. code-block:: sh + + Every 2.0s: /opt/vyatta/bin/vyatta-op-cmd-wrapper Sun Mar 26 02:49:46 2019 + + Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down + Interface IP Address S/L Description + --------- ---------- --- ----------- + eth0 192.168.1.1/24 u/u + eth0.5 198.51.100.4/24 u/u WAN + lo 127.0.0.1/8 u/u + ::1/128 + vti0 172.32.254.2/30 u/u + vti1 172.32.254.9/30 u/u + +Clear Command +------------- + +Sometimes you need to clear counters or statistics to troubleshoot better. + +To do this use the ``clear`` command in Operational mode. + +to clear the console output + +.. code-block:: sh + + vyos@vyos:~$ clear console + +to clear interface counters + +.. code-block:: sh + + # clear all interfaces + vyos@vyos:~$ clear interface ethernet counters + # clear specific interface + vyos@vyos:~$ clear interface ehternet eth0 counters + +The command follow the same logic as the ``set`` command in configuration mode. + +.. code-block:: sh + + # clear all counters of a interface type + vyos@vyos:~$ clear interface counters + # clear counter of a interface in interface_type + vyos@vyos:~$ clear interface counters + + +to clear counters on firewall rulesets or single rules + +.. code-block:: sh + + vyos@vyos:~$ clear firewall name counters + vyos@vyos:~$ clear firewall name rule counters + + vyos@vyos:~$ clear firewall ipv6-name counters + vyos@vyos:~$ clear firewall ipv6-name rule counters + + +Basic System Information +------------------------ + +Boot steps +^^^^^^^^^^ + +VyOS 1.2 uses `Debian Jessie`_ as the base Linux operating system. Jessie was +the first version of Debian that uses systemd_ as the default init system. + +These are the boot steps for VyOS 1.2 + +1. The BIOS loads Grub (or isolinux for the Live CD) +2. Grub then starts the Linux boot and loads the Linux Kernel ``/boot/vmlinuz`` +3. Kernel Launches Systemd ``/lib/systemd/systemd`` +4. Systemd loads the VyOS service file + ``/lib/systemd/system/vyos-router.service`` +5. The service file launches the VyOS router init script + ``/usr/libexec/vyos/init/vyos-router`` - this is part of the vyatta-cfg_ + Debian package + + 1. Starts FRR_ - successor to `GNU Zebra`_ and Quagga_ + + 2. Initialises the boot configuration file - copies over + ``config.boot.default`` if there is no configuration + 3. Runs the configuration migration, if the configuration is for an older + version of VyOS + 4. Runs The pre-config script, if there is one + ``/config/scripts/vyos-preconfig-bootup.script`` + 5. If the config file was upgraded, runs any post upgrade scripts + ``/config/scripts/post-upgrade.d`` + 6. Starts ``rl-system`` and ``firewall`` + 7. Mounts the ``/boot`` partition + 8. The boot configuration file is then applied by ``/opt/vyatta/sbin/ + vyatta-boot-config-loader/opt/vyatta/etc/config/config.boot`` + + 1. The config loader script writes log entries to + ``/var/log/vyatta-config-loader.log`` + + 10. Runs ``telinit q`` to tell the init system to reload ``/etc/inittab`` + 11. Finally it runs the post-config script + ``/config/scripts/vyos-postconfig-bootup.script`` + +.. _Quagga: http://www.quagga.net/ +.. _`GNU Zebra`: https://www.gnu.org/software/zebra/ +.. _FRR: https://frrouting.org/ +.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg +.. _systemd: https://freedesktop.org/wiki/Software/systemd/ +.. _`Debian Jessie`: https://www.debian.org/releases/jessie/ +.. _mtr: http://www.bitwizard.nl/mtr/ +.. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html +.. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html -- cgit v1.2.3