.. _troubleshooting: Troubleshooting =============== Sometimes things break or don't work as expected. This section describes several troubleshooting tools provided by VyOS that can help when something goes wrong. Basic Connectivity Verification ------------------------------- Verifying connectivity can be done with the familiar `ping` and `traceroute` commands. The options for each are shown (the options for each command were displayed using the built-in help as described in the :ref:`cli` section and are omitted from the output here): .. code-block:: none vyos@vyos:~$ ping Possible completions: Send Internet Control Message Protocol (ICMP) echo request Several options are available when more extensive troubleshooting is needed: .. code-block:: none vyos@vyos:~$ ping 8.8.8.8 Possible completions: Execute the current command adaptive Ping options allow-broadcast audible bypass-route count deadline flood interface interval mark no-loopback numeric pattern quiet record-route size timestamp tos ttl verbose .. code-block:: none vyos@vyos:~$ traceroute Possible completions: Track network path to specified node ipv4 Track network path to ipv6 Track network path to However, another tool, mtr_, is available which combines ping and traceroute into a single tool. An example of its output is shown: .. code-block:: none vyos@vyos:~$ mtr 10.62.212.12 My traceroute [v0.85] vyos (0.0.0.0) Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1 2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1 3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1 4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0 .. note:: The output of ``mtr`` consumes the screen and will replace your command prompt. Several options are available for changing the display output. Press `h` to invoke the built in help system. To quit, just press `q` and you'll be returned to the VyOS command prompt. Interface names --------------- If you find the names of your interfaces have changed, this could be because your MAC addresses have changed. * For example, you have a VyOS VM with 4 Ethernet interfaces named eth0, eth1, eth2 and eth3. Then, you migrate your VyOS VM to a different host and find your interfaces now are eth4, eth5, eth6 and eth7. One way to fix this issue **taking control of the MAC addresses** is: Log into VyOS and run this command to display your interface settings. .. code-block:: none show interfaces detail Take note of MAC addresses. Now, in order to update a MAC address in the configuration, run this command specifying the interface name and MAC address you want. .. code-block:: none set interfaces eth0 hw-id 00:0c:29:da:a4:fe If it is a VM, go into the settings of the host and set the MAC address to the settings found in the config.boot file. You can also set the MAC to static if the host allows so. * Another example could be when cloning VyOS VMs in GNS3 and you get into the same issue: interface names have changed. And **a more generic way to fix it** is just deleting every MAC address at the configuration file of the cloned machine. They will be correctly regenerated automatically. Monitoring ---------- Network Interfaces ^^^^^^^^^^^^^^^^^^ It's possible to monitor network traffic, either at the flow level or protocol level. This can be useful when troubleshooting a variety of protocols and configurations. The following interface types can be monitored: .. code-block:: none vyos@vyos:~$ monitor interfaces Possible completions: Execute the current command bonding Monitor a bonding interface bridge Monitor a bridge interface ethernet Monitor a ethernet interface loopback Monitor a loopback interface openvpn Monitor an openvpn interface pppoe Monitor pppoe interface pseudo-ethernet Monitor a pseudo-ethernet interface tunnel Monitor a tunnel interface vrrp Monitor a vrrp interface vti Monitor a vti interface wireless Monitor wireless interface To monitor traffic flows, issue the :code:`monitor interfaces flow` command, replacing `` and `` with your desired interface type and name, respectively. Output looks like the following: .. code-block:: none 12.5Kb 25.0Kb 37.5Kb 50.0Kb 62.5Kb ???????????????????????????????????????????????????????????????????????????????????????????????????? 10.11.111.255 => 10.11.110.37 0b 0b 0b <= 624b 749b 749b 10.11.110.29 => 10.62.200.11 0b 198b 198b <= 0b 356b 356b 255.255.255.255 => 10.11.110.47 0b 0b 0b <= 724b 145b 145b 10.11.111.255 => 10.11.110.47 0b 0b 0b <= 724b 145b 145b 10.11.111.255 => 10.11.110.255 0b 0b 0b <= 680b 136b 136b ???????????????????????????????????????????????????????????????????????????????????????????????????? TX: cumm: 26.7KB peak: 40.6Kb rates: 23.2Kb 21.4Kb 21.4Kb RX: 67.5KB 63.6Kb 54.6Kb 54.0Kb 54.0Kb TOTAL: 94.2KB 104Kb 77.8Kb 75.4Kb 75.4Kb Several options are available for changing the display output. Press `h` to invoke the built in help system. To quit, just press `q` and you'll be returned to the VyOS command prompt. To monitor interface traffic, issue the :code:`monitor interfaces traffic` command, replacing `` and `` with your desired interface type and name, respectively. This command invokes the familiar tshark_ utility and the following options are available: .. code-block:: none vyos@vyos:~$ monitor interfaces ethernet eth0 traffic Possible completions: Execute the current command detail Monitor detailed traffic for the specified ethernet interface filter Monitor filtered traffic for the specified ethernet interface save Save monitored traffic to a file unlimited Monitor traffic for the specified ethernet interface To quit monitoring, press `Ctrl-c` and you'll be returned to the VyOS command prompt. The `detail` keyword provides verbose output of the traffic seen on the monitored interface. The `filter` keyword accepts valid `PCAP filter expressions`_, enclosed in single or double quotes (e.g. "port 25" or "port 161 and udp"). The `save` keyword allows you to save the traffic dump to a file. The `unlimited` keyword is used to specify that an unlimited number of packets can be captured (by default, 1,000 packets are captured and you're returned to the VyOS command prompt). Interface Bandwith ^^^^^^^^^^^^^^^^^^ to take a quick view on the used bandwith of an interface use the ``monitor bandwith`` command .. code-block:: none vyos@vyos:~$ monitor bandwidth interface eth0 show the following: .. code-block:: none eth0 bmon 3.5 Interfaces │ RX bps pps %│ TX bps pps % >eth0 │ 141B 2 │ 272B 1 ───────────────────────────────┴───────────────────────┴──────────────────────────────────────────────────────────────── B (RX Bytes/second) 198.00 .|....|..................................................... 165.00 .|....|..................................................... 132.00 ||..|.|..................................................... 99.00 ||..|.|..................................................... 66.00 |||||||..................................................... 33.00 |||||||..................................................... 1 5 10 15 20 25 30 35 40 45 50 55 60 KiB (TX Bytes/second) 3.67 ......|..................................................... 3.06 ......|..................................................... 2.45 ......|..................................................... 1.84 ......|..................................................... 1.22 ......|..................................................... 0.61 :::::||..................................................... 1 5 10 15 20 25 30 35 40 45 50 55 60 ───────────────────────────────────────── Press d to enable detailed statistics ──────────────────────────────────────── ─────────────────────────────────────── Press i to enable additional information ─────────────────────────────────────── Wed Apr 3 14:46:59 2019 Press ? for help | Press ``d`` for more detailed informations or ``i`` for additional information. | To exit press ``q`` and than ``y`` Interface performance ^^^^^^^^^^^^^^^^^^^^^ To take a look on the network bandwith between two nodes, the ``monitor bandwidth-test`` command is used to run iperf. .. code-block:: none vyos@vyos:~$ monitor bandwidth-test Possible completions: accept Wait for bandwidth test connections (port TCP/5001) initiate Initiate a bandwidth test | The ``accept`` command open a listen iperf server on TCP Port 5001 | The ``initiate`` command conncet to this server. .. code-block:: none vyos@vyos:~$ monitor bandwidth-test initiate Possible completions: Initiate a bandwidth test to specified host (port TCP/5001) Monitor command ^^^^^^^^^^^^^^^ The ``monitor command`` command allows you to repeatedly run a command to view a continuously refreshed output. The command is run and output every 2 seconds, allowing you to monitor the output continuously without having to re-run the command. This can be useful to follow routing adjacency formation. .. code-block:: none vyos@router:~$ monitor command "show interfaces" Will clear the screen and show you the output of ``show interfaces`` every 2 seconds. .. code-block:: none Every 2.0s: /opt/vyatta/bin/vyatta-op-cmd-wrapper s... Sun Mar 26 02:49:46 2019 Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down Interface IP Address S/L Description --------- ---------- --- ----------- eth0 192.168.1.1/24 u/u eth0.5 198.51.100.4/24 u/u WAN lo 127.0.0.1/8 u/u ::1/128 vti0 172.32.254.2/30 u/u vti1 172.32.254.9/30 u/u Clear Command ------------- Sometimes you need to clear counters or statistics to troubleshoot better. To do this use the ``clear`` command in Operational mode. to clear the console output .. code-block:: none vyos@vyos:~$ clear console to clear interface counters .. code-block:: none # clear all interfaces vyos@vyos:~$ clear interface ethernet counters # clear specific interface vyos@vyos:~$ clear interface ehternet eth0 counters The command follow the same logic as the ``set`` command in configuration mode. .. code-block:: none # clear all counters of a interface type vyos@vyos:~$ clear interface counters # clear counter of a interface in interface_type vyos@vyos:~$ clear interface counters to clear counters on firewall rulesets or single rules .. code-block:: none vyos@vyos:~$ clear firewall name counters vyos@vyos:~$ clear firewall name rule counters vyos@vyos:~$ clear firewall ipv6-name counters vyos@vyos:~$ clear firewall ipv6-name rule counters Basic System Information ------------------------ .. _boot-steps: Boot steps ^^^^^^^^^^ VyOS 1.2.0+ uses `Debian Jessie`_ as the base Linux operating system. Jessie was the first version of Debian that uses `systemd`_ as the default init system. These are the boot steps for VyOS 1.2.0+ 1. The BIOS loads Grub (or isolinux for the Live CD) 2. Grub then starts the Linux boot and loads the Linux Kernel ``/boot/vmlinuz`` 3. Kernel Launches Systemd ``/lib/systemd/systemd`` 4. Systemd loads the VyOS service file ``/lib/systemd/system/vyos-router.service`` 5. The service file launches the VyOS router init script ``/usr/libexec/vyos/init/vyos-router`` - this is part of the `vyatta-cfg`_ Debian package 1. Starts FRR_ - successor to `GNU Zebra`_ and `Quagga`_ 2. Initialises the boot configuration file - copies over ``config.boot.default`` if there is no configuration 3. Runs the configuration migration, if the configuration is for an older version of VyOS 4. Runs The pre-config script, if there is one ``/config/scripts/vyos-preconfig-bootup.script`` 5. If the config file was upgraded, runs any post upgrade scripts ``/config/scripts/post-upgrade.d`` 6. Starts **rl-system** and **firewall** 7. Mounts the ``/boot`` partition 8. The boot configuration file is then applied by ``/opt/vyatta/sbin/vyatta-boot-config-loader /opt/vyatta/etc/config/config.boot`` 1. The config loader script writes log entries to ``/var/log/vyatta-config-loader.log`` 10. Runs ``telinit q`` to tell the init system to reload ``/etc/inittab`` 11. Finally it runs the post-config script ``/config/scripts/vyos-postconfig-bootup.script`` .. _Quagga: http://www.quagga.net/ .. _`GNU Zebra`: https://www.gnu.org/software/zebra/ .. _FRR: https://frrouting.org/ .. _vyatta-cfg: https://github.com/vyos/vyatta-cfg .. _systemd: _https://freedesktop.org/wiki/Software/systemd/ .. _`Debian Jessie`: https://www.debian.org/releases/jessie/ .. _mtr: http://www.bitwizard.nl/mtr/ .. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html .. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html