1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
|
.. _troubleshooting:
Troubleshooting
===============
Sometimes things break or don't work as expected. This section describes
several troubleshooting tools provided by VyOS that can help when something
goes wrong.
Basic Connectivity Verification
-------------------------------
Verifying connectivity can be done with the familiar `ping` and `traceroute`
commands. The options for each are shown (the options for each command were
displayed using the built-in help as described in the :ref:`cli`
section and are omitted from the output here):
.. code-block:: none
vyos@vyos:~$ ping
Possible completions:
<hostname> Send Internet Control Message Protocol (ICMP) echo request
<x.x.x.x>
<h:h:h:h:h:h:h:h>
Several options are available when more extensive troubleshooting is needed:
.. code-block:: none
vyos@vyos:~$ ping 8.8.8.8
Possible completions:
<Enter> Execute the current command
adaptive Ping options
allow-broadcast
audible
bypass-route
count
deadline
flood
interface
interval
mark
no-loopback
numeric
pattern
quiet
record-route
size
timestamp
tos
ttl
verbose
.. code-block:: none
vyos@vyos:~$ traceroute
Possible completions:
<hostname> Track network path to specified node
<x.x.x.x>
<h:h:h:h:h:h:h:h>
ipv4 Track network path to <hostname|IPv4 address>
ipv6 Track network path to <hostname|IPv6 address>
However, another tool, mtr_, is available which combines ping and traceroute
into a single tool. An example of its output is shown:
.. code-block:: none
vyos@vyos:~$ mtr 10.62.212.12
My traceroute [v0.85]
vyos (0.0.0.0)
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1
2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1
3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1
4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0
.. note:: The output of ``mtr`` consumes the screen and will replace your
command prompt.
Several options are available for changing the display output. Press `h` to
invoke the built in help system. To quit, just press `q` and you'll be returned
to the VyOS command prompt.
Monitoring
----------
Network Interfaces
^^^^^^^^^^^^^^^^^^
It's possible to monitor network traffic, either at the flow level or protocol
level. This can be useful when troubleshooting a variety of protocols and
configurations. The following interface types can be monitored:
.. code-block:: none
vyos@vyos:~$ monitor interfaces
Possible completions:
<Enter> Execute the current command
bonding Monitor a bonding interface
bridge Monitor a bridge interface
ethernet Monitor a ethernet interface
loopback Monitor a loopback interface
openvpn Monitor an openvpn interface
pppoe Monitor pppoe interface
pseudo-ethernet
Monitor a pseudo-ethernet interface
tunnel Monitor a tunnel interface
vrrp Monitor a vrrp interface
vti Monitor a vti interface
wireless Monitor wireless interface
To monitor traffic flows, issue the :code:`monitor interfaces <type> <name> flow`
command, replacing `<type>` and `<name>` with your desired interface type and
name, respectively. Output looks like the following:
.. code-block:: none
12.5Kb 25.0Kb 37.5Kb 50.0Kb 62.5Kb
????????????????????????????????????????????????????????????????????????????????????????????????????
10.11.111.255 => 10.11.110.37 0b 0b 0b
<= 624b 749b 749b
10.11.110.29 => 10.62.200.11 0b 198b 198b
<= 0b 356b 356b
255.255.255.255 => 10.11.110.47 0b 0b 0b
<= 724b 145b 145b
10.11.111.255 => 10.11.110.47 0b 0b 0b
<= 724b 145b 145b
10.11.111.255 => 10.11.110.255 0b 0b 0b
<= 680b 136b 136b
????????????????????????????????????????????????????????????????????????????????????????????????????
TX: cumm: 26.7KB peak: 40.6Kb rates: 23.2Kb 21.4Kb 21.4Kb
RX: 67.5KB 63.6Kb 54.6Kb 54.0Kb 54.0Kb
TOTAL: 94.2KB 104Kb 77.8Kb 75.4Kb 75.4Kb
Several options are available for changing the display output. Press `h` to
invoke the built in help system. To quit, just press `q` and you'll be returned
to the VyOS command prompt.
To monitor interface traffic, issue the :code:`monitor interfaces <type> <name>
traffic` command, replacing `<type>` and `<name>` with your desired interface
type and name, respectively. This command invokes the familiar tshark_ utility
and the following options are available:
.. code-block:: none
vyos@vyos:~$ monitor interfaces ethernet eth0 traffic
Possible completions:
<Enter> Execute the current command
detail Monitor detailed traffic for the specified ethernet interface
filter Monitor filtered traffic for the specified ethernet interface
save Save monitored traffic to a file
unlimited Monitor traffic for the specified ethernet interface
To quit monitoring, press `Ctrl-c` and you'll be returned to the VyOS command
prompt. The `detail` keyword provides verbose output of the traffic seen on
the monitored interface. The `filter` keyword accepts valid `PCAP filter
expressions`_, enclosed in single or double quotes (e.g. "port 25" or "port 161
and udp"). The `save` keyword allows you to save the traffic dump to a file.
The `unlimited` keyword is used to specify that an unlimited number of packets
can be captured (by default, 1,000 packets are captured and you're returned to
the VyOS command prompt).
Interface Bandwith
^^^^^^^^^^^^^^^^^^
to take a quick view on the used bandwith of an interface use the ``monitor bandwith`` command
.. code-block:: none
vyos@vyos:~$ monitor bandwidth interface eth0
show the following:
.. code-block:: none
eth0 bmon 3.5
Interfaces │ RX bps pps %│ TX bps pps %
>eth0 │ 141B 2 │ 272B 1
───────────────────────────────┴───────────────────────┴────────────────────────────────────────────────────────────────
B (RX Bytes/second)
198.00 .|....|.....................................................
165.00 .|....|.....................................................
132.00 ||..|.|.....................................................
99.00 ||..|.|.....................................................
66.00 |||||||.....................................................
33.00 |||||||.....................................................
1 5 10 15 20 25 30 35 40 45 50 55 60
KiB (TX Bytes/second)
3.67 ......|.....................................................
3.06 ......|.....................................................
2.45 ......|.....................................................
1.84 ......|.....................................................
1.22 ......|.....................................................
0.61 :::::||.....................................................
1 5 10 15 20 25 30 35 40 45 50 55 60
───────────────────────────────────────── Press d to enable detailed statistics ────────────────────────────────────────
─────────────────────────────────────── Press i to enable additional information ───────────────────────────────────────
Wed Apr 3 14:46:59 2019 Press ? for help
| Press ``d`` for more detailed informations or ``i`` for additional information.
| To exit press ``q`` and than ``y``
Interface performance
^^^^^^^^^^^^^^^^^^^^^
To take a look on the network bandwith between two nodes, the ``monitor bandwidth-test`` command is used to run iperf.
.. code-block:: none
vyos@vyos:~$ monitor bandwidth-test
Possible completions:
accept Wait for bandwidth test connections (port TCP/5001)
initiate Initiate a bandwidth test
| The ``accept`` command open a listen iperf server on TCP Port 5001
| The ``initiate`` command conncet to this server.
.. code-block:: none
vyos@vyos:~$ monitor bandwidth-test initiate
Possible completions:
<hostname> Initiate a bandwidth test to specified host (port TCP/5001)
<x.x.x.x>
<h:h:h:h:h:h:h:h>
Monitor command
^^^^^^^^^^^^^^^
The ``monitor command`` command allows you to repeatedly run a command to view a continuously refreshed output.
The command is run and output every 2 seconds, allowing you to monitor the output continuously without having to re-run the command. This can be useful to follow routing adjacency formation.
.. code-block:: none
vyos@router:~$ monitor command "show interfaces"
Will clear the screen and show you the output of ``show interfaces`` every 2 seconds.
.. code-block:: none
Every 2.0s: /opt/vyatta/bin/vyatta-op-cmd-wrapper s... Sun Mar 26 02:49:46 2019
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 192.168.1.1/24 u/u
eth0.5 198.51.100.4/24 u/u WAN
lo 127.0.0.1/8 u/u
::1/128
vti0 172.32.254.2/30 u/u
vti1 172.32.254.9/30 u/u
Clear Command
-------------
Sometimes you need to clear counters or statistics to troubleshoot better.
To do this use the ``clear`` command in Operational mode.
to clear the console output
.. code-block:: none
vyos@vyos:~$ clear console
to clear interface counters
.. code-block:: none
# clear all interfaces
vyos@vyos:~$ clear interface ethernet counters
# clear specific interface
vyos@vyos:~$ clear interface ehternet eth0 counters
The command follow the same logic as the ``set`` command in configuration mode.
.. code-block:: none
# clear all counters of a interface type
vyos@vyos:~$ clear interface <interface_type> counters
# clear counter of a interface in interface_type
vyos@vyos:~$ clear interface <interface_type> <interace_name> counters
to clear counters on firewall rulesets or single rules
.. code-block:: none
vyos@vyos:~$ clear firewall name <ipv4 ruleset name> counters
vyos@vyos:~$ clear firewall name <ipv4 ruleset name> rule <rule#> counters
vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> counters
vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> rule <rule#> counters
Basic System Information
------------------------
Boot steps
^^^^^^^^^^
VyOS 1.2.0+ uses `Debian Jessie`_ as the base Linux operating system.
Jessie was the first version of Debian that uses `systemd`_ as the default init system.
These are the boot steps for VyOS 1.2.0+
1. The BIOS loads Grub (or isolinux for the Live CD)
2. Grub then starts the Linux boot and loads the Linux Kernel ``/boot/vmlinuz``
3. Kernel Launches Systemd ``/lib/systemd/systemd``
4. Systemd loads the VyOS service file ``/lib/systemd/system/vyos-router.service``
5. The service file launches the VyOS router init script ``/usr/libexec/vyos/init/vyos-router`` - this is part of the `vyatta-cfg`_ Debian package
1. Starts FRR_ - successor to `GNU Zebra`_ and `Quagga`_
2. Initialises the boot configuration file - copies over ``config.boot.default`` if there is no configuration
3. Runs the configuration migration, if the configuration is for an older version of VyOS
4. Runs The pre-config script, if there is one ``/config/scripts/vyos-preconfig-bootup.script``
5. If the config file was upgraded, runs any post upgrade scripts ``/config/scripts/post-upgrade.d``
6. Starts **rl-system** and **firewall**
7. Mounts the ``/boot`` partition
8. The boot configuration file is then applied by ``/opt/vyatta/sbin/vyatta-boot-config-loader /opt/vyatta/etc/config/config.boot``
1. The config loader script writes log entries to ``/var/log/vyatta-config-loader.log``
10. Runs ``telinit q`` to tell the init system to reload ``/etc/inittab``
11. Finally it runs the post-config script ``/config/scripts/vyos-postconfig-bootup.script``
.. _Quagga: http://www.quagga.net/
.. _`GNU Zebra`: https://www.gnu.org/software/zebra/
.. _FRR: https://frrouting.org/
.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg
.. _systemd: _https://freedesktop.org/wiki/Software/systemd/
.. _`Debian Jessie`: https://www.debian.org/releases/jessie/
.. _mtr: http://www.bitwizard.nl/mtr/
.. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html
.. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html
|
