1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
|
.. _debugging:
#########
Debugging
#########
There are two flags available to aid in debugging configuration scripts.
Since configuration loading issues will manifest during boot, the flags are
passed as kernel boot parameters.
ISO image build
===============
When having trouble compiling your own ISO image or debugging Jenkins issues
you can follow the steps at :ref:`iso_build_issues`.
System Startup
==============
The system startup can be debugged (like loading in the configuration
file from ``/config/config.boot``. This can be achieve by extending the
Kernel command-line in the bootloader.
Kernel
------
* ``vyos-debug`` - Adding the parameter to the linux boot line will produce
timing results for the execution of scripts during commit. If one is seeing
an unexpected delay during manual or boot commit, this may be useful in
identifying bottlenecks. The internal flag is ``VYOS_DEBUG``, and is found
in vyatta-cfg_. Output is directed to ``/var/log/vyatta/cfg-stdout.log``.
* ``vyos-config-debug`` - During development, coding errors can lead to a
commit failure on boot, possibly resulting in a failed initialization of the
CLI. In this circumstance, the kernel boot parameter ``vyos-config-debug``
will ensure access to the system as user ``vyos``, and will log a Python
stack trace to the file ``/tmp/boot-config-trace``.
File ``boot-config-trace`` will generate only if config loaded with a failure
status.
Live System
===========
A number of flags can be set up to change the behaviour of VyOS at runtime.
These flags can be toggled using either environment variables or creating
files.
For each feature, a file called ``vyos.feature.debug`` can be created to
toggle the feature on. If a parameter is required it can be placed inside
the file as its first line.
The file can be placed in ``/tmp`` for one time debugging (as the file
will be removed on reboot) or placed in '/config' to stay permanently.
For example, ``/tmp/vyos.ifconfig.debug`` can be created to enable
interface debugging.
It is also possible to set up the debugging using environment variables.
In that case, the name will be (in uppercase) VYOS_FEATURE_DEBUG.
For example running, ``export VYOS_IFCONFIG_DEBUG=""`` on your vbash,
will have the same effect as ``touch /tmp/vyos.ifconfig.debug``.
* ``ifconfig`` - Once set, all commands used, and their responses received
from the OS, will be presented on the screen for inspection.
* ``command`` - Once set, all commands used, and their responses received
from the OS, will be presented on the screen for inspection.
* ``developer`` - Should a command fail, instead of printing a message to the
user explaining how to report issues, the python interpreter will start a
PBD post-mortem session to allow the developer to debug the issue. As the
debugger will wait from input from the developer, it has the capacity to
prevent a router to boot and therefore should only be permanently set up
on production if you are ready to see the OS fail to boot.
* ``log`` - In some rare cases, it may be useful to see what the OS is doing,
including during boot. This option sends all commands used by VyOS to a
file. The default file is ``/tmp/full-log`` but it can be changed.
.. note:: In order to retrieve the debug output on the command-line you need to
disable ``vyos-configd`` in addition. This can be run either one-time by
calling ``sudo systemctl stop vyos-configd`` or make this reboot-safe by
calling ``sudo systemctl disable vyos-configd``.
FRR
---
Recent versions use the ``vyos.frr`` framework. The Python class is located
inside our ``vyos-1x:python/vyos/frr.py``. It comes with an embedded debugging/
(print style) debugger as vyos.ifconfig does.
To enable debugging just run: ``$ touch /tmp/vyos.frr.debug``
Debugging Python Code with PDB
------------------------------
Sometimes it might be useful to debug Python code interactively on the live
system rather than a IDE. This can be achieved using pdb.
Let us assume you want to debug a Python script that is called by an op-mode
command. After you found the script by looking up the op-mode-defitions you
can edit the script in the live system using e.g. vi:
``vi /usr/libexec/vyos/op_mode/show_xyz.py``
Insert the following statement right before the section where you want to
investigate a problem (e.g. a statement you see in a backtrace):
``import pdb; pdb.set_trace()``
Optionally you can surrounded this statement by an ``if`` which only triggers
under the condition you are interested in.
Once you run ``show xyz`` and your condition is triggered you should be dropped
into the python debugger:
.. code-block:: none
> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process()
-> rule_type = rule.get('type', '')
(Pdb)
You can type ``help`` to get an overview of the available commands, and
``help command`` to get more information on each command.
Useful commands are:
* examine variables using ``pp(var)``
* continue execution using ``cont``
* get a backtrace using ``bt``
Config Migration Scripts
------------------------
When writing a new configuration migrator it may happen that you see an error
when you try to invoke it manually on a development system. This error will
look like:
.. code-block:: none
vyos@vyos:~$ /opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1 /tmp/config.boot
Traceback (most recent call last):
File "/opt/vyatta/etc/config-migrate/migrate/ssh/0-to-1", line 31, in <module>
config = ConfigTree(config_file)
File "/usr/lib/python3/dist-packages/vyos/configtree.py", line 134, in __init__
raise ValueError("Failed to parse config: {0}".format(msg))
ValueError: Failed to parse config: Syntax error on line 240, character 1: Invalid syntax.
The reason is that the configuration migration backend is rewritten and uses
a new form of "magic string" which is applied on demand when real config
migration is run on boot. When running individual migrators for testing,
you need to convert the "magic string" on your own by:
.. code-block:: none
vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --virtual --set-vintage vyos /tmp/config.boot
Configuration Error on System Boot
----------------------------------
Being brave and running the latest rolling releases will sometimes trigger
bugs due to corner cases we missed in our design. Those bugs should be filed
via Phabricator_ but you can help us to narrow down the issue. Login to your
VyOS system and change into configuration mode by typing ``configure``. Now
re-load your boot configuration by simply typing ``load`` followed by return.
You should now see a Python backtrace which will help us to handle the issue,
please attach it to the Phabricator_ task.
Boot Timing
-----------
During the migration and extensive rewrite of functionality from Perl into
Python a significant increase in the overall system boottime was noticed. The
system boot time can be analysed and a graph can be generated in the end which
shows in detail who called whom during the system startup phase.
This is done by utilizing the ``systemd-bootchart`` package which is now
installed by default on the VyOS 1.3 (equuleus) branch. The configuration is
also versioned so we get comparable results. ``systemd-bootchart`` is configured
using this file: bootchart.conf_
To enable boot time graphing change the Kernel commandline and add the following
string: ``init=/usr/lib/systemd/systemd-bootchart``
This can also be done permanently by changing ``/boot/grub/grub.cfg``.
Priorities
==========
VyOS CLI is all about priorities. Every CLI node has a corresponding
``node.def`` file and possibly an attached script that is executed when the
node is present. Nodes can have a priority, and on system bootup - or any
other ``commit`` to the config all scripts are executed from lowest to highest
priority. This is good as this gives a deterministic behavior.
To debug issues in priorities or to see what's going on in the background
you can use the ``/opt/vyatta/sbin/priority.pl`` script which lists to you
the execution order of the scripts.
.. stop_vyoslinter
.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg
.. _bootchart.conf: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf
.. include:: /_include/common-references.txt
.. start_vyoslinter
|
