diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/contributing/md-development.md | 543 | ||||
| -rw-r--r-- | docs/operation/md-upgrade-recovery.md | 63 | ||||
| -rw-r--r-- | docs/vpp/configuration/dataplane/md-buffers.md | 90 | ||||
| -rw-r--r-- | docs/vpp/configuration/dataplane/md-unix.md | 54 |
4 files changed, 750 insertions, 0 deletions
diff --git a/docs/contributing/md-development.md b/docs/contributing/md-development.md new file mode 100644 index 00000000..4decbea3 --- /dev/null +++ b/docs/contributing/md-development.md @@ -0,0 +1,543 @@ +--- +lastproofread: '2025-12-12' +--- + +(development)= + +# Development + +Learn how to contribute to VyOS. + +(architecture-overview)= + +## Architecture overview + +VyOS source code is hosted on GitHub in the VyOS organization: +<https://github.com/vyos> + +VyOS is composed of multiple modules spread across different +repositories. Some modules contain forks of upstream +packages and are periodically synced. +VyOS consolidates most packages into the +[vyos-1x](https://github.com/vyos/vyos-1x) +repository while maintaining a consistent structure. +The base code is being rewritten +from Perl and Bash to Python using an XML-based CLI interface definition. + +VyOS ISO build scripts are hosted in the +[vyos-build](https://github.com/vyos/vyos-build) repository. See the +`vyos-build` repository +[README.md file](https://github.com/vyos/vyos-build/blob/current/README.md) +for more information on building VyOS ISO images. + +## Contributing code + +:::{warning} +You must sign the {doc}`Contributor License Agreement<cla>` +for your contributions to be accepted. +::: + +VyOS is open-source and welcomes patches. +All submissions must adhere to these guidelines: + +- Each commit addresses a single issue or feature. +- Each commit message references a [Phabricator](https://vyos.dev/) task ID + (for example, `T1234`). +- Each commit is associated with a username and email address + to identify the author (see [Configure your Git identity](configure-your-git-identity)). +- Only submit bugfixes in packages other than <https://github.com/vyos/vyos-1x>. +- Commits follow the [coding guidelines](coding-guidelines) outlined below. + +### Determining package ownership + +To determine which VyOS package contains a file you want to modify, use Debian's +`dpkg -S` command on your running VyOS installation. + +### Submitting your code + +Fork the repository and submit a GitHub pull request. This is the preferred way +to contribute changes to VyOS. + +To fork a VyOS repository: + +1. Append `/fork` to the repository URL on GitHub. For example, to fork + `vyos-1x`, use: <https://github.com/vyos/vyos-1x/fork> + +2. Clone your fork or add it as a remote to your local repository: + + - Clone: `git clone https://github.com/<user>/vyos-1x.git` + - Add remote: `git remote add myfork https://github.com/<user>/vyos-1x.git` + +(configure-your-git-identity)= + +3. Configure your Git identity: + + ```none + git config --global user.name "J. Random Hacker" + git config --global user.email "jrhacker@example.net" + ``` + +4. Make your changes and add files to the Git index: + + - Single file: `git add myfile` + - Directory: `git add somedir/*` + +5. Commit your changes with a meaningful headline and [Phabricator](https://vyos.dev/) reference: + + `git commit` + +6. Push to your fork and create a GitHub pull request: + + `git push` + +Alternatively, you can export commits as patches and send them to +<mailto:maintainers@vyos.net> or attach them directly to the [Phabricator](https://vyos.dev/) task: + +- Export last commit: `git format-patch` +- Export last two commits: `git format-patch -2` + +## Commit messages + +For guidance on writing commit messages, review the file history +with `git log path/to/file.txt`. + +Every change must be associated with a task number (prefixed with **T**) and +a component. If no bug report or feature request exists for your changes, +create a [Phabricator](https://vyos.dev/) task first. Reference the task ID in your commit message: + +- `ddclient: T1030: auto create runtime directories` +- `Jenkins: add current Git commit ID to build description` + +If your pull request lacks a [Phabricator](https://vyos.dev/) reference, maintainers will request +that you amend the commit message. + +### Writing good commit messages + +Follow the format described in +the [Git documentation](https://git-scm.com/book/ch5-2.html) +and [Chris Beams' guide](https://chris.beams.io/posts/git-commit/). + +Commit message format: + +1. **Summary line** (50 characters recommended, 80 maximum): Include the + component + prefix and [Phabricator](https://vyos.dev/) reference (for example, `snmp: T1111:` or + `ethernet: T2222:`). Concatenate multiple components with colons + (for example, `snmp: ethernet: T3333`). +2. **Blank line**: Separate the summary from the body. + This blank line is critical. + +4) **Message body** with details: + + - Describe what changed, why, and how. This helps with `git bisect`. + - Wrap text at 72 characters for readability with `git log` on an 80x25 + terminal. + - Reference previous commits when applicable: + `After commit abcd12ef ("snmp: this is a headline") + a Python import statement is missing, throwing the following exception: + ABCDEF` + +5) **Cherry-pick option**: Always use the `-x` option when back-porting or + forward-porting commits: + + `git cherry-pick -x <commit>` + + This appends `(cherry picked from commit <ID>)` to the commit message, + making bisecting easier. + +6) **Single responsibility**: Each commit must be self-contained. Do not fix + multiple bugs in a single commit. Use `git add --patch` to stage only + the parts related to one issue. + +Constraints: + +- Bugfixes are only accepted for packages other than + <https://github.com/vyos/vyos-1x>. + New functionality must use the new XML/Python interface, not old-style + templates (`node.def` files and Perl/Bash code). + +(coding-guidelines)= + +## Coding guidelines + +VyOS maintains consistent coding standards to help contributors navigate the +codebase and understand its logic. + +### Formatting + +- **Python**: Use 4 spaces per indentation level. Tabs **must not** be used. +- **XML**: Use 2 spaces per indentation level. Tabs **must not** be used. + +Use tools like VIM extensions (xmllint) to enforce correct indentation. Add this +to your `.vimrc` file: +```none +au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null +``` +Then use `gg=G` in command mode to run the linter. + +### Text generation + +Use a template processor for generating config files: + +- **Jinja2** is the default template processor for VyOS code. +- Built-in string formatting **may** be used for simple line-oriented formats + (for example, iptables rules) where every line is self-contained. +- Template processors **must** be used for structured, multi-line formats + (for example, ISC DHCPd configuration). + +### Python code + +Configuration scripts and operation mode scripts written in Python3 should +follow these guidelines: + +- Wrap lines at 80 characters. This improves readability when browsing + GitHub on mobile devices and reads well in side-by-side diffs. + +Structure your scripts with these functions: +```python +#!/usr/bin/env python3 +# +# Copyright (C) 2020 VyOS maintainers and contributors +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 or later as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +import sys + +from vyos.config import Config +from vyos import ConfigError + +def get_config(config=None): + if config: + conf = config + else: + conf = Config() + + # Base path to CLI nodes + base = ['...', '...'] + # Convert the VyOS config to an abstract internal representation + config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) + return config_data + +def verify(config): + # Verify that configuration is valid + if invalid: + raise ConfigError("Descriptive message") + +def generate(config): + # Generate daemon configs + pass + +def apply(config): + # Apply the generated configs to the live system + pass + +try: + c = get_config() + verify(c) + generate(c) + apply(c) +except ConfigError as e: + print(e) + sys.exit(1) +``` +`get_config()`: This function converts a VyOS config object to an abstract +internal representation. No other function may call the `vyos.config.Config` +object directly. Limiting config reads to one function makes it easier to +modify the config syntax in the future. Additionally, this design improves +testability since you can construct an internal representation by hand rather +than mocking the entire config subsystem. + +`verify()`: This function validates the internal representation. It must +raise `ConfigError` with a descriptive message if the config is invalid. It +**must not** make any changes to the system. This design enables future features +like commit dry-run ("commit test" as in JunOS) where the system can abort a +commit before making changes. + +`generate()`: This function generates config files for system components. + +`apply()`: This function applies the generated configuration to the live +system. Prefer non-disruptive reload when possible. Disruptive operations like +daemon restarts are acceptable only when: + +- The component does not support non-disruptive reload, or +- The expected service degradation is minimal (for example, auxiliary services + like LLDPd) + +For high-impact services (VPN daemons, routing protocols), make effort to +determine if changes can be applied non-disruptively before resorting to +restarts. + +Never modify active configuration directly unless absolutely necessary. Instead, +generate configuration files and apply them with a single command like service +reload through systemd. For example, save iptables rules to a file and load them +with `iptables-restore` rather than executing iptables commands one by one. + +The `apply()` and `generate()` functions may raise `ConfigError` if the +daemon fails to start with the updated config. However, this is not a substitute +for proper config validation in the `verify()` function. Make reasonable +effort to verify that generated configuration is valid and will be accepted by +the daemon, including cross-checks with other VyOS configuration subtrees when +necessary. + +Exceptions like `VyOSError` (raised by `vyos.config.Config` on improper +operations) should not be silenced or caught. While this may produce less +polished error output for users, it generates better bug reports and helps +maintainers debug issues. + +For reference implementations, see `ntp.py` or `interfaces-bonding.py` (for +tag nodes) in the [vyos-1x](https://github.com/vyos/vyos-1x) repository. + +### Other considerations: `vyos-configd` + +All scripts now run under the config daemon and must conform to these +requirements: + +1. The signature and first four lines of `get_config(...)` **must** be as + specified above. +2. Each of `get_config`, `verify`, `apply`, and `generate` **must** + appear + with the correct signatures, even if they are a no-op. +3. `Config` objects other than those in `get_config` **must not** appear. +4. The legacy function `my_set` **must not** appear. Modifications to active + config **should not** appear in new code (alternative mechanisms may be used + if absolutely necessary). + +## XML for CLI definitions + +XML interface definitions define the VyOS CLI structure. +Before VyOS `1.2` (crux), these +files were created manually. After a redesign, new-style templates are +automatically generated from XML input files. + +VyOS interface definitions come with a RelaxNG schema located in the +[vyos-1x](https://github.com/vyos/vyos-1x/tree/current/schema) +repository. This schema is a modified version from `VyConf` (VyOS `2.0`). +VyOS `1.2.x` +interface definitions are reusable in future VyOS versions with minimal changes. + +Schemas provide two benefits: + +- Complete grammar verification +- Automatic validation against the schema + +The [build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) +script converts XML definitions to +old-style templates and verifies them against the schema. A bad definition +causes the package build to fail. While the XML format is verbose, no other +format provides this level of verification. Specialized XML editors can help +manage verbosity. + +Example XML interface definition: +```xml +<?xml version="1.0"?> +<!-- Cron configuration --> +<interfaceDefinition> + <node name="system"> + <children> + <node name="task-scheduler"> + <properties> + <help>Task scheduler settings</help> + </properties> + <children> + <tagNode name="task" owner="${vyos_conf_scripts_dir}/task_scheduler.py"> + <properties> + <help>Scheduled task</help> + <valueHelp> + <format><string></format> + <description>Task name</description> + </valueHelp> + <priority>999</priority> + </properties> + <children> + <leafNode name="crontab-spec"> + <properties> + <help>UNIX crontab time specification string</help> + </properties> + </leafNode> + <leafNode name="interval"> + <properties> + <help>Execution interval</help> + <valueHelp> + <format><minutes></format> + <description>Execution interval in minutes</description> + </valueHelp> + <valueHelp> + <format><minutes>m</format> + <description>Execution interval in minutes</description> + </valueHelp> + <valueHelp> + <format><hours>h</format> + <description>Execution interval in hours</description> + </valueHelp> + <valueHelp> + <format><days>d</format> + <description>Execution interval in days</description> + </valueHelp> + <constraint> + <regex>[1-9]([0-9]*)([mhd]{0,1})</regex> + </constraint> + </properties> + </leafNode> + <node name="executable"> + <properties> + <help>Executable path and arguments</help> + </properties> + <children> + <leafNode name="path"> + <properties> + <help>Path to executable</help> + </properties> + </leafNode> + <leafNode name="arguments"> + <properties> + <help>Arguments passed to the executable</help> + </properties> + </leafNode> + </children> + </node> + </children> + </tagNode> + </children> + </node> + </children> + </node> +</interfaceDefinition> +``` +XML definitions are purely declarative and contain no logic. All logic for +generating config files, restarting services, and related tasks is implemented +in configuration scripts. + +### Template Processors + +XML interface definition files use the `.xml.in` file extension (implemented +in {vytask}`T1843`). These files use the GCC preprocessor to reduce code +duplication in common areas: + +- VIF (including VIF-S and VIF-C) +- Address configuration +- Description +- Enabled/Disabled state + +Instead of repeating XML nodes, use include files with predefined features: + +- [IPv4, IPv6, and DHCP(v6)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i) + address assignment. +- [IPv4 and IPv6](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i) + address assignment. +- [VLAN (VIF)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/accel-ppp/vlan.xml.i) + definition. +- [MAC address](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/firewall/mac-address.xml.i) + assignment. + +The `.in` files are preprocessed and stored in the [interface-definitions](https://github.com/vyos/vyos-1x/tree/current/interface-definitions) +folder. The [scripts/build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) +script then operates on this folder to generate all required CLI nodes. + +Example preprocessor output: +```none +$ make interface_definitions +install -d -m 0755 build/interface-definitions +install -d -m 0755 build/op-mode-definitions +Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in +Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in +Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in +Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in +Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in +Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in +[...] +``` + +### Command Definition Guidelines + +#### Use of Numbers + +Avoid using numbers in command names unless the number is part of a protocol +name or similar. For example, `protocols ospfv3` is appropriate, +but `server-1` is questionable. + +#### Help Strings + +Follow these guidelines for consistent, readable help strings: + +##### Capitalization and Punctuation + +- Capitalize the first word of every help string. +- Do not use a period at the end of help strings. + +This standard mirrors network device CLIs and improves aesthetics. + +Examples: + +- Good: "Frobnication algorithm" +- Bad: "frobnication algorithm" +- Bad: "Frobnication algorithm." +- Incorrect: "frobnication algorithm." + +##### Abbreviations and Acronyms + +- Capitalize all abbreviations and acronyms. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "tcp connection timeout" +- Bad: "Tcp connection timeout" +- Capitalize acronyms to distinguish them from normal words. + +Examples: + +- Good: RADIUS (remote authentication for dial-in user services) +- Bad: radius (unless referring to circular distance) +- Follow accepted spelling conventions for mixed-case abbreviations. If it + contains "over" or "version", use lowercase. Follow RFC or standard spellings + when they exist. + +Examples: + +- Good: PPPoE, IPsec +- Bad: PPPOE, IPSEC +- Bad: pppoe, ipsec + +##### Verbs + +- Avoid verbs. If a verb can be omitted, omit it. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "Set TCP connection timeout" +- When a verb is essential, use it. For example: "Disable IPv6 forwarding on + all interfaces" for `set system ipv6 disable-forwarding`. +- Use infinitive form for necessary verbs. + +Examples: + +- Good: "Disable IPv6 forwarding" +- Bad: "Disables IPv6 forwarding" + +## C++ Backend Code + +The VyOS CLI parser combines bash, bash-completion helpers, and the C++ backend +library [vyatta-cfg](https://github.com/vyos/vyatta-cfg). This section +references common CLI commands and their C/C++ entry points: + +`set`: + +- <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352> +- <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549> + +`commit`: + +- <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252> + + diff --git a/docs/operation/md-upgrade-recovery.md b/docs/operation/md-upgrade-recovery.md new file mode 100644 index 00000000..a8754d3b --- /dev/null +++ b/docs/operation/md-upgrade-recovery.md @@ -0,0 +1,63 @@ +--- +lastproofread: '2025-11-20' +--- + +(upgrade-recovery)= + +# Recovery after Failed Upgrades + +Use **VyOS upgrade recovery** to restore the system to the last working +version after a failed upgrade. + +- {ref}`Configuration: <configuration>` How to enable upgrade recovery +- {ref}`How it works: <how-it-works>` Overview of the recovery process +- {ref}`Cancelling recovery: <cancelling-recovery>` Overview of the recovery + process + +(configuration)= + +## Configuration + +:::{warning} +Upgrade recovery is disabled by default. To use it, +**enable it first**. +::: + +To enable upgrade recovery, run the following command: + +```{cfgcmd} set system option reboot-on-upgrade-failure [timeout \<min\>] +``` + +- `timeout <min>:` The time in minutes (5 - 30) to cancel upgrade + recovery before VyOS reboots. + See {ref}`Cancelling Recovery <cancelling-recovery>`. +(how-it-works)= + +## How it works +After a VyOS upgrade, the system monitors the boot process. Upon detecting a +boot failure, VyOS initiates a revert to the last working version and displays +the following warning: +```none +Booting failed, reverting to previous image +Automatic reboot in xx minutes +Use "reboot cancel" to cancel +``` +If no action is taken, the reboot happens automatically after the configured +timeout. Upon successful recovery and reboot, the following message appears: +```none +WARNING: Image update to "VyOS 1.5.xxxx" failed +Please check the logs: +/usr/lib/live/mount/persistence/boot/NAME/rw/var/log +Message is cleared on next reboot! +``` +(cancelling-recovery)= + +## Cancelling recovery +Upon detecting a boot failure, you have the predefined timeout to cancel +upgrade recovery. This is useful if you want to troubleshoot the faulty VyOS +version on your own. + +To cancel upgrade recovery, run the following command: +```none +reboot cancel +``` diff --git a/docs/vpp/configuration/dataplane/md-buffers.md b/docs/vpp/configuration/dataplane/md-buffers.md new file mode 100644 index 00000000..e9bddec9 --- /dev/null +++ b/docs/vpp/configuration/dataplane/md-buffers.md @@ -0,0 +1,90 @@ +--- +lastproofread: '2026-02-23' +--- + +(vpp-config-dataplane-buffers)= + +```{include} /_include/need_improvement.txt +``` + +# VPP Dataplane Buffers Configuration + +Buffers are essential for handling network packets efficiently. Proper +configuration enhances performance and reliability, and is mandatory for +VPP to work. Buffers temporarily store packets during processing. Therefore, +their configuration must be in sync with NIC configuration, CPU threads, and +overall system resources. + +:::{important} +VPP buffers are allocated from the physical memory pool (`physmem`). The +total amount of memory available for buffer allocation is controlled by the +`physmem-max-size` setting, while the buffer configuration parameters +below control how that memory is used for buffer allocation. + +See {ref}`VPP Physical Memory Configuration <vpp-config-dataplane-physmem>` +for details on configuring `physmem`. +::: + +## Buffer Configuration Parameters + +The following parameters can be configured for VPP buffers: + +### buffers-per-numa +Number of buffers allocated per NUMA node. This setting optimizes +memory access patterns for multi-CPU systems. + +Typically, you need to tune this value if: +- The system has many interfaces +- NICs have many queues +- NICs have large descriptor sizes + +Set this value carefully to balance memory usage and performance. +```{cfgcmd} set vpp settings resource-allocation buffers buffers-per-numa \<value\> +``` +The common approach for the calculation is to use the formula: +```none +buffers-per-numa = (num-rx-queues * num-rx-desc) + (num-tx-queues * num-tx-desc) +``` +Calculate this formula for each NIC and sum the results. Multiply the +total by 2.5 to get the minimum recommended value for +`buffers-per-numa`. + +Avoid setting this value too low to prevent packet drops. + +### data-size +This value sets how much payload data can be stored in a single buffer +allocated by VPP. Larger values reduce buffer chains for large packets, +while smaller values conserve memory for environments handling mostly +small packets. +```{cfgcmd} set vpp settings resource-allocation buffers data-size \<value\> +``` +Optimal size depends on the typical packet size in your network. If +unsure, use the largest MTU in your network plus overhead (for example, +128 bytes). + +### page-size +A memory pages type used for buffer allocation. Common values are 4K, 2M, or 1G. + +Use page sizes configured in your system settings. +```{cfgcmd} set vpp settings resource-allocation buffers page-size \<value\> +``` + +## Potential Issues and Troubleshooting + +Improper buffer configuration can lead to issues such as: + +- Increased latency and packet loss +- Inefficient CPU utilization +- Interface initialization failures + +Indicators of such issues are: + +- Errors during interfaces initialization in VPP logs +- Packet drops observed in VPP statistics + +To troubleshoot buffer-related issues, consider the following steps: + +- Review VPP logs for errors related to buffer allocation. Look for + error `-5` messages. +- Tune available buffers by adjusting the `buffers-per-numa` and + `data-size` parameters. diff --git a/docs/vpp/configuration/dataplane/md-unix.md b/docs/vpp/configuration/dataplane/md-unix.md new file mode 100644 index 00000000..9832b86d --- /dev/null +++ b/docs/vpp/configuration/dataplane/md-unix.md @@ -0,0 +1,54 @@ +--- +lastproofread: '2026-02-27' +--- + +(vpp-config-dataplane-unix)= + +```{include} /_include/need_improvement.txt +``` + +# VPP Unix Dataplane Configuration +The UNIX configuration section is used to control VPP's interaction +with the underlying operating system, including operations scheduling. + +VPP relies on the polling mechanism to efficiently manage I/O operations +and system events. By default VPP continuously polls for events, which +leads to permanent 100% CPU usage by all cores assigned to VPP dataplane. +This is optimal for performance, but may not be desirable in all +environments, especially where power consumption is a concern or where VPP +is running inside a hypervisor, especially if the VM has burstable +thresholds and CPU usage limits. + +To mitigate this, VPP provides a configurable polling delay that allows +reducing CPU usage by introducing a delay between polling cycles. This +introduces a trade-off between CPU usage and latency, as longer delays +can lead to increased latency in processing events. + +You can configure the polling delay using the following command in the +VyOS CLI: +```{cfgcmd} set vpp settings poll-sleep-usec \<delay\> +``` + +Sets the polling delay in microseconds. A value of 0 means no delay +(default), while higher values introduce a delay between polling cycles. + +## Troubleshooting + +Setting the polling delay too high can lead to increased latency and +reduced performance, as VPP may not respond to events as quickly. +Conversely, setting it too low may result in high CPU usage, which can be +problematic in resource-constrained environments. + +Symptoms of improper configuration may include: + +- Increased latency in packet processing +- Higher CPU usage than expected +- Packets lost due to buffer overruns + +If you do not need to reduce CPU usage, it is recommended to leave the +polling delay at its default value of 0 for optimal performance. + +If you need to reduce CPU usage, you may also consider using `interrupt` or +`adaptive` {ref}`DPDK driver modes <vpp-config-dataplane-interface-rx-mode>`, +which can provide a balance between performance and resource utilization +without affecting polling behavior. |