diff options
Diffstat (limited to 'docs/contributing')
| -rw-r--r-- | docs/contributing/md-cla.md | 45 | ||||
| -rw-r--r-- | docs/contributing/md-debugging.md | 204 | ||||
| -rw-r--r-- | docs/contributing/md-development.md | 541 | ||||
| -rw-r--r-- | docs/contributing/md-index.md | 13 | ||||
| -rw-r--r-- | docs/contributing/md-issues-features.md | 122 | ||||
| -rw-r--r-- | docs/contributing/md-testing.md | 206 | ||||
| -rw-r--r-- | docs/contributing/md-upstream-packages.md | 149 |
7 files changed, 1280 insertions, 0 deletions
diff --git a/docs/contributing/md-cla.md b/docs/contributing/md-cla.md new file mode 100644 index 00000000..01323111 --- /dev/null +++ b/docs/contributing/md-cla.md @@ -0,0 +1,45 @@ +--- +lastproofread: '2025-12-05' +--- + +(cla)= + +# Contributor License Agreement + +Before we can accept your contributions to VyOS, you must sign a **Contributor +License Agreement (CLA)**. + +This is a standard open-source practice that protects both you and the project. + +The process is straightforward and fully automated: + +1. **Review the CLA document** + + Find the CLA text in our + [GitHub repository](https://github.com/vyos/vyos-cla-signatures/). + +2. **Submit a pull request** + + When you open a pull request, a CLA bot automatically checks whether all + commit authors have signed the CLA. + +3. **Follow the bot's instructions** + + If the CLA has not been signed, the bot leaves a comment with instructions. + Reply to that comment with the suggested text to sign the CLA. + +4. **Wait for confirmation** + + The CLA bot verifies your response and updates the pull request status. + Once all commit authors have signed, the bot confirms that the CLA + requirement is met and unlocks the pull request for merging. + +:::{note} +Each commit author must sign the CLA. + +If your pull request includes commits from multiple contributors, each one +must sign the CLA before the pull request can be accepted. +::: + +Once you sign the CLA, it remains valid for all your past and future +contributions to VyOS under the same GitHub identity. diff --git a/docs/contributing/md-debugging.md b/docs/contributing/md-debugging.md new file mode 100644 index 00000000..d3b4b513 --- /dev/null +++ b/docs/contributing/md-debugging.md @@ -0,0 +1,204 @@ +--- +lastproofread: '2025-12-05' +--- + +(debugging)= + +# Debugging + +Two flags are available to help debug configuration scripts. Configuration +loading issues manifest during boot, so these flags are passed as kernel boot +parameters. + +## ISO image build + +If you have trouble compiling your own ISO image or debugging Jenkins issues, +follow the steps at {ref}`iso_build_issues`. + +## System Startup + +Debug system startup by examining the configuration file loading from +`/config/config.boot`. Extend the kernel command-line in the bootloader to +enable this. + +### Kernel + +- `vyos-debug` - Add this parameter to the Linux boot line to produce + timing results for script execution during commit. If you see an unexpected + delay during manual or boot commit, this parameter helps identify bottlenecks. + The internal flag is `VYOS_DEBUG`, found in [vyatta-cfg]. Output is directed + to `/var/log/vyatta/cfg-stdout.log`. +- `vyos-config-debug` - During development, coding errors can cause commit + failures on boot, potentially preventing CLI initialization. This kernel boot + parameter ensures access to the system as user `vyos` and logs a Python + stack trace to `/tmp/boot-config-trace`. The file is created only if the + configuration load fails. + +## Live System + +Several flags can be set to change VyOS behavior at runtime. Toggle these flags +using environment variables or by creating files. + +For each feature, create a file called `vyos.feature.debug` to enable it. +If a parameter is required, place it as the first line inside the file. + +Place the file in `/tmp` for one-time debugging (the file is removed on +reboot) or in `/config` to persist permanently. + +For example, `/tmp/vyos.ifconfig.debug` can be created to enable +interface debugging. + +You can also enable debugging using environment variables. +The environment variable name follows the convention `VYOS_FEATURE_DEBUG`. + +For example, `export VYOS_IFCONFIG_DEBUG=""` in your vbash has the same effect +as `touch /tmp/vyos.ifconfig.debug`. + +- `ifconfig` - Display all commands and their responses from the OS on + screen for inspection. +- `command` - Display all commands and their responses from the OS on screen + for inspection. +- `developer` - When a command fails, start a PDB post-mortem session instead + of showing a standard error message. This allows developers to debug issues + interactively. Because the debugger waits for input, it can prevent the router + from booting, so only enable this permanently on production systems if you are + ready for potential boot failures. +- `log` - Send all commands used by VyOS to a log file for inspection. This + is useful in rare cases when you need to see what the OS is doing, including + during boot. The default file is `/tmp/full-log`, but you can change it. + +:::{note} +To retrieve debug output on the command line, disable `vyos-configd` +in addition. You can do this one-time with +`sudo systemctl stop vyos-configd` +or permanently with `sudo systemctl disable vyos-configd`. +::: + +### FRR + +Recent versions use the `vyos.frr` framework. The Python class is located in +`vyos-1x:python/vyos/frr.py`. It includes an embedded debugger similar to the +one in `vyos.ifconfig`. + +Enable debugging by running: `touch /tmp/vyos.frr.debug` + +### Debug Python code with PDB + +Sometimes it is useful to debug Python code interactively on the live system +rather than in an IDE. You can do this using pdb. + +Assuming you want to debug a Python script called by an op-mode command, find +the script by looking up the op-mode definitions, then edit it on the live +system using vi: +`vi /usr/libexec/vyos/op_mode/show_xyz.py` + +Insert the following statement right before the section where you want to +investigate a problem (for example, a statement you see in a backtrace): +`import pdb; pdb.set_trace()` + +Optionally, surround this statement with an `if` condition that triggers only +for the conditions you are interested in. + +When you run `show xyz` and your condition triggers, you enter the Python +debugger: + +```none +> /usr/libexec/vyos/op_mode/show_nat_translations.py(109)process() +-> rule_type = rule.get('type', '') +(Pdb) +``` + +You can type `help` to get an overview of the available commands, and +`help command` to get more information on each command. + +Common useful commands include: + +- examine variables using `pp(var)` +- continue execution using `cont` +- get a backtrace using `bt` + +### Config Migration Scripts + +Starting with VyOS 1.5, a new mechanism is used for config migration that +improves migration performance. New migrators use only the new format with a +`migration()` function. + +```python +from vyos.configtree import ConfigTree +base = ['vpn', 'ipsec'] +def migrate(config: ConfigTree) -> None: + if not config.exists(base): + # Nothing to do + return + # do your stuff here +``` + +New-style migration scripts can no longer run on their own. However, the new +migration subsystem handler includes a test kit: + +```none +vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --help +usage: run-config-migration.py [-h] [--test-script TEST_SCRIPT] [--output-file OUTPUT_FILE] [--force] config_file + +positional arguments: + config_file configuration file to migrate + +options: + -h, --help show this help message and exit + --test-script TEST_SCRIPT + test named script + --output-file OUTPUT_FILE + write to named output file instead of config file + --force force run of all migration scripts +``` + +To test your migration, run: + +```none +vyos@vyos:~$ /usr/libexec/vyos/run-config-migration.py --test-script /opt/vyatta/etc/config-migrate/migrate/quagga/11-to-12 --output-file /tmp/foo /tmp/static-route-basic +vyos@vyos:~$ cat /tmp/foo +``` + +The file `/tmp/foo` contains the migrated configuration. + +### Configuration Error on System Boot + +Running the latest rolling releases sometimes exposes bugs due to edge cases +missed in design. File these bugs via [Phabricator](https://vyos.dev/), but you can help narrow +down the issue by following these steps: + +1. Log in to your VyOS system. +2. Enter configuration mode: `configure` +3. Reload your boot configuration: `load` + +You should see a Python backtrace that helps identify the issue. Attach it to +the [Phabricator](https://vyos.dev/) task. + +### Boot Timing + +During the migration and rewrite of functionality from Perl to Python, system +boot time increased significantly. You can analyze and graph boot time to see +detailed call sequences during startup. + +This uses the `systemd-bootchart` package, which is installed by default on +VyOS 1.3 (equuleus) and later. Configuration is versioned for comparable +results. Refer to [bootchart.conf] for the configuration file. + +To enable boot time graphing, add the following to the kernel command line: +`init=/usr/lib/systemd/systemd-bootchart` + +You can also make this permanent by editing `/boot/grub/grub.cfg`. + +## Priorities + +VyOS CLI depends heavily on priorities. Every CLI node has a corresponding +`node.def` file and possibly an attached script. Nodes can have priorities, +and on system bootup or any `commit` to the configuration, scripts execute +from lowest to highest priority. This provides deterministic behavior. + +To debug priority issues or see script execution order, use the +`/opt/vyatta/sbin/priority.pl` script, which lists the execution order of +scripts. + +[bootchart.conf]: https://github.com/vyos/vyos-build/blob/current/data/live-build-config/includes.chroot/etc/systemd/bootchart.conf +[vyatta-cfg]: https://github.com/vyos/vyatta-cfg diff --git a/docs/contributing/md-development.md b/docs/contributing/md-development.md new file mode 100644 index 00000000..8581a28e --- /dev/null +++ b/docs/contributing/md-development.md @@ -0,0 +1,541 @@ +--- +lastproofread: '2025-12-12' +--- + +(development)= + +# Development + +Learn how to contribute to VyOS. + +(architecture-overview)= + +## Architecture overview + +VyOS source code is hosted on GitHub in the VyOS organization: +<https://github.com/vyos> + +VyOS is composed of multiple modules spread across different +repositories. Some modules contain forks of upstream +packages and are periodically synced. +VyOS consolidates most packages into the +[vyos-1x](https://github.com/vyos/vyos-1x) +repository while maintaining a consistent structure. +The base code is being rewritten +from Perl and Bash to Python using an XML-based CLI interface definition. + +VyOS ISO build scripts are hosted in the +[vyos-build](https://github.com/vyos/vyos-build) repository. See the +`vyos-build` repository +[README.md file](https://github.com/vyos/vyos-build/blob/current/README.md) +for more information on building VyOS ISO images. + +## Contributing code + +:::{warning} +You must sign the {doc}`Contributor License Agreement<cla>` +for your contributions to be accepted. +::: + +VyOS is open-source and welcomes patches. +All submissions must adhere to these guidelines: + +- Each commit addresses a single issue or feature. +- Each commit message references a [Phabricator](https://vyos.dev/) task ID + (for example, `T1234`). +- Each commit is associated with a username and email address + to identify the author (see [Configure your Git identity](configure-your-git-identity)). +- Only submit bugfixes in packages other than <https://github.com/vyos/vyos-1x>. +- Commits follow the [coding guidelines](coding-guidelines) outlined below. + +### Determining package ownership + +To determine which VyOS package contains a file you want to modify, use Debian's +`dpkg -S` command on your running VyOS installation. + +### Submitting your code + +Fork the repository and submit a GitHub pull request. This is the preferred way +to contribute changes to VyOS. + +To fork a VyOS repository: + +1. Append `/fork` to the repository URL on GitHub. For example, to fork + `vyos-1x`, use: <https://github.com/vyos/vyos-1x/fork> + +2. Clone your fork or add it as a remote to your local repository: + + - Clone: `git clone https://github.com/<user>/vyos-1x.git` + - Add remote: `git remote add myfork https://github.com/<user>/vyos-1x.git` + +(configure-your-git-identity)= + +3. Configure your Git identity: + + ```none + git config --global user.name "J. Random Hacker" + git config --global user.email "jrhacker@example.net" + ``` + +4. Make your changes and add files to the Git index: + + - Single file: `git add myfile` + - Directory: `git add somedir/*` + +5. Commit your changes with a meaningful headline and [Phabricator](https://vyos.dev/) reference: + + `git commit` + +6. Push to your fork and create a GitHub pull request: + + `git push` + +Alternatively, you can export commits as patches and send them to +<mailto:maintainers@vyos.net> or attach them directly to the [Phabricator](https://vyos.dev/) task: + +- Export last commit: `git format-patch` +- Export last two commits: `git format-patch -2` + +## Commit messages + +For guidance on writing commit messages, review the file history +with `git log path/to/file.txt`. + +Every change must be associated with a task number (prefixed with **T**) and +a component. If no bug report or feature request exists for your changes, +create a [Phabricator](https://vyos.dev/) task first. Reference the task ID in your commit message: + +- `ddclient: T1030: auto create runtime directories` +- `Jenkins: add current Git commit ID to build description` + +If your pull request lacks a [Phabricator](https://vyos.dev/) reference, maintainers will request +that you amend the commit message. + +### Writing good commit messages + +Follow the format described in +the [Git documentation](https://git-scm.com/book/ch5-2.html) +and [Chris Beams' guide](https://chris.beams.io/posts/git-commit/). + +Commit message format: + +1. **Summary line** (50 characters recommended, 80 maximum): Include the + component + prefix and [Phabricator](https://vyos.dev/) reference (for example, `snmp: T1111:` or + `ethernet: T2222:`). Concatenate multiple components with colons + (for example, `snmp: ethernet: T3333`). +2. **Blank line**: Separate the summary from the body. + This blank line is critical. + +4) **Message body** with details: + + - Describe what changed, why, and how. This helps with `git bisect`. + - Wrap text at 72 characters for readability with `git log` on an 80x25 + terminal. + - Reference previous commits when applicable: + `After commit abcd12ef ("snmp: this is a headline") + a Python import statement is missing, throwing the following exception: + ABCDEF` + +5) **Cherry-pick option**: Always use the `-x` option when back-porting or + forward-porting commits: + + `git cherry-pick -x <commit>` + + This appends `(cherry picked from commit <ID>)` to the commit message, + making bisecting easier. + +6) **Single responsibility**: Each commit must be self-contained. Do not fix + multiple bugs in a single commit. Use `git add --patch` to stage only + the parts related to one issue. + +Constraints: + +- Bugfixes are only accepted for packages other than + <https://github.com/vyos/vyos-1x>. + New functionality must use the new XML/Python interface, not old-style + templates (`node.def` files and Perl/Bash code). + +## Coding guidelines + +VyOS maintains consistent coding standards to help contributors navigate the +codebase and understand its logic. + +### Formatting + +- **Python**: Use 4 spaces per indentation level. Tabs **must not** be used. +- **XML**: Use 2 spaces per indentation level. Tabs **must not** be used. + +Use tools like VIM extensions (xmllint) to enforce correct indentation. Add this +to your `.vimrc` file: +```none +au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null +``` +Then use `gg=G` in command mode to run the linter. + +### Text generation + +Use a template processor for generating config files: + +- **Jinja2** is the default template processor for VyOS code. +- Built-in string formatting **may** be used for simple line-oriented formats + (for example, iptables rules) where every line is self-contained. +- Template processors **must** be used for structured, multi-line formats + (for example, ISC DHCPd configuration). + +### Python code + +Configuration scripts and operation mode scripts written in Python3 should +follow these guidelines: + +- Wrap lines at 80 characters. This improves readability when browsing + GitHub on mobile devices and reads well in side-by-side diffs. + +Structure your scripts with these functions: +```python +#!/usr/bin/env python3 +# +# Copyright (C) 2020 VyOS maintainers and contributors +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 or later as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +import sys + +from vyos.config import Config +from vyos import ConfigError + +def get_config(config=None): + if config: + conf = config + else: + conf = Config() + + # Base path to CLI nodes + base = ['...', '...'] + # Convert the VyOS config to an abstract internal representation + config_data = conf.get_config_dict(base, key_mangling=('-', '_'), get_first_key=True) + return config_data + +def verify(config): + # Verify that configuration is valid + if invalid: + raise ConfigError("Descriptive message") + +def generate(config): + # Generate daemon configs + pass + +def apply(config): + # Apply the generated configs to the live system + pass + +try: + c = get_config() + verify(c) + generate(c) + apply(c) +except ConfigError as e: + print(e) + sys.exit(1) +``` +`get_config()`: This function converts a VyOS config object to an abstract +internal representation. No other function may call the `vyos.config.Config` +object directly. Limiting config reads to one function makes it easier to +modify the config syntax in the future. Additionally, this design improves +testability since you can construct an internal representation by hand rather +than mocking the entire config subsystem. + +`verify()`: This function validates the internal representation. It must +raise `ConfigError` with a descriptive message if the config is invalid. It +**must not** make any changes to the system. This design enables future features +like commit dry-run ("commit test" as in JunOS) where the system can abort a +commit before making changes. + +`generate()`: This function generates config files for system components. + +`apply()`: This function applies the generated configuration to the live +system. Prefer non-disruptive reload when possible. Disruptive operations like +daemon restarts are acceptable only when: + +- The component does not support non-disruptive reload, or +- The expected service degradation is minimal (for example, auxiliary services + like LLDPd) + +For high-impact services (VPN daemons, routing protocols), make effort to +determine if changes can be applied non-disruptively before resorting to +restarts. + +Never modify active configuration directly unless absolutely necessary. Instead, +generate configuration files and apply them with a single command like service +reload through systemd. For example, save iptables rules to a file and load them +with `iptables-restore` rather than executing iptables commands one by one. + +The `apply()` and `generate()` functions may raise `ConfigError` if the +daemon fails to start with the updated config. However, this is not a substitute +for proper config validation in the `verify()` function. Make reasonable +effort to verify that generated configuration is valid and will be accepted by +the daemon, including cross-checks with other VyOS configuration subtrees when +necessary. + +Exceptions like `VyOSError` (raised by `vyos.config.Config` on improper +operations) should not be silenced or caught. While this may produce less +polished error output for users, it generates better bug reports and helps +maintainers debug issues. + +For reference implementations, see `ntp.py` or `interfaces-bonding.py` (for +tag nodes) in the [vyos-1x](https://github.com/vyos/vyos-1x) repository. + +### Other considerations: `vyos-configd` + +All scripts now run under the config daemon and must conform to these +requirements: + +1. The signature and first four lines of `get_config(...)` **must** be as + specified above. +2. Each of `get_config`, `verify`, `apply`, and `generate` **must** + appear + with the correct signatures, even if they are a no-op. +3. `Config` objects other than those in `get_config` **must not** appear. +4. The legacy function `my_set` **must not** appear. Modifications to active + config **should not** appear in new code (alternative mechanisms may be used + if absolutely necessary). + +## XML for CLI definitions + +XML interface definitions define the VyOS CLI structure. +Before VyOS `1.2` (crux), these +files were created manually. After a redesign, new-style templates are +automatically generated from XML input files. + +VyOS interface definitions come with a RelaxNG schema located in the +[vyos-1x](https://github.com/vyos/vyos-1x/tree/current/schema) +repository. This schema is a modified version from `VyConf` (VyOS `2.0`). +VyOS `1.2.x` +interface definitions are reusable in future VyOS versions with minimal changes. + +Schemas provide two benefits: + +- Complete grammar verification +- Automatic validation against the schema + +The [build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) +script converts XML definitions to +old-style templates and verifies them against the schema. A bad definition +causes the package build to fail. While the XML format is verbose, no other +format provides this level of verification. Specialized XML editors can help +manage verbosity. + +Example XML interface definition: +```xml +<?xml version="1.0"?> +<!-- Cron configuration --> +<interfaceDefinition> + <node name="system"> + <children> + <node name="task-scheduler"> + <properties> + <help>Task scheduler settings</help> + </properties> + <children> + <tagNode name="task" owner="${vyos_conf_scripts_dir}/task_scheduler.py"> + <properties> + <help>Scheduled task</help> + <valueHelp> + <format><string></format> + <description>Task name</description> + </valueHelp> + <priority>999</priority> + </properties> + <children> + <leafNode name="crontab-spec"> + <properties> + <help>UNIX crontab time specification string</help> + </properties> + </leafNode> + <leafNode name="interval"> + <properties> + <help>Execution interval</help> + <valueHelp> + <format><minutes></format> + <description>Execution interval in minutes</description> + </valueHelp> + <valueHelp> + <format><minutes>m</format> + <description>Execution interval in minutes</description> + </valueHelp> + <valueHelp> + <format><hours>h</format> + <description>Execution interval in hours</description> + </valueHelp> + <valueHelp> + <format><days>d</format> + <description>Execution interval in days</description> + </valueHelp> + <constraint> + <regex>[1-9]([0-9]*)([mhd]{0,1})</regex> + </constraint> + </properties> + </leafNode> + <node name="executable"> + <properties> + <help>Executable path and arguments</help> + </properties> + <children> + <leafNode name="path"> + <properties> + <help>Path to executable</help> + </properties> + </leafNode> + <leafNode name="arguments"> + <properties> + <help>Arguments passed to the executable</help> + </properties> + </leafNode> + </children> + </node> + </children> + </tagNode> + </children> + </node> + </children> + </node> +</interfaceDefinition> +``` +XML definitions are purely declarative and contain no logic. All logic for +generating config files, restarting services, and related tasks is implemented +in configuration scripts. + +### Template Processors + +XML interface definition files use the `.xml.in` file extension (implemented +in {vytask}`T1843`). These files use the GCC preprocessor to reduce code +duplication in common areas: + +- VIF (including VIF-S and VIF-C) +- Address configuration +- Description +- Enabled/Disabled state + +Instead of repeating XML nodes, use include files with predefined features: + +- [IPv4, IPv6, and DHCP(v6)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6-dhcp.xml.i) + address assignment. +- [IPv4 and IPv6](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/interface/address-ipv4-ipv6.xml.i) + address assignment. +- [VLAN (VIF)](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/accel-ppp/vlan.xml.i) + definition. +- [MAC address](https://github.com/vyos/vyos-1x/blob/current/interface-definitions/include/firewall/mac-address.xml.i) + assignment. + +The `.in` files are preprocessed and stored in the [interface-definitions](https://github.com/vyos/vyos-1x/tree/current/interface-definitions) +folder. The [scripts/build-command-templates](https://github.com/vyos/vyos-1x/blob/current/scripts/build-command-templates) +script then operates on this folder to generate all required CLI nodes. + +Example preprocessor output: +```none +$ make interface_definitions +install -d -m 0755 build/interface-definitions +install -d -m 0755 build/op-mode-definitions +Generating build/interface-definitions/intel_qat.xml from interface-definitions/intel_qat.xml.in +Generating build/interface-definitions/interfaces-bonding.xml from interface-definitions/interfaces-bonding.xml.in +Generating build/interface-definitions/cron.xml from interface-definitions/cron.xml.in +Generating build/interface-definitions/pppoe-server.xml from interface-definitions/pppoe-server.xml.in +Generating build/interface-definitions/mdns-repeater.xml from interface-definitions/mdns-repeater.xml.in +Generating build/interface-definitions/tftp-server.xml from interface-definitions/tftp-server.xml.in +[...] +``` + +### Command Definition Guidelines + +#### Use of Numbers + +Avoid using numbers in command names unless the number is part of a protocol +name or similar. For example, `protocols ospfv3` is appropriate, +but `server-1` is questionable. + +#### Help Strings + +Follow these guidelines for consistent, readable help strings: + +##### Capitalization and Punctuation + +- Capitalize the first word of every help string. +- Do not use a period at the end of help strings. + +This standard mirrors network device CLIs and improves aesthetics. + +Examples: + +- Good: "Frobnication algorithm" +- Bad: "frobnication algorithm" +- Bad: "Frobnication algorithm." +- Incorrect: "frobnication algorithm." + +##### Abbreviations and Acronyms + +- Capitalize all abbreviations and acronyms. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "tcp connection timeout" +- Bad: "Tcp connection timeout" +- Capitalize acronyms to distinguish them from normal words. + +Examples: + +- Good: RADIUS (remote authentication for dial-in user services) +- Bad: radius (unless referring to circular distance) +- Follow accepted spelling conventions for mixed-case abbreviations. If it + contains "over" or "version", use lowercase. Follow RFC or standard spellings + when they exist. + +Examples: + +- Good: PPPoE, IPsec +- Bad: PPPOE, IPSEC +- Bad: pppoe, ipsec + +##### Verbs + +- Avoid verbs. If a verb can be omitted, omit it. + +Examples: + +- Good: "TCP connection timeout" +- Bad: "Set TCP connection timeout" +- When a verb is essential, use it. For example: "Disable IPv6 forwarding on + all interfaces" for `set system ipv6 disable-forwarding`. +- Use infinitive form for necessary verbs. + +Examples: + +- Good: "Disable IPv6 forwarding" +- Bad: "Disables IPv6 forwarding" + +## C++ Backend Code + +The VyOS CLI parser combines bash, bash-completion helpers, and the C++ backend +library [vyatta-cfg](https://github.com/vyos/vyatta-cfg). This section +references common CLI commands and their C/C++ entry points: + +`set`: + +- <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L352> +- <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/cstore/cstore.cpp#L2549> + +`commit`: + +- <https://github.com/vyos/vyatta-cfg/blob/0f42786a0b3/src/commit/commit-algorithm.cpp#L1252> + + diff --git a/docs/contributing/md-index.md b/docs/contributing/md-index.md new file mode 100644 index 00000000..f26a6b70 --- /dev/null +++ b/docs/contributing/md-index.md @@ -0,0 +1,13 @@ +# Contributing + +```{toctree} +:maxdepth: 1 + +build-vyos +development +cla +issues-features +upstream-packages +debugging +testing +``` diff --git a/docs/contributing/md-issues-features.md b/docs/contributing/md-issues-features.md new file mode 100644 index 00000000..ab235326 --- /dev/null +++ b/docs/contributing/md-issues-features.md @@ -0,0 +1,122 @@ +--- +lastproofread: '2025-12-08' +--- + +(issues_features)= + +# Issues/Feature requests + +(bug_report)= + +## Bug Report/Issue + +Issues and bugs occur in every software project, and VyOS is no exception. + +### I found a bug, what should I do? + +When you find a potential bug, first: + +- Consult the [documentation] to ensure you configured your system + correctly. +- Check if the VyOS community has identified a workaround for the bug through + [Slack] or the VyOS [Forum]. + +### Ensure the bug is reproducible + +Include the following information when reporting a bug: + +- A sequence of configuration commands or a complete configuration file needed + to recreate the bug. Avoid partial configurations: a sequence of commands is + easy to paste and a complete configuration is easy to load, but a partial + config is hard to reconstruct. +- Describe the expected behavior and how it differs from what you observe. + Include command outputs or traffic dumps. Explain briefly why these outputs + are incorrect and what the correct behavior should be. +- A sequence of actions that trigger the bug. While not always possible, this + helps developers and community members confirm the issue and verify fixes. +- If the bug is a regression, specify the VyOS version where the feature worked + correctly (any working version is acceptable). Identify the exact version + that the feature stopped working, if possible. + +If you are uncertain whether the behavior is a bug or what the correct behavior +is, or if you lack a reliable reproducing procedure, post on the forum or ask in +chat first. If you have a subscription, create a support ticket. The team and +community can help identify the issue, work around it, and create an actionable +bug report. + +### Report a Bug + +To open a bug report or feature request, create an account on +[vyos.dev](https://vyos.dev), the public issue tracker for VyOS. + +When creating a new issue, select the appropriate project and: + +- Provide as much information as you can. +- Specify which VyOS version you are using: `run show version`. +- Explain how to reproduce the bug. + +(feature-request)= + +## Feature Requests + +Have an idea to improve VyOS or need a feature that would benefit all users? +Before submitting a feature request, search the public issue tracker +[vyos.dev](https://vyos.dev) to check if a request already exists. You can +also enhance an existing request by providing additional information. + +Create a task before starting work on a feature, +even if it is a trivial feature. +The task tracker generates release notes, so all work must be reflected +in the tracker. + +Include at least the following information: + +- Provide a detailed description of the feature: what it is, how it works, and + how you would use it. Maintainers may not have experience with every feature, + protocol, and tool in VyOS. Detailed information helps VyOS contributors and + maintainers test new features they are unfamiliar with. +- Include proposed CLI syntax if the feature requires new commands. Provide both + configuration and operational mode commands if both are needed. + +Consider including the following information: + +- Is the feature already supported by the underlying component + (FreeRangeRouting, nftables, Kea, etc.)? +- How would you configure the feature manually within that component? +- Are there any limitations to using the feature + (hardware support, resource usage)? +- Are there any adverse or non-obvious interactions with other features? Should + the feature be mutually exclusive? +- Any relevant documentation or references about the feature. + +You do not need to provide all this information, but if you can, it simplifies +developers' work considerably. Research these questions when possible. + +## Task auto-closing + +A special task status exists for when all work by maintainers and contributors +is complete: **Needs reporter action**. + +VyOS assigns this status to: + +- Feature requests that do not include required information and need + clarification. +- Bug reports that lack reproducing procedures. +- Tasks that are implemented and tested by the implementation author, + but require testing in the real-world environment that only the reporter + can replicate (for example, hardware VyOS does not support or specific + network conditions). + +When a task is set to **Needs reporter action**: + +- If the reporter does not respond within two weeks, the task bot adds a comment + ("Any news?") to remind the reporter. +- If there is still no response after another two weeks, + the task is closed automatically. + +We do not auto-close tasks with any other status and do not close tasks due to +lack of maintainer activity. + +[documentation]: https://docs.vyos.io +[forum]: https://forum.vyos.io +[slack]: https://slack.vyos.io diff --git a/docs/contributing/md-testing.md b/docs/contributing/md-testing.md new file mode 100644 index 00000000..5e2371d6 --- /dev/null +++ b/docs/contributing/md-testing.md @@ -0,0 +1,206 @@ +--- +lastproofread: '2025-12-02' +--- + +(testing)= + +# Testing + +One of the major features introduced in VyOS 1.3 is an automated test +framework. When you assemble an ISO image, several things can go wrong. +VyOS uses this framework to detect issues before they cause downstream problems. + +This section describes how the automated testing process at VyOS works. + +## Smoketests + +Smoketests execute predefined VyOS CLI commands and check if the desired +daemon or service configuration is rendered. + +When an ISO image is assembled by the [VyOS CI](https://ci.vyos.net), the `BUILD_SMOKETEST` +parameter is enabled by default. This extends the ISO configuration line +with the following packages: + +```python +def CUSTOM_PACKAGES = '' + if (params.BUILD_SMOKETESTS) + CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest' +``` + +If you plan to build your own custom ISO image and want to use VyOS's +smoketests, ensure that you have the `vyos-1x-smoketest` package installed. + +The `make test` command from the [vyos-build](https://github.com/vyos/vyos-build) repository launches a new +QEMU instance, and the ISO image is first installed to the virtual hard disk. + +After the first boot into the newly installed system, the main Smoketest script +is executed. It can be found at `/usr/bin/vyos-smoketest`. + +The script searches for executable test cases under +`/usr/libexec/vyos/tests/smoke/cli/` and executes them one by one. + +:::{note} +Smoketests will alter the system configuration. If you are logged +in remotely, you may lose your connection to the system. +::: + +:::{note} +To enable smoketest debugging (print the CLI set commands used), +run: `touch /tmp/vyos.smoketest.debug`. +::: + +### Manual Smoketest Run + +Each test is contained in its own file, so you can execute a single Smoketest +manually by running the Python test script. + +Example: + +```none +vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py +test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok +test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok +test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok +test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok +test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok +test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok +test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok +test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok +test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok +test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok +test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok +test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok +test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok + +---------------------------------------------------------------------- +Ran 13 tests in 348.191s + +OK +``` + +### Interface-based tests + +Our smoketests not only test daemons and services, but also check if interface +configuration works as expected. There is a common base class named +`base_interfaces_test.py` that holds all the common code for interface tests. + +These common tests consist of: + +- Add one or more IP addresses + +- DHCP client and DHCPv6 prefix delegation + +- MTU size + +- IP and IPv6 options + +- Port description + +- Port disable + +- VLANs (QinQ and regular 802.1q) + +- ... + +:::{note} +When you are working on interface configuration and want to test +if the Smoketests pass, you would normally lose the remote SSH connection +to your {abbr}`DUT (Device Under Test)`. To handle this, some interface-based +tests can be called with an environment variable beforehand to limit the +number of interfaces used in the test. By default, all interfaces (e.g., all +Ethernet interfaces) are used. +::: + +```none +vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py +test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok +test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok +test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok +test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok +test_bonding_min_links (__main__.BondingInterfaceTest) ... ok +test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok +test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok +test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok +test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok +test_interface_description (__main__.BondingInterfaceTest) ... ok +test_interface_disable (__main__.BondingInterfaceTest) ... ok +test_interface_ip_options (__main__.BondingInterfaceTest) ... ok +test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok +test_interface_mtu (__main__.BondingInterfaceTest) ... ok +test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok +test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok +test_span_mirror (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok +test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok +test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok +test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok + +---------------------------------------------------------------------- +Ran 23 tests in 244.694s + +OK +``` + +This will limit the `bond` interface test to use only `eth1` and `eth2` +as member ports. + +## Config Load Tests + +The other part of our tests are called "config load tests." Config load tests +sequentially load arbitrary configuration files to verify that configuration +migration scripts work as designed and that a given set of functionality can +still be loaded with a fresh VyOS ISO image. + +The configurations are all derived from production systems and can act as +test cases or as references for enabling certain features. The configurations +can be found here: +<https://github.com/vyos/vyos-1x/tree/current/smoketest/configs> + +The entire test is controlled by the main wrapper script +`/usr/bin/vyos-configtest`. +It behaves in the same way as the main smoketest script. It scans the folder +for potential configuration files and issues a `load` command for each file. + +### Manual config load test + +You do not have to load all configurations sequentially; you can also load +individual test configurations manually. + +```none +vyos@vyos:~$ configure +load[edit] + +vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small +Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small' +Load complete. Use 'commit' to make changes effective. +[edit] +vyos@vyos# compare +[edit interfaces ethernet eth0] +-hw-id 00:50:56:bf:c5:6d +[edit interfaces ethernet eth1] ++duplex auto +-hw-id 00:50:56:b3:38:c5 ++speed auto +[edit interfaces] +-ethernet eth2 { +- hw-id 00:50:56:b3:9c:1d +-} +-vti vti1 { +- address 192.0.2.1/30 +-} +... + +vyos@vyos# commit +vyos@vyos# +``` + +:::{note} +Some configurations have preconditions that must be met. These most +likely include generation of cryptographic keys before the config can be +applied; otherwise, you will get a commit error. If you are interested in +how those preconditions are fulfilled, check the [vyos-build](https://github.com/vyos/vyos-build) repository and +the `scripts/check-qemu-install` file. +::: + diff --git a/docs/contributing/md-upstream-packages.md b/docs/contributing/md-upstream-packages.md new file mode 100644 index 00000000..c7da9066 --- /dev/null +++ b/docs/contributing/md-upstream-packages.md @@ -0,0 +1,149 @@ +--- +lastproofread: '2026-01-30' +--- + +(upstream-packages)= + +# Upstream Packages + +Many base system packages are pulled straight from Debian's `main` and +`contrib` repositories, but there are exceptions. If you only want to build +a fresh ISO image, you can skip +this section. This information may be useful for a deeper dive into VyOS. + +System packages that are not directly pulled from Debian are built through a +separate build system, `build.py` in the [vyos-build](https://github.com/vyos/vyos-build/tree/current/scripts/package-build) repository. + +## Overview + +Previously, VyOS used Jenkins for building upstream packages. With the move away +from Jenkins, the build system was replaced with a Python-based solution using +`build.py` and `package.toml` configuration files. + +Each package directory contains: + +- A `package.toml` configuration file that defines how the package is built. +- A symlink to the common `build.py` script in the build system. + +## Building Packages + +To build a package, navigate to the package directory and execute the +build script: + +```console +cd package-build/<package-name> +./build.py +``` + +The script will: + +1. Check out the source code from the configured repository. +2. Apply any patches defined in the configuration. +3. Execute pre-build hooks (if configured). +4. Build the package using the specified build command. +5. Generate both binary (`.deb`) packages and source tarballs. + +## Package Configuration (package.toml) + +Each package directory contains a `package.toml` file that defines the build +parameters. The key configuration fields are: + +**name** + +: The package name (e.g., `frr`) + +**commit_id** + +: The specific commit, tag, or branch to check out from the source repository + (e.g., `stable/10.5`) + +**scm_url** + +: The Git URL of the upstream source repository + (e.g., `https://github.com/FRRouting/frr.git`) + +**build_cmd** + +: The command to execute for building the package. This replaces what was + previously defined in the Jenkins `Jenkinsfile`. + + Default if not specified: `dpkg-buildpackage -uc -us -tc -F --source-option=--tar-ignore=.git --source-option=--tar-ignore=.github` + + Example with custom build command: + + ```toml + build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" + ``` + +**pre_build_hook** (Optional) + +: A shell command or script that executes after the repository is checked out + and before the build process begins. This allows you to perform preparatory + tasks such as: + + - Creating directories + - Copying files + - Running custom setup scripts + - Installing dependencies + + Single command example: + + ```toml + pre_build_hook = "echo 'Preparing build environment'" + ``` + + Multi-line commands example: + + ```toml + pre_build_hook = """ + mkdir -p ../hello/vyos + mkdir -p ../vyos + cp example.txt ../vyos + """ + ``` + + Combined commands and scripts: + + ```toml + pre_build_hook = "ls -l; ./script.sh" + ``` + +**apply_patches** (Optional) + +: Boolean flag to control whether patches should be applied. Defaults to + `True`. + + ```toml + apply_patches = false + ``` + +**prepare_package** (Optional) + +: Boolean flag to enable package preparation. When set to `True`, the + `install_data` configuration is used. + +**install_data** (Optional) + +: Data used for package preparation when `prepare_package` is enabled. + +## Example package.toml file + +Here's an example configuration for the FRRouting (FRR) package: +```toml +name = "frr" +commit_id = "stable/10.5" +scm_url = "https://github.com/FRRouting/frr.git" +build_cmd = "sudo dpkg -i ../*.deb; dpkg-buildpackage -us -uc -tc -b -Ppkg.frr.rtrlib,pkg.frr.lua" +``` + +## Build Output + +After running `./build.py`, the following artifacts are generated in the +package directory: + +- `.deb` files - Binary Debian packages ready for installation +- `.tar.gz` files - Source tarballs of the checked-out repositories +- Additional build artifacts as produced by the Debian build system + +The build script also creates build dependency packages (`*build-deps*.deb`), +which are automatically cleaned up after the build completes. |