diff options
| -rw-r--r-- | .env.example | 16 | ||||
| -rw-r--r-- | CHANGELOG.md | 109 | ||||
| -rw-r--r-- | CONTRIBUTING.md | 71 | ||||
| -rw-r--r-- | LICENSE | 2 | ||||
| -rw-r--r-- | README.md | 272 | ||||
| -rw-r--r-- | docs/getting-started.md | 4 | ||||
| -rw-r--r-- | docs/index.md | 4 |
7 files changed, 342 insertions, 136 deletions
diff --git a/.env.example b/.env.example index 328f4c0..d3a2448 100644 --- a/.env.example +++ b/.env.example @@ -1,5 +1,19 @@ +# pyvyos device configuration. +# Copy this file to .env and fill in the values. + +# Hostname or IP address of the VyOS device. VYDEVICE_HOSTNAME= + +# API key configured on the device with: +# set service https api keys id <name> key <secret> VYDEVICE_APIKEY= + +# HTTPS port of the VyOS API (default: 443). VYDEVICE_PORT=443 + +# Protocol: "https" (recommended) or "http". VYDEVICE_PROTOCOL=https -VYDEVICE_VERIFY_SSL=False
\ No newline at end of file + +# Verify the TLS certificate of the device. +# Set to "false" ONLY in lab environments using self-signed certificates. +VYDEVICE_VERIFY_SSL=true diff --git a/CHANGELOG.md b/CHANGELOG.md index 15b9314..3132e02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,50 +5,97 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.4.0] - 2025-01-XX +## [0.4.0] - 2025-11-20 + +`0.4.0` is a cleanup and consolidation release. It is the first version +since `0.2.2` published to PyPI. The git tag `v0.4.0` previously pointed at +the intermediate refactor commit `6b4e901`; it has been moved to the +release commit that includes the cleanup described below. + +This release **does not change HTTP payload generation or request/response +behavior**. ### Added -- Exception hierarchy (`SDKError`, `HttpError`, `ApiError`, `ValidationError`) in `pyvyos.exceptions` -- Utility functions in `pyvyos.utils`: - - `json.redact_key()` and `json.safe_dumps()` for secure JSON handling - - `ids.request_id()` for request tracing - - `paths.build_path()` for building configuration paths -- Structured logging in `RestClient` with request ID tracking and elapsed time -- Optional Pydantic validation models in `pyvyos.specs.commands.*` for request/response validation -- Development documentation: - - Architecture guide (`docs/development/architecture.md`) - - Refactor roadmap (`docs/development/refactor-roadmap.md`) - - Quality and utils guidelines (`docs/development/quality-and-utils.md`) -- Comprehensive test suite for backward compatibility (19 tests for shims, 16 tests for utils, 6 tests for exceptions) -- `[tool.uv] package = true` in `pyproject.toml` for editable installation via `uv sync` +- `pyvyos/py.typed` PEP 561 marker, advertising the package as typed. +- `.pre-commit-config.yaml` with neutral hooks (trailing whitespace, + end-of-file fixer, YAML/TOML syntax check, large-file guard). +- GitHub Actions ecosystem entry in `.github/dependabot.yml` so workflow + versions stay current. +- Documented public API stability and deprecation timeline in `README.md`. ### Changed -- Moved JSON API specifications from `pyvyos/vyos-api/` to `docs/development/vyos_api/` (reference only) -- Updated `pyproject.toml` to include optional `validation` dependency group for Pydantic -- Enhanced `RestClient` logging with structured fields (request_id, command, op, status, elapsed_ms) -- Refactored internal structure to `pyvyos.core.*` while maintaining full backward compatibility via shims +- `pyvyos.core.*` is now the internal implementation layer; the supported + public API is `from pyvyos import VyDevice, ApiResponse`. The legacy + `pyvyos.device` and `pyvyos.rest` shims continue to work without + warnings. +- `RestClient` logging is now plain and structured: command, op, status, + elapsed milliseconds. No request-ID field. +- `pyproject.toml`: + - declared the wheel package explicitly, + - removed duplicated dependency blocks, + - dropped the unused `validation` extra, + - raised the pytest floor for Python 3.13. +- PR validation workflow upgraded to `actions/checkout@v4`, + `actions/setup-python@v5`, and `astral-sh/setup-uv@v3`. +- `.env.example` default flipped to `VYDEVICE_VERIFY_SSL=true`; commented + with field descriptions. + +### Removed +- `pyvyos.specs` package (experimental Pydantic models). It was never + imported by the runtime and had 0% test coverage. Pydantic is no longer + an optional dependency. +- `pyvyos.exceptions` module (`SDKError`, `HttpError`, `ApiError`, + `ValidationError`). The hierarchy was defined but never raised anywhere + in the codebase. Error reporting continues through `ApiResponse.error`. +- `pyvyos.utils.ids.request_id` helper. The generated UUIDs were attached + to log records but never propagated to callers — half-implemented + tracing is worse than none. It will be reintroduced if and when real + observability hooks land. +- `pyvyos.utils` no longer re-exports `request_id`. +- Obsolete tooling: `Makefile` (hard-coded `env/bin/python`), + `run_tests.sh`, `run_tests.py`. +- Dead workflow `.github/workflows/python-app.yml` (Python 3.12, only ran + flake8 with pytest commented out, referenced a non-existent + `requirements.txt`). +- `sphinx/` source tree and `.readthedocs.yaml`: the RTD configuration + pointed to `docs/source/conf.py` while the Sphinx tree lived under + `sphinx/source/`, so the build never worked and no documentation was + ever published. The hand-written Markdown docs under `docs/` are + retained. +- Stale development notes under `docs/development/` (architecture, + refactor roadmap, quality-and-utils) — they described the pre-cleanup + proposal that included specs/exceptions/request_id. +- Tests for the removed modules (`tests/test_exceptions.py`, + `tests/utils/test_ids.py`). ### Fixed -- **Fixed #25**: `config_file_save()` and `config_file_load()` now correctly include `path: []` in payload as required by VyOS API -- Path parameter handling for `config-file` commands (always includes `path: []`) -- Secret redaction in logs and sanitized payloads +- `LICENSE` copyright now reads `2023 GravScale, Roberto Bertó`. -### Security -- API keys are automatically redacted in logs and response payloads +### Compatibility +- Public imports are unchanged: + - `from pyvyos import VyDevice, ApiResponse` + - `from pyvyos.device import VyDevice` + - `from pyvyos.rest import RestClient, ApiResponse` +- HTTP payload generation is unchanged. +- Request and response behaviour is unchanged. +- No compatibility shim has been deprecated in this release. ### Notes -- This release maintains 100% backward compatibility with version 0.3.0 -- All existing imports and code will continue to work without changes -- New internal structure (`pyvyos.core.*`) is available but not required for existing code +- This release prepares the project for the upcoming `0.5.x` work: + contract tests for public payloads, fixes to the public method edges + (`image_add`/`image_delete`, `timeout`, mutable defaults), stdlib-based + validators, and a tidier internal core. ## [0.3.0] - 2024-XX-XX +Tagged in git but never published to PyPI. Released to PyPI as part of +`0.4.0`. + ### Added -- Initial release -- Core functionality for VyOS REST API interaction -- Support for configure, retrieve, show, generate, reset, config-file, reboot, poweroff, and image operations +- Initial public release of the SDK structure. +- Core functionality for VyOS HTTPS API: configure, retrieve, show, + generate, reset, config-file, reboot, poweroff, and image operations. [Unreleased]: https://github.com/vyos-contrib/pyvyos/compare/v0.4.0...HEAD -[0.4.0]: https://github.com/vyos-contrib/pyvyos/compare/v0.3.0...v0.4.0 +[0.4.0]: https://github.com/vyos-contrib/pyvyos/compare/v0.2.2...v0.4.0 [0.3.0]: https://github.com/vyos-contrib/pyvyos/releases/tag/v0.3.0 - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..2bfaaa0 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,71 @@ +# Contributing to pyvyos + +Thanks for considering a contribution. `pyvyos` is a small library with a +narrow scope, which makes review and maintenance easier when contributions +follow a few simple rules. + +## Scope + +`pyvyos` is a thin Python wrapper around the VyOS HTTPS API. Changes that +fit naturally into this scope are welcome. Anything that adds a new +dependency, introduces a parallel client (SSH, NETCONF, …), or expands +beyond the HTTPS API surface is best discussed in an issue first. + +## Before opening a pull request + +1. **Open an issue first** for anything beyond a small fix or a typo. It + saves everyone time if the direction is agreed before the patch lands. +2. **Keep the diff focused.** One topic per pull request. Refactors and + bug fixes are easier to review separately. +3. **Do not change HTTP payload generation in passing.** Payload changes + require a deliberate review pass; they should be their own pull + request and include tests. +4. **Run the test suite locally:** `uv run pytest`. +5. **Match the existing style.** No formatter is enforced yet; just keep + diffs minimal and readable. + +## Development setup + +```bash +git clone https://github.com/vyos-contrib/pyvyos.git +cd pyvyos +uv sync --extra dev +uv run pytest +``` + +Optional pre-commit hooks (whitespace, EOF, YAML/TOML syntax): + +```bash +pip install pre-commit +pre-commit install +``` + +## Public API stability + +The supported public API is: + +```python +from pyvyos import VyDevice, ApiResponse +``` + +Compatibility shims at `pyvyos.device` and `pyvyos.rest` are kept while +the cost is trivial. Anything under `pyvyos.core.*` is internal and may +change between minor releases. + +If your change touches the public API surface, please flag it explicitly +in the pull request description. + +## Reporting bugs + +When reporting a bug, please include: + +- The `pyvyos` version (`pip show pyvyos`). +- The Python version. +- The VyOS version of the device. +- A minimal reproducer and the resulting error (or, for behaviour bugs, + the observed vs expected behaviour). + +## License + +By contributing you agree that your contribution will be licensed under +the [MIT License](LICENSE). @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2023 GravScale +Copyright (c) 2023 GravScale, Roberto Bertó Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -1,149 +1,225 @@ -# pyvyos Documentation +# pyvyos -pyvyos is a Python library for interacting with VyOS devices via their API. This documentation provides a guide on how to use pyvyos to manage your VyOS devices programmatically. +[](https://pypi.org/project/pyvyos/) +[](https://pypi.org/project/pyvyos/) +[](LICENSE) +[](https://github.com/vyos-contrib/pyvyos/actions/workflows/python-pr-validation.yml) -You can find the complete pyvyos documentation on [Read the Docs](https://pyvyos.readthedocs.io/en/latest/). +Python SDK for the [VyOS](https://vyos.io/) HTTPS API. -## Installation +`pyvyos` is a small, focused library that wraps the VyOS HTTPS API in an +idiomatic Python interface. It is intended for automation scripts, internal +tooling, and integrations with configuration management systems. -You can install pyvyos using pip https://pypi.org/project/pyvyos/: +## Installation ```bash pip install pyvyos ``` -## Getting Started +Requires **Python 3.13 or newer**. -### Importing and Disabling Warnings for verify=False -Before using pyvyos, it's a good practice to disable urllib3 warnings and import the required modules, IF you use verify=False: - -``` -import urllib3 -urllib3.disable_warnings() -``` +## Quick start -### Using API Response Class -pyvyos uses a custom ApiResponse data class to handle API responses: +Enable the HTTPS API on the VyOS device and create an API key: -``` -@dataclass -class ApiResponse: - status: int - request: dict - result: dict - error: str +```text +set service https api keys id my-key key 'your-secret-key' +commit ``` -### Initializing a VyDevice Object +Then, from Python: +```python +import os +from pyvyos import VyDevice -#### Configuring Your Environment for VyDevice -1. Rename the file .env.example to .env. -1. Open the .env file in a text editor. -1. Replace the placeholder values with your VyOS device credentials: - - **VYDEVICE_HOSTNAME**: Your device's hostname or IP address. - - **VYDEVICE_APIKEY**: Your API key for authentication. - - **VYDEVICE_PORT**: The port number for the API. Default 443 - - **VYDEVICE_PROTOCOL**: The protocol (e.g., http or https). Default https - - **VYDEVICE_VERIFY_SSL**: Set to True or False for SSL verification. +device = VyDevice( + hostname=os.environ["VYDEVICE_HOSTNAME"], + apikey=os.environ["VYDEVICE_APIKEY"], + port=int(os.environ.get("VYDEVICE_PORT", "443")), + protocol=os.environ.get("VYDEVICE_PROTOCOL", "https"), + verify=os.environ.get("VYDEVICE_VERIFY_SSL", "true").lower() == "true", +) +response = device.show(path=["system", "image"]) +if response.error: + print(f"Error {response.status}: {response.error}") +else: + print(response.result) +``` +If you use self-signed certificates, set `verify=False` **only in lab +environments** and silence the urllib3 warning explicitly: + +```python +import urllib3 +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) ``` -# Retrieve VyOS device connection details from environment variables and configure VyDevice -from dotenv import load_dotenv -load_dotenv() -hostname = os.getenv('VYDEVICE_HOSTNAME') -apikey = os.getenv('VYDEVICE_APIKEY') -port = os.getenv('VYDEVICE_PORT') -protocol = os.getenv('VYDEVICE_PROTOCOL') -verify_ssl = os.getenv('VYDEVICE_VERIFY_SSL') +## Environment variables -# Convert the verify_ssl value to a boolean -verify = verify_ssl.lower() == "true" if verify_ssl else True +A `.env.example` is shipped with the project. The recognised variables are: -device = VyDevice(hostname=hostname, apikey=apikey, port=port, protocol=protocol, verify=verify) -``` +| Variable | Default | Purpose | +| ---------------------- | ------- | -------------------------------------------------- | +| `VYDEVICE_HOSTNAME` | — | Hostname or IP address of the VyOS device. | +| `VYDEVICE_APIKEY` | — | API key configured on the device. | +| `VYDEVICE_PORT` | `443` | HTTPS port of the VyOS API. | +| `VYDEVICE_PROTOCOL` | `https` | `https` (recommended) or `http`. | +| `VYDEVICE_VERIFY_SSL` | `true` | Verify the TLS certificate of the device. | -## Using pyvyos +`pyvyos` does not read these variables on its own — your application is +responsible for loading them (for example with `python-dotenv`) and passing +the values to `VyDevice`. -### configure, then set -The configure_set method sets a VyOS configuration: +## API overview -``` -# Set a VyOS configuration -response = device.configure_set(path=["interfaces", "ethernet", "eth0", "address", "192.168.1.1/24"]) +All methods return an `ApiResponse` dataclass with four fields: -# Check for errors and print the result -if not response.error: - print(response.result) -``` -### configure, then show a single OBJECT value -``` -# Retrieve VyOS return values for a specific interface -response = device.retrieve_return_values(path=["interfaces", "dummy", "dum1", "address"]) -print(response.result) +```python +@dataclass +class ApiResponse: + status: int # HTTP status code + request: dict # the request payload (API key removed) + result: dict | list # parsed `data` field from the response + error: str | bool # error message, or False on success ``` -### configure, then show OBJECT -The retrieve_show_config method retrieves the VyOS configuration: +The recommended usage pattern is: +```python +response = device.retrieve_show_config(path=["interfaces"]) +if response.error: + raise RuntimeError(response.error) +do_something_with(response.result) ``` -# Retrieve the VyOS configuration -response = device.retrieve_show_config(path=[]) -# Check for errors and print the result -if not response.error: - print(response.result) -``` +### Configuration -### configure, then delete OBJECT -``` -# Delete a VyOS interface configuration -response = device.configure_delete(path=["interfaces", "dummy", "dum1"]) +```python +device.configure_set(path=["interfaces", "ethernet", "eth0", "address", "192.0.2.1/24"]) +device.configure_delete(path=["interfaces", "dummy", "dum1"]) +device.configure_multiple_op(path=[ + {"op": "set", "path": ["interfaces", "dummy", "dum2", "address", "203.0.113.1/24"]}, + {"op": "delete", "path": ["interfaces", "dummy", "dum1"]}, +]) ``` -### configure, then save -``` -# Save VyOS configuration without specifying a file (default location) -response = device.config_file_save() -``` +### Retrieval -### configure, then save FILE +```python +device.retrieve_show_config(path=["system"]) +device.retrieve_return_values(path=["interfaces", "dummy", "dum1", "address"]) ``` -# Save VyOS configuration to a specific file -response = device.config_file_save(file="/config/test300.config") + +### Operational + +```python +device.show(path=["system", "image"]) +device.generate(path=["ssh", "client-key", "/tmp/key"]) +device.reset(path=["conntrack-sync", "internal-cache"]) ``` -## show OBJECT +### Configuration files + +```python +device.config_file_save() # default location +device.config_file_save(file="/config/backup.config") +device.config_file_load(file="/config/backup.config") ``` -# Show VyOS system image information -response = device.show(path=["system", "image"]) -print(response.result) + +### System control + +```python +device.reboot() # equivalent to device.reboot(path=["now"]) +device.poweroff() # equivalent to device.poweroff(path=["now"]) ``` -### generate OBJECT +### Image management + +```python +device.image_add(url="https://downloads.vyos.io/.../vyos-1.4-image.iso") +device.image_delete(name="1.4-rolling-...") ``` -# Generate an SSH key with a random string in the name -randstring = ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(20)) -keyrand = f'/tmp/key_{randstring}' -response = device.generate(path=["ssh", "client-key", keyrand]) + +## Public API stability + +The supported public API of pyvyos is: + +```python +from pyvyos import VyDevice, ApiResponse ``` -### reset OBJECT -The reset method allows you to run a reset command: +These compatibility imports continue to work without warnings and will be +kept while the migration cost remains trivial: +```python +from pyvyos.device import VyDevice +from pyvyos.rest import RestClient, ApiResponse ``` -# Execute the reset command -response = device.reset(path=["conntrack-sync", "internal-cache"]) -# Check for errors and print the result -if not response.error: - print(response.result) +Anything under `pyvyos.core.*` is internal implementation detail and may +change between minor releases. + +The deprecation timeline is: + +| Release | Status | +| ------- | ----------------------------------------------------------------- | +| `0.4.x` | Compatibility shims work without warnings. | +| `0.5.x` | Internal solidity work; shims still silent. | +| `0.6.x` | Compatibility shims emit a `DeprecationWarning`. | +| `1.0.0` | Shims are removed or kept, depending on observed usage. | + +## Logging + +`pyvyos` uses the standard `logging` module under the `pyvyos` namespace and +attaches a `NullHandler` so a default install does not print anything. + +To see request/response activity, configure the logger in your application: + +```python +import logging +logging.basicConfig(level=logging.INFO) +logging.getLogger("pyvyos").setLevel(logging.DEBUG) ``` -### configure, then load FILE +Request payloads are sanitised before logging — the `key` field is replaced +with `***REDACTED***`. + +## VyOS compatibility + +Tested against: + +- VyOS 1.4 LTS (stable) +- VyOS 1.5 rolling + +The library only depends on the HTTPS API surface, so versions that expose +the same endpoints should work without changes. + +## Development + +The project uses [uv](https://docs.astral.sh/uv/) for environment +management: + +```bash +uv sync --extra dev +uv run pytest ``` -# Load VyOS configuration from a specific file -response = device.config_file_load(file="/config/test300.config") + +Optional code-style hooks: + +```bash +pip install pre-commit +pre-commit install ``` + +## Contributing + +Bug reports and pull requests are welcome. Please open an issue first to +discuss anything beyond a small fix, and keep changes focused — payload and +public-API changes go through a separate review cycle. + +## License + +MIT — see [LICENSE](LICENSE). diff --git a/docs/getting-started.md b/docs/getting-started.md index 183c109..3cf4891 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -21,7 +21,7 @@ VYDEVICE_HOSTNAME=192.168.1.1 VYDEVICE_APIKEY=your-api-key-here VYDEVICE_PORT=443 VYDEVICE_PROTOCOL=https -VYDEVICE_VERIFY_SSL=False +VYDEVICE_VERIFY_SSL=true ``` ## Step 3: Create Your First Script @@ -54,7 +54,7 @@ If using `verify=False`, disable urllib3 warnings: ```python import urllib3 -urllib3.disable_warnings() +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) ``` ## Understanding API Responses diff --git a/docs/index.md b/docs/index.md index 1bc9cb7..7576f20 100644 --- a/docs/index.md +++ b/docs/index.md @@ -37,7 +37,7 @@ device = VyDevice( apikey="your-api-key", port=443, protocol="https", - verify=False + verify=True ) response = device.show(path=["system", "image"]) @@ -48,8 +48,6 @@ print(response.result) - [GitHub Repository](https://github.com/vyos-contrib/pyvyos) - [PyPI Package](https://pypi.org/project/pyvyos/) -- [Read the Docs](https://pyvyos.readthedocs.io/) - ## License MIT License - See LICENSE file for details |