| Age | Commit message (Collapse) | Author |
|---|
|
As part of IN_PLACE testing, /etc/cloud/cloud.cfg.d get overwritten by
what's in the source tree. This can cause problems when the directory is
mounted in, because tests need the ability to modify files in
/etc/cloud. Attempting to 'lxc file push' instead will fail on LXD VMs
because the LXD agent isn't available yet.
If such functionality is desired, one can temporarily use the
'lxd_setup' mark while writing a test and push the files manually.
|
|
* Wrap the log fetching code in a try/except in case file is missing
* Stop checking NoCloud seed dir when testing datasource detection
|
|
Since lxc bind mounts will be read-only as nobody:nogroup
we don't want to bind mount /etc/cloud/cloud.cfg.d into the
instance because some tests add artifacts to /etc/cloud/cloud.cfg.d.
Also make LXD push_file pull_file methods assert that the
file transfer was a success, otherwise we miss the root-cause
for test failures.
This resulted in failed Jenkins runs in test_lxd_discovery with a
symptom of NoCloud being detected instead of LXD datasource.
The root-case was that instance.file_push failed due to permission
errors for root on the bind mounted /etc/cloud/cloud.cfg.d.
Also bump pycloudlib commitish to get Azure Jammy image support.
|
|
* Include CI and Fixtures sections in integration test docs
* Incorporate additional variable annotations
* Remove unnecessary IntegrationInstance subclasses
* Move setup_image teardown into its fixture
|
|
Migrate from legacy schema or define new schema in
cloud-init-schema.json, adding extensive schema tests for:
- cc_apt_configure
- cc_bootcmd
- cc_byobu
- cc_ca_certs
- cc_chef
- cc_debug
- cc_disable_ec2_metadata
- cc_disk_setup
Deprecate config hyphenated schema keys in favor of underscores:
- ca_certs and ca_certs.remove_defaults instead of
ca-certs and ca-certs.remove-defaults
- Continue to honor deprecated config keys but emit DEPRECATION
warnings in logs for continued use of the deprecated keys:
- apt_sources key
- any apt v1 or v2 keys
- use or ca-certs or ca_certs.remove-defaults
- Extend apt_configure schema
- Define more strict schema below object opaque keys using
patternProperties
- create common $def apt_configure.mirror for reuse in 'primary'
and 'security' schema definitions within cc_apt_configure
Co-Authored-by: James Falcon <james.falcon@canonical.com>
|
|
Refactor _report_ready_if_needed() to work for both Savable PPS
and Runnable PPS:
* rename _report_ready_if_needed() to _report_ready_for_pps()
* return interface name from lease to support _poll_imds() behavior
without changing it.
* fixes an issue where reporting ready return value was silently
ignored for Savable PPS.
* add explicit handling for failure to obtain DHCP lease to
result in sources.InvalidMetaDataException.
Refactor _poll_imds():
* use _report_ready_for_pps() for reporting ready, removing this logic
to simplify loop logic.
* move netlink and vnetswitch out of while loop to simplify loop logic,
leaving only reprovision polling in loop.
* add explicit handling for failure to obtain DHCP lease and
retry in the next iteration.
Signed-off-by: Chris Patterson cpatterson@microsoft.com
|
|
Consolidate _should_reprovision_after_nic_attach() with
_should_reprovision() into the following:
_write_reprovision_marker() to write provisioning marker for
reboot-during-provisioning case.
PPSType enum and _determine_pps_type() for determining which to
provisioning mode, if any, we're running under.
PPSType.UNKNOWN is when the reprovisioning marker is found and we
do not have the context to know what the original mode was. In this
scenario, we must resort to polling for reprovision data.
Tests:
Introduce a simple data source fixture to for fine-grain
control of mocking with pytest without unittest.
Migrate relevant _should_reprovision() tests into a combination of
TestDeterminePPSTypeScenarios cases.
Signed-off-by: Chris Patterson cpatterson@microsoft.com
|
|
When obtaining information from "ip addr", default to using
"ip --json addr" rather than using regex to parse "ip addr show"
as json is machine readable as less prone to error.
Deprecate but leave fallback to use "ip addr" for older iproute2
tooling which does not support --json param.
Fix regex parsing of "ip addr" to support peer addresses and
metrics.
|
|
According to the documentation in the tests:
```
We expect 3 calls to report_failure_to_fabric,
because we try 3 different methods of calling report failure.
The different methods are attempted in the following order:
1. Using cached ephemeral dhcp context to report failure to Azure
2. Using new ephemeral dhcp to report failure to Azure
3. Using fallback lease to report failure to Azure
```
Case 1 and 2 make sense. If networking is established, use it.
Should failure occur using current network configuration, retry
with fresh DHCP.
Case 3 suggests that we can fall back to a lease file and retry.
Given that:
1. The wireserver address has never changed to date.
2. The wireserver address should be in the DHCP lease.
3. Parsing the lease file does not improve connectivity over the
prior attempts.
...we can safely remove this case without regression.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
|
|
load_azure_ds_dir() always returns a tuple. Instead of saving this
tuple as ret, expand it immediately as md, userdata_raw, cfg, files.
This allows for more fine-grained passing of data before getting
expanded later.
- Update _should_reprovision methods to use cfg instead of tuple.
- Update _should_reprovision methods to remove the ovf_md guard.
This should be a safe refactor as the OVF is not required, and the
config is initialized to an empty dict. In practice, a mount failure
would have initialized ret anyways if the OVF was not found. If a
mount failure wasn't seen and ret was None, this guard could be
causing other failures by ignoring the PPS state that should be
available from IMDS metadata.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Avoid series-specific log formatting in tests by using
caplog.record_tuples.
Added benefit, we can easily check log-level WARNING too.
Fixes unit tests from Focal Package builds at:
https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/\
+build/23076508
|
|
Dailies are always available for stable releases and the devel release.
Moreover testing against dailies will warn us earlier about issues.
|
|
snapd currrently looks for 'not run' from cloud-init status[1].
Avoid changing this behavior and revert "not-run" value to
"not run".
This avoids having to get snapd to change implementation and
release updates as far back as Bionic to handle a hyphenated
not-run string.
[1]: https://github.com/snapcore/snapd/blob/master/sysconfig/\
cloudinit.go#L802
|
|
The if-statement set ovf_is_accessible to True if the OVF is read
from /dev/sr0, but not from other data sources. It defaults to
True, but may get flipped to False while processing an invalid
source, and never get set back to True when reading from the data
directory.
Instead, default ovf_is_accessible to False, and only set it to
True once we've read an OVF successfully (and end the search).
This fixes an error when OVF is read from data_dir and IMDS
data is unavailable (failing with "No OVF or IMDS available").
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
- Update EphemeralDHCPv4WithReporting to subclass EphemeralDHCPv4 for
consistency (non-functional change).
- Replace all usage of EphemeralDHCPv4 with EphemeralDHCPv4WithReporting.
- Converging to one DHCP class exposed an issue with ExitStack patches
being mixed with decorators. Specifically, it appeared that tests
that did not enable azure.EphemeralDHCPv4WithReporting mocks had it
applied anyways from previous tests.
Presumably ExitStack was overwriting the actual value with the
mock provided by the decorator? For now, remove some mock patches
that trigger failures, but future work should move towards a
consistent approach to prevent undetected effects.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.
Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.
This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.
The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.
Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:
- get_schema: source base schema file from cloud-init-schema.json
and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param
Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI
|
|
https://pycloudlib.readthedocs.io/en/latest/configuration.html#configuration
|
|
In 2c52e6e88b19f5db8d55eb7280ee27703e05d75f, the order of
reading network config was changed for Oracle due to initramfs
needing to take lower precedence than the datasource. However,
this also bumped system_cfg to a lower precedence than ds, which
means that any network configuration specified in /etc/cloud will not
be applied. system_cfg should instead be moved above ds so network
configuration in /etc/cloud takes precedence.
LP: #1956788
|
|
Reduce template rendering test runtime
|
|
This fix has two elements:
- cloud-init status will not correctly report 'not-run' prior to systemd
generator running. Only report "disabled" when generator has run
and /run/cloud-init/disabled exists.
- Expose not-run and disabled state in cloud-id responses
- Add unique error codes from cloud-id for error, disabled and not-run.
The new cloud-id exit codes:
0: success
1: error
2: cloud-init is in disabled state
3: cloud-init generator has not run yet
|
|
distutils is getting deprecated soon. Let's replace it with suggested
alternatives as suggested in:
https://www.python.org/dev/peps/pep-0632/
Remove `requests` version check and related code from url_helper.py
as the versions specified are old enough to no longer be relevant.
Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
|
|
Adds a new module to allow setting keyboard layout,
for use-cases in which cloud-init is used to configure
OS images meant for physical computers instead
of the cloud.
This initial release only implements support
for Linux distributions that allow layout to be
set through systemd's localectl.
LP: #1951593
|
|
Format tweak to match naming conventions for classes & enums.
No functional changes.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Includes:
- Update tox.ini and .travis.yml accordingly
- Cleanup tox.ini with new tox syntax and cloud-init dependencies
- Update documentation accordingly
- Replace/remove xenial references where additional testing isn't required
- Remove xenial checks in integration tests
- Replace yield_fixture with fixture in pytest tests
Sections of code commented with lines like "Remove when Xenial is no
longer supported" still exist as they're require additional testing.
|
|
On LXD containers /dev/lxd/sock will always exist.
Mock dscheck_LXD to return 1 (NOT_FOUND) to avoid leaking into
test environment and returning LXD as detected.
We have integration tests covering proper LXD datasource detection
so we don't need a Unit test validating the [ -S /dev/lxd/sock] that
is in ds-identify.
|
|
Warn during boot when an empty config is provided. Likewise,
`cloud-init devel schema --annotate` should not throw exception, return
something meaningful instead.
|
|
If get_imds_data_with_api_fallback() falls back to the minimum required
API version, it is effectively pinned to the old API version forever.
Remove the failed_desired_api_version property to prevent persistence of
the flag between calls and/or reboots.
The continued presence of this flag in obj.pkl should be harmless.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
cloud-init currently makes calls to ubuntu_advantage without assume-yes.
some ua enable commands, such as ua enable fips, have prompts. In an
automated environment, calling ua enable without --assume-yes will
result in errors and not applying the change. This sets --assume-yes by
default for all enable commands. This capability was added two years ago
in ua commit 576e605ceb5f so should be safe for use in all systems at
this time.
LP: #1954842
|
|
Applied Black and isort, fixed any linting issues, updated tox.ini
and CI.
|
|
test_handle_args_error_on_invalid_vaname_paths() would fail when run
as root due to invocation of load_userdata() on the default user/vendor
data locations under the instance link.
- Mock load_userdata() for this test case to avoid loads.
- Update _setup_paths() to configure cloud_dir in temporary location.
- Add new test case to verify that the default locations are loaded
when unspecified.
LP: #1825027
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
* Use proper logging
* Add parsing for patternProperties
* Add label to annotate patternProperties
* Log warning if schema parsing fails during metaschema processing
* Some schema test fixes
|
|
`sysctl -n hw.disknames` returns a trailing `\n`. We need to clean
this up. In addition, the criteria matching system is a source of
problem because:
- we don't have a way to look up the label of the partition
- we've got situation where an ISO image can be exposed through a virtio
block device.
So we just totally ignore the value of `criteria`. We end-up with a
slightly longer loop of mount-retry. But this way we're sure we don't
miss a configuration disk.
Tested on Kubvirt with the help of Brady Pratt @jbpratt.
|
|
Thanks to [1], the hostname is set prior to network bring-up.
The Azure data source has been bouncing the hostname during
setup(), occurring after the hostname has already been
properly configured.
Note that this doesn't prevent leaking the image's hostname
during Azure's _get_data() when it brings up ephemeral DHCP.
However, as are not guaranteed to have the hostname metadata
available from a truly "local" source, this behavior is to
be expected unless we disable `send host-name` from dhclient
config.
[1]: https://github.com/canonical/cloud-init/commit/133ad2cb327ad17b7b81319fac8f9f14577c04df
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
When the metadata is an ISO image and is exposed through a disk,
the device is called `/dev/sd?a` internally. For instance `/dev/sd1a`.
It can then be mounted with `mount_cd9660 /dev/sd1a /mnt`.
Metadata in the FAT32 format are exposed as `/dev/sd?i`.
With this change, we try to mount `/dev/sd?a` in addition to `/dev/sd?i`.
Closes: https://github.com/ContainerCraft/kmi/issues/12
|
|
Add a test helper to get top level directory
Many tests need to get the location of files & dirs within the cloud-init
project directory.
Tests implement this in various different ways, and often those ways
depend on the current working directory of the pytest invocation.
Create helper functions (and tests) that gets the path of the top
directory or any sub directory under the top directory. This function
does not depend on the environment.
|
|
Parametrized pytest tests get named by on their parameters. If a name
has random characters, it can break the test collection of when
using pytest-xdist. Replace random name with deterministic name.
|
|
|
|
|
|
Improve schema validation.
This adds strict validation of config module definitions at testing
time, with plumbing included for future runtime validation. This
eliminates a class of bugs resulting from schemas that have definitions
that are incorrect, but get interpreted by jsonschema as
"additionalProperties" that are therefore ignored.
- Add strict meta-schema for jsonschema unit test validation
- Separate schema from module metadata structure
- Improve type annotations for various functions and data types
Cleanup:
- Remove unused jsonschema "required" elements
- Eliminate manual memoization in schema.py:get_schema(),
reference module.__doc__ directly
|
|
If we set a dhcp server side like this:
$ cat /var/tmp/cloud-init/cloud-init-dhcp-f0rie5tm/dhcp.leases
lease {
...
option classless-static-routes 31.169.254.169.254 0.0.0.0,31.169.254.169.254
10.112.143.127,22.10.112.140 0.0.0.0,0 10.112.140.1;
...
}
cloud-init fails to configure the routes via 'ip route add' because to there are
two different routes for 169.254.169.254:
$ ip -4 route add 192.168.1.1/32 via 0.0.0.0 dev eth0
$ ip -4 route add 192.168.1.1/32 via 10.112.140.248 dev eth0
But NetworkManager can handle such scenario successfully as it uses "ip route append".
So change cloud-init to also use "ip route append" to fix the issue:
$ ip -4 route append 192.168.1.1/32 via 0.0.0.0 dev eth0
$ ip -4 route append 192.168.1.1/32 via 10.112.140.248 dev eth0
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
RHBZ: #2003231
|
|
This attempts to standardize unit test file location under test/unittests/
such that any source file located at cloudinit/path/to/file.py may have a
corresponding unit test file at test/unittests/path/to/test_file.py.
Noteworthy Comments:
====================
Four different duplicate test files existed:
test_{gpg,util,cc_mounts,cc_resolv_conf}.py
Each of these duplicate file pairs has been merged together. This is a
break in git history for these files.
The test suite appears to have a dependency on test order. Changing test
order causes some tests to fail. This should be rectified, but for now
some tests have been modified in
tests/unittests/config/test_set_passwords.py.
A helper class name starts with "Test" which causes pytest to try
executing it as a test case, which then throws warnings "due to Class
having __init__()". Silence by changing the name of the class.
# helpers.py is imported in many test files, import paths change
cloudinit/tests/helpers.py -> tests/unittests/helpers.py
# Move directories:
cloudinit/distros/tests -> tests/unittests/distros
cloudinit/cmd/devel/tests -> tests/unittests/cmd/devel
cloudinit/cmd/tests -> tests/unittests/cmd/
cloudinit/sources/helpers/tests -> tests/unittests/sources/helpers
cloudinit/sources/tests -> tests/unittests/sources
cloudinit/net/tests -> tests/unittests/net
cloudinit/config/tests -> tests/unittests/config
cloudinit/analyze/tests/ -> tests/unittests/analyze/
# Standardize tests already in tests/unittests/
test_datasource -> sources
test_distros -> distros
test_vmware -> sources/vmware
test_handler -> config # this contains cloudconfig module tests
test_runs -> runs
|
|
Given that there are additional network management tools that we haven't
yet supported with activators, we should log a warning and continue
without network activation here, especially since this was a no-op for
years.
LP: #1948681
|
|
(#1123)
Allow #cloud-config and cloud-init query to use underscore-delimited
"jinja-safe" key aliases for any instance-data.json keys
containing jinja operator characters.
This provides a means to use Jinja's dot-notation instead of square brackets
and quoting to reference "unsafe" obtain attribute names.
Support for these aliased keys is available to both #cloud-config user-data and
`cloud-init query`.
For example #cloud-config alias access can look like:
{{ ds.config.user_network_config }}
- instead of -
{{ ds.config["user.network-config"] }}
|
|
|
|
GCE currently fetches metadata after network has come up. There's no
reason we can't fetch at init-local time, so update GCE to fetch at
init-local time to be more performant and consistent with other
datasources.
|
|
When cloud-init is configured to show SSH user key fingerprints during
boot two of the same message appears for each user. This appears to be as
the util.multi_log call defaults to send to both console directly and to
stderr (which also goes to console).
This change sends them only to console directly.
|
|
Some references were missed in the removal of the agent command
in PR #799. This simply removes the remaining references.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
testing: monkeypatch system_info call in unit tests
system_info can make calls that read or write from the filesystem, which
should require special mocking. It is also decorated with 'lru_cache',
which means test authors often don't realize they need to be mocking.
Also, we don't actually want the results from the user's local
machine, so monkeypatching it across all tests should be reasonable.
Additionally, moved some of 'system_info` into a helper function to
reduce the surface area of the monkeypatch, added tests for the new
function (and fixed a bug as a result), and removed related mocks that
should be no longer needed.
|
|
Also simplify a path and fix a spelling error while in the file
|
