| Age | Commit message (Collapse) | Author |
|---|
|
Once a valid datasource is detected, publish the following artifacts
to expedite cloud-identification without having to invoke cloud-id from
shell scripts or sheling out from python.
These files can also be relied on in systemd ConditionPathExists
directives to limit execution of services and units to specific
clouds.
/run/cloud-init/cloud-id:
- A symlink with content that is the canonical cloud-id of the
datasource detected. This content is the same lower-case value
as the output of /usr/bin/cloud-id.
/run/cloud-init/cloud-id-<canonical-cloud-id>:
- A single file which will contain the canonical cloud-id encoded
in the filename
|
|
|
|
Split _get_public_ssh_keys_and_source() into
_get_public_keys_from_imds() and _get_public_keys_from_ovf().
Set _get_public_keys_from_imds() to take a parameter of the
IMDS metadata rather than assuming it is already set in
self.metadata. This will allow us to move negotation into
local phase where self.metadata may not be set yet. Update this
method to raise KeyError if IMDS metadata is missing/malformed,
and ValueError if SSH key format is not supported. Update
get_public_ssh_keys() to catch these errors and fall back to the
OVF/Wireserver keys as needed.
To improve clarity, update register_with_azure_and_fetch_data()
to return the list of SSH keys, rather than bundling them into
a dictionary for updating against the metadata dictionary.
There should be no change in behavior with this refactor.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
|
|
When the datasource was originally submitted, EphemeralDHCPv4 was not
yet available. Also avoid race conditions by skipping network
configuration if metadata service can be reached.
Signed-off-by: Markus Schade <markus.schade@hetzner.com>
|
|
|
|
This change converts the IPv6 netmask from the network_data.json[1]
format to the CIDR style, <IPv6_addr>/<prefix>.
Using an IPv6 address like ffff:ffff:ffff:ffff:: does not work with
NetworkManager, nor networkscripts.
NetworkManager will ignore the route, logging:
ifcfg-rh: ignoring invalid route at \
"::/:: via fd00:fd00:fd00:2::fffe dev $DEV" \
(/etc/sysconfig/network-scripts/route6-$DEV:3): \
Argument for "::/::" is not ADDR/PREFIX format
Similarly if using networkscripts, ip route fail with error:
Error: inet6 prefix is expected rather than \
"fd00:fd00:fd00::/ffff:ffff:ffff:ffff::".
Also a bit of refactoring ...
cloudinit.net.sysconfig.Route.to_string:
* Move a couple of lines around to reduce repeated code.
* if "ADDRESS" not in key -> continute, so that the
code block following it can be de-indented.
cloudinit.net.network_state:
* Refactors the ipv4_mask_to_net_prefix, ipv6_mask_to_net_prefix
removes mask_to_net_prefix methods. Utilize ipaddress library to
do some of the heavy lifting.
LP: #1959148
|
|
Remove debug print that snuck in on a previous fixup.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Test pycloudlib's BaseInstance.pull_file doesn't return a Result
object. So we can't call ok() on the response in integration tests.
Leave the try/except handling as pull_file will raise an
IOError if there is an error connecting via paramiko's sftp.get.
|
|
Introduce:
- _setup_ephemeral_networking() to bring up networking.
If no iface is specified, it will use net.find_fallback_nic()
which is consistent with the previous usage of fallback_interface.
This method now tracks the encoded address of the wireserver
with a new property `_wireserver_endpoint`. Introduce a
timeout parameter to allow for retrying for a specified amount
of time.
- _teardown_ephemeral_networking() to bring down networking.
- _is_ephemeral_networking_up() to check status.
Ephemeral networking is now:
- Brought up prior to checking IMDS.
- Torn down following metadata crawl.
- For Savable PPS, torn down prior to waiting for NIC detach.
The link must be torn down in advance or we will see errors
from cleaning up network after the interface is unplugged.
- For Running PPS, torn down after waiting for media switch.
The link must be up for media switch to be detected.
- For all PPS, after network switch is complete, networking is
brought back up to poll for reprovision data and report ready.
It will be torn down after metadata crawl is complete like
non-PPS paths.
Additionally:
- Remove EphemeralDHCPv4WithReporting variant in favor of directly
using EphemeralDHCPv4. The reporting was only for __enter__ usage
which is no longer a used path. Continue to use dhcp_log_cb
callback.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
This fixes a bug that prevents the salt module from enabling the salt minion in rc.conf.
For more details:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254339
|
|
Currently _check_if_nic_is_primary() checks for imds_md is None,
but imds_md is returned as an empty dictionary on error fetching
metdata.
Fix this check and the tests that are incorrectly vetting IMDS
polling code.
Additionally, use response.contents instead of str(response) when
loding the JSON data returned from readurl. This helps simplify the
mocking and avoids an unncessary conversion.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
The ephemeral disk depends on a functional network to be mounted. Even
though it depends on cloud-init.service, sometimes an ordering cycle is
noticed on the instance. If the option "_netdev" is added the problem is
gone.
rhbz: #1998445
Signed-off-by: Eduardo Otubo otubo@redhat.com
|
|
As part of IN_PLACE testing, /etc/cloud/cloud.cfg.d get overwritten by
what's in the source tree. This can cause problems when the directory is
mounted in, because tests need the ability to modify files in
/etc/cloud. Attempting to 'lxc file push' instead will fail on LXD VMs
because the LXD agent isn't available yet.
If such functionality is desired, one can temporarily use the
'lxd_setup' mark while writing a test and push the files manually.
|
|
* Wrap the log fetching code in a try/except in case file is missing
* Stop checking NoCloud seed dir when testing datasource detection
|
|
|
|
|
|
Since lxc bind mounts will be read-only as nobody:nogroup
we don't want to bind mount /etc/cloud/cloud.cfg.d into the
instance because some tests add artifacts to /etc/cloud/cloud.cfg.d.
Also make LXD push_file pull_file methods assert that the
file transfer was a success, otherwise we miss the root-cause
for test failures.
This resulted in failed Jenkins runs in test_lxd_discovery with a
symptom of NoCloud being detected instead of LXD datasource.
The root-case was that instance.file_push failed due to permission
errors for root on the bind mounted /etc/cloud/cloud.cfg.d.
Also bump pycloudlib commitish to get Azure Jammy image support.
|
|
* Include CI and Fixtures sections in integration test docs
* Incorporate additional variable annotations
* Remove unnecessary IntegrationInstance subclasses
* Move setup_image teardown into its fixture
|
|
|
|
Fix spelling errors
- Add Makefile target that checks for spelling errors
- Add Makefile target that fixes spelling errors
- Add spelling check to travis doc tests
- Fix various spelling errors in the docs
|
|
|
|
Migrate from legacy schema or define new schema in
cloud-init-schema.json, adding extensive schema tests for:
- cc_apt_configure
- cc_bootcmd
- cc_byobu
- cc_ca_certs
- cc_chef
- cc_debug
- cc_disable_ec2_metadata
- cc_disk_setup
Deprecate config hyphenated schema keys in favor of underscores:
- ca_certs and ca_certs.remove_defaults instead of
ca-certs and ca-certs.remove-defaults
- Continue to honor deprecated config keys but emit DEPRECATION
warnings in logs for continued use of the deprecated keys:
- apt_sources key
- any apt v1 or v2 keys
- use or ca-certs or ca_certs.remove-defaults
- Extend apt_configure schema
- Define more strict schema below object opaque keys using
patternProperties
- create common $def apt_configure.mirror for reuse in 'primary'
and 'security' schema definitions within cc_apt_configure
Co-Authored-by: James Falcon <james.falcon@canonical.com>
|
|
LP: #1959118
|
|
|
|
Refactor _report_ready_if_needed() to work for both Savable PPS
and Runnable PPS:
* rename _report_ready_if_needed() to _report_ready_for_pps()
* return interface name from lease to support _poll_imds() behavior
without changing it.
* fixes an issue where reporting ready return value was silently
ignored for Savable PPS.
* add explicit handling for failure to obtain DHCP lease to
result in sources.InvalidMetaDataException.
Refactor _poll_imds():
* use _report_ready_for_pps() for reporting ready, removing this logic
to simplify loop logic.
* move netlink and vnetswitch out of while loop to simplify loop logic,
leaving only reprovision polling in loop.
* add explicit handling for failure to obtain DHCP lease and
retry in the next iteration.
Signed-off-by: Chris Patterson cpatterson@microsoft.com
|
|
The partner archive is now obsolete.
LP: #1959343
|
|
|
|
Consolidate _should_reprovision_after_nic_attach() with
_should_reprovision() into the following:
_write_reprovision_marker() to write provisioning marker for
reboot-during-provisioning case.
PPSType enum and _determine_pps_type() for determining which to
provisioning mode, if any, we're running under.
PPSType.UNKNOWN is when the reprovisioning marker is found and we
do not have the context to know what the original mode was. In this
scenario, we must resort to polling for reprovision data.
Tests:
Introduce a simple data source fixture to for fine-grain
control of mocking with pytest without unittest.
Migrate relevant _should_reprovision() tests into a combination of
TestDeterminePPSTypeScenarios cases.
Signed-off-by: Chris Patterson cpatterson@microsoft.com
|
|
|
|
|
|
LP: #1959149
|
|
When obtaining information from "ip addr", default to using
"ip --json addr" rather than using regex to parse "ip addr show"
as json is machine readable as less prone to error.
Deprecate but leave fallback to use "ip addr" for older iproute2
tooling which does not support --json param.
Fix regex parsing of "ip addr" to support peer addresses and
metrics.
|
|
Signed-off-by: Louis Sautier <sautier.louis@gmail.com>
|
|
|
|
According to the documentation in the tests:
```
We expect 3 calls to report_failure_to_fabric,
because we try 3 different methods of calling report failure.
The different methods are attempted in the following order:
1. Using cached ephemeral dhcp context to report failure to Azure
2. Using new ephemeral dhcp to report failure to Azure
3. Using fallback lease to report failure to Azure
```
Case 1 and 2 make sense. If networking is established, use it.
Should failure occur using current network configuration, retry
with fresh DHCP.
Case 3 suggests that we can fall back to a lease file and retry.
Given that:
1. The wireserver address has never changed to date.
2. The wireserver address should be in the DHCP lease.
3. Parsing the lease file does not improve connectivity over the
prior attempts.
...we can safely remove this case without regression.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Avoid requirement of getattr() and ensure _ephemeral_dhcp_ctx isn't
persisted in the cache.
Signed-off-by: Chris Patterson cpatterson@microsoft.com
|
|
|
|
load_azure_ds_dir() always returns a tuple. Instead of saving this
tuple as ret, expand it immediately as md, userdata_raw, cfg, files.
This allows for more fine-grained passing of data before getting
expanded later.
- Update _should_reprovision methods to use cfg instead of tuple.
- Update _should_reprovision methods to remove the ovf_md guard.
This should be a safe refactor as the OVF is not required, and the
config is initialized to an empty dict. In practice, a mount failure
would have initialized ret anyways if the OVF was not found. If a
mount failure wasn't seen and ret was None, this guard could be
causing other failures by ignoring the PPS state that should be
available from IMDS metadata.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Avoid series-specific log formatting in tests by using
caplog.record_tuples.
Added benefit, we can easily check log-level WARNING too.
Fixes unit tests from Focal Package builds at:
https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/\
+build/23076508
|
|
We were seeing issues where if anything showed up before the
expected first adapter, booting could fail. This switches to seeking
for a working interface to handle edge cases.
Also fixes region code handling.
|
|
Dailies are always available for stable releases and the devel release.
Moreover testing against dailies will warn us earlier about issues.
|
|
snapd currrently looks for 'not run' from cloud-init status[1].
Avoid changing this behavior and revert "not-run" value to
"not run".
This avoids having to get snapd to change implementation and
release updates as far back as Bionic to handle a hyphenated
not-run string.
[1]: https://github.com/snapcore/snapd/blob/master/sysconfig/\
cloudinit.go#L802
|
|
|
|
The if-statement set ovf_is_accessible to True if the OVF is read
from /dev/sr0, but not from other data sources. It defaults to
True, but may get flipped to False while processing an invalid
source, and never get set back to True when reading from the data
directory.
Instead, default ovf_is_accessible to False, and only set it to
True once we've read an OVF successfully (and end the search).
This fixes an error when OVF is read from data_dir and IMDS
data is unavailable (failing with "No OVF or IMDS available").
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
This enables the OVF enviroment transport via ISO variable as defined
in the Open Virtualization Format Specification Version: 2.1.1.
8.1 VirtualHardwareSection
11.1 Transport media
Co-authored-by: Gabriel Mainberger <gabriel.mainberger@vshn.net>
|
|
- Update EphemeralDHCPv4WithReporting to subclass EphemeralDHCPv4 for
consistency (non-functional change).
- Replace all usage of EphemeralDHCPv4 with EphemeralDHCPv4WithReporting.
- Converging to one DHCP class exposed an issue with ExitStack patches
being mixed with decorators. Specifically, it appeared that tests
that did not enable azure.EphemeralDHCPv4WithReporting mocks had it
applied anyways from previous tests.
Presumably ExitStack was overwriting the actual value with the
mock provided by the decorator? For now, remove some mock patches
that trigger failures, but future work should move towards a
consistent approach to prevent undetected effects.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.
Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.
This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.
The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.
Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:
- get_schema: source base schema file from cloud-init-schema.json
and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param
Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI
|
|
Cloud-init includes the capability to take the network-config from a
separate key. This removes the need to merge the network config in
the user-data and make it more transparent in some cases.
Reference:
https://github.com/canonical/cloud-init/blob/42b938e8ff4c50833ff7b8f5acc1d9ab3f43ab18/cloudinit/sources/DataSourceOVF.py#L557
|
|
https://pycloudlib.readthedocs.io/en/latest/configuration.html#configuration
|
