| Age | Commit message (Collapse) | Author |
|---|
|
#342 (70dbccbb) introduced the ability to determine route-metrics based on
the `device-number` provided by the EC2 IMDS. Not all datasources that
subclass EC2 will have this attribute, so allow the old behavior if
`device-number` is not present.
LP: #1917875
|
|
`get_interfaces` is used to in two ways, broadly: firstly, to determine
the available interfaces when converting cloud network configuration
formats to cloud-init's network configuration formats; and, secondly, to
ensure that any interfaces which are specified in network configuration
are (a) available, and (b) named correctly. The first of these is
unaffected by this commit, as no clouds support Open vSwitch
configuration in their network configuration formats.
For the second, we check that MAC addresses of physical devices are
unique. In some OVS configurations, there are OVS-created devices which
have duplicate MAC addresses, either with each other or with physical
devices. As these interfaces are created by OVS, we can be confident
that (a) they will be available when appropriate, and (b) that OVS will
name them correctly. As such, this commit excludes any OVS-internal
interfaces from the set of interfaces returned by `get_interfaces`.
LP: #1912844
|
|
Add flexibility to IMDS api-version by having both a desired IMDS
api-version and a minimum api-version. The desired api-version will
be used first, and if that fails it will fall back to the minimum
api-version.
|
|
Changes:
* Only merge in default Azure cloud ephemeral disk configs
during DataSourceAzure._get_data() if the ephemeral disk
exists.
* DataSourceAzure.address_ephemeral_resize() (which is
invoked in DataSourceAzure.activate() should only set up
the ephemeral disk if the disk exists.
Azure VMs may or may not come with ephemeral resource disks
depending on the VM SKU. For VM SKUs that come with
ephemeral resource disks, the Azure platform guarantees that
the ephemeral resource disk is attached to the VM before
the VM is booted. For VM SKUs that do not come with
ephemeral resource disks, cloud-init currently attempts
to wait and set up a non-existent ephemeral resource
disk, which wastes boot time. It also causes disk setup
modules to fail (due to non-existent references to the
ephemeral resource disk).
udevadm settle is invoked by cloud-init very early in boot.
udevadm settle is invoked very early, before
DataSourceAzure's _get_data() and activate() methods.
Within DataSourceAzure's _get_data() and activate() methods,
the ephemeral resource disk path should exist if the
VM SKU comes with an ephemeral resource disk.
The ephemeral resource disk path should not exist if the
VM SKU does not come with an ephemeral resource disk.
LP: #1901011
|
|
Kernel's newer than 4.15 present /sys/dmi/id/product_uuid as a
lowercase value. Previously UUID was uppercase.
Azure datasource reads the product_uuid directly as their platform's
instance-id. This presents a problem if a kernel is either
upgraded or downgraded across the 4.15 kernel version boundary because
the case of the UUID will change, resulting in cloud-init seeing a
"new" instance id and re-running all modules.
Re-running cc_ssh in cloud-init deletes and regenerates ssh_host keys
on a system which can cause concern on long-running instances that
somethingnefarious has happened.
Also add:
- An integration test for this for Azure Bionic Ubuntu FIPS upgrading from
a FIPS kernel with uppercase UUID to a lowercase UUID in linux-azure
- A new pytest.mark.sru_next to collect all integration tests related to our
next SRU
LP: #1835584
|
|
New datasource utilizing UpCloud metadata API, including relevant unit
tests and documentation.
|
|
Add support for openstack's dynamic vendor data, which appears under openstack/latest/vendor_data2.json
This adds vendor_data2 to all pathways; it should be a no-op for non-OpenStack providers.
LP: #1841104
|
|
If cloud-init is enabled on VMware platform, cloud-init will wait until
its configuration file is ready and currently the max wait is 90
seconds by default. With our test, this configuration file should be
ready within 1 second, so change it to 15 seconds for better
performance. Also update the documentation about how to change the
default value in cloud-init configuration file.
|
|
This feature will modify VMware datasource to read from meta data and user data which are specified by VMware vSphere user. If meta data/user data are found in cloud-init configuration directory, datasource will parse the meta data/network and user data from the configuration file, otherwise it will continue to parse them from traditional customization configuration file as before. The supported meta data file is in json or yaml format.
|
|
With the changes for SSH public keys to be retrieved from IMDS as a
first option, when a key is passed through not in the raw SSH public key
format it causes an issue and the key is not added to the user's
authorized_keys file.
This PR will temporarily disable this behavior until a permanent fix is
put in place.
|
|
|
|
Prevent network interfaces without IP addresses from being added to the
generated network configuration.
|
|
Adds the ability to run the Azure preprovisioned VMs as NIC-less and
then hot-attach them when assigned for reprovision.
The NIC on the preprovisioned VM is hot-detached as soon as it reports
ready and goes into wait for one or more interfaces to be hot-attached.
Once they are attached, cloud-init gets the expected number of NICs (in
case there are more than one) that will be attached from IMDS and waits
until all of them are attached. After all the NICs are attached,
reprovision proceeds as usual.
|
|
Pushing dmesg log to KVP to help troubleshoot VM boot issues
|
|
cc_set_password will only update the password for the default user if
cfg['password'] is set. The existing code of datasource Azure will fail
to update the default user's password because it does not set that
metadata. If the default user doesn't exist in the image, the current
code works fine because the password is set during user create and
not in cc_set_password
|
|
Increase Azure Endpoint HTTP retries to handle
occasional platform network blips.
Introduce a common method http_with_retries
in the azure.py helper, which will serve as
the common HTTP request handler for
all HTTP requests with the Azure endpoint.
This method has builtin retries and
reporting diagnostics logic.
|
|
On systems where the Azure datasource
is a viable platform for crawling metadata,
cloud-init occasionally encounters fatal
irrecoverable errors during the crawling
of the Azure datasource.
When this happens, cloud-init crashes,
and Azure VM provisioning would fail.
However, instead of failing immediately,
the user will continue seeing provisioning
for a long time until it times out with
"OS Provisioning Timed Out" message.
In these situations, cloud-init should
report failure to the Azure datasource
endpoint indicating provisioning failure.
The user will immediately see provisioning
terminate, giving them a much better
failure experience instead of pointlessly
waiting for OS provisioning timeout.
|
|
This is a new builtin variable that appeared in Ubuntu in
5.1~rc2-1ubuntu1 and started causing daily build failures.
|
|
Largely speaking, ds-identify protects from this scenario being
hit, but if DataSourceRbxCloud ran and there was no metadata
disks found (LABEL=CLOUDMD), then it would stacktrace.
The fix is just to clean up the get_md function a little
bit, and the explicitly check for False as a return value in _get_data.
|
|
This allows the cloud-init log to be pushed multiple times during boot,
with the latest lines being pushed each time.
|
|
* Include both Ubuntu-style cfg file, and Debian-style directory in ENI
* Add WebSpider as contributor
|
|
This just separates the reading of dmi values into its own file.
Some things of note:
* left import of util in dmi.py only for 'is_container'
It'd be good if is_container was not in util.
* just the use of 'util.is_x86' to dmi.py
* open() is used directly rather than load_file.
|
|
Fixes erroneous string/int comparison introduced in 1431c8a
metadata['instance-id'] is an integer but the value read from smbios is
a string. The comparision would cause TypeError.
|
|
Hetzner Cloud also provides the instance ID in SMBIOS information. Use
it to locally check_instance_id and to compared with instance_id from
metadata service.
LP: #1885527
|
|
The static and static6 subnet types for network_data.json were
being ignored by the Openstack handler, this would cause the code to
break and not function properly.
As of today, if a static6 configuration is chosen, the interface will
still eventually be available to receive router advertisements or be set
from NetworkManager to wait for them and cycle the interface in negative
case.
It is safe to assume that if the interface is manually configured to use
static ipv6 address, there's no need to wait for router advertisements.
This patch will set automatically IPV6_AUTOCONF and IPV6_FORCE_ACCEPT_RA
both to "no" in this case.
This patch fixes the specific behavior only for RHEL flavor and
sysconfig renderer. It also introduces new unit tests for the specific
case as well as adjusts some existent tests to be compatible with the
new options. This patch also addresses this problem by assigning the
appropriate subnet type for each case on the openstack handler.
rhbz: #1889635
rhbz: #1889635
Signed-off-by: Eduardo Otubo otubo@redhat.com
|
|
DataSourceAzure previously writes the preprovisioning
reported ready marker file before it goes through the
report ready workflow. On certain VM instances, the
marker file is successfully written but then reporting
ready fails.
Upon rare VM reboots by the platform, cloud-init sees
that the report ready marker file already exists.
The existence of this marker file tells cloud-init
not to report ready again (because it mistakenly
assumes that it already reported ready in
preprovisioning).
In this scenario, cloud-init instead erroneously
takes the reprovisioning workflow instead of
reporting ready again.
|
|
Consider valid product names as valid chassis asset tags when detecting
OpenStack platform before crawling for OpenStack metadata.
As `ds-identify` tool uses product name as valid chassis asset tags,
let's replicate the behaviour in the OpenStack platform detection too.
This change should be backwards compatible and a temporary fix for the
current limitations on the OpenStack platform detection.
LP: #1895976
|
|
This moves logging into `report_diagnostic_event`, to clean up its callsites.
|
|
enumeration of physical network devices (#591)
|
|
fails (#549)
Azure datasource's `parse_network_config` throws a fatal uncaught exception when an exception is raised during generation of network config from IMDS metadata. This happens when IMDS metadata is invalid/corrupted (such as when it is missing network or interface metadata). This causes the rest of provisioning to fail.
This changes `parse_network_config` to be a non-fatal implementation. Additionally, when generating network config from IMDS metadata fails, fall back on generating fallback network config (`_generate_network_config_from_fallback_config`).
This also changes fallback network config generation (`_generate_network_config_from_fallback_config`) to blacklist an additional driver: `mlx5_core`.
|
|
|
|
Co-authored-by: Rick Harding <rharding@mitechie.com>
|
|
* pull ssh keys from imds first and fall back to ovf if unavailable
* refactor log and diagnostic messages
* refactor the OpenSSLManager instantiation and certificate usage
* fix unit test where exception was being silenced for generate cert
* fix tests now that certificate is not always generated
* add documentation for ssh key retrieval
* add ability to check if http client has security enabled
* refactor certificate logic to GoalState
|
|
This reverts commit 8d25d5e6fac39ab3319ec5d37d23196429fb0c95.
|
|
Changes:
tox: bump the pylint version to 2.6.0 in the default run
Fix pylint 2.6.0 W0707 warnings (raise-missing-from)
|
|
This fixes a long delay during boot of some instances. For Azure instance types using SR-IOV via the Hyper-V netvsc network driver, two network interfaces are created that share the same MAC, but only the virtual device should be configured and used. Updating the netplan configuration to filter on the hv_netvsc driver prevents netplan from trying to figure both devices.
LP: #1830740
|
|
Push the cloud-init.log file (Up to 500KB at once) to the KVP before reporting ready to the Azure platform.
Based on the analysis done on a large sample of cloud-init.log files, Here's the statistics collected on the log file size:
P50 P90 P95 P99 P99.9 P99.99
137K 423K 537K 3.5MB 6MB 16MB
This change limits the size of cloud-init.log file data that gets dumped to KVP to 500KB. So for ~95% of the cases, the whole log file will be dumped and for the remaining ~5%, we will get the last 500KB of the cloud-init.log file.
To asses the performance of the 500KB limit, 250 VM were deployed with a 500KB cloud-init.log file and the time taken to compress, encode and dump the entries to KVP was measured. Here's the time in milliseconds percentiles:
P50 P99 P999
75.705 232.701 1169.636
Another 250 VMs were deployed with this logic dumping their normal cloud-init.log file to KVP, the same timing was measured as above. Here's the time in milliseconds percentiles:
P50 P99 P999
1.88 5.277 6.992
Added excluded_handlers to the report_event function to be able to opt-out from reporting the events of the compressed cloud-init.log file to the cloud-init.log file.
The KVP break_down logic had a bug, where it will reuse the same key for all the split chunks of KVP which results in overwriting the split KVPs by the last one when consumed by Hyper-V. I added the split chunk index as a differentiator to the KVP key.
The Hyper-V consumes the KVPs from the KVP file as chunks whose key is 512KB and value is 2048KB but the Azure platform expects the value to be 1024KB, thus I introduced the Azure value limit.
|
|
|
|
This PR refactors Azure report ready code to include more robust tests and telemetry.
|
|
* v2 of the API is now default with fallback to v1.
* Refactored the Oracle datasource to fetch version, instance, and vnic metadata simultaneously.
|
|
The /opc/v1/ metadata endpoints[0] are universally available in Oracle
Cloud Infrastructure and the OpenStack endpoints are considered
deprecated, so we can refactor the data source to use the OPC endpoints
exclusively. This simplifies the datasource code substantially, and
enables use of OPC-specific attributes in future.
[0] https://docs.cloud.oracle.com/en-us/iaas/Content/Compute/Tasks/gettingmetadata.htm
|
|
Update DataSourceNoCloud and ds-identify to recognize LABEL_FATBOOT labels from blkid.
Also updated associated tests.
LP: #1841466
|
|
DataSourceAzure: Gracefully handle the case of set hostname failure during provisioning
|
|
Add support for VMware's vCD configuration setting DEFAULT-RUN-POST-CUST-SCRIPT.
When set True, it will default vms to run post customization scripts if the VM has not been configured in VMTools with "enable-custom-scripts" set False.
Add datasource documentation with a bit more context about this interaction on VMware products.
With this fix, the behavior will be:
* If VM administrator doesn't want others to execute a script on this VM, VMtools can set "enable-custom-scripts" to false from the utility "vmware-toolbox-cmd".
* If VM administrator doesn't set value to "enable-custom-scripts", then by default this script is disabled for security purpose.
* For VMware's vCD product , the preference is to enable the script if "enable-custom-scripts" is not set. vCD will generate a configuration file with "DEFAULT-RUN-POST-CUST-SCRIPT" set to true. This flag works for both VMware customization engine and cloud-init.
|
|
JSONDecodeError is only available in Python 3.5+. When it isn't available (i.e. on Python 3.4, which cloud-init still supports) use the more generic ValueError.
|
|
* cloudinit: remove global disable of pylint W0107 and fix errors
This includes removing a test class which contained no tests but wasn't
detected as empty because of an errant pass statement.
* .pylintrc: update disable comment to match arguments
|
|
This includes a fix to a test that had a string concatenation issue, and
so was only testing a prefix of what was intended.
|
|
Specifically:
* disable E1102 in cloudinit/sources/helpers/openstack.py for reasons
described in a comment, and
* refactor `abs_join` to require at least one positional argument; this
matches os.path.join's signature, and that mismatch is what was
causing pylint to emit a warning
* bump to pylint 2.4.2
|
|
As the first refactor PR, this also includes the initial structure for tests.
LP: #1884619
|
|
Changes are made that simplify code and aim to properly support FreeBSD:
- use `util.find_devs_with` instead call directly `blkid`, because on FreeBSD is not supported well and `util.find_devs_with` have solution for FreeBSD for that
- introduction of an additional name on FAT file system, which is used in FreeBSD
- drop shell to use default value, because FreeBSD – by default – does not have `/bin/bash`
|
