| Age | Commit message (Collapse) | Author |
|---|
|
|
|
PyYAML upgraded from 5.4.1 to 6.0.0. 6.0.0 always requires a `Loader`
arg to `yaml.load()`
|
|
Also added supporting distro/datasource classes and updated tests
that have a `get_cloud` call.
|
|
Due to multiarch, the libdeployPkgPlugin.so is deployed into dir
/usr/lib/<multiarch name>/open-vm-tools, we need to add this path
into search_paths.
LP: #1944946
|
|
Allow comments in runcmd and report failed commands correctly
A `runcmd` script may fail to parse properly, but does not mark
`runcmd` as failed when that occurs. Additionally `shellify()` fails
to correctly parse scripts that contain a comment line.
Rectify both issues and add unit tests to verify correct behavior.
LP: #1853146
|
|
In #919 (81299de), we refactored some of the code used to bring up
networks across distros. Previously, the call to bring up network
interfaces during 'init' stage unintentionally resulted in a no-op
such that network interfaces were NEVER brought up by cloud-init, even
if new network interfaces were found after crawling the metadata.
The code was altered to bring up these discovered network interfaces.
On ubuntu, this results in a 'netplan apply' call during 'init' stage
for any ubuntu-based distro on a datasource that has a NETWORK
dependency. On GCE, this additional 'netplan apply' conflicts with the
google-guest-agent service, resulting in an instance that can no
be connected to.
This commit adds a 'disable_network_activation' option that can be
enabled in /etc/cloud.cfg to disable the activation of network
interfaces in 'init' stage.
LP: #1938299
|
|
OpenNebula 6.1.80 (current dev. version) is introducing new IPv6 gateway
contextualization variable ETHx_IP6_GATEWAY, which mimics existing
variable ETHx_GATEWAY6. The ETHx_GATEWAY6 used until now will
be depracated in future relase (ET spring 2022).
See:
- new variable - https://github.com/OpenNebula/one/commit/e4d2cc11b9f3c6d01b53774b831f48d9d089c1cc
- deprecation tracking issue - https://github.com/OpenNebula/one/issues/5536
Also, added support for SET_HOSTNAME context variable, which is
currently widely used variable to configure guest VM hostname. See
https://docs.opennebula.io/6.0/management_and_operations/references/template.html#context-section
|
|
Cloud tests have been replaced with integration tests
|
|
|
|
Offload Vultr's vendordata assembly to the backend, correct vendordata
storage and parsing, allow passing critical data via the useragent,
better networking configuration for additional interfaces.
|
|
tox: bump the pinned flake8 and pylint version
* pylint: fix W1406 (redundant-u-string-prefix)
The u prefix for strings is no longer necessary in Python >=3.0.
* pylint: disable W1514 (unspecified-encoding)
From https://www.python.org/dev/peps/pep-0597/ (Python 3.10):
The new warning stems form https://www.python.org/dev/peps/pep-0597,
which says:
Developers using macOS or Linux may forget that the default encoding
is not always UTF-8. [...] Even Python experts may assume that the
default encoding is UTF-8. This creates bugs that only happen on Windows.
The warning could be fixed by always specifying encoding='utf-8',
however we should be careful to not break environments which are not
utf-8 (or explicitly state that only utf-8 is supported). Let's silence
the warning for now.
* _quick_read_instance_id: cover the case where load_yaml() returns None
Spotted by pylint:
- E1135 (unsupported-membership-test)
- E1136 (unsubscriptable-object)
LP: #1944414
|
|
In #1006, we set Azure to apply networking config every
BOOT_NEW_INSTANCE because the BOOT_LEGACY option was causing problems
applying networking the second time per boot. However,
BOOT_NEW_INSTANCE is also wrong as Azure needs to apply networking
once per boot, during init-local phase.
|
|
|
|
* Update test_combined.py to allow either valid LXD subplatform
* Split jinja templated tests into separate module as they can be more
fragile
* Move checks for warnings and tracebacks into dedicated utility
function. This allows us to work around persistent and expected
tracebacks/warnings on particular clouds.
* Update test_upgrade.py to allow either valid Azure datasource.
/var/lib/waagent or a mounted device are both valid.
* Add specificity to test_ntp_servers.py
Clouds will often specify their own ntp servers in the ntp
configuration files, so make the tests manually specify their own.
* Account for additional keys on system in test_ssh_keysfiles.py
* Update tests to account for invalid cache
test_user_events.py and test_version_change.py both have tests that
assume we will have valid ds cache when rebooting.
In test_user_events.py, subsequent boots should block applying
network on boot if boot event is denied. However, if the cache is
invalid, it is valid to apply networking config that boot.
In test_version_change.py no cache found won't trigger the expected
debug log. Additionally, the pickle used for that test on an older
release triggered an unexpected issue that took a different error
path.
* Ignore bionic in hotplug tests (LP: #1942247)
On Bionic, we traceback when attempting to detect the hotplugged
device in the updated metadata. This is because Bionic is
specifically configured not to provide network metadata.
See LP: #1942247 for more details.
* Fix date used in test_final_message.
In test_final_message, we ensured the variable substitution works as
expected. For $timestamp, we compared against the current date. It's
possible for the host date to be massively different from the client
date, so obtain date on client rather than host.
* Remove module success from lp1813396 test. Module may fail
unrelatedly (in this case apt-get update is failing), but the test
should still pass.
* Skip testing events if network is disabled
* Ensure we install expected version of cloud-init
As part of test setup, we can install cloud-init from various
sources, including PROPOSED, PPAs, etc. We were never checking that
this install completes successfully, and on OCI, it wasn't
completing successfully because of apt locking issues. Code has
been updated to retry, and then fail loudly if we can't complete the
install.
* Remove ubuntu-azure-fips metapkg which mandates FIPS-flavour kernel
In test_lp1835584.py
* Update test_user_events.py to account for Azure behavior
since Azure has a separate service to clear the pickled metadata
every boot
* Change failure to warning in test_upgrade.py if initial boot errors
If there's already a pre-existing cause for warnings or tracebacks,
that shouldn't cause the new version to fail.
* Add retry to test_random_passwords_emitted_to_serial_console
It's possible we haven't retrieved the entire log when the call returns,
so retry a few times if the output isn't empty.
|
|
openEuler Homepage: https://www.openeuler.org/en/
|
|
As specified in #LP 1845552,
In cloudinit/ssh_util.py, in parse_ssh_config_lines(), we attempt to
parse each line of sshd_config. This function expects each line to
be one of the following forms:
\# comment
key value
key=value
However, options like DenyGroups and DenyUsers are specified to
*optionally* accepts values in sshd_config.
Cloud-init should comply to this and skip the option if a value
is not provided.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
|
|
https://www.cloudlinux.com/
|
|
The current code starts the puppet agent and also sets autostart
in all cases. This conflicts with a common pattern where puppet
itself manages the agent and autostart state.
For example, in my deploy puppet disables the puppet agent
and replaces it with a cron. This causes various races
both within this cloud-init unit and within puppet itself
while cloud-init and puppet fight over whether or not
to enable the service.
|
|
Using flake8 inplace of pyflakes
Renamed run-pyflakes -> run-flake8
Changed target name to flake8 in Makefile
With pyflakes we can't suppress warnings/errors in few required places.
flake8 is flexible in that regard. Hence using flake8 seems to be a
better choice here.
flake8 does the job of pep8 anyway.
So, removed pep8 target from Makefile along with tools/run-pep8 script.
Included setup.py in flake8 checks
|
|
Also fix search path in networkd
|
|
In the nic attach path, we skip doing dhcp since we already did it
when bringing the interface up. However when polling for
reprovisiondata, it is possible for the request to timeout due to
platform issues. In those cases we still need to do dhcp and try again
since we tear down the context. We can only skip the first dhcp
attempt.
|
|
Home directory permissions changed in hirsute. The integration test
assumed permissions from earlier releases. Test was fixed to take both
permissions into account
|
|
before rebinding again (#990)
Add 10 second polling loop in wait_for_link_up after performing
an unbind and re-bind of primary NIC in hv_netvsc driver.
Also reduce cloud-init logging levels to debug for these operations.
|
|
Fix home permissions modified by ssh module
In #956, we updated the file and directory permissions for keys not in
the user's home directory. We also unintentionally modified the
permissions within the home directory as well. These should not change,
and this commit changes that back.
LP: #1940233
|
|
Ensure jinja templates work for both instance-data.json and
instance-data-sensitive.json. Test for LP: #1931392
Also removed test_runcmd.py as it's made redundant by this change.
|
|
|
|
The issues we see on Bionic VMs don't appear anywhere else, including
when invoking kvm directly. It likely has to do with the extra
LXD agent setup happening on bionic. Given that we still have Bionic
covered on all other platforms, the risk of skipping bionic for LXD VM
tests seems low.
|
|
Alters hotplug hook to have a query mechanism checking if the
functionality is enabled. This allows us to avoid using the hotplug
socket and service when hotplug is disabled.
|
|
Add tests for cc_resolv_conf handler
|
|
* Replace broken httpretty tests with mock
Certain versions of python/httpretty don't work correctly using https
URIs. #960 recently added httpretty tests using https. This commit
replaces the httpretty tests that were failing on https with mocks of
readurl instead.
|
|
When bringing interface up by unbinding and then binding hv_netvsc
driver, it might take a short delay after binding for the link to be
up. So before trying unbind/bind again after sleep, check if the link
is up. This is a corner case when a preprovisioned VM is reused and
the NICs are hot-attached.
|
|
|
|
- update the puppet module to support AIO installations by setting
`install_type` to `aio`
- make the install collection configurable through the `collection`
parameter; by default the rolling `puppet` collection will be used,
which installs the latest version)
- when `install_type` is `aio`, puppetlabs repos will be purged after
installation; set `cleanup` to `False` to prevent this
- AIO installations are performed by downloading and executing a shell
script; the URL for this script can be overridden using the
`aio_install_url` parameter
- make it possible to run puppet agent after installation/configuration
via the `exec` key
- by default, puppet agent will run with the `--test` argument; this can
be overridden via the `exec_args` key
|
|
This patch finally introduces the Cloud-Init Datasource for VMware
GuestInfo as a part of cloud-init proper. This datasource has existed
since 2018, and rapidly became the de facto datasource for developers
working with Packer, Terraform, for projects like kube-image-builder,
and the de jure datasource for Photon OS.
The major change to the datasource from its previous incarnation is
the name. Now named DatasourceVMware, this new version of the
datasource will allow multiple transport types in addition to
GuestInfo keys.
This datasource includes several unique features developed to address
real-world situations:
* Support for reading any key (metadata, userdata, vendordata) both
from the guestinfo table when running on a VM in vSphere as well as
from an environment variable when running inside of a container,
useful for rapid dev/test.
* Allows booting with DHCP while still providing full participation
in Cloud-Init instance data and Jinja queries. The netifaces library
provides the ability to inspect the network after it is online,
and the runtime network configuration is then merged into the
existing metadata and persisted to disk.
* Advertises the local_ipv4 and local_ipv6 addresses via guestinfo
as well. This is useful as Guest Tools is not always able to
identify what would be considered the local address.
The primary author and current steward of this datasource spoke at
Cloud-Init Con 2020 where there was interest in contributing this datasource
to the Cloud-Init codebase.
The datasource currently lives in its own GitHub repository at
https://github.com/vmware/cloud-init-vmware-guestinfo. Once the datasource
is merged into Cloud-Init, the old repository will be deprecated.
|
|
|
|
In /etc/ssh/sshd_config, it is possible to define a custom
authorized_keys file that will contain the keys allowed to access the
machine via the AuthorizedKeysFile option. Cloudinit is able to add
user-specific keys to the existing ones, but we need to be careful on
which of the authorized_keys files listed to pick.
Chosing a file that is shared by all user will cause security
issues, because the owner of that key can then access also other users.
We therefore pick an authorized_keys file only if it satisfies the
following conditions:
1. it is not a "global" file, ie it must be defined in
AuthorizedKeysFile with %u, %h or be in /home/<user>. This avoids
security issues.
2. it must comply with ssh permission requirements, otherwise the ssh
agent won't use that file.
If it doesn't meet either of those conditions, write to
~/.ssh/authorized_keys
We also need to consider the case when the chosen authorized_keys file
does not exist. In this case, the existing behavior of cloud-init is
to create the new file. We therefore need to be sure that the file
complies with ssh permissions too, by setting:
- the actual file to permission 600, and owned by the user
- the directories in the path that do not exist must be root owned and
with permission 755.
|
|
(SC-191) (#955)
This should enable us to remove the cloud-tests entirely.
|
|
|
|
Implement missing device_aliases feature
The device_aliases key has been documented as part of disk_setup for
years, however the feature was never implemented. This implements the
feature as documented allowing usercfg (rather than dsconfig) to create
a mapping of device names.
This is not to be confused with disk_aliases, a very similar map but
existing solely for use by datasources.
LP: #1867532
|
|
test_ssh_import_id.py occassionally fails because cloud-init finishes
before the keys have been fully imported. A retry has been added to the
test.
|
|
Currently cloud-init generates fallback network config on various
scenarios.
For example:
1. When no DS found
2. There is no 'network' info given in DS metadata.
3. If a DS gives a network config once and upon reboot if DS doesn't
give any network info, previously set network data will be
overridden.
A newly introduced key in cloud.cfg.tmpl can be used to control this
behavior on PhotonOS.
Also, if OS comes with a set of default network files(configs), like in
PhotonOS, cloud-init should not overwrite them by default.
This change also includes some nitpicking changes of reorganizing few
config variables.
Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
|
|
Virtuozzo Linux is a distro based off of CentOS 8, similar to Alma Linux and Rocky Linux.
|
|
Details:
1. Support guest set network config through guestinfo.ovfEnv using OVF
2. 'network-config' Property is optional
3. 'network-config' Property's value has to be base64 encoded
Added unittests and updated ovf-env.xml example
|
|
In CI run against pylint 2.9.3 and fix occurrences of:
- W0237 (arguments-renamed)
- W0402 (deprecated-module)
The W0402 deprecated-module was about module `imp`:
cloudinit/patcher.py:9: [W0402(deprecated-module), ]
Uses of a deprecated module 'imp'
The imp module is deprecated and replaced by importlib, which according
to the documentation has no replacement for acquire_lock() and
release_lock(), which are the only reason why `imp` is imported.
Nothing about the code using this lock that actually requires it.
Let's remove the locking code and the import altogether.
Dropping the locking makes patcher.patch() an empty wrapper around
_patch_logging(). Rename _patch_logging() to patch_logging() and
call it directly instead. Drop patch().
|
|
With a few exceptions, Azure VM deployments receive provisioning
metadata through the provisioning iso presented as a cdrom device
(/dev/sr0). The existing code attempts to find this device by calling
blkid to find all devices that have either type iso9660 or udf. This
can be very expensive if the VM has a lot of disks. This commit will
attempt to mount the default iso location first and only tries to use
blkid to locate the iso location if the default mounting location fails
|
|
Adds a udev script which will invoke a hotplug hook script on all net
add events. The script will write some udev arguments to a systemd FIFO
socket (to ensure we have only instance of cloud-init running at a
time), which is then read by a new service that calls a new 'cloud-init
devel hotplug-hook' command to handle the new event.
This hotplug-hook command will:
- Fetch the pickled datsource
- Verify that the hotplug event is supported/enabled
- Update the metadata for the datasource
- Ensure the hotplugged device exists within the datasource
- Apply the config change on the datasource metadata
- Bring up the new interface (or apply global network configuration)
- Save the updated metadata back to the pickle cache
Also scattered in some unrelated typing where helpful
|
|
Python 3.6 added a new `policy` attribute to `MIMEMultipart`.
MIMEMultipart may be part of the cached object pickle of a datasource.
Upgrading from an old version of python to 3.6+ will cause the
datasource to be invalid after pickle load.
This commit uses the upgrade framework to attempt to access the mime
message and fail early (thus discarding the cache) if we cannot.
Commit 78e89b03 should fix this issue more generally.
|
|
defined in AuthorizedKeysFile (#937)
This patch aims to fix LP1911680, by analyzing the files provided
in sshd_config and merge all keys into an user-specific file. Also
introduces additional tests to cover this specific case.
The file is picked by analyzing the path given in AuthorizedKeysFile.
If it points inside the current user folder (path is /home/user/*), it
means it is an user-specific file, so we can copy all user-keys there.
If it contains a %u or %h, it means that there will be a specific
authorized_keys file for each user, so we can copy all user-keys there.
If no path points to an user-specific file, for example when only
/etc/ssh/authorized_keys is given, default to ~/.ssh/authorized_keys.
Note that if there are more than a single user-specific file, the last
one will be picked.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Co-authored-by: James Falcon <therealfalcon@gmail.com>
LP: #1911680
RHBZ:1862967
|
|
Add a new switch allow_raw_data to control raw data feature, update
the documentation. Fix bugs about max_wait.
|
|
test_upgrade.py was outputting a ton of stuff that had to be manually
collected and verified. This commit adds more assertions to the test
and outputs directly to the logs rather than separate files.
|
