| Age | Commit message (Collapse) | Author |
|---|
|
Allow users to provide '## template: jinja' as the first line or their
#cloud-config or custom script user-data parts. When this header exists,
the cloud-config or script will be rendered as a jinja template.
All instance metadata keys and values present in
/run/cloud-init/instance-data.json will be available as jinja variables
for the template. This means any cloud-config module or script can
reference any standardized instance data in templates and scripts.
Additionally, any standardized instance-data.json keys scoped below a
'<v#>' key will be promoted as a top-level key for ease of reference in
templates. This means that '{{ local_hostname }}' is the same as using the
latest '{{ v#.local_hostname }}'.
Since instance-data is written to /run/cloud-init/instance-data.json, make
sure it is persisted across reboots when the cached datasource opject is
reloaded.
LP: #1791781
|
|
Cloud config can now disable ssh access to non-root users.
When defining the 'users' list in cloud-configuration a boolean
'ssh_redirect_user: true' can be provided to disable ssh logins for
that user. Any ssh 'public-keys' defined in cloud meta-data will be added
and disabled in .ssh/authorized_keys. Any attempts to ssh as this user
using acceptable ssh keys will be presented with a message like the
following:
Please login as the user "ubuntu" rather than the user "youruser".
|
|
In many cases, cloud-init uses 'util.subp' to run a subprocess.
This is not really desirable in our unit tests as it makes the tests
dependent upon existance of those utilities.
The change here is to modify the base test case class (CiTestCase) to
raise exception any time subp is called. Then, fix all callers.
For cases where subp is necessary or actually desired, we can use it
via
a.) context hander CiTestCase.allow_subp(value)
b.) class level self.allowed_subp = value
Both cases the value is a list of acceptable executable names that
will be called (essentially argv[0]).
Some cleanups in AltCloud were done as the code was being updated.
|
|
Multiple distros use sysconfig format but have different content
and paths to certain files. Update distros to specify these
template paths in their renderer_configs dictionary.
|
|
The issue is when customize a VM with static IPv4 and without gateway, it
will still extend route list and will loop a gateways list which is None.
This fix is to make sure when no gateway is here, it will not extend route
list.
LP: #1766538
|
|
Linux guests can provide information to Hyper-V hosts via KVP.
KVP allows the guests to provide any string key-value-pairs back to the
host's registry. On linux, kvp communication pools are presented as pool
files in /var/lib/hyperv/.kvp_pool_#.
The following reporting configuration can enable this kvp reporting in
addition to default logging if the pool files exist:
reporting:
logging:
type: log
telemetry:
type: hyperv
|
|
The snap test requires access to a proxy and currently the integration
tests do not handle this scenario. I am disabling the test until I can
loop back around and fix this.
|
|
The snap test requires access to a proxy and currently the integration
tests do not handle this scenario. I am disabling the test untill I can
loop back around and fix this.
The write_files test, specifically, the binary test is failing on cosmic
because the "binary" file we were writting was not a complete elf
executable, but we expected 'file' to identify it as such.
The change here is to simply use some 24 bytes of random, non-utf data
and check that file was written correctly via expected checksum.
|
|
This adds a Oracle specific datasource that functions with OCI.
It is a simplified version of the OpenStack metadata server
with support for vendor-data.
It does not support the OCI-C (classic) platform.
Also here is a move of BrokenMetadata to common 'sources'
as this was the third occurrence of that class.
|
|
Azure datasource now queries IMDS metadata service for network
configuration at link local address
http://169.254.169.254/metadata/instance?api-version=2017-12-01. The
azure metadata service presents a list of macs and allocated ip addresses
associated with this instance. Azure will now also regenerate network
configuration on every boot because it subscribes to EventType.BOOT
maintenance events as well as the 'first boot'
EventType.BOOT_NEW_INSTANCE.
For testing add azure-imds --kind to cloud-init devel net_convert tool
for debugging IMDS metadata.
Also refactor _get_data into 3 discrete methods:
- is_platform_viable: check quickly whether the datasource is
potentially compatible with the platform on which is is running
- crawl_metadata: walk all potential metadata candidates, returning a
structured dict of all metadata and userdata. Raise InvalidMetaData on
error.
- _get_data: call crawl_metadata and process results or error. Cache
instance data on class attributes: metadata, userdata_raw etc.
|
|
DEP_NETWORK is removed since the network_config must
run at each boot. New EventType.BOOT event is used
for that.
Network is brought up early to fetch the metadata which
is required to configure the network (ipv4 and/or v6).
Adds unittests for the following and fixes test_common for
LOCAL and NETWORK sets.
|
|
When converting network config v1 to netplan, we were not correctly
rendering the 'macaddress' key on a bond. Not that the difference
in spelling between v1 'mac_address' and v2 'macaddress' is intentional.
Also fixed here is rendering of the macaddress for bridges.
LP: #1784699
|
|
Move the tools/net-convert.py to be exposed as part of 'cloud-init devel'
subcommands.
It can now be called like:
$ cloud-init devel net-convert
Or, if you just have checked out source (and no cli executable):
$ python3 -m cloudinit.cmd.devel.net_convert
or
$ python3 -m cloudinit.cmd.main devel net-convert
|
|
The OpenNebula data source generates an invalid netplan yaml file
if the IPv6 gateway is not defined in context.sh.
LP: #1768547
|
|
The OpenStack datasource in 18.3 changed to detect data in the
init-local stage instead of init-network and attempted to redetect
OpenStackLocal datasource on Oracle across reboots. The function
detect_openstack was added to quickly detect whether a platform is
OpenStack based on dmi product_name or chassis_asset_tag and it was
a bit too strict for Oracle in checking for 'OpenStack Nova'/'Compute'
DMI product_name.
Oracle's DMI product_name reports 'SAtandard PC (i440FX + PIIX, 1996)'
and DMI chassis_asset_tag is 'OracleCloud.com'.
detect_openstack function now adds 'OracleCloud.com' as a supported value
'OracleCloud.com' to valid chassis-asset-tags for the OpenStack
datasource.
LP: #1784685
|
|
Described in bug 1783198 we have seen some transient failures when
using pylxd -> lxd api.
This does:
* adds a str() representation of LXDInstance
* checks the value of the pylxd_container object on instantion
* sets pylxd_container object to None on deletion.
* adds retry logic to shutdown()
|
|
This adds a script to always get the /etc/cloud/build.info file
if it exists, and a hook when preparing the image to log the information
if it is available.
INFO - setting up ubuntu-cosmic (build_name=server serial=20180718)
This is just useful for debug and reproduce.
|
|
Pylint 2.0.0 was recently released and complains more about
logging-not-lazy than it used to. I've fixed those warnings, here.
The changes in rh_subscription are more extensive. pylint may be
complaining incorrectly there, but the tests were not correctly un-doing
all of their mock/patching. This cleans those up and makes pylint happy.
|
|
A recent commit added get_linux_distro to replace the deprecated python
platform.dist module behavior before it is dropped from python. It added
behavior that was compliant on OpenSuSE and SLES, by returning
(<distro_name>, <distro_version>, <cpu-arch>).
Fix get_linux_distro to behave more like the specific distribution's
platform.dist on ubuntu, centos and debian, which will return the
distribution release codename as the third element instead of <cpu-arch>.
SLES and OpenSUSE will retain their current behavior.
Examples follow:
('sles', '15', 'x86_64')
('opensuse', '42.3', 'x86_64')
('debian', '9', 'stretch')
('ubuntu', '16.04', 'xenial')
('centos', '7', 'Core')
LP: #1780481
|
|
The salt minion integration test as we had it did not do a whole lot
more than the unit tests on that module did. Additionally, it caused
some transient failures at least in Ubuntu 18.04.
At a future date we may choose to add an integration test that installs
salt-minion and salt server and configures it to be a better test.
LP: #1778737
|
|
To deny a user elevated access, you can omit the `sudo` key from the
`users` dictionary. This works fine however it's implicitly defined
based on defaults of `cloud-init`. If the project moves to have `sudo`
access allowed for all by default (quite unlikely but still possible)
this will catch a few people out.
This introduces the ability to define an explicit `sudo: False` in the
`users` dictionary and it will prevent `sudo` access. The behaviour is
identical to omitting the key.
LP: #1771468
|
|
Newer versions (3.0.1+) of lxd create the 'lxdbr0' network when
'lxd init --auto' is invoked.
When cloud-init is given a network configuration to pass on to
lxc and that config had no name specified or 'lxdbr0', then cloud-init
would fail to create the network as it already exists.
Similarly, we need to remove the device from the default profile
so that the attach code can work.
Also, add a _lxc method and use it to make sure we're getting the
--force-local flag everywhere.
LP: #1776958
|
|
OpenStack datasource is now discovered in init-local stage. In order to
probe whether OpenStack metadata is present, it performs a costly
sandboxed dhclient setup and metadata probe against http://169.254.169.254
for openstack data.
Cloud-init properly detects non-OpenStack on EC2, but it spends precious
time probing the metadata service also resulting in a confusing WARNING
log about 'metadata not present'. To avoid the wasted cycles, and
confusing warning, get_data will call a detect_openstack function to
quickly determine whether the platform looks like OpenStack before trying
to setup network to probe and crawl the metadata service.
LP: #1776701
|
|
In /etc/cloud/cloud.cfg, users and imagees can configure which modules run
during a specific cloud-init stage by modifying one of the following
lists: cloud_init_modules, cloud_init_modules, cloud_init_final_modules.
If any of the configured module lists are absent or empty, cloud-init will
emit the same message it already does for existing lists that only contain
modules which are not unsupported on that platform:
No 'config' modules to run under section 'cloud_config_modules'
LP: #1770462
|
|
When creating the multipart mime message that is written as
user-data.txt.i, cloud-init losing data on conversion to some things
as a string.
LP: #1768600
Author: Scott Moser <smoser@ubuntu.com>
Co-Authored-By: Chad Smith <chad.smith@canonical.com>
|
|
There is no requirement that the environment of a process contains
only utf-8 data. This modifies get_proc_env to support it reading
data as binary and decoding if provided with an encoding.
The default case is now that we now do:
contents.decode('utf-8', 'replace')
rather than
contents.decode('utf-8', 'strict')
LP: #1775371
|
|
In ubuntu, the salt-minion package version 2017.7.4+dfsg1-1 or later
automatically moves any seed keys from /etc/salt/pki/minion/ to
/var/lib/salt/pki/minion/. Fix integration tests to collect
either files in either /etc/salt/pki/minion/ or
/var/lib/salt/pki/minion/.
|
|
Integration tests will now provide a brief summary for test failures
listed by platform and distribution. The failure summary will only consist
of failed test name and assert error message.
Drop the verbose dictionary of all integration test output because this
content is unreadable given the large number of integration test results
listed within this dictionary.
|
|
A fix for chrony support per LP: #1589780 is not expected in Artful or
older series. Skip the chrony suite of tests when running on a container
and ubuntu series represented is <= artful as errors are expected.
|
|
By default, integration tests destroy the test instances after each
test run. To aid debug and development of integration tests, support a
--preserve-instance argument which will leave the modified test instance
in a stopped state for further debug.
|
|
When network configuration for any interface defines maximum transmission
values (MTU) the netplan, eni and sysconfig renders will take into account
any device-level, or subnet-level mtu values.
When network configuration has conflicting device-level and ipv4 subnet
mtu values, the subnet-specific value is honored and a warning will be
logged about any ignored device-level setting.
LP: #1774666
|
|
This adds 'combine_capture' argument as was present in curtin's
subp. It is useful to get interleaved output of a command. I noticed
a need for it when looking at user_data_rhevm in DataSourceAltCloud.
That will run a subcommand, logging its stdout but swallowing its stderr.
Another thing to change to use this would be in udevadm_settle which
currently just returns the subp() call.
Also, add the docstring copied from curtin's subp.
|
|
- Updated datadict reference URL
- Store sdc:routes metadata in DatasourceSmartOS
- Map sdc:routes values to per-interface subnet configuration
- Added unittest
Co-authored-by: Mike Gerdts <mike.gerdts@joyent.com>
LP: #1763512
|
|
Network has not yet been configured in the init-local stage so the
openstack datasource will use dhcp-client to temporarily obtain an ipv4
address and query the metadata service at http://169.254.169.254 to get
network_data.json configuration. If present, the datasource will return
network_config version 1 config based on that network_data.json content.
Previously OpenStack datasource only setup dhcp on the fallback interface
so this represents a change in behavior to react to the full config
provided by openstack.
Also significant to OpenStack is the separation of a _crawl_data operation
from get_data(). crawl_data walks the available metadata services and
returns a dict of discovered content. get_data consumes the crawled_data,
caches it in the datasource and reacts to that data.
/run/cloud-init/instance-data.json now published network_data.json or
ec2_metadata key if that data is present on any datasource.
The main reasons for the separation of crawl from get_data:
* Enable performance metrics of cloud-init's metadata crawls on each
* Enable cloud-init modules and scripts to query and consume metadata
content which may have updated/changed after cloud-init's initial cache
during instance boot. (Think hotplug)
Also generalize common logic to base DataSource class/module:
* Move to a common UNSET variable up into base datasource module fix EC2,
ConfigDrive, OpenStack, SmartOS to use the global.
* Drop get_url_settings from Ec2, CloudStack and OpenStack and generalize
DataSource.get_url_params(). Allow subclasses to override url_max_wait,
url_timeout and url_retries params.
* Rename get_network_metadata bool to perform_dhcp_setup as it designates
whether EphemeralDHCPv4 setup is required before crawling metadata.
LP: #1749717
|
|
On OpenSuSE 42.3, we would get errors running
tests/unittests/test_handler/test_handler_chef.py
- test_myhttps_nonet raises a UnmockedError
No mocking was registered, and real connections are not allowed
- test_myhttps_net raises SSLError
("bad handshake: SysCallError(32, 'EPIPE')",)
This fixes the errors by just using http instead of https.
Also it modifies the HttprettyTestCase to do the httpretty activate
and deactivate itself in setUp and tearDown. Then we don't have to
decorate individual test_ methods. Also, we set
httpretty.HTTPretty.allow_net_connect = False
Test cases here should not reach out to a network resource.
LP: #1771659
|
|
Yaml tracebacks are generally hard to read for average users. Add a bit of
logic to util.yaml_load and schema validation to look for
YAMLError.context_marker or problem_marker line and column counts.
No longer log the full exceeption traceback from the yaml_load error,
instead just LOG.warning for the specific error and point to the offending
line and column where the problem exists.
|
|
The Azure data source provides a method to check whether a NTFS partition
on the ephemeral disk is safe for reformatting to ext4. The method checks
to see if there are customer data files on the disk. However, mounting
the partition fails on systems that do not have the capability of
mounting NTFS. Note that in this case, it is also very unlikely that the
NTFS partition would have been used by the system (since it can't mount
it). The only case would be where an update to the system removed the
capability to mount NTFS, the likelihood of which is also very small.
This change allows the reformatting of the ephemeral disk to ext4 on
systems where mounting NTFS is not supported.
|
|
This modifies version.version_string to support having the package
build write the *packaged* version in with a easy replace.
Then, when cloud-init reports its version it will include the full
packaged version.
Also modified here are upstream package build files to get that done.
Note part of the trickery in packages/debian/rules.in was to avoid
the 'basic' templater consuming the '$variable' variable names.
LP: #1770712
|
|
Do not add new entries to /etc/fstab for devices that already have an
existing fstab entry.
Resolves: rhbz#1542578
|
|
SuSE builds were not getting a PATH set in generator's environment.
This may seem like mis-configuration on the system, but caused ds-identify
to fail to find blkid (or any other program).
The change here just ensures that we get /sbin /usr/sbin /bin /usr/bin
into the PATH when main is run.
LP: #1771382
|
|
|
|
The result of a read_file_or_url on a file and on a url would differ
in behavior.
str(UrlResponse) would return UrlResponse.contents.decode('utf-8')
while
str(FileResponse) would return str(FileResponse.contents)
The difference being "b'foo'" versus "foo".
As part of the general goal of cleaning util, move read_file_or_url
into url_helper.
|
|
pylint missed finding a typo in the lxd platform because it could not
determine that the variable was being used was a string. The variable
was set by loading a yaml file which pylint couldn't know that it
would be a string. In these cases, we can be more explicit.
|
|
The SSH function was retrying and waiting for SSH for over an
hour when an SSH connection was failing to be established. This
reduces the amount of retries and time between each retry to
prevent tests from running for hours.
Also restructures how waiting for the system works: the system
will attempt to SSH up to the boot timeout time by catching
SSH connection failures and retrying until the timeout is
reached. If the limit is reached now an exception is thrown
to abort the test.
Drive by - this also fixes printing of the instance name when
collecting the console log, rather than showing a Python object
address.
Fixes LP: #1758409
|
|
In playing with a SmartOS container I found that ds-identify did
not identify the container there as a container. Systemd-detect-virt
identifies it as 'container-other'.
Also here are tests for ds-identify for the SmartOS platform
identification, and some indentation fixes in ds-identify.
|
|
Make test_net.TestGenerateFallbackConfig.test_unstable_names mock
the value of /proc/cmdline in the same way as the existing
test_unstable_names_disabled test.
LP: #1769952
|
|
When attempting to apply network configuration for SmartOS's container
platform, cloud-init would not identify nics. The nics on provided
in this container service do not have 'addr_assign_type'. That
was being interpreted as being a "stolen" mac, and would be filtered
out by get_interfaces.
|
|
package_update_upgrade_install was failing as htop is now included in
Bionic images. Switch this test to install 'sl' instead.
ca_certs integration test fails on cert_count test because bionic
update-ca-certificates on bionic generates less symlinks for a given cert.
Integration tests now collect dpkg-query --show output on every instance.
Add a new assertPackageInstalled helper method which finds the package or
package version installed on the instance.
Adapt existing byobu, package_update_upgrade_install, ntp and salt_minion
tests to use assertPackageInstalled method.
LP: #1769985
|
|
Fix remaining pycodesytle warnings related to invalid string literals
introduced in more recent pycodeflakes versions
https://bugs.python.org/issue27364 .
Also stop using flake8 in tox as it is incompatible with newer versions of
pyflakes. Instead we now add tox environments for pycodestyle and pyflakes
individually.
Set the versions in both pycodestyle and pyflakes to the currently
available versions.
|
|
This change is for Azure VM Preprovisioning. A bug was found when after
azure VMs report ready the first time, during the time when VM is polling
indefinitely for the new ovf-env.xml from Instance Metadata Service
(IMDS), if a reboot happens, we send another report ready signal to the
fabric, which deletes the reprovisioning data on the node.
This marker file is used to fix this issue so that we will only send a
report ready signal to the fabric when no marker file is present. Then,
create a marker file so that when a reboot does occur, we check if a
marker file has been created and decide whether we would like to send the
repot ready signal.
LP: #1765214
|
