| Age | Commit message (Collapse) | Author |
|---|
|
Allow users to provide '## template: jinja' as the first line or their
#cloud-config or custom script user-data parts. When this header exists,
the cloud-config or script will be rendered as a jinja template.
All instance metadata keys and values present in
/run/cloud-init/instance-data.json will be available as jinja variables
for the template. This means any cloud-config module or script can
reference any standardized instance data in templates and scripts.
Additionally, any standardized instance-data.json keys scoped below a
'<v#>' key will be promoted as a top-level key for ease of reference in
templates. This means that '{{ local_hostname }}' is the same as using the
latest '{{ v#.local_hostname }}'.
Since instance-data is written to /run/cloud-init/instance-data.json, make
sure it is persisted across reboots when the cached datasource opject is
reloaded.
LP: #1791781
|
|
Cloud config can now disable ssh access to non-root users.
When defining the 'users' list in cloud-configuration a boolean
'ssh_redirect_user: true' can be provided to disable ssh logins for
that user. Any ssh 'public-keys' defined in cloud meta-data will be added
and disabled in .ssh/authorized_keys. Any attempts to ssh as this user
using acceptable ssh keys will be presented with a message like the
following:
Please login as the user "ubuntu" rather than the user "youruser".
|
|
|
|
In many cases, cloud-init uses 'util.subp' to run a subprocess.
This is not really desirable in our unit tests as it makes the tests
dependent upon existance of those utilities.
The change here is to modify the base test case class (CiTestCase) to
raise exception any time subp is called. Then, fix all callers.
For cases where subp is necessary or actually desired, we can use it
via
a.) context hander CiTestCase.allow_subp(value)
b.) class level self.allowed_subp = value
Both cases the value is a list of acceptable executable names that
will be called (essentially argv[0]).
Some cleanups in AltCloud were done as the code was being updated.
|
|
Multiple distros use sysconfig format but have different content
and paths to certain files. Update distros to specify these
template paths in their renderer_configs dictionary.
|
|
These tests focus on the apply_credentials method and the ssh setup for
root and a distro default user.
|
|
this version uses unittest2 skipIf which is present in our python 2.6
environment.
|
|
The error message when read-vesion is not very useful and does not help
the end-user know how to overcome the issue. This adds a short message
explaining that the user does not have the latest upstream tags and how
to get those tags.
|
|
Switch the implementation to a daemon thread which uses a
blocking get from the Queue. No additional locking or flag checking
is needed since the Queue itself handles acquiring the lock as needed.
cloud-init only has a single producer (the main thread calling publish)
and the consumer will read all events in the queue and write them out.
Using the daemon mode of the thread handles flushing the queue on
main exit in python3; in python2.7 we handle the EOFError that results
when the publish thread calls to get() fails indicating the main thread
has exited.
The result is that the handler is no longer spawing a thread on each
publish event but rather creates a single thread when we start up
the reporter and we remove any additional use of separate locks and
flags as we only have a single Queue object and we're only calling
queue.put() from main thread and queue.get() from consuming thread.
|
|
The issue is when customize a VM with static IPv4 and without gateway, it
will still extend route list and will loop a gateways list which is None.
This fix is to make sure when no gateway is here, it will not extend route
list.
LP: #1766538
|
|
Linux guests can provide information to Hyper-V hosts via KVP.
KVP allows the guests to provide any string key-value-pairs back to the
host's registry. On linux, kvp communication pools are presented as pool
files in /var/lib/hyperv/.kvp_pool_#.
The following reporting configuration can enable this kvp reporting in
addition to default logging if the pool files exist:
reporting:
logging:
type: log
telemetry:
type: hyperv
|
|
The snap test requires access to a proxy and currently the integration
tests do not handle this scenario. I am disabling the test until I can
loop back around and fix this.
|
|
The snap test requires access to a proxy and currently the integration
tests do not handle this scenario. I am disabling the test untill I can
loop back around and fix this.
The write_files test, specifically, the binary test is failing on cosmic
because the "binary" file we were writting was not a complete elf
executable, but we expected 'file' to identify it as such.
The change here is to simply use some 24 bytes of random, non-utf data
and check that file was written correctly via expected checksum.
|
|
This adds a Oracle specific datasource that functions with OCI.
It is a simplified version of the OpenStack metadata server
with support for vendor-data.
It does not support the OCI-C (classic) platform.
Also here is a move of BrokenMetadata to common 'sources'
as this was the third occurrence of that class.
|
|
Azure datasource now queries IMDS metadata service for network
configuration at link local address
http://169.254.169.254/metadata/instance?api-version=2017-12-01. The
azure metadata service presents a list of macs and allocated ip addresses
associated with this instance. Azure will now also regenerate network
configuration on every boot because it subscribes to EventType.BOOT
maintenance events as well as the 'first boot'
EventType.BOOT_NEW_INSTANCE.
For testing add azure-imds --kind to cloud-init devel net_convert tool
for debugging IMDS metadata.
Also refactor _get_data into 3 discrete methods:
- is_platform_viable: check quickly whether the datasource is
potentially compatible with the platform on which is is running
- crawl_metadata: walk all potential metadata candidates, returning a
structured dict of all metadata and userdata. Raise InvalidMetaData on
error.
- _get_data: call crawl_metadata and process results or error. Cache
instance data on class attributes: metadata, userdata_raw etc.
|
|
DEP_NETWORK is removed since the network_config must
run at each boot. New EventType.BOOT event is used
for that.
Network is brought up early to fetch the metadata which
is required to configure the network (ipv4 and/or v6).
Adds unittests for the following and fixes test_common for
LOCAL and NETWORK sets.
|
|
Fix a typo in in documentation that showed 'analyze blame' but
intended to show 'analyze dump'.
|
|
When converting network config v1 to netplan, we were not correctly
rendering the 'macaddress' key on a bond. Not that the difference
in spelling between v1 'mac_address' and v2 'macaddress' is intentional.
Also fixed here is rendering of the macaddress for bridges.
LP: #1784699
|
|
Move the tools/net-convert.py to be exposed as part of 'cloud-init devel'
subcommands.
It can now be called like:
$ cloud-init devel net-convert
Or, if you just have checked out source (and no cli executable):
$ python3 -m cloudinit.cmd.devel.net_convert
or
$ python3 -m cloudinit.cmd.main devel net-convert
|
|
This changes redhat's default behavior to remove the ssh keys on
new instance (ssh_deletekeys will now be at its default true value).
On redhat systems, cloud-init.service has both:
Wants=sshd-keygen.service
Before=sshd-keygen.serviceh
Which is why 'ssh_genkeytypes' is set to None
(yaml '~' == yaml null == python none).
I've changed that to be null as it seems more clear and we do not
use the tilda anywhere else in configs.
LP: #1781094
rhbz: https://bugzilla.redhat.com/show_bug.cgi?id=1598832
|
|
Bash and most other "bourne-like" shells allow declaring function
local variables via 'local'. ksh does not. Instead of using 'local'
always, use 'typeset' when the KSH_VERSION variable is present in
environment.
LP: #1784713
|
|
The OpenNebula data source generates an invalid netplan yaml file
if the IPv6 gateway is not defined in context.sh.
LP: #1768547
|
|
The OpenStack datasource in 18.3 changed to detect data in the
init-local stage instead of init-network and attempted to redetect
OpenStackLocal datasource on Oracle across reboots. The function
detect_openstack was added to quickly detect whether a platform is
OpenStack based on dmi product_name or chassis_asset_tag and it was
a bit too strict for Oracle in checking for 'OpenStack Nova'/'Compute'
DMI product_name.
Oracle's DMI product_name reports 'SAtandard PC (i440FX + PIIX, 1996)'
and DMI chassis_asset_tag is 'OracleCloud.com'.
detect_openstack function now adds 'OracleCloud.com' as a supported value
'OracleCloud.com' to valid chassis-asset-tags for the OpenStack
datasource.
LP: #1784685
|
|
Described in bug 1783198 we have seen some transient failures when
using pylxd -> lxd api.
This does:
* adds a str() representation of LXDInstance
* checks the value of the pylxd_container object on instantion
* sets pylxd_container object to None on deletion.
* adds retry logic to shutdown()
|
|
The comment in update_metadata() that explains how a datasource should
enable network reconfig on every boot presumes that
EventType.BOOT_NEW_INSTANCE is a subset of EventType.BOOT. That's not
the case, and as such a datasource that needs to configure networking
when it is a new instance and every boot needs to include both event
types.
To make the situation above easier to debug, update_metadata() now
logs when it returns false.
To make it so that datasources do not need to test before appending to
the update_events['network'], it is changed from a list to a set.
test_update_metadata_only_acts_on_supported_update_events is updated
to allow datasources to support EventType.BOOT.
Author: Mike Gerdts <mike.gerdts@joyent.com>
|
|
This adds a script to always get the /etc/cloud/build.info file
if it exists, and a hook when preparing the image to log the information
if it is available.
INFO - setting up ubuntu-cosmic (build_name=server serial=20180718)
This is just useful for debug and reproduce.
|
|
Pylint 2.0.0 was recently released and complains more about
logging-not-lazy than it used to. I've fixed those warnings, here.
The changes in rh_subscription are more extensive. pylint may be
complaining incorrectly there, but the tests were not correctly un-doing
all of their mock/patching. This cleans those up and makes pylint happy.
|
|
Add examples and tests for RHEL values of redhat-release and os-release.
These examples were collected from IBMCloud images.
on rhel systems 'platform.dist()' returns 'redhat' rather than 'rhel'
so we have adjusted the response to align there.
|
|
An empty /etc/os-release exists on some redhat images, most notably
the COPR build images of centos6 and rawhide. On platforms missing
/etc/os-release or having an empty /etc/os-release file, use
_parse_redhat_release on rhel-based images to obtain distribution and
release codename information.
LP: #1781229
|
|
In order to see some of the WARNING messages added by bug 1774666
I wanted logging output of tools/net-convert. This does:
a.) add '--debug' and make it print the network state and read yaml only
if --debug is provided.
b.) set up basic logging so warnings goes to console by default and
debug goes to console if --debug is provided.
|
|
Paramiko version 2.4.0 had a CVE (CVE-2018-7750) against it.
It is not likely particularly worrisome for our integration tests,
but we might as well bump it.
|
|
LP: #1727876
|
|
A recent commit added get_linux_distro to replace the deprecated python
platform.dist module behavior before it is dropped from python. It added
behavior that was compliant on OpenSuSE and SLES, by returning
(<distro_name>, <distro_version>, <cpu-arch>).
Fix get_linux_distro to behave more like the specific distribution's
platform.dist on ubuntu, centos and debian, which will return the
distribution release codename as the third element instead of <cpu-arch>.
SLES and OpenSUSE will retain their current behavior.
Examples follow:
('sles', '15', 'x86_64')
('opensuse', '42.3', 'x86_64')
('debian', '9', 'stretch')
('ubuntu', '16.04', 'xenial')
('centos', '7', 'Core')
LP: #1780481
|
|
|
|
Very basic type definitions are now defined to distinguish 'boot'
events from 'new instance (first boot)'. Event types will now be handed
to a datasource.update_metadata method which can determine whether
to refresh its metadata and re-render configuration based on that
source event.
A datasource can 'subscribe' to an event by setting up the update_events
attribute on the datasource class which describe what config scope is
updated by a list of matching events. By default datasources will have
the following update_events: {'network': [EventType.BOOT_NEW_INSTANCE]}
This setting says the datasource will re-write network configuration only
on first boot of a new instance or when the instance id changes.
New methods are now present on the datasource:
- clear_cached_attrs: Resets cached datasource attributes to values
listed in datasource.cached_attr_defaults. This is performed prior to
processing a fresh metadata process to avoid keeping old/invalid
cached data around.
- update_metadata: accepts source_event_types to determine if the
metadata should be crawled again and processed
|
|
The salt minion integration test as we had it did not do a whole lot
more than the unit tests on that module did. Additionally, it caused
some transient failures at least in Ubuntu 18.04.
At a future date we may choose to add an integration test that installs
salt-minion and salt server and configures it to be a better test.
LP: #1778737
|
|
When cloud-init tries to read a key from a keyserver, it will now
retry twice with 1 second in between each.
Retries of import are done by default because keyservers can be
unreliable. Additionally, there is no way to determine the difference
between a non-existant key and a failure. In both cases gpg (at least
2.2.4) exits with status 2 and stderr: "keyserver receive failed: No data"
It is assumed that a key provided to cloud-init exists on the keyserver so
re-trying makes better sense than failing.
Examples of things that made receive keys particularly unreliable:
https://bitbucket.org/skskeyserver/sks-keyserver/issues/57
https://bitbucket.org/skskeyserver/sks-keyserver/issues/60
There is also a change here from 'gpg --recv' to the longer
'gpg --recv-keys'. That option is functional and working back to
centos 6 (gpg 2.0.14) and ubuntu 14.04 (gpg 1.4.16).
|
|
If run-container was called without --package or --binary-package, then
it would still try to copy out artifacts and would fail doing so as
there were no artifacts to collect.
Also fix a bug when only --source-package without --package.
|
|
standargs -> standards.
|
|
Simplestreams is moving to git. Use the git repo rather than bzr.
|
|
Bump the version in cloudinit/version.py to be 18.3 and update ChangeLog.
LP: #1777743
|
|
|
|
To deny a user elevated access, you can omit the `sudo` key from the
`users` dictionary. This works fine however it's implicitly defined
based on defaults of `cloud-init`. If the project moves to have `sudo`
access allowed for all by default (quite unlikely but still possible)
this will catch a few people out.
This introduces the ability to define an explicit `sudo: False` in the
`users` dictionary and it will prevent `sudo` access. The behaviour is
identical to omitting the key.
LP: #1771468
|
|
Newer versions (3.0.1+) of lxd create the 'lxdbr0' network when
'lxd init --auto' is invoked.
When cloud-init is given a network configuration to pass on to
lxc and that config had no name specified or 'lxdbr0', then cloud-init
would fail to create the network as it already exists.
Similarly, we need to remove the device from the default profile
so that the attach code can work.
Also, add a _lxc method and use it to make sure we're getting the
--force-local flag everywhere.
LP: #1776958
|
|
OpenStack datasource is now discovered in init-local stage. In order to
probe whether OpenStack metadata is present, it performs a costly
sandboxed dhclient setup and metadata probe against http://169.254.169.254
for openstack data.
Cloud-init properly detects non-OpenStack on EC2, but it spends precious
time probing the metadata service also resulting in a confusing WARNING
log about 'metadata not present'. To avoid the wasted cycles, and
confusing warning, get_data will call a detect_openstack function to
quickly determine whether the platform looks like OpenStack before trying
to setup network to probe and crawl the metadata service.
LP: #1776701
|
|
In /etc/cloud/cloud.cfg, users and imagees can configure which modules run
during a specific cloud-init stage by modifying one of the following
lists: cloud_init_modules, cloud_init_modules, cloud_init_final_modules.
If any of the configured module lists are absent or empty, cloud-init will
emit the same message it already does for existing lists that only contain
modules which are not unsupported on that platform:
No 'config' modules to run under section 'cloud_config_modules'
LP: #1770462
|
|
When creating the multipart mime message that is written as
user-data.txt.i, cloud-init losing data on conversion to some things
as a string.
LP: #1768600
Author: Scott Moser <smoser@ubuntu.com>
Co-Authored-By: Chad Smith <chad.smith@canonical.com>
|
|
There is no requirement that the environment of a process contains
only utf-8 data. This modifies get_proc_env to support it reading
data as binary and decoding if provided with an encoding.
The default case is now that we now do:
contents.decode('utf-8', 'replace')
rather than
contents.decode('utf-8', 'strict')
LP: #1775371
|
|
In ubuntu, the salt-minion package version 2017.7.4+dfsg1-1 or later
automatically moves any seed keys from /etc/salt/pki/minion/ to
/var/lib/salt/pki/minion/. Fix integration tests to collect
either files in either /etc/salt/pki/minion/ or
/var/lib/salt/pki/minion/.
|
|
Integration tests will now provide a brief summary for test failures
listed by platform and distribution. The failure summary will only consist
of failed test name and assert error message.
Drop the verbose dictionary of all integration test output because this
content is unreadable given the large number of integration test results
listed within this dictionary.
|
