| Age | Commit message (Collapse) | Author |
|---|
|
Drop remaining python six usage
|
|
Add support for the NetBSD Operating System.
Features in this branch:
* Add BSD distro parent class from which NetBSD and FreeBSD can
specialize
* Add *bsd util functions to cloudinit.net and cloudinit.net.bsd_utils
* subclass cloudinit.distro.freebsd.Distro from bsd.Distro
* Add new cloudinit.distro.netbsd and cloudinit.net.renderer for
netbsd
* Add lru_cached util.is_NetBSD functions
* Add NetBSD detection for ConfigDrive and NoCloud datasources
This branch has been tested with:
- NoCloud and OpenStack (with and without config-drive)
- NetBSD 8.1. and 9.0
- FreeBSD 11.2 and 12.1
- Python 3.7 only, because of the dependency oncrypt.METHOD_BLOWFISH.
This version is available in NetBSD 7, 8 and 9 anyway
|
|
Cloud-config userdata provided as jinja templates are now distro,
platform and merged cloud config aware. The cloud-init query command
will also surface this config data.
Now users can selectively render portions of cloud-config based on:
* distro name, version, release
* python version
* merged cloud config values
* machine platform
* kernel
To support template handling of this config, add new top-level
keys to /run/cloud-init/instance-data.json.
The new 'merged_cfg' key represents merged cloud config from
/etc/cloud/cloud.cfg and /etc/cloud/cloud.cfg.d/*.
The new 'sys_info' key which captures distro and platform
info from cloudinit.util.system_info.
Cloud config userdata templates can render conditional content
based on these additional environmental checks such as the following
simple example:
```
## template: jinja
#cloud-config
runcmd:
{% if distro == 'opensuse' %}
- sh /custom-setup-sles
{% elif distro == 'centos' %}
- sh /custom-setup-centos
{% elif distro == 'debian' %}
- sh /custom-setup-debian
{% endif %}
```
To see all values: sudo cloud-init query --all
Any keys added to the standardized v1 keys are guaranteed to not
change or drop on future released of cloud-init. 'v1' keys will be retained
for backward-compatibility even if a new standardized 'v2' set of keys
are introduced
The following standardized v1 keys are added:
* distro, distro_release, distro_version, kernel_version, machine,
python_version, system_platform, variant
LP: #1865969
|
|
As noticed by Seth Arnold, non-deterministic SystemRandom should be
used when creating security sensitive random strings.
|
|
This makes it clearer that we should only use this in code paths that
will definitely have dpkg available to them.
- Rename get_architecture -> get_dpkg_architecture
- Add docstring to get_dpkg_architecture
|
|
* Ensure util.get_architecture() runs only once
util.get_architecture() recently was wrapped using python3's lru_cache()
which will cache the result so we only invoke 'dpkg --print-architecture'
once. In practice, cloud-init.log will show multiple invocations of the
command. The source of this was that the debian Distro object implements
the get_primary_arch() with this command, but it was not calling it from
util, but issuing a util.subp() directly. This branch also updates
cc_apt_configure methods to fetch the arch value from the distro class,
and then ensure that the methods apt_configure calls pass the arch value
around.
* utils: remove lsb_release and get_architecture wrappers
The original lsb_release wrapper was used to prevent polluting the
single value we cached, however lru_cache() already handles this
case by using args, kwargs values to cache different calls to the
method.
* rename_apt_list: use all positional parameters
|
|
Also fix bugs:
- pass binary instead of string to sysctlbyname(), and
- unpack the "return value" in a struct, rather than in single integer.
LP: #1853160
Co-Authored-By: Ryan Harper <ryan.harper@canonical.com>
|
|
* Make DistroChecker test work with Python 3.8
In Python 3.8, `platform.linux_distribution` has been removed. This was
anticipated, and the cloud-init code uses its own
`util.get_linux_distro` instead, which works fine w/o
`platform.linux_distribution`. However, these tests still try to mock
the platform function, which fails if it doesn't exist (Python 3.8).
Instead, mock the new function here, as this is a test for code that
depends on it rather than the function itself.
* Make GetLinuxDistro tests work with Python 3.8
In Python 3.8, `platform.dist` was removed, so allow mock to create the
function by setting `create=True`.
* Make linter happy in Python 3.8
Suppress E1101(no-member) as this function was removed.
|
|
Headers param was accidentally omitted and no longer passed through to
readurl due to a previous commit.
To avoid this omission of params in the future, drop positional param
definitions from read_file_or_url and pass all kwargs through to readurl
when we are not operating on a file.
In util:read_seeded, correct the case where invalid positional param
file_retries was being passed into read_file_or_url.
Also drop duplicated file:// prefix addition from read_seeded because
read_file_or_url does that work anyway.
LP: #1854084
|
|
This makes for a slightly prettier and less confusing log.
|
|
Since `is_FreeBSD()` is used a lot, which uses `system_info()`, which uses `get_linux_distro()` we add caching, by decorating the following functions with `@lru_cache`:
- get_architecture()
- _lsb_release()
- is_FreeBSD
- get_linux_distro
- system_info()
- _get_cmdline()
Since [functools](https://docs.python.org/3/library/functools.html) only exists in Python 3, only python 3 will benefit from this improvement. For python 2, our shim is just a pass-thru. Too bad, but, also… https://pythonclock.org/
The main motivation here was, at first, to cache more, following the style of _lsb_release.
That is now consolidated under this very same roof.
LP: #1815030
|
|
Here we replace uses of the pyyaml module directly with functions
provided by cloudinit.safeyaml. Also, change/move
cloudinit.util.yaml_dumps
to
cloudinit.safeyaml.dumps
LP: #1849640
|
|
- Detect Arch Linux and set variant accordingly in `system_info()`
- Allow setting render-cloudcfg variant parameter to 'arch'
- Adjust some basic settings for Arch Linux in the cloud.cfg.tmpl
The template might need some additional Arch-specific tweaks in the
future, but at least for now the generated config works and contains
the most relevant modules.
Also:
- Sort distro variant lists when adding Arch
- Add debian to known variants in render-cloudcfg
|
|
Since python 2.7 doesn't handle UnicodeDecodeErrors with the default
handler
LP: #1801364
|
|
- UFS file system support
- GPT partition table support
- add support for newfs's -L parameter (label)
- move freebsd specific test from Azure to freebsd
|
|
On FreeBSD, mount_cd9660 does not accept the sync option that is enabled
by default. In addition, the sync is only useful with the `rw` mode.
However the `rw` mode was never used.
This patch removes the `rw` and `sync` parameter of `mount_cb` to
simplify the code base and resolve the FreeBSD issue.
LP: #1645824
|
|
The ubuntu_drivers config module enables usage of the 'ubuntu-drivers'
command. At this point it only serves as a way of installing NVIDIA
drivers for general purpose graphics processing unit (GPGPU)
functionality.
Also, a small usability improvement to get_cfg_by_path to allow it to
take a string for the key path
"toplevel/second/mykey"
in addition to the original:
("toplevel", "second", "mykey")
|
|
- Remove the last few places that use `if PY26`
- Replace our Python version detection logic with six's (which we were
already using in most places)
|
|
six already provides this for us, and we're already paying the cost to
determine it there; no need to do it twice.
|
|
Cloud-init rendered netplan with duplicate aliases if a network config
included "global" nameserver/search values. Netplan uses can read yaml
files which do use aliaes but cloud-init did not render a single yaml
dictionary, instead it combined yaml sections into a single document
which sometimes resulted in duplicate aliases being present.
This branch introduces a yaml SafeDumper class which can set the
'ignore_aliases' attribute. This is not enabled by default but callers
to util.yaml_dumps can pass a boolean to toggle this. The netplan
render uses noalias=True and the resulting yaml output does not contain
any aliases.
LP: #1815051
|
|
cloud-init uses dhclient to fetch the DHCP lease so it can extract
DHCP options. dhclient creates the leasefile, then writes to it;
simply waiting for the leasefile to appear creates a race between
dhclient and cloud-init. Instead, wait for dhclient to be parented by
init. At that point, we know it has written to the leasefile, so it's
safe to copy the file and kill the process.
cloud-init creates a temporary directory in which to execute dhclient,
and deletes that directory after it has killed the process. If
cloud-init abandons waiting for dhclient to daemonize, it will still
attempt to delete the temporary directory, but will not report an
exception should that attempt fail.
LP: #1794399
|
|
openSUSE has changed the way the distribution is identified in
os-release. Add support detecting for openSUSE Leap 42.3, Leap 15
and TumbleWeed.
Reference: boo#1111427
|
|
Add the following instance-data.json standardized keys:
* v1._beta_keys: List any v1 keys in beta development,
e.g. ['subplatform'].
* v1.public_ssh_keys: List of any cloud-provided ssh keys for the
instance.
* v1.platform: String representing the cloud platform api supporting the
datasource. For example: 'ec2' for aws, aliyun and brightbox cloud
names.
* v1.subplatform: String with more details about the source of the
metadata consumed. For example, metadata uri, config drive device path
or seed directory.
To support the new platform and subplatform standardized instance-data,
DataSource and its subclasses grew platform and subplatform attributes.
The platform attribute defaults to the lowercase string datasource name at
self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive
datasources.
The subplatform attribute calls a _get_subplatform method which will
return a string containing a simple slug for subplatform type such as
metadata, seed-dir or config-drive followed by a detailed uri, device or
directory path where the datasource consumed its configuration.
As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata
have been refactored for a few reasons:
- crawl_metadata is now a read-only operation, persisting no attributes on
the datasource instance and returns a dictionary of consumed metadata.
- crawl_metadata now closely represents the raw stucture of the ec2
metadata consumed, so that end-users can leverage public ec2 metadata
documentation where possible.
- crawl_metadata adds a '_metadata_api_version' key to the crawled
ds.metadata to advertise what version of EC2's api was consumed by
cloud-init.
- _get_data now does all the processing of crawl_metadata and saves
datasource instance attributes userdata_raw, metadata etc.
Additional drive-bys:
* unit test rework for test_altcloud and test_azure to simplify mocks
and make use of existing util and test_helpers functions.
|
|
Add examples and tests for RHEL values of redhat-release and os-release.
These examples were collected from IBMCloud images.
on rhel systems 'platform.dist()' returns 'redhat' rather than 'rhel'
so we have adjusted the response to align there.
|
|
An empty /etc/os-release exists on some redhat images, most notably
the COPR build images of centos6 and rawhide. On platforms missing
/etc/os-release or having an empty /etc/os-release file, use
_parse_redhat_release on rhel-based images to obtain distribution and
release codename information.
LP: #1781229
|
|
A recent commit added get_linux_distro to replace the deprecated python
platform.dist module behavior before it is dropped from python. It added
behavior that was compliant on OpenSuSE and SLES, by returning
(<distro_name>, <distro_version>, <cpu-arch>).
Fix get_linux_distro to behave more like the specific distribution's
platform.dist on ubuntu, centos and debian, which will return the
distribution release codename as the third element instead of <cpu-arch>.
SLES and OpenSUSE will retain their current behavior.
Examples follow:
('sles', '15', 'x86_64')
('opensuse', '42.3', 'x86_64')
('debian', '9', 'stretch')
('ubuntu', '16.04', 'xenial')
('centos', '7', 'Core')
LP: #1780481
|
|
OpenStack datasource is now discovered in init-local stage. In order to
probe whether OpenStack metadata is present, it performs a costly
sandboxed dhclient setup and metadata probe against http://169.254.169.254
for openstack data.
Cloud-init properly detects non-OpenStack on EC2, but it spends precious
time probing the metadata service also resulting in a confusing WARNING
log about 'metadata not present'. To avoid the wasted cycles, and
confusing warning, get_data will call a detect_openstack function to
quickly determine whether the platform looks like OpenStack before trying
to setup network to probe and crawl the metadata service.
LP: #1776701
|
|
There is no requirement that the environment of a process contains
only utf-8 data. This modifies get_proc_env to support it reading
data as binary and decoding if provided with an encoding.
The default case is now that we now do:
contents.decode('utf-8', 'replace')
rather than
contents.decode('utf-8', 'strict')
LP: #1775371
|
|
This adds 'combine_capture' argument as was present in curtin's
subp. It is useful to get interleaved output of a command. I noticed
a need for it when looking at user_data_rhevm in DataSourceAltCloud.
That will run a subcommand, logging its stdout but swallowing its stderr.
Another thing to change to use this would be in udevadm_settle which
currently just returns the subp() call.
Also, add the docstring copied from curtin's subp.
|
|
Allow the user to set the distribution with --distro argument to setup.py.
Fall back is to read /etc/os-release. Final backup is to use
platform.dist() Python function. The platform.dist() function is
deprecated and will be removed in Python 3.7
LP: #1745235
|
|
A newer version of pyflakes (2.0.0) was released.
It identifed some unused variables that version 1.6.0 did not identify.
The change here merely fixes those unused variables.
|
|
Yaml tracebacks are generally hard to read for average users. Add a bit of
logic to util.yaml_load and schema validation to look for
YAMLError.context_marker or problem_marker line and column counts.
No longer log the full exceeption traceback from the yaml_load error,
instead just LOG.warning for the specific error and point to the offending
line and column where the problem exists.
|
|
The Azure data source provides a method to check whether a NTFS partition
on the ephemeral disk is safe for reformatting to ext4. The method checks
to see if there are customer data files on the disk. However, mounting
the partition fails on systems that do not have the capability of
mounting NTFS. Note that in this case, it is also very unlikely that the
NTFS partition would have been used by the system (since it can't mount
it). The only case would be where an update to the system removed the
capability to mount NTFS, the likelihood of which is also very small.
This change allows the reformatting of the ephemeral disk to ext4 on
systems where mounting NTFS is not supported.
|
|
The result of a read_file_or_url on a file and on a url would differ
in behavior.
str(UrlResponse) would return UrlResponse.contents.decode('utf-8')
while
str(FileResponse) would return str(FileResponse.contents)
The difference being "b'foo'" versus "foo".
As part of the general goal of cleaning util, move read_file_or_url
into url_helper.
|
|
In looking at some boot time for Xenial, Artful and Bionic, we noticed
some long amounts of time that appeared to be part of the DataSource but
we related to resolving URLs. In Artful and Bionic, there was an issue
(bug: #1739672) that resulted in slow getaddrinfo() calls when
systemd-resolved was in use. This patch adds two events that track time
for datasource.setup_datasource() and datasource.activate_datasource()
Additionally use log_time() to wrapper util.is_resolvable_url() which
leaves info in cloud-init.log about how much time was spent.
|
|
The cloud-init-local.service expects that any network device name changes
have already been completed by the kernel or udev daemon.
In some situations we've found that the renaming of interfaces from kernel
names (eth0, eth1, etc) to their persistent names (eno1, ens3, enp0s1,
etc) may happen after cloud-init-local has started where it reads values
from sysfs about what network devices are present, and which device to use
as a fallback nic.
Subsequently, cloud-init-local would write out network configuration for a
kernel device name which would no longer be present by the time that
networking services start to bring up the devices. The result is that the
instance does not get networking configured. Prior to use of
systemd-networkd, the Ubuntu 'networking.service' unit included a call to
udevadm settle which is why this race is not seen on a Xenial system.
This change adds the ability to detect if an interface has a stable name,
if if we find one without stable names and stable names have not been
disabled (net.ifnames=0 in /proc/cmdline), then cloud-init will invoke
udevadm settle.
LP: #1766287
|
|
This enables warnings produced by pylint for unused variables (W0612),
and fixes the existing errors.
|
|
Python has deprecated these invalid string literals now
https://bugs.python.org/issue27364
and pycodestyle is identifying them with a W605 warning.
https://github.com/PyCQA/pycodestyle/pull/676
So basically, any use of \ not followed by one of [\'"abfnrtv]
or \ooo (octal) \xhh (hex) or a newline is invalid. This is most
comomnly seen for us in regex. To solve, you either:
a.) use a raw string r'...'
b.) correctly escape the \ that was not intended to be interpreted.
|
|
The zfs/zpool commands will hang for 10 seconds if /dev/zfs is not
present (bug 1760173). This is a common occurence for containers
using zfs as rootfs. Additionally handle missing zpool command or
other errors that may occur while executing the zpool command.
|
|
Previously there was no support at all for zfs file system. With this
change it is now possible to use the resizefs module to grow a zpool to
its maximum partition size on FreeBSD.
LP: #1721243
|
|
This adds a specific IBM Cloud datasource.
IBM Cloud is identified by:
a.) running on xen
b.) one of a LABEL=METADATA disk or a LABEL=config-2 disk with
UUID=9796-932E
The datasource contains its own config-drive reader that reads
only the currently supported portion of config-drive needed for
ibm cloud.
During the provisioning boot, cloud-init is disabled.
See the docstring in DataSourceIBMCloud.py for more more information.
|
|
Support installing and configuring snaps on ubuntu systems. Now,
cloud-config files can provide a list or dictionary of snap:assertions
which will be allow configuration of snapd on a system via 'snap ack'
calls. The snap:commands configuration option supports arbitrary system
commands intended to interact with snappy's cli. This allows users to run
arbitrary snappy commands to create users, download, install and
configure snap packages and snapd.
This branch also deprecates old snappy and snap_config modules leaving
warnings in documentation and runtime for consumers of these modules.
Deprecated snap* modules will be dropped in cloud-init v.18.2 release.
|
|
The command provided to subp can either be a string or a list. This patch
fixes a regression which raised CalledProcessError whenever providing a
string to subp.
LP: #1755965
|
|
When instance meta-data provides hostname information, run
cc_set_hostname in the init-local or init-net stage before network
comes up.
Prevent an initial DHCP request which leaks the stock cloud-image default
hostname before the meta-data provided hostname was processed.
A leaked cloud-image hostname adversely affects Dynamic DNS which
would reallocate 'ubuntu' hostname in DNS to every instance brought up by
cloud-init. These instances would only update DNS to the cloud-init
configured hostname upon DHCP lease renewal.
This branch extends the get_hostname methods in datasource, cloud and
util to limit results to metadata_only to avoid extra cost of querying
the distro for hostname information if metadata does not provide that
information.
LP: #1746455
|
|
This makes 2 changes to shellify's behavior:
a.) raise a TypeError rather than a RuntimeError.
b.) raise a TypeError if input is not a list or tuple.
|
|
Just replace a couple things like:
if b > a and b < c:
with:
if a < b < c:
|
|
This will provide a small performance improvement and shorter code.
|
|
Make puppet installation more configurable by:
- Adding a package_name parameter
- Exposing the puppet configuration and puppet ssl directories as
parameters. These default to the previous values if unset, but can be set
to the new values puppetlabs requires for its puppet 4.x packages. This
way puppet 4 configuration is now possible.
LP: #1446804
|
|
If python starts up without a locale set, then its default encoding
ends up set as ascii. That is not easily changed with the likes of
setlocale. In order to avoid UnicodeDecodeErrors cloud-init will
encode to bytes a python3 string or python2 basestring so that the
values passed to Popen are already bytes.
LP: #1751051
|
|
While addressing undeclared variable in 'cloud-init status', I also fixed
the errors raised by automated code reviews against cloud-init master at
https://lgtm.com/projects/g/cloud-init/cloud-init/alerts
The following items are addressed:
* Fix 'cloud-init status':
* Only report 'running' state when any stage in
/run/cloud-init/status.json has a start time but no finished time.
Default start time to 0 if null.
* undeclared variable 'reason' now reports 'Cloud-init enabled by
systemd cloud-init-generator' when systemd enables cloud-init
* cc_rh_subscription.py util.subp return values aren't set during if an
exception is raised, use ProcessExecution as e instead.
* distros/freebsd.py:
* Drop repetitive looping over ipv4 and ipv6 nic lists.
* Initialize bsddev to 'NOTFOUND' in the event that no devs are
discovered
* declare nics_with_addresses = set() in broader scope outside
check_downable conditional
* cloudinit/util.py: Raise TypeError if mtype parameter isn't string,
iterable or None.
LP: #1744796
|
