| Age | Commit message (Collapse) | Author |
|---|
|
This change will enable azure vms to report provisioning has completed
twice, first to tell the fabric it has completed then a second time to
enable customer settings. The datasource for the second provisioning is
the Instance Metadata Service (IMDS),and the VM will poll indefinitely for
the new ovf-env.xml from IMDS.
This branch introduces EphemeralDHCPv4 which encapsulates common logic
used by both DataSourceEc2 an DataSourceAzure for temporary DHCP
interactions without side-effects.
LP: #1734991
|
|
Resize of btrfs fails if the mount point for the file system we are trying
to resize, i.e. the root of the filesystem is read only. With this change
we use a known (currently snapper specific) rw location to work around a
flaw that blocks resizing of the ro filesystem.
LP: #1734787
|
|
Network configuration in OpenNebula would only work if the host correctly
guessed the names of the devices in the guest. OpenNebula provided data
in its context.sh like 'ETH0_NETWORK', but if the guest named devices
differently then results were not predictable. This would occur with
Predictable Network Interface Names. To address this,
newer versions (of OpenNebula provide the mac address ETH0_MAC.
This function is present in 4.14 and documented officially in 5.0 docs.
This provides support for reading the mac addresses from the context.sh.
It also fixes cases where context.sh provided a field (ETH0_NETWORK
or ETH0_MASK) with a empty string. Previously the empty string would
be used rather than falling back to the default.
LP: #1719157, #1716397, #1736750
|
|
The previous commit added a test that would attempt to create and use
/run/cloud-init/. This just modifies it to use a temp dir instead.
|
|
The behavior changes and improvements include:
- Only import keys into the default user that contain the name of the
default user ('ubuntu', or 'centos') or that contain 'cloudinit'.
- Use instance or project level keys based on GCE convention.
- Respect expiration time when keys are set.
Do not import expired keys.
- Support ssh-keys in project level metadata (the GCE default).
As part of this change, we also update the request header when talking
to the metadata server based on the documentation:
https://cloud.google.com/compute/docs/storing-retrieving-metadata#querying
LP: #1670456, #1707033, #1707037, #1707039
|
|
When subp raised a ProcessExecutionError, that exception's stderr and
stdout might end up being the string '-' rather than bytes.
This mean that:
try:
subp(mycommand, decode=False)
except ProcessExecutionError as e:
pass
Would have 'e.stdout' set to '-' while the caller would expect bytes.
Also reduce the try/except block in subp to a specifically the two lines
that may raise an OSError.
|
|
New mkfs.vfat and fatlabel tools included in the dosfsutils package no
longer support creating vfat disks with lowercase labels. They silently
default to an all uppercase label eg CONFIG-2 instead of config-2. This
change makes cloud-init handle either upper or lower case.
LP: #1598783
|
|
This issue was first identified when manual_cache_clean was set, as
ds-identify would write /run/cloud-init/cloud.cfg with
# manual_cache_clean
that would generate a warning as cloud-init expected to load a dict.
Any other "empty" config would also log such a warning.
Also fix reading of di_report to allow it to be None, as ds-identify
would write:
di_report:
# manual_cache_clean
which reads as 'di_report: None' rather than di_report: {}.
LP: #1742479
|
|
This stores a hash of the OAuth tokens as an 'id' for the maas
datasource. Since new instances get new tokens created and those tokens
are written by curtin into datasource system config this will provide
a way to identify a new "instance" (install).
LP: #1712680
|
|
This simply increases test coverage in rendering of network config
for SUSE and SLES.
|
|
This fixes a traceback when attempting to bounce the network after
hostname resets.
In artful and bionic ifupdown package is no longer installed in default
cloud images. As such, Azure can't use those tools to bounce the network
informing DDNS about hostname changes. This doesn't affect DDNS updates
though because systemd-networkd is now watching hostname deltas and with
default behavior to SendHostname=True over dhcp for all hostname updates
which publishes DDNS for us.
LP: #1722668
|
|
The cli help docs and argument parser allow the 'init' mode value
which caused a traceback.
Fix the cli to support 'init', 'config' and 'final' modes for the
cloud-init modules subcommand.
Add a check in the cli to raise a ValueError if a new
subcommand ends up allowing an unsupported/unimplemented modes.
Drive by unit test additions for a bit better coverage of error
handling.
LP: #1736600
|
|
The previous OVF datasource change added a debug message that referenced
an un-used variable. The failure path would be triggered if an image was
booted with a iso9660 filesystem attached to a device that was not a
cdrom.
A unit test is added for the specific failure found.
Additional safety to avoid 'cidata' labels is also added to the OVF
checker.
LP: #1737704
|
|
The instance identity document is a better source for region information,
partly because region isn't actually in meta-data at all, only
availability-zone, which happens to be named similarly.
Reviewed-by: Ethan Faust <efaust@amazon.com>
Reviewed-by: Cyle Riggs <cyler@amazon.com>
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
[ajorgens@amazon.com: rebase onto 0.7.9]
[ajorgens@amazon.com: changes per merge proposal discussions]
|
|
Make sure that some temporary files used by the config drive tests get
cleaned up properly.
|
|
Previously the OVF transport would not be identified except for when
config files set 'ovf_vmware_guest_customization'. It would also
return DS_MAYBE almost always.
The change here is to add support to ds-identify for storing the
iso9660 filesystems that it finds (ISO9660_DEVS). Then the OVF check
will check that the iso9660 filesystem has ovf-env.xml on it. The least
wonderful part of this is that the check is done by 'grep' for case
insensitive ovf-env.xml.
Future improvement would be to identify VMware's OVF by label or UUID
so we could avoid the grep.
LP: #1731868
|
|
In the VMware customization workflow, we have some options for the user
to upload scripts for additional customization. Based on user request,
those custom scripts can be either run before regular customization or
after. For post customization scripts, we decide whether to run the scripts
just after customization or post system reboot.
|
|
The motivation for this is that
a.) 1.7.1 runs with python 3.6 (bionic)
b.) we want to run pylint on tests/ and tools for the same reasons
that we want to run it on cloudinit/
The changes are described below.
- Update tox.ini to invoke pylint v1.7.1.
- Modify .pylintrc generated-members ignore mocked object members (m_.*)
- Replace "dangerous" params defaulting to {}
- Fix up cloud_tests use of platforms
- Cast some instance objects to with dict()
- Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated)
- Update use of assertEqual(<boolean>, value) to assert<Boolean>(value)
- replace depricated assertRegexp -> assertRegex
- Remove useless test-class calls to super class
- Assign class property accessors a result and use it
- Fix missing class member in CepkoResultTests
- Fix Cheetah test import
|
|
Each DataSource subclass must define its own get_data method. This branch
formalizes our DataSource class to require that subclasses define an
explicit dsname for sourcing cloud-config datasource configuration.
Subclasses must also override the _get_data method or a
NotImplementedError is raised.
The branch also writes /run/cloud-init/instance-data.json. This file
contains all meta-data, user-data and vendor-data and a standardized set
of metadata keys in a json blob which other utilities with root-access
could make use of. Because some meta-data or user-data is potentially
sensitive the file is only readable by root.
Generally most metadata content types should be json serializable. If
specific keys or values are not serializable, those specific values will
be base64encoded and the key path will be listed under the top-level key
'base64-encoded-keys' in instance-data.json. If json writing fails due to
other TypeErrors or UnicodeDecodeErrors, a warning log will be emitted to
/var/log/cloud-init.log and no instance-data.json will be created.
|
|
The 'cloud-init clean' command allows a user or script to clear cloud-init
artifacts from the system so that cloud-init sees the system as
unconfigured upon reboot. Optional parameters can be provided to remove
cloud-init logs and reboot after clean.
The 'cloud-init status' command allows the user or script to check whether
cloud-init has finished all configuration stages and whether errors
occurred. An optional --wait argument will poll on a 0.25 second interval
until cloud-init configuration is complete. The benefit here is scripts
can block on cloud-init completion before performing post-config tasks.
|
|
There is a race condition where our sandboxed dhclient properly writes a
lease file but has not yet written a pid file. If the sandbox temporary
directory is torn down before the dhclient subprocess writes a pidfile
DataSourceEc2Local gets a traceback and the instance will fallback to
DataSourceEc2 in the init-network stage. This wastes boot cycles we'd
rather not spend.
Fix handling of sandboxed dhclient to wait for both pidfile and leasefile
before proceding. If either file doesn't show in 5 seconds, log a warning
and return empty lease results {}.
LP: #1735331
|
|
Currently when dns and dns search info is provided, it is not rendered
when outputting to sysconfig format.
This patch causes the DNS and DOMAIN lines to be written out rendering
sysconfig.
LP: #1705804
|
|
If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.
This branch does two things:
- Add a fallback_interface property to DatasourceEC2 to support reading the
old .fallback_nic attribute if it was set. New versions will
call net.find_fallback_nic() if there has not been one found.
- Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
metadata
LP: #1732917
|
|
Add opensuse distro support to cc_ntp module.
LP: #1726572
|
|
Without this the entire stage can fail, which will leave an instance
unaccessible.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
Reviewed-by: Ben Cressey <bcressey@amazon.com>
|
|
The etc/hosts file is was not properly setup for openSUSE or SLES
when manage_etc_hosts is set in the config file.
Improve the doc to address the fact that the 'localhost' ip is
distribution dependent (not always 127.0.0.1).
LP: #1731022
|
|
The rh_subscription module doesn't perform null checks when attempting to
iterate on the enabled and disable repos arrays. When only one is
specified, cloud-init fails to run.
|
|
VPC instances have the option to specific local only IPv4 addresses. Allow
Ec2Datasource to enable dhcp4 on instances even if local-ipv4s is
configured on an instance.
Also limit network_configuration to only the primary (fallback) nic.
LP: #1728152
|
|
A recent cleanup of the resizefs module broke resizing when a system was
booted with root=PARTUUID=<uuid> and the device /dev/root does not exist.
This path is exposed with the Ubuntu 16.04 but not with Ubuntu 17.10. A
recreate exists under bug 1684869.
LP: #1725067
|
|
There was fallout in a full integration test run from my adding of
test_no_warnings_in_log which asserted that there could not be a WARNING
found in the /var/log/cloud-init.log
This fixes 2 of the cases:
* TestCommandOutputSimple had a valid WARNING written, so adjust its
test case to allow for that.
* TestLxdDir had a valid config in the test but the module would
log a WARNING, so fix the module.
Also updates lxd unit tests to look for WARN themselves.
|
|
Fix three things related to the ntp module:
1. Fix invalid cloud-config schema in the integration test which
provided empty dicts instead of emptylists for pools and servers
2. Correct logic in the ntp module to allow support for the minimal
cloud-config 'ntp:' without raising a RuntimeError. Docs and schema
definitions already describe that cloud-config's ntp can be empty.
An ntp configuration with neither pools nor servers will be
configured with a default set of ntp pools. As such, the ntp module
now officially allows the following ntp cloud-configs:
- ntp:
- ntp: {}
- ntp:
servers: []
pools: []
3. Add a simple unit test which validates all cloud-config provided to
our integration tests to ensure it adheres to any defined module
schema so as more jsonschema definitions are added, we validate our
integration test configs.
LP: #1724951
|
|
Update unit tests to pass a 0 instead of 'off' to validate that network
state is properly written.
|
|
Update network_state to store the bridge_stp value as a boolean.
The various renderers then can map the boolean value to the correct
output as needed; eni uses 'on/off', sysconfig uses 'yes/no' and
netplan will use the boolean directly.
Update unittest values for sysconfig and netplan. Both contained the
network_state string value which resulted in not correctly enable/disable
STP in the target system.
Update network_state comment (fd -> forward-delay, add stp as boolean) on
bridge commands to match the expected format of a netplan bridge command.
LP: #1721157
|
|
This adds a config module so support for adding zypper repositories
via cloud-config.
LP: #1718675
|
|
Systems that used systemd-networkd's dhcp client would not be able to get
information on the Azure endpoint (placed in Option 245) or the CloudStack
server (in 'server_address').
The change here supports reading these files in /run/systemd/netif/leases.
The files declare that "This is private data. Do not parse.", but at this
point we do not have another option.
LP: #1718029
|
|
Previously we had hard coded paths in /sbin for the udevadm and modprobe
programs invoked by AltCloud. Its more flexible to expect the PATH to
be set correctly.
Debian: #852564
|
|
DataSourceOVF attempts to find iso files via walking os.listdir('/dev/')
which is far too wide. This approach is too invasive and can sometimes
race with systemd attempting to fsck and mount devices.
Instead, utilize cloudinit.util.find_devs_with to filter devices by
criteria (which uses blkid under the covers). This results in fewer
attempts to mount block devices which do not contain iso filesystems.
Unittest changes include:
- cloudinit.tests.helpers; introduce add_patch() helper
- Add unittest coverage for DataSourceOVF use of transport_iso9660
LP: #1718287
|
|
The bootcmd test was leaving files in the tmpdir named
ci-FakeExtendedTempFile.XXXXXX.
This cleans those up.
|
|
Create a copy of each modules schema attribute when generating sphinx docs
to avoid altering the actual module dict in memory. This avoids illegible
rendering of module examples and distros where each character of a list
was represented on a separate line by itself.
Fixes ntp, resizefs, runcmd and bootcmd docs.
|
|
The variable DI_EC2_STRICT_ID_DEFAULT was not being set in unit tests
so when 16.04 built, which changed that setting in patches the tests
would unexpectedly fail.
|
|
This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data that was read in
self.userdata_raw, rather than self.userdata. That is consistent
with other datasources and ulitimately how it was before the refactor.
The main is updated to address the fact that user-data is binary data
and may not be able to be printed.
LP: #1717598
|
|
Add a new collect-logs sub command to the cloud-init CLI. This script
will collect all logs pertinent to a cloud-init run and store them in a
compressed tar-gzipped file. This tarfile can be attached to any
cloud-init bug filed in order to aid in bug triage and resolution.
A cloudinit.apport module is also added that allows apport interaction.
Here is an example bug filed via ubuntu-bug cloud-init: LP: #1716975.
Once the apport launcher is packaged in cloud-init, bugs can be filed
against cloud-init with the following command:
ubuntu-bug cloud-init
LP: #1607345
|
|
A regression in 'get_latest_lease' made it ignore files starting with
'dhclient-' rather than just 'dhclient.'. The fix here is to allow those
files to be considered.
There is a lot more we could do here to better ensure that we pick the
most recent lease, but this change fixes the regression.
LP: #1717147
|
|
As root user, os.access(<path>, os.W_OK) will always return True so that
path will never get executed. Also avoid a warning if the root is
overlayroot, which is the common case on a MAAS booted 'ephemeral' system.
|
|
Modules can optionally define a list of supported distros on which they can run
by declaring a distros attribute in the cc_*py module. This branch fixes
handling of cloudinit.stages.Modules.run_section. The behavior of run_section
is now the following:
- always run a module if the module doesn't declare a distros attribute
- always run a module if the module declares distros = [ALL_DISTROS]
- skip a module if the distribution on which we run isn't in module.distros
- force a run of a skipped module if unverified_modules configuration contains
the module name
LP: #1715738
LP: #1715690
|
|
Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.
This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.
LP: #1462693
|
|
Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.
This branch also does the following:
- drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
option only created the specified directory and didn't make use of
that directory for any resize operations.
- Drop yaml.dumps calls from schema documentation generation to avoid
yaml import costs on module load
- Add __doc__ = get_schema_doc(schema) definitions it each module to
supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
cc_resizefs
- Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
contain more than one example
|
|
For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.
This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.
Modify the code to read the customization configuration and return the
converted data.
Added few tests.
LP: #1675063
|
|
This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:
a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)
b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.
The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.
Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.
LP: #1715128
|
|
During boot, the usage of /tmp is not safe. In systemd systems,
systemd-tmpfiles-clean may run at any point and clear out a temp file
while cloud-init is using it. The solution here is to use
/run/cloud-init/tmp.
LP: #1707222
|
