| Age | Commit message (Collapse) | Author |
|---|
|
Cloud-init integration tests should not depend on a curtin test ppa.
We already had a cloud-init test ppa for explicitly this purpose.
Just use it instead.
|
|
If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.
This branch does two things:
- Add a fallback_interface property to DatasourceEC2 to support reading the
old .fallback_nic attribute if it was set. New versions will
call net.find_fallback_nic() if there has not been one found.
- Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
metadata
LP: #1732917
|
|
Add opensuse distro support to cc_ntp module.
LP: #1726572
|
|
Without this the entire stage can fail, which will leave an instance
unaccessible.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
Reviewed-by: Ben Cressey <bcressey@amazon.com>
|
|
The etc/hosts file is was not properly setup for openSUSE or SLES
when manage_etc_hosts is set in the config file.
Improve the doc to address the fact that the 'localhost' ip is
distribution dependent (not always 127.0.0.1).
LP: #1731022
|
|
The rh_subscription module doesn't perform null checks when attempting to
iterate on the enabled and disable repos arrays. When only one is
specified, cloud-init fails to run.
|
|
We had used some dns records in i9n.brickies.net (my personal domain)
as a temporary solution until we got names registered in the cloud-init.io
namespace.
We now have CNAME records for:
ubuntu.i9n.cloud-init.io
cloudinit1.cloud-init.io
cloudinit2.cloud-init.io
|
|
3 things here:
a.) link to a bug that we opened to track what made us add
dns entries for hostname of our guests.
b.) spelling fix.
c.) raise an instance of a NotImplementedError not the class.
|
|
Integration test harness changes:
* Enable collection of console log in nocloud-kvm and lxd.
* Collect the console log to results for all test runs.
* change 'tmpfile' to pick name locally instead of using 'mktemp'.
* drop the 'instance' attribute from nocloud-kvm Image and
demote LXDImage.instance to a private attribute.
This is because Images do not actually have instances.
(LXDImage internally uses a booted system to modify the image).
* Add 'TargetBase' as a superclass of Image and Instance providing
implementations of execute, read_data, write_data, pull_file,
and push_file. These all depend on an implementation of _execute.
* Improve '_execute' implementations to support accepting stdin.
* execute supports 'rcs=False' meaning 'do not raise exception'.
* Drop support for pylxd < 2.2. older versions cannot determine
exit code of 'execute', which makes them unusable.
* make NoCloudKVMInstance._execute run as root via sudo. This required
some changes so that 'hostname' could be reverse-looked up in order
to avoid sudo taking a long time (~20 seconds).
* re-use existing ssh connection in nocloud-kvm.
Test changes here:
* do not use /tmp, but rather /var/tmp (LP: #1707222)
* make keys_to_console assertions more strict.
* change user test cases to always add default (ubuntu) user
so that nocloud-kvm's execute which operates over ssh can work.
|
|
VPC instances have the option to specific local only IPv4 addresses. Allow
Ec2Datasource to enable dhcp4 on instances even if local-ipv4s is
configured on an instance.
Also limit network_configuration to only the primary (fallback) nic.
LP: #1728152
|
|
A recent cleanup of the resizefs module broke resizing when a system was
booted with root=PARTUUID=<uuid> and the device /dev/root does not exist.
This path is exposed with the Ubuntu 16.04 but not with Ubuntu 17.10. A
recreate exists under bug 1684869.
LP: #1725067
|
|
There was fallout in a full integration test run from my adding of
test_no_warnings_in_log which asserted that there could not be a WARNING
found in the /var/log/cloud-init.log
This fixes 2 of the cases:
* TestCommandOutputSimple had a valid WARNING written, so adjust its
test case to allow for that.
* TestLxdDir had a valid config in the test but the module would
log a WARNING, so fix the module.
Also updates lxd unit tests to look for WARN themselves.
|
|
Tests are currently run by creating a temporary subclass of each class
and then executing it (in get_suites). When running the tests suite
the output would contain the temporary class name. That was less than
useful, and made batch runs almost impossible to identify which
test case had an error.
This change goes from output of:
FAIL: test_no_warnings_in_log \
(tests.cloud_tests.testcases.get_suite.<locals>.tmp)
To
FAIL: test_no_warnings_in_log \
(tests.cloud_tests.testcases.modules.ntp.TestNtp)
|
|
Fix three things related to the ntp module:
1. Fix invalid cloud-config schema in the integration test which
provided empty dicts instead of emptylists for pools and servers
2. Correct logic in the ntp module to allow support for the minimal
cloud-config 'ntp:' without raising a RuntimeError. Docs and schema
definitions already describe that cloud-config's ntp can be empty.
An ntp configuration with neither pools nor servers will be
configured with a default set of ntp pools. As such, the ntp module
now officially allows the following ntp cloud-configs:
- ntp:
- ntp: {}
- ntp:
servers: []
pools: []
3. Add a simple unit test which validates all cloud-config provided to
our integration tests to ensure it adheres to any defined module
schema so as more jsonschema definitions are added, we validate our
integration test configs.
LP: #1724951
|
|
When operating in expected path, cloud-init should avoid logging with
warning. That causes 'WARNING' messages in /var/log/cloud-init.log.
By default, warnings also go to the console.
Since jsonschema is a optional dependency, and not present on xenial
and zesty, cloud-init should not warn there.
Also here:
* Add a test to integration tests to assert that there are no
warnings in /var/log/cloud-init.log.
* Update one integration test that did show warning and the related
documentation and examples.
LP: #1724354
|
|
Update unit tests to pass a 0 instead of 'off' to validate that network
state is properly written.
|
|
Update network_state to store the bridge_stp value as a boolean.
The various renderers then can map the boolean value to the correct
output as needed; eni uses 'on/off', sysconfig uses 'yes/no' and
netplan will use the boolean directly.
Update unittest values for sysconfig and netplan. Both contained the
network_state string value which resulted in not correctly enable/disable
STP in the target system.
Update network_state comment (fd -> forward-delay, add stp as boolean) on
bridge commands to match the expected format of a netplan bridge command.
LP: #1721157
|
|
This adds a config module so support for adding zypper repositories
via cloud-config.
LP: #1718675
|
|
Combine the configs and testcases directories, so all files are
together in one place. Update the test config location as well.
|
|
Systems that used systemd-networkd's dhcp client would not be able to get
information on the Azure endpoint (placed in Option 245) or the CloudStack
server (in 'server_address').
The change here supports reading these files in /run/systemd/netif/leases.
The files declare that "This is private data. Do not parse.", but at this
point we do not have another option.
LP: #1718029
|
|
This removes shlex and converts the subprocess commands to use a
list over a string.
|
|
Previously we had hard coded paths in /sbin for the udevadm and modprobe
programs invoked by AltCloud. Its more flexible to expect the PATH to
be set correctly.
Debian: #852564
|
|
DataSourceOVF attempts to find iso files via walking os.listdir('/dev/')
which is far too wide. This approach is too invasive and can sometimes
race with systemd attempting to fsck and mount devices.
Instead, utilize cloudinit.util.find_devs_with to filter devices by
criteria (which uses blkid under the covers). This results in fewer
attempts to mount block devices which do not contain iso filesystems.
Unittest changes include:
- cloudinit.tests.helpers; introduce add_patch() helper
- Add unittest coverage for DataSourceOVF use of transport_iso9660
LP: #1718287
|
|
The bootcmd test was leaving files in the tmpdir named
ci-FakeExtendedTempFile.XXXXXX.
This cleans those up.
|
|
Create a copy of each modules schema attribute when generating sphinx docs
to avoid altering the actual module dict in memory. This avoids illegible
rendering of module examples and distros where each character of a list
was represented on a separate line by itself.
Fixes ntp, resizefs, runcmd and bootcmd docs.
|
|
The variable DI_EC2_STRICT_ID_DEFAULT was not being set in unit tests
so when 16.04 built, which changed that setting in patches the tests
would unexpectedly fail.
|
|
This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data that was read in
self.userdata_raw, rather than self.userdata. That is consistent
with other datasources and ulitimately how it was before the refactor.
The main is updated to address the fact that user-data is binary data
and may not be able to be printed.
LP: #1717598
|
|
Add a new collect-logs sub command to the cloud-init CLI. This script
will collect all logs pertinent to a cloud-init run and store them in a
compressed tar-gzipped file. This tarfile can be attached to any
cloud-init bug filed in order to aid in bug triage and resolution.
A cloudinit.apport module is also added that allows apport interaction.
Here is an example bug filed via ubuntu-bug cloud-init: LP: #1716975.
Once the apport launcher is packaged in cloud-init, bugs can be filed
against cloud-init with the following command:
ubuntu-bug cloud-init
LP: #1607345
|
|
A regression in 'get_latest_lease' made it ignore files starting with
'dhclient-' rather than just 'dhclient.'. The fix here is to allow those
files to be considered.
There is a lot more we could do here to better ensure that we pick the
most recent lease, but this change fixes the regression.
LP: #1717147
|
|
As root user, os.access(<path>, os.W_OK) will always return True so that
path will never get executed. Also avoid a warning if the root is
overlayroot, which is the common case on a MAAS booted 'ephemeral' system.
|
|
The NoCloud KVM platform includes:
* Downloads daily Ubuntu images using streams and store in
/srv/images
* Image customization, if required, is done using
mount-image-callback otherwise image is untouched
* Launches KVM via the xkvm script, a wrapper around
qemu-system, and sets custom port for SSH
* Generation and inject an SSH (RSA 4096) key pair to use for
communication with the guest to collect test artifacts
* Add method to produce safe shell strings by base64 encoding
the command
Additional Changes:
* Set default backend to use LXD
* Verify not running script as root in order to prevent images
from becoming owned by root
* Removed extra quotes around that were added when collecting
the cloud-init version from the image
* Added info about each release as previously the lxd backend
was able to query that information from pylxd image info,
however, other backends will not be able to obtain the same
information as easily
|
|
Modules can optionally define a list of supported distros on which they can run
by declaring a distros attribute in the cc_*py module. This branch fixes
handling of cloudinit.stages.Modules.run_section. The behavior of run_section
is now the following:
- always run a module if the module doesn't declare a distros attribute
- always run a module if the module declares distros = [ALL_DISTROS]
- skip a module if the distribution on which we run isn't in module.distros
- force a run of a skipped module if unverified_modules configuration contains
the module name
LP: #1715738
LP: #1715690
|
|
Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.
This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.
LP: #1462693
|
|
If a string is passed to execute, then invoke 'bash', '-c',
'string'. That allows the less verbose execution of simple
commands:
image.execute("ls /run")
compared to the more explicit but longer winded:
image.execute(["ls", "/run"])
If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.
|
|
Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.
This branch also does the following:
- drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
option only created the specified directory and didn't make use of
that directory for any resize operations.
- Drop yaml.dumps calls from schema documentation generation to avoid
yaml import costs on module load
- Add __doc__ = get_schema_doc(schema) definitions it each module to
supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
cc_resizefs
- Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
contain more than one example
|
|
For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.
This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.
Modify the code to read the customization configuration and return the
converted data.
Added few tests.
LP: #1675063
|
|
This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:
a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)
b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.
The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.
Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.
LP: #1715128
|
|
During boot, the usage of /tmp is not safe. In systemd systems,
systemd-tmpfiles-clean may run at any point and clear out a temp file
while cloud-init is using it. The solution here is to use
/run/cloud-init/tmp.
LP: #1707222
|
|
OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.
An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.
LP: #1715241
|
|
This missed mock in test_openstack resulted in a costly unit test timeout.
LP: #1714376
|
|
This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.
|
|
The ubuntu-init-switch module allowed the use to launch an instance that
was booted with upstart and have it switch its init system to systemd and
then reboot itself. It was only useful for the time period when Ubuntu was
transitioning to systemd but only produced images using upstart.
Also, do not run setup with --init-system=upstart. This means that by
default, debian packages built with packages/bddeb will not have upstart
unit files included. No other removal is done here.
|
|
DataSourceEc2 behavior changed to first check a minimum acceptable
metadata version uri http://169.154.169.254/<min_version>/instance-id,
retrying on 404, until the metadata service is available. After the
metadata service is up, the datasource inspects preferred
extended_metadata_versions for availability. Unit tests only mocked the
preferred extended_metadata_version so all Ec2 tests were retrying
attempts against
http://169.254.169.254/meta-data/<min-version>/instance-id adding a lot of
time cost to the unit test runs.
This branch uses httpretty to properly mock the following:
- 404s from metadata on undesired extended_metadata_version test routes
- https://169.254.169.254/meta-data/2016-09-02/instance-id
- full metadata dictionary represented on min_metadata_version
- https://169.254.169.254/meta-data/2016-09-02/*
The branch also tightens httpretty to raise a MockError for any URL which
isn't mocked via httpretty.HTTPretty.allow_net_connect=False.
LP: #1714117
|
|
Currently the cloud-init default locale (en_US.UTF-8) is set by
the base datasource class. This patch allows a distro to overide
the fallback value with one that's available in the distro but continues
to respect an image which has preconfigured a locale.
- Distro object now has a get_locale method which will return a
preconfigure locale setting by checking the distros locale system
configuration file. If not set or not present, return the default
locale of en_US.UTF-8 which retains behavior of all previous cloud-init
releases.
- Apply locale now handles regenerating locales or system configuration
files as needed.
- Adjust apply_locale logic to skip locale-regen if the specified LANG
value is C.UTF-8,C, or POSIX; they do not require regeneration.
- Further add unittests to exercise the default paths for Ubuntu and
non-ubuntu paths to validate they get the LANG expected.
|
|
test_set_locale_sles and test_set_locale_sles_default were incorrectly
testing for truth of <distro_object>.uses_systemd rather than calling
that function and checking its result.
The error was only seen if the system running the tests was not using
systemd.
|
|
This gets initial opensuse and SLES support back to a working state.
Still missing is more complete network file writing and unit tests.
|
|
DataSourceEc2 now parses the metadata for each nic to determine if
configured for ipv6 and/or ipv4 addresses. In AWS for metadata version
2016-09-02, nics configured for ipv4 or ipv6 addresses will have non-zero
values stored in metadata at network/interfaces/macs/<MAC>/public-ipv4 or
ipv6s respectively. Those metadata files are only non-zero when an ipv4 or
ipv6 ip is associated to the specific nic. A new
DataSourceEc2.network_config property is added which parses the metadata
and renders a network version 1 dictionary representing both dhcp4 and
dhcp6 configuration for associated nics.
The network configuration returned from the datasource will also 'pin' the
nic name to the name presented on the instance for each nic.
LP: #1639030
|
|
Currently the python logging module will default to a local time which may
contain an TZ offset in the values it produces, but the logged time format
does not contain the offset. Switching to UTC time for logging produces
consistent values in the cloud-init.log file and avoids issues when the
timezone is changed during boot.
LP: #1713158
|
|
A patch to allow scripts missing a #! to run by using shell=True was
proposed but rejected. Instead we emit a log message to help the user
understand what went wrong.
|
|
In an effort to save file load cost during system boot, certain
subcommands, analyze and devel, do not get loaded unless the subcommand is
specified on the commandline. Because setup.py entrypoint for cloud-init
script doesn't specify sysv_args parameter when calling the CLI's main()
we need main to read sys.argv into sysv_args so our subparser loading
continues to work.
LP: #1712676
|
