| Age | Commit message (Collapse) | Author |
|---|
|
The 'merged_cfg' key introduced in 71af48d was incorrectly referenced
to as 'ci_cfg' in the json validation test for ec2.
|
|
Cloud-config userdata provided as jinja templates are now distro,
platform and merged cloud config aware. The cloud-init query command
will also surface this config data.
Now users can selectively render portions of cloud-config based on:
* distro name, version, release
* python version
* merged cloud config values
* machine platform
* kernel
To support template handling of this config, add new top-level
keys to /run/cloud-init/instance-data.json.
The new 'merged_cfg' key represents merged cloud config from
/etc/cloud/cloud.cfg and /etc/cloud/cloud.cfg.d/*.
The new 'sys_info' key which captures distro and platform
info from cloudinit.util.system_info.
Cloud config userdata templates can render conditional content
based on these additional environmental checks such as the following
simple example:
```
## template: jinja
#cloud-config
runcmd:
{% if distro == 'opensuse' %}
- sh /custom-setup-sles
{% elif distro == 'centos' %}
- sh /custom-setup-centos
{% elif distro == 'debian' %}
- sh /custom-setup-debian
{% endif %}
```
To see all values: sudo cloud-init query --all
Any keys added to the standardized v1 keys are guaranteed to not
change or drop on future released of cloud-init. 'v1' keys will be retained
for backward-compatibility even if a new standardized 'v2' set of keys
are introduced
The following standardized v1 keys are added:
* distro, distro_release, distro_version, kernel_version, machine,
python_version, system_platform, variant
LP: #1865969
|
|
Adapt the test to the new capitalization introduced in
8116493950e7c47af0ce66fc1bb5d799ce5e477a.
|
|
cloud-init has moved to cc_snap module and a top-level
config key 'snap'. cc_snap_config was deprecated in
cloud-init version 18.2
Co-authored-by: Daniel Watkins <daniel@daniel-watkins.co.uk>
|
|
* cc_snappy: remove deprecated module
* cloud_tests: remove cc_snappy tests (and references)
This module was deprecated in favor of cc_snap in cloud-init v.18.2
|
|
The apt_pipelining test-cases were broken but until cloud-init
changed it's default behavior to not disable, these silently passed
as both only ever checked if pipelinging was disabled.
First, the tests used the 'apt' namespace, which is not for
configuring pipelining, rather that requires 'apt_pipelining'
as the namespace.
Second, the 'os' variant needs to check that cloud-init does not
write a configuration file; it was a copy-and-paste error from the
disable test-case.
This branch fixes the config and collection to validate both
scenarios.
|
|
LP: #1797231
|
|
Make integration test for flexible using regexp in case disk changes.
LP: #1797199
|
|
Add the following instance-data.json standardized keys:
* v1._beta_keys: List any v1 keys in beta development,
e.g. ['subplatform'].
* v1.public_ssh_keys: List of any cloud-provided ssh keys for the
instance.
* v1.platform: String representing the cloud platform api supporting the
datasource. For example: 'ec2' for aws, aliyun and brightbox cloud
names.
* v1.subplatform: String with more details about the source of the
metadata consumed. For example, metadata uri, config drive device path
or seed directory.
To support the new platform and subplatform standardized instance-data,
DataSource and its subclasses grew platform and subplatform attributes.
The platform attribute defaults to the lowercase string datasource name at
self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive
datasources.
The subplatform attribute calls a _get_subplatform method which will
return a string containing a simple slug for subplatform type such as
metadata, seed-dir or config-drive followed by a detailed uri, device or
directory path where the datasource consumed its configuration.
As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata
have been refactored for a few reasons:
- crawl_metadata is now a read-only operation, persisting no attributes on
the datasource instance and returns a dictionary of consumed metadata.
- crawl_metadata now closely represents the raw stucture of the ec2
metadata consumed, so that end-users can leverage public ec2 metadata
documentation where possible.
- crawl_metadata adds a '_metadata_api_version' key to the crawled
ds.metadata to advertise what version of EC2's api was consumed by
cloud-init.
- _get_data now does all the processing of crawl_metadata and saves
datasource instance attributes userdata_raw, metadata etc.
Additional drive-bys:
* unit test rework for test_altcloud and test_azure to simplify mocks
and make use of existing util and test_helpers functions.
|
|
Commit d3e803ad316e6796e5d83e7e8f8f4f7224b92df9 added deb-src comments to
the cloud-init apt templates. This doubled the number of matching entries
seen in /etc/apt/sources.list in apt_configure_primary integration test.
This test was really asserting that GaTech urls were present in
/etc//apt/sources.list instead of archive.ubuntu.com. Fix the test to be a
bit more flexible in case cloud-init changes its bas apt template again.
|
|
Individual skipTest or setUp SkipTest will still launch the instance.
This allows us to stop the running of the instance so we don't
waste cycles or boot systems that are known to fail.
Also replace remaining unittest usage in tests/cloud_tests/
with unittest2.
|
|
Skip lxd tests on cosmic for two reasons:
a.) bug 1795036 - 'lxd init' fails on cosmic kernel.
b.) apt install lxd installs via snap which can be slow
as that will download core snap and lxd.
|
|
Git commitish fc4b966ba928b30b1c586407e752e0b51b1031e8 changed integration
test dependencies from unittest to unittest2. Use unittest2.SkipTest in
test_chrony to avoid causing tracebacks.
|
|
Relax expectation on path to lxc and lxd. The deb path still does
install them in /usr/bin/ but that is overly pedantic.
Add a 'lxd waitready' (present since lxd 0.5) to wait until lxd
is ready before operating on it.
|
|
Commitish c7555762f3a30190ce7726b4d013bc3e83c7e4b6 changed the variable
names in instance-data.json from hyphenated to underscore delimited. In
the shuffle, meta-data -> meta_data was missed.
|
|
Cloud-init caches any cloud metadata crawled during boot in the file
/run/cloud-init/instance-data.json. Cloud-init also standardizes some of
that metadata across all clouds. The command 'cloud-init query' surfaces a
simple CLI to query or format any cached instance metadata so that scripts
or end-users do not have to write tools to crawl metadata themselves.
Since 'cloud-init query' is runnable by non-root users, redact any
sensitive data from instance-data.json and provide a root-readable
unredacted instance-data-sensitive.json. Datasources can now define a
sensitive_metadata_keys tuple which will redact any matching keys
which could contain passwords or credentials from instance-data.json.
Also add the following standardized 'v1' instance-data.json keys:
- user_data: The base64encoded user-data provided at instance launch
- vendor_data: Any vendor_data provided to the instance at launch
- underscore_delimited versions of existing hyphenated keys:
instance_id, local_hostname, availability_zone, cloud_name
|
|
Allow users to provide '## template: jinja' as the first line or their
#cloud-config or custom script user-data parts. When this header exists,
the cloud-config or script will be rendered as a jinja template.
All instance metadata keys and values present in
/run/cloud-init/instance-data.json will be available as jinja variables
for the template. This means any cloud-config module or script can
reference any standardized instance data in templates and scripts.
Additionally, any standardized instance-data.json keys scoped below a
'<v#>' key will be promoted as a top-level key for ease of reference in
templates. This means that '{{ local_hostname }}' is the same as using the
latest '{{ v#.local_hostname }}'.
Since instance-data is written to /run/cloud-init/instance-data.json, make
sure it is persisted across reboots when the cached datasource opject is
reloaded.
LP: #1791781
|
|
The snap test requires access to a proxy and currently the integration
tests do not handle this scenario. I am disabling the test until I can
loop back around and fix this.
|
|
The snap test requires access to a proxy and currently the integration
tests do not handle this scenario. I am disabling the test untill I can
loop back around and fix this.
The write_files test, specifically, the binary test is failing on cosmic
because the "binary" file we were writting was not a complete elf
executable, but we expected 'file' to identify it as such.
The change here is to simply use some 24 bytes of random, non-utf data
and check that file was written correctly via expected checksum.
|
|
The salt minion integration test as we had it did not do a whole lot
more than the unit tests on that module did. Additionally, it caused
some transient failures at least in Ubuntu 18.04.
At a future date we may choose to add an integration test that installs
salt-minion and salt server and configures it to be a better test.
LP: #1778737
|
|
In ubuntu, the salt-minion package version 2017.7.4+dfsg1-1 or later
automatically moves any seed keys from /etc/salt/pki/minion/ to
/var/lib/salt/pki/minion/. Fix integration tests to collect
either files in either /etc/salt/pki/minion/ or
/var/lib/salt/pki/minion/.
|
|
A fix for chrony support per LP: #1589780 is not expected in Artful or
older series. Skip the chrony suite of tests when running on a container
and ubuntu series represented is <= artful as errors are expected.
|
|
package_update_upgrade_install was failing as htop is now included in
Bionic images. Switch this test to install 'sl' instead.
ca_certs integration test fails on cert_count test because bionic
update-ca-certificates on bionic generates less symlinks for a given cert.
Integration tests now collect dpkg-query --show output on every instance.
Add a new assertPackageInstalled helper method which finds the package or
package version installed on the instance.
Adapt existing byobu, package_update_upgrade_install, ntp and salt_minion
tests to use assertPackageInstalled method.
LP: #1769985
|
|
This enables warnings produced by pylint for unused variables (W0612),
and fixes the existing errors.
|
|
Python has deprecated these invalid string literals now
https://bugs.python.org/issue27364
and pycodestyle is identifying them with a W605 warning.
https://github.com/PyCQA/pycodestyle/pull/676
So basically, any use of \ not followed by one of [\'"abfnrtv]
or \ooo (octal) \xhh (hex) or a newline is invalid. This is most
comomnly seen for us in regex. To solve, you either:
a.) use a raw string r'...'
b.) correctly escape the \ that was not intended to be interpreted.
|
|
Add a base NTP client configuration dictionary and allow Distro
specific changes to be merged. Add a select client function which
implements logic to preferr installed clients over clients which
need to be installed. Also allow distributions to override the
cloud-init defaults.
LP: #1749722
|
|
Fix integraiton test logic for ec2 to look for network and
availability-zone data under the key path
'ds'=>'meta-data' instead of just 'ds' when parsing instance-data.json.
|
|
LP: #1420018
|
|
Support installing and configuring snaps on ubuntu systems. Now,
cloud-config files can provide a list or dictionary of snap:assertions
which will be allow configuration of snapd on a system via 'snap ack'
calls. The snap:commands configuration option supports arbitrary system
commands intended to interact with snappy's cli. This allows users to run
arbitrary snappy commands to create users, download, install and
configure snap packages and snapd.
This branch also deprecates old snappy and snap_config modules leaving
warnings in documentation and runtime for consumers of these modules.
Deprecated snap* modules will be dropped in cloud-init v.18.2 release.
|
|
Previously the module was not working under FreeBSD due to a different
package name and some different paths. The module now has OS specific
default values which can even be customized via corresponding cloud config
variables.
LP: #1721503
|
|
While salt grains can be configured in the minion config file, it is
usually better to configure it in the /etc/salt/grains file.
This allows that to be done.
|
|
This adds collection a gzip compressed systemd journal on systemd systems.
The file can later be reviewed with:
zcat system.journal.gz > system.journal
journalctl --file=system.journal [-o short-monotonic ..]
To support this:
* modify test harness infrastructure to not assume content is utf-8.
* fix lxd platform to support make '_execute' return bytes rather
than a string. https://github.com/lxc/pylxd/issues/268
Also switched the base collectors to use /bin/sh as others already did.
|
|
Tests run on EC2 would successfully resolve the ipv4 dns address
and that caused false positives on failure reports.
Basically, dns lookup of 172.16.15.14 would return
ip-172-16-15-14.us-east-2.compute.internal.
which then shows up in the ntpq output unless you provide -n.
|
|
Due to additional platforms getting added this test was not taking into
account platform specific mirrors nor was it checking that no additional
entries were added.
|
|
The motivation for this is that
a.) 1.7.1 runs with python 3.6 (bionic)
b.) we want to run pylint on tests/ and tools for the same reasons
that we want to run it on cloudinit/
The changes are described below.
- Update tox.ini to invoke pylint v1.7.1.
- Modify .pylintrc generated-members ignore mocked object members (m_.*)
- Replace "dangerous" params defaulting to {}
- Fix up cloud_tests use of platforms
- Cast some instance objects to with dict()
- Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated)
- Update use of assertEqual(<boolean>, value) to assert<Boolean>(value)
- replace depricated assertRegexp -> assertRegex
- Remove useless test-class calls to super class
- Assign class property accessors a result and use it
- Fix missing class member in CepkoResultTests
- Fix Cheetah test import
|
|
Cloud-init integration tests should not depend on a curtin test ppa.
We already had a cloud-init test ppa for explicitly this purpose.
Just use it instead.
|
|
We had used some dns records in i9n.brickies.net (my personal domain)
as a temporary solution until we got names registered in the cloud-init.io
namespace.
We now have CNAME records for:
ubuntu.i9n.cloud-init.io
cloudinit1.cloud-init.io
cloudinit2.cloud-init.io
|
|
Integration test harness changes:
* Enable collection of console log in nocloud-kvm and lxd.
* Collect the console log to results for all test runs.
* change 'tmpfile' to pick name locally instead of using 'mktemp'.
* drop the 'instance' attribute from nocloud-kvm Image and
demote LXDImage.instance to a private attribute.
This is because Images do not actually have instances.
(LXDImage internally uses a booted system to modify the image).
* Add 'TargetBase' as a superclass of Image and Instance providing
implementations of execute, read_data, write_data, pull_file,
and push_file. These all depend on an implementation of _execute.
* Improve '_execute' implementations to support accepting stdin.
* execute supports 'rcs=False' meaning 'do not raise exception'.
* Drop support for pylxd < 2.2. older versions cannot determine
exit code of 'execute', which makes them unusable.
* make NoCloudKVMInstance._execute run as root via sudo. This required
some changes so that 'hostname' could be reverse-looked up in order
to avoid sudo taking a long time (~20 seconds).
* re-use existing ssh connection in nocloud-kvm.
Test changes here:
* do not use /tmp, but rather /var/tmp (LP: #1707222)
* make keys_to_console assertions more strict.
* change user test cases to always add default (ubuntu) user
so that nocloud-kvm's execute which operates over ssh can work.
|
|
There was fallout in a full integration test run from my adding of
test_no_warnings_in_log which asserted that there could not be a WARNING
found in the /var/log/cloud-init.log
This fixes 2 of the cases:
* TestCommandOutputSimple had a valid WARNING written, so adjust its
test case to allow for that.
* TestLxdDir had a valid config in the test but the module would
log a WARNING, so fix the module.
Also updates lxd unit tests to look for WARN themselves.
|
|
Tests are currently run by creating a temporary subclass of each class
and then executing it (in get_suites). When running the tests suite
the output would contain the temporary class name. That was less than
useful, and made batch runs almost impossible to identify which
test case had an error.
This change goes from output of:
FAIL: test_no_warnings_in_log \
(tests.cloud_tests.testcases.get_suite.<locals>.tmp)
To
FAIL: test_no_warnings_in_log \
(tests.cloud_tests.testcases.modules.ntp.TestNtp)
|
|
Fix three things related to the ntp module:
1. Fix invalid cloud-config schema in the integration test which
provided empty dicts instead of emptylists for pools and servers
2. Correct logic in the ntp module to allow support for the minimal
cloud-config 'ntp:' without raising a RuntimeError. Docs and schema
definitions already describe that cloud-config's ntp can be empty.
An ntp configuration with neither pools nor servers will be
configured with a default set of ntp pools. As such, the ntp module
now officially allows the following ntp cloud-configs:
- ntp:
- ntp: {}
- ntp:
servers: []
pools: []
3. Add a simple unit test which validates all cloud-config provided to
our integration tests to ensure it adheres to any defined module
schema so as more jsonschema definitions are added, we validate our
integration test configs.
LP: #1724951
|
|
When operating in expected path, cloud-init should avoid logging with
warning. That causes 'WARNING' messages in /var/log/cloud-init.log.
By default, warnings also go to the console.
Since jsonschema is a optional dependency, and not present on xenial
and zesty, cloud-init should not warn there.
Also here:
* Add a test to integration tests to assert that there are no
warnings in /var/log/cloud-init.log.
* Update one integration test that did show warning and the related
documentation and examples.
LP: #1724354
|
|
Combine the configs and testcases directories, so all files are
together in one place. Update the test config location as well.
|
|
The locale integration test started failing with commit 0ef61b28.
This was just because the test's expectations on the content/formatting
of /etc/default/locale were too strict. The change here is to read
the file as a set of shell variables and assert that the values are
set correctly.
|
|
The test is currently importing the incorrect keyid. It specifies
the curtin developers ppa, rather than the cloud-init ppa. On
Artful this causes failures as a check is made to verify the
correct key is imported for the ppa, whereas on previous releases
only a warning was issued.
Also, change to use a full key fingerprint.
LP: #1702717
|
|
I want to be able to add additional SSH keys to my account, therefore I
should not be limiting these tests to look for one specific key. Instead
we confirm that the comment in authorized_users has the specified users.
|
|
Recent change to ntp in artful has added the sntp package whenever
ntp is installed. The tests, rather poorly, did a dpkg -l instead
of checking with `which`. This fixes the ntp tests to all use
`which` over expecting a certain number of lines using dpkg and
as a result make the tests OS independent.
|
|
Massive update to clean up and greatly enhance the integration testing
framework developed by Wesley Wiedenmeier.
- Updated tox environment to run integration test 'citest' to utilize
pylxd 2.2.3
- Add support for distro feature flags
- add framework for feature flags to release config with feature groups
and overrides allowed in any release conf override level
- add support for feature flags in platform and config handling
- during collect, skip testcases that require features not supported by
the image with a warning message
- Enable additional distros (i.e. centos, debian)
- Add 'bddeb' command to build a deb from the current working tree
cleanly in a container, so deps do not have to be installed on host
- Adds a command line option '--preserve-data' that ensures that
collected data will be left after tests run. This also allows the
directory to store collected data in during the run command to be
specified using '--data-dir'.
- Updated Read the Docs testing page and doc strings for pep 257
compliance
|
|
Snapd does not start on artful or on the versions in proposed. This
changes the behavior of the test to confirm that snapd is installed, not
that it is started, while the snap team fixes the issue (LP: ##1690880).
|
|
|
