| Age | Commit message (Collapse) | Author |
|---|
|
validate_cloudconfig_schema with strict=True would not actually validate
if there was no jsonschema available. That seems kind of strange.
The change here is to make it raise an exception if strict was passed in.
And then to fix the one test that needed a skipIfJsonSchema wrapper.
|
|
This enables warnings produced by pylint for unused variables (W0612),
and fixes the existing errors.
|
|
runcmd, bootcmd, snap/commands, ubuntu-advantage/commands would
log warning (and fail if strict) on duplicate values in the commands.
But those should be allowed. Example, it is perfectly valid to do:
runcmd: ['sleep 1', 'sleep 1']
LP: #1764264
|
|
If a file passed to render_from_file had non-ascii text then
jinja in python2 would decode as ascii, which would cause
UnicodeDecodeError. This issue can be re-created in python2
with just:
'can\xe2\x80\x99t'.decode()
The solution here is to explicitly pass in unicode supporting
type (py3 str, py2 unicode). Those are six.text_type.
Then jinja does not try to decode.
The reason we hit this is that load_file calls decode_binary.
decode_binary believes it has no work to do if it got a six.string_types.
isinstance('can\xe2\x80\x99t', six.string_types) == True
So it returns the original string which will blow up for jinja.
Our fix here then is to load the file in binary mode and explicitly
decode it to utf-8. Then in python2 we'll have a unicode type
and in python3 we'll have a string type.
|
|
Add a base NTP client configuration dictionary and allow Distro
specific changes to be merged. Add a select client function which
implements logic to preferr installed clients over clients which
need to be installed. Also allow distributions to override the
cloud-init defaults.
LP: #1749722
|
|
Previously there was no support at all for zfs file system. With this
change it is now possible to use the resizefs module to grow a zpool to
its maximum partition size on FreeBSD.
LP: #1721243
|
|
ubuntu-advantage-tools is a package for enabling and disabling extended
support services such as Extended Security Maintenance (ESM), Canonical
Livepatch and FIPS certified PPAs. Simplify Ubuntu Advantage setup on
machines by allowing users to provide a list of ubuntu-advantage commands
in cloud-config.
|
|
Support installing and configuring snaps on ubuntu systems. Now,
cloud-config files can provide a list or dictionary of snap:assertions
which will be allow configuration of snapd on a system via 'snap ack'
calls. The snap:commands configuration option supports arbitrary system
commands intended to interact with snappy's cli. This allows users to run
arbitrary snappy commands to create users, download, install and
configure snap packages and snapd.
This branch also deprecates old snappy and snap_config modules leaving
warnings in documentation and runtime for consumers of these modules.
Deprecated snap* modules will be dropped in cloud-init v.18.2 release.
|
|
Older unittest2.TestCase (as seen in CentOS 6) do not have an
assertRaisesRegex method. They only have the now-deprecated
assertRaisesRegexp.
We need our unit tests to work there and on newer python (3.6).
Simply making assertRaisesRegex = assertRaisesRegexp makes pylint
complain as described in https://github.com/PyCQA/pylint/issues/1946 .
What was here before this commit was actually broken. This commit
makes assertRaisesRegex functional in CentOS 6 and works around
the invalid Deprecated warning from pylint.
To prove this, we use assertRaisesRegex in a unit test which will
be exectued in py27, py3 and py26.
|
|
When instance meta-data provides hostname information, run
cc_set_hostname in the init-local or init-net stage before network
comes up.
Prevent an initial DHCP request which leaks the stock cloud-image default
hostname before the meta-data provided hostname was processed.
A leaked cloud-image hostname adversely affects Dynamic DNS which
would reallocate 'ubuntu' hostname in DNS to every instance brought up by
cloud-init. These instances would only update DNS to the cloud-init
configured hostname upon DHCP lease renewal.
This branch extends the get_hostname methods in datasource, cloud and
util to limit results to metadata_only to avoid extra cost of querying
the distro for hostname information if metadata does not provide that
information.
LP: #1746455
|
|
This just centralizes a hunk of duplicated code and uses it from the
new location.
|
|
This makes 2 changes to shellify's behavior:
a.) raise a TypeError rather than a RuntimeError.
b.) raise a TypeError if input is not a list or tuple.
|
|
When we moved some tests to live under cloudinit/ we inadvertantly
failed to change all things that would run nose to include that
directory.
This changes all the 'nose' invocations to consistently run with
tests/unittests and cloudinit/.
Also, it works around, more correctly this time, a python2.6-ism with
the following code:
with assertRaises(SystemExit) as cm:
sys.exit(2)
|
|
Resize of btrfs fails if the mount point for the file system we are trying
to resize, i.e. the root of the filesystem is read only. With this change
we use a known (currently snapper specific) rw location to work around a
flaw that blocks resizing of the ro filesystem.
LP: #1734787
|
|
The motivation for this is that
a.) 1.7.1 runs with python 3.6 (bionic)
b.) we want to run pylint on tests/ and tools for the same reasons
that we want to run it on cloudinit/
The changes are described below.
- Update tox.ini to invoke pylint v1.7.1.
- Modify .pylintrc generated-members ignore mocked object members (m_.*)
- Replace "dangerous" params defaulting to {}
- Fix up cloud_tests use of platforms
- Cast some instance objects to with dict()
- Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated)
- Update use of assertEqual(<boolean>, value) to assert<Boolean>(value)
- replace depricated assertRegexp -> assertRegex
- Remove useless test-class calls to super class
- Assign class property accessors a result and use it
- Fix missing class member in CepkoResultTests
- Fix Cheetah test import
|
|
Add opensuse distro support to cc_ntp module.
LP: #1726572
|
|
The etc/hosts file is was not properly setup for openSUSE or SLES
when manage_etc_hosts is set in the config file.
Improve the doc to address the fact that the 'localhost' ip is
distribution dependent (not always 127.0.0.1).
LP: #1731022
|
|
A recent cleanup of the resizefs module broke resizing when a system was
booted with root=PARTUUID=<uuid> and the device /dev/root does not exist.
This path is exposed with the Ubuntu 16.04 but not with Ubuntu 17.10. A
recreate exists under bug 1684869.
LP: #1725067
|
|
There was fallout in a full integration test run from my adding of
test_no_warnings_in_log which asserted that there could not be a WARNING
found in the /var/log/cloud-init.log
This fixes 2 of the cases:
* TestCommandOutputSimple had a valid WARNING written, so adjust its
test case to allow for that.
* TestLxdDir had a valid config in the test but the module would
log a WARNING, so fix the module.
Also updates lxd unit tests to look for WARN themselves.
|
|
Fix three things related to the ntp module:
1. Fix invalid cloud-config schema in the integration test which
provided empty dicts instead of emptylists for pools and servers
2. Correct logic in the ntp module to allow support for the minimal
cloud-config 'ntp:' without raising a RuntimeError. Docs and schema
definitions already describe that cloud-config's ntp can be empty.
An ntp configuration with neither pools nor servers will be
configured with a default set of ntp pools. As such, the ntp module
now officially allows the following ntp cloud-configs:
- ntp:
- ntp: {}
- ntp:
servers: []
pools: []
3. Add a simple unit test which validates all cloud-config provided to
our integration tests to ensure it adheres to any defined module
schema so as more jsonschema definitions are added, we validate our
integration test configs.
LP: #1724951
|
|
This adds a config module so support for adding zypper repositories
via cloud-config.
LP: #1718675
|
|
The bootcmd test was leaving files in the tmpdir named
ci-FakeExtendedTempFile.XXXXXX.
This cleans those up.
|
|
As root user, os.access(<path>, os.W_OK) will always return True so that
path will never get executed. Also avoid a warning if the root is
overlayroot, which is the common case on a MAAS booted 'ephemeral' system.
|
|
Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.
This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.
LP: #1462693
|
|
Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.
This branch also does the following:
- drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
option only created the specified directory and didn't make use of
that directory for any resize operations.
- Drop yaml.dumps calls from schema documentation generation to avoid
yaml import costs on module load
- Add __doc__ = get_schema_doc(schema) definitions it each module to
supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
cc_resizefs
- Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
contain more than one example
|
|
This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.
|
|
Currently the cloud-init default locale (en_US.UTF-8) is set by
the base datasource class. This patch allows a distro to overide
the fallback value with one that's available in the distro but continues
to respect an image which has preconfigured a locale.
- Distro object now has a get_locale method which will return a
preconfigure locale setting by checking the distros locale system
configuration file. If not set or not present, return the default
locale of en_US.UTF-8 which retains behavior of all previous cloud-init
releases.
- Apply locale now handles regenerating locales or system configuration
files as needed.
- Adjust apply_locale logic to skip locale-regen if the specified LANG
value is C.UTF-8,C, or POSIX; they do not require regeneration.
- Further add unittests to exercise the default paths for Ubuntu and
non-ubuntu paths to validate they get the LANG expected.
|
|
test_set_locale_sles and test_set_locale_sles_default were incorrectly
testing for truth of <distro_object>.uses_systemd rather than calling
that function and checking its result.
The error was only seen if the system running the tests was not using
systemd.
|
|
This gets initial opensuse and SLES support back to a working state.
Still missing is more complete network file writing and unit tests.
|
|
Both landscape and puppet modules had issues with the way they wrote
/etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or
python2. This branch adds initial unit tests for both modules which will
get better exercise under both python2 and python3.
The unit tests shed light on a few issues:
- In the cc_landscape module py3 can't provide six.StringIO content to
ConfigParser.write, so we need to use six.BytesIO instead
- In the cc_puppet module, python <= 2.7 doesn't support using
six.StringIO as a context manager, so we drop the context manager
fanciness and directly set outputstream = StringIO().
- The docstring in cc_puppet is fixed to document the 'conf'
sub-key requiring valid puppet section names for each
key-value list.
LP: #1699282
LP: #1710932
|
|
This branch does a few things:
- Add 'schema' subcommand to cloud-init CLI for validating
cloud-config files against strict module jsonschema definitions
- Add --annotate parameter to 'cloud-init schema' to annotate
existing cloud-config file content with validation errors
- Add jsonschema definition to cc_runcmd
- Add unit test coverage for cc_runcmd
- Update CLI capabilities documentation
This branch only imports development (and analyze) subparsers when the
specific subcommand is provided on the CLI to avoid adding costly unused
file imports during cloud-init system boot.
The schema command allows a person to quickly validate a cloud-config text
file against cloud-init's known module schemas to avoid costly roundtrips
deploying instances in their cloud of choice. As of this branch, only
cc_ntp and cc_runcmd cloud-config modules define schemas. Schema
validation will ignore all undefined config keys until all modules define
a strict schema.
To perform validation of runcmd and ntp sections of a cloud-config file:
$ cat > cloud.cfg <<EOF
runcmd: bogus
EOF
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg \
--annotate
Once jsonschema is defined for all ~55 cc modules, we will move this
schema subcommand up as a proper subcommand of the cloud-init CLI.
|
|
Some systems like Ubuntu-Core do not provide an ntp package for
installation but do include systemd-timesyncd (an ntp client).
On such systems cloud-init will generate a timesyncd configuration
using the 'servers' and 'pools' values as ntp hosts for timesyncd to use.
LP: #1686485
|
|
Instead of passing around a 'log' reference to functions, just import
logging and use that. This is the pattern that is now more common in
cloud-init.
|
|
- Simplify the logic of 'variant' in util.system_info
much of the data from
https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
- fix get_resource_disk_on_freebsd when running on a system without
an Azure resource disk.
- fix tools/build-on-freebsd to replace oauth with oauthlib and add
bash which is a dependency for tests.
- update a fiew places that were checking for freebsd but not using
the util.is_FreeBSD()
|
|
Unix file modes are usually represented as octal, but they were being
interpreted as decimal, for example 0o644 would be printed as '420'.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
|
|
These changes are all in an effort to get tools/run-centos using
read-dependencies rather than the 'setup-centos' script with a separate
set of dependencies listed.
- tools/read-dependencies: support taking multiple --requirements
options. This allows run-centos to get both test and build
dependencies. Ultimately, I think it might be nicer for
read-dependencies to take a list of "goals" (build, test, run or
test-tox) rather than having the caller need to know to provide
multiple --requirements.
- packages/pkg-deps.json: drop the version on the sudo package.
centos 6 has newer (1.8.6p3) version than listed, so its not a problem.
- test_handler_disk_setup.py: a test case here was using assertLogs
which is not present in the version of unittest2 that is available in
centos 6 epel. We just adjust it to use with_logs = True.
- tools/run-cents:
- improve usage with example
- add 'inside_as_cd' to provide the dir you want to cd first to.
- avoid the intermediate tarball on disk in the container.
- add 'prep' subcommand and use it to install pre-dependencies.
- use read-dependencies.
|
|
The jsonschema package is used only when available, but the lint
check thinks the import is unused across pyflakes and flake8. In
order to avoid having exceptions for both assert that the import
works right after and the import is considered used.
The '# NOQA' doesn't affect pyflakes (only flake8).
LP: #1695918
|
|
On some build environments we don't have python-jsonschema installed.
Since this dependency is an optional runtime dependency, we can also make
it an optional unit test dependency. Add a skip of related unittests when
jsonschema is not present.
Also, KeyError messages on CentOs don't have single quotes around the
missing 'key-name'. Make our KeyError assertion a bit more flexible with
the assertIn call.
LP: #1695318
|
|
cloud-config files are very flexible and permissive.
This adds a jsonsschema definition to the cc_ntp module and validation
functions in cloudinit/config/schema which will log warnings about
invalid configuration values in the ntp section.
A cmdline tools/cloudconfig-schema is added which can be used in our dev
environments to quickly attempt to exercise the ntp schema.
It is also exposed as a main in cloudinit.config.schema.
(python3 -m cloudinit.config.schema)
LP: #1692916
|
|
This attempts to use udevadm settle to wait until devices have been
fully "realized". If a device exists, there may still be events in
the udev queue that would create its partition table entries.
We need to wait until those have been processed also.
LP: #1692093
|
|
python2.6 unittest.TestCase does not have the assertIsNone or
assertIsNotNone. We just have to explicitly use the unittest2
version, which we get from helpers.
The desire to use assertIsNone comes from flake8 (through hacking,
I believe).
Also, fix "{}.format('foo')" which is not valid in python2.6.
|
|
Any CiTestCase subclass can now set a class attribute with_logs = True and
tests can now make assertions on self.logs.getvalue(). This branch
restructures a bit of cc_ntp module to get better test coverage of the
module. It also restructures the handler_cc_ntp unit tests to avoid nested
mocks where possible. Deeply nested mocks cause a couple of issues:
- greater risk: mocks are permanent within the scope, so multiple
call-sites could be affected by package mocks
- less legible tests: each mock doesn't advertise the actual call-site
- tight coupling: the unit test logic to tightly bound to the actual
implementation in remote (unrelated) modules which makes it more
costly to maintain code
- false success: we should be testing the expected behavior not specific
remote method names as we want to know if that underlying behavior
changes and breaks us.
LP: #1692794
|
|
This just moves flake8 and related tools up to newer versions and fixes
the complaints associated with that.
We added to the list of flake8 ignores:
H102: do not put vim info in source files
H304: no relative imports
Also updates and pins the following in the flake8 environment:
pep8: 1.7.0 => drop (although hacking still pulls it in).
pyflakes 1.1.0 => 1.5.0
hacking 0.10.2 => 0.13.0
flake8 2.5.4 => 3.3.0
pycodestyle none => 2.3.1
|
|
This fixes several shortcomings of disk_setup with gpt disks.
* 'sgdisk -p' was being used to determine the size of a disk.
this can fail if it believes there is a bad gpt partition table.
Instead we just use blockdev now for both mbr or gpt disks.
* parsing of sgdisk -p output assumed that the 'name' of the partition
type would not have any spaces (Microsoft basic data)
* interaction with sgdisk did not realize that sgdisk wants input
of '8300' rather than '83' and will output the same.
LP: #1692087
|
|
On systems which installed ntp and specified servers or pools in the
config ntpd didn't notice the updated configuration file and didn't
use the correct configuration. Resolve this by rendering the template
first which allows the package install to use the existing
configuration. Additionally add a service restart to handle the case
where ntp does not need to be installed but it may not have started.
Add an integration test to confirm that cc_ntp enables ntp to use the
specific servers and pools in the cloud-config.
LP: #1645644
|
|
A recent merge that added a mkfs.ext4 tests has a hard coded location
for the binary of mkfs.ext4. On CentOS 7 the test failed because the
command in a different location than Ubuntu.
LP: #1691517
|
|
Large instance types have a different disk format on the newly
partitioned ephemeral drive. So we have to adjust the logic in the
Azure datasource to recognize that a disk with 2 partitions and
an empty ntfs filesystem on the second one is acceptable.
This also adjusts the datasources's builtin fs_setup config to remove
the 'replace_fs' entry. This entry was previously ignored, and confusing.
I've clarified the doc on that also.
LP: #1686514
|
|
Apt related tests were broken when running on centos becasue apt is not
available. This fixes the unit test, with a small re-work of apt_configure.
Also in 'tox -e centos6' only run nose on tests/unittests as tests/
also contain integration tests that should not be run.
|
|
This patch targets to make FreeBSD 10.3 or 11 work on Azure. The
modifications abide by the rule of:
* making as less modification as possible
* delegate to the distro or datasource where possible.
The main modifications are:
1. network configuration improvements, and movement into distro path.
2. Fix setting of password.
Password setting through "pw" can only work through pipe.
3. Add 'root:wheel' to syslog_fix_perms field.
4. Support resizing default file system (ufs)
5. copy cloud.cfg for freebsd to /etc/cloud/cloud.cfg rather than
/usr/local/etc/cloud/cloud.cfg.
6. Azure specific changes:
a. When reading the azure endpoint, search in a different path
and read a different option name (option-245 vs. unknown-245).
so, the lease file path should be generated according to platform.
b. adjust the handling of ephemeral mounts for ufs filesystem and
for finding the ephemeral device.
c. fix mounting of cdrom
LP: #1636345
|
|
If 'cmd' is provided to a fs_setup entry, then cloud-init was trying
to execute the rendered string as a single name, rather than
splitting the string. The change here will pass the string to
shell for interpretation so that it is split there.
Also fix some documentation errors and warn when fs_opts or overwrite
is provided along with 'cmd'.
LP: #1687712
|
