| Age | Commit message (Collapse) | Author |
|---|
|
Resize of btrfs fails if the mount point for the file system we are trying
to resize, i.e. the root of the filesystem is read only. With this change
we use a known (currently snapper specific) rw location to work around a
flaw that blocks resizing of the ro filesystem.
LP: #1734787
|
|
The motivation for this is that
a.) 1.7.1 runs with python 3.6 (bionic)
b.) we want to run pylint on tests/ and tools for the same reasons
that we want to run it on cloudinit/
The changes are described below.
- Update tox.ini to invoke pylint v1.7.1.
- Modify .pylintrc generated-members ignore mocked object members (m_.*)
- Replace "dangerous" params defaulting to {}
- Fix up cloud_tests use of platforms
- Cast some instance objects to with dict()
- Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated)
- Update use of assertEqual(<boolean>, value) to assert<Boolean>(value)
- replace depricated assertRegexp -> assertRegex
- Remove useless test-class calls to super class
- Assign class property accessors a result and use it
- Fix missing class member in CepkoResultTests
- Fix Cheetah test import
|
|
Add opensuse distro support to cc_ntp module.
LP: #1726572
|
|
The etc/hosts file is was not properly setup for openSUSE or SLES
when manage_etc_hosts is set in the config file.
Improve the doc to address the fact that the 'localhost' ip is
distribution dependent (not always 127.0.0.1).
LP: #1731022
|
|
A recent cleanup of the resizefs module broke resizing when a system was
booted with root=PARTUUID=<uuid> and the device /dev/root does not exist.
This path is exposed with the Ubuntu 16.04 but not with Ubuntu 17.10. A
recreate exists under bug 1684869.
LP: #1725067
|
|
There was fallout in a full integration test run from my adding of
test_no_warnings_in_log which asserted that there could not be a WARNING
found in the /var/log/cloud-init.log
This fixes 2 of the cases:
* TestCommandOutputSimple had a valid WARNING written, so adjust its
test case to allow for that.
* TestLxdDir had a valid config in the test but the module would
log a WARNING, so fix the module.
Also updates lxd unit tests to look for WARN themselves.
|
|
Fix three things related to the ntp module:
1. Fix invalid cloud-config schema in the integration test which
provided empty dicts instead of emptylists for pools and servers
2. Correct logic in the ntp module to allow support for the minimal
cloud-config 'ntp:' without raising a RuntimeError. Docs and schema
definitions already describe that cloud-config's ntp can be empty.
An ntp configuration with neither pools nor servers will be
configured with a default set of ntp pools. As such, the ntp module
now officially allows the following ntp cloud-configs:
- ntp:
- ntp: {}
- ntp:
servers: []
pools: []
3. Add a simple unit test which validates all cloud-config provided to
our integration tests to ensure it adheres to any defined module
schema so as more jsonschema definitions are added, we validate our
integration test configs.
LP: #1724951
|
|
This adds a config module so support for adding zypper repositories
via cloud-config.
LP: #1718675
|
|
The bootcmd test was leaving files in the tmpdir named
ci-FakeExtendedTempFile.XXXXXX.
This cleans those up.
|
|
As root user, os.access(<path>, os.W_OK) will always return True so that
path will never get executed. Also avoid a warning if the root is
overlayroot, which is the common case on a MAAS booted 'ephemeral' system.
|
|
Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.
This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.
LP: #1462693
|
|
Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.
This branch also does the following:
- drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
option only created the specified directory and didn't make use of
that directory for any resize operations.
- Drop yaml.dumps calls from schema documentation generation to avoid
yaml import costs on module load
- Add __doc__ = get_schema_doc(schema) definitions it each module to
supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
cc_resizefs
- Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
contain more than one example
|
|
This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.
|
|
Currently the cloud-init default locale (en_US.UTF-8) is set by
the base datasource class. This patch allows a distro to overide
the fallback value with one that's available in the distro but continues
to respect an image which has preconfigured a locale.
- Distro object now has a get_locale method which will return a
preconfigure locale setting by checking the distros locale system
configuration file. If not set or not present, return the default
locale of en_US.UTF-8 which retains behavior of all previous cloud-init
releases.
- Apply locale now handles regenerating locales or system configuration
files as needed.
- Adjust apply_locale logic to skip locale-regen if the specified LANG
value is C.UTF-8,C, or POSIX; they do not require regeneration.
- Further add unittests to exercise the default paths for Ubuntu and
non-ubuntu paths to validate they get the LANG expected.
|
|
test_set_locale_sles and test_set_locale_sles_default were incorrectly
testing for truth of <distro_object>.uses_systemd rather than calling
that function and checking its result.
The error was only seen if the system running the tests was not using
systemd.
|
|
This gets initial opensuse and SLES support back to a working state.
Still missing is more complete network file writing and unit tests.
|
|
Both landscape and puppet modules had issues with the way they wrote
/etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or
python2. This branch adds initial unit tests for both modules which will
get better exercise under both python2 and python3.
The unit tests shed light on a few issues:
- In the cc_landscape module py3 can't provide six.StringIO content to
ConfigParser.write, so we need to use six.BytesIO instead
- In the cc_puppet module, python <= 2.7 doesn't support using
six.StringIO as a context manager, so we drop the context manager
fanciness and directly set outputstream = StringIO().
- The docstring in cc_puppet is fixed to document the 'conf'
sub-key requiring valid puppet section names for each
key-value list.
LP: #1699282
LP: #1710932
|
|
This branch does a few things:
- Add 'schema' subcommand to cloud-init CLI for validating
cloud-config files against strict module jsonschema definitions
- Add --annotate parameter to 'cloud-init schema' to annotate
existing cloud-config file content with validation errors
- Add jsonschema definition to cc_runcmd
- Add unit test coverage for cc_runcmd
- Update CLI capabilities documentation
This branch only imports development (and analyze) subparsers when the
specific subcommand is provided on the CLI to avoid adding costly unused
file imports during cloud-init system boot.
The schema command allows a person to quickly validate a cloud-config text
file against cloud-init's known module schemas to avoid costly roundtrips
deploying instances in their cloud of choice. As of this branch, only
cc_ntp and cc_runcmd cloud-config modules define schemas. Schema
validation will ignore all undefined config keys until all modules define
a strict schema.
To perform validation of runcmd and ntp sections of a cloud-config file:
$ cat > cloud.cfg <<EOF
runcmd: bogus
EOF
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg \
--annotate
Once jsonschema is defined for all ~55 cc modules, we will move this
schema subcommand up as a proper subcommand of the cloud-init CLI.
|
|
Some systems like Ubuntu-Core do not provide an ntp package for
installation but do include systemd-timesyncd (an ntp client).
On such systems cloud-init will generate a timesyncd configuration
using the 'servers' and 'pools' values as ntp hosts for timesyncd to use.
LP: #1686485
|
|
Instead of passing around a 'log' reference to functions, just import
logging and use that. This is the pattern that is now more common in
cloud-init.
|
|
- Simplify the logic of 'variant' in util.system_info
much of the data from
https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
- fix get_resource_disk_on_freebsd when running on a system without
an Azure resource disk.
- fix tools/build-on-freebsd to replace oauth with oauthlib and add
bash which is a dependency for tests.
- update a fiew places that were checking for freebsd but not using
the util.is_FreeBSD()
|
|
Unix file modes are usually represented as octal, but they were being
interpreted as decimal, for example 0o644 would be printed as '420'.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
|
|
These changes are all in an effort to get tools/run-centos using
read-dependencies rather than the 'setup-centos' script with a separate
set of dependencies listed.
- tools/read-dependencies: support taking multiple --requirements
options. This allows run-centos to get both test and build
dependencies. Ultimately, I think it might be nicer for
read-dependencies to take a list of "goals" (build, test, run or
test-tox) rather than having the caller need to know to provide
multiple --requirements.
- packages/pkg-deps.json: drop the version on the sudo package.
centos 6 has newer (1.8.6p3) version than listed, so its not a problem.
- test_handler_disk_setup.py: a test case here was using assertLogs
which is not present in the version of unittest2 that is available in
centos 6 epel. We just adjust it to use with_logs = True.
- tools/run-cents:
- improve usage with example
- add 'inside_as_cd' to provide the dir you want to cd first to.
- avoid the intermediate tarball on disk in the container.
- add 'prep' subcommand and use it to install pre-dependencies.
- use read-dependencies.
|
|
The jsonschema package is used only when available, but the lint
check thinks the import is unused across pyflakes and flake8. In
order to avoid having exceptions for both assert that the import
works right after and the import is considered used.
The '# NOQA' doesn't affect pyflakes (only flake8).
LP: #1695918
|
|
On some build environments we don't have python-jsonschema installed.
Since this dependency is an optional runtime dependency, we can also make
it an optional unit test dependency. Add a skip of related unittests when
jsonschema is not present.
Also, KeyError messages on CentOs don't have single quotes around the
missing 'key-name'. Make our KeyError assertion a bit more flexible with
the assertIn call.
LP: #1695318
|
|
cloud-config files are very flexible and permissive.
This adds a jsonsschema definition to the cc_ntp module and validation
functions in cloudinit/config/schema which will log warnings about
invalid configuration values in the ntp section.
A cmdline tools/cloudconfig-schema is added which can be used in our dev
environments to quickly attempt to exercise the ntp schema.
It is also exposed as a main in cloudinit.config.schema.
(python3 -m cloudinit.config.schema)
LP: #1692916
|
|
This attempts to use udevadm settle to wait until devices have been
fully "realized". If a device exists, there may still be events in
the udev queue that would create its partition table entries.
We need to wait until those have been processed also.
LP: #1692093
|
|
python2.6 unittest.TestCase does not have the assertIsNone or
assertIsNotNone. We just have to explicitly use the unittest2
version, which we get from helpers.
The desire to use assertIsNone comes from flake8 (through hacking,
I believe).
Also, fix "{}.format('foo')" which is not valid in python2.6.
|
|
Any CiTestCase subclass can now set a class attribute with_logs = True and
tests can now make assertions on self.logs.getvalue(). This branch
restructures a bit of cc_ntp module to get better test coverage of the
module. It also restructures the handler_cc_ntp unit tests to avoid nested
mocks where possible. Deeply nested mocks cause a couple of issues:
- greater risk: mocks are permanent within the scope, so multiple
call-sites could be affected by package mocks
- less legible tests: each mock doesn't advertise the actual call-site
- tight coupling: the unit test logic to tightly bound to the actual
implementation in remote (unrelated) modules which makes it more
costly to maintain code
- false success: we should be testing the expected behavior not specific
remote method names as we want to know if that underlying behavior
changes and breaks us.
LP: #1692794
|
|
This just moves flake8 and related tools up to newer versions and fixes
the complaints associated with that.
We added to the list of flake8 ignores:
H102: do not put vim info in source files
H304: no relative imports
Also updates and pins the following in the flake8 environment:
pep8: 1.7.0 => drop (although hacking still pulls it in).
pyflakes 1.1.0 => 1.5.0
hacking 0.10.2 => 0.13.0
flake8 2.5.4 => 3.3.0
pycodestyle none => 2.3.1
|
|
This fixes several shortcomings of disk_setup with gpt disks.
* 'sgdisk -p' was being used to determine the size of a disk.
this can fail if it believes there is a bad gpt partition table.
Instead we just use blockdev now for both mbr or gpt disks.
* parsing of sgdisk -p output assumed that the 'name' of the partition
type would not have any spaces (Microsoft basic data)
* interaction with sgdisk did not realize that sgdisk wants input
of '8300' rather than '83' and will output the same.
LP: #1692087
|
|
On systems which installed ntp and specified servers or pools in the
config ntpd didn't notice the updated configuration file and didn't
use the correct configuration. Resolve this by rendering the template
first which allows the package install to use the existing
configuration. Additionally add a service restart to handle the case
where ntp does not need to be installed but it may not have started.
Add an integration test to confirm that cc_ntp enables ntp to use the
specific servers and pools in the cloud-config.
LP: #1645644
|
|
A recent merge that added a mkfs.ext4 tests has a hard coded location
for the binary of mkfs.ext4. On CentOS 7 the test failed because the
command in a different location than Ubuntu.
LP: #1691517
|
|
Large instance types have a different disk format on the newly
partitioned ephemeral drive. So we have to adjust the logic in the
Azure datasource to recognize that a disk with 2 partitions and
an empty ntfs filesystem on the second one is acceptable.
This also adjusts the datasources's builtin fs_setup config to remove
the 'replace_fs' entry. This entry was previously ignored, and confusing.
I've clarified the doc on that also.
LP: #1686514
|
|
Apt related tests were broken when running on centos becasue apt is not
available. This fixes the unit test, with a small re-work of apt_configure.
Also in 'tox -e centos6' only run nose on tests/unittests as tests/
also contain integration tests that should not be run.
|
|
This patch targets to make FreeBSD 10.3 or 11 work on Azure. The
modifications abide by the rule of:
* making as less modification as possible
* delegate to the distro or datasource where possible.
The main modifications are:
1. network configuration improvements, and movement into distro path.
2. Fix setting of password.
Password setting through "pw" can only work through pipe.
3. Add 'root:wheel' to syslog_fix_perms field.
4. Support resizing default file system (ufs)
5. copy cloud.cfg for freebsd to /etc/cloud/cloud.cfg rather than
/usr/local/etc/cloud/cloud.cfg.
6. Azure specific changes:
a. When reading the azure endpoint, search in a different path
and read a different option name (option-245 vs. unknown-245).
so, the lease file path should be generated according to platform.
b. adjust the handling of ephemeral mounts for ufs filesystem and
for finding the ephemeral device.
c. fix mounting of cdrom
LP: #1636345
|
|
If 'cmd' is provided to a fs_setup entry, then cloud-init was trying
to execute the rendered string as a single name, rather than
splitting the string. The change here will pass the string to
shell for interpretation so that it is split there.
Also fix some documentation errors and warn when fs_opts or overwrite
is provided along with 'cmd'.
LP: #1687712
|
|
ConfigObj produces configuration files that are incompatible with yum if
multiple values are listed for a configuration key. Switch to the builtin
configparser, and ConfigParser (Python 2) which correctly handles this
case.
Add additional test case for array values in yum_repos definition
LP: #1592150
|
|
Accordingly to the documentation:
The ``partition`` option may also be set to ``auto``, in which this
module will search for the existance of a filesystem matching the
``label``, ``type`` and ``device`` of the ``fs_setup`` entry and
will skip creating the filesystem if one is found.
However, using this "auto" flag always recreates the partition no matter
if it has been done before or not.
This commit fixes a bug in which the "partition" attribute was always
set to None although in some cases it should not.
LP: #1634678
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
The version of sfdisk in wily (and onwards) only accepts sectors as a
valid disk size. As such, this refactors the MBR code path in
cc_disk_setup to use sectors.
- use --unit=S: while newer versions of sfdisk assume --unit=S, older
versions do not so we specifically pass it in. Versions of sfdisk
found in supported OSes such as centos6 wont assume --unit=S.
- add --force: this exists back to centos 6 (2.17.2), so it should
be fine, and is what we ultimately want.
"do what I say, even if it is stupid"
- keep --Linux. Even though this has been deprecated for quite some
time, we keep it until versions that want it are unsupported.
If necessary at some point we could check for util linux version
and if it had --Linux and use it in those cases.
Additionally, improve usefulness of some log messages.
LP: #1460715
|
|
Ubuntu Core images use the `snap create-user` to add users to an
Ubuntu Core system. Add support for creating snap users by adding
a key to the users dictionary.
users:
- name: bob
snapuser: bob@bobcom.io
Or via the 'snappy' dictionary:
snappy:
email: bob@bobcom.io
Users may also create a snap user without contacting the SSO by
providing a 'system-user' assertion by importing them into snapd.
Additionally, Ubuntu Core systems have a read-only /etc/passwd such that
the normal useradd/groupadd commands do not function without an additional
flag, '--extrausers', which redirects the pwd to /var/lib/extrausers.
Move the system_is_snappy() check from cc_snappy module to util for
re-use and then update the Distro class to append '--extrausers' if
the system is Ubuntu Core.
|
|
This gets the tests running in centos 6.
* ProcessExecutionError: remove setting of .message
Nothing in cloud-init seems to use .message anywhere, so
it does not seem necessary.
The reason to change it is that on 2.6 it spits out:
cloudinit/util.py:286: DeprecationWarning: BaseException.message
* tox.ini: add a centos6 environment
the tox versions listed here replicate a centos6 install with
packages from EPEL.
You will still need a python2.6 to run this env so we do not
enable it by default.
|
|
Prior to LXD 2.3, the bridge configuration was done through distro
packaging. Thus, lxd module interacted with debconf.
With 2.3 and higher, this is now done inside LXD itself, so we
need to use "lxc network" there.
For now, this perfectly matches what we had before with debconf and
doesn't cover any of the new options. We can always add those later.
A set of tests similar to what we had for debconf has been added to make
sure things look good.
This is tested in Yakkety container running LXD 2.3 and all options seem
to be passed through as expected, giving me the bridge I defined.
Signed-off-by: Stéphane Graber <stgraber@ubuntu.com>
|
|
In order for a caller to use 'env' argument of subp, they
will realistically do:
env = os.environ.copy()
env['FOO'] = 'BZR'
subp(cmd, env=env)
This shortens that to be:
subp(cmd, update_env={'FOO': 'BZR'})
Add tests, and update growpart tests to use mock when playing with
os.environ.
|
|
test_handler_apt_source_v3.py was not cleaning up all its tmp dirs.
|
|
Old behavior allowed a user to provide:
apt_mirror: ""
And that was the same as:
apt_mirror: null
and the same as having not specified apt_mirror at all. This maintains
that behavior for all old string values.
LP: #1621180
|
|
This allows both v1/2 and and v3 formats to exist in config.
If both are present, then prefer v3. If values are not the same
then a ValueError is raised.
LP: #1616831
|
|
Spacewalk is used by some peopel to manage connections into redhat package
management systems and kickstart and various other tasks, so having a
system be able to do the needed tasks on first boot to integrate with that
system would be very useful (to some).
See: https://fedorahosted.org/spacewalk/
|
|
This adds an improved apt configuration format that is fully backwards
compatible with previous behavior. This is mostly copied from curtin's
implementation.
It does:
* clean up and centralizes many of the top level 'apt_*' values that
previously existed into a single top level 'apt'key.
* support a 'source' in apt/sources/entry that has only a key
* documents new features and adds tests.
See the added doc/examples/cloud-config-apt.txt for more information.
|
