summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2017-07-17Scaleway: add datasource with user and vendor data for Scaleway.Julien Castets
Here we add and enable by default a datasource for Scaleway cloud. The datasource quickly exits unless one of three things: a.) 'Scaleway' found as the system vendor b.) 'scaleway' found on the kernel command line. c.) the directory /var/run/scaleway exists (this is currently created by the scaleway initramfs module). One interesting bit of this particular datasource is that it requires the source port of the http request to be < 1024.
2017-07-17Support comments in content read by load_shell_content.Scott Moser
load_shell_content previously would not allow shell comment characters in the content being parsed. If comments=True is not passed then an exception would previously be raised as the line would not be guaranteed to have an '=' in it.
2017-07-17cloudinitlocal fail to run during bootHongjiang Zhang
There is a circular dependence in cloudinitlocal, which caused it to fail. As a result, cloud-init failed to find data source on Azure.
2017-07-17doc: fix disk setup example table_type optionsSandor Zeestraten
This fixes the disk setup example doc which specifies that the only currently supported table_type option is 'mbr' by adding the 'gpt' option which got supported as of 0.7.7. LP: #1703789
2017-07-17tools: Fix exception handling.Joonas Kylmälä
We should be expecting IndexError instead of KeyError because we are using a list (key_ids) and not a dictionary. Also, thanks to Emmanuel Kasper for pointing out the wrong response code. LP: #1701527
2017-07-13tests: fix usage of mock in GCE test.Scott Moser
The usage of mock in this test was simply invalid and only worked by happenstance.
2017-07-13test_gce: Fix invalid mock of platform_reports_gce to return FalseChad Smith
The mock of platform_reports_gce is created with a True return value in tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.setUp(). But, the final test_get_data_returns_false_if_not_on_gce incorrectly attempts to override the mocked return_value of True to False by setting self.m_platform_gce.return_value = False. But, since the mock is already initialized, the updated False is not honored. Instead we should use the patch decorator on the specific unit test to override the return_value of DataSourceGCE.platform_reports_gce to False. A False from platform_reports_gce allows DataSourceGCE.get_data to immediately return False instead of trying to contact metadata.google.internal as the related bug references.
2017-07-11test: fix incorrect keyid for apt repository.Joshua Powers
The test is currently importing the incorrect keyid. It specifies the curtin developers ppa, rather than the cloud-init ppa. On Artful this causes failures as a check is made to verify the correct key is imported for the ppa, whereas on previous releases only a warning was issued. Also, change to use a full key fingerprint. LP: #1702717
2017-07-11tests: Update version of pylxdJoshua Powers
With the upgrade to lxd 2.15, pylxd version 2.2.3 broke. Upgrading to version 2.2.4 fixes issues with missing attributes.
2017-07-11write_files: Remove log from helper function signatures.Andrew Jorgensen
Instead of passing around a 'log' reference to functions, just import logging and use that. This is the pattern that is now more common in cloud-init.
2017-07-11doc: document the cmdline options to NoCloudBrian Candler
Add permitted keys to documentation on seeding NoCloud.
2017-06-29read_dmi_data: always return None when inside a container.Scott Moser
This fixes stacktrace and warning message that would be printed to the log if running inside a container and read_dmi_data tried to access a key that was not present. In a container, the /sys/class/dmi/id data is not relevant to the but to the host. Additionally an unpriviledged container might see strange behavior: # cd /sys/class/dmi/id/ # id -u 0 # ls -l chassis_serial -r-------- 1 nobody nogroup 4096 Jun 29 16:49 chassis_serial # cat chassis_serial cat: /sys/class/dmi/id/chassis_serial: Permission denied The solution here is to just always return None when running in a container. LP: #1701325
2017-06-29requirements.txt: remove trailing white space.Scott Moser
The 'jsonschema' line had trailing white space. Remove it.
2017-06-27Azure: Add network-config, Refactor net layer to handle duplicate macs.Ryan Harper
On systems with network devices with duplicate mac addresses, cloud-init will fail to rename the devices according to the specified network configuration. Refactor net layer to search by device driver and device id if available. Azure systems may have duplicate mac addresses by design. Update Azure datasource to run at init-local time and let Azure datasource generate a fallback networking config to handle advanced networking configurations. Lastly, add a 'setup' method to the datasources that is called before userdata/vendordata is processed but after networking is up. That is used here on Azure to interact with the 'fabric'.
2017-06-27Tests: Simplify the check on ssh-import-idJoshua Powers
I want to be able to add additional SSH keys to my account, therefore I should not be limiting these tests to look for one specific key. Instead we confirm that the comment in authorized_users has the specified users.
2017-06-27tests: update ntp tests after sntp addedJoshua Powers
Recent change to ntp in artful has added the sntp package whenever ntp is installed. The tests, rather poorly, did a dpkg -l instead of checking with `which`. This fixes the ntp tests to all use `which` over expecting a certain number of lines using dpkg and as a result make the tests OS independent.
2017-06-15FreeBSD: Make freebsd a variant, fix unittests and tools/build-on-freebsd.Scott Moser
- Simplify the logic of 'variant' in util.system_info much of the data from https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version - fix get_resource_disk_on_freebsd when running on a system without an Azure resource disk. - fix tools/build-on-freebsd to replace oauth with oauthlib and add bash which is a dependency for tests. - update a fiew places that were checking for freebsd but not using the util.is_FreeBSD()
2017-06-15FreeBSD: fix test failureScott Moser
The previous commit caused test failure. This separates out _check_freebsd_cdrom and mocks it in a test rather than patching open.
2017-06-15FreeBSD: replace ifdown/ifup with "ifconfig down" and "ifconfig up".Hongjiang Zhang
Fix the issue caused by different commands on Linux and FreeBSD. On Linux, we used ifdown and ifup to enable and disable a NIC, but on FreeBSD, the counterpart is "ifconfig down" and "ifconfig up". LP: #1697815
2017-06-15FreeBSD: fix cdrom mounting failure if /mnt/cdrom/secure did not exist.Hongjiang Zhang
The current method is to attempt to mount the cdrom (/dev/cd0), if it is successful, /dev/cd0 is configured, otherwise, it is not configured. The problem is it forgets to check whether the mounting destination folder is created or not. As a result, mounting attempt failed even if cdrom is ready. LP: #1696295
2017-06-15main: Don't use templater to format the welcome messageAndrew Jorgensen
Some versions of Cheetah returned everything as unicode by default (not utf-8 or ascii) and some varieties of syslog would choke on unicode. Jinja2 is probably fine, but Python's format() is perfectly adequate for a short message like the welcome message. Reviewed-by: Tom Kirchner <tjk@amazon.com> Reviewed-by: Ben Cressey <bcressey@amazon.com>
2017-06-15docs: Automatically generate module docs form schema if present.Chad Smith
We have started adding jsonschema definitions for cloudconfig modules (cc_ntp). This branch allows us render sphinx docs using the module's shema definition instead of using the module's docstring. This allows us to avoid duplicating schema documentation in the module-level docstring and schema definition. The corresponding module documentation is extended a bit to differentiate between config schema and potential examples.
2017-06-15debian: fix path comment in /etc/hosts template.Jens Sandmann
The comments in the debian template file of /etc/hosts still pointed to a general template file instead of the debian one. LP: #1606406
2017-06-15suse: add hostname and fully qualified domain to template.Jens Sandmann
This just adds an entry for hostname and fqdn to 127.0.0.1 in templates/hosts.suse.tmpl.
2017-06-15write_file(s): Print permissions as octal, not decimalAndrew Jorgensen
Unix file modes are usually represented as octal, but they were being interpreted as decimal, for example 0o644 would be printed as '420'. Reviewed-by: Tom Kirchner <tjk@amazon.com>
2017-06-14ci deps: Add --test-distro to read-dependencies to install all depsChad Smith
read-dependencies now takes --test-distro param to indicate we want to install all system package depenencies to allow for testing and building for our continous integration environment. It allows us to install all needed deps on a fresh system with: python3 ./tools/read-dependencies --distro ubuntu --test-distro [--dry-run]. Additionally read-dependencies now looks at what version of python is running the script (py2 vs p3) and opts to install python 2 or 3 system deps respectively. This behavior can still be overridden with python3 ./tools/read-dependencies ... --python-version 2. There are also some distro-specific packaging and test dependencies, like devscripts, tox and libssl-dev on debian or ubuntu. Those pkg dependencies have now been broken out from common pkg deps to avoid trying to install them on centos/redhat/suse.
2017-06-14tools/run-centos: cleanups and move to using read-dependenciesScott Moser
These changes are all in an effort to get tools/run-centos using read-dependencies rather than the 'setup-centos' script with a separate set of dependencies listed. - tools/read-dependencies: support taking multiple --requirements options. This allows run-centos to get both test and build dependencies. Ultimately, I think it might be nicer for read-dependencies to take a list of "goals" (build, test, run or test-tox) rather than having the caller need to know to provide multiple --requirements. - packages/pkg-deps.json: drop the version on the sudo package. centos 6 has newer (1.8.6p3) version than listed, so its not a problem. - test_handler_disk_setup.py: a test case here was using assertLogs which is not present in the version of unittest2 that is available in centos 6 epel. We just adjust it to use with_logs = True. - tools/run-cents: - improve usage with example - add 'inside_as_cd' to provide the dir you want to cd first to. - avoid the intermediate tarball on disk in the container. - add 'prep' subcommand and use it to install pre-dependencies. - use read-dependencies.
2017-06-13pkg build ci: Add make ci-deps-<distro> target to install pkgsChad Smith
This change adds a couple of makefile targets for ci environments to install all necessary dependencies for package builds and test runs. It adds a number of arguments to ./tools/read-dependencies to facilitate reading pip dependencies, translating pip deps to system package names and optionally installing needed system-package dependencies on the local system. This relocates all package dependency and translation logic into ./tools/read-dependencies instead of duplication found in packages/brpm and packages/bddeb. In this branch, we also define buildrequires as including all runtime requires when rendering cloud-init.spec.in and debian/control files because our package build infrastructure will also be running all unit test during the package build process so we need runtime deps at build time. Additionally, this branch converts packages/(redhat|suse)/cloud-init.spec.in from cheetah templates to jinja to allow building python3 envs.
2017-06-13systemd: make cloud-final.service run before apt daily services.Scott Moser
This changes all cloud-init systemd units to run 'Before' the apt processes that run daily and may cause a lock on the apt database. apt-daily-upgrade.service contains 'After=apt-daily.service'. Thus following order is enforced, so we can just be 'Before' the first. apt-daily.service apt-daily-upgrade.service Note that this means only that apt-daily* will not run until cloud-init has entirely finished. Any other processes running apt-get operations are still affected by the global lock. LP: #1693361
2017-06-12selinux: Allow restorecon to be non-fatal.Ryan Harper
On some systems with python-libselinux a bug[1] related to recursive restorecon fails but the distro release does not yet include an update. This change will accept the error and log a warning. 1. https://bugzilla.redhat.com/show_bug.cgi?id=1406520 LP: #1686751
2017-06-12net: Allow netinfo subprocesses to return 0 or 1.Ryan Harper
On systems with selinux enabled, some of the networking commands executed successfully do not return 0. Allow these commands to return 1 since the output is valid. Ultimately we need to get this information in some way so that we can display it correctly. For now, work around the stack trace when selinux does not allow us to collect it. LP: #1686751
2017-06-12net: Allow for NetworkManager configurationRyan McCabe
In cases where the config json specifies nameserver entries, if there are interfaces configured to use dhcp, NetworkManager, if enabled, will clobber the /etc/resolv.conf that cloud-init has produced, which can break dns. If there are no interfaces configured to use dhcp, NetworkManager could clobber /etc/resolv.conf with an empty file. This patch adds a mechanism for dropping additional configuration into /etc/NetworkManager/conf.d/ and disables management of /etc/resolv.conf by NetworkManager when nameserver information is provided in the config. LP: #1693251 Signed-off-by: Ryan McCabe <rmccabe@redhat.com>
2017-06-09Use distro release version to determine if we use systemd in redhat specRyan Harper
The typical rpm build process will examine the spec file to determine which packages should be installed in the boot root. This requires the specfile to declare that it needs system. Provide this information by checking which version in which the rpm is being built and exporting requirements for systemd.
2017-06-08net: normalize data in network_state objectScott Moser
The network_state object's network and route keys would have different information depending upon how the network_state object was populated. This change cleans that up. Now: * address will always contain an IP address. * prefix will always include an integer value that is the network_prefix for the address. * netmask will be present only if the address is ipv4, and its value will always correlate to the 'prefix'.
2017-06-08Integration Testing: tox env, pyxld 2.2.3, and revamp frameworkWesley Wiedenmeier
Massive update to clean up and greatly enhance the integration testing framework developed by Wesley Wiedenmeier. - Updated tox environment to run integration test 'citest' to utilize pylxd 2.2.3 - Add support for distro feature flags - add framework for feature flags to release config with feature groups and overrides allowed in any release conf override level - add support for feature flags in platform and config handling - during collect, skip testcases that require features not supported by the image with a warning message - Enable additional distros (i.e. centos, debian) - Add 'bddeb' command to build a deb from the current working tree cleanly in a container, so deps do not have to be installed on host - Adds a command line option '--preserve-data' that ensures that collected data will be left after tests run. This also allows the directory to store collected data in during the run command to be specified using '--data-dir'. - Updated Read the Docs testing page and doc strings for pep 257 compliance
2017-06-08Chef: Update omnibus url to chef.io, minor doc changes.JJ Asghar
- Updated to standard chef.io url - Removed the port 4000, due to that has been deprecated - Added Note about the run_list not being required Signed-off-by: JJ Asghar <jj@chef.io>
2017-06-08tools: add centos scripts to build and testJoshua Powers
The added 'run-centos' does: - Creates centos 6 or 7 lxd container * Sets http_proxy variable for yum if set locally * Creates centos user - Push local tree * Tar's up working directory * Pushes to container and untars - Installs pip and yum dependencies - As user centos it can then based on flags: * runs unittests * run ./packages/brpm * run ./packages/brpm --srpm * artifact the built *.rpm
2017-06-08Drop cheetah python module as it is not needed by trunkRyan Harper
Cloud-init's template renderer does not require the use of cheetah (which is python2 only) so do not put it in the list of package requirements.
2017-06-08rhel/centos spec cleanups.Scott Moser
Many changes here to get us able to build rpms on CentOS 5 or 6 and RHEL. * add 'Requires' as 'BuildRequires' also. This allows us to run cloud-init tools in the build environment, and also will allow us to run tests in the build process. * build for both systemd and upstart (centos 5) init systems. * Add 'centos' as a variant Adding the variant means we can use the 'centos' user as default on centos rather than a 'fedora' or 'rhel'. * drop argparse from the requirements. On any system other than python 2.6, having a 'requirements' that mentions argparse just causes problems. Instead we add that Requires to the spec directly. * list dependency on dmidecode (as redhat distro spec had) * remove duplicate line in files section ({_unitdir}/cloud-*) * Use rpm macros for init-system chunks and drop use of init_system variable template * Add el6 only build-req on python-argparse * python-cheetah is not required in the build environment as the the spec is already rendered. (We will soon move the spec to jinja).
2017-06-08cloud.cfg: move to a template. setup.py changes along the way.Scott Moser
Here we move the config/cloud.cfg to be rendered as a template. That allows us to maintain deltas between distros in one place. Currently we use 'variant' variable to make decisions. A tools/render-cloudcfg is provided to render the file. There were changes to setup.py, MANIFEST.in to allow us to put all files into a virtual env installation and to render the cloud-config file in 'install' or 'bdist' targets. We have also included some config changes that were found in the redhat distro spec. * include some config changes from the redhat distro spec. The rendered cloud.cfg has some differences. Ubuntu: white space and comment changes only. Freebsd: - whitespace changes and comment changes - datasource_list definition moved to be closer to 'datasource'. - enable modules: migrator, write_files - move package-update-upgrade-install to final. The initial work was done by Josh Harlow.
2017-06-08Makefile: add deb-src and srpm targets. use PYVER more places.Scott Moser
This just adds targets for deb-src and srpm, and uses PYVER anywhere where we run a python program.
2017-06-08makefile: fix python 2/3 detection in the MakefileChad Smith
Fix detection of python in a non-python3 environment. The old path always used python3. The 2 fixes here are: a.) escape the '$' before the subshell. b.) use shell builtin 'command -v' rather than 'which' in case 'which' is not available.
2017-06-07snap: Removing snapcraft plug lineJoshua Powers
The snap is a classic snap and does not need to specify any plugs because it is already unconfined. LP: #1695333
2017-06-07RHEL/CentOS: Fix default routes for IPv4/IPv6 configuration.Andreas Karis
Since f38fa413176, default routes get added to both ifcfg-* and route-* and route6-* files. Default routes should only go to ifcfg-* files, otherwise the information is redundant. LP: #1696176
2017-06-07test: Fix pyflakes complaint of unused import.Joshua Powers
The jsonschema package is used only when available, but the lint check thinks the import is unused across pyflakes and flake8. In order to avoid having exceptions for both assert that the import works right after and the import is considered used. The '# NOQA' doesn't affect pyflakes (only flake8). LP: #1695918
2017-06-06NoCloud: support seed of nocloud from smbios informationVladimir Pouzanov
This allows the user to seed NoCloud in a trivial way from qemu/libvirt, by using a stock image and passing a single command line flag. No custom command line, no filesystem modification, no bootstrap disk image. This is particularly handy now that Ec2 backend is discouraged from use under bug 1660385. LP: #1691772
2017-06-06net: when selecting a network device, use natural sort orderMarc-Aurèle Brothier
The code deciding which interface to choose as the default to request the IP address through DHCP does not sort the interfaces correctly. On Ubuntu Xenial images for example, the interfaces are named ens1, ens2, ens3..., ens11, ... depending on the pci bus address. The python sorting will list 'ens11' before 'ens3' for example despite the fact that 'ens3' should be before 'ens11'. This patch address this issue and sort the interface names according to a human sorting. Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2017-06-06fix typos and remove whitespace in various docsStephan Telling
2017-06-05systemd: Fix typo in comment in cloud-init.target.Chen-Han Hsiao
Simply fix a typo in a comment. No functional change.
2017-06-02Tests: Skip jsonschema related unit tests when dependency is absent.Chad Smith
On some build environments we don't have python-jsonschema installed. Since this dependency is an optional runtime dependency, we can also make it an optional unit test dependency. Add a skip of related unittests when jsonschema is not present. Also, KeyError messages on CentOs don't have single quotes around the missing 'key-name'. Make our KeyError assertion a bit more flexible with the assertIn call. LP: #1695318