summaryrefslogtreecommitdiff
path: root/cloudinit
AgeCommit message (Collapse)Author
2017-08-25Configure logging module to always use UTC time.Ryan Harper
Currently the python logging module will default to a local time which may contain an TZ offset in the values it produces, but the logged time format does not contain the offset. Switching to UTC time for logging produces consistent values in the cloud-init.log file and avoids issues when the timezone is changed during boot. LP: #1713158
2017-08-25Log a helpful message if a user script does not include shebang.Andrew Jorgensen
A patch to allow scripts missing a #! to run by using shell=True was proposed but rejected. Instead we emit a log message to help the user understand what went wrong.
2017-08-25cli: Fix command line parsing of coniditionally loaded subcommands.Chad Smith
In an effort to save file load cost during system boot, certain subcommands, analyze and devel, do not get loaded unless the subcommand is specified on the commandline. Because setup.py entrypoint for cloud-init script doesn't specify sysv_args parameter when calling the CLI's main() we need main to read sys.argv into sysv_args so our subparser loading continues to work. LP: #1712676
2017-08-23cc_landscape & cc_puppet: Fix six.StringIO use in writing configsChad Smith
Both landscape and puppet modules had issues with the way they wrote /etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or python2. This branch adds initial unit tests for both modules which will get better exercise under both python2 and python3. The unit tests shed light on a few issues: - In the cc_landscape module py3 can't provide six.StringIO content to ConfigParser.write, so we need to use six.BytesIO instead - In the cc_puppet module, python <= 2.7 doesn't support using six.StringIO as a context manager, so we drop the context manager fanciness and directly set outputstream = StringIO(). - The docstring in cc_puppet is fixed to document the 'conf' sub-key requiring valid puppet section names for each key-value list. LP: #1699282 LP: #1710932
2017-08-22schema cli: Add schema subcommand to cloud-init cli and cc_runcmd schemaChad Smith
This branch does a few things: - Add 'schema' subcommand to cloud-init CLI for validating cloud-config files against strict module jsonschema definitions - Add --annotate parameter to 'cloud-init schema' to annotate existing cloud-config file content with validation errors - Add jsonschema definition to cc_runcmd - Add unit test coverage for cc_runcmd - Update CLI capabilities documentation This branch only imports development (and analyze) subparsers when the specific subcommand is provided on the CLI to avoid adding costly unused file imports during cloud-init system boot. The schema command allows a person to quickly validate a cloud-config text file against cloud-init's known module schemas to avoid costly roundtrips deploying instances in their cloud of choice. As of this branch, only cc_ntp and cc_runcmd cloud-config modules define schemas. Schema validation will ignore all undefined config keys until all modules define a strict schema. To perform validation of runcmd and ntp sections of a cloud-config file: $ cat > cloud.cfg <<EOF runcmd: bogus EOF $ python -m cloudinit.cmd.main schema --config-file cloud.cfg $ python -m cloudinit.cmd.main schema --config-file cloud.cfg \ --annotate Once jsonschema is defined for all ~55 cc modules, we will move this schema subcommand up as a proper subcommand of the cloud-init CLI.
2017-08-21tools: Add tooling for basic cloud-init performance analysis.Chad Smith
This branch adds cloudinit-analyze into cloud-init proper. It adds an "analyze" subcommand to the cloud-init command line utility for quick performance assessment of cloud-init stages and events. On a cloud-init configured instance, running "cloud-init analyze blame" will now report which cloud-init events cost the most wall time. This allows for quick assessment of the most costly stages of cloud-init. This functionality is pulled from Ryan Harper's analyze work. The cloudinit-analyze main script itself has been refactored a bit for inclusion as a subcommand of cloud-init CLI. There will be a followup branch at some point which will optionally instrument detailed strace profiling, but that approach needs a bit more discussion first. This branch also adds: * additional debugging topic to the sphinx-generated docs describing cloud-init analyze, dump and show as well as cloud-init single usage. * Updates the Makefile unittests target to include cloudinit directory because we now have unittests within that package. LP: #1709761
2017-08-15network: add v2 passthrough and fix parsing v2 config with bonds/bridge paramsRyan Harper
If the network-config sent to cloud-init is in version: 2 format then when rendering netplan, we can pass the content through and avoid consuming network_state elements. This removes the need for trying to map many v2 features onto network state where other renderers won't be able to use anyhow (for example match parameters for multi-interface configuration and wifi configuration support). Additionally ensure we retain bond/bridge v2 configuration in network state so when rendering to eni or sysconfig we don't lose the configuration - Drop the NotImplemented wifi exception, log a warning that it works for netplan only - Adjust unittests to new code path and output - Fix issue with v2 macaddress values getting dropped - Add unittests for consuming/validating v2 configurations LP: #1709180
2017-08-15vcloud directory: Guest Customization support for passwordsMaitreyee Saikia
This feature enables the following VMware VCloud Director functionality: 1. Setting admin password 2. Expire password. 3. Set admin password and expire. Password configuration is triggered only as part of a full recustomization, that happens either on first power on or when "poweron and full recustomization" is selected. Full customization flow is determined by marker files. Unique marker ids are generated when full recustomization is requested. And marker file based on these marker ids help to determine if we need to execute the above configuration.
2017-08-09ec2: Allow Ec2 to run in init-local using dhclient in a sandbox.Chad Smith
This branch is a prerequisite for IPv6 support in AWS by allowing Ec2 datasource to query the metadata source version 2016-09-02 about whether or not it needs to configure IPv6 on interfaces. If version 2016-09-02 is not present, fallback to the min_metadata_version of 2009-04-04. The DataSourceEc2Local not run on FreeBSD because dhclient in doesn't support the -sf flag allowing us to run dhclient without filesystem side-effects. To query AWS' metadata address @ 169.254.169.254, the instance must have a dhcp-allocated address configured. Configuring IPv4 link-local addresses result in timeouts from the metadata service. We introduced a DataSourceEc2Local subclass which will perform a sandboxed dhclient discovery which obtains an authorized IP address on eth0 and crawl metadata about full instance network configuration. Since ec2 IPv6 metadata is not sufficient in itself to tell us all the ipv6 knownledge we need, it only be used as a boolean to tell us which nics need IPv6. Cloud-init will then configure desired interfaces to DHCPv6 versus DHCPv4. Performance side note: Shifting the dhcp work into init-local for Ec2 actually gets us 1 second faster deployments by skipping init-network phase of alternate datasource checks because Ec2Local is configured in an ealier boot stage. In 3 test runs prior to this change: cloud-init runs were 5.5 seconds, with the change we now average 4.6 seconds. This efficiency could be even further improved if we avoiding dhcp discovery in order to talk to the metadata service from an AWS authorized dhcp address if there were some way to advertize the dhcp configuration via DMI/SMBIOS or system environment variables. Inspecting time costs of the dhclient setup/teardown in 3 live runs the time cost for the dhcp setup round trip on AWS is: test 1: 76 milliseconds dhcp discovery + metadata: 0.347 seconds metadata alone: 0.271 seconds test 2: 88 milliseconds dhcp discovery + metadata: 0.388 seconds metadata alone: 0.300 seconds test 3: 75 milliseconds dhcp discovery + metadata: 0.366 seconds metadata alone: 0.291 seconds LP: #1709772
2017-08-04cc_ntp: fallback on timesyncd configuration if ntp is not installableRyan Harper
Some systems like Ubuntu-Core do not provide an ntp package for installation but do include systemd-timesyncd (an ntp client). On such systems cloud-init will generate a timesyncd configuration using the 'servers' and 'pools' values as ntp hosts for timesyncd to use. LP: #1686485
2017-08-03net: Reduce duplicate code. Have get_interfaces_by_mac use get_interfaces.Scott Moser
get_interfaces_by_mac and get_interfaces just looked much alike. This makes get_interfaces_by_mac call get_interfaces.
2017-08-01sysconfig: Dont repeat header when rendering resolv.confRyan Harper
The sysconfig renderer duplicates the cloud-init header string when rendering resolv.conf file. This leads to resolv.conf file growing with every reboot of a system. Fix this by checking for the header when loading content from existing file. Update one of the sysconfig unittests with multiple render calls to simulate the reboot to check that we don't repeat the header. LP: #1701420
2017-08-01archlinux: Fix bug with empty dns, do not render 'lo' devices.Scott Moser
If no dns nameservers were provided a stack trace would occur. The changes here add some unit tests for the arch distro. Also avoids rendering an 'lo' interface. LP: #1663045 LP: #1706593
2017-07-28cloudinit.net: add initialize_network_device function and testsChad Smith
This is not yet called, but will be called in a subsequent Ec2-related branch to manually initialize a network interface with the responses using dhcp discovery without any dhcp-script side-effects. The functionality has been tested on Ec2 ubuntu and CentOS vms to ensure that network interface initialization works in both OS-types. Since there was poor unit test coverage for the cloudinit.net.__init__ module, this branch adds a bunch of coverage to the functions in cloudinit.net.__init. We can also now have unit tests local to the cloudinit modules. The benefits of having unittests under cloudinit module: - Proximity of unittest to cloudinit module makes it easier for ongoing devs to know where to augment unit tests. The tests.unittest directory is organizated such that it - Allows for 1 to 1 name mapping module -> tests/test_module.py - Improved test and module isolation, if we find unit tests have to import from a number of modules besides the module under test, it will better prompt resturcturing of the module. This also branch touches: - tox.ini to run unit tests found in cloudinit as well as include all test-requirements for pylint since we now have unit tests living within cloudinit package - setup.py to exclude any test modules under cloudinit when packaging
2017-07-25tests: Add initial tests for EC2 and improve a docstring.Scott Moser
EC2 was the original, but this adds some initial tests for that datasource. Also updates a docstring for an internal method.
2017-07-25locale: Do not re-run locale-gen if provided locale is system default.Scott Moser
If the system configure default in /etc/default/locale is set to the same value that is provided for cloud-init's "locale" setting, then do not re-run locale-gen. This allows images built with a locale already generated to not re-run locale-gen (which can be very heavy). Also here is a fix to invoke update-locale correctly and remove the internal writing of /etc/default/locale. We were calling update-locale <locale> This ends up having no affect. The more correct invocation is: update-locale LANG=<locale> Also added some support here should we ever want to change setting LANG to setting LC_ALL (or any other key). Lastly, a test change to allow us to use assert_not_called from mock. Versions of mock in CentOS 6 do not have assert_not_called.
2017-07-24archlinux: fix set hostname usage of write_file.Joshua Powers
cloud-init fails to set the hostname on Arch Linux because that _write_hostname passes conf instead of str(conf) to util.write_file. LP: #1705306
2017-07-22sysconfig: support subnet type of 'manual'.Scott Moser
The subnet type 'manual' was used as a way to declare a device and set an MTU for it but not assign network addresses. This updates the manual example config to handle that case and provides expected rendered output for sysconfig, eni, and netplan.
2017-07-21Drop rand_str() usage in DNS redirection detectionBob Aman
Making lots of random invalid DNS queries interferes with the ability of security teams to identify malicious or anomalous behavior from DNS logs. The same goal should be achievable with a consistent query for a name that is disallowed. LP: #1088611
2017-07-20sysconfig: use MACADDR on bonds/bridges to configure mac_addressRyan Harper
Previously, sysconfig rendered HWADDR for all interface types, but that value is only used to identify physical devices. Instead use MACADDR to configure the MAC on virtual devices, like bonds and bridges. - Sort bond slave list to ensure consistent ordering in sysconfig rendered files. - Add unittests for sysconfig rendering of bonds/bridges with mac_address LP: #1701417
2017-07-20net: eni route rendering missed ipv6 default route configRyan Harper
In some network configurations a network value of '::' and a netmask value of '::' were used to indicate a default IPV6 gateway. Commit d00da2d5 removed ipv6 'netmask' attributes and calculate a prefix length value instead. The eni route rendering failed to update the check to use prefix value of 0 to indicate the presence of an IPV6 default route. A broken ipv6 default route rendered like: post-up route add -net :: netmask :: gw 2001:4800:78ff:1b::1 || true And with this patch, it now renders like: post-up route add -A inet6 default gw 2001:4800:78ff:1b::1 || true LP: #1701097
2017-07-20sysconfig: enable mtu set per subnet, including ipv6 mtuRyan Harper
Render MTU values if present in subnet and route configurations for v4 and v6. LP: #1702513
2017-07-20sysconfig: handle manual type subnetsRyan Harper
Implement manual control for sysconfig by using ONBOOT=N. This allows an interface to be configured but not brought up. Note that ONBOOT is per-interface not per address. LP: #1687725
2017-07-20sysconfig: fix ipv6 gateway routesRyan Harper
Currently only the subnet is checked for 'ipv6' setting, however, the routes array may include a mix of v4 or v6 configurations, in particular, the gateway in a route may be ipv6, and if so, should export the value via IPV6_DEFAULTGW in the ifcfg-XXXX file. Additionally, if the route is v6, it should rendering a routes6-XXXX file; this is present but missing the 'dev <interface>' scoping. LP: #1694801
2017-07-20sysconfig: fix rendering of bond, bridge and vlan types.Ryan Harper
Previously, virtual types (bond, bridge, vlan) were almost completely broken. They would not get any network configuration (ip addresses or dhcp config) and or routes rendered. This fixes those issues. For bonds we now correctly render BONDING_SLAVE entries. Also add tests for simple bond, bridge and vlan. LP: #1695092
2017-07-19sysconfig: ipv6 and default gateway fixes.Ryan Harper
With this change, entries in IPV6ADDR and IPV6ADDR_SECONDARIES will now always be in format addr/prefix. When a subnet has a gateway will be written. If the gateway is ipv6, use the key IPV6_DEFAULTGW rather than GATEWAY. LP: #1704872
2017-07-19net: fix renaming of nics to support mac addresses written in upper case.Scott Moser
The network device renaming code previously required the case of the mac address input to match that of the data read from the system. For example, if user provided network config with mac address in upper case, then cloud-init would not rename the device correctly as /sys/class/net/address stores lower case values. The fix here is to always compare lower case mac addresses. LP: #1705147
2017-07-19tests: fixes for issues uncovered when moving to python 3.6.Scott Moser
This includes a few fixes found when testing with python 3.6. - fix eni renderer when target is None This just uses the util.target_path() in the event that target is None. - change test cases to not rely on the cached result of util.get_cmdline() and other cached globals. Update the base TestCase to unset that cache. - mock calls to system_is_snappy from the create_users test cases. - drop unused _pp_root in test_simple_run.py LP: #1703697
2017-07-18sysconfig: include GATEWAY value if set in subnetRyan Harper
Render the GATEWAY= value in interface files which have a gateway in the subnet configuration. LP: #1686856
2017-07-17Scaleway: add datasource with user and vendor data for Scaleway.Julien Castets
Here we add and enable by default a datasource for Scaleway cloud. The datasource quickly exits unless one of three things: a.) 'Scaleway' found as the system vendor b.) 'scaleway' found on the kernel command line. c.) the directory /var/run/scaleway exists (this is currently created by the scaleway initramfs module). One interesting bit of this particular datasource is that it requires the source port of the http request to be < 1024.
2017-07-17Support comments in content read by load_shell_content.Scott Moser
load_shell_content previously would not allow shell comment characters in the content being parsed. If comments=True is not passed then an exception would previously be raised as the line would not be guaranteed to have an '=' in it.
2017-07-11write_files: Remove log from helper function signatures.Andrew Jorgensen
Instead of passing around a 'log' reference to functions, just import logging and use that. This is the pattern that is now more common in cloud-init.
2017-06-29read_dmi_data: always return None when inside a container.Scott Moser
This fixes stacktrace and warning message that would be printed to the log if running inside a container and read_dmi_data tried to access a key that was not present. In a container, the /sys/class/dmi/id data is not relevant to the but to the host. Additionally an unpriviledged container might see strange behavior: # cd /sys/class/dmi/id/ # id -u 0 # ls -l chassis_serial -r-------- 1 nobody nogroup 4096 Jun 29 16:49 chassis_serial # cat chassis_serial cat: /sys/class/dmi/id/chassis_serial: Permission denied The solution here is to just always return None when running in a container. LP: #1701325
2017-06-27Azure: Add network-config, Refactor net layer to handle duplicate macs.Ryan Harper
On systems with network devices with duplicate mac addresses, cloud-init will fail to rename the devices according to the specified network configuration. Refactor net layer to search by device driver and device id if available. Azure systems may have duplicate mac addresses by design. Update Azure datasource to run at init-local time and let Azure datasource generate a fallback networking config to handle advanced networking configurations. Lastly, add a 'setup' method to the datasources that is called before userdata/vendordata is processed but after networking is up. That is used here on Azure to interact with the 'fabric'.
2017-06-15FreeBSD: Make freebsd a variant, fix unittests and tools/build-on-freebsd.Scott Moser
- Simplify the logic of 'variant' in util.system_info much of the data from https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version - fix get_resource_disk_on_freebsd when running on a system without an Azure resource disk. - fix tools/build-on-freebsd to replace oauth with oauthlib and add bash which is a dependency for tests. - update a fiew places that were checking for freebsd but not using the util.is_FreeBSD()
2017-06-15FreeBSD: fix test failureScott Moser
The previous commit caused test failure. This separates out _check_freebsd_cdrom and mocks it in a test rather than patching open.
2017-06-15FreeBSD: replace ifdown/ifup with "ifconfig down" and "ifconfig up".Hongjiang Zhang
Fix the issue caused by different commands on Linux and FreeBSD. On Linux, we used ifdown and ifup to enable and disable a NIC, but on FreeBSD, the counterpart is "ifconfig down" and "ifconfig up". LP: #1697815
2017-06-15FreeBSD: fix cdrom mounting failure if /mnt/cdrom/secure did not exist.Hongjiang Zhang
The current method is to attempt to mount the cdrom (/dev/cd0), if it is successful, /dev/cd0 is configured, otherwise, it is not configured. The problem is it forgets to check whether the mounting destination folder is created or not. As a result, mounting attempt failed even if cdrom is ready. LP: #1696295
2017-06-15main: Don't use templater to format the welcome messageAndrew Jorgensen
Some versions of Cheetah returned everything as unicode by default (not utf-8 or ascii) and some varieties of syslog would choke on unicode. Jinja2 is probably fine, but Python's format() is perfectly adequate for a short message like the welcome message. Reviewed-by: Tom Kirchner <tjk@amazon.com> Reviewed-by: Ben Cressey <bcressey@amazon.com>
2017-06-15write_file(s): Print permissions as octal, not decimalAndrew Jorgensen
Unix file modes are usually represented as octal, but they were being interpreted as decimal, for example 0o644 would be printed as '420'. Reviewed-by: Tom Kirchner <tjk@amazon.com>
2017-06-12selinux: Allow restorecon to be non-fatal.Ryan Harper
On some systems with python-libselinux a bug[1] related to recursive restorecon fails but the distro release does not yet include an update. This change will accept the error and log a warning. 1. https://bugzilla.redhat.com/show_bug.cgi?id=1406520 LP: #1686751
2017-06-12net: Allow netinfo subprocesses to return 0 or 1.Ryan Harper
On systems with selinux enabled, some of the networking commands executed successfully do not return 0. Allow these commands to return 1 since the output is valid. Ultimately we need to get this information in some way so that we can display it correctly. For now, work around the stack trace when selinux does not allow us to collect it. LP: #1686751
2017-06-12net: Allow for NetworkManager configurationRyan McCabe
In cases where the config json specifies nameserver entries, if there are interfaces configured to use dhcp, NetworkManager, if enabled, will clobber the /etc/resolv.conf that cloud-init has produced, which can break dns. If there are no interfaces configured to use dhcp, NetworkManager could clobber /etc/resolv.conf with an empty file. This patch adds a mechanism for dropping additional configuration into /etc/NetworkManager/conf.d/ and disables management of /etc/resolv.conf by NetworkManager when nameserver information is provided in the config. LP: #1693251 Signed-off-by: Ryan McCabe <rmccabe@redhat.com>
2017-06-08net: normalize data in network_state objectScott Moser
The network_state object's network and route keys would have different information depending upon how the network_state object was populated. This change cleans that up. Now: * address will always contain an IP address. * prefix will always include an integer value that is the network_prefix for the address. * netmask will be present only if the address is ipv4, and its value will always correlate to the 'prefix'.
2017-06-08Chef: Update omnibus url to chef.io, minor doc changes.JJ Asghar
- Updated to standard chef.io url - Removed the port 4000, due to that has been deprecated - Added Note about the run_list not being required Signed-off-by: JJ Asghar <jj@chef.io>
2017-06-08rhel/centos spec cleanups.Scott Moser
Many changes here to get us able to build rpms on CentOS 5 or 6 and RHEL. * add 'Requires' as 'BuildRequires' also. This allows us to run cloud-init tools in the build environment, and also will allow us to run tests in the build process. * build for both systemd and upstart (centos 5) init systems. * Add 'centos' as a variant Adding the variant means we can use the 'centos' user as default on centos rather than a 'fedora' or 'rhel'. * drop argparse from the requirements. On any system other than python 2.6, having a 'requirements' that mentions argparse just causes problems. Instead we add that Requires to the spec directly. * list dependency on dmidecode (as redhat distro spec had) * remove duplicate line in files section ({_unitdir}/cloud-*) * Use rpm macros for init-system chunks and drop use of init_system variable template * Add el6 only build-req on python-argparse * python-cheetah is not required in the build environment as the the spec is already rendered. (We will soon move the spec to jinja).
2017-06-08cloud.cfg: move to a template. setup.py changes along the way.Scott Moser
Here we move the config/cloud.cfg to be rendered as a template. That allows us to maintain deltas between distros in one place. Currently we use 'variant' variable to make decisions. A tools/render-cloudcfg is provided to render the file. There were changes to setup.py, MANIFEST.in to allow us to put all files into a virtual env installation and to render the cloud-config file in 'install' or 'bdist' targets. We have also included some config changes that were found in the redhat distro spec. * include some config changes from the redhat distro spec. The rendered cloud.cfg has some differences. Ubuntu: white space and comment changes only. Freebsd: - whitespace changes and comment changes - datasource_list definition moved to be closer to 'datasource'. - enable modules: migrator, write_files - move package-update-upgrade-install to final. The initial work was done by Josh Harlow.
2017-06-07RHEL/CentOS: Fix default routes for IPv4/IPv6 configuration.Andreas Karis
Since f38fa413176, default routes get added to both ifcfg-* and route-* and route6-* files. Default routes should only go to ifcfg-* files, otherwise the information is redundant. LP: #1696176
2017-06-06NoCloud: support seed of nocloud from smbios informationVladimir Pouzanov
This allows the user to seed NoCloud in a trivial way from qemu/libvirt, by using a stock image and passing a single command line flag. No custom command line, no filesystem modification, no bootstrap disk image. This is particularly handy now that Ec2 backend is discouraged from use under bug 1660385. LP: #1691772
2017-06-06net: when selecting a network device, use natural sort orderMarc-Aurèle Brothier
The code deciding which interface to choose as the default to request the IP address through DHCP does not sort the interfaces correctly. On Ubuntu Xenial images for example, the interfaces are named ens1, ens2, ens3..., ens11, ... depending on the pci bus address. The python sorting will list 'ens11' before 'ens3' for example despite the fact that 'ens3' should be before 'ens11'. This patch address this issue and sort the interface names according to a human sorting. Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>