summaryrefslogtreecommitdiff
path: root/cloudinit
AgeCommit message (Collapse)Author
2018-09-05tests: Disallow use of util.subp except for where needed.Scott Moser
In many cases, cloud-init uses 'util.subp' to run a subprocess. This is not really desirable in our unit tests as it makes the tests dependent upon existance of those utilities. The change here is to modify the base test case class (CiTestCase) to raise exception any time subp is called. Then, fix all callers. For cases where subp is necessary or actually desired, we can use it via   a.) context hander CiTestCase.allow_subp(value)   b.) class level self.allowed_subp = value Both cases the value is a list of acceptable executable names that will be called (essentially argv[0]). Some cleanups in AltCloud were done as the code was being updated.
2018-09-05sysconfig: refactor sysconfig to accept distro specific templates pathsRyan Harper
Multiple distros use sysconfig format but have different content and paths to certain files. Update distros to specify these template paths in their renderer_configs dictionary.
2018-09-01Add unit tests for config/cc_ssh.pyFrancis Ginther
These tests focus on the apply_credentials method and the ssh setup for root and a distro default user.
2018-08-31Fix the built-in cloudinit/tests/helpers:skipIfScott Moser
this version uses unittest2 skipIf which is present in our python 2.6 environment.
2018-08-31hyperv_reporting_handler: simplify threaded publisherRyan Harper
Switch the implementation to a daemon thread which uses a blocking get from the Queue. No additional locking or flag checking is needed since the Queue itself handles acquiring the lock as needed. cloud-init only has a single producer (the main thread calling publish) and the consumer will read all events in the queue and write them out. Using the daemon mode of the thread handles flushing the queue on main exit in python3; in python2.7 we handle the EOFError that results when the publish thread calls to get() fails indicating the main thread has exited. The result is that the handler is no longer spawing a thread on each publish event but rather creates a single thread when we start up the reporter and we remove any additional use of separate locks and flags as we only have a single Queue object and we're only calling queue.put() from main thread and queue.get() from consuming thread.
2018-08-28VMWare: Fix a network config bug in vm with static IPv4 and no gateway.Pengpeng Sun
The issue is when customize a VM with static IPv4 and without gateway, it will still extend route list and will loop a gateways list which is None. This fix is to make sure when no gateway is here, it will not extend route list. LP: #1766538
2018-08-24logging: Add logging config type hyperv for reporting via Azure KVPAndy Liu
Linux guests can provide information to Hyper-V hosts via KVP. KVP allows the guests to provide any string key-value-pairs back to the host's registry. On linux, kvp communication pools are presented as pool files in /var/lib/hyperv/.kvp_pool_#. The following reporting configuration can enable this kvp reporting in addition to default logging if the pool files exist: reporting:     logging:         type: log     telemetry:         type: hyperv
2018-08-17Add datasource Oracle Compute Infrastructure (OCI).Scott Moser
This adds a Oracle specific datasource that functions with OCI. It is a simplified version of the OpenStack metadata server with support for vendor-data. It does not support the OCI-C (classic) platform. Also here is a move of BrokenMetadata to common 'sources' as this was the third occurrence of that class.
2018-08-17azure: allow azure to generate network configuration from IMDS per boot.Chad Smith
Azure datasource now queries IMDS metadata service for network configuration at link local address http://169.254.169.254/metadata/instance?api-version=2017-12-01. The azure metadata service presents a list of macs and allocated ip addresses associated with this instance. Azure will now also regenerate network configuration on every boot because it subscribes to EventType.BOOT maintenance events as well as the 'first boot' EventType.BOOT_NEW_INSTANCE. For testing add azure-imds --kind to cloud-init devel net_convert tool for debugging IMDS metadata. Also refactor _get_data into 3 discrete methods:   - is_platform_viable: check quickly whether the datasource is     potentially compatible with the platform on which is is running   - crawl_metadata: walk all potential metadata candidates, returning a     structured dict of all metadata and userdata. Raise InvalidMetaData on     error.   - _get_data: call crawl_metadata and process results or error. Cache     instance data on class attributes: metadata, userdata_raw etc.
2018-08-17Scaleway: Add network configuration to the DataSourceLouis Bouchard
DEP_NETWORK is removed since the network_config must run at each boot. New EventType.BOOT event is used for that. Network is brought up early to fetch the metadata which is required to configure the network (ipv4 and/or v6). Adds unittests for the following and fixes test_common for LOCAL and NETWORK sets.
2018-08-06netplan: Correctly render macaddress on a bonds and bridges when provided.Scott Moser
When converting network config v1 to netplan, we were not correctly rendering the 'macaddress' key on a bond. Not that the difference in spelling between v1 'mac_address' and v2 'macaddress' is intentional. Also fixed here is rendering of the macaddress for bridges. LP: #1784699
2018-08-06tools: Add 'net-convert' subcommand command to 'cloud-init devel'.Scott Moser
Move the tools/net-convert.py to be exposed as part of 'cloud-init devel' subcommands. It can now be called like: $ cloud-init devel net-convert Or, if you just have checked out source (and no cli executable):   $ python3 -m cloudinit.cmd.devel.net_convert or   $ python3 -m cloudinit.cmd.main devel net-convert
2018-08-03OpenNebula: Fix null gateway6Akihiko Ota
The OpenNebula data source generates an invalid netplan yaml file if the IPv6 gateway is not defined in context.sh. LP: #1768547
2018-07-31oracle: fix detect_openstack to report True on OracleCloud.com DMI dataChad Smith
The OpenStack datasource in 18.3 changed to detect data in the init-local stage instead of init-network and attempted to redetect OpenStackLocal datasource on Oracle across reboots. The function detect_openstack was added to quickly detect whether a platform is OpenStack based on dmi product_name or chassis_asset_tag and it was a bit too strict for Oracle in checking for 'OpenStack Nova'/'Compute' DMI product_name. Oracle's DMI product_name reports 'SAtandard PC (i440FX + PIIX, 1996)' and DMI chassis_asset_tag is 'OracleCloud.com'. detect_openstack function now adds 'OracleCloud.com' as a supported value 'OracleCloud.com' to valid chassis-asset-tags for the OpenStack datasource. LP: #1784685
2018-07-26update_metadata re-config on every boot comments and tests not quite rightMike Gerdts
The comment in update_metadata() that explains how a datasource should enable network reconfig on every boot presumes that EventType.BOOT_NEW_INSTANCE is a subset of EventType.BOOT. That's not the case, and as such a datasource that needs to configure networking when it is a new instance and every boot needs to include both event types. To make the situation above easier to debug, update_metadata() now logs when it returns false. To make it so that datasources do not need to test before appending to the update_events['network'], it is changed from a list to a set. test_update_metadata_only_acts_on_supported_update_events is updated to allow datasources to support EventType.BOOT. Author: Mike Gerdts <mike.gerdts@joyent.com>
2018-07-21pylint: Fix pylint warnings reported in pylint 2.0.0.Scott Moser
Pylint 2.0.0 was recently released and complains more about logging-not-lazy than it used to. I've fixed those warnings, here. The changes in rh_subscription are more extensive. pylint may be complaining incorrectly there, but the tests were not correctly un-doing all of their mock/patching. This cleans those up and makes pylint happy.
2018-07-20get_linux_distro: add support for rhel via redhat-release.Scott Moser
Add examples and tests for RHEL values of redhat-release and os-release. These examples were collected from IBMCloud images. on rhel systems 'platform.dist()' returns 'redhat' rather than 'rhel' so we have adjusted the response to align there.
2018-07-20get_linux_distro: add support for centos6 and rawhide flavors of redhatChad Smith
An empty /etc/os-release exists on some redhat images, most notably the COPR build images of centos6 and rawhide. On platforms missing /etc/os-release or having an empty /etc/os-release file, use _parse_redhat_release on rhel-based images to obtain distribution and release codename information. LP: #1781229
2018-07-09docs: note in rtd about avoiding /tmp when writing filesChad Smith
LP: #1727876
2018-07-09ubuntu,centos,debian: get_linux_distro to align with platform.distChad Smith
A recent commit added get_linux_distro to replace the deprecated python platform.dist module behavior before it is dropped from python. It added behavior that was compliant on OpenSuSE and SLES, by returning (<distro_name>, <distro_version>, <cpu-arch>). Fix get_linux_distro to behave more like the specific distribution's platform.dist on ubuntu, centos and debian, which will return the distribution release codename as the third element instead of <cpu-arch>. SLES and OpenSUSE will retain their current behavior. Examples follow: ('sles', '15', 'x86_64') ('opensuse', '42.3', 'x86_64') ('debian', '9', 'stretch') ('ubuntu', '16.04', 'xenial') ('centos', '7', 'Core') LP: #1780481
2018-07-01update_metadata: a datasource can support network re-config every bootChad Smith
Very basic type definitions are now defined to distinguish 'boot' events from 'new instance (first boot)'. Event types will now be handed to a datasource.update_metadata method which can determine whether to refresh its metadata and re-render configuration based on that source event. A datasource can 'subscribe' to an event by setting up the update_events attribute on the datasource class which describe what config scope is updated by a list of matching events. By default datasources will have the following update_events: {'network': [EventType.BOOT_NEW_INSTANCE]} This setting says the datasource will re-write network configuration only on first boot of a new instance or when the instance id changes. New methods are now present on the datasource: - clear_cached_attrs: Resets cached datasource attributes to values listed in datasource.cached_attr_defaults. This is performed prior to processing a fresh metadata process to avoid keeping old/invalid cached data around. - update_metadata: accepts source_event_types to determine if the metadata should be crawled again and processed
2018-06-28Retry on failed import of gpg receive keys.Scott Moser
When cloud-init tries to read a key from a keyserver, it will now retry twice with 1 second in between each. Retries of import are done by default because keyservers can be unreliable. Additionally, there is no way to determine the difference between a non-existant key and a failure. In both cases gpg (at least 2.2.4) exits with status 2 and stderr: "keyserver receive failed: No data" It is assumed that a key provided to cloud-init exists on the keyserver so re-trying makes better sense than failing. Examples of things that made receive keys particularly unreliable:   https://bitbucket.org/skskeyserver/sks-keyserver/issues/57   https://bitbucket.org/skskeyserver/sks-keyserver/issues/60 There is also a change here from 'gpg --recv' to the longer 'gpg --recv-keys'. That option is functional and working back to centos 6 (gpg 2.0.14) and ubuntu 14.04 (gpg 1.4.16).
2018-06-28docs: Fix a small spelling error.Oz N Tiram
standargs -> standards.
2018-06-19release 18.3Chad Smith
Bump the version in cloudinit/version.py to be 18.3 and update ChangeLog. LP: #1777743
2018-06-19docs: represent sudo:false in docs for user_groups config moduleChad Smith
2018-06-19Explicitly prevent `sudo` access for user moduleJacob Bednarz
To deny a user elevated access, you can omit the `sudo` key from the `users` dictionary. This works fine however it's implicitly defined based on defaults of `cloud-init`. If the project moves to have `sudo` access allowed for all by default (quite unlikely but still possible) this will catch a few people out. This introduces the ability to define an explicit `sudo: False` in the `users` dictionary and it will prevent `sudo` access. The behaviour is identical to omitting the key. LP: #1771468
2018-06-15lxd: Delete default network and detach device if lxd-init created them.Scott Moser
Newer versions (3.0.1+) of lxd create the 'lxdbr0' network when 'lxd init --auto' is invoked. When cloud-init is given a network configuration to pass on to lxc and that config had no name specified or 'lxdbr0', then cloud-init would fail to create the network as it already exists. Similarly, we need to remove the device from the default profile so that the attach code can work. Also, add a _lxc method and use it to make sure we're getting the --force-local flag everywhere. LP: #1776958
2018-06-15openstack: avoid unneeded metadata probe on non-openstack platformsChad Smith
OpenStack datasource is now discovered in init-local stage. In order to probe whether OpenStack metadata is present, it performs a costly sandboxed dhclient setup and metadata probe against http://169.254.169.254 for openstack data. Cloud-init properly detects non-OpenStack on EC2, but it spends precious time probing the metadata service also resulting in a confusing WARNING log about 'metadata not present'. To avoid the wasted cycles, and confusing warning, get_data will call a detect_openstack function to quickly determine whether the platform looks like OpenStack before trying to setup network to probe and crawl the metadata service. LP: #1776701
2018-06-15stages: fix tracebacks if a module stage is undefined or emptyRobert Schweikert
In /etc/cloud/cloud.cfg, users and imagees can configure which modules run during a specific cloud-init stage by modifying one of the following lists: cloud_init_modules, cloud_init_modules, cloud_init_final_modules. If any of the configured module lists are absent or empty, cloud-init will emit the same message it already does for existing lists that only contain modules which are not unsupported on that platform: No 'config' modules to run under section 'cloud_config_modules' LP: #1770462
2018-06-13Be more safe on string/bytes when writing multipart user-data to disk.Scott Moser
When creating the multipart mime message that is written as user-data.txt.i, cloud-init losing data on conversion to some things as a string. LP: #1768600 Author: Scott Moser <smoser@ubuntu.com> Co-Authored-By: Chad Smith <chad.smith@canonical.com>
2018-06-13Fix get_proc_env for pids that have non-utf8 content in environment.Scott Moser
There is no requirement that the environment of a process contains only utf-8 data. This modifies get_proc_env to support it reading data as binary and decoding if provided with an encoding. The default case is now that we now do: contents.decode('utf-8', 'replace') rather than contents.decode('utf-8', 'strict') LP: #1775371
2018-06-12netplan: fix mtu if provided by network config for all rendered typesChad Smith
When network configuration for any interface defines maximum transmission values (MTU) the netplan, eni and sysconfig renders will take into account any device-level, or subnet-level mtu values. When network configuration has conflicting device-level and ipv4 subnet mtu values, the subnet-specific value is honored and a warning will be logged about any ignored device-level setting. LP: #1774666
2018-06-08subp: support combine_capture argument.Scott Moser
This adds 'combine_capture' argument as was present in curtin's subp. It is useful to get interleaved output of a command. I noticed a need for it when looking at user_data_rhevm in DataSourceAltCloud. That will run a subcommand, logging its stdout but swallowing its stderr. Another thing to change to use this would be in udevadm_settle which currently just returns the subp() call. Also, add the docstring copied from curtin's subp.
2018-05-29util: add get_linux_distro function to replace platform.distRobert Schweikert
Allow the user to set the distribution with --distro argument to setup.py. Fall back is to read /etc/os-release. Final backup is to use platform.dist() Python function. The platform.dist() function is deprecated and will be removed in Python 3.7 LP: #1745235
2018-05-29pyflakes: fix unused variable references identified by pyflakes 2.0.0.Scott Moser
A newer version of pyflakes (2.0.0) was released. It identifed some unused variables that version 1.6.0 did not identify. The change here merely fixes those unused variables.
2018-05-24Enable SmartOS network metadata to work with netplan via per-subnet routesDan McDonald
- Updated datadict reference URL - Store sdc:routes metadata in DatasourceSmartOS - Map sdc:routes values to per-interface subnet configuration - Added unittest Co-authored-by: Mike Gerdts <mike.gerdts@joyent.com> LP: #1763512
2018-05-23openstack: Allow discovery in init-local using dhclient in a sandbox.Chad Smith
Network has not yet been configured in the init-local stage so the openstack datasource will use dhcp-client to temporarily obtain an ipv4 address and query the metadata service at http://169.254.169.254 to get network_data.json configuration. If present, the datasource will return network_config version 1 config based on that network_data.json content. Previously OpenStack datasource only setup dhcp on the fallback interface so this represents a change in behavior to react to the full config provided by openstack. Also significant to OpenStack is the separation of a _crawl_data operation from get_data(). crawl_data walks the available metadata services and returns a dict of discovered content. get_data consumes the crawled_data,  caches it in the datasource and reacts to that data. /run/cloud-init/instance-data.json now published network_data.json or ec2_metadata key if that data is present on any datasource. The main reasons for the separation of crawl from get_data:  * Enable performance metrics of cloud-init's metadata crawls on each  * Enable cloud-init modules and scripts to query and consume metadata    content which may have updated/changed after cloud-init's initial cache    during instance boot. (Think hotplug) Also generalize common logic to base DataSource class/module:  * Move to a common UNSET variable up into base datasource module fix EC2,    ConfigDrive, OpenStack, SmartOS to use the global.  * Drop get_url_settings from Ec2, CloudStack and OpenStack and generalize    DataSource.get_url_params(). Allow subclasses to override url_max_wait,    url_timeout and url_retries params.  * Rename get_network_metadata bool to perform_dhcp_setup as it designates    whether EphemeralDHCPv4 setup is required before crawling metadata. LP: #1749717
2018-05-23tests: Avoid using https in httpretty, improve HttPretty test case.Scott Moser
On OpenSuSE 42.3, we would get errors running tests/unittests/test_handler/test_handler_chef.py  - test_myhttps_nonet raises a UnmockedError    No mocking was registered, and real connections are not allowed  - test_myhttps_net raises SSLError    ("bad handshake: SysCallError(32, 'EPIPE')",) This fixes the errors by just using http instead of https. Also it modifies the HttprettyTestCase to do the httpretty activate and deactivate itself in setUp and tearDown. Then we don't have to decorate individual test_ methods. Also, we set    httpretty.HTTPretty.allow_net_connect = False Test cases here should not reach out to a network resource. LP: #1771659
2018-05-23yaml_load/schema: Add invalid line and column nums to error messageChad Smith
Yaml tracebacks are generally hard to read for average users. Add a bit of logic to util.yaml_load and schema validation to look for YAMLError.context_marker or problem_marker line and column counts. No longer log the full exceeption traceback from the yaml_load error, instead just LOG.warning for the specific error and point to the offending line and column where the problem exists.
2018-05-23Azure: Ignore NTFS mount errors when checking ephemeral drivePaul Meyer
The Azure data source provides a method to check whether a NTFS partition on the ephemeral disk is safe for reformatting to ext4. The method checks to see if there are customer data files on the disk. However, mounting the partition fails on systems that do not have the capability of mounting NTFS. Note that in this case, it is also very unlikely that the NTFS partition would have been used by the system (since it can't mount it). The only case would be where an update to the system removed the capability to mount NTFS, the likelihood of which is also very small. This change allows the reformatting of the ephemeral disk to ext4 on systems where mounting NTFS is not supported.
2018-05-22Update version.version_string to contain packaged version.Scott Moser
This modifies version.version_string to support having the package build write the *packaged* version in with a easy replace. Then, when cloud-init reports its version it will include the full packaged version. Also modified here are upstream package build files to get that done. Note part of the trickery in packages/debian/rules.in was to avoid the 'basic' templater consuming the '$variable' variable names. LP: #1770712
2018-05-22cc_mounts: Do not add devices to fstab that are already present.Lars Kellogg-Stedman
Do not add new entries to /etc/fstab for devices that already have an existing fstab entry. Resolves: rhbz#1542578
2018-05-17read_file_or_url: move to url_helper, fix bug in its FileResponse.Scott Moser
The result of a read_file_or_url on a file and on a url would differ in behavior. str(UrlResponse) would return UrlResponse.contents.decode('utf-8') while str(FileResponse) would return str(FileResponse.contents) The difference being "b'foo'" versus "foo". As part of the general goal of cleaning util, move read_file_or_url into url_helper.
2018-05-16flake8: fix flake8 errors in previous commit.Scott Moser
2018-05-16typos: Fix spelling mistakes in cc_mounts.py log messagesStephen Ford
This also makes some of the messages more consistent.
2018-05-15ds-identify: recognize container-other as a container, test SmartOS.Scott Moser
In playing with a SmartOS container I found that ds-identify did not identify the container there as a container. Systemd-detect-virt identifies it as 'container-other'. Also here are tests for ds-identify for the SmartOS platform identification, and some indentation fixes in ds-identify.
2018-05-09SmartOS: fix get_interfaces for nics that do not have addr_assign_type.Scott Moser
When attempting to apply network configuration for SmartOS's container platform, cloud-init would not identify nics. The nics on provided in this container service do not have 'addr_assign_type'. That was being interpreted as being a "stolen" mac, and would be filtered out by get_interfaces.
2018-05-04pycodestyle: Fix deprecated string literals, move away from flake8.Chad Smith
Fix remaining pycodesytle warnings related to invalid string literals introduced in more recent pycodeflakes versions https://bugs.python.org/issue27364 . Also stop using flake8 in tox as it is incompatible with newer versions of pyflakes. Instead we now add tox environments for pycodestyle and pyflakes individually. Set the versions in both pycodestyle and pyflakes to the currently available versions.
2018-05-03azure: Add reported ready marker file.Joshua Chan
This change is for Azure VM Preprovisioning. A bug was found when after azure VMs report ready the first time, during the time when VM is polling indefinitely for the new ovf-env.xml from Instance Metadata Service (IMDS), if a reboot happens, we send another report ready signal to the fabric, which deletes the reprovisioning data on the node. This marker file is used to fix this issue so that we will only send a report ready signal to the fabric when no marker file is present. Then, create a marker file so that when a reboot does occur, we check if a marker file has been created and decide whether we would like to send the repot ready signal. LP: #1765214
2018-05-01FreeBSD: Invoke growfs on ufs filesystems such that it does not prompt.Harm Weites
By default, FreeBSD's growfs runs interactively asking a question which can be mitigated using the '-y' command line option. The fix here is simply to pass -y to growfs to avoid the prompt. LP: #1404745