summaryrefslogtreecommitdiff
path: root/cloudinit/sources
AgeCommit message (Collapse)Author
2019-04-18mount_cb: do not pass sync and rw options to mountGonéri Le Bouder
On FreeBSD, mount_cd9660 does not accept the sync option that is enabled by default. In addition, the sync is only useful with the `rw` mode. However the `rw` mode was never used. This patch removes the `rw` and `sync` parameter of `mount_cb` to simplify the code base and resolve the FreeBSD issue. LP: #1645824
2019-04-10Revert "DataSource: move update_events from a class to an instance..."Daniel Watkins
Moving update_events from a class attribute to an instance attribute means that it doesn't exist on DataSource objects that are unpickled, causing tracebacks on cloud-init upgrade. As this change is only required for cloud-init installations which don't utilise ds-identify, we're backing it out to be reintroduced once the upgrade path bug has been addressed. This reverts commit f2fd6eac4407e60d0e98826ab03847dda4cde138.
2019-04-09Change DataSourceNoCloud to ignore file system label's case.Risto Oikarinen
NoCloud data source now accepts both 'cidata' and 'CIDATA' as filesystem labels. This is similar to DataSourceConfigDrive's support for 'config-2' and 'CONFIG-2'.
2019-04-03Azure: Treat _unset network configuration as if it were absentJason Zions (MSFT)
When the Azure datasource persists all of its metadata to the instance directory, it deliberately sets the self.network_config value to be the sources.UNSET value. The goal is to ensure that each time the system boots, fresh network configuration data is fetched from the cloud platform so that any control plane changes will take effect. When a VM is first created, there's no pickled instance to restore, so self._network_config is None, resulting in self.network_config() properly building a new config. Azure suffered from LP: #1801364 which prevented ds from being stored in obj.pkl in the instance directory, so subsequent reboots always regenerated their network configuration. Commit 0dc3a77f41f4544e4cb5a41637af7693410d4cdf introduced a new bug in which self.network_config() assumed the self._network_config value was either None or trustable; when the config was unpickled, that value was _unset, thus breaking the assumption. LP: #1823084
2019-04-03DatasourceAzure: add additional logging for azure datasourceAnh Vo
Create an Azure logging decorator and use additional ReportEventStack context managers to provide additional logging details.
2019-03-26Azure: Ensure platform random_seed is always serializable as JSON.Jason Zions (MSFT)
The Azure platform surfaces random bytes into /sys via Hyper-V. Python 2.7 json.dump() raises an exception if asked to convert a str with non-character content, and python 3.0 json.dump() won't serialize a "bytes" value. As a result, c-i instance data is often not written by Azure, making reboots slower (c-i has to repeat work). The random data is base64-encoded and then decoded into a string (str or unicode depending on the version of Python in use). The base64 string has just as many bits of entropy, so we're not throwing away useful "information", but we can be certain json.dump() will correctly serialize the bits.
2019-03-14DataSource: move update_events from a class to an instance attributeDaniel Watkins
Currently, DataSourceAzure updates self.update_events in __init__. As update_events is a class attribute on DataSource, this updates it for all instances of classes derived from DataSource including those for other clouds. This means that if DataSourceAzure is even instantiated, its behaviour is applied to whichever data source ends up being used for boot. To address this, update_events is moved from a class attribute to an instance attribute (that is therefore populated at instantiation time). This retains the defaults for all DataSource sub-class instances, but avoids them being able to mutate the state in instances of other DataSource sub-classes. update_events is only ever referenced on an instance of DataSource (or a sub-class); no code relies on it being a class attribute. (In fact, it's only used within methods on DataSource or its sub-classes, so it doesn't even _need_ to remain public, though I think it's appropriate for it to be public.) DataSourceScaleway is also updated to move update_events from a class attribute to an instance attribute, as the class attribute would now be masked by the DataSource instance attribute. LP: #1819913
2019-03-12DataSourceEc2: update RELEASE_BLOCKER to be more accurateDaniel Watkins
Our previous understanding of the upgrade issue was incomplete; it turns out the only change we need is the one now outlined.
2019-03-04helpers/openstack: Treat unknown link types as physicalDaniel Watkins
Some deployments of OpenStack expose link types to the guest which cloud-init doesn't recognise. These will almost always be physical, so we can operate more robustly if we assume that they are (whilst warning the user that we're seeing something unexpected). LP: #1639263
2019-02-26util: don't determine string_types ourselvesDaniel Watkins
six already provides this for us, and we're already paying the cost to determine it there; no need to do it twice.
2019-02-22azure: Filter list of ssh keys pulled from fabricJason Zions (MSFT)
The Azure data source is expected to expose a list of ssh keys for the user-to-be-provisioned in the crawled metadata. When configured to use the __builtin__ agent this list is built by the WALinuxAgentShim. The shim retrieves the full set of certificates and public keys exposed to the VM from the wireserver, extracts any ssh keys it can, and returns that list. This fix reduces that list of ssh keys to just the ones whose fingerprints appear in the "administrative user" section of the ovf-env.xml file. The Azure control plane exposes other ssh keys to the VM for other reasons, but those should not be added to the authorized_keys file for the provisioned user.
2019-02-14EC2: Rewrite network config on AWS Classic instances every bootGuilherme G. Piccoli
AWS EC2 instances' network come in 2 basic flavors: Classic and VPC (Virtual Private Cloud). The former has an interesting behavior of having its MAC address changed whenever the instance is stopped/restarted. This behavior is not observed in VPC instances. In Ubuntu 18.04 (Bionic) the network "management" changed from ENI-style (etc/network/interfaces) to netplan, and when using netplan we observe the following block present in /etc/netplan/50-cloud-init.yaml: match: macaddress: aa:bb:cc:dd:ee:ff Jani Ollikainen noticed in Launchpad bug #1802073 that the EC2 Classic instances were booting without network access in Bionic after stop/restart procedure, due to their MAC address change behavior. It was narrowed down to the netplan MAC match block, that kept the old MAC address after stopping and restarting an instance, since the network configuration writing happens by default only once in EC2 instances, in the first boot. This patch changes the network configuration write to every boot in EC2 Classic instances, by checking against the "vpc-id" metadata information provided only in the VPC instances - if we don't have this metadata value, cloud-init will rewrite the network configuration file in every boot. This was tested in an EC2 Classic instance and proved to fix the issue; unit tests were also added for the new method is_classic_instance(). LP: #1802073 Reported-by: Jani Ollikainen <jani.ollikainen@ik.fi> Suggested-by: Ryan Harper <ryan.harper@canonical.com> Co-developed-by: Chad Smith <chad.smith@canonical.com> Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
2019-01-30Resolve flake8 comparison and pycodestyle over-ident issuesParide Legovini
Fixes: - flake8: use ==/!= to compare str, bytes, and int literals - pycodestyle: E117 over-indented
2019-01-28opennebula: also exclude epochseconds from changed environment varsChad Smith
In addition to EPOCHREALTIME there is also an EPOCHSECONDS environment variable that OpenNebula needs to exclude as it is expected to change. This commit supplements the other exclusion in commit d1a2fe7307e9cf2251d1f9a666c12d71d3f522d6. Without this fix, unittests will intermittently fail if parse_shell_config is run across a timing boundary where the EPOCHSECONDS changes mid-test. LP: #1813641
2019-01-26opennebula: exclude EPOCHREALTIME as known bash env variable with a deltaChad Smith
This branch is needed to allow cloud-init to sbuild on Ubuntu Disco. OpenNebula:parse_shell_config tries to do a comparison of bash environment values, excluding expected environment variables which are known to change. Bash on Ubuntu Disco surfaces a new EPOCHREALTIME environment variable which wasn't in previous bash environments, this var needs to be ignored by parse_shell_config too. LP: #1813383
2019-01-15[Azure] Increase retries when talking to Wireserver during metadata walkJason Zions
Testing startup of large numbers of VMs (of varying distros) in Azure shows that 3 retries results in a small percentage of failed VMs. Increasing that by a few dramatically decreases the occurrence of provisioning timeout errors. The initial choice of "3 retries" was uninformed by heavy testing. Also, the alternate provisioning mechanism for Azure (waagent) retries the Wireserver crawl without limit. 10 retries seems a more reasonable choice.
2018-12-20Scaleway: Support ssh keys provided inside an instance tag.PORTE Loïc
The change here will utilize ssh keys found inside an instance's tag. The tag value must start with 'AUTHORIZED_KEY'.
2018-12-20OVF: simplify expected return values of transport functions.Scott Moser
Transport functions (transport_iso9660 and transport_vmware_guestinfo) would return a tuple of 3 values, but only the first was ever used outside of test. The other values (device and filename) were just ignored. This just simplifies the transport functions to now return content (in string format) or None indicating that the transport was not found.
2018-12-20Vmware: Add support for the com.vmware.guestInfo OVF transport.Scott Moser
This adds support for reading OVF information over the 'com.vmware.guestInfo' tranport. The current implementation requires vmware-rpctool be installed in the system. LP: #1807466
2018-12-14Update to pylint 2.2.2.Scott Moser
The tip-pylint tox target correctly reported the invalid use of string formatting. The change here is to: a.) Fix the error that was caught. b.) move to pylint 2.2.2 for the default 'pylint' target.
2018-12-03NoCloud: Allow top level 'network' key in network-config.Scott Moser
NoCloud's 'network-config' file was originally expected to contain network configuration without the top level 'network' key. This was because the file was named 'network-config' so specifying 'network' seemed redundant. However, JuJu is currently providing a top level 'network' config when it tries to disable networking ({"network": {"config": "disabled"}). Other users have also been surprised/confused by the fact that a network config in /etc/cloud/cloud.cfg.d/network.cfg differed from what was expected in 'network-config'. LP: #1798117
2018-12-03ovf: Fix ovf network config generation gateway/routesRyan Harper
Move routes under the nic's subnet rather than use top-level ("global") route config ensuring all net renderers will provide the configured route. Also updated cloudinit/cmd/devel/net_convert.py:  - Add input type 'vmware-imc' for OVF customization config files  - Fix bug when output-type was netplan which invoked netplan   generate/apply and attempted to write to   /etc/netplan/50-cloud-init.yaml instead of joining with the   output directory. LP: #1806103
2018-11-29azure: detect vnet migration via netlink media change eventTamilmani Manoharan
Replace Azure pre-provision polling on IMDS with a blocking call which watches for netlink link state change messages. The media change event happens when a pre-provisioned VM has been activated and is connected to the users virtual network and cloud-init can then resume operation to complete image instantiation.
2018-11-29Azure: fix copy/paste error in error handling when reading azure ovf.Adam DePue
Check the appropriate variables based on code review. Correcting what seems to be a copy/paste mistake for the error handling from a few lines above.
2018-11-15azure: _poll_imds only retry on 404. Fail on TimeoutChad Smith
Upon URL timeout, _poll_imds is expected to re-dhcp to get updated IP configuration. We don't want to indefinitely retry because the instance likely has invalid IP configuration. LP: #1803598
2018-11-13azure: retry imds polling on requests.TimeoutChad Smith
There is an infrequent race when the booting instance can hit the IMDS service before it is fully available. This results in a requests.ConnectTimeout being raised. Azure's retry_callback logic now retries on either 404s or Timeouts. LP:1800223
2018-11-12azure: Accept variation in error msg from mount for ntfs volumesJason Zions
If Azure detects an ntfs filesystem type during mount attempt, it should still report the resource device as reformattable. There are slight differences in error message format on RedHat and SuSE. This patch simplifies the expected error match to work on both distributions. LP: #1799338
2018-11-12azure: fix regression introduced when persisting ephemeral dhcp leaseasakkurr
In commitish 9073951 azure datasource tried to leverage stale DHCP information obtained from EphemeralDHCPv4 context manager to report updated provisioning status to the fabric earlier in the boot process. Unfortunately the stale ephemeral network configuration had already been torn down in preparation to bring up IMDS network config so the report attempt failed on timeout. This branch introduces obtain_lease and clean_network public methods on EphemeralDHCPv4 to allow for setup and teardown of ephemeral network configuration without using a context manager. Azure datasource now uses this to persist ephemeral network configuration across multiple contexts during provisioning to avoid multiple DHCP roundtrips.
2018-11-01azure: remove /etc/netplan/90-hotplug-azure.yaml when net from IMDSChad Smith
There was a typo in the seeded filename s/azure-hotplug/hotplug-azure/.
2018-10-31azure: report ready to fabric after reprovision and reduce loggingasakkurr
When reusing a preprovisioned VM, report ready to Azure fabric as soon as we get the reprovision data and the goal state so that we are not delayed by the cloud-init stage switch, saving 2-3 seconds. Also reduce logging when polling IMDS for reprovision data. LP: #1799594
2018-10-17azure: Add apply_network_config option to disable network from IMDSChad Smith
Azure generates network configuration from the IMDS service and removes any preexisting hotplug network scripts which exist in Azure cloud images. Add a datasource configuration option which allows for writing a default network configuration which sets up dhcp on eth0 and leave the hotplug handling to the cloud-image scripts. To disable network-config from Azure IMDS, add the following to /etc/cloud/cloud.cfg.d/99-azure-no-imds-network.cfg: datasource:   Azure:     apply_network_config: False LP: #1798424
2018-10-09tools: Add cloud-id command line utilityChad Smith
Add a quick cloud lookup utility in order to more easily determine the cloud on which an instance is running. The utility parses standardized attributes from /run/cloud-init/instance-data.json to print the canonical cloud-id for the instance. It uses known region maps if necessary to determine on which specific cloud the instance is running. Examples: aws, aws-gov, aws-china, rackspace, azure-china, lxd, openstack, unknown
2018-10-09instance-data: Add standard keys platform and subplatform. Refactor ec2.Chad Smith
Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed by a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. Additional drive-bys: * unit test rework for test_altcloud and test_azure to simplify mocks and make use of existing util and test_helpers functions.
2018-09-26docs: surface experimental doc in instance-data.jsonChad Smith
2018-09-26Add support for Infiniband network interfaces (IPoIB).Mark Goddard
OpenStack ironic references Infiniband interfaces via a 6 byte 'MAC address' formed from bytes 13-15 and 18-20 of interface's hardware address. This address is used as the ethernet_mac_address of Infiniband links in network_data.json in configdrives generated by OpenStack nova. We can use this address to map links in network_data.json to their corresponding interface names. When generating interface configuration files, we need to use the interface's full hardware address as the HWADDR, rather than the 6 byte MAC address provided by network_data.json. This change allows IB interfaces to be referenced in this dual mode - by MAC address and hardware address, depending on the context. Support TYPE=InfiniBand for sysconfig configuration of IB interfaces.
2018-09-25cli: add cloud-init query subcommand to query instance metadataChad Smith
Cloud-init caches any cloud metadata crawled during boot in the file /run/cloud-init/instance-data.json. Cloud-init also standardizes some of that metadata across all clouds. The command 'cloud-init query' surfaces a simple CLI to query or format any cached instance metadata so that scripts or end-users do not have to write tools to crawl metadata themselves. Since 'cloud-init query' is runnable by non-root users, redact any sensitive data from instance-data.json and provide a root-readable unredacted instance-data-sensitive.json. Datasources can now define a sensitive_metadata_keys tuple which will redact any matching keys which could contain passwords or credentials from instance-data.json. Also add the following standardized 'v1' instance-data.json keys:   - user_data: The base64encoded user-data provided at instance launch   - vendor_data: Any vendor_data provided to the instance at launch   - underscore_delimited versions of existing hyphenated keys:     instance_id, local_hostname, availability_zone, cloud_name
2018-09-20Remove dead-code _write_network distro implementations.Scott Moser
Any distro that has a '_write_nework_config' method should no longer get their _write_network called at all. So lets drop that code and raise a RuntimeError any time we got there. Replace the one caller of 'apply_network' (legacy openstack path) with a call to apply_network_config after converting the ENI to network config.
2018-09-14OpenStack: Support setting mac address on bond.Fabian Wiesel
Fix a bug where setting of mac address on a bond device was ignored when provided in OpenStack network_config.json. LP: #1682064
2018-09-13EphemeralIPv4Network: Be more explicit when adding default route.Scott Moser
On OpenStack based OVH public cloud, we got DHCP response with   fixed-address 54.36.113.86;   option subnet-mask 255.255.255.255;   option routers 54.36.112.1; The router clearly is not on the subnet. So 'ip' would fail when we tried to add the default route. The solution here is to add an explicit route on that interface to the router and then add the default route. Also add 'bgpovs' to the list of 'physical' types for OpenStack network configuration. That type is used on OVH public cloud. LP: #1792415
2018-09-13OpenStack: support reading of newer versions of metdata.Scott Moser
Mark as supported for reading some newer versions of openstack metadata: 2016-06-30 : Newton one 2016-10-06 : Newton two 2017-02-22 : Ocata 2018-08-27 : Rocky
2018-09-12OpenStack: fix bug causing 'latest' version to be used from network.Scott Moser
Cloud-init was reading a list of versions from the OpenStack metadata service (http://169.254.169.254/openstack/) and attempt to select the newest known supported version. The problem was that the list of versions was not being decoded, so we were comparing a list of bytes (found versions) to a list of strings (known versions). LP: #1792157
2018-09-11user-data: jinja template to render instance-data.json in cloud-configChad Smith
Allow users to provide '## template: jinja' as the first line or their #cloud-config or custom script user-data parts. When this header exists, the cloud-config or script will be rendered as a jinja template. All instance metadata keys and values present in /run/cloud-init/instance-data.json will be available as jinja variables for the template. This means any cloud-config module or script can reference any standardized instance data in templates and scripts. Additionally, any standardized instance-data.json keys scoped below a '<v#>' key will be promoted as a top-level key for ease of reference in templates. This means that '{{ local_hostname }}' is the same as using the latest '{{ v#.local_hostname }}'. Since instance-data is written to /run/cloud-init/instance-data.json, make sure it is persisted across reboots when the cached datasource opject is reloaded. LP: #1791781
2018-09-05tests: Disallow use of util.subp except for where needed.Scott Moser
In many cases, cloud-init uses 'util.subp' to run a subprocess. This is not really desirable in our unit tests as it makes the tests dependent upon existance of those utilities. The change here is to modify the base test case class (CiTestCase) to raise exception any time subp is called. Then, fix all callers. For cases where subp is necessary or actually desired, we can use it via   a.) context hander CiTestCase.allow_subp(value)   b.) class level self.allowed_subp = value Both cases the value is a list of acceptable executable names that will be called (essentially argv[0]). Some cleanups in AltCloud were done as the code was being updated.
2018-08-28VMWare: Fix a network config bug in vm with static IPv4 and no gateway.Pengpeng Sun
The issue is when customize a VM with static IPv4 and without gateway, it will still extend route list and will loop a gateways list which is None. This fix is to make sure when no gateway is here, it will not extend route list. LP: #1766538
2018-08-17Add datasource Oracle Compute Infrastructure (OCI).Scott Moser
This adds a Oracle specific datasource that functions with OCI. It is a simplified version of the OpenStack metadata server with support for vendor-data. It does not support the OCI-C (classic) platform. Also here is a move of BrokenMetadata to common 'sources' as this was the third occurrence of that class.
2018-08-17azure: allow azure to generate network configuration from IMDS per boot.Chad Smith
Azure datasource now queries IMDS metadata service for network configuration at link local address http://169.254.169.254/metadata/instance?api-version=2017-12-01. The azure metadata service presents a list of macs and allocated ip addresses associated with this instance. Azure will now also regenerate network configuration on every boot because it subscribes to EventType.BOOT maintenance events as well as the 'first boot' EventType.BOOT_NEW_INSTANCE. For testing add azure-imds --kind to cloud-init devel net_convert tool for debugging IMDS metadata. Also refactor _get_data into 3 discrete methods:   - is_platform_viable: check quickly whether the datasource is     potentially compatible with the platform on which is is running   - crawl_metadata: walk all potential metadata candidates, returning a     structured dict of all metadata and userdata. Raise InvalidMetaData on     error.   - _get_data: call crawl_metadata and process results or error. Cache     instance data on class attributes: metadata, userdata_raw etc.
2018-08-17Scaleway: Add network configuration to the DataSourceLouis Bouchard
DEP_NETWORK is removed since the network_config must run at each boot. New EventType.BOOT event is used for that. Network is brought up early to fetch the metadata which is required to configure the network (ipv4 and/or v6). Adds unittests for the following and fixes test_common for LOCAL and NETWORK sets.
2018-08-03OpenNebula: Fix null gateway6Akihiko Ota
The OpenNebula data source generates an invalid netplan yaml file if the IPv6 gateway is not defined in context.sh. LP: #1768547
2018-07-31oracle: fix detect_openstack to report True on OracleCloud.com DMI dataChad Smith
The OpenStack datasource in 18.3 changed to detect data in the init-local stage instead of init-network and attempted to redetect OpenStackLocal datasource on Oracle across reboots. The function detect_openstack was added to quickly detect whether a platform is OpenStack based on dmi product_name or chassis_asset_tag and it was a bit too strict for Oracle in checking for 'OpenStack Nova'/'Compute' DMI product_name. Oracle's DMI product_name reports 'SAtandard PC (i440FX + PIIX, 1996)' and DMI chassis_asset_tag is 'OracleCloud.com'. detect_openstack function now adds 'OracleCloud.com' as a supported value 'OracleCloud.com' to valid chassis-asset-tags for the OpenStack datasource. LP: #1784685
2018-07-26update_metadata re-config on every boot comments and tests not quite rightMike Gerdts
The comment in update_metadata() that explains how a datasource should enable network reconfig on every boot presumes that EventType.BOOT_NEW_INSTANCE is a subset of EventType.BOOT. That's not the case, and as such a datasource that needs to configure networking when it is a new instance and every boot needs to include both event types. To make the situation above easier to debug, update_metadata() now logs when it returns false. To make it so that datasources do not need to test before appending to the update_events['network'], it is changed from a list to a set. test_update_metadata_only_acts_on_supported_update_events is updated to allow datasources to support EventType.BOOT. Author: Mike Gerdts <mike.gerdts@joyent.com>