From f0bc02d7e221c9aa5982b267739481420c761ead Mon Sep 17 00:00:00 2001 From: Chad Smith Date: Tue, 9 Oct 2018 21:46:35 +0000 Subject: instance-data: Add standard keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed by a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. Additional drive-bys: * unit test rework for test_altcloud and test_azure to simplify mocks and make use of existing util and test_helpers functions. --- doc/rtd/topics/instancedata.rst | 183 ++++++++++++++++++++++++++++++---------- 1 file changed, 137 insertions(+), 46 deletions(-) (limited to 'doc') diff --git a/doc/rtd/topics/instancedata.rst b/doc/rtd/topics/instancedata.rst index 634e1807..5d2dc948 100644 --- a/doc/rtd/topics/instancedata.rst +++ b/doc/rtd/topics/instancedata.rst @@ -90,24 +90,46 @@ There are three basic top-level keys: The standardized keys present: -+----------------------+-----------------------------------------------+---------------------------+ -| Key path | Description | Examples | -+======================+===============================================+===========================+ -| v1.cloud_name | The name of the cloud provided by metadata | aws, openstack, azure, | -| | key 'cloud-name' or the cloud-init datasource | configdrive, nocloud, | -| | name which was discovered. | ovf, etc. | -+----------------------+-----------------------------------------------+---------------------------+ -| v1.instance_id | Unique instance_id allocated by the cloud | i- | -+----------------------+-----------------------------------------------+---------------------------+ -| v1.local_hostname | The internal or local hostname of the system | ip-10-41-41-70, | -| | | | -+----------------------+-----------------------------------------------+---------------------------+ -| v1.region | The physical region/datacenter in which the | us-east-2 | -| | instance is deployed | | -+----------------------+-----------------------------------------------+---------------------------+ -| v1.availability_zone | The physical availability zone in which the | us-east-2b, nova, null | -| | instance is deployed | | -+----------------------+-----------------------------------------------+---------------------------+ ++----------------------+-----------------------------------------------+-----------------------------------+ +| Key path | Description | Examples | ++======================+===============================================+===================================+ +| v1._beta_keys | List of standardized keys still in 'beta'. | [subplatform] | +| | The format, intent or presence of these keys | | +| | can change. Do not consider them | | +| | production-ready. | | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.cloud_name | Where possible this will indicate the 'name' | aws, openstack, azure, | +| | of the cloud this system is running on. This | configdrive, nocloud, | +| | is specifically different than the 'platform' | ovf, etc. | +| | below. As an example, the name of Amazon Web | | +| | Services is 'aws' while the platform is 'ec2'.| | +| | | | +| | If no specific name is determinable or | | +| | provided in meta-data, then this field may | | +| | contain the same content as 'platform'. | | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.instance_id | Unique instance_id allocated by the cloud | i- | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.local_hostname | The internal or local hostname of the system | ip-10-41-41-70, | +| | | | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.platform | An attempt to identify the cloud platform | ec2, openstack, lxd, gce | +| | instance that the system is running on. | nocloud, ovf | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.subplatform | Additional platform details describing the | metadata (http://168.254.169.254),| +| | specific source or type of metadata used. | seed-dir (/path/to/seed-dir/), | +| | The format of subplatform will be: | config-disk (/dev/cd0), | +| | () | configdrive (/dev/sr0) | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.public_ssh_keys | A list of ssh keys provided to the instance | ['ssh-rsa AA...', ...] | +| | by the datasource metadata. | | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.region | The physical region/datacenter in which the | us-east-2 | +| | instance is deployed | | ++----------------------+-----------------------------------------------+-----------------------------------+ +| v1.availability_zone | The physical availability zone in which the | us-east-2b, nova, null | +| | instance is deployed | | ++----------------------+-----------------------------------------------+-----------------------------------+ Below is an example of ``/run/cloud-init/instance_data.json`` on an EC2 @@ -117,10 +139,75 @@ instance: { "base64_encoded_keys": [], - "sensitive_keys": [], "ds": { - "meta_data": { - "ami-id": "ami-014e1416b628b0cbf", + "_doc": "EXPERIMENTAL: The structure and format of content scoped under the 'ds' key may change in subsequent releases of cloud-init.", + "_metadata_api_version": "2016-09-02", + "dynamic": { + "instance-identity": { + "document": { + "accountId": "437526006925", + "architecture": "x86_64", + "availabilityZone": "us-east-2b", + "billingProducts": null, + "devpayProductCodes": null, + "imageId": "ami-079638aae7046bdd2", + "instanceId": "i-075f088c72ad3271c", + "instanceType": "t2.micro", + "kernelId": null, + "marketplaceProductCodes": null, + "pendingTime": "2018-10-05T20:10:43Z", + "privateIp": "10.41.41.95", + "ramdiskId": null, + "region": "us-east-2", + "version": "2017-09-30" + }, + "pkcs7": [ + "MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggHbewog", + "ICJkZXZwYXlQcm9kdWN0Q29kZXMiIDogbnVsbCwKICAibWFya2V0cGxhY2VQcm9kdWN0Q29kZXMi", + "IDogbnVsbCwKICAicHJpdmF0ZUlwIiA6ICIxMC40MS40MS45NSIsCiAgInZlcnNpb24iIDogIjIw", + "MTctMDktMzAiLAogICJpbnN0YW5jZUlkIiA6ICJpLTA3NWYwODhjNzJhZDMyNzFjIiwKICAiYmls", + "bGluZ1Byb2R1Y3RzIiA6IG51bGwsCiAgImluc3RhbmNlVHlwZSIgOiAidDIubWljcm8iLAogICJh", + "Y2NvdW50SWQiIDogIjQzNzUyNjAwNjkyNSIsCiAgImF2YWlsYWJpbGl0eVpvbmUiIDogInVzLWVh", + "c3QtMmIiLAogICJrZXJuZWxJZCIgOiBudWxsLAogICJyYW1kaXNrSWQiIDogbnVsbCwKICAiYXJj", + "aGl0ZWN0dXJlIiA6ICJ4ODZfNjQiLAogICJpbWFnZUlkIiA6ICJhbWktMDc5NjM4YWFlNzA0NmJk", + "ZDIiLAogICJwZW5kaW5nVGltZSIgOiAiMjAxOC0xMC0wNVQyMDoxMDo0M1oiLAogICJyZWdpb24i", + "IDogInVzLWVhc3QtMiIKfQAAAAAAADGCARcwggETAgEBMGkwXDELMAkGA1UEBhMCVVMxGTAXBgNV", + "BAgTEFdhc2hpbmd0b24gU3RhdGUxEDAOBgNVBAcTB1NlYXR0bGUxIDAeBgNVBAoTF0FtYXpvbiBX", + "ZWIgU2VydmljZXMgTExDAgkAlrpI2eVeGmcwCQYFKw4DAhoFAKBdMBgGCSqGSIb3DQEJAzELBgkq", + "hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE4MTAwNTIwMTA0OFowIwYJKoZIhvcNAQkEMRYEFK0k", + "Tz6n1A8/zU1AzFj0riNQORw2MAkGByqGSM44BAMELjAsAhRNrr174y98grPBVXUforN/6wZp8AIU", + "JLZBkrB2GJA8A4WJ1okq++jSrBIAAAAAAAA=" + ], + "rsa2048": [ + "MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwGggCSABIIB", + "23sKICAiZGV2cGF5UHJvZHVjdENvZGVzIiA6IG51bGwsCiAgIm1hcmtldHBsYWNlUHJvZHVjdENv", + "ZGVzIiA6IG51bGwsCiAgInByaXZhdGVJcCIgOiAiMTAuNDEuNDEuOTUiLAogICJ2ZXJzaW9uIiA6", + "ICIyMDE3LTA5LTMwIiwKICAiaW5zdGFuY2VJZCIgOiAiaS0wNzVmMDg4YzcyYWQzMjcxYyIsCiAg", + "ImJpbGxpbmdQcm9kdWN0cyIgOiBudWxsLAogICJpbnN0YW5jZVR5cGUiIDogInQyLm1pY3JvIiwK", + "ICAiYWNjb3VudElkIiA6ICI0Mzc1MjYwMDY5MjUiLAogICJhdmFpbGFiaWxpdHlab25lIiA6ICJ1", + "cy1lYXN0LTJiIiwKICAia2VybmVsSWQiIDogbnVsbCwKICAicmFtZGlza0lkIiA6IG51bGwsCiAg", + "ImFyY2hpdGVjdHVyZSIgOiAieDg2XzY0IiwKICAiaW1hZ2VJZCIgOiAiYW1pLTA3OTYzOGFhZTcw", + "NDZiZGQyIiwKICAicGVuZGluZ1RpbWUiIDogIjIwMTgtMTAtMDVUMjA6MTA6NDNaIiwKICAicmVn", + "aW9uIiA6ICJ1cy1lYXN0LTIiCn0AAAAAAAAxggH/MIIB+wIBATBpMFwxCzAJBgNVBAYTAlVTMRkw", + "FwYDVQQIExBXYXNoaW5ndG9uIFN0YXRlMRAwDgYDVQQHEwdTZWF0dGxlMSAwHgYDVQQKExdBbWF6", + "b24gV2ViIFNlcnZpY2VzIExMQwIJAM07oeX4xevdMA0GCWCGSAFlAwQCAQUAoGkwGAYJKoZIhvcN", + "AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgxMDA1MjAxMDQ4WjAvBgkqhkiG9w0B", + "CQQxIgQgkYz0pZk3zJKBi4KP4egeOKJl/UYwu5UdE7id74pmPwMwDQYJKoZIhvcNAQEBBQAEggEA", + "dC3uIGGNul1OC1mJKSH3XoBWsYH20J/xhIdftYBoXHGf2BSFsrs9ZscXd2rKAKea4pSPOZEYMXgz", + "lPuT7W0WU89N3ZKviy/ReMSRjmI/jJmsY1lea6mlgcsJXreBXFMYucZvyeWGHdnCjamoKWXkmZlM", + "mSB1gshWy8Y7DzoKviYPQZi5aI54XK2Upt4kGme1tH1NI2Cq+hM4K+adxTbNhS3uzvWaWzMklUuU", + "QHX2GMmjAVRVc8vnA8IAsBCJJp+gFgYzi09IK+cwNgCFFPADoG6jbMHHf4sLB3MUGpiA+G9JlCnM", + "fmkjI2pNRB8spc0k4UG4egqLrqCz67WuK38tjwAAAAAAAA==" + ], + "signature": [ + "Tsw6h+V3WnxrNVSXBYIOs1V4j95YR1mLPPH45XnhX0/Ei3waJqf7/7EEKGYP1Cr4PTYEULtZ7Mvf", + "+xJpM50Ivs2bdF7o0c4vnplRWe3f06NI9pv50dr110j/wNzP4MZ1pLhJCqubQOaaBTF3LFutgRrt", + "r4B0mN3p7EcqD8G+ll0=" + ] + } + }, + "meta-data": { + "ami-id": "ami-079638aae7046bdd2", "ami-launch-index": "0", "ami-manifest-path": "(unknown)", "block-device-mapping": { @@ -129,31 +216,31 @@ instance: "ephemeral1": "sdc", "root": "/dev/sda1" }, - "hostname": "ip-10-41-41-70.us-east-2.compute.internal", + "hostname": "ip-10-41-41-95.us-east-2.compute.internal", "instance-action": "none", - "instance-id": "i-04fa31cfc55aa7976", + "instance-id": "i-075f088c72ad3271c", "instance-type": "t2.micro", - "local-hostname": "ip-10-41-41-70.us-east-2.compute.internal", - "local-ipv4": "10.41.41.70", - "mac": "06:b6:92:dd:9d:24", + "local-hostname": "ip-10-41-41-95.us-east-2.compute.internal", + "local-ipv4": "10.41.41.95", + "mac": "06:74:8f:39:cd:a6", "metrics": { "vhostmd": "" }, "network": { "interfaces": { "macs": { - "06:b6:92:dd:9d:24": { + "06:74:8f:39:cd:a6": { "device-number": "0", - "interface-id": "eni-08c0c9fdb99b6e6f4", + "interface-id": "eni-052058bbd7831eaae", "ipv4-associations": { - "18.224.22.43": "10.41.41.70" + "18.218.221.122": "10.41.41.95" }, - "local-hostname": "ip-10-41-41-70.us-east-2.compute.internal", - "local-ipv4s": "10.41.41.70", - "mac": "06:b6:92:dd:9d:24", + "local-hostname": "ip-10-41-41-95.us-east-2.compute.internal", + "local-ipv4s": "10.41.41.95", + "mac": "06:74:8f:39:cd:a6", "owner-id": "437526006925", - "public-hostname": "ec2-18-224-22-43.us-east-2.compute.amazonaws.com", - "public-ipv4s": "18.224.22.43", + "public-hostname": "ec2-18-218-221-122.us-east-2.compute.amazonaws.com", + "public-ipv4s": "18.218.221.122", "security-group-ids": "sg-828247e9", "security-groups": "Cloud-init integration test secgroup", "subnet-id": "subnet-282f3053", @@ -171,16 +258,14 @@ instance: "availability-zone": "us-east-2b" }, "profile": "default-hvm", - "public-hostname": "ec2-18-224-22-43.us-east-2.compute.amazonaws.com", - "public-ipv4": "18.224.22.43", + "public-hostname": "ec2-18-218-221-122.us-east-2.compute.amazonaws.com", + "public-ipv4": "18.218.221.122", "public-keys": { "cloud-init-integration": [ - "ssh-rsa - AAAAB3NzaC1yc2EAAAADAQABAAABAQDSL7uWGj8cgWyIOaspgKdVy0cKJ+UTjfv7jBOjG2H/GN8bJVXy72XAvnhM0dUM+CCs8FOf0YlPX+Frvz2hKInrmRhZVwRSL129PasD12MlI3l44u6IwS1o/W86Q+tkQYEljtqDOo0a+cOsaZkvUNzUyEXUwz/lmYa6G4hMKZH4NBj7nbAAF96wsMCoyNwbWryBnDYUr6wMbjRR1J9Pw7Xh7WRC73wy4Va2YuOgbD3V/5ZrFPLbWZW/7TFXVrql04QVbyei4aiFR5n//GvoqwQDNe58LmbzX/xvxyKJYdny2zXmdAhMxbrpFQsfpkJ9E/H5w0yOdSvnWbUoG5xNGoOB - cloud-init-integration" + "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSL7uWGj8cgWyIOaspgKdVy0cKJ+UTjfv7jBOjG2H/GN8bJVXy72XAvnhM0dUM+CCs8FOf0YlPX+Frvz2hKInrmRhZVwRSL129PasD12MlI3l44u6IwS1o/W86Q+tkQYEljtqDOo0a+cOsaZkvUNzUyEXUwz/lmYa6G4hMKZH4NBj7nbAAF96wsMCoyNwbWryBnDYUr6wMbjRR1J9Pw7Xh7WRC73wy4Va2YuOgbD3V/5ZrFPLbWZW/7TFXVrql04QVbyei4aiFR5n//GvoqwQDNe58LmbzX/xvxyKJYdny2zXmdAhMxbrpFQsfpkJ9E/H5w0yOdSvnWbUoG5xNGoOB cloud-init-integration" ] }, - "reservation-id": "r-06ab75e9346f54333", + "reservation-id": "r-0594a20e31f6cfe46", "security-groups": "Cloud-init integration test secgroup", "services": { "domain": "amazonaws.com", @@ -188,16 +273,22 @@ instance: } } }, + "sensitive_keys": [], "v1": { + "_beta_keys": [ + "subplatform" + ], "availability-zone": "us-east-2b", "availability_zone": "us-east-2b", - "cloud-name": "aws", "cloud_name": "aws", - "instance-id": "i-04fa31cfc55aa7976", - "instance_id": "i-04fa31cfc55aa7976", - "local-hostname": "ip-10-41-41-70", - "local_hostname": "ip-10-41-41-70", - "region": "us-east-2" + "instance_id": "i-075f088c72ad3271c", + "local_hostname": "ip-10-41-41-95", + "platform": "ec2", + "public_ssh_keys": [ + "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSL7uWGj8cgWyIOaspgKdVy0cKJ+UTjfv7jBOjG2H/GN8bJVXy72XAvnhM0dUM+CCs8FOf0YlPX+Frvz2hKInrmRhZVwRSL129PasD12MlI3l44u6IwS1o/W86Q+tkQYEljtqDOo0a+cOsaZkvUNzUyEXUwz/lmYa6G4hMKZH4NBj7nbAAF96wsMCoyNwbWryBnDYUr6wMbjRR1J9Pw7Xh7WRC73wy4Va2YuOgbD3V/5ZrFPLbWZW/7TFXVrql04QVbyei4aiFR5n//GvoqwQDNe58LmbzX/xvxyKJYdny2zXmdAhMxbrpFQsfpkJ9E/H5w0yOdSvnWbUoG5xNGoOB cloud-init-integration" + ], + "region": "us-east-2", + "subplatform": "metadata (http://169.254.169.254)" } } -- cgit v1.2.3 From 1d5e9aefdab06a2574d78e644deed6c6fa1da171 Mon Sep 17 00:00:00 2001 From: Chad Smith Date: Wed, 17 Oct 2018 18:47:35 +0000 Subject: azure: Add apply_network_config option to disable network from IMDS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Azure generates network configuration from the IMDS service and removes any preexisting hotplug network scripts which exist in Azure cloud images. Add a datasource configuration option which allows for writing a default network configuration which sets up dhcp on eth0 and leave the hotplug handling to the cloud-image scripts. To disable network-config from Azure IMDS, add the following to /etc/cloud/cloud.cfg.d/99-azure-no-imds-network.cfg: datasource:   Azure:     apply_network_config: False LP: #1798424 --- cloudinit/sources/DataSourceAzure.py | 11 +++++- doc/rtd/topics/datasources/azure.rst | 46 ++++++++++++++++++++++ tests/unittests/test_datasource/test_azure.py | 56 +++++++++++++++++++++++++-- 3 files changed, 107 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py index 39391d01..d0358e96 100644 --- a/cloudinit/sources/DataSourceAzure.py +++ b/cloudinit/sources/DataSourceAzure.py @@ -207,7 +207,9 @@ BUILTIN_DS_CONFIG = { }, 'disk_aliases': {'ephemeral0': RESOURCE_DISK_PATH}, 'dhclient_lease_file': LEASE_FILE, + 'apply_network_config': True, # Use IMDS published network configuration } +# RELEASE_BLOCKER: Xenial and earlier apply_network_config default is False BUILTIN_CLOUD_CONFIG = { 'disk_setup': { @@ -458,7 +460,8 @@ class DataSourceAzure(sources.DataSource): except sources.InvalidMetaDataException as e: LOG.warning('Could not crawl Azure metadata: %s', e) return False - if self.distro and self.distro.name == 'ubuntu': + if (self.distro and self.distro.name == 'ubuntu' and + self.ds_cfg.get('apply_network_config')): maybe_remove_ubuntu_network_config_scripts() # Process crawled data and augment with various config defaults @@ -619,7 +622,11 @@ class DataSourceAzure(sources.DataSource): the blacklisted devices. """ if not self._network_config: - self._network_config = parse_network_config(self._metadata_imds) + if self.ds_cfg.get('apply_network_config'): + nc_src = self._metadata_imds + else: + nc_src = None + self._network_config = parse_network_config(nc_src) return self._network_config diff --git a/doc/rtd/topics/datasources/azure.rst b/doc/rtd/topics/datasources/azure.rst index 559011ef..f73c3694 100644 --- a/doc/rtd/topics/datasources/azure.rst +++ b/doc/rtd/topics/datasources/azure.rst @@ -57,6 +57,52 @@ in order to use waagent.conf with cloud-init, the following settings are recomme ResourceDisk.MountPoint=/mnt +Configuration +------------- +The following configuration can be set for the datasource in system +configuration (in `/etc/cloud/cloud.cfg` or `/etc/cloud/cloud.cfg.d/`). + +The settings that may be configured are: + + * **agent_command**: Either __builtin__ (default) or a command to run to getcw + metadata. If __builtin__, get metadata from walinuxagent. Otherwise run the + provided command to obtain metadata. + * **apply_network_config**: Boolean set to True to use network configuration + described by Azure's IMDS endpoint instead of fallback network config of + dhcp on eth0. Default is True. For Ubuntu 16.04 or earlier, default is False. + * **data_dir**: Path used to read metadata files and write crawled data. + * **dhclient_lease_file**: The fallback lease file to source when looking for + custom DHCP option 245 from Azure fabric. + * **disk_aliases**: A dictionary defining which device paths should be + interpreted as ephemeral images. See cc_disk_setup module for more info. + * **hostname_bounce**: A dictionary Azure hostname bounce behavior to react to + metadata changes. + * **hostname_bounce**: A dictionary Azure hostname bounce behavior to react to + metadata changes. Azure will throttle ifup/down in some cases after metadata + has been updated to inform dhcp server about updated hostnames. + * **set_hostname**: Boolean set to True when we want Azure to set the hostname + based on metadata. + +An example configuration with the default values is provided below: + +.. sourcecode:: yaml + + datasource: + Azure: + agent_command: __builtin__ + apply_network_config: true + data_dir: /var/lib/waagent + dhclient_lease_file: /var/lib/dhcp/dhclient.eth0.leases + disk_aliases: + ephemeral0: /dev/disk/cloud/azure_resource + hostname_bounce: + interface: eth0 + command: builtin + policy: true + hostname_command: hostname + set_hostname: true + + Userdata -------- Userdata is provided to cloud-init inside the ovf-env.xml file. Cloud-init diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py index 0f4b7bf7..cd6e7e74 100644 --- a/tests/unittests/test_datasource/test_azure.py +++ b/tests/unittests/test_datasource/test_azure.py @@ -256,7 +256,8 @@ scbus-1 on xpt0 bus 0 ]) return dsaz - def _get_ds(self, data, agent_command=None, distro=None): + def _get_ds(self, data, agent_command=None, distro=None, + apply_network=None): def dsdevs(): return data.get('dsdevs', []) @@ -312,6 +313,8 @@ scbus-1 on xpt0 bus 0 data.get('sys_cfg', {}), distro=distro, paths=self.paths) if agent_command is not None: dsrc.ds_cfg['agent_command'] = agent_command + if apply_network is not None: + dsrc.ds_cfg['apply_network_config'] = apply_network return dsrc @@ -434,14 +437,26 @@ fdescfs /dev/fd fdescfs rw 0 0 def test_get_data_on_ubuntu_will_remove_network_scripts(self): """get_data will remove ubuntu net scripts on Ubuntu distro.""" + sys_cfg = {'datasource': {'Azure': {'apply_network_config': True}}} odata = {'HostName': "myhost", 'UserName': "myuser"} data = {'ovfcontent': construct_valid_ovf_env(data=odata), - 'sys_cfg': {}} + 'sys_cfg': sys_cfg} dsrc = self._get_ds(data, distro='ubuntu') dsrc.get_data() self.m_remove_ubuntu_network_scripts.assert_called_once_with() + def test_get_data_on_ubuntu_will_not_remove_network_scripts_disabled(self): + """When apply_network_config false, do not remove scripts on Ubuntu.""" + sys_cfg = {'datasource': {'Azure': {'apply_network_config': False}}} + odata = {'HostName': "myhost", 'UserName': "myuser"} + data = {'ovfcontent': construct_valid_ovf_env(data=odata), + 'sys_cfg': sys_cfg} + + dsrc = self._get_ds(data, distro='ubuntu') + dsrc.get_data() + self.m_remove_ubuntu_network_scripts.assert_not_called() + def test_crawl_metadata_returns_structured_data_and_caches_nothing(self): """Return all structured metadata and cache no class attributes.""" yaml_cfg = "{agent_command: my_command}\n" @@ -523,8 +538,10 @@ fdescfs /dev/fd fdescfs rw 0 0 def test_network_config_set_from_imds(self): """Datasource.network_config returns IMDS network data.""" + sys_cfg = {'datasource': {'Azure': {'apply_network_config': True}}} odata = {} - data = {'ovfcontent': construct_valid_ovf_env(data=odata)} + data = {'ovfcontent': construct_valid_ovf_env(data=odata), + 'sys_cfg': sys_cfg} expected_network_config = { 'ethernets': { 'eth0': {'set-name': 'eth0', @@ -803,9 +820,10 @@ fdescfs /dev/fd fdescfs rw 0 0 @mock.patch('cloudinit.net.generate_fallback_config') def test_imds_network_config(self, mock_fallback): """Network config is generated from IMDS network data when present.""" + sys_cfg = {'datasource': {'Azure': {'apply_network_config': True}}} odata = {'HostName': "myhost", 'UserName': "myuser"} data = {'ovfcontent': construct_valid_ovf_env(data=odata), - 'sys_cfg': {}} + 'sys_cfg': sys_cfg} dsrc = self._get_ds(data) ret = dsrc.get_data() @@ -821,6 +839,36 @@ fdescfs /dev/fd fdescfs rw 0 0 self.assertEqual(expected_cfg, dsrc.network_config) mock_fallback.assert_not_called() + @mock.patch('cloudinit.net.get_interface_mac') + @mock.patch('cloudinit.net.get_devicelist') + @mock.patch('cloudinit.net.device_driver') + @mock.patch('cloudinit.net.generate_fallback_config') + def test_imds_network_ignored_when_apply_network_config_false( + self, mock_fallback, mock_dd, mock_devlist, mock_get_mac): + """When apply_network_config is False, use fallback instead of IMDS.""" + sys_cfg = {'datasource': {'Azure': {'apply_network_config': False}}} + odata = {'HostName': "myhost", 'UserName': "myuser"} + data = {'ovfcontent': construct_valid_ovf_env(data=odata), + 'sys_cfg': sys_cfg} + fallback_config = { + 'version': 1, + 'config': [{ + 'type': 'physical', 'name': 'eth0', + 'mac_address': '00:11:22:33:44:55', + 'params': {'driver': 'hv_netsvc'}, + 'subnets': [{'type': 'dhcp'}], + }] + } + mock_fallback.return_value = fallback_config + + mock_devlist.return_value = ['eth0'] + mock_dd.return_value = ['hv_netsvc'] + mock_get_mac.return_value = '00:11:22:33:44:55' + + dsrc = self._get_ds(data) + self.assertTrue(dsrc.get_data()) + self.assertEqual(dsrc.network_config, fallback_config) + @mock.patch('cloudinit.net.get_interface_mac') @mock.patch('cloudinit.net.get_devicelist') @mock.patch('cloudinit.net.device_driver') -- cgit v1.2.3 From 532ff0f0a4f12225d5785bc98d7a4528e0c779d6 Mon Sep 17 00:00:00 2001 From: Tomer Cohen Date: Thu, 25 Oct 2018 20:48:05 +0000 Subject: docs: remove colon from network v1 config example. The docs for network v1 config contained a errant ':'. Simply drop it. --- doc/rtd/topics/network-config-format-v1.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/rtd/topics/network-config-format-v1.rst b/doc/rtd/topics/network-config-format-v1.rst index 3b0148ca..83520000 100644 --- a/doc/rtd/topics/network-config-format-v1.rst +++ b/doc/rtd/topics/network-config-format-v1.rst @@ -332,7 +332,7 @@ the following keys: - type: static address: 192.168.23.14/27 gateway: 192.168.23.1 - - type: nameserver: + - type: nameserver address: - 192.168.23.2 - 8.8.8.8 -- cgit v1.2.3 From d74d3f0ff5c8d453f626b113f4e6065322f822fa Mon Sep 17 00:00:00 2001 From: Chad Smith Date: Tue, 30 Oct 2018 20:02:38 +0000 Subject: query: better error when missing read permission on instance-data Emit a permissions error instead of "Missing instance-data.json" when non-root user doesn't have read-permission on /run/cloud-init/instance-data.json --- cloudinit/cmd/devel/render.py | 12 ++++++++---- cloudinit/cmd/query.py | 8 ++++++-- cloudinit/cmd/tests/test_query.py | 27 +++++++++++++++++++++++---- cloudinit/handlers/jinja_template.py | 10 +++++++++- doc/rtd/topics/network-config-format-v1.rst | 2 +- tests/unittests/test_builtin_handlers.py | 25 +++++++++++++++++++++++++ 6 files changed, 72 insertions(+), 12 deletions(-) (limited to 'doc') diff --git a/cloudinit/cmd/devel/render.py b/cloudinit/cmd/devel/render.py index 4d3ec958..1bc22406 100755 --- a/cloudinit/cmd/devel/render.py +++ b/cloudinit/cmd/devel/render.py @@ -71,10 +71,14 @@ def handle_args(name, args): except IOError: LOG.error('Missing user-data file: %s', args.user_data) return 1 - rendered_payload = render_jinja_payload_from_file( - payload=user_data, payload_fn=args.user_data, - instance_data_file=instance_data_fn, - debug=True if args.debug else False) + try: + rendered_payload = render_jinja_payload_from_file( + payload=user_data, payload_fn=args.user_data, + instance_data_file=instance_data_fn, + debug=True if args.debug else False) + except RuntimeError as e: + LOG.error('Cannot render from instance data: %s', str(e)) + return 1 if not rendered_payload: LOG.error('Unable to render user-data file: %s', args.user_data) return 1 diff --git a/cloudinit/cmd/query.py b/cloudinit/cmd/query.py index ff03de94..1d888b9d 100644 --- a/cloudinit/cmd/query.py +++ b/cloudinit/cmd/query.py @@ -3,6 +3,7 @@ """Query standardized instance metadata from the command line.""" import argparse +from errno import EACCES import os import six import sys @@ -106,8 +107,11 @@ def handle_args(name, args): try: instance_json = util.load_file(instance_data_fn) - except IOError: - LOG.error('Missing instance-data.json file: %s', instance_data_fn) + except (IOError, OSError) as e: + if e.errno == EACCES: + LOG.error("No read permission on '%s'. Try sudo", instance_data_fn) + else: + LOG.error('Missing instance-data file: %s', instance_data_fn) return 1 instance_data = util.load_json(instance_json) diff --git a/cloudinit/cmd/tests/test_query.py b/cloudinit/cmd/tests/test_query.py index 241f5413..28738b1e 100644 --- a/cloudinit/cmd/tests/test_query.py +++ b/cloudinit/cmd/tests/test_query.py @@ -1,5 +1,6 @@ # This file is part of cloud-init. See LICENSE file for license information. +import errno from six import StringIO from textwrap import dedent import os @@ -51,10 +52,28 @@ class TestQuery(CiTestCase): with mock.patch('sys.stderr', new_callable=StringIO) as m_stderr: self.assertEqual(1, query.handle_args('anyname', args)) self.assertIn( - 'ERROR: Missing instance-data.json file: %s' % absent_fn, + 'ERROR: Missing instance-data file: %s' % absent_fn, self.logs.getvalue()) self.assertIn( - 'ERROR: Missing instance-data.json file: %s' % absent_fn, + 'ERROR: Missing instance-data file: %s' % absent_fn, + m_stderr.getvalue()) + + def test_handle_args_error_when_no_read_permission_instance_data(self): + """When instance_data file is unreadable, log an error.""" + noread_fn = self.tmp_path('unreadable', dir=self.tmp) + write_file(noread_fn, 'thou shall not pass') + args = self.args( + debug=False, dump_all=True, format=None, instance_data=noread_fn, + list_keys=False, user_data='ud', vendor_data='vd', varname=None) + with mock.patch('sys.stderr', new_callable=StringIO) as m_stderr: + with mock.patch('cloudinit.cmd.query.util.load_file') as m_load: + m_load.side_effect = OSError(errno.EACCES, 'Not allowed') + self.assertEqual(1, query.handle_args('anyname', args)) + self.assertIn( + "ERROR: No read permission on '%s'. Try sudo" % noread_fn, + self.logs.getvalue()) + self.assertIn( + "ERROR: No read permission on '%s'. Try sudo" % noread_fn, m_stderr.getvalue()) def test_handle_args_defaults_instance_data(self): @@ -71,10 +90,10 @@ class TestQuery(CiTestCase): self.assertEqual(1, query.handle_args('anyname', args)) json_file = os.path.join(run_dir, INSTANCE_JSON_FILE) self.assertIn( - 'ERROR: Missing instance-data.json file: %s' % json_file, + 'ERROR: Missing instance-data file: %s' % json_file, self.logs.getvalue()) self.assertIn( - 'ERROR: Missing instance-data.json file: %s' % json_file, + 'ERROR: Missing instance-data file: %s' % json_file, m_stderr.getvalue()) def test_handle_args_root_fallsback_to_instance_data(self): diff --git a/cloudinit/handlers/jinja_template.py b/cloudinit/handlers/jinja_template.py index 3fa4097e..ce3accf6 100644 --- a/cloudinit/handlers/jinja_template.py +++ b/cloudinit/handlers/jinja_template.py @@ -1,5 +1,6 @@ # This file is part of cloud-init. See LICENSE file for license information. +from errno import EACCES import os import re @@ -76,7 +77,14 @@ def render_jinja_payload_from_file( raise RuntimeError( 'Cannot render jinja template vars. Instance data not yet' ' present at %s' % instance_data_file) - instance_data = load_json(load_file(instance_data_file)) + try: + instance_data = load_json(load_file(instance_data_file)) + except (IOError, OSError) as e: + if e.errno == EACCES: + raise RuntimeError( + 'Cannot render jinja template vars. No read permission on' + " '%s'. Try sudo" % instance_data_file) + rendered_payload = render_jinja_payload( payload, payload_fn, instance_data, debug) if not rendered_payload: diff --git a/doc/rtd/topics/network-config-format-v1.rst b/doc/rtd/topics/network-config-format-v1.rst index 83520000..3b0148ca 100644 --- a/doc/rtd/topics/network-config-format-v1.rst +++ b/doc/rtd/topics/network-config-format-v1.rst @@ -332,7 +332,7 @@ the following keys: - type: static address: 192.168.23.14/27 gateway: 192.168.23.1 - - type: nameserver + - type: nameserver: address: - 192.168.23.2 - 8.8.8.8 diff --git a/tests/unittests/test_builtin_handlers.py b/tests/unittests/test_builtin_handlers.py index abe820e1..b92ffc79 100644 --- a/tests/unittests/test_builtin_handlers.py +++ b/tests/unittests/test_builtin_handlers.py @@ -3,6 +3,7 @@ """Tests of the built-in user data handlers.""" import copy +import errno import os import shutil import tempfile @@ -202,6 +203,30 @@ class TestJinjaTemplatePartHandler(CiTestCase): os.path.exists(script_file), 'Unexpected file created %s' % script_file) + def test_jinja_template_handle_errors_on_unreadable_instance_data(self): + """If instance-data is unreadable, raise an error from handle_part.""" + script_handler = ShellScriptPartHandler(self.paths) + instance_json = os.path.join(self.run_dir, 'instance-data.json') + util.write_file(instance_json, util.json_dumps({})) + h = JinjaTemplatePartHandler( + self.paths, sub_handlers=[script_handler]) + with mock.patch(self.mpath + 'load_file') as m_load: + with self.assertRaises(RuntimeError) as context_manager: + m_load.side_effect = OSError(errno.EACCES, 'Not allowed') + h.handle_part( + data='data', ctype="!" + handlers.CONTENT_START, + filename='part01', + payload='## template: jinja \n#!/bin/bash\necho himom', + frequency='freq', headers='headers') + script_file = os.path.join(script_handler.script_dir, 'part01') + self.assertEqual( + 'Cannot render jinja template vars. No read permission on' + " '{rdir}/instance-data.json'. Try sudo".format(rdir=self.run_dir), + str(context_manager.exception)) + self.assertFalse( + os.path.exists(script_file), + 'Unexpected file created %s' % script_file) + @skipUnlessJinja() def test_jinja_template_handle_renders_jinja_content(self): """When present, render jinja variables from instance-data.json.""" -- cgit v1.2.3 From 2e93cc62c544fe07afaf36c473e84e6730928998 Mon Sep 17 00:00:00 2001 From: Tomer Cohen Date: Wed, 28 Nov 2018 14:56:40 +0000 Subject: doc: Change dns_nameserver property to dns_nameservers. According to the examples in the page, v1 network config DNS should be defined using the dns_nameservers. The singular dns_nameserver is undefined. --- doc/rtd/topics/network-config-format-v1.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/rtd/topics/network-config-format-v1.rst b/doc/rtd/topics/network-config-format-v1.rst index 3b0148ca..9723d689 100644 --- a/doc/rtd/topics/network-config-format-v1.rst +++ b/doc/rtd/topics/network-config-format-v1.rst @@ -384,7 +384,7 @@ Valid keys for ``subnets`` include the following: - ``address``: IPv4 or IPv6 address. It may include CIDR netmask notation. - ``netmask``: IPv4 subnet mask in dotted format or CIDR notation. - ``gateway``: IPv4 address of the default gateway for this subnet. -- ``dns_nameserver``: Specify a list of IPv4 dns server IPs to end up in +- ``dns_nameservers``: Specify a list of IPv4 dns server IPs to end up in resolv.conf. - ``dns_search``: Specify a list of search paths to be included in resolv.conf. -- cgit v1.2.3