vyos-cloud-init.git/cloudinit/sources/tests, branch crux

tools: Add cloud-id command line utility

2018-10-09T22:19:20+00:00

Add a quick cloud lookup utility in order to more easily determine the cloud on which an instance is running. The utility parses standardized attributes from /run/cloud-init/instance-data.json to print the canonical cloud-id for the instance. It uses known region maps if necessary to determine on which specific cloud the instance is running. Examples: aws, aws-gov, aws-china, rackspace, azure-china, lxd, openstack, unknown

instance-data: Add standard keys platform and subplatform. Refactor ec2.

2018-10-09T21:46:35+00:00

Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed by a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. Additional drive-bys: * unit test rework for test_altcloud and test_azure to simplify mocks and make use of existing util and test_helpers functions.

docs: surface experimental doc in instance-data.json

2018-09-26T21:18:24+00:00

cli: add cloud-init query subcommand to query instance metadata

2018-09-25T21:59:16+00:00

Cloud-init caches any cloud metadata crawled during boot in the file /run/cloud-init/instance-data.json. Cloud-init also standardizes some of that metadata across all clouds. The command 'cloud-init query' surfaces a simple CLI to query or format any cached instance metadata so that scripts or end-users do not have to write tools to crawl metadata themselves. Since 'cloud-init query' is runnable by non-root users, redact any sensitive data from instance-data.json and provide a root-readable unredacted instance-data-sensitive.json. Datasources can now define a sensitive_metadata_keys tuple which will redact any matching keys which could contain passwords or credentials from instance-data.json. Also add the following standardized 'v1' instance-data.json keys: - user_data: The base64encoded user-data provided at instance launch - vendor_data: Any vendor_data provided to the instance at launch - underscore_delimited versions of existing hyphenated keys: instance_id, local_hostname, availability_zone, cloud_name

user-data: jinja template to render instance-data.json in cloud-config

2018-09-11T17:31:46+00:00

Allow users to provide '## template: jinja' as the first line or their #cloud-config or custom script user-data parts. When this header exists, the cloud-config or script will be rendered as a jinja template. All instance metadata keys and values present in /run/cloud-init/instance-data.json will be available as jinja variables for the template. This means any cloud-config module or script can reference any standardized instance data in templates and scripts. Additionally, any standardized instance-data.json keys scoped below a '' key will be promoted as a top-level key for ease of reference in templates. This means that '{{ local_hostname }}' is the same as using the latest '{{ v#.local_hostname }}'. Since instance-data is written to /run/cloud-init/instance-data.json, make sure it is persisted across reboots when the cached datasource opject is reloaded. LP: #1791781

Add datasource Oracle Compute Infrastructure (OCI).

2018-08-17T20:25:31+00:00

This adds a Oracle specific datasource that functions with OCI. It is a simplified version of the OpenStack metadata server with support for vendor-data. It does not support the OCI-C (classic) platform. Also here is a move of BrokenMetadata to common 'sources' as this was the third occurrence of that class.

update_metadata re-config on every boot comments and tests not quite right

2018-07-26T15:31:52+00:00

The comment in update_metadata() that explains how a datasource should enable network reconfig on every boot presumes that EventType.BOOT_NEW_INSTANCE is a subset of EventType.BOOT. That's not the case, and as such a datasource that needs to configure networking when it is a new instance and every boot needs to include both event types. To make the situation above easier to debug, update_metadata() now logs when it returns false. To make it so that datasources do not need to test before appending to the update_events['network'], it is changed from a list to a set. test_update_metadata_only_acts_on_supported_update_events is updated to allow datasources to support EventType.BOOT. Author: Mike Gerdts

update_metadata: a datasource can support network re-config every boot

2018-07-01T22:46:23+00:00

Very basic type definitions are now defined to distinguish 'boot' events from 'new instance (first boot)'. Event types will now be handed to a datasource.update_metadata method which can determine whether to refresh its metadata and re-render configuration based on that source event. A datasource can 'subscribe' to an event by setting up the update_events attribute on the datasource class which describe what config scope is updated by a list of matching events. By default datasources will have the following update_events: {'network': [EventType.BOOT_NEW_INSTANCE]} This setting says the datasource will re-write network configuration only on first boot of a new instance or when the instance id changes. New methods are now present on the datasource: - clear_cached_attrs: Resets cached datasource attributes to values listed in datasource.cached_attr_defaults. This is performed prior to processing a fresh metadata process to avoid keeping old/invalid cached data around. - update_metadata: accepts source_event_types to determine if the metadata should be crawled again and processed

openstack: Allow discovery in init-local using dhclient in a sandbox.

2018-05-23T22:08:43+00:00

Network has not yet been configured in the init-local stage so the openstack datasource will use dhcp-client to temporarily obtain an ipv4 address and query the metadata service at http://169.254.169.254 to get network_data.json configuration. If present, the datasource will return network_config version 1 config based on that network_data.json content. Previously OpenStack datasource only setup dhcp on the fallback interface so this represents a change in behavior to react to the full config provided by openstack. Also significant to OpenStack is the separation of a _crawl_data operation from get_data(). crawl_data walks the available metadata services and returns a dict of discovered content. get_data consumes the crawled_data, caches it in the datasource and reacts to that data. /run/cloud-init/instance-data.json now published network_data.json or ec2_metadata key if that data is present on any datasource. The main reasons for the separation of crawl from get_data: * Enable performance metrics of cloud-init's metadata crawls on each * Enable cloud-init modules and scripts to query and consume metadata content which may have updated/changed after cloud-init's initial cache during instance boot. (Think hotplug) Also generalize common logic to base DataSource class/module: * Move to a common UNSET variable up into base datasource module fix EC2, ConfigDrive, OpenStack, SmartOS to use the global. * Drop get_url_settings from Ec2, CloudStack and OpenStack and generalize DataSource.get_url_params(). Allow subclasses to override url_max_wait, url_timeout and url_retries params. * Rename get_network_metadata bool to perform_dhcp_setup as it designates whether EphemeralDHCPv4 setup is required before crawling metadata. LP: #1749717

pylint: pay attention to unused variable warnings.

2018-04-20T03:30:08+00:00

This enables warnings produced by pylint for unused variables (W0612), and fixes the existing errors.