From a5727fe1477c9cc4288d1ac41f70bd1ab7d7928a Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 8 Jan 2014 17:16:24 -0700 Subject: Significant re-working of the userdata handling and introduction of vendordata. Vendordata is a datasource provided userdata-like blob that is parsed similiarly to userdata, execept at the user's pleasure. cloudinit/config/cc_scripts_vendor.py: added vendor script cloud config cloudinit/config/cc_vendor_scripts_per_boot.py: added vendor per boot cloud config cloudinit/config/cc_vendor_scripts_per_instance.py: added vendor per instance vendor cloud config cloudinit/config/cc_vendor_scripts_per_once.py: added per once vendor cloud config script doc/examples/cloud-config-vendor-data.txt: documentation of vendor-data examples doc/vendordata.txt: documentation of vendordata for vendors (RENAMED) tests/unittests/test_userdata.py => tests/unittests/test_userdata.py TO: tests/unittests/test_userdata.py => tests/unittests/test_data.py: userdata test cases are not expanded to confirm superiority over vendor data. bin/cloud-init: change instances of 'consume_userdata' to 'consume_data' cloudinit/handlers/cloud_config.py: Added vendor script handling to default cloud-config modules cloudinit/handlers/shell_script.py: Added ability to change the path key to support vendor provided 'vendor-scripts'. Defaults to 'script'. cloudinit/helpers.py: - Changed ConfigMerger to include handling of vendordata. - Changed helpers to include paths for vendordata. cloudinit/sources/__init__.py: Added functions for helping vendordata - get_vendordata_raw(): returns vendordata unprocessed - get_vendordata(): returns vendordata through userdata processor - has_vendordata(): indicator if vendordata is present - consume_vendordata(): datasource directive for indicating explict user approval of vendordata consumption. Defaults to 'false' cloudinit/stages.py: Re-jiggered for handling of vendordata - _initial_subdirs(): added vendor script definition - update(): added self._store_vendordata() - [ADDED] _store_vendordata(): store vendordata - _get_default_handlers(): modified to allow for filtering which handlers will run against vendordata - [ADDED] _do_handlers(): moved logic from consume_userdata to _do_handlers(). This allows _consume_vendordata() and _consume_userdata() to use the same code path. - [RENAMED] consume_userdata() to _consume_userdata() - [ADDED] _consume_vendordata() for handling vendordata - run after userdata to get user cloud-config - uses ConfigMerger to get the configuration from the instance perspective about whether or not to use vendordata - [ADDED] consume_data() to call _consume_{user,vendor}data cloudinit/util.py: - [ADDED] get_nested_option_as_list() used by cc_vendor* for getting a nested value from a dict and returned as a list - runparts(): added 'exe_prefix' for running exe with a prefix, used by cc_vendor* config/cloud.cfg: Added vendor script execution as default tests/unittests/test_runs/test_merge_run.py: changed consume_userdata() to consume_data() tests/unittests/test_runs/test_simple_run.py: changed consume_userdata() to consume_data() --- doc/vendordata.txt | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 doc/vendordata.txt (limited to 'doc/vendordata.txt') diff --git a/doc/vendordata.txt b/doc/vendordata.txt new file mode 100644 index 00000000..63a6c999 --- /dev/null +++ b/doc/vendordata.txt @@ -0,0 +1,93 @@ +=== Overview === +Vendordata is data provided by the entity that launches an instance. +The cloud provider makes this data available to the instance via in one +way or another. + +Vendordata follows the same rules as user-data, with the following +caveauts: + 1. Users have ultimate control over vendordata + 2. By default it only runs on first boot + 3. Vendordata runs at the users pleasure. If the use of + vendordata is required for the instance to run, then + vendordata should not be used. + 4. Most vendor operations should be done either via script, + boot_hook or upstart job. + +Vendors utilizing the vendordata channel are strongly advised to +use the #cloud-config-jsonp method, otherwise they risk that a +user can accidently override choices. + +Further, we strongly advise vendors to not 'be evil'. By evil, we +mean any action that could compromise a system. Since users trust +you, please take care to make sure that any vendordata is safe, +atomic, indopenant and does not put your users at risk. + +cloud-init can read this input and act on it in different ways. + +=== Input Formats === +cloud-init will download and cache to filesystem any vendor-data that it +finds. However, certain types of vendor-data are handled specially. + + * Gzip Compressed Content + content found to be gzip compressed will be uncompressed, and + these rules applied to the uncompressed data + + * Mime Multi Part archive + This list of rules is applied to each part of this multi-part file + Using a mime-multi part file, the user can specify more than one + type of data. For example, both a user data script and a + cloud-config type could be specified. + + * vendor-data Script + begins with: #! or Content-Type: text/x-shellscript + script will be executed at "rc.local-like" level during first boot. + rc.local-like means "very late in the boot sequence" + + * Include File + begins with #include or Content-Type: text/x-include-url + This content is a "include" file. The file contains a list of + urls, one per line. Each of the URLs will be read, and their content + will be passed through this same set of rules. Ie, the content + read from the URL can be gzipped, mime-multi-part, or plain text + +* Include File Once + begins with #include-once or Content-Type: text/x-include-once-url + This content is a "include" file. The file contains a list of + urls, one per line. Each of the URLs will be read, and their content + will be passed through this same set of rules. Ie, the content + read from the URL can be gzipped, mime-multi-part, or plain text + This file will just be downloaded only once per instance, and its + contents cached for subsequent boots. This allows you to pass in + one-time-use or expiring URLs. + + * Cloud Config Data + begins with #cloud-config or Content-Type: text/cloud-config + + This content is "cloud-config" data. See the examples for a + commented example of supported config formats. + + * Upstart Job + begins with #upstart-job or Content-Type: text/upstart-job + + Content is placed into a file in /etc/init, and will be consumed + by upstart as any other upstart job. + + * Cloud Boothook + begins with #cloud-boothook or Content-Type: text/cloud-boothook + + This content is "boothook" data. It is stored in a file under + /var/lib/cloud and then executed immediately. + + This is the earliest "hook" available. Note, that there is no + mechanism provided for running only once. The boothook must take + care of this itself. It is provided with the instance id in the + environment variable "INSTANCE_ID". This could be made use of to + provide a 'once-per-instance' + +=== Examples === +There are examples in the examples subdirectory. +Additionally, the 'tools' directory contains 'write-mime-multipart', +which can be used to easily generate mime-multi-part files from a list +of input files. That data can then be given to an instance. + +See 'write-mime-multipart --help' for usage. -- cgit v1.2.3 From 91d15b934b655fc56af416309ee020caac24ca3f Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Fri, 17 Jan 2014 08:27:09 -0700 Subject: pep8 and pylint fixes; typo fix for documentation --- cloudinit/stages.py | 11 +++++------ doc/vendordata.txt | 2 +- tests/unittests/test_data.py | 4 +--- 3 files changed, 7 insertions(+), 10 deletions(-) (limited to 'doc/vendordata.txt') diff --git a/cloudinit/stages.py b/cloudinit/stages.py index 5dced998..8ae0bc0d 100644 --- a/cloudinit/stages.py +++ b/cloudinit/stages.py @@ -26,8 +26,7 @@ import copy import os import sys -from cloudinit.settings import (PER_ALWAYS, PER_INSTANCE, FREQUENCIES, - CLOUD_CONFIG) +from cloudinit.settings import (PER_INSTANCE, FREQUENCIES, CLOUD_CONFIG) from cloudinit import handlers @@ -510,13 +509,13 @@ class Init(object): vdcfg = {'enabled': False} LOG.warn("invalid 'vendor_data' setting. resetting to: %s", vdcfg) - if not util.is_true(vdcfg.get('enabled')): - LOG.debug("vendordata consumption is disabled.") - return - enabled = vdcfg.get('enabled') no_handlers = vdcfg.get('disabled_handlers', None) + if not util.is_true(enabled): + LOG.debug("vendordata consumption is disabled.") + return + LOG.debug("vendor data will be consumed. disabled_handlers=%s", no_handlers) diff --git a/doc/vendordata.txt b/doc/vendordata.txt index 63a6c999..530e3b4c 100644 --- a/doc/vendordata.txt +++ b/doc/vendordata.txt @@ -20,7 +20,7 @@ user can accidently override choices. Further, we strongly advise vendors to not 'be evil'. By evil, we mean any action that could compromise a system. Since users trust you, please take care to make sure that any vendordata is safe, -atomic, indopenant and does not put your users at risk. +atomic, idempotent and does not put your users at risk. cloud-init can read this input and act on it in different ways. diff --git a/tests/unittests/test_data.py b/tests/unittests/test_data.py index a753debf..68729c57 100644 --- a/tests/unittests/test_data.py +++ b/tests/unittests/test_data.py @@ -264,13 +264,11 @@ vendor_data: freq=PER_INSTANCE) mods = stages.Modules(initer) (_which_ran, _failures) = mods.run_section('cloud_init_modules') - cfg = mods.cfg + _cfg = mods.cfg vendor_script = initer.paths.get_ipath_cur('vendor_scripts') vendor_script_fns = "%s%s/part-001" % (new_root, vendor_script) self.assertTrue(os.path.exists(vendor_script_fns)) - - def test_merging_cloud_config(self): blob = ''' #cloud-config -- cgit v1.2.3 From fd5c4c76746cf66df35d7860a6fa2f86454c7596 Mon Sep 17 00:00:00 2001 From: Scott Moser Date: Fri, 17 Jan 2014 13:09:31 -0500 Subject: doc improvements --- doc/vendordata.txt | 108 +++++++++++++++++------------------------------------ 1 file changed, 34 insertions(+), 74 deletions(-) (limited to 'doc/vendordata.txt') diff --git a/doc/vendordata.txt b/doc/vendordata.txt index 530e3b4c..9acbe41c 100644 --- a/doc/vendordata.txt +++ b/doc/vendordata.txt @@ -1,88 +1,48 @@ === Overview === -Vendordata is data provided by the entity that launches an instance. -The cloud provider makes this data available to the instance via in one -way or another. +Vendordata is data provided by the entity that launches an instance +(for example, the cloud provider). This data can be used to +customize the image to fit into the particular environment it is +being run in. Vendordata follows the same rules as user-data, with the following -caveauts: - 1. Users have ultimate control over vendordata +caveats: + 1. Users have ultimate control over vendordata. They can disable its + execution or disable handling of specific parts of multipart input. 2. By default it only runs on first boot - 3. Vendordata runs at the users pleasure. If the use of - vendordata is required for the instance to run, then - vendordata should not be used. - 4. Most vendor operations should be done either via script, - boot_hook or upstart job. - -Vendors utilizing the vendordata channel are strongly advised to -use the #cloud-config-jsonp method, otherwise they risk that a -user can accidently override choices. - + 3. Vendordata can be disabled by the user. If the use of vendordata is + required for the instance to run, then vendordata should not be + used. + 4. user supplied cloud-config is merged over cloud-config from + vendordata. + +Users providing cloud-config data can use the '#cloud-config-jsonp' method +to more finely control their modifications to the vendor supplied +cloud-config. For example, if both vendor and user have provided +'runcnmd' then the default merge handler will cause the user's runcmd to +override the one provided by the vendor. To append to 'runcmd', the user +could better provide multipart input with a cloud-config-jsonp part like: + #cloud-config-jsonp + [{ "op": "add", "path": "/runcmd", "value": ["my", "command", "here"]}] + Further, we strongly advise vendors to not 'be evil'. By evil, we mean any action that could compromise a system. Since users trust you, please take care to make sure that any vendordata is safe, atomic, idempotent and does not put your users at risk. -cloud-init can read this input and act on it in different ways. - === Input Formats === cloud-init will download and cache to filesystem any vendor-data that it -finds. However, certain types of vendor-data are handled specially. - - * Gzip Compressed Content - content found to be gzip compressed will be uncompressed, and - these rules applied to the uncompressed data - - * Mime Multi Part archive - This list of rules is applied to each part of this multi-part file - Using a mime-multi part file, the user can specify more than one - type of data. For example, both a user data script and a - cloud-config type could be specified. - - * vendor-data Script - begins with: #! or Content-Type: text/x-shellscript - script will be executed at "rc.local-like" level during first boot. - rc.local-like means "very late in the boot sequence" - - * Include File - begins with #include or Content-Type: text/x-include-url - This content is a "include" file. The file contains a list of - urls, one per line. Each of the URLs will be read, and their content - will be passed through this same set of rules. Ie, the content - read from the URL can be gzipped, mime-multi-part, or plain text - -* Include File Once - begins with #include-once or Content-Type: text/x-include-once-url - This content is a "include" file. The file contains a list of - urls, one per line. Each of the URLs will be read, and their content - will be passed through this same set of rules. Ie, the content - read from the URL can be gzipped, mime-multi-part, or plain text - This file will just be downloaded only once per instance, and its - contents cached for subsequent boots. This allows you to pass in - one-time-use or expiring URLs. - - * Cloud Config Data - begins with #cloud-config or Content-Type: text/cloud-config - - This content is "cloud-config" data. See the examples for a - commented example of supported config formats. - - * Upstart Job - begins with #upstart-job or Content-Type: text/upstart-job - - Content is placed into a file in /etc/init, and will be consumed - by upstart as any other upstart job. - - * Cloud Boothook - begins with #cloud-boothook or Content-Type: text/cloud-boothook - - This content is "boothook" data. It is stored in a file under - /var/lib/cloud and then executed immediately. - - This is the earliest "hook" available. Note, that there is no - mechanism provided for running only once. The boothook must take - care of this itself. It is provided with the instance id in the - environment variable "INSTANCE_ID". This could be made use of to - provide a 'once-per-instance' +finds. Vendordata is handled exactly like user-data. That means that +the vendor can supply multipart input and have those parts acted on +in the same way as user-data. + +The only differences are: + * user-scripts are stored in a different location than user-scripts (to + avoid namespace collision) + * user can disable part handlers by cloud-config settings. + For example, to disable handling of 'part-handlers' in vendor-data, + the user could provide user-data like this: + #cloud-config + vendordata: {excluded: 'text/part-handler'} === Examples === There are examples in the examples subdirectory. -- cgit v1.2.3