Significant re-working of the userdata handling and introduction of

vendordata.

Vendordata is a datasource provided userdata-like blob that is parsed
similiarly to userdata, execept at the user's pleasure.


cloudinit/config/cc_scripts_vendor.py: added vendor script cloud config

cloudinit/config/cc_vendor_scripts_per_boot.py: added vendor per boot
    cloud config

cloudinit/config/cc_vendor_scripts_per_instance.py: added vendor per
    instance vendor cloud config

cloudinit/config/cc_vendor_scripts_per_once.py: added per once vendor
    cloud config script

doc/examples/cloud-config-vendor-data.txt: documentation of vendor-data
    examples

doc/vendordata.txt: documentation of vendordata for vendors

(RENAMED) tests/unittests/test_userdata.py => tests/unittests/test_userdata.py
      TO: tests/unittests/test_userdata.py => tests/unittests/test_data.py:
    userdata test cases are not expanded to confirm superiority over vendor
    data.

bin/cloud-init: change instances of 'consume_userdata' to 'consume_data'

cloudinit/handlers/cloud_config.py: Added vendor script handling to default
    cloud-config modules

cloudinit/handlers/shell_script.py: Added ability to change the path key to
    support vendor provided 'vendor-scripts'. Defaults to 'script'.

cloudinit/helpers.py:
    - Changed ConfigMerger to include handling of vendordata.
    - Changed helpers to include paths for vendordata.

cloudinit/sources/__init__.py: Added functions for helping vendordata
    - get_vendordata_raw(): returns vendordata unprocessed
    - get_vendordata(): returns vendordata through userdata processor
    - has_vendordata(): indicator if vendordata is present
    - consume_vendordata(): datasource directive for indicating explict
        user approval of vendordata consumption. Defaults to 'false'

cloudinit/stages.py: Re-jiggered for handling of vendordata
    - _initial_subdirs(): added vendor script definition
    - update(): added self._store_vendordata()
    - [ADDED] _store_vendordata(): store vendordata
    - _get_default_handlers(): modified to allow for filtering
        which handlers will run against vendordata
    - [ADDED] _do_handlers(): moved logic from consume_userdata
        to _do_handlers(). This allows _consume_vendordata() and
        _consume_userdata() to use the same code path.
    - [RENAMED] consume_userdata() to _consume_userdata()
    - [ADDED] _consume_vendordata() for handling vendordata
        - run after userdata to get user cloud-config
        - uses ConfigMerger to get the configuration from the
            instance perspective about whether or not to use
            vendordata
    - [ADDED] consume_data() to call _consume_{user,vendor}data

cloudinit/util.py:
    - [ADDED] get_nested_option_as_list() used by cc_vendor* for
        getting a nested value from a dict and returned as a list
    - runparts(): added 'exe_prefix' for running exe with a prefix,
        used by cc_vendor*

config/cloud.cfg: Added vendor script execution as default

tests/unittests/test_runs/test_merge_run.py: changed consume_userdata() to
    consume_data()

tests/unittests/test_runs/test_simple_run.py: changed consume_userdata() to
    consume_data()

2 files changed, 109 insertions, 0 deletions

diff --git a/doc/examples/cloud-config-vendor-data.txt b/doc/examples/cloud-config-vendor-data.txt
new file mode 100644
index 00000000..7f90847b
--- /dev/null
+++ b/doc/examples/cloud-config-vendor-data.txt
@@ -0,0 +1,16 @@
+#cloud-config
+#
+# This explains how to control vendordata via a cloud-config
+#
+# On select Datasources, vendors have a channel for the consumptions
+# of all support user-data types via a special channel called
+# vendordata. Users of the end system are given ultimate control.
+#
+vendor_data:
+    enabled: True
+    prefix: /usr/bin/ltrace
+
+# enabled: whether it is enabled or not
+# prefix: the command to run before any vendor scripts.
+#   Note: this is a fairly weak method of containment. It should
+#         be used to profile a script, not to prevent its run
diff --git a/doc/vendordata.txt b/doc/vendordata.txt
new file mode 100644
index 00000000..63a6c999
--- /dev/null
+++ b/doc/vendordata.txt
@@ -0,0 +1,93 @@
+=== Overview ===
+Vendordata is data provided by the entity that launches an instance.
+The cloud provider makes this data available to the instance via in one
+way or another.
+
+Vendordata follows the same rules as user-data, with the following
+caveauts:
+    1. Users have ultimate control over vendordata
+    2. By default it only runs on first boot
+    3. Vendordata runs at the users pleasure. If the use of
+        vendordata is required for the instance to run, then
+        vendordata should not be used.
+    4. Most vendor operations should be done either via script,
+        boot_hook or upstart job.
+
+Vendors utilizing the vendordata channel are strongly advised to
+use the #cloud-config-jsonp method, otherwise they risk that a
+user can accidently override choices.
+
+Further, we strongly advise vendors to not 'be evil'. By evil, we
+mean any action that could compromise a system. Since users trust
+you, please take care to make sure that any vendordata is safe,
+atomic, indopenant and does not put your users at risk.
+
+cloud-init can read this input and act on it in different ways.
+
+=== Input Formats ===
+cloud-init will download and cache to filesystem any vendor-data that it
+finds.  However, certain types of vendor-data are handled specially.
+
+ * Gzip Compressed Content
+   content found to be gzip compressed will be uncompressed, and
+   these rules applied to the uncompressed data
+
+ * Mime Multi Part archive
+   This list of rules is applied to each part of this multi-part file
+   Using a mime-multi part file, the user can specify more than one
+   type of data.  For example, both a user data script and a
+   cloud-config type could be specified.
+
+ * vendor-data Script
+   begins with: #!            or Content-Type: text/x-shellscript
+   script will be executed at "rc.local-like" level during first boot.
+   rc.local-like means "very late in the boot sequence"
+
+ * Include File
+   begins with  #include      or Content-Type: text/x-include-url
+   This content is a "include" file.  The file contains a list of
+   urls, one per line.  Each of the URLs will be read, and their content
+   will be passed through this same set of rules.  Ie, the content
+   read from the URL can be gzipped, mime-multi-part, or plain text
+
+* Include File Once
+   begins with  #include-once      or Content-Type: text/x-include-once-url
+   This content is a "include" file.  The file contains a list of
+   urls, one per line.  Each of the URLs will be read, and their content
+   will be passed through this same set of rules.  Ie, the content
+   read from the URL can be gzipped, mime-multi-part, or plain text
+   This file will just be downloaded only once per instance, and its
+   contents cached for subsequent boots.  This allows you to pass in
+   one-time-use or expiring URLs.
+
+ * Cloud Config Data
+   begins with  #cloud-config or Content-Type: text/cloud-config
+
+   This content is "cloud-config" data.  See the examples for a
+   commented example of supported config formats.
+
+ * Upstart Job
+   begins with  #upstart-job  or Content-Type: text/upstart-job
+
+   Content is placed into a file in /etc/init, and will be consumed
+   by upstart as any other upstart job.
+
+ * Cloud Boothook
+   begins with #cloud-boothook or Content-Type: text/cloud-boothook
+
+   This content is "boothook" data.  It is stored in a file under
+   /var/lib/cloud and then executed immediately.
+
+   This is the earliest "hook" available.  Note, that there is no
+   mechanism provided for running only once.  The boothook must take
+   care of this itself.  It is provided with the instance id in the
+   environment variable "INSTANCE_ID".  This could be made use of to
+   provide a 'once-per-instance'
+
+=== Examples ===
+There are examples in the examples subdirectory.
+Additionally, the 'tools' directory contains 'write-mime-multipart',
+which can be used to easily generate mime-multi-part files from a list
+of input files.  That data can then be given to an instance.
+
+See 'write-mime-multipart --help' for usage.