merge from trunk

8 files changed, 298 insertions, 224 deletions

diff --git a/doc/examples/cloud-config-datasources.txt b/doc/examples/cloud-config-datasources.txt
index 65a3cdf5..3bde4aac 100644
--- a/doc/examples/cloud-config-datasources.txt
+++ b/doc/examples/cloud-config-datasources.txt
@@ -49,7 +49,6 @@ datasource:
     hostname_bounce:
       interface: eth0
       policy: on # [can be 'on', 'off' or 'force']
-    }
 
   SmartOS:
     # Smart OS datasource works over a serial console interacting with
diff --git a/doc/examples/cloud-config-disk-setup.txt b/doc/examples/cloud-config-disk-setup.txt
index 6ad61c33..0dfef8e8 100644
--- a/doc/examples/cloud-config-disk-setup.txt
+++ b/doc/examples/cloud-config-disk-setup.txt
@@ -1,24 +1,24 @@
-Cloud-init supports the creation of simple partition tables and file systems
-on devices.
+# Cloud-init supports the creation of simple partition tables and file systems
+# on devices.
 
-Default disk definitions for AWS
---------------------------------
-(Not implemented yet, but provided for future documentation)
+# Default disk definitions for AWS
+# --------------------------------
+# (Not implemented yet, but provided for future documentation)
 
- disk_setup:
-    ephmeral0:
-        type: 'mbr'
-        layout: True
-        overwrite: False
+disk_setup:
+   ephmeral0:
+       type: 'mbr'
+       layout: True
+       overwrite: False
 
- fs_setup:
-    - label: None,
-      filesystem: ext3
-      device: ephemeral0
-      partition: auto
+fs_setup:
+   - label: None,
+     filesystem: ext3
+     device: ephemeral0
+     partition: auto
 
-Default disk definitions for Windows Azure
-------------------------------------------
+# Default disk definitions for Windows Azure
+# ------------------------------------------
 
 device_aliases: {'ephemeral0': '/dev/sdb'}
 disk_setup:
@@ -34,8 +34,8 @@ fs_setup:
       replace_fs: ntfs
 
 
-Default disk definitions for SmartOS
-------------------------------------
+# Default disk definitions for SmartOS
+# ------------------------------------
 
 device_aliases: {'ephemeral0': '/dev/sdb'}
 disk_setup:
@@ -49,203 +49,203 @@ fs_setup:
       filesystem: ext3
       device: ephemeral0.0
 
-Cavaut for SmartOS: if ephemeral disk is not defined, then the disk will
-    not be automatically added to the mounts.
-
-
-The default definition is used to make sure that the ephemeral storage is
-setup properly.
-
-"disk_setup": disk partitioning
---------------------------------
-
-The disk_setup directive instructs Cloud-init to partition a disk. The format is:
-
- disk_setup:
-    ephmeral0:
-        type: 'mbr'
-        layout: 'auto'
-    /dev/xvdh:
-        type: 'mbr'
-        layout:
-            - 33
-            - [33, 82]
-            - 33
-        overwrite: True
-
-The format is a list of dicts of dicts. The first value is the name of the
-device and the subsequent values define how to create and layout the partition.
-
-The general format is:
-    disk_setup:
-        <DEVICE>:
-            type: 'mbr'
-            layout: <LAYOUT|BOOL>
-            overwrite: <BOOL>
-
-Where:
-    <DEVICE>: The name of the device. 'ephemeralX' and 'swap' are special
-                values which are specific to the cloud. For these devices
-                Cloud-init will look up what the real devices is and then
-                use it.
-
-                For other devices, the kernel device name is used. At this
-                time only simply kernel devices are supported, meaning
-                that device mapper and other targets may not work.
-
-                Note: At this time, there is no handling or setup of
-                device mapper targets.
+# Cavaut for SmartOS: if ephemeral disk is not defined, then the disk will
+#    not be automatically added to the mounts.
 
-    type=<TYPE>: Currently the following are supported:
-                    'mbr': default and setups a MS-DOS partition table
 
-                Note: At this time only 'mbr' partition tables are allowed.
-                    It is anticipated in the future that we'll have GPT as
-                    option in the future, or even "RAID" to create a mdadm
-                    RAID.
+# The default definition is used to make sure that the ephemeral storage is
+# setup properly.
 
-    layout={...}: The device layout. This is a list of values, with the
-                percentage of disk that partition will take.
-                Valid options are:
-                    [<SIZE>, [<SIZE>, <PART_TYPE]]
+# "disk_setup": disk partitioning
+# --------------------------------
 
-                Where <SIZE> is the _percentage_ of the disk to use, while
-                <PART_TYPE> is the numerical value of the partition type.
+# The disk_setup directive instructs Cloud-init to partition a disk. The format is:
 
-                The following setups two partitions, with the first
-                partition having a swap label, taking 1/3 of the disk space
-                and the remainder being used as the second partition.
-                    /dev/xvdh':
-                        type: 'mbr'
-                        layout:
-                            - [33,82]
-                            - 66
-                        overwrite: True
-
-                When layout is "true" it means single partition the entire
-                device.
-
-                When layout is "false" it means don't partition or ignore
-                existing partitioning.
-
-                If layout is set to "true" and overwrite is set to "false",
-                it will skip partitioning the device without a failure.
-
-    overwrite=<BOOL>: This describes whether to ride with saftey's on and
-                everything holstered.
-
-                'false' is the default, which means that:
-                    1. The device will be checked for a partition table
-                    2. The device will be checked for a file system
-                    3. If either a partition of file system is found, then
-                        the operation will be _skipped_.
-
-                'true' is cowboy mode. There are no checks and things are
-                    done blindly. USE with caution, you can do things you
-                    really, really don't want to do.
-
-
-fs_setup: Setup the file system
--------------------------------
-
-fs_setup describes the how the file systems are supposed to look.
+disk_setup:
+   ephmeral0:
+       type: 'mbr'
+       layout: 'auto'
+   /dev/xvdh:
+       type: 'mbr'
+       layout:
+           - 33
+           - [33, 82]
+           - 33
+       overwrite: True
+
+# The format is a list of dicts of dicts. The first value is the name of the
+# device and the subsequent values define how to create and layout the
+# partition.
+# The general format is:
+#    disk_setup:
+#        <DEVICE>:
+#            type: 'mbr'
+#            layout: <LAYOUT|BOOL>
+#            overwrite: <BOOL>
+#
+# Where:
+#    <DEVICE>: The name of the device. 'ephemeralX' and 'swap' are special
+#                values which are specific to the cloud. For these devices
+#                Cloud-init will look up what the real devices is and then
+#                use it.
+#
+#                For other devices, the kernel device name is used. At this
+#                time only simply kernel devices are supported, meaning
+#                that device mapper and other targets may not work.
+#
+#                Note: At this time, there is no handling or setup of
+#                device mapper targets.
+#
+#    type=<TYPE>: Currently the following are supported:
+#                    'mbr': default and setups a MS-DOS partition table
+#
+#                Note: At this time only 'mbr' partition tables are allowed.
+#                    It is anticipated in the future that we'll have GPT as
+#                    option in the future, or even "RAID" to create a mdadm
+#                    RAID.
+#
+#    layout={...}: The device layout. This is a list of values, with the
+#                percentage of disk that partition will take.
+#                Valid options are:
+#                    [<SIZE>, [<SIZE>, <PART_TYPE]]
+#
+#                Where <SIZE> is the _percentage_ of the disk to use, while
+#                <PART_TYPE> is the numerical value of the partition type.
+#
+#                The following setups two partitions, with the first
+#                partition having a swap label, taking 1/3 of the disk space
+#                and the remainder being used as the second partition.
+#                    /dev/xvdh':
+#                        type: 'mbr'
+#                        layout:
+#                            - [33,82]
+#                            - 66
+#                        overwrite: True
+#
+#                When layout is "true" it means single partition the entire
+#                device.
+#
+#                When layout is "false" it means don't partition or ignore
+#                existing partitioning.
+#
+#                If layout is set to "true" and overwrite is set to "false",
+#                it will skip partitioning the device without a failure.
+#
+#    overwrite=<BOOL>: This describes whether to ride with saftey's on and
+#                everything holstered.
+#
+#                'false' is the default, which means that:
+#                    1. The device will be checked for a partition table
+#                    2. The device will be checked for a file system
+#                    3. If either a partition of file system is found, then
+#                        the operation will be _skipped_.
+#
+#                'true' is cowboy mode. There are no checks and things are
+#                    done blindly. USE with caution, you can do things you
+#                    really, really don't want to do.
+#
+#
+# fs_setup: Setup the file system
+# -------------------------------
+#
+# fs_setup describes the how the file systems are supposed to look.
 
- fs_setup:
-    - label: ephemeral0
-      filesystem: 'ext3'
-      device: 'ephemeral0'
-      partition: 'auto'
-    - label:  mylabl2
-      filesystem: 'ext4'
-      device: '/dev/xvda1'
-    - special:
-      cmd: mkfs -t %(FILESYSTEM)s -L %(LABEL)s %(DEVICE)s
-      filesystem: 'btrfs'
-      device: '/dev/xvdh'
-
-The general format is:
-    fs_setup:
-        - label: <LABEL>
-          filesystem: <FS_TYPE>
-          device: <DEVICE>
-          partition: <PART_VALUE>
-          overwrite: <OVERWRITE>
-          replace_fs: <FS_TYPE>
-
-Where:
-    <LABEL>: The file system label to be used. If set to None, no label is
-        used.
-
-    <FS_TYPE>: The file system type. It is assumed that the there
-        will be a "mkfs.<FS_TYPE>" that behaves likes "mkfs". On a standard
-        Ubuntu Cloud Image, this means that you have the option of ext{2,3,4},
-        and vfat by default.
-
-    <DEVICE>: The device name. Special names of 'ephemeralX' or 'swap'
-        are allowed and the actual device is acquired from the cloud datasource.
-        When using 'ephemeralX' (i.e. ephemeral0), make sure to leave the
-        label as 'ephemeralX' otherwise there may be issues with the mounting
-        of the ephemeral storage layer.
-
-        If you define the device as 'ephemeralX.Y' then Y will be interpetted
-        as a partition value. However, ephermalX.0 is the _same_ as ephemeralX.
-
-    <PART_VALUE>:
-        Partition definitions are overwriten if you use the '<DEVICE>.Y' notation.
-
-        The valid options are:
-        "auto|any": tell cloud-init not to care whether there is a partition
-            or not. Auto will use the first partition that does not contain a
-            file system already. In the absence of a partition table, it will
-            put it directly on the disk.
-
-            "auto": If a file system that matches the specification in terms of
-            label, type and device, then cloud-init will skip the creation of
-            the file system.
-
-            "any": If a file system that matches the file system type and device,
-            then cloud-init will skip the creation of the file system.
-
-            Devices are selected based on first-detected, starting with partitions
-            and then the raw disk. Consider the following:
-                NAME     FSTYPE LABEL
-                xvdb
-                |-xvdb1  ext4
-                |-xvdb2
-                |-xvdb3  btrfs  test
-                \-xvdb4  ext4   test
-
-            If you ask for 'auto', label of 'test, and file system of 'ext4'
-            then cloud-init will select the 2nd partition, even though there
-            is a partition match at the 4th partition.
-
-            If you ask for 'any' and a label of 'test', then cloud-init will
-            select the 1st partition.
-
-            If you ask for 'auto' and don't define label, then cloud-init will
-            select the 1st partition.
-
-            In general, if you have a specific partition configuration in mind,
-            you should define either the device or the partition number. 'auto'
-            and 'any' are specifically intended for formating ephemeral storage or
-            for simple schemes.
-
-        "none": Put the file system directly on the device.
-
-        <NUM>: where NUM is the actual partition number.
-
-    <OVERWRITE>: Defines whether or not to overwrite any existing
-        filesystem.
-
-        "true": Indiscriminately destroy any pre-existing file system. Use at
-            your own peril.
-
-        "false": If an existing file system exists, skip the creation.
-
-    <REPLACE_FS>: This is a special directive, used for Windows Azure that
-        instructs cloud-init to replace a file system of <FS_TYPE>. NOTE:
-        unless you define a label, this requires the use of the 'any' partition
-        directive.
-
-Behavior Caveat: The default behavior is to _check_ if the file system exists.
-    If a file system matches the specification, then the operation is a no-op.
+fs_setup:
+   - label: ephemeral0
+     filesystem: 'ext3'
+     device: 'ephemeral0'
+     partition: 'auto'
+   - label:  mylabl2
+     filesystem: 'ext4'
+     device: '/dev/xvda1'
+   - special:
+     cmd: mkfs -t %(FILESYSTEM)s -L %(LABEL)s %(DEVICE)s
+     filesystem: 'btrfs'
+     device: '/dev/xvdh'
+
+# The general format is:
+#    fs_setup:
+#        - label: <LABEL>
+#          filesystem: <FS_TYPE>
+#          device: <DEVICE>
+#          partition: <PART_VALUE>
+#          overwrite: <OVERWRITE>
+#          replace_fs: <FS_TYPE>
+#
+# Where:
+#     <LABEL>: The file system label to be used. If set to None, no label is
+#        used.
+#
+#    <FS_TYPE>: The file system type. It is assumed that the there
+#        will be a "mkfs.<FS_TYPE>" that behaves likes "mkfs". On a standard
+#        Ubuntu Cloud Image, this means that you have the option of ext{2,3,4},
+#        and vfat by default.
+#
+#    <DEVICE>: The device name. Special names of 'ephemeralX' or 'swap'
+#        are allowed and the actual device is acquired from the cloud datasource.
+#        When using 'ephemeralX' (i.e. ephemeral0), make sure to leave the
+#        label as 'ephemeralX' otherwise there may be issues with the mounting
+#        of the ephemeral storage layer.
+#
+#        If you define the device as 'ephemeralX.Y' then Y will be interpetted
+#        as a partition value. However, ephermalX.0 is the _same_ as ephemeralX.
+#
+#    <PART_VALUE>:
+#        Partition definitions are overwriten if you use the '<DEVICE>.Y' notation.
+#
+#        The valid options are:
+#        "auto|any": tell cloud-init not to care whether there is a partition
+#            or not. Auto will use the first partition that does not contain a
+#            file system already. In the absence of a partition table, it will
+#            put it directly on the disk.
+#
+#            "auto": If a file system that matches the specification in terms of
+#            label, type and device, then cloud-init will skip the creation of
+#            the file system.
+#
+#            "any": If a file system that matches the file system type and device,
+#            then cloud-init will skip the creation of the file system.
+#
+#            Devices are selected based on first-detected, starting with partitions
+#            and then the raw disk. Consider the following:
+#                NAME     FSTYPE LABEL
+#                xvdb
+#                |-xvdb1  ext4
+#                |-xvdb2
+#                |-xvdb3  btrfs  test
+#                \-xvdb4  ext4   test
+#
+#            If you ask for 'auto', label of 'test, and file system of 'ext4'
+#            then cloud-init will select the 2nd partition, even though there
+#            is a partition match at the 4th partition.
+#
+#            If you ask for 'any' and a label of 'test', then cloud-init will
+#            select the 1st partition.
+#
+#            If you ask for 'auto' and don't define label, then cloud-init will
+#            select the 1st partition.
+#
+#            In general, if you have a specific partition configuration in mind,
+#            you should define either the device or the partition number. 'auto'
+#            and 'any' are specifically intended for formating ephemeral storage or
+#            for simple schemes.
+#
+#        "none": Put the file system directly on the device.
+#
+#        <NUM>: where NUM is the actual partition number.
+#
+#    <OVERWRITE>: Defines whether or not to overwrite any existing
+#        filesystem.
+#
+#        "true": Indiscriminately destroy any pre-existing file system. Use at
+#            your own peril.
+#
+#        "false": If an existing file system exists, skip the creation.
+#
+#    <REPLACE_FS>: This is a special directive, used for Windows Azure that
+#        instructs cloud-init to replace a file system of <FS_TYPE>. NOTE:
+#        unless you define a label, this requires the use of the 'any' partition
+#        directive.
+#
+# Behavior Caveat: The default behavior is to _check_ if the file system exists.
+#    If a file system matches the specification, then the operation is a no-op.
diff --git a/doc/examples/cloud-config-growpart.txt b/doc/examples/cloud-config-growpart.txt
index a459573d..393d5164 100644
--- a/doc/examples/cloud-config-growpart.txt
+++ b/doc/examples/cloud-config-growpart.txt
@@ -5,12 +5,10 @@
 #
 #  mode:
 #    values:
-#     * auto: use any option possible (growpart or parted)
+#     * auto: use any option possible (any available)
 #             if none are available, do not warn, but debug.
 #     * growpart: use growpart to grow partitions
 #             if growpart is not available, this is an error.
-#     * parted: use parted (parted resizepart) to resize partitions
-#             if parted is not available, this is an error.
 #     * off, false
 #
 # devices:
diff --git a/doc/examples/cloud-config-landscape.txt b/doc/examples/cloud-config-landscape.txt
index e4d23cc9..74e07b62 100644
--- a/doc/examples/cloud-config-landscape.txt
+++ b/doc/examples/cloud-config-landscape.txt
@@ -6,6 +6,9 @@
 # 
 # Note: 'tags' should be specified as a comma delimited string
 # rather than a list.
+#
+# You can get example key/values by running 'landscape-config',
+# answer question, then look at /etc/landscape/client.config
 landscape:
   client:
     url: "https://landscape.canonical.com/message-system"
@@ -13,3 +16,7 @@ landscape:
     data_path: "/var/lib/landscape/client"
     http_proxy: "http://my.proxy.com/foobar"
     tags: "server,cloud"
+    computer_title = footitle
+    https_proxy = fooproxy
+    registration_key = fookey
+    account_name = fooaccount
diff --git a/doc/examples/cloud-config-vendor-data.txt b/doc/examples/cloud-config-vendor-data.txt
new file mode 100644
index 00000000..7f90847b
--- /dev/null
+++ b/doc/examples/cloud-config-vendor-data.txt
@@ -0,0 +1,16 @@
+#cloud-config
+#
+# This explains how to control vendordata via a cloud-config
+#
+# On select Datasources, vendors have a channel for the consumptions
+# of all support user-data types via a special channel called
+# vendordata. Users of the end system are given ultimate control.
+#
+vendor_data:
+    enabled: True
+    prefix: /usr/bin/ltrace
+
+# enabled: whether it is enabled or not
+# prefix: the command to run before any vendor scripts.
+#   Note: this is a fairly weak method of containment. It should
+#         be used to profile a script, not to prevent its run
diff --git a/doc/examples/cloud-config.txt b/doc/examples/cloud-config.txt
index b35b084d..61fa6065 100644
--- a/doc/examples/cloud-config.txt
+++ b/doc/examples/cloud-config.txt
@@ -74,7 +74,7 @@ apt_preserve_sources_list: true
 
 # 'source' entries in apt-sources that match this python regex
 # expression will be passed to add-apt-repository
-add_apt_repo_match = "^[\w-]+:\w"
+add_apt_repo_match: '^[\w-]+:\w'
 
 apt_sources:
  - source: "deb http://ppa.launchpad.net/byobu/ppa/ubuntu karmic main"
@@ -147,8 +147,13 @@ apt_sources:
 #     '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet']
 #
 # apt_get_upgrade_subcommand:
-# Specify a different subcommand for 'upgrade. The default is 'dist-upgrade'.
-# This is the subcommand that is invoked if package_upgrade is set to true above.
+#  Specify a different subcommand for 'upgrade. The default is 'dist-upgrade'.
+#  This is the subcommand that is invoked if package_upgrade is set to true above.
+#
+# apt_get_wrapper:
+#   command: eatmydata
+#   enabled: [True, False, "auto"]
+#  
 
 # Install additional packages on first boot
 #
diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst
index 5543ed34..cc0d0ede 100644
--- a/doc/rtd/topics/datasources.rst
+++ b/doc/rtd/topics/datasources.rst
@@ -130,10 +130,6 @@ To see which versions are supported from your cloud provider use the following U
     ...
     latest
         
-**Note:** internally in cloudinit the `boto`_ library used to fetch the instance
-userdata and instance metadata, feel free to check that library out, it provides
-many other useful EC2 functionality.
-
 ---------------------------
 Config Drive
 ---------------------------
diff --git a/doc/vendordata.txt b/doc/vendordata.txt
new file mode 100644
index 00000000..9acbe41c
--- /dev/null
+++ b/doc/vendordata.txt
@@ -0,0 +1,53 @@
+=== Overview ===
+Vendordata is data provided by the entity that launches an instance
+(for example, the cloud provider).  This data can be used to
+customize the image to fit into the particular environment it is
+being run in.
+
+Vendordata follows the same rules as user-data, with the following
+caveats:
+    1. Users have ultimate control over vendordata. They can disable its
+       execution or disable handling of specific parts of multipart input.
+    2. By default it only runs on first boot
+    3. Vendordata can be disabled by the user. If the use of vendordata is
+       required for the instance to run, then vendordata should not be
+       used.
+    4. user supplied cloud-config is merged over cloud-config from
+       vendordata.
+
+Users providing cloud-config data can use the '#cloud-config-jsonp' method
+to more finely control their modifications to the vendor supplied
+cloud-config.  For example, if both vendor and user have provided
+'runcnmd' then the default merge handler will cause the user's runcmd to
+override the one provided by the vendor.  To append to 'runcmd', the user
+could better provide multipart input with a cloud-config-jsonp part like:
+ #cloud-config-jsonp
+ [{ "op": "add", "path": "/runcmd", "value": ["my", "command", "here"]}]
+ 
+Further, we strongly advise vendors to not 'be evil'. By evil, we
+mean any action that could compromise a system. Since users trust
+you, please take care to make sure that any vendordata is safe,
+atomic, idempotent and does not put your users at risk.
+
+=== Input Formats ===
+cloud-init will download and cache to filesystem any vendor-data that it
+finds.  Vendordata is handled exactly like user-data.  That means that
+the vendor can supply multipart input and have those parts acted on
+in the same way as user-data.
+
+The only differences are:
+ * user-scripts are stored in a different location than user-scripts (to
+   avoid namespace collision)
+ * user can disable part handlers by cloud-config settings.
+   For example, to disable handling of 'part-handlers' in vendor-data,
+   the user could provide user-data like this:
+    #cloud-config
+    vendordata: {excluded: 'text/part-handler'}
+
+=== Examples ===
+There are examples in the examples subdirectory.
+Additionally, the 'tools' directory contains 'write-mime-multipart',
+which can be used to easily generate mime-multi-part files from a list
+of input files.  That data can then be given to an instance.
+
+See 'write-mime-multipart --help' for usage.