merge from trunk

9 files changed, 625 insertions, 0 deletions

diff --git a/doc/rtd/topics/availability.rst b/doc/rtd/topics/availability.rst
new file mode 100644
index 00000000..2d58f808
--- /dev/null
+++ b/doc/rtd/topics/availability.rst
@@ -0,0 +1,20 @@
+============
+Availability
+============
+
+It is currently installed in the `Ubuntu Cloud Images`_ and also in the official `Ubuntu`_ images available on EC2.
+
+Versions for other systems can be (or have been) created for the following distributions:
+
+- Ubuntu
+- Fedora
+- Debian
+- RHEL
+- CentOS
+- *and more...*
+
+So ask your distribution provider where you can obtain an image with it built-in if one is not already available ☺
+
+
+.. _Ubuntu Cloud Images: http://cloud-images.ubuntu.com/
+.. _Ubuntu: http://www.ubuntu.com/
diff --git a/doc/rtd/topics/capabilities.rst b/doc/rtd/topics/capabilities.rst
new file mode 100644
index 00000000..63b34270
--- /dev/null
+++ b/doc/rtd/topics/capabilities.rst
@@ -0,0 +1,24 @@
+=====================
+Capabilities
+=====================
+
+- Setting a default locale
+- Setting a instance hostname
+- Generating instance ssh private keys
+- Adding ssh keys to a users ``.ssh/authorized_keys`` so they can log in
+- Setting up ephemeral mount points
+
+User configurability
+--------------------
+
+`Cloud-init`_ 's behavior can be configured via user-data.
+
+    User-data can be given by the user at instance launch time.
+
+This is done via the ``--user-data`` or ``--user-data-file`` argument to ec2-run-instances for example.
+
+* Check your local clients documentation for how to provide a `user-data` string
+  or `user-data` file for usage by cloud-init on instance creation.
+
+
+.. _Cloud-init: https://launchpad.net/cloud-init
diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst
new file mode 100644
index 00000000..59c58805
--- /dev/null
+++ b/doc/rtd/topics/datasources.rst
@@ -0,0 +1,192 @@
+.. _datasources:
+
+=========
+Datasources
+=========
+----------
+ What is a datasource?
+----------
+
+Datasources are sources of configuration data for cloud-init that typically come
+from the user (aka userdata) or come from the stack that created the configuration
+drive (aka metadata). Typical userdata would include files, yaml, and shell scripts
+while typical metadata would include server name, instance id, display name and other
+cloud specific details. Since there are multiple ways to provide this data (each cloud
+solution seems to prefer its own way) internally a datasource abstract class was
+created to allow for a single way to access the different cloud systems methods 
+to provide this data through the typical usage of subclasses.
+
+The current interface that a datasource object must provide is the following:
+
+.. sourcecode:: python
+    
+    # returns a mime multipart message that contains
+    # all the various fully-expanded components that
+    # were found from processing the raw userdata string
+    # - when filtering only the mime messages targeting
+    #   this instance id will be returned (or messages with
+    #   no instance id)
+    def get_userdata(self, apply_filter=False)
+    
+    # returns the raw userdata string (or none)
+    def get_userdata_raw(self)
+    
+    # returns a integer (or none) which can be used to identify
+    # this instance in a group of instances which are typically
+    # created from a single command, thus allowing programatic
+    # filtering on this launch index (or other selective actions)
+    @property
+    def launch_index(self)
+    
+    # the data sources' config_obj is a cloud-config formated
+    # object that came to it from ways other than cloud-config
+    # because cloud-config content would be handled elsewhere
+    def get_config_obj(self)
+    
+    #returns a list of public ssh keys
+    def get_public_ssh_keys(self)
+    
+    # translates a device 'short' name into the actual physical device
+    # fully qualified name (or none if said physical device is not attached
+    # or does not exist)
+    def device_name_to_device(self, name)
+    
+    # gets the locale string this instance should be applying 
+    # which typically used to adjust the instances locale settings files
+    def get_locale(self)
+    
+    @property
+    def availability_zone(self)
+    
+    # gets the instance id that was assigned to this instance by the 
+    # cloud provider or when said instance id does not exist in the backing
+    # metadata this will return 'iid-datasource'
+    def get_instance_id(self)
+    
+    # gets the fully qualified domain name that this host should  be using
+    # when configuring network or hostname releated settings, typically
+    # assigned either by the cloud provider or the user creating the vm
+    def get_hostname(self, fqdn=False)
+    
+    def get_package_mirror_info(self)
+
+---------------------------
+EC2
+---------------------------
+
+The EC2 datasource is the oldest and most widely used datasource that cloud-init
+supports. This datasource interacts with a *magic* ip that is provided to the 
+instance by the cloud provider. Typically this ip is ``169.254.169.254`` of which
+at this ip a http server is provided to the instance so that the instance can make
+calls to get instance userdata and instance metadata.
+
+Metadata is accessible via the following URL:
+
+::
+    
+    GET http://169.254.169.254/2009-04-04/meta-data/
+    ami-id
+    ami-launch-index
+    ami-manifest-path
+    block-device-mapping/
+    hostname
+    instance-id
+    instance-type
+    local-hostname
+    local-ipv4
+    placement/
+    public-hostname
+    public-ipv4
+    public-keys/
+    reservation-id
+    security-groups
+
+Userdata is accessible via the following URL:
+
+::
+    
+    GET http://169.254.169.254/2009-04-04/user-data
+    1234,fred,reboot,true | 4512,jimbo, | 173,,,
+
+Note that there are multiple versions of this data provided, cloud-init
+by default uses **2009-04-04** but newer versions can be supported with
+relative ease (newer versions have more data exposed, while maintaining
+backward compatibility with the previous versions). 
+
+To see which versions are supported from your cloud provider use the following URL:
+
+::
+    
+    GET http://169.254.169.254/
+    1.0
+    2007-01-19
+    2007-03-01
+    2007-08-29
+    2007-10-10
+    2007-12-15
+    2008-02-01
+    2008-09-01
+    2009-04-04
+    ...
+    latest
+        
+**Note:** internally in cloudinit the `boto`_ library used to fetch the instance
+userdata and instance metadata, feel free to check that library out, it provides
+many other useful EC2 functionality.
+
+---------------------------
+Config Drive
+---------------------------
+
+.. include:: ../../sources/configdrive/README.rst
+
+---------------------------
+Alt cloud
+---------------------------
+
+.. include:: ../../sources/altcloud/README.rst
+
+---------------------------
+No cloud
+---------------------------
+
+.. include:: ../../sources/nocloud/README.rst
+
+---------------------------
+MAAS
+---------------------------
+
+*TODO*
+
+For now see: http://maas.ubuntu.com/
+
+---------------------------
+CloudStack
+---------------------------
+
+*TODO*
+
+---------------------------
+OVF
+---------------------------
+
+*TODO*
+
+For now see: https://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/sources/ovf/
+
+---------------------------
+Fallback/None
+---------------------------
+
+This is the fallback datasource when no other datasource can be selected. It is
+the equivalent of a *empty* datasource in that it provides a empty string as userdata
+and a empty dictionary as metadata. It is useful for testing as well as for when
+you do not have a need to have an actual datasource to meet your instance 
+requirements (ie you just want to run modules that are not concerned with any
+external data). It is typically put at the end of the datasource search list
+so that if all other datasources are not matched, then this one will be so that
+the user is not left with an inaccessible instance.
+
+**Note:** the instance id that this datasource provides is ``iid-datasource-none``.
+
+.. _boto: http://docs.pythonboto.org/en/latest/
diff --git a/doc/rtd/topics/dir_layout.rst b/doc/rtd/topics/dir_layout.rst
new file mode 100644
index 00000000..8815d33d
--- /dev/null
+++ b/doc/rtd/topics/dir_layout.rst
@@ -0,0 +1,81 @@
+=========
+Directory layout
+=========
+
+Cloudinits's directory structure is somewhat different from a regular application::
+
+  /var/lib/cloud/
+      - data/
+         - instance-id  
+         - previous-instance-id
+         - datasource
+         - previous-datasource
+         - previous-hostname
+      - handlers/
+      - instance
+      - instances/
+          i-00000XYZ/
+            - boot-finished
+            - cloud-config.txt
+            - datasource
+            - handlers/
+            - obj.pkl
+            - scripts/
+            - sem/
+            - user-data.txt
+            - user-data.txt.i
+      - scripts/
+         - per-boot/
+         - per-instance/
+         - per-once/
+      - seed/
+      - sem/
+
+``/var/lib/cloud``
+
+  The main directory containing the cloud-init specific subdirectories.
+  It is typically located at ``/var/lib`` but there are certain configuration
+  scenarios where this can be altered. 
+
+  TBD, describe this overriding more.
+
+``data/``
+
+  Contains information releated to instance ids, datasources and hostnames of the previous
+  and current instance if they are different. These can be examined as needed to
+  determine any information releated to a previous boot (if applicable).
+
+``handlers/``
+
+  Custom ``part-handlers`` code is written out here. Files that end up here are written
+  out with in the scheme of ``part-handler-XYZ`` where ``XYZ`` is the handler number (the
+  first handler found starts at 0).
+
+
+``instance``
+
+  A symlink to the current ``instances/`` subdirectory that points to the currently
+  active instance (which is active is dependent on the datasource loaded).
+
+``instances/``
+
+  All instances that were created using this image end up with instance identifer
+  subdirectories (and corresponding data for each instance). The currently active
+  instance will be symlinked the the ``instance`` symlink file defined previously.
+
+``scripts/``
+
+  Scripts that are downloaded/created by the corresponding ``part-handler`` will end up
+  in one of these subdirectories.
+
+``seed/``
+
+  TBD
+
+``sem/``
+
+  Cloud-init has a concept of a module sempahore, which basically consists
+  of the module name and its frequency. These files are used to ensure a module
+  is only ran `per-once`, `per-instance`, `per-always`. This folder contains 
+  sempaphore `files` which are only supposed to run `per-once` (not tied to the instance id).
+
diff --git a/doc/rtd/topics/examples.rst b/doc/rtd/topics/examples.rst
new file mode 100644
index 00000000..36508bde
--- /dev/null
+++ b/doc/rtd/topics/examples.rst
@@ -0,0 +1,133 @@
+.. _yaml_examples:
+
+=========
+Cloud config examples
+=========
+
+Including users and groups
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-user-groups.txt
+   :language: yaml
+   :linenos:
+
+
+Writing out arbitrary files
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-write-files.txt
+   :language: yaml
+   :linenos:
+
+
+Adding a yum repository
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-yum-repo.txt
+   :language: yaml
+   :linenos:
+
+Configure an instances trusted CA certificates
+------------------------------------------------------
+
+.. literalinclude:: ../../examples/cloud-config-ca-certs.txt
+   :language: yaml
+   :linenos:
+
+Configure an instances resolv.conf
+------------------------------------------------------
+
+*Note:* when using a config drive and a RHEL like system resolv.conf
+will also be managed 'automatically' due to the available information
+provided for dns servers in the config drive network format. For those
+that wish to have different settings use this module.
+
+.. literalinclude:: ../../examples/cloud-config-resolv-conf.txt
+   :language: yaml
+   :linenos:
+
+Install and run `chef`_ recipes
+------------------------------------------------------
+
+.. literalinclude:: ../../examples/cloud-config-chef.txt
+   :language: yaml
+   :linenos:
+
+Setup and run `puppet`_
+------------------------------------------------------
+
+.. literalinclude:: ../../examples/cloud-config-puppet.txt
+   :language: yaml
+   :linenos:
+
+Add apt repositories
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-add-apt-repos.txt
+   :language: yaml
+   :linenos:
+
+Run commands on first boot
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-boot-cmds.txt
+   :language: yaml
+   :linenos:
+
+.. literalinclude:: ../../examples/cloud-config-run-cmds.txt
+   :language: yaml
+   :linenos:
+
+
+Alter the completion message
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-final-message.txt
+   :language: yaml
+   :linenos:
+
+Install arbitrary packages
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-install-packages.txt
+   :language: yaml
+   :linenos:
+
+Run apt or yum upgrade
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-update-packages.txt
+   :language: yaml
+   :linenos:
+
+Adjust mount points mounted
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-mount-points.txt
+   :language: yaml
+   :linenos:
+
+Call a url when finished
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-phone-home.txt
+   :language: yaml
+   :linenos:
+
+Reboot/poweroff when finished
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-power-state.txt
+   :language: yaml
+   :linenos:
+
+Configure instances ssh-keys
+---------------------------
+
+.. literalinclude:: ../../examples/cloud-config-ssh-keys.txt
+   :language: yaml
+   :linenos:
+   
+
+.. _chef: http://www.opscode.com/chef/
+.. _puppet: http://puppetlabs.com/
diff --git a/doc/rtd/topics/format.rst b/doc/rtd/topics/format.rst
new file mode 100644
index 00000000..eba9533f
--- /dev/null
+++ b/doc/rtd/topics/format.rst
@@ -0,0 +1,159 @@
+=========
+Formats
+=========
+
+User data that will be acted upon by cloud-init must be in one of the following types.
+
+Gzip Compressed Content
+------------------------
+
+Content found to be gzip compressed will be uncompressed.
+The uncompressed data will then be used as if it were not compressed. 
+This is typically is useful because user-data is limited to ~16384 [#]_ bytes.
+
+Mime Multi Part Archive
+------------------------
+
+This list of rules is applied to each part of this multi-part file. 
+Using a mime-multi part file, the user can specify more than one type of data.
+
+For example, both a user data script and a cloud-config type could be specified.
+
+Supported content-types:
+
+- text/x-include-once-url
+- text/x-include-url
+- text/cloud-config-archive
+- text/upstart-job
+- text/cloud-config
+- text/part-handler
+- text/x-shellscript
+- text/cloud-boothook
+
+Helper script to generate mime messages
+~~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+   #!/usr/bin/python
+   
+   import sys
+   
+   from email.mime.multipart import MIMEMultipart
+   from email.mime.text import MIMEText
+   
+   if len(sys.argv) == 1:
+       print("%s input-file:type ..." % (sys.argv[0]))
+       sys.exit(1)
+   
+   combined_message = MIMEMultipart()
+   for i in sys.argv[1:]:
+       (filename, format_type) = i.split(":", 1)
+       with open(filename) as fh:
+           contents = fh.read()
+       sub_message = MIMEText(contents, format_type, sys.getdefaultencoding())
+       sub_message.add_header('Content-Disposition', 'attachment; filename="%s"' % (filename))
+       combined_message.attach(sub_message)
+   
+   print(combined_message)
+
+
+User-Data Script
+------------------------
+
+Typically used by those who just want to execute a shell script.
+
+Begins with: ``#!`` or ``Content-Type: text/x-shellscript`` when using a MIME archive.
+
+Example
+~~~~~~~
+
+::
+
+  $ cat myscript.sh
+  
+  #!/bin/sh
+  echo "Hello World.  The time is now $(date -R)!" | tee /root/output.txt
+  
+  $ euca-run-instances --key mykey --user-data-file myscript.sh ami-a07d95c9 
+
+Include File
+------------
+
+This content is a ``include`` file.
+
+The file contains a list of urls, one per line.
+Each of the URLs will be read, and their content will be passed through this same set of rules.
+Ie, the content read from the URL can be gzipped, mime-multi-part, or plain text.
+
+Begins with: ``#include`` or ``Content-Type: text/x-include-url``  when using a MIME archive.
+
+Cloud Config Data
+-----------------
+
+Cloud-config is the simplest way to accomplish some things
+via user-data. Using cloud-config syntax, the user can specify certain things in a human friendly format. 
+
+These things include:
+
+- apt upgrade should be run on first boot
+- a different apt mirror should be used
+- additional apt sources should be added
+- certain ssh keys should be imported
+- *and many more...*
+
+**Note:** The file must be valid yaml syntax.
+
+See the :ref:`yaml_examples` section for a commented set of examples of supported cloud config formats.
+
+Begins with: ``#cloud-config`` or ``Content-Type: text/cloud-config`` when using a MIME archive.
+
+Upstart Job
+-----------
+
+Content is placed into a file in ``/etc/init``, and will be consumed by upstart as any other upstart job.
+
+Begins with: ``#upstart-job`` or ``Content-Type: text/upstart-job`` when using a MIME archive.
+
+Cloud Boothook
+--------------
+
+This content is ``boothook`` data. It is stored in a file under ``/var/lib/cloud`` and then executed immediately.
+This is the earliest ``hook`` available. Note, that there is no mechanism provided for running only once. The boothook must take care of this itself.
+It is provided with the instance id in the environment variable ``INSTANCE_I``. This could be made use of to provide a 'once-per-instance' type of functionality.
+
+Begins with: ``#cloud-boothook`` or ``Content-Type: text/cloud-boothook`` when using a MIME archive.
+
+Part Handler
+------------
+
+This is a ``part-handler``. It will be written to a file in ``/var/lib/cloud/data`` based on its filename (which is generated).
+This must be python code that contains a ``list_types`` method and a ``handle_type`` method. 
+Once the section is read the ``list_types`` method will be called. It must return a list of mime-types that this part-handler handles.
+
+The ``handle_type`` method must be like:
+
+.. code-block:: python
+
+    def handle_part(data, ctype, filename, payload):
+      # data = the cloudinit object
+      # ctype = "__begin__", "__end__", or the mime-type of the part that is being handled.
+      # filename = the filename of the part (or a generated filename if none is present in mime data)
+      # payload = the parts' content
+
+Cloud-init will then call the ``handle_type`` method once at begin, once per part received, and once at end.
+The ``begin`` and ``end`` calls are to allow the part handler to do initialization or teardown.
+
+Begins with: ``#part-handler`` or ``Content-Type: text/part-handler`` when using a MIME archive.
+
+Example
+~~~~~~~
+
+.. literalinclude:: ../../examples/part-handler.txt
+   :language: python
+   :linenos:
+
+Also this `blog`_ post offers another example for more advanced usage.
+
+.. [#] See your cloud provider for applicable user-data size limitations...
+.. _blog: http://foss-boss.blogspot.com/2011/01/advanced-cloud-init-custom-handlers.html
diff --git a/doc/rtd/topics/hacking.rst b/doc/rtd/topics/hacking.rst
new file mode 100644
index 00000000..96ab88ef
--- /dev/null
+++ b/doc/rtd/topics/hacking.rst
@@ -0,0 +1 @@
+.. include:: ../../../HACKING.rst
diff --git a/doc/rtd/topics/modules.rst b/doc/rtd/topics/modules.rst
new file mode 100644
index 00000000..d4dd55df
--- /dev/null
+++ b/doc/rtd/topics/modules.rst
@@ -0,0 +1,3 @@
+=========
+Modules
+=========
diff --git a/doc/rtd/topics/moreinfo.rst b/doc/rtd/topics/moreinfo.rst
new file mode 100644
index 00000000..19e96af0
--- /dev/null
+++ b/doc/rtd/topics/moreinfo.rst
@@ -0,0 +1,12 @@
+=========
+More information
+=========
+
+Useful external references
+-------------------------
+
+- `The beauty of cloudinit`_
+- `Introduction to cloud-init`_ (video)
+
+.. _Introduction to cloud-init: http://www.youtube.com/watch?v=-zL3BdbKyGY
+.. _The beauty of cloudinit: http://brandon.fuller.name/archives/2011/05/02/06.40.57/