Merge rework branch in [Joshua Harlow]

 - unified binary that activates the various stages
   - Now using argparse + subcommands to specify the various CLI options
 - a stage module that clearly separates the stages of the different
   components (also described how they are used and in what order in the
   new unified binary)
 - user_data is now a module that just does user data processing while the
   actual activation and 'handling' of the processed user data is done via
   a separate set of files (and modules) with the main 'init' stage being the
   controller of this
   - creation of boot_hook, cloud_config, shell_script, upstart_job version 2
     modules (with classes that perform there functionality) instead of those
     having functionality that is attached to the cloudinit object (which
     reduces reuse and limits future functionality, and makes testing harder)
 - removal of global config that defined paths, shared config, now this is
   via objects making unit testing testing and global side-effects a non issue
 - creation of a 'helpers.py' 
   - this contains an abstraction for the 'lock' like objects that the various 
     module/handler running stages use to avoid re-running a given 
     module/handler for a given frequency. this makes it separated from 
     the actual usage of that object (thus helpful for testing and clear lines
     usage and how the actual job is accomplished)
     - a common 'runner' class is the main entrypoint using these locks to
       run function objects passed in (along with there arguments) and there
       frequency
   - add in a 'paths' object that provides access to the previously global
     and/or config based paths (thus providing a single entrypoint object/type
     that provides path information)
       - this also adds in the ability to change the path when constructing 
       that path 'object' and adding in additional config that can be used to 
       alter the root paths of 'joins' (useful for testing or possibly useful
       in chroots?)
        - config options now avaiable that can alter the 'write_root' and the 
         'read_root' when backing code uses the paths join() function
   - add a config parser subclass that will automatically add unknown sections
     and return default values (instead of throwing exceptions for these cases)
   - a new config merging class that will be the central object that knows
     how to do the common configuration merging from the various configuration
     sources. The order is the following:
     - cli config files override environment config files
       which override instance configs which override datasource
       configs which override base configuration which overrides
       default configuration.
 - remove the passing around of the 'cloudinit' object as a 'cloud' variable
   and instead pass around an 'interface' object that can be given to modules
   and handlers as there cloud access layer while the backing of that
   object can be varied (good for abstraction and testing)
 - use a single set of functions to do importing of modules
 - add a function in which will search for a given set of module names with
   a given set of attributes and return those which are found
 - refactor logging so that instead of using a single top level 'log' that
   instead each component/module can use its own logger (if desired), this
   should be backwards compatible with handlers and config modules that used
   the passed in logger (its still passed in)
   - ensure that all places where exception are caught and where applicable
     that the util logexc() is called, so that no exceptions that may occur
     are dropped without first being logged (where it makes sense for this 
     to happen)
 - add a 'requires' file that lists cloud-init dependencies
   - applying it in package creation (bdeb and brpm) as well as using it
     in the modified setup.py to ensure dependencies are installed when
     using that method of packaging
 - add a 'version.py' that lists the active version (in code) so that code
   inside cloud-init can report the version in messaging and other config files
 - cleanup of subprocess usage so that all subprocess calls go through the
   subp() utility method, which now has an exception type that will provide
   detailed information on python 2.6 and 2.7
 - forced all code loading, moving, chmod, writing files and other system
   level actions to go through standard set of util functions, this greatly 
   helps in debugging and determining exactly which system actions cloud-init is
   performing
 - switching out the templating engine cheetah for tempita since tempita has
   no external dependencies (minus python) while cheetah has many dependencies
   which makes it more difficult to adopt cloud-init in distros that may not
   have those dependencies
 - adjust url fetching and url trying to go through a single function that
   reads urls in the new 'url helper' file, this helps in tracing, debugging
   and knowing which urls are being called and/or posted to from with-in 
   cloud-init code
   - add in the sending of a 'User-Agent' header for all urls fetched that
     do not provide there own header mapping, derive this user-agent from
     the following template, 'Cloud-Init/{version}' where the version is the
     cloud-init version number
 - using prettytable for netinfo 'debug' printing since it provides a standard
   and defined output that should be easier to parse than a custom format
 - add a set of distro specific classes, that handle distro specific actions
   that modules and or handler code can use as needed, this is organized into
   a base abstract class with child classes that implement the shared 
   functionality. config determines exactly which subclass to load, so it can
   be easily extended as needed.
   - current functionality
      - network interface config file writing
      - hostname setting/updating
      - locale/timezone/ setting
      - updating of /etc/hosts (with templates or generically)
      - package commands (ie installing, removing)/mirror finding
      - interface up/down activating
   - implemented a debian + ubuntu subclass
   - implemented a redhat + fedora subclass
 - adjust the root 'cloud.cfg' file to now have distrobution/path specific 
   configuration values in it. these special configs are merged as the normal
   config is, but the system level config is not passed into modules/handlers
   - modules/handlers must go through the path and distro object instead
 - have the cloudstack datasource test the url before calling into boto to 
   avoid the long wait for boto to finish retrying and finally fail when
   the gateway meta-data address is unavailable
 - add a simple mock ec2 meta-data python based http server that can serve a
   very simple set of ec2 meta-data back to callers
      - useful for testing or for understanding what the ec2 meta-data 
        service can provide in terms of data or functionality
 - for ssh key and authorized key file parsing add in classes and util functions
   that maintain the state of individual lines, allowing for a clearer 
   separation of parsing and modification (useful for testing and tracing)
 - add a set of 'base' init.d scripts that can be used on systems that do
   not have full upstart or systemd support (or support that does not match
   the standard fedora/ubuntu implementation)
   - currently these are being tested on RHEL 6.2
 - separate the datasources into there own subdirectory (instead of being
   a top-level item), this matches how config 'modules' and user-data 'handlers'
   are also in there own subdirectory (thus helping new developers and others
   understand the code layout in a quicker manner)
 - add the building of rpms based off a new cli tool and template 'spec' file
   that will templatize and perform the necessary commands to create a source
   and binary package to be used with a cloud-init install on a 'rpm' supporting
   system
   - uses the new standard set of requires and converts those pypi requirements
     into a local set of package requirments (that are known to exist on RHEL
     systems but should also exist on fedora systems)
 - adjust the bdeb builder to be a python script (instead of a shell script) and
   make its 'control' file a template that takes in the standard set of pypi 
   dependencies and uses a local mapping (known to work on ubuntu) to create the
   packages set of dependencies (that should also work on ubuntu-like systems)
 - pythonify a large set of various pieces of code
   - remove wrapping return statements with () when it has no effect
   - upper case all constants used
   - correctly 'case' class and method names (where applicable)
   - use os.path.join (and similar commands) instead of custom path creation
   - use 'is None' instead of the frowned upon '== None' which picks up a large
     set of 'true' cases than is typically desired (ie for objects that have
     there own equality)
   - use context managers on locks, tempdir, chdir, file, selinux, umask, 
     unmounting commands so that these actions do not have to be closed and/or
     cleaned up manually in finally blocks, which is typically not done and will
     eventually be a bug in the future
   - use the 'abc' module for abstract classes base where possible
      - applied in the datasource root class, the distro root class, and the
        user-data v2 root class
 - when loading yaml, check that the 'root' type matches a predefined set of
   valid types (typically just 'dict') and throw a type error if a mismatch
   occurs, this seems to be a good idea to do when loading user config files
 - when forking a long running task (ie resizing a filesytem) use a new util
   function that will fork and then call a callback, instead of having to
   implement all that code in a non-shared location (thus allowing it to be
   used by others in the future)
 - when writing out filenames, go through a util function that will attempt to
   ensure that the given filename is 'filesystem' safe by replacing '/' with
   '_' and removing characters which do not match a given whitelist of allowed
   filename characters
 - for the varying usages of the 'blkid' command make a function in the util
   module that can be used as the single point of entry for interaction with
   that command (and its results) instead of having X separate implementations
 - place the rfc 8222 time formatting and uptime repeated pieces of code in the
   util module as a set of function with the name 'time_rfc2822'/'uptime'
 - separate the pylint+pep8 calling from one tool into two indivudal tools so
   that they can be called independently, add make file sections that can be 
   used to call these independently
 - remove the support for the old style config that was previously located in
   '/etc/ec2-init/ec2-config.cfg', no longer supported!
 - instead of using a altered config parser that added its own 'dummy' section
   on in the 'mcollective' module, use configobj which handles the parsing of
   config without sections better (and it also maintains comments instead of
   removing them)
 - use the new defaulting config parser (that will not raise errors on sections
   that do not exist or return errors when values are fetched that do not exist)
   in the 'puppet' module
 - for config 'modules' add in the ability for the module to provide a list of 
   distro names which it is known to work with, if when ran and the distro being
   used name does not match one of those in this list, a warning will be written
   out saying that this module may not work correctly on this distrobution
 - for all dynamically imported modules ensure that they are fixed up before 
   they are used by ensuring that they have certain attributes, if they do not
   have those attributes they will be set to a sensible set of defaults instead
 - adjust all 'config' modules and handlers to use the adjusted util functions
   and the new distro objects where applicable so that those pieces of code can 
   benefit from the unified and enhanced functionality being provided in that
   util module
 - fix a potential bug whereby when a #includeonce was encountered it would
   enable checking of urls against a cache, if later a #include was encountered
   it would continue checking against that cache, instead of refetching (which
   would likely be the expected case)
 - add a openstack/nova based pep8 extension utility ('hacking.py') that allows
   for custom checks (along with the standard pep8 checks) to occur when running
   'make pep8' and its derivatives

140 files changed, 11102 insertions, 5859 deletions

diff --git a/ChangeLog b/ChangeLog
index 1f1160d5..c3f71b9c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,196 @@
+0.7.0:
+ - unified binary that activates the various stages
+   - Now using argparse + subcommands to specify the various CLI options
+ - a stage module that clearly separates the stages of the different
+   components (also described how they are used and in what order in the
+   new unified binary)
+ - user_data is now a module that just does user data processing while the
+   actual activation and 'handling' of the processed user data is done via
+   a separate set of files (and modules) with the main 'init' stage being the
+   controller of this
+   - creation of boot_hook, cloud_config, shell_script, upstart_job version 2
+     modules (with classes that perform there functionality) instead of those
+     having functionality that is attached to the cloudinit object (which
+     reduces reuse and limits future functionality, and makes testing harder)
+ - removal of global config that defined paths, shared config, now this is
+   via objects making unit testing testing and global side-effects a non issue
+ - creation of a 'helpers.py' 
+   - this contains an abstraction for the 'lock' like objects that the various 
+     module/handler running stages use to avoid re-running a given 
+     module/handler for a given frequency. this makes it separated from 
+     the actual usage of that object (thus helpful for testing and clear lines
+     usage and how the actual job is accomplished)
+     - a common 'runner' class is the main entrypoint using these locks to
+       run function objects passed in (along with there arguments) and there
+       frequency
+   - add in a 'paths' object that provides access to the previously global
+     and/or config based paths (thus providing a single entrypoint object/type
+     that provides path information)
+       - this also adds in the ability to change the path when constructing 
+       that path 'object' and adding in additional config that can be used to 
+       alter the root paths of 'joins' (useful for testing or possibly useful
+       in chroots?)
+        - config options now avaiable that can alter the 'write_root' and the 
+         'read_root' when backing code uses the paths join() function
+   - add a config parser subclass that will automatically add unknown sections
+     and return default values (instead of throwing exceptions for these cases)
+   - a new config merging class that will be the central object that knows
+     how to do the common configuration merging from the various configuration
+     sources. The order is the following:
+     - cli config files override environment config files
+       which override instance configs which override datasource
+       configs which override base configuration which overrides
+       default configuration.
+ - remove the passing around of the 'cloudinit' object as a 'cloud' variable
+   and instead pass around an 'interface' object that can be given to modules
+   and handlers as there cloud access layer while the backing of that
+   object can be varied (good for abstraction and testing)
+ - use a single set of functions to do importing of modules
+ - add a function in which will search for a given set of module names with
+   a given set of attributes and return those which are found
+ - refactor logging so that instead of using a single top level 'log' that
+   instead each component/module can use its own logger (if desired), this
+   should be backwards compatible with handlers and config modules that used
+   the passed in logger (its still passed in)
+   - ensure that all places where exception are caught and where applicable
+     that the util logexc() is called, so that no exceptions that may occur
+     are dropped without first being logged (where it makes sense for this 
+     to happen)
+ - add a 'requires' file that lists cloud-init dependencies
+   - applying it in package creation (bdeb and brpm) as well as using it
+     in the modified setup.py to ensure dependencies are installed when
+     using that method of packaging
+ - add a 'version.py' that lists the active version (in code) so that code
+   inside cloud-init can report the version in messaging and other config files
+ - cleanup of subprocess usage so that all subprocess calls go through the
+   subp() utility method, which now has an exception type that will provide
+   detailed information on python 2.6 and 2.7
+ - forced all code loading, moving, chmod, writing files and other system
+   level actions to go through standard set of util functions, this greatly 
+   helps in debugging and determining exactly which system actions cloud-init is
+   performing
+ - switching out the templating engine cheetah for tempita since tempita has
+   no external dependencies (minus python) while cheetah has many dependencies
+   which makes it more difficult to adopt cloud-init in distros that may not
+   have those dependencies
+ - adjust url fetching and url trying to go through a single function that
+   reads urls in the new 'url helper' file, this helps in tracing, debugging
+   and knowing which urls are being called and/or posted to from with-in 
+   cloud-init code
+   - add in the sending of a 'User-Agent' header for all urls fetched that
+     do not provide there own header mapping, derive this user-agent from
+     the following template, 'Cloud-Init/{version}' where the version is the
+     cloud-init version number
+ - using prettytable for netinfo 'debug' printing since it provides a standard
+   and defined output that should be easier to parse than a custom format
+ - add a set of distro specific classes, that handle distro specific actions
+   that modules and or handler code can use as needed, this is organized into
+   a base abstract class with child classes that implement the shared 
+   functionality. config determines exactly which subclass to load, so it can
+   be easily extended as needed.
+   - current functionality
+      - network interface config file writing
+      - hostname setting/updating
+      - locale/timezone/ setting
+      - updating of /etc/hosts (with templates or generically)
+      - package commands (ie installing, removing)/mirror finding
+      - interface up/down activating
+   - implemented a debian + ubuntu subclass
+   - implemented a redhat + fedora subclass
+ - adjust the root 'cloud.cfg' file to now have distrobution/path specific 
+   configuration values in it. these special configs are merged as the normal
+   config is, but the system level config is not passed into modules/handlers
+   - modules/handlers must go through the path and distro object instead
+ - have the cloudstack datasource test the url before calling into boto to 
+   avoid the long wait for boto to finish retrying and finally fail when
+   the gateway meta-data address is unavailable
+ - add a simple mock ec2 meta-data python based http server that can serve a
+   very simple set of ec2 meta-data back to callers
+      - useful for testing or for understanding what the ec2 meta-data 
+        service can provide in terms of data or functionality
+ - for ssh key and authorized key file parsing add in classes and util functions
+   that maintain the state of individual lines, allowing for a clearer 
+   separation of parsing and modification (useful for testing and tracing)
+ - add a set of 'base' init.d scripts that can be used on systems that do
+   not have full upstart or systemd support (or support that does not match
+   the standard fedora/ubuntu implementation)
+   - currently these are being tested on RHEL 6.2
+ - separate the datasources into there own subdirectory (instead of being
+   a top-level item), this matches how config 'modules' and user-data 'handlers'
+   are also in there own subdirectory (thus helping new developers and others
+   understand the code layout in a quicker manner)
+ - add the building of rpms based off a new cli tool and template 'spec' file
+   that will templatize and perform the necessary commands to create a source
+   and binary package to be used with a cloud-init install on a 'rpm' supporting
+   system
+   - uses the new standard set of requires and converts those pypi requirements
+     into a local set of package requirments (that are known to exist on RHEL
+     systems but should also exist on fedora systems)
+ - adjust the bdeb builder to be a python script (instead of a shell script) and
+   make its 'control' file a template that takes in the standard set of pypi 
+   dependencies and uses a local mapping (known to work on ubuntu) to create the
+   packages set of dependencies (that should also work on ubuntu-like systems)
+ - pythonify a large set of various pieces of code
+   - remove wrapping return statements with () when it has no effect
+   - upper case all constants used
+   - correctly 'case' class and method names (where applicable)
+   - use os.path.join (and similar commands) instead of custom path creation
+   - use 'is None' instead of the frowned upon '== None' which picks up a large
+     set of 'true' cases than is typically desired (ie for objects that have
+     there own equality)
+   - use context managers on locks, tempdir, chdir, file, selinux, umask, 
+     unmounting commands so that these actions do not have to be closed and/or
+     cleaned up manually in finally blocks, which is typically not done and will
+     eventually be a bug in the future
+   - use the 'abc' module for abstract classes base where possible
+      - applied in the datasource root class, the distro root class, and the
+        user-data v2 root class
+ - when loading yaml, check that the 'root' type matches a predefined set of
+   valid types (typically just 'dict') and throw a type error if a mismatch
+   occurs, this seems to be a good idea to do when loading user config files
+ - when forking a long running task (ie resizing a filesytem) use a new util
+   function that will fork and then call a callback, instead of having to
+   implement all that code in a non-shared location (thus allowing it to be
+   used by others in the future)
+ - when writing out filenames, go through a util function that will attempt to
+   ensure that the given filename is 'filesystem' safe by replacing '/' with
+   '_' and removing characters which do not match a given whitelist of allowed
+   filename characters
+ - for the varying usages of the 'blkid' command make a function in the util
+   module that can be used as the single point of entry for interaction with
+   that command (and its results) instead of having X separate implementations
+ - place the rfc 8222 time formatting and uptime repeated pieces of code in the
+   util module as a set of function with the name 'time_rfc2822'/'uptime'
+ - separate the pylint+pep8 calling from one tool into two indivudal tools so
+   that they can be called independently, add make file sections that can be 
+   used to call these independently
+ - remove the support for the old style config that was previously located in
+   '/etc/ec2-init/ec2-config.cfg', no longer supported!
+ - instead of using a altered config parser that added its own 'dummy' section
+   on in the 'mcollective' module, use configobj which handles the parsing of
+   config without sections better (and it also maintains comments instead of
+   removing them)
+ - use the new defaulting config parser (that will not raise errors on sections
+   that do not exist or return errors when values are fetched that do not exist)
+   in the 'puppet' module
+ - for config 'modules' add in the ability for the module to provide a list of 
+   distro names which it is known to work with, if when ran and the distro being
+   used name does not match one of those in this list, a warning will be written
+   out saying that this module may not work correctly on this distrobution
+ - for all dynamically imported modules ensure that they are fixed up before 
+   they are used by ensuring that they have certain attributes, if they do not
+   have those attributes they will be set to a sensible set of defaults instead
+ - adjust all 'config' modules and handlers to use the adjusted util functions
+   and the new distro objects where applicable so that those pieces of code can 
+   benefit from the unified and enhanced functionality being provided in that
+   util module
+ - fix a potential bug whereby when a #includeonce was encountered it would
+   enable checking of urls against a cache, if later a #include was encountered
+   it would continue checking against that cache, instead of refetching (which
+   would likely be the expected case)
+ - add a openstack/nova based pep8 extension utility ('hacking.py') that allows
+   for custom checks (along with the standard pep8 checks) to occur when running
+   'make pep8' and its derivatives
 0.6.4:
  - support relative path in AuthorizedKeysFile (LP: #970071).
  - make apt-get update run with --quiet (suitable for logging) (LP: #1012613)
diff --git a/Makefile b/Makefile
index 0fc6c46b..a96d6b5b 100644
--- a/Makefile
+++ b/Makefile
@@ -1,14 +1,33 @@
+CWD=$(shell pwd)
+PY_FILES=$(shell find cloudinit bin -name "*.py")
+PY_FILES+="bin/cloud-init"
 
 all: test
 
+pep8:
+	$(CWD)/tools/run-pep8 $(PY_FILES)
+
 pylint:
-	pylint cloudinit
+	$(CWD)/tools/run-pylint $(PY_FILES)
 
 pyflakes:
-	pyflakes .
+	pyflakes $(PY_FILES)
 
 test:
-	nosetests tests/unittests/
+	nosetests $(noseopts) tests/unittests/
+
+2to3:
+	2to3 $(PY_FILES)
+
+clean:
+	rm -rf /var/log/cloud-init.log \
+		   /var/lib/cloud/
+
+rpm:
+	cd packages && ./brpm
+
+deb:
+	cd packages && ./bddeb
 
-.PHONY: test pylint pyflakes
+.PHONY: test pylint pyflakes 2to3 clean pep8 rpm deb
 
diff --git a/Requires b/Requires
new file mode 100644
index 00000000..10be0155
--- /dev/null
+++ b/Requires
@@ -0,0 +1,30 @@
+# Pypi requirements for cloud-init to work
+
+# Used for templating any files or strings that are considered 
+# to be templates, not cheetah since it pulls in alot of extra libs.
+# This one is pretty dinky and does want we want (var substituion)
+Tempita
+
+# This is used for any pretty printing of tabular data.
+PrettyTable
+
+# This one is currently only used by the MAAS datasource. If that
+# datasource is removed, this is no longer needed
+oauth
+
+# This is used to fetch the ec2 metadata into a easily
+# parseable format, instead of having to have cloud-init perform
+# those same fetchs and decodes and signing (...) that ec2 requires.
+boto
+
+# This is only needed for places where we need to support configs in a manner
+# that the built-in config parser is not sufficent (ie
+# when we need to preserve comments, or do not have a top-level
+# section)...
+configobj
+
+# All new style configurations are in the yaml format
+pyyaml
+
+# The new main entrypoint uses argparse instead of optparse
+argparse
diff --git a/TODO b/TODO
index 568bdb07..1725db00 100644
--- a/TODO
+++ b/TODO
@@ -1,14 +1,37 @@
-- consider 'failsafe' DataSource
+- Consider a 'failsafe' DataSource
   If all others fail, setting a default that
    - sets the user password, writing it to console
    - logs to console that this happened
-- consider 'previous' DataSource
+- Consider a 'previous' DataSource
   If no other data source is found, fall back to the 'previous' one
   keep a indication of what instance id that is in /var/lib/cloud
-- rewrite "cloud-init-query"
-  have DataSource and cloudinit expose explicit fields
+- Rewrite "cloud-init-query" (currently not implemented)
+  Possibly have DataSource and cloudinit expose explicit fields
    - instance-id
    - hostname
    - mirror
    - release
    - ssh public keys
+- Remove the conversion of the ubuntu network interface format conversion
+  to a RH/fedora format and replace it with a top level format that uses
+  the netcf libraries format instead (which itself knows how to translate
+  into the specific formats)
+- Replace the 'apt*' modules with variants that now use the distro classes
+  to perform distro independent packaging commands (where possible)
+- Canonicalize the semaphore/lock name for modules and user data handlers
+  a. It is most likely a bug that currently exists that if a module in config
+     alters its name and it has already ran, then it will get ran again since
+     the lock name hasn't be canonicalized
+- Replace some the LOG.debug calls with a LOG.info where appropriate instead
+  of how right now there is really only 2 levels (WARN and DEBUG)
+- Remove the 'cc_' for config modules, either have them fully specified (ie
+  'cloudinit.config.resizefs') or by default only look in the 'cloudinit.config'
+  for these modules (or have a combination of the above), this avoids having
+  to understand where your modules are coming from (which can be altered by
+  the current python inclusion path)
+- Depending on if people think the wrapper around 'os.path.join' provided
+  by the 'paths' object is useful (allowing us to modify based off a 'read'
+  and 'write' configuration based 'root') or is just to confusing, it might be 
+  something to remove later, and just recommend using 'chroot' instead (or the X 
+  different other options which are similar to 'chroot'), which is might be more 
+  natural and less confusing...
diff --git a/bin/cloud-init b/bin/cloud-init
new file mode 100755
index 00000000..c7863db1
--- /dev/null
+++ b/bin/cloud-init
@@ -0,0 +1,474 @@
+#!/usr/bin/python
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import os
+import sys
+import traceback
+
+# This is more just for running from the bin folder so that
+# cloud-init binary can find the cloudinit module
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(
+        sys.argv[0]), os.pardir, os.pardir))
+if os.path.exists(os.path.join(possible_topdir, "cloudinit", "__init__.py")):
+    sys.path.insert(0, possible_topdir)
+
+from cloudinit import log as logging
+from cloudinit import netinfo
+from cloudinit import sources
+from cloudinit import stages
+from cloudinit import templater
+from cloudinit import util
+from cloudinit import version
+
+from cloudinit.settings import (PER_INSTANCE, PER_ALWAYS, PER_ONCE,
+                                CLOUD_CONFIG)
+
+
+# Pretty little welcome message template
+WELCOME_MSG_TPL = ("Cloud-init v. {{version}} running '{{action}}' at "
+                   "{{timestamp}}. Up {{uptime}} seconds.")
+
+# Module section template
+MOD_SECTION_TPL = "cloud_%s_modules"
+
+# Things u can query on
+QUERY_DATA_TYPES = [
+    'data',
+    'data_raw',
+    'instance_id',
+]
+
+# Frequency shortname to full name
+# (so users don't have to remember the full name...)
+FREQ_SHORT_NAMES = {
+    'instance': PER_INSTANCE,
+    'always': PER_ALWAYS,
+    'once': PER_ONCE,
+}
+
+LOG = logging.getLogger()
+
+
+# Used for when a logger may not be active
+# and we still want to print exceptions...
+def print_exc(msg=''):
+    if msg:
+        sys.stderr.write("%s\n" % (msg))
+    sys.stderr.write('-' * 60)
+    sys.stderr.write("\n")
+    traceback.print_exc(file=sys.stderr)
+    sys.stderr.write('-' * 60)
+    sys.stderr.write("\n")
+
+
+def welcome(action):
+    tpl_params = {
+        'version': version.version_string(),
+        'uptime': util.uptime(),
+        'timestamp': util.time_rfc2822(),
+        'action': action,
+    }
+    tpl_msg = templater.render_string(WELCOME_MSG_TPL, tpl_params)
+    util.multi_log("%s\n" % (tpl_msg),
+                   console=False, stderr=True)
+
+
+def extract_fns(args):
+    # Files are already opened so lets just pass that along
+    # since it would of broke if it couldn't have
+    # read that file already...
+    fn_cfgs = []
+    if args.files:
+        for fh in args.files:
+            # The realpath is more useful in logging
+            # so lets resolve to that...
+            fn_cfgs.append(os.path.realpath(fh.name))
+    return fn_cfgs
+
+
+def run_module_section(mods, action_name, section):
+    full_section_name = MOD_SECTION_TPL % (section)
+    (which_ran, failures) = mods.run_section(full_section_name)
+    total_attempted = len(which_ran) + len(failures)
+    if total_attempted == 0:
+        msg = ("No '%s' modules to run"
+               " under section '%s'") % (action_name, full_section_name)
+        sys.stderr.write("%s\n" % (msg))
+        LOG.debug(msg)
+        return 0
+    else:
+        LOG.debug("Ran %s modules with %s failures",
+                  len(which_ran), len(failures))
+        return len(failures)
+
+
+def main_init(name, args):
+    deps = [sources.DEP_FILESYSTEM, sources.DEP_NETWORK]
+    if args.local:
+        deps = [sources.DEP_FILESYSTEM]
+
+    if not args.local:
+        # See doc/kernel-cmdline.txt
+        #
+        # This is used in maas datasource, in "ephemeral" (read-only root)
+        # environment where the instance netboots to iscsi ro root.
+        # and the entity that controls the pxe config has to configure
+        # the maas datasource.
+        #
+        # Could be used elsewhere, only works on network based (not local).
+        root_name = "%s.d" % (CLOUD_CONFIG)
+        target_fn = os.path.join(root_name, "91_kernel_cmdline_url.cfg")
+        util.read_write_cmdline_url(target_fn)
+
+    # Cloud-init 'init' stage is broken up into the following sub-stages
+    # 1. Ensure that the init object fetches its config without errors
+    # 2. Setup logging/output redirections with resultant config (if any)
+    # 3. Initialize the cloud-init filesystem
+    # 4. Check if we can stop early by looking for various files
+    # 5. Fetch the datasource
+    # 6. Connect to the current instance location + update the cache
+    # 7. Consume the userdata (handlers get activated here)
+    # 8. Construct the modules object
+    # 9. Adjust any subsequent logging/output redirections using
+    #    the modules objects configuration
+    # 10. Run the modules for the 'init' stage
+    # 11. Done!
+    welcome(name)
+    init = stages.Init(deps)
+    # Stage 1
+    init.read_cfg(extract_fns(args))
+    # Stage 2
+    outfmt = None
+    errfmt = None
+    try:
+        LOG.debug("Closing stdin")
+        util.close_stdin()
+        (outfmt, errfmt) = util.fixup_output(init.cfg, name)
+    except:
+        util.logexc(LOG, "Failed to setup output redirection!")
+        print_exc("Failed to setup output redirection!")
+    if args.debug:
+        # Reset so that all the debug handlers are closed out
+        LOG.debug(("Logging being reset, this logger may no"
+                    " longer be active shortly"))
+        logging.resetLogging()
+    logging.setupLogging(init.cfg)
+    # Stage 3
+    try:
+        init.initialize()
+    except Exception:
+        util.logexc(LOG, "Failed to initialize, likely bad things to come!")
+    # Stage 4
+    path_helper = init.paths
+    if not args.local:
+        sys.stderr.write("%s\n" % (netinfo.debug_info()))
+        LOG.debug(("Checking to see if files that we need already"
+                   " exist from a previous run that would allow us"
+                   " to stop early."))
+        stop_files = [
+            os.path.join(path_helper.get_cpath("data"), "no-net"),
+            path_helper.get_ipath_cur("obj_pkl"),
+        ]
+        existing_files = []
+        for fn in stop_files:
+            try:
+                c = util.load_file(fn)
+                if len(c):
+                    existing_files.append((fn, len(c)))
+            except Exception:
+                pass
+        if existing_files:
+            LOG.debug("Exiting early due to the existence of %s files",
+                      existing_files)
+            return 0
+    else:
+        # The cache is not instance specific, so it has to be purged
+        # but we want 'start' to benefit from a cache if
+        # a previous start-local populated one...
+        manual_clean = util.get_cfg_option_bool(init.cfg,
+                                                'manual_cache_clean', False)
+        if manual_clean:
+            LOG.debug("Not purging instance link, manual cleaning enabled")
+            init.purge_cache(False)
+        else:
+            init.purge_cache()
+        # Delete the non-net file as well
+        util.del_file(os.path.join(path_helper.get_cpath("data"), "no-net"))
+    # Stage 5
+    try:
+        init.fetch()
+    except sources.DataSourceNotFoundException:
+        util.logexc(LOG, ("No instance datasource found!"
+                          " Likely bad things to come!"))
+        # In the case of cloud-init (net mode) it is a bit
+        # more likely that the user would consider it
+        # failure if nothing was found. When using
+        # upstart it will also mentions job failure
+        # in console log if exit code is != 0.
+        if not args.force:
+            if args.local:
+                return 0
+            else:
+                return 1
+    # Stage 6
+    iid = init.instancify()
+    LOG.debug("%s will now be targeting instance id: %s", name, iid)
+    init.update()
+    # Stage 7
+    try:
+        # Attempt to consume the data per instance.
+        # This may run user-data handlers and/or perform
+        # url downloads and such as needed.
+        (ran, _results) = init.cloudify().run('consume_userdata',
+                                             init.consume_userdata,
+                                             args=[PER_INSTANCE],
+                                             freq=PER_INSTANCE)
+        if not ran:
+            # Just consume anything that is set to run per-always
+            # if nothing ran in the per-instance code
+            #
+            # See: https://bugs.launchpad.net/bugs/819507 for a little
+            # reason behind this...
+            init.consume_userdata(PER_ALWAYS)
+    except Exception:
+        util.logexc(LOG, "Consuming user data failed!")
+        return 1
+    # Stage 8 - TODO - do we really need to re-extract our configs?
+    mods = stages.Modules(init, extract_fns(args))
+    # Stage 9 - TODO is this really needed??
+    try:
+        outfmt_orig = outfmt
+        errfmt_orig = errfmt
+        (outfmt, errfmt) = util.get_output_cfg(mods.cfg, name)
+        if outfmt_orig != outfmt or errfmt_orig != errfmt:
+            LOG.warn("Stdout, stderr changing to (%s, %s)", outfmt, errfmt)
+            (outfmt, errfmt) = util.fixup_output(mods.cfg, name)
+    except:
+        util.logexc(LOG, "Failed to re-adjust output redirection!")
+    # Stage 10
+    return run_module_section(mods, name, name)
+
+
+def main_modules(action_name, args):
+    name = args.mode
+    # Cloud-init 'modules' stages are broken up into the following sub-stages
+    # 1. Ensure that the init object fetches its config without errors
+    # 2. Get the datasource from the init object, if it does
+    #    not exist then that means the main_init stage never
+    #    worked, and thus this stage can not run.
+    # 3. Construct the modules object
+    # 4. Adjust any subsequent logging/output redirections using
+    #    the modules objects configuration
+    # 5. Run the modules for the given stage name
+    # 6. Done!
+    welcome("%s:%s" % (action_name, name))
+    init = stages.Init(ds_deps=[])
+    # Stage 1
+    init.read_cfg(extract_fns(args))
+    # Stage 2
+    try:
+        init.fetch()
+    except sources.DataSourceNotFoundException:
+        # There was no datasource found, theres nothing to do
+        util.logexc(LOG, ('Can not apply stage %s, '
+                          'no datasource found!'
+                          " Likely bad things to come!"), name)
+        print_exc(('Can not apply stage %s, '
+                   'no datasource found!'
+                   " Likely bad things to come!") % (name))
+        if not args.force:
+            return 1
+    # Stage 3
+    mods = stages.Modules(init, extract_fns(args))
+    # Stage 4
+    try:
+        LOG.debug("Closing stdin")
+        util.close_stdin()
+        util.fixup_output(mods.cfg, name)
+    except:
+        util.logexc(LOG, "Failed to setup output redirection!")
+    if args.debug:
+        # Reset so that all the debug handlers are closed out
+        LOG.debug(("Logging being reset, this logger may no"
+                    " longer be active shortly"))
+        logging.resetLogging()
+    logging.setupLogging(mods.cfg)
+    # Stage 5
+    return run_module_section(mods, name, name)
+
+
+def main_query(name, _args):
+    raise NotImplementedError(("Action '%s' is not"
+                               " currently implemented") % (name))
+
+
+def main_single(name, args):
+    # Cloud-init single stage is broken up into the following sub-stages
+    # 1. Ensure that the init object fetches its config without errors
+    # 2. Attempt to fetch the datasource (warn if it doesn't work)
+    # 3. Construct the modules object
+    # 4. Adjust any subsequent logging/output redirections using
+    #    the modules objects configuration
+    # 5. Run the single module
+    # 6. Done!
+    mod_name = args.name
+    welcome("%s:%s" % (name, mod_name))
+    init = stages.Init(ds_deps=[])
+    # Stage 1
+    init.read_cfg(extract_fns(args))
+    # Stage 2
+    try:
+        init.fetch()
+    except sources.DataSourceNotFoundException:
+        # There was no datasource found,
+        # that might be bad (or ok) depending on
+        # the module being ran (so continue on)
+        util.logexc(LOG, ("Failed to fetch your datasource,"
+                          " likely bad things to come!"))
+        print_exc(("Failed to fetch your datasource,"
+                   " likely bad things to come!"))
+        if not args.force:
+            return 1
+    # Stage 3
+    mods = stages.Modules(init, extract_fns(args))
+    mod_args = args.module_args
+    if mod_args:
+        LOG.debug("Using passed in arguments %s", mod_args)
+    mod_freq = args.frequency
+    if mod_freq:
+        LOG.debug("Using passed in frequency %s", mod_freq)
+        mod_freq = FREQ_SHORT_NAMES.get(mod_freq)
+    # Stage 4
+    try:
+        LOG.debug("Closing stdin")
+        util.close_stdin()
+        util.fixup_output(mods.cfg, None)
+    except:
+        util.logexc(LOG, "Failed to setup output redirection!")
+    if args.debug:
+        # Reset so that all the debug handlers are closed out
+        LOG.debug(("Logging being reset, this logger may no"
+                   " longer be active shortly"))
+        logging.resetLogging()
+    logging.setupLogging(mods.cfg)
+    # Stage 5
+    (which_ran, failures) = mods.run_single(mod_name,
+                                            mod_args,
+                                            mod_freq)
+    if failures:
+        LOG.warn("Ran %s but it failed!", mod_name)
+        return 1
+    elif not which_ran:
+        LOG.warn("Did not run %s, does it exist?", mod_name)
+        return 1
+    else:
+        # Guess it worked
+        return 0
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    # Top level args
+    parser.add_argument('--version', '-v', action='version',
+                        version='%(prog)s ' + (version.version_string()))
+    parser.add_argument('--file', '-f', action='append',
+                        dest='files',
+                        help=('additional yaml configuration'
+                              ' files to use'),
+                        type=argparse.FileType('rb'))
+    parser.add_argument('--debug', '-d', action='store_true',
+                        help=('show additional pre-action'
+                              ' logging (default: %(default)s)'),
+                        default=False)
+    parser.add_argument('--force', action='store_true',
+                        help=('force running even if no datasource is'
+                              ' found (use at your own risk)'),
+                        dest='force',
+                        default=False)
+    subparsers = parser.add_subparsers()
+
+    # Each action and its sub-options (if any)
+    parser_init = subparsers.add_parser('init',
+                                        help=('initializes cloud-init and'
+                                              ' performs initial modules'))
+    parser_init.add_argument("--local", '-l', action='store_true',
+                             help="start in local mode (default: %(default)s)",
+                             default=False)
+    # This is used so that we can know which action is selected +
+    # the functor to use to run this subcommand
+    parser_init.set_defaults(action=('init', main_init))
+
+    # These settings are used for the 'config' and 'final' stages
+    parser_mod = subparsers.add_parser('modules',
+                                      help=('activates modules '
+                                            'using a given configuration key'))
+    parser_mod.add_argument("--mode", '-m', action='store',
+                             help=("module configuration name "
+                                    "to use (default: %(default)s)"),
+                             default='config',
+                             choices=('init', 'config', 'final'))
+    parser_mod.set_defaults(action=('modules', main_modules))
+
+    # These settings are used when you want to query information
+    # stored in the cloud-init data objects/directories/files
+    parser_query = subparsers.add_parser('query',
+                                         help=('query information stored '
+                                               'in cloud-init'))
+    parser_query.add_argument("--name", '-n', action="store",
+                              help="item name to query on",
+                              required=True,
+                              choices=QUERY_DATA_TYPES)
+    parser_query.set_defaults(action=('query', main_query))
+
+    # This subcommand allows you to run a single module
+    parser_single = subparsers.add_parser('single',
+                                         help=('run a single module '))
+    parser_single.set_defaults(action=('single', main_single))
+    parser_single.add_argument("--name", '-n', action="store",
+                              help="module name to run",
+                              required=True)
+    parser_single.add_argument("--frequency", action="store",
+                              help=("frequency of the module"),
+                              required=False,
+                              choices=list(FREQ_SHORT_NAMES.keys()))
+    parser_single.add_argument("module_args", nargs="*",
+                              metavar='argument',
+                              help=('any additional arguments to'
+                                    ' pass to this module'))
+    parser_single.set_defaults(action=('single', main_single))
+
+    args = parser.parse_args()
+
+    # Setup basic logging to start (until reinitialized)
+    # iff in debug mode...
+    if args.debug:
+        logging.setupBasicLogging()
+
+    (name, functor) = args.action
+    return functor(name, args)
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/cloud-init-cfg.py b/cloud-init-cfg.py
deleted file mode 100755
index 3a475c1c..00000000
--- a/cloud-init-cfg.py
+++ /dev/null
@@ -1,115 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import sys
-import cloudinit
-import cloudinit.util as util
-import cloudinit.CloudConfig as CC
-import logging
-import os
-
-
-def Usage(out=sys.stdout):
-    out.write("Usage: %s name\n" % sys.argv[0])
-
-
-def main():
-    # expect to be called with
-    #   name [ freq [ args ]
-    #   run the cloud-config job 'name' at with given args
-    # or
-    #   read cloud config jobs from config (builtin -> system)
-    #   and run all in order
-
-    util.close_stdin()
-
-    modename = "config"
-
-    if len(sys.argv) < 2:
-        Usage(sys.stderr)
-        sys.exit(1)
-    if sys.argv[1] == "all":
-        name = "all"
-        if len(sys.argv) > 2:
-            modename = sys.argv[2]
-    else:
-        freq = None
-        run_args = []
-        name = sys.argv[1]
-        if len(sys.argv) > 2:
-            freq = sys.argv[2]
-            if freq == "None":
-                freq = None
-        if len(sys.argv) > 3:
-            run_args = sys.argv[3:]
-
-    cfg_path = cloudinit.get_ipath_cur("cloud_config")
-    cfg_env_name = cloudinit.cfg_env_name
-    if cfg_env_name in os.environ:
-        cfg_path = os.environ[cfg_env_name]
-
-    cloud = cloudinit.CloudInit(ds_deps=[])  # ds_deps=[], get only cached
-    try:
-        cloud.get_data_source()
-    except cloudinit.DataSourceNotFoundException as e:
-        # there was no datasource found, theres nothing to do
-        sys.exit(0)
-
-    cc = CC.CloudConfig(cfg_path, cloud)
-
-    try:
-        (outfmt, errfmt) = CC.get_output_cfg(cc.cfg, modename)
-        CC.redirect_output(outfmt, errfmt)
-    except Exception as e:
-        err("Failed to get and set output config: %s\n" % e)
-
-    cloudinit.logging_set_from_cfg(cc.cfg)
-    log = logging.getLogger()
-    log.info("cloud-init-cfg %s" % sys.argv[1:])
-
-    module_list = []
-    if name == "all":
-        modlist_cfg_name = "cloud_%s_modules" % modename
-        module_list = CC.read_cc_modules(cc.cfg, modlist_cfg_name)
-        if not len(module_list):
-            err("no modules to run in cloud_config [%s]" % modename, log)
-            sys.exit(0)
-    else:
-        module_list.append([name, freq] + run_args)
-
-    failures = CC.run_cc_modules(cc, module_list, log)
-    if len(failures):
-        err("errors running cloud_config [%s]: %s" % (modename, failures), log)
-    sys.exit(len(failures))
-
-
-def err(msg, log=None):
-    if log:
-        log.error(msg)
-    sys.stderr.write(msg + "\n")
-
-
-def fail(msg, log=None):
-    err(msg, log)
-    sys.exit(1)
-
-if __name__ == '__main__':
-    main()
diff --git a/cloud-init-query.py b/cloud-init-query.py
deleted file mode 100755
index 856cf462..00000000
--- a/cloud-init-query.py
+++ /dev/null
@@ -1,56 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import sys
-import cloudinit
-import cloudinit.CloudConfig
-
-
-def Usage(out=sys.stdout):
-    out.write("Usage: %s name\n" % sys.argv[0])
-
-
-def main():
-    # expect to be called with name of item to fetch
-    if len(sys.argv) != 2:
-        Usage(sys.stderr)
-        sys.exit(1)
-
-    cfg_path = cloudinit.get_ipath_cur("cloud_config")
-    cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
-    data = {
-        'user_data': cc.cloud.get_userdata(),
-        'user_data_raw': cc.cloud.get_userdata_raw(),
-        'instance_id': cc.cloud.get_instance_id(),
-    }
-
-    name = sys.argv[1].replace('-', '_')
-
-    if name not in data:
-        sys.stderr.write("unknown name '%s'.  Known values are:\n  %s\n" %
-            (sys.argv[1], ' '.join(data.keys())))
-        sys.exit(1)
-
-    print data[name]
-    sys.exit(0)
-
-if __name__ == '__main__':
-    main()
diff --git a/cloud-init.py b/cloud-init.py
deleted file mode 100755
index 2acea3f8..00000000
--- a/cloud-init.py
+++ /dev/null
@@ -1,229 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import subprocess
-import sys
-
-import cloudinit
-import cloudinit.util as util
-import cloudinit.CloudConfig as CC
-import cloudinit.DataSource as ds
-import cloudinit.netinfo as netinfo
-import time
-import traceback
-import logging
-import errno
-import os
-
-
-def warn(wstr):
-    sys.stderr.write("WARN:%s" % wstr)
-
-
-def main():
-    util.close_stdin()
-
-    cmds = ("start", "start-local")
-    deps = {"start": (ds.DEP_FILESYSTEM, ds.DEP_NETWORK),
-            "start-local": (ds.DEP_FILESYSTEM, )}
-
-    cmd = ""
-    if len(sys.argv) > 1:
-        cmd = sys.argv[1]
-
-    cfg_path = None
-    if len(sys.argv) > 2:
-        # this is really for debugging only
-        # but you can invoke on development system with ./config/cloud.cfg
-        cfg_path = sys.argv[2]
-
-    if not cmd in cmds:
-        sys.stderr.write("bad command %s. use one of %s\n" % (cmd, cmds))
-        sys.exit(1)
-
-    now = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.gmtime())
-    try:
-        uptimef = open("/proc/uptime")
-        uptime = uptimef.read().split(" ")[0]
-        uptimef.close()
-    except IOError as e:
-        warn("unable to open /proc/uptime\n")
-        uptime = "na"
-
-    cmdline_msg = None
-    cmdline_exc = None
-    if cmd == "start":
-        target = "%s.d/%s" % (cloudinit.system_config,
-            "91_kernel_cmdline_url.cfg")
-        if os.path.exists(target):
-            cmdline_msg = "cmdline: %s existed" % target
-        else:
-            cmdline = util.get_cmdline()
-            try:
-                (key, url, content) = cloudinit.get_cmdline_url(
-                    cmdline=cmdline)
-                if key and content:
-                    util.write_file(target, content, mode=0600)
-                    cmdline_msg = ("cmdline: wrote %s from %s, %s" %
-                        (target, key, url))
-                elif key:
-                    cmdline_msg = ("cmdline: %s, %s had no cloud-config" %
-                        (key, url))
-            except Exception:
-                cmdline_exc = ("cmdline: '%s' raised exception\n%s" %
-                    (cmdline, traceback.format_exc()))
-                warn(cmdline_exc)
-
-    try:
-        cfg = cloudinit.get_base_cfg(cfg_path)
-    except Exception as e:
-        warn("Failed to get base config. falling back to builtin: %s\n" % e)
-        try:
-            cfg = cloudinit.get_builtin_cfg()
-        except Exception as e:
-            warn("Unable to load builtin config\n")
-            raise
-
-    try:
-        (outfmt, errfmt) = CC.get_output_cfg(cfg, "init")
-        CC.redirect_output(outfmt, errfmt)
-    except Exception as e:
-        warn("Failed to get and set output config: %s\n" % e)
-
-    cloudinit.logging_set_from_cfg(cfg)
-    log = logging.getLogger()
-
-    if cmdline_exc:
-        log.debug(cmdline_exc)
-    elif cmdline_msg:
-        log.debug(cmdline_msg)
-
-    try:
-        cloudinit.initfs()
-    except Exception as e:
-        warn("failed to initfs, likely bad things to come: %s\n" % str(e))
-
-    nonet_path = "%s/%s" % (cloudinit.get_cpath("data"), "no-net")
-
-    if cmd == "start":
-        print netinfo.debug_info()
-
-        stop_files = (cloudinit.get_ipath_cur("obj_pkl"), nonet_path)
-        # if starting as the network start, there are cases
-        # where everything is already done for us, and it makes
-        # most sense to exit early and silently
-        for f in stop_files:
-            try:
-                fp = open(f, "r")
-                fp.close()
-            except:
-                continue
-
-            log.debug("no need for cloud-init start to run (%s)\n", f)
-            sys.exit(0)
-    elif cmd == "start-local":
-        # cache is not instance specific, so it has to be purged
-        # but we want 'start' to benefit from a cache if
-        # a previous start-local populated one
-        manclean = util.get_cfg_option_bool(cfg, 'manual_cache_clean', False)
-        if manclean:
-            log.debug("not purging cache, manual_cache_clean = True")
-        cloudinit.purge_cache(not manclean)
-
-        try:
-            os.unlink(nonet_path)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise
-
-    msg = "cloud-init %s running: %s. up %s seconds" % (cmd, now, uptime)
-    sys.stderr.write(msg + "\n")
-    sys.stderr.flush()
-
-    log.info(msg)
-
-    cloud = cloudinit.CloudInit(ds_deps=deps[cmd])
-
-    try:
-        cloud.get_data_source()
-    except cloudinit.DataSourceNotFoundException as e:
-        sys.stderr.write("no instance data found in %s\n" % cmd)
-        sys.exit(0)
-
-    # set this as the current instance
-    cloud.set_cur_instance()
-
-    # store the metadata
-    cloud.update_cache()
-
-    msg = "found data source: %s" % cloud.datasource
-    sys.stderr.write(msg + "\n")
-    log.debug(msg)
-
-    # parse the user data (ec2-run-userdata.py)
-    try:
-        ran = cloud.sem_and_run("consume_userdata", cloudinit.per_instance,
-            cloud.consume_userdata, [cloudinit.per_instance], False)
-        if not ran:
-            cloud.consume_userdata(cloudinit.per_always)
-    except:
-        warn("consuming user data failed!\n")
-        raise
-
-    cfg_path = cloudinit.get_ipath_cur("cloud_config")
-    cc = CC.CloudConfig(cfg_path, cloud)
-
-    # if the output config changed, update output and err
-    try:
-        outfmt_orig = outfmt
-        errfmt_orig = errfmt
-        (outfmt, errfmt) = CC.get_output_cfg(cc.cfg, "init")
-        if outfmt_orig != outfmt or errfmt_orig != errfmt:
-            warn("stdout, stderr changing to (%s,%s)" % (outfmt, errfmt))
-            CC.redirect_output(outfmt, errfmt)
-    except Exception as e:
-        warn("Failed to get and set output config: %s\n" % e)
-
-    # send the cloud-config ready event
-    cc_path = cloudinit.get_ipath_cur('cloud_config')
-    cc_ready = cc.cfg.get("cc_ready_cmd",
-        ['initctl', 'emit', 'cloud-config',
-         '%s=%s' % (cloudinit.cfg_env_name, cc_path)])
-    if cc_ready:
-        if isinstance(cc_ready, str):
-            cc_ready = ['sh', '-c', cc_ready]
-        subprocess.Popen(cc_ready).communicate()
-
-    module_list = CC.read_cc_modules(cc.cfg, "cloud_init_modules")
-
-    failures = []
-    if len(module_list):
-        failures = CC.run_cc_modules(cc, module_list, log)
-    else:
-        msg = "no cloud_init_modules to run"
-        sys.stderr.write(msg + "\n")
-        log.debug(msg)
-        sys.exit(0)
-
-    sys.exit(len(failures))
-
-if __name__ == '__main__':
-    main()
diff --git a/cloudinit/CloudConfig/__init__.py b/cloudinit/CloudConfig/__init__.py
deleted file mode 100644
index d2d1035a..00000000
--- a/cloudinit/CloudConfig/__init__.py
+++ /dev/null
@@ -1,274 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2008-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Chuck Short <chuck.short@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-import yaml
-import cloudinit
-import cloudinit.util as util
-import sys
-import traceback
-import os
-import subprocess
-import time
-
-per_instance = cloudinit.per_instance
-per_always = cloudinit.per_always
-per_once = cloudinit.per_once
-
-
-class CloudConfig():
-    cfgfile = None
-    cfg = None
-
-    def __init__(self, cfgfile, cloud=None, ds_deps=None):
-        if cloud == None:
-            self.cloud = cloudinit.CloudInit(ds_deps)
-            self.cloud.get_data_source()
-        else:
-            self.cloud = cloud
-        self.cfg = self.get_config_obj(cfgfile)
-
-    def get_config_obj(self, cfgfile):
-        try:
-            cfg = util.read_conf(cfgfile)
-        except:
-            # TODO: this 'log' could/should be passed in
-            cloudinit.log.critical("Failed loading of cloud config '%s'. "
-                                   "Continuing with empty config\n" % cfgfile)
-            cloudinit.log.debug(traceback.format_exc() + "\n")
-            cfg = None
-        if cfg is None:
-            cfg = {}
-
-        try:
-            ds_cfg = self.cloud.datasource.get_config_obj()
-        except:
-            ds_cfg = {}
-
-        cfg = util.mergedict(cfg, ds_cfg)
-        return(util.mergedict(cfg, self.cloud.cfg))
-
-    def handle(self, name, args, freq=None):
-        try:
-            mod = __import__("cc_" + name.replace("-", "_"), globals())
-            def_freq = getattr(mod, "frequency", per_instance)
-            handler = getattr(mod, "handle")
-
-            if not freq:
-                freq = def_freq
-
-            self.cloud.sem_and_run("config-" + name, freq, handler,
-                [name, self.cfg, self.cloud, cloudinit.log, args])
-        except:
-            raise
-
-
-# reads a cloudconfig module list, returns
-# a 2 dimensional array suitable to pass to run_cc_modules
-def read_cc_modules(cfg, name):
-    if name not in cfg:
-        return([])
-    module_list = []
-    # create 'module_list', an array of arrays
-    # where array[0] = config
-    #       array[1] = freq
-    #       array[2:] = arguemnts
-    for item in cfg[name]:
-        if isinstance(item, str):
-            module_list.append((item,))
-        elif isinstance(item, list):
-            module_list.append(item)
-        else:
-            raise TypeError("failed to read '%s' item in config")
-    return(module_list)
-
-
-def run_cc_modules(cc, module_list, log):
-    failures = []
-    for cfg_mod in module_list:
-        name = cfg_mod[0]
-        freq = None
-        run_args = []
-        if len(cfg_mod) > 1:
-            freq = cfg_mod[1]
-        if len(cfg_mod) > 2:
-            run_args = cfg_mod[2:]
-
-        try:
-            log.debug("handling %s with freq=%s and args=%s" %
-                (name, freq, run_args))
-            cc.handle(name, run_args, freq=freq)
-        except:
-            log.warn(traceback.format_exc())
-            log.error("config handling of %s, %s, %s failed\n" %
-                (name, freq, run_args))
-            failures.append(name)
-
-    return(failures)
-
-
-# always returns well formated values
-# cfg is expected to have an entry 'output' in it, which is a dictionary
-# that includes entries for 'init', 'config', 'final' or 'all'
-#   init: /var/log/cloud.out
-#   config: [ ">> /var/log/cloud-config.out", /var/log/cloud-config.err ]
-#   final:
-#     output: "| logger -p"
-#     error: "> /dev/null"
-# this returns the specific 'mode' entry, cleanly formatted, with value
-# None if if none is given
-def get_output_cfg(cfg, mode="init"):
-    ret = [None, None]
-    if not 'output' in cfg:
-        return ret
-
-    outcfg = cfg['output']
-    if mode in outcfg:
-        modecfg = outcfg[mode]
-    else:
-        if 'all' not in outcfg:
-            return ret
-        # if there is a 'all' item in the output list
-        # then it applies to all users of this (init, config, final)
-        modecfg = outcfg['all']
-
-    # if value is a string, it specifies stdout and stderr
-    if isinstance(modecfg, str):
-        ret = [modecfg, modecfg]
-
-    # if its a list, then we expect (stdout, stderr)
-    if isinstance(modecfg, list):
-        if len(modecfg) > 0:
-            ret[0] = modecfg[0]
-        if len(modecfg) > 1:
-            ret[1] = modecfg[1]
-
-    # if it is a dictionary, expect 'out' and 'error'
-    # items, which indicate out and error
-    if isinstance(modecfg, dict):
-        if 'output' in modecfg:
-            ret[0] = modecfg['output']
-        if 'error' in modecfg:
-            ret[1] = modecfg['error']
-
-    # if err's entry == "&1", then make it same as stdout
-    # as in shell syntax of "echo foo >/dev/null 2>&1"
-    if ret[1] == "&1":
-        ret[1] = ret[0]
-
-    swlist = [">>", ">", "|"]
-    for i in range(len(ret)):
-        if not ret[i]:
-            continue
-        val = ret[i].lstrip()
-        found = False
-        for s in swlist:
-            if val.startswith(s):
-                val = "%s %s" % (s, val[len(s):].strip())
-                found = True
-                break
-        if not found:
-            # default behavior is append
-            val = "%s %s" % (">>", val.strip())
-        ret[i] = val
-
-    return(ret)
-
-
-# redirect_output(outfmt, errfmt, orig_out, orig_err)
-#  replace orig_out and orig_err with filehandles specified in outfmt or errfmt
-#  fmt can be:
-#   > FILEPATH
-#   >> FILEPATH
-#   | program [ arg1 [ arg2 [ ... ] ] ]
-#
-#   with a '|', arguments are passed to shell, so one level of
-#   shell escape is required.
-def redirect_output(outfmt, errfmt, o_out=sys.stdout, o_err=sys.stderr):
-    if outfmt:
-        (mode, arg) = outfmt.split(" ", 1)
-        if mode == ">" or mode == ">>":
-            owith = "ab"
-            if mode == ">":
-                owith = "wb"
-            new_fp = open(arg, owith)
-        elif mode == "|":
-            proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
-            new_fp = proc.stdin
-        else:
-            raise TypeError("invalid type for outfmt: %s" % outfmt)
-
-        if o_out:
-            os.dup2(new_fp.fileno(), o_out.fileno())
-        if errfmt == outfmt:
-            os.dup2(new_fp.fileno(), o_err.fileno())
-            return
-
-    if errfmt:
-        (mode, arg) = errfmt.split(" ", 1)
-        if mode == ">" or mode == ">>":
-            owith = "ab"
-            if mode == ">":
-                owith = "wb"
-            new_fp = open(arg, owith)
-        elif mode == "|":
-            proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
-            new_fp = proc.stdin
-        else:
-            raise TypeError("invalid type for outfmt: %s" % outfmt)
-
-        if o_err:
-            os.dup2(new_fp.fileno(), o_err.fileno())
-    return
-
-
-def run_per_instance(name, func, args, clear_on_fail=False):
-    semfile = "%s/%s" % (cloudinit.get_ipath_cur("data"), name)
-    if os.path.exists(semfile):
-        return
-
-    util.write_file(semfile, str(time.time()))
-    try:
-        func(*args)
-    except:
-        if clear_on_fail:
-            os.unlink(semfile)
-        raise
-
-
-# apt_get top level command (install, update...), and args to pass it
-def apt_get(tlc, args=None):
-    if args is None:
-        args = []
-    e = os.environ.copy()
-    e['DEBIAN_FRONTEND'] = 'noninteractive'
-    cmd = ['apt-get', '--option', 'Dpkg::Options::=--force-confold',
-           '--assume-yes', '--quiet', tlc]
-    cmd.extend(args)
-    subprocess.check_call(cmd, env=e)
-
-
-def update_package_sources():
-    run_per_instance("update-sources", apt_get, ("update",))
-
-
-def install_packages(pkglist):
-    update_package_sources()
-    apt_get("install", pkglist)
diff --git a/cloudinit/CloudConfig/cc_apt_pipelining.py b/cloudinit/CloudConfig/cc_apt_pipelining.py
deleted file mode 100644
index 0286a9ae..00000000
--- a/cloudinit/CloudConfig/cc_apt_pipelining.py
+++ /dev/null
@@ -1,53 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#
-#    Author: Ben Howard <ben.howard@canonical.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-
-frequency = per_instance
-default_file = "/etc/apt/apt.conf.d/90cloud-init-pipelining"
-
-
-def handle(_name, cfg, _cloud, log, _args):
-
-    apt_pipe_value = util.get_cfg_option_str(cfg, "apt_pipelining", False)
-    apt_pipe_value = str(apt_pipe_value).lower()
-
-    if apt_pipe_value == "false":
-        write_apt_snippet("0", log)
-
-    elif apt_pipe_value in ("none", "unchanged", "os"):
-        return
-
-    elif apt_pipe_value in str(range(0, 6)):
-        write_apt_snippet(apt_pipe_value, log)
-
-    else:
-        log.warn("Invalid option for apt_pipeling: %s" % apt_pipe_value)
-
-
-def write_apt_snippet(setting, log, f_name=default_file):
-    """ Writes f_name with apt pipeline depth 'setting' """
-
-    acquire_pipeline_depth = 'Acquire::http::Pipeline-Depth "%s";\n'
-    file_contents = ("//Written by cloud-init per 'apt_pipelining'\n"
-                     + (acquire_pipeline_depth % setting))
-
-    util.write_file(f_name, file_contents)
-
-    log.debug("Wrote %s with APT pipeline setting" % f_name)
diff --git a/cloudinit/CloudConfig/cc_apt_update_upgrade.py b/cloudinit/CloudConfig/cc_apt_update_upgrade.py
deleted file mode 100644
index a7049bce..00000000
--- a/cloudinit/CloudConfig/cc_apt_update_upgrade.py
+++ /dev/null
@@ -1,241 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-import subprocess
-import traceback
-import os
-import glob
-import cloudinit.CloudConfig as cc
-
-
-def handle(_name, cfg, cloud, log, _args):
-    update = util.get_cfg_option_bool(cfg, 'apt_update', False)
-    upgrade = util.get_cfg_option_bool(cfg, 'apt_upgrade', False)
-
-    release = get_release()
-
-    mirror = find_apt_mirror(cloud, cfg)
-
-    log.debug("selected mirror at: %s" % mirror)
-
-    if not util.get_cfg_option_bool(cfg, \
-        'apt_preserve_sources_list', False):
-        generate_sources_list(release, mirror)
-        old_mir = util.get_cfg_option_str(cfg, 'apt_old_mirror', \
-            "archive.ubuntu.com/ubuntu")
-        rename_apt_lists(old_mir, mirror)
-
-    # set up proxy
-    proxy = cfg.get("apt_proxy", None)
-    proxy_filename = "/etc/apt/apt.conf.d/95cloud-init-proxy"
-    if proxy:
-        try:
-            contents = "Acquire::HTTP::Proxy \"%s\";\n"
-            with open(proxy_filename, "w") as fp:
-                fp.write(contents % proxy)
-        except Exception as e:
-            log.warn("Failed to write proxy to %s" % proxy_filename)
-    elif os.path.isfile(proxy_filename):
-        os.unlink(proxy_filename)
-
-    # process 'apt_sources'
-    if 'apt_sources' in cfg:
-        errors = add_sources(cfg['apt_sources'],
-                             {'MIRROR': mirror, 'RELEASE': release})
-        for e in errors:
-            log.warn("Source Error: %s\n" % ':'.join(e))
-
-    dconf_sel = util.get_cfg_option_str(cfg, 'debconf_selections', False)
-    if dconf_sel:
-        log.debug("setting debconf selections per cloud config")
-        try:
-            util.subp(('debconf-set-selections', '-'), dconf_sel)
-        except:
-            log.error("Failed to run debconf-set-selections")
-            log.debug(traceback.format_exc())
-
-    pkglist = util.get_cfg_option_list_or_str(cfg, 'packages', [])
-
-    errors = []
-    if update or len(pkglist) or upgrade:
-        try:
-            cc.update_package_sources()
-        except subprocess.CalledProcessError as e:
-            log.warn("apt-get update failed")
-            log.debug(traceback.format_exc())
-            errors.append(e)
-
-    if upgrade:
-        try:
-            cc.apt_get("upgrade")
-        except subprocess.CalledProcessError as e:
-            log.warn("apt upgrade failed")
-            log.debug(traceback.format_exc())
-            errors.append(e)
-
-    if len(pkglist):
-        try:
-            cc.install_packages(pkglist)
-        except subprocess.CalledProcessError as e:
-            log.warn("Failed to install packages: %s " % pkglist)
-            log.debug(traceback.format_exc())
-            errors.append(e)
-
-    if len(errors):
-        raise errors[0]
-
-    return(True)
-
-
-def mirror2lists_fileprefix(mirror):
-    string = mirror
-    # take of http:// or ftp://
-    if string.endswith("/"):
-        string = string[0:-1]
-    pos = string.find("://")
-    if pos >= 0:
-        string = string[pos + 3:]
-    string = string.replace("/", "_")
-    return string
-
-
-def rename_apt_lists(omirror, new_mirror, lists_d="/var/lib/apt/lists"):
-    oprefix = "%s/%s" % (lists_d, mirror2lists_fileprefix(omirror))
-    nprefix = "%s/%s" % (lists_d, mirror2lists_fileprefix(new_mirror))
-    if(oprefix == nprefix):
-        return
-    olen = len(oprefix)
-    for filename in glob.glob("%s_*" % oprefix):
-        os.rename(filename, "%s%s" % (nprefix, filename[olen:]))
-
-
-def get_release():
-    stdout, _stderr = subprocess.Popen(['lsb_release', '-cs'],
-                                       stdout=subprocess.PIPE).communicate()
-    return(str(stdout).strip())
-
-
-def generate_sources_list(codename, mirror):
-    util.render_to_file('sources.list', '/etc/apt/sources.list', \
-        {'mirror': mirror, 'codename': codename})
-
-
-def add_sources(srclist, searchList=None):
-    """
-    add entries in /etc/apt/sources.list.d for each abbreviated
-    sources.list entry in 'srclist'.  When rendering template, also
-    include the values in dictionary searchList
-    """
-    if searchList is None:
-        searchList = {}
-    elst = []
-
-    for ent in srclist:
-        if 'source' not in ent:
-            elst.append(["", "missing source"])
-            continue
-
-        source = ent['source']
-        if source.startswith("ppa:"):
-            try:
-                util.subp(["add-apt-repository", source])
-            except:
-                elst.append([source, "add-apt-repository failed"])
-            continue
-
-        source = util.render_string(source, searchList)
-
-        if 'filename' not in ent:
-            ent['filename'] = 'cloud_config_sources.list'
-
-        if not ent['filename'].startswith("/"):
-            ent['filename'] = "%s/%s" % \
-                ("/etc/apt/sources.list.d/", ent['filename'])
-
-        if ('keyid' in ent and 'key' not in ent):
-            ks = "keyserver.ubuntu.com"
-            if 'keyserver' in ent:
-                ks = ent['keyserver']
-            try:
-                ent['key'] = util.getkeybyid(ent['keyid'], ks)
-            except:
-                elst.append([source, "failed to get key from %s" % ks])
-                continue
-
-        if 'key' in ent:
-            try:
-                util.subp(('apt-key', 'add', '-'), ent['key'])
-            except:
-                elst.append([source, "failed add key"])
-
-        try:
-            util.write_file(ent['filename'], source + "\n", omode="ab")
-        except:
-            elst.append([source, "failed write to file %s" % ent['filename']])
-
-    return(elst)
-
-
-def find_apt_mirror(cloud, cfg):
-    """ find an apt_mirror given the cloud and cfg provided """
-
-    # TODO: distro and defaults should be configurable
-    distro = "ubuntu"
-    defaults = {
-        'ubuntu': "http://archive.ubuntu.com/ubuntu",
-        'debian': "http://archive.debian.org/debian",
-    }
-    mirror = None
-
-    cfg_mirror = cfg.get("apt_mirror", None)
-    if cfg_mirror:
-        mirror = cfg["apt_mirror"]
-    elif "apt_mirror_search" in cfg:
-        mirror = util.search_for_mirror(cfg['apt_mirror_search'])
-    else:
-        if cloud:
-            mirror = cloud.get_mirror()
-
-        mydom = ""
-
-        doms = []
-
-        if not mirror and cloud:
-            # if we have a fqdn, then search its domain portion first
-            (_hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
-            mydom = ".".join(fqdn.split(".")[1:])
-            if mydom:
-                doms.append(".%s" % mydom)
-
-        if not mirror:
-            doms.extend((".localdomain", "",))
-
-            mirror_list = []
-            mirrorfmt = "http://%s-mirror%s/%s" % (distro, "%s", distro)
-            for post in doms:
-                mirror_list.append(mirrorfmt % post)
-
-            mirror = util.search_for_mirror(mirror_list)
-
-    if not mirror:
-        mirror = defaults[distro]
-
-    return mirror
diff --git a/cloudinit/CloudConfig/cc_bootcmd.py b/cloudinit/CloudConfig/cc_bootcmd.py
deleted file mode 100644
index f584da02..00000000
--- a/cloudinit/CloudConfig/cc_bootcmd.py
+++ /dev/null
@@ -1,48 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-import cloudinit.util as util
-import subprocess
-import tempfile
-import os
-from cloudinit.CloudConfig import per_always
-frequency = per_always
-
-
-def handle(_name, cfg, cloud, log, _args):
-    if "bootcmd" not in cfg:
-        return
-
-    try:
-        content = util.shellify(cfg["bootcmd"])
-        tmpf = tempfile.TemporaryFile()
-        tmpf.write(content)
-        tmpf.seek(0)
-    except:
-        log.warn("failed to shellify bootcmd")
-        raise
-
-    try:
-        env = os.environ.copy()
-        env['INSTANCE_ID'] = cloud.get_instance_id()
-        subprocess.check_call(['/bin/sh'], env=env, stdin=tmpf)
-        tmpf.close()
-    except:
-        log.warn("failed to run commands from bootcmd")
-        raise
diff --git a/cloudinit/CloudConfig/cc_chef.py b/cloudinit/CloudConfig/cc_chef.py
deleted file mode 100644
index 941e04fe..00000000
--- a/cloudinit/CloudConfig/cc_chef.py
+++ /dev/null
@@ -1,119 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Avishai Ish-Shalom <avishai@fewbytes.com>
-#    Author: Mike Moulton <mike@meltmedia.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import subprocess
-import json
-import cloudinit.CloudConfig as cc
-import cloudinit.util as util
-
-ruby_version_default = "1.8"
-
-
-def handle(_name, cfg, cloud, log, _args):
-    # If there isn't a chef key in the configuration don't do anything
-    if 'chef' not in cfg:
-        return
-    chef_cfg = cfg['chef']
-
-    # ensure the chef directories we use exist
-    mkdirs(['/etc/chef', '/var/log/chef', '/var/lib/chef',
-            '/var/cache/chef', '/var/backups/chef', '/var/run/chef'])
-
-    # set the validation key based on the presence of either 'validation_key'
-    # or 'validation_cert'. In the case where both exist, 'validation_key'
-    # takes precedence
-    for key in ('validation_key', 'validation_cert'):
-        if key in chef_cfg and chef_cfg[key]:
-            with open('/etc/chef/validation.pem', 'w') as validation_key_fh:
-                validation_key_fh.write(chef_cfg[key])
-            break
-
-    # create the chef config from template
-    util.render_to_file('chef_client.rb', '/etc/chef/client.rb',
-        {'server_url': chef_cfg['server_url'],
-         'node_name': util.get_cfg_option_str(chef_cfg, 'node_name',
-                                          cloud.datasource.get_instance_id()),
-         'environment': util.get_cfg_option_str(chef_cfg, 'environment',
-                                                '_default'),
-         'validation_name': chef_cfg['validation_name']})
-
-    # set the firstboot json
-    with open('/etc/chef/firstboot.json', 'w') as firstboot_json_fh:
-        initial_json = {}
-        if 'run_list' in chef_cfg:
-            initial_json['run_list'] = chef_cfg['run_list']
-        if 'initial_attributes' in chef_cfg:
-            initial_attributes = chef_cfg['initial_attributes']
-            for k in initial_attributes.keys():
-                initial_json[k] = initial_attributes[k]
-        firstboot_json_fh.write(json.dumps(initial_json))
-
-    # If chef is not installed, we install chef based on 'install_type'
-    if not os.path.isfile('/usr/bin/chef-client'):
-        install_type = util.get_cfg_option_str(chef_cfg, 'install_type',
-                                               'packages')
-        if install_type == "gems":
-            # this will install and run the chef-client from gems
-            chef_version = util.get_cfg_option_str(chef_cfg, 'version', None)
-            ruby_version = util.get_cfg_option_str(chef_cfg, 'ruby_version',
-                                                   ruby_version_default)
-            install_chef_from_gems(ruby_version, chef_version)
-            # and finally, run chef-client
-            log.debug('running chef-client')
-            subprocess.check_call(['/usr/bin/chef-client', '-d', '-i', '1800',
-                                   '-s', '20'])
-        else:
-            # this will install and run the chef-client from packages
-            cc.install_packages(('chef',))
-
-
-def get_ruby_packages(version):
-    # return a list of packages needed to install ruby at version
-    pkgs = ['ruby%s' % version, 'ruby%s-dev' % version]
-    if version == "1.8":
-        pkgs.extend(('libopenssl-ruby1.8', 'rubygems1.8'))
-    return(pkgs)
-
-
-def install_chef_from_gems(ruby_version, chef_version=None):
-    cc.install_packages(get_ruby_packages(ruby_version))
-    if not os.path.exists('/usr/bin/gem'):
-        os.symlink('/usr/bin/gem%s' % ruby_version, '/usr/bin/gem')
-    if not os.path.exists('/usr/bin/ruby'):
-        os.symlink('/usr/bin/ruby%s' % ruby_version, '/usr/bin/ruby')
-    if chef_version:
-        subprocess.check_call(['/usr/bin/gem', 'install', 'chef',
-                               '-v %s' % chef_version, '--no-ri',
-                               '--no-rdoc', '--bindir', '/usr/bin', '-q'])
-    else:
-        subprocess.check_call(['/usr/bin/gem', 'install', 'chef',
-                               '--no-ri', '--no-rdoc', '--bindir',
-                               '/usr/bin', '-q'])
-
-
-def ensure_dir(d):
-    if not os.path.exists(d):
-        os.makedirs(d)
-
-
-def mkdirs(dirs):
-    for d in dirs:
-        ensure_dir(d)
diff --git a/cloudinit/CloudConfig/cc_final_message.py b/cloudinit/CloudConfig/cc_final_message.py
deleted file mode 100644
index abb4ca32..00000000
--- a/cloudinit/CloudConfig/cc_final_message.py
+++ /dev/null
@@ -1,58 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-from cloudinit.CloudConfig import per_always
-import sys
-from cloudinit import util, boot_finished
-import time
-
-frequency = per_always
-
-final_message = "cloud-init boot finished at $TIMESTAMP. Up $UPTIME seconds"
-
-
-def handle(_name, cfg, _cloud, log, args):
-    if len(args) != 0:
-        msg_in = args[0]
-    else:
-        msg_in = util.get_cfg_option_str(cfg, "final_message", final_message)
-
-    try:
-        uptimef = open("/proc/uptime")
-        uptime = uptimef.read().split(" ")[0]
-        uptimef.close()
-    except IOError as e:
-        log.warn("unable to open /proc/uptime\n")
-        uptime = "na"
-
-    try:
-        ts = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.gmtime())
-    except:
-        ts = "na"
-
-    try:
-        subs = {'UPTIME': uptime, 'TIMESTAMP': ts}
-        sys.stdout.write("%s\n" % util.render_string(msg_in, subs))
-    except Exception as e:
-        log.warn("failed to render string to stdout: %s" % e)
-
-    fp = open(boot_finished, "wb")
-    fp.write(uptime + "\n")
-    fp.close()
diff --git a/cloudinit/CloudConfig/cc_keys_to_console.py b/cloudinit/CloudConfig/cc_keys_to_console.py
deleted file mode 100644
index 73a477c0..00000000
--- a/cloudinit/CloudConfig/cc_keys_to_console.py
+++ /dev/null
@@ -1,42 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-from cloudinit.CloudConfig import per_instance
-import cloudinit.util as util
-import subprocess
-
-frequency = per_instance
-
-
-def handle(_name, cfg, _cloud, log, _args):
-    cmd = ['/usr/lib/cloud-init/write-ssh-key-fingerprints']
-    fp_blacklist = util.get_cfg_option_list_or_str(cfg,
-        "ssh_fp_console_blacklist", [])
-    key_blacklist = util.get_cfg_option_list_or_str(cfg,
-        "ssh_key_console_blacklist", ["ssh-dss"])
-    try:
-        confp = open('/dev/console', "wb")
-        cmd.append(','.join(fp_blacklist))
-        cmd.append(','.join(key_blacklist))
-        subprocess.call(cmd, stdout=confp)
-        confp.close()
-    except:
-        log.warn("writing keys to console value")
-        raise
diff --git a/cloudinit/CloudConfig/cc_locale.py b/cloudinit/CloudConfig/cc_locale.py
deleted file mode 100644
index 2bb22fdb..00000000
--- a/cloudinit/CloudConfig/cc_locale.py
+++ /dev/null
@@ -1,54 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-import os.path
-import subprocess
-import traceback
-
-
-def apply_locale(locale, cfgfile):
-    if os.path.exists('/usr/sbin/locale-gen'):
-        subprocess.Popen(['locale-gen', locale]).communicate()
-    if os.path.exists('/usr/sbin/update-locale'):
-        subprocess.Popen(['update-locale', locale]).communicate()
-
-    util.render_to_file('default-locale', cfgfile, {'locale': locale})
-
-
-def handle(_name, cfg, cloud, log, args):
-    if len(args) != 0:
-        locale = args[0]
-    else:
-        locale = util.get_cfg_option_str(cfg, "locale", cloud.get_locale())
-
-    locale_cfgfile = util.get_cfg_option_str(cfg, "locale_configfile",
-                                             "/etc/default/locale")
-
-    if not locale:
-        return
-
-    log.debug("setting locale to %s" % locale)
-
-    try:
-        apply_locale(locale, locale_cfgfile)
-    except Exception as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("failed to apply locale %s" % locale)
diff --git a/cloudinit/CloudConfig/cc_mcollective.py b/cloudinit/CloudConfig/cc_mcollective.py
deleted file mode 100644
index a2a6230c..00000000
--- a/cloudinit/CloudConfig/cc_mcollective.py
+++ /dev/null
@@ -1,99 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Marc Cluet <marc.cluet@canonical.com>
-#    Based on code by Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import subprocess
-import StringIO
-import ConfigParser
-import cloudinit.CloudConfig as cc
-import cloudinit.util as util
-
-pubcert_file = "/etc/mcollective/ssl/server-public.pem"
-pricert_file = "/etc/mcollective/ssl/server-private.pem"
-
-
-# Our fake header section
-class FakeSecHead(object):
-    def __init__(self, fp):
-        self.fp = fp
-        self.sechead = '[nullsection]\n'
-
-    def readline(self):
-        if self.sechead:
-            try:
-                return self.sechead
-            finally:
-                self.sechead = None
-        else:
-            return self.fp.readline()
-
-
-def handle(_name, cfg, _cloud, _log, _args):
-    # If there isn't a mcollective key in the configuration don't do anything
-    if 'mcollective' not in cfg:
-        return
-    mcollective_cfg = cfg['mcollective']
-    # Start by installing the mcollective package ...
-    cc.install_packages(("mcollective",))
-
-    # ... and then update the mcollective configuration
-    if 'conf' in mcollective_cfg:
-        # Create object for reading server.cfg values
-        mcollective_config = ConfigParser.ConfigParser()
-        # Read server.cfg values from original file in order to be able to mix
-        # the rest up
-        mcollective_config.readfp(FakeSecHead(open('/etc/mcollective/'
-                                                   'server.cfg')))
-        for cfg_name, cfg in mcollective_cfg['conf'].iteritems():
-            if cfg_name == 'public-cert':
-                util.write_file(pubcert_file, cfg, mode=0644)
-                mcollective_config.set(cfg_name,
-                    'plugin.ssl_server_public', pubcert_file)
-                mcollective_config.set(cfg_name, 'securityprovider', 'ssl')
-            elif cfg_name == 'private-cert':
-                util.write_file(pricert_file, cfg, mode=0600)
-                mcollective_config.set(cfg_name,
-                    'plugin.ssl_server_private', pricert_file)
-                mcollective_config.set(cfg_name, 'securityprovider', 'ssl')
-            else:
-                # Iterate throug the config items, we'll use ConfigParser.set
-                # to overwrite or create new items as needed
-                for o, v in cfg.iteritems():
-                    mcollective_config.set(cfg_name, o, v)
-        # We got all our config as wanted we'll rename
-        # the previous server.cfg and create our new one
-        os.rename('/etc/mcollective/server.cfg',
-                  '/etc/mcollective/server.cfg.old')
-        outputfile = StringIO.StringIO()
-        mcollective_config.write(outputfile)
-        # Now we got the whole file, write to disk except first line
-        # Note below, that we've just used ConfigParser because it generally
-        # works.  Below, we remove the initial 'nullsection' header
-        # and then change 'key = value' to 'key: value'.  The global
-        # search and replace of '=' with ':' could be problematic though.
-        # this most likely needs fixing.
-        util.write_file('/etc/mcollective/server.cfg',
-            outputfile.getvalue().replace('[nullsection]\n', '').replace(' =',
-                                                                         ':'),
-            mode=0644)
-
-    # Start mcollective
-    subprocess.check_call(['service', 'mcollective', 'start'])
diff --git a/cloudinit/CloudConfig/cc_puppet.py b/cloudinit/CloudConfig/cc_puppet.py
deleted file mode 100644
index 6fc475f6..00000000
--- a/cloudinit/CloudConfig/cc_puppet.py
+++ /dev/null
@@ -1,108 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import os.path
-import pwd
-import socket
-import subprocess
-import StringIO
-import ConfigParser
-import cloudinit.CloudConfig as cc
-import cloudinit.util as util
-
-
-def handle(_name, cfg, cloud, log, _args):
-    # If there isn't a puppet key in the configuration don't do anything
-    if 'puppet' not in cfg:
-        return
-    puppet_cfg = cfg['puppet']
-    # Start by installing the puppet package ...
-    cc.install_packages(("puppet",))
-
-    # ... and then update the puppet configuration
-    if 'conf' in puppet_cfg:
-        # Add all sections from the conf object to puppet.conf
-        puppet_conf_fh = open('/etc/puppet/puppet.conf', 'r')
-        # Create object for reading puppet.conf values
-        puppet_config = ConfigParser.ConfigParser()
-        # Read puppet.conf values from original file in order to be able to
-        # mix the rest up
-        puppet_config.readfp(StringIO.StringIO(''.join(i.lstrip() for i in
-                                               puppet_conf_fh.readlines())))
-        # Close original file, no longer needed
-        puppet_conf_fh.close()
-        for cfg_name, cfg in puppet_cfg['conf'].iteritems():
-            # ca_cert configuration is a special case
-            # Dump the puppetmaster ca certificate in the correct place
-            if cfg_name == 'ca_cert':
-                # Puppet ssl sub-directory isn't created yet
-                # Create it with the proper permissions and ownership
-                os.makedirs('/var/lib/puppet/ssl')
-                os.chmod('/var/lib/puppet/ssl', 0771)
-                os.chown('/var/lib/puppet/ssl',
-                         pwd.getpwnam('puppet').pw_uid, 0)
-                os.makedirs('/var/lib/puppet/ssl/certs/')
-                os.chown('/var/lib/puppet/ssl/certs/',
-                         pwd.getpwnam('puppet').pw_uid, 0)
-                ca_fh = open('/var/lib/puppet/ssl/certs/ca.pem', 'w')
-                ca_fh.write(cfg)
-                ca_fh.close()
-                os.chown('/var/lib/puppet/ssl/certs/ca.pem',
-                         pwd.getpwnam('puppet').pw_uid, 0)
-                util.restorecon_if_possible('/var/lib/puppet', recursive=True)
-            else:
-                #puppet_conf_fh.write("\n[%s]\n" % (cfg_name))
-                # If puppet.conf already has this section we don't want to
-                # write it again
-                if puppet_config.has_section(cfg_name) == False:
-                    puppet_config.add_section(cfg_name)
-                # Iterate throug the config items, we'll use ConfigParser.set
-                # to overwrite or create new items as needed
-                for o, v in cfg.iteritems():
-                    if o == 'certname':
-                        # Expand %f as the fqdn
-                        v = v.replace("%f", socket.getfqdn())
-                        # Expand %i as the instance id
-                        v = v.replace("%i",
-                              cloud.datasource.get_instance_id())
-                        # certname needs to be downcase
-                        v = v.lower()
-                    puppet_config.set(cfg_name, o, v)
-                    #puppet_conf_fh.write("%s=%s\n" % (o, v))
-            # We got all our config as wanted we'll rename
-            # the previous puppet.conf and create our new one
-            os.rename('/etc/puppet/puppet.conf', '/etc/puppet/puppet.conf.old')
-            with open('/etc/puppet/puppet.conf', 'wb') as configfile:
-                puppet_config.write(configfile)
-            util.restorecon_if_possible('/etc/puppet/puppet.conf')
-    # Set puppet to automatically start
-    if os.path.exists('/etc/default/puppet'):
-        subprocess.check_call(['sed', '-i',
-                               '-e', 's/^START=.*/START=yes/',
-                               '/etc/default/puppet'])
-    elif os.path.exists('/bin/systemctl'):
-        subprocess.check_call(['/bin/systemctl', 'enable', 'puppet.service'])
-    elif os.path.exists('/sbin/chkconfig'):
-        subprocess.check_call(['/sbin/chkconfig', 'puppet', 'on'])
-    else:
-        log.warn("Do not know how to enable puppet service on this system")
-    # Start puppetd
-    subprocess.check_call(['service', 'puppet', 'start'])
diff --git a/cloudinit/CloudConfig/cc_resizefs.py b/cloudinit/CloudConfig/cc_resizefs.py
deleted file mode 100644
index 2dc66def..00000000
--- a/cloudinit/CloudConfig/cc_resizefs.py
+++ /dev/null
@@ -1,108 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-import subprocess
-import os
-import stat
-import sys
-import time
-import tempfile
-from cloudinit.CloudConfig import per_always
-
-frequency = per_always
-
-
-def handle(_name, cfg, _cloud, log, args):
-    if len(args) != 0:
-        resize_root = False
-        if str(args[0]).lower() in ['true', '1', 'on', 'yes']:
-            resize_root = True
-    else:
-        resize_root = util.get_cfg_option_str(cfg, "resize_rootfs", True)
-
-    if str(resize_root).lower() in ['false', '0']:
-        return
-
-    # we use mktemp rather than mkstemp because early in boot nothing
-    # else should be able to race us for this, and we need to mknod.
-    devpth = tempfile.mktemp(prefix="cloudinit.resizefs.", dir="/run")
-
-    try:
-        st_dev = os.stat("/").st_dev
-        dev = os.makedev(os.major(st_dev), os.minor(st_dev))
-        os.mknod(devpth, 0400 | stat.S_IFBLK, dev)
-    except:
-        if util.is_container():
-            log.debug("inside container, ignoring mknod failure in resizefs")
-            return
-        log.warn("Failed to make device node to resize /")
-        raise
-
-    cmd = ['blkid', '-c', '/dev/null', '-sTYPE', '-ovalue', devpth]
-    try:
-        (fstype, _err) = util.subp(cmd)
-    except subprocess.CalledProcessError as e:
-        log.warn("Failed to get filesystem type of maj=%s, min=%s via: %s" %
-            (os.major(st_dev), os.minor(st_dev), cmd))
-        log.warn("output=%s\nerror=%s\n", e.output[0], e.output[1])
-        os.unlink(devpth)
-        raise
-
-    if str(fstype).startswith("ext"):
-        resize_cmd = ['resize2fs', devpth]
-    elif fstype == "xfs":
-        resize_cmd = ['xfs_growfs', devpth]
-    else:
-        os.unlink(devpth)
-        log.debug("not resizing unknown filesystem %s" % fstype)
-        return
-
-    if resize_root == "noblock":
-        fid = os.fork()
-        if fid == 0:
-            try:
-                do_resize(resize_cmd, devpth, log)
-                os._exit(0)  # pylint: disable=W0212
-            except Exception as exc:
-                sys.stderr.write("Failed: %s" % exc)
-                os._exit(1)  # pylint: disable=W0212
-    else:
-        do_resize(resize_cmd, devpth, log)
-
-    log.debug("resizing root filesystem (type=%s, maj=%i, min=%i, val=%s)" %
-        (str(fstype).rstrip("\n"), os.major(st_dev), os.minor(st_dev),
-         resize_root))
-
-    return
-
-
-def do_resize(resize_cmd, devpth, log):
-    try:
-        start = time.time()
-        util.subp(resize_cmd)
-    except subprocess.CalledProcessError as e:
-        log.warn("Failed to resize filesystem (%s)" % resize_cmd)
-        log.warn("output=%s\nerror=%s\n", e.output[0], e.output[1])
-        os.unlink(devpth)
-        raise
-
-    os.unlink(devpth)
-    log.debug("resize took %s seconds" % (time.time() - start))
diff --git a/cloudinit/CloudConfig/cc_rightscale_userdata.py b/cloudinit/CloudConfig/cc_rightscale_userdata.py
deleted file mode 100644
index 5ed0848f..00000000
--- a/cloudinit/CloudConfig/cc_rightscale_userdata.py
+++ /dev/null
@@ -1,78 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-##
-## The purpose of this script is to allow cloud-init to consume
-## rightscale style userdata.  rightscale user data is key-value pairs
-## in a url-query-string like format.
-##
-## for cloud-init support, there will be a key named
-## 'CLOUD_INIT_REMOTE_HOOK'.
-##
-## This cloud-config module will
-## - read the blob of data from raw user data, and parse it as key/value
-## - for each key that is found, download the content to
-##   the local instance/scripts directory and set them executable.
-## - the files in that directory will be run by the user-scripts module
-##   Therefore, this must run before that.
-##
-##
-
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-from cloudinit import get_ipath_cur
-from urlparse import parse_qs
-
-frequency = per_instance
-my_name = "cc_rightscale_userdata"
-my_hookname = 'CLOUD_INIT_REMOTE_HOOK'
-
-
-def handle(_name, _cfg, cloud, log, _args):
-    try:
-        ud = cloud.get_userdata_raw()
-    except:
-        log.warn("failed to get raw userdata in %s" % my_name)
-        return
-
-    try:
-        mdict = parse_qs(ud)
-        if not my_hookname in mdict:
-            return
-    except:
-        log.warn("failed to urlparse.parse_qa(userdata_raw())")
-        raise
-
-    scripts_d = get_ipath_cur('scripts')
-    i = 0
-    first_e = None
-    for url in mdict[my_hookname]:
-        fname = "%s/rightscale-%02i" % (scripts_d, i)
-        i = i + 1
-        try:
-            content = util.readurl(url)
-            util.write_file(fname, content, mode=0700)
-        except Exception as e:
-            if not first_e:
-                first_e = None
-            log.warn("%s failed to read %s: %s" % (my_name, url, e))
-
-    if first_e:
-        raise(e)
diff --git a/cloudinit/CloudConfig/cc_ssh.py b/cloudinit/CloudConfig/cc_ssh.py
deleted file mode 100644
index 48eb58bc..00000000
--- a/cloudinit/CloudConfig/cc_ssh.py
+++ /dev/null
@@ -1,106 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-import cloudinit.SshUtil as sshutil
-import os
-import glob
-import subprocess
-
-DISABLE_ROOT_OPTS = "no-port-forwarding,no-agent-forwarding," \
-"no-X11-forwarding,command=\"echo \'Please login as the user \\\"$USER\\\" " \
-"rather than the user \\\"root\\\".\';echo;sleep 10\""
-
-
-def handle(_name, cfg, cloud, log, _args):
-
-    # remove the static keys from the pristine image
-    if cfg.get("ssh_deletekeys", True):
-        for f in glob.glob("/etc/ssh/ssh_host_*key*"):
-            try:
-                os.unlink(f)
-            except:
-                pass
-
-    if "ssh_keys" in cfg:
-        # if there are keys in cloud-config, use them
-        key2file = {
-            "rsa_private": ("/etc/ssh/ssh_host_rsa_key", 0600),
-            "rsa_public": ("/etc/ssh/ssh_host_rsa_key.pub", 0644),
-            "dsa_private": ("/etc/ssh/ssh_host_dsa_key", 0600),
-            "dsa_public": ("/etc/ssh/ssh_host_dsa_key.pub", 0644),
-            "ecdsa_private": ("/etc/ssh/ssh_host_ecdsa_key", 0600),
-            "ecdsa_public": ("/etc/ssh/ssh_host_ecdsa_key.pub", 0644),
-        }
-
-        for key, val in cfg["ssh_keys"].items():
-            if key in key2file:
-                util.write_file(key2file[key][0], val, key2file[key][1])
-
-        priv2pub = {'rsa_private': 'rsa_public', 'dsa_private': 'dsa_public',
-                    'ecdsa_private': 'ecdsa_public', }
-
-        cmd = 'o=$(ssh-keygen -yf "%s") && echo "$o" root@localhost > "%s"'
-        for priv, pub in priv2pub.iteritems():
-            if pub in cfg['ssh_keys'] or not priv in cfg['ssh_keys']:
-                continue
-            pair = (key2file[priv][0], key2file[pub][0])
-            subprocess.call(('sh', '-xc', cmd % pair))
-            log.debug("generated %s from %s" % pair)
-    else:
-        # if not, generate them
-        for keytype in util.get_cfg_option_list_or_str(cfg, 'ssh_genkeytypes',
-                                                      ['rsa', 'dsa', 'ecdsa']):
-            keyfile = '/etc/ssh/ssh_host_%s_key' % keytype
-            if not os.path.exists(keyfile):
-                subprocess.call(['ssh-keygen', '-t', keytype, '-N', '',
-                                 '-f', keyfile])
-
-    util.restorecon_if_possible('/etc/ssh', recursive=True)
-
-    try:
-        user = util.get_cfg_option_str(cfg, 'user')
-        disable_root = util.get_cfg_option_bool(cfg, "disable_root", True)
-        disable_root_opts = util.get_cfg_option_str(cfg, "disable_root_opts",
-            DISABLE_ROOT_OPTS)
-        keys = cloud.get_public_ssh_keys()
-
-        if "ssh_authorized_keys" in cfg:
-            cfgkeys = cfg["ssh_authorized_keys"]
-            keys.extend(cfgkeys)
-
-        apply_credentials(keys, user, disable_root, disable_root_opts, log)
-    except:
-        util.logexc(log)
-        log.warn("applying credentials failed!\n")
-
-
-def apply_credentials(keys, user, disable_root,
-                      disable_root_opts=DISABLE_ROOT_OPTS, log=None):
-    keys = set(keys)
-    if user:
-        sshutil.setup_user_keys(keys, user, '', log)
-
-    if disable_root:
-        key_prefix = disable_root_opts.replace('$USER', user)
-    else:
-        key_prefix = ''
-
-    sshutil.setup_user_keys(keys, 'root', key_prefix, log)
diff --git a/cloudinit/CloudConfig/cc_timezone.py b/cloudinit/CloudConfig/cc_timezone.py
deleted file mode 100644
index e5c9901b..00000000
--- a/cloudinit/CloudConfig/cc_timezone.py
+++ /dev/null
@@ -1,67 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-from cloudinit.CloudConfig import per_instance
-from cloudinit import util
-import os.path
-import shutil
-
-frequency = per_instance
-tz_base = "/usr/share/zoneinfo"
-
-
-def handle(_name, cfg, _cloud, log, args):
-    if len(args) != 0:
-        timezone = args[0]
-    else:
-        timezone = util.get_cfg_option_str(cfg, "timezone", False)
-
-    if not timezone:
-        return
-
-    tz_file = "%s/%s" % (tz_base, timezone)
-
-    if not os.path.isfile(tz_file):
-        log.debug("Invalid timezone %s" % tz_file)
-        raise Exception("Invalid timezone %s" % tz_file)
-
-    try:
-        fp = open("/etc/timezone", "wb")
-        fp.write("%s\n" % timezone)
-        fp.close()
-    except:
-        log.debug("failed to write to /etc/timezone")
-        raise
-    if os.path.exists("/etc/sysconfig/clock"):
-        try:
-            with open("/etc/sysconfig/clock", "w") as fp:
-                fp.write('ZONE="%s"\n' % timezone)
-        except:
-            log.debug("failed to write to /etc/sysconfig/clock")
-            raise
-
-    try:
-        shutil.copy(tz_file, "/etc/localtime")
-    except:
-        log.debug("failed to copy %s to /etc/localtime" % tz_file)
-        raise
-
-    log.debug("set timezone to %s" % timezone)
-    return
diff --git a/cloudinit/CloudConfig/cc_update_etc_hosts.py b/cloudinit/CloudConfig/cc_update_etc_hosts.py
deleted file mode 100644
index 6ad2fca8..00000000
--- a/cloudinit/CloudConfig/cc_update_etc_hosts.py
+++ /dev/null
@@ -1,87 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_always
-import StringIO
-
-frequency = per_always
-
-
-def handle(_name, cfg, cloud, log, _args):
-    (hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
-
-    manage_hosts = util.get_cfg_option_str(cfg, "manage_etc_hosts", False)
-    if manage_hosts in ("True", "true", True, "template"):
-        # render from template file
-        try:
-            if not hostname:
-                log.info("manage_etc_hosts was set, but no hostname found")
-                return
-
-            util.render_to_file('hosts', '/etc/hosts',
-                                {'hostname': hostname, 'fqdn': fqdn})
-        except Exception:
-            log.warn("failed to update /etc/hosts")
-            raise
-    elif manage_hosts == "localhost":
-        log.debug("managing 127.0.1.1 in /etc/hosts")
-        update_etc_hosts(hostname, fqdn, log)
-        return
-    else:
-        if manage_hosts not in ("False", False):
-            log.warn("Unknown value for manage_etc_hosts.  Assuming False")
-        else:
-            log.debug("not managing /etc/hosts")
-
-
-def update_etc_hosts(hostname, fqdn, _log):
-    with open('/etc/hosts', 'r') as etchosts:
-        header = "# Added by cloud-init\n"
-        hosts_line = "127.0.1.1\t%s %s\n" % (fqdn, hostname)
-        need_write = False
-        need_change = True
-        new_etchosts = StringIO.StringIO()
-        for line in etchosts:
-            split_line = [s.strip() for s in line.split()]
-            if len(split_line) < 2:
-                new_etchosts.write(line)
-                continue
-            if line == header:
-                continue
-            ip, hosts = split_line[0], split_line[1:]
-            if ip == "127.0.1.1":
-                if sorted([hostname, fqdn]) == sorted(hosts):
-                    need_change = False
-                if need_change == True:
-                    line = "%s%s" % (header, hosts_line)
-                    need_change = False
-                    need_write = True
-            new_etchosts.write(line)
-        etchosts.close()
-        if need_change == True:
-            new_etchosts.write("%s%s" % (header, hosts_line))
-            need_write = True
-        if need_write == True:
-            new_etcfile = open('/etc/hosts', 'wb')
-            new_etcfile.write(new_etchosts.getvalue())
-            new_etcfile.close()
-        new_etchosts.close()
-    return
diff --git a/cloudinit/CloudConfig/cc_update_hostname.py b/cloudinit/CloudConfig/cc_update_hostname.py
deleted file mode 100644
index b9d1919a..00000000
--- a/cloudinit/CloudConfig/cc_update_hostname.py
+++ /dev/null
@@ -1,101 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2011 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.util as util
-import subprocess
-import errno
-from cloudinit.CloudConfig import per_always
-
-frequency = per_always
-
-
-def handle(_name, cfg, cloud, log, _args):
-    if util.get_cfg_option_bool(cfg, "preserve_hostname", False):
-        log.debug("preserve_hostname is set. not updating hostname")
-        return
-
-    (hostname, _fqdn) = util.get_hostname_fqdn(cfg, cloud)
-    try:
-        prev = "%s/%s" % (cloud.get_cpath('data'), "previous-hostname")
-        update_hostname(hostname, prev, log)
-    except Exception:
-        log.warn("failed to set hostname\n")
-        raise
-
-
-# read hostname from a 'hostname' file
-# allow for comments and stripping line endings.
-# if file doesn't exist, or no contents, return default
-def read_hostname(filename, default=None):
-    try:
-        fp = open(filename, "r")
-        lines = fp.readlines()
-        fp.close()
-        for line in lines:
-            hpos = line.find("#")
-            if hpos != -1:
-                line = line[0:hpos]
-            line = line.rstrip()
-            if line:
-                return line
-    except IOError as e:
-        if e.errno != errno.ENOENT:
-            raise
-    return default
-
-
-def update_hostname(hostname, prev_file, log):
-    etc_file = "/etc/hostname"
-
-    hostname_prev = None
-    hostname_in_etc = None
-
-    try:
-        hostname_prev = read_hostname(prev_file)
-    except Exception as e:
-        log.warn("Failed to open %s: %s" % (prev_file, e))
-
-    try:
-        hostname_in_etc = read_hostname(etc_file)
-    except:
-        log.warn("Failed to open %s" % etc_file)
-
-    update_files = []
-    if not hostname_prev or hostname_prev != hostname:
-        update_files.append(prev_file)
-
-    if (not hostname_in_etc or
-        (hostname_in_etc == hostname_prev and hostname_in_etc != hostname)):
-        update_files.append(etc_file)
-
-    try:
-        for fname in update_files:
-            util.write_file(fname, "%s\n" % hostname, 0644)
-            log.debug("wrote %s to %s" % (hostname, fname))
-    except:
-        log.warn("failed to write hostname to %s" % fname)
-
-    if hostname_in_etc and hostname_prev and hostname_in_etc != hostname_prev:
-        log.debug("%s differs from %s. assuming user maintained" %
-                  (prev_file, etc_file))
-
-    if etc_file in update_files:
-        log.debug("setting hostname to %s" % hostname)
-        subprocess.Popen(['hostname', hostname]).communicate()
diff --git a/cloudinit/DataSource.py b/cloudinit/DataSource.py
deleted file mode 100644
index e2a9150d..00000000
--- a/cloudinit/DataSource.py
+++ /dev/null
@@ -1,214 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-
-DEP_FILESYSTEM = "FILESYSTEM"
-DEP_NETWORK = "NETWORK"
-
-import cloudinit.UserDataHandler as ud
-import cloudinit.util as util
-import socket
-
-
-class DataSource:
-    userdata = None
-    metadata = None
-    userdata_raw = None
-    cfgname = ""
-    # system config (passed in from cloudinit,
-    # cloud-config before input from the DataSource)
-    sys_cfg = {}
-    # datasource config, the cloud-config['datasource']['__name__']
-    ds_cfg = {}  # datasource config
-
-    def __init__(self, sys_cfg=None):
-        if not self.cfgname:
-            name = str(self.__class__).split(".")[-1]
-            if name.startswith("DataSource"):
-                name = name[len("DataSource"):]
-            self.cfgname = name
-        if sys_cfg:
-            self.sys_cfg = sys_cfg
-
-        self.ds_cfg = util.get_cfg_by_path(self.sys_cfg,
-                          ("datasource", self.cfgname), self.ds_cfg)
-
-    def get_userdata(self):
-        if self.userdata == None:
-            self.userdata = ud.preprocess_userdata(self.userdata_raw)
-        return self.userdata
-
-    def get_userdata_raw(self):
-        return(self.userdata_raw)
-
-    # the data sources' config_obj is a cloud-config formated
-    # object that came to it from ways other than cloud-config
-    # because cloud-config content would be handled elsewhere
-    def get_config_obj(self):
-        return({})
-
-    def get_public_ssh_keys(self):
-        keys = []
-        if 'public-keys' not in self.metadata:
-            return([])
-
-        if isinstance(self.metadata['public-keys'], str):
-            return(str(self.metadata['public-keys']).splitlines())
-
-        if isinstance(self.metadata['public-keys'], list):
-            return(self.metadata['public-keys'])
-
-        for _keyname, klist in self.metadata['public-keys'].items():
-            # lp:506332 uec metadata service responds with
-            # data that makes boto populate a string for 'klist' rather
-            # than a list.
-            if isinstance(klist, str):
-                klist = [klist]
-            for pkey in klist:
-                # there is an empty string at the end of the keylist, trim it
-                if pkey:
-                    keys.append(pkey)
-
-        return(keys)
-
-    def device_name_to_device(self, _name):
-        # translate a 'name' to a device
-        # the primary function at this point is on ec2
-        # to consult metadata service, that has
-        #  ephemeral0: sdb
-        # and return 'sdb' for input 'ephemeral0'
-        return(None)
-
-    def get_locale(self):
-        return('en_US.UTF-8')
-
-    def get_local_mirror(self):
-        return None
-
-    def get_instance_id(self):
-        if 'instance-id' not in self.metadata:
-            return "iid-datasource"
-        return(self.metadata['instance-id'])
-
-    def get_hostname(self, fqdn=False):
-        defdomain = "localdomain"
-        defhost = "localhost"
-
-        domain = defdomain
-        if not 'local-hostname' in self.metadata:
-
-            # this is somewhat questionable really.
-            # the cloud datasource was asked for a hostname
-            # and didn't have one. raising error might be more appropriate
-            # but instead, basically look up the existing hostname
-            toks = []
-
-            hostname = socket.gethostname()
-
-            fqdn = util.get_fqdn_from_hosts(hostname)
-
-            if fqdn and fqdn.find(".") > 0:
-                toks = str(fqdn).split(".")
-            elif hostname:
-                toks = [hostname, defdomain]
-            else:
-                toks = [defhost, defdomain]
-
-        else:
-            # if there is an ipv4 address in 'local-hostname', then
-            # make up a hostname (LP: #475354) in format ip-xx.xx.xx.xx
-            lhost = self.metadata['local-hostname']
-            if is_ipv4(lhost):
-                toks = "ip-%s" % lhost.replace(".", "-")
-            else:
-                toks = lhost.split(".")
-
-        if len(toks) > 1:
-            hostname = toks[0]
-            domain = '.'.join(toks[1:])
-        else:
-            hostname = toks[0]
-
-        if fqdn:
-            return "%s.%s" % (hostname, domain)
-        else:
-            return hostname
-
-
-# return a list of classes that have the same depends as 'depends'
-# iterate through cfg_list, loading "DataSourceCollections" modules
-# and calling their "get_datasource_list".
-# return an ordered list of classes that match
-#
-# - modules must be named "DataSource<item>", where 'item' is an entry
-#   in cfg_list
-# - if pkglist is given, it will iterate try loading from that package
-#   ie, pkglist=[ "foo", "" ]
-#     will first try to load foo.DataSource<item>
-#     then DataSource<item>
-def list_sources(cfg_list, depends, pkglist=None):
-    if pkglist is None:
-        pkglist = []
-    retlist = []
-    for ds_coll in cfg_list:
-        for pkg in pkglist:
-            if pkg:
-                pkg = "%s." % pkg
-            try:
-                mod = __import__("%sDataSource%s" % (pkg, ds_coll))
-                if pkg:
-                    mod = getattr(mod, "DataSource%s" % ds_coll)
-                lister = getattr(mod, "get_datasource_list")
-                retlist.extend(lister(depends))
-                break
-            except:
-                raise
-    return(retlist)
-
-
-# depends is a list of dependencies (DEP_FILESYSTEM)
-# dslist is a list of 2 item lists
-# dslist = [
-#   ( class, ( depends-that-this-class-needs ) )
-# }
-# it returns a list of 'class' that matched these deps exactly
-# it is a helper function for DataSourceCollections
-def list_from_depends(depends, dslist):
-    retlist = []
-    depset = set(depends)
-    for elem in dslist:
-        (cls, deps) = elem
-        if depset == set(deps):
-            retlist.append(cls)
-    return(retlist)
-
-
-def is_ipv4(instr):
-    """ determine if input string is a ipv4 address. return boolean"""
-    toks = instr.split('.')
-    if len(toks) != 4:
-        return False
-
-    try:
-        toks = [x for x in toks if (int(x) < 256 and int(x) > 0)]
-    except:
-        return False
-
-    return (len(toks) == 4)
diff --git a/cloudinit/DataSourceCloudStack.py b/cloudinit/DataSourceCloudStack.py
deleted file mode 100644
index 5afdf7b6..00000000
--- a/cloudinit/DataSourceCloudStack.py
+++ /dev/null
@@ -1,92 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2012 Canonical Ltd.
-#    Copyright (C) 2012 Cosmin Luta
-#
-#    Author: Cosmin Luta <q4break@gmail.com>
-#    Author: Scott Moser <scott.moser@canonical.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.DataSource as DataSource
-
-from cloudinit import seeddir as base_seeddir
-from cloudinit import log
-import cloudinit.util as util
-from socket import inet_ntoa
-import time
-import boto.utils as boto_utils
-from struct import pack
-
-
-class DataSourceCloudStack(DataSource.DataSource):
-    api_ver = 'latest'
-    seeddir = base_seeddir + '/cs'
-    metadata_address = None
-
-    def __init__(self, sys_cfg=None):
-        DataSource.DataSource.__init__(self, sys_cfg)
-        # Cloudstack has its metadata/userdata URLs located at
-        # http://<default-gateway-ip>/latest/
-        self.metadata_address = "http://%s/" % self.get_default_gateway()
-
-    def get_default_gateway(self):
-        """ Returns the default gateway ip address in the dotted format
-        """
-        with open("/proc/net/route", "r") as f:
-            for line in f.readlines():
-                items = line.split("\t")
-                if items[1] == "00000000":
-                    # found the default route, get the gateway
-                    gw = inet_ntoa(pack("<L", int(items[2], 16)))
-                    log.debug("found default route, gateway is %s" % gw)
-                    return gw
-
-    def __str__(self):
-        return "DataSourceCloudStack"
-
-    def get_data(self):
-        seedret = {}
-        if util.read_optional_seed(seedret, base=self.seeddir + "/"):
-            self.userdata_raw = seedret['user-data']
-            self.metadata = seedret['meta-data']
-            log.debug("using seeded cs data in %s" % self.seeddir)
-            return True
-
-        try:
-            start = time.time()
-            self.userdata_raw = boto_utils.get_instance_userdata(self.api_ver,
-                None, self.metadata_address)
-            self.metadata = boto_utils.get_instance_metadata(self.api_ver,
-                self.metadata_address)
-            log.debug("crawl of metadata service took %ds" %
-                (time.time() - start))
-            return True
-        except Exception as e:
-            log.exception(e)
-            return False
-
-    def get_instance_id(self):
-        return self.metadata['instance-id']
-
-    def get_availability_zone(self):
-        return self.metadata['availability-zone']
-
-datasources = [
-  (DataSourceCloudStack, (DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK)),
-]
-
-
-# return a list of data sources that match this set of dependencies
-def get_datasource_list(depends):
-    return DataSource.list_from_depends(depends, datasources)
diff --git a/cloudinit/DataSourceConfigDrive.py b/cloudinit/DataSourceConfigDrive.py
deleted file mode 100644
index 2db4a76a..00000000
--- a/cloudinit/DataSourceConfigDrive.py
+++ /dev/null
@@ -1,231 +0,0 @@
-#    Copyright (C) 2012 Canonical Ltd.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.DataSource as DataSource
-
-from cloudinit import seeddir as base_seeddir
-from cloudinit import log
-import cloudinit.util as util
-import os.path
-import os
-import json
-import subprocess
-
-DEFAULT_IID = "iid-dsconfigdrive"
-
-
-class DataSourceConfigDrive(DataSource.DataSource):
-    seed = None
-    seeddir = base_seeddir + '/config_drive'
-    cfg = {}
-    userdata_raw = None
-    metadata = None
-    dsmode = "local"
-
-    def __str__(self):
-        mstr = "DataSourceConfigDrive[%s]" % self.dsmode
-        mstr = mstr + " [seed=%s]" % self.seed
-        return(mstr)
-
-    def get_data(self):
-        found = None
-        md = {}
-        ud = ""
-
-        defaults = {"instance-id": DEFAULT_IID, "dsmode": "pass"}
-
-        if os.path.isdir(self.seeddir):
-            try:
-                (md, ud) = read_config_drive_dir(self.seeddir)
-                found = self.seeddir
-            except nonConfigDriveDir:
-                pass
-
-        if not found:
-            dev = cfg_drive_device()
-            if dev:
-                try:
-                    (md, ud) = util.mount_callback_umount(dev,
-                        read_config_drive_dir)
-                    found = dev
-                except (nonConfigDriveDir, util.mountFailedError):
-                    pass
-
-        if not found:
-            return False
-
-        if 'dsconfig' in md:
-            self.cfg = md['dscfg']
-
-        md = util.mergedict(md, defaults)
-
-        # update interfaces and ifup only on the local datasource
-        # this way the DataSourceConfigDriveNet doesn't do it also.
-        if 'network-interfaces' in md and self.dsmode == "local":
-            if md['dsmode'] == "pass":
-                log.info("updating network interfaces from configdrive")
-            else:
-                log.debug("updating network interfaces from configdrive")
-
-            util.write_file("/etc/network/interfaces",
-                md['network-interfaces'])
-            try:
-                (out, err) = util.subp(['ifup', '--all'])
-                if len(out) or len(err):
-                    log.warn("ifup --all had stderr: %s" % err)
-
-            except subprocess.CalledProcessError as exc:
-                log.warn("ifup --all failed: %s" % (exc.output[1]))
-
-        self.seed = found
-        self.metadata = md
-        self.userdata_raw = ud
-
-        if md['dsmode'] == self.dsmode:
-            return True
-
-        log.debug("%s: not claiming datasource, dsmode=%s" %
-            (self, md['dsmode']))
-        return False
-
-    def get_public_ssh_keys(self):
-        if not 'public-keys' in self.metadata:
-            return([])
-        return(self.metadata['public-keys'])
-
-    # the data sources' config_obj is a cloud-config formated
-    # object that came to it from ways other than cloud-config
-    # because cloud-config content would be handled elsewhere
-    def get_config_obj(self):
-        return(self.cfg)
-
-
-class DataSourceConfigDriveNet(DataSourceConfigDrive):
-    dsmode = "net"
-
-
-class nonConfigDriveDir(Exception):
-    pass
-
-
-def cfg_drive_device():
-    """ get the config drive device.  return a string like '/dev/vdb'
-        or None (if there is no non-root device attached). This does not
-        check the contents, only reports that if there *were* a config_drive
-        attached, it would be this device.
-        per config_drive documentation, this is
-         "associated as the last available disk on the instance"
-    """
-
-    if 'CLOUD_INIT_CONFIG_DRIVE_DEVICE' in os.environ:
-        return(os.environ['CLOUD_INIT_CONFIG_DRIVE_DEVICE'])
-
-    # we are looking for a raw block device (sda, not sda1) with a vfat
-    # filesystem on it.
-
-    letters = "abcdefghijklmnopqrstuvwxyz"
-    devs = util.find_devs_with("TYPE=vfat")
-
-    # filter out anything not ending in a letter (ignore partitions)
-    devs = [f for f in devs if f[-1] in letters]
-
-    # sort them in reverse so "last" device is first
-    devs.sort(reverse=True)
-
-    if len(devs):
-        return(devs[0])
-
-    return(None)
-
-
-def read_config_drive_dir(source_dir):
-    """
-    read_config_drive_dir(source_dir):
-       read source_dir, and return a tuple with metadata dict and user-data
-       string populated.  If not a valid dir, raise a nonConfigDriveDir
-    """
-    md = {}
-    ud = ""
-
-    flist = ("etc/network/interfaces", "root/.ssh/authorized_keys", "meta.js")
-    found = [f for f in flist if os.path.isfile("%s/%s" % (source_dir, f))]
-    keydata = ""
-
-    if len(found) == 0:
-        raise nonConfigDriveDir("%s: %s" % (source_dir, "no files found"))
-
-    if "etc/network/interfaces" in found:
-        with open("%s/%s" % (source_dir, "/etc/network/interfaces")) as fp:
-            md['network-interfaces'] = fp.read()
-
-    if "root/.ssh/authorized_keys" in found:
-        with open("%s/%s" % (source_dir, "root/.ssh/authorized_keys")) as fp:
-            keydata = fp.read()
-
-    meta_js = {}
-
-    if "meta.js" in found:
-        content = ''
-        with open("%s/%s" % (source_dir, "meta.js")) as fp:
-            content = fp.read()
-        md['meta_js'] = content
-        try:
-            meta_js = json.loads(content)
-        except ValueError:
-            raise nonConfigDriveDir("%s: %s" %
-                (source_dir, "invalid json in meta.js"))
-
-    keydata = meta_js.get('public-keys', keydata)
-
-    if keydata:
-        lines = keydata.splitlines()
-        md['public-keys'] = [l for l in lines
-            if len(l) and not l.startswith("#")]
-
-    for copy in ('dsmode', 'instance-id', 'dscfg'):
-        if copy in meta_js:
-            md[copy] = meta_js[copy]
-
-    if 'user-data' in meta_js:
-        ud = meta_js['user-data']
-
-    return(md, ud)
-
-datasources = (
-  (DataSourceConfigDrive, (DataSource.DEP_FILESYSTEM, )),
-  (DataSourceConfigDriveNet,
-    (DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK)),
-)
-
-
-# return a list of data sources that match this set of dependencies
-def get_datasource_list(depends):
-    return(DataSource.list_from_depends(depends, datasources))
-
-if __name__ == "__main__":
-    def main():
-        import sys
-        import pprint
-        print cfg_drive_device()
-        (md, ud) = read_config_drive_dir(sys.argv[1])
-        print "=== md ==="
-        pprint.pprint(md)
-        print "=== ud ==="
-        print(ud)
-
-    main()
-
-# vi: ts=4 expandtab
diff --git a/cloudinit/DataSourceEc2.py b/cloudinit/DataSourceEc2.py
deleted file mode 100644
index 7051ecda..00000000
--- a/cloudinit/DataSourceEc2.py
+++ /dev/null
@@ -1,217 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.DataSource as DataSource
-
-from cloudinit import seeddir as base_seeddir
-from cloudinit import log
-import cloudinit.util as util
-import socket
-import time
-import boto.utils as boto_utils
-import os.path
-
-
-class DataSourceEc2(DataSource.DataSource):
-    api_ver = '2009-04-04'
-    seeddir = base_seeddir + '/ec2'
-    metadata_address = "http://169.254.169.254"
-
-    def __str__(self):
-        return("DataSourceEc2")
-
-    def get_data(self):
-        seedret = {}
-        if util.read_optional_seed(seedret, base=self.seeddir + "/"):
-            self.userdata_raw = seedret['user-data']
-            self.metadata = seedret['meta-data']
-            log.debug("using seeded ec2 data in %s" % self.seeddir)
-            return True
-
-        try:
-            if not self.wait_for_metadata_service():
-                return False
-            start = time.time()
-            self.userdata_raw = boto_utils.get_instance_userdata(self.api_ver,
-                None, self.metadata_address)
-            self.metadata = boto_utils.get_instance_metadata(self.api_ver,
-                self.metadata_address)
-            log.debug("crawl of metadata service took %ds" % (time.time() -
-                                                              start))
-            return True
-        except Exception as e:
-            print e
-            return False
-
-    def get_instance_id(self):
-        return(self.metadata['instance-id'])
-
-    def get_availability_zone(self):
-        return(self.metadata['placement']['availability-zone'])
-
-    def get_local_mirror(self):
-        return(self.get_mirror_from_availability_zone())
-
-    def get_mirror_from_availability_zone(self, availability_zone=None):
-        # availability is like 'us-west-1b' or 'eu-west-1a'
-        if availability_zone == None:
-            availability_zone = self.get_availability_zone()
-
-        fallback = None
-
-        if self.is_vpc():
-            return fallback
-
-        try:
-            host = "%s.ec2.archive.ubuntu.com" % availability_zone[:-1]
-            socket.getaddrinfo(host, None, 0, socket.SOCK_STREAM)
-            return 'http://%s/ubuntu/' % host
-        except:
-            return fallback
-
-    def wait_for_metadata_service(self):
-        mcfg = self.ds_cfg
-
-        if not hasattr(mcfg, "get"):
-            mcfg = {}
-
-        max_wait = 120
-        try:
-            max_wait = int(mcfg.get("max_wait", max_wait))
-        except Exception:
-            util.logexc(log)
-            log.warn("Failed to get max wait. using %s" % max_wait)
-
-        if max_wait == 0:
-            return False
-
-        timeout = 50
-        try:
-            timeout = int(mcfg.get("timeout", timeout))
-        except Exception:
-            util.logexc(log)
-            log.warn("Failed to get timeout, using %s" % timeout)
-
-        def_mdurls = ["http://169.254.169.254", "http://instance-data:8773"]
-        mdurls = mcfg.get("metadata_urls", def_mdurls)
-
-        # Remove addresses from the list that wont resolve.
-        filtered = [x for x in mdurls if util.is_resolvable_url(x)]
-
-        if set(filtered) != set(mdurls):
-            log.debug("removed the following from metadata urls: %s" %
-                list((set(mdurls) - set(filtered))))
-
-        if len(filtered):
-            mdurls = filtered
-        else:
-            log.warn("Empty metadata url list! using default list")
-            mdurls = def_mdurls
-
-        urls = []
-        url2base = {False: False}
-        for url in mdurls:
-            cur = "%s/%s/meta-data/instance-id" % (url, self.api_ver)
-            urls.append(cur)
-            url2base[cur] = url
-
-        starttime = time.time()
-        url = util.wait_for_url(urls=urls, max_wait=max_wait,
-                                timeout=timeout, status_cb=log.warn)
-
-        if url:
-            log.debug("Using metadata source: '%s'" % url2base[url])
-        else:
-            log.critical("giving up on md after %i seconds\n" %
-                         int(time.time() - starttime))
-
-        self.metadata_address = url2base[url]
-        return (bool(url))
-
-    def device_name_to_device(self, name):
-        # consult metadata service, that has
-        #  ephemeral0: sdb
-        # and return 'sdb' for input 'ephemeral0'
-        if 'block-device-mapping' not in self.metadata:
-            return(None)
-
-        found = None
-        for entname, device in self.metadata['block-device-mapping'].items():
-            if entname == name:
-                found = device
-                break
-            # LP: #513842 mapping in Euca has 'ephemeral' not 'ephemeral0'
-            if entname == "ephemeral" and name == "ephemeral0":
-                found = device
-        if found == None:
-            log.debug("unable to convert %s to a device" % name)
-            return None
-
-        # LP: #611137
-        # the metadata service may believe that devices are named 'sda'
-        # when the kernel named them 'vda' or 'xvda'
-        # we want to return the correct value for what will actually
-        # exist in this instance
-        mappings = {"sd": ("vd", "xvd")}
-        ofound = found
-        short = os.path.basename(found)
-
-        if not found.startswith("/"):
-            found = "/dev/%s" % found
-
-        if os.path.exists(found):
-            return(found)
-
-        for nfrom, tlist in mappings.items():
-            if not short.startswith(nfrom):
-                continue
-            for nto in tlist:
-                cand = "/dev/%s%s" % (nto, short[len(nfrom):])
-                if os.path.exists(cand):
-                    log.debug("remapped device name %s => %s" % (found, cand))
-                    return(cand)
-
-        # on t1.micro, ephemeral0 will appear in block-device-mapping from
-        # metadata, but it will not exist on disk (and never will)
-        # at this pint, we've verified that the path did not exist
-        # in the special case of 'ephemeral0' return None to avoid bogus
-        # fstab entry (LP: #744019)
-        if name == "ephemeral0":
-            return None
-        return ofound
-
-    def is_vpc(self):
-        # per comment in LP: #615545
-        ph = "public-hostname"
-        p4 = "public-ipv4"
-        if ((ph not in self.metadata or self.metadata[ph] == "") and
-            (p4 not in self.metadata or self.metadata[p4] == "")):
-            return True
-        return False
-
-
-datasources = [
-  (DataSourceEc2, (DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK)),
-]
-
-
-# return a list of data sources that match this set of dependencies
-def get_datasource_list(depends):
-    return(DataSource.list_from_depends(depends, datasources))
diff --git a/cloudinit/DataSourceMAAS.py b/cloudinit/DataSourceMAAS.py
deleted file mode 100644
index e3e62057..00000000
--- a/cloudinit/DataSourceMAAS.py
+++ /dev/null
@@ -1,345 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2012 Canonical Ltd.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import cloudinit.DataSource as DataSource
-
-from cloudinit import seeddir as base_seeddir
-from cloudinit import log
-import cloudinit.util as util
-import errno
-import oauth.oauth as oauth
-import os.path
-import urllib2
-import time
-
-
-MD_VERSION = "2012-03-01"
-
-
-class DataSourceMAAS(DataSource.DataSource):
-    """
-    DataSourceMAAS reads instance information from MAAS.
-    Given a config metadata_url, and oauth tokens, it expects to find
-    files under the root named:
-      instance-id
-      user-data
-      hostname
-    """
-    seeddir = base_seeddir + '/maas'
-    baseurl = None
-
-    def __str__(self):
-        return("DataSourceMAAS[%s]" % self.baseurl)
-
-    def get_data(self):
-        mcfg = self.ds_cfg
-
-        try:
-            (userdata, metadata) = read_maas_seed_dir(self.seeddir)
-            self.userdata_raw = userdata
-            self.metadata = metadata
-            self.baseurl = self.seeddir
-            return True
-        except MAASSeedDirNone:
-            pass
-        except MAASSeedDirMalformed as exc:
-            log.warn("%s was malformed: %s\n" % (self.seeddir, exc))
-            raise
-
-        try:
-            # if there is no metadata_url, then we're not configured
-            url = mcfg.get('metadata_url', None)
-            if url == None:
-                return False
-
-            if not self.wait_for_metadata_service(url):
-                return False
-
-            self.baseurl = url
-
-            (userdata, metadata) = read_maas_seed_url(self.baseurl,
-                self.md_headers)
-            self.userdata_raw = userdata
-            self.metadata = metadata
-            return True
-        except Exception:
-            util.logexc(log)
-            return False
-
-    def md_headers(self, url):
-        mcfg = self.ds_cfg
-
-        # if we are missing token_key, token_secret or consumer_key
-        # then just do non-authed requests
-        for required in ('token_key', 'token_secret', 'consumer_key'):
-            if required not in mcfg:
-                return({})
-
-        consumer_secret = mcfg.get('consumer_secret', "")
-
-        return(oauth_headers(url=url, consumer_key=mcfg['consumer_key'],
-            token_key=mcfg['token_key'], token_secret=mcfg['token_secret'],
-            consumer_secret=consumer_secret))
-
-    def wait_for_metadata_service(self, url):
-        mcfg = self.ds_cfg
-
-        max_wait = 120
-        try:
-            max_wait = int(mcfg.get("max_wait", max_wait))
-        except Exception:
-            util.logexc(log)
-            log.warn("Failed to get max wait. using %s" % max_wait)
-
-        if max_wait == 0:
-            return False
-
-        timeout = 50
-        try:
-            timeout = int(mcfg.get("timeout", timeout))
-        except Exception:
-            util.logexc(log)
-            log.warn("Failed to get timeout, using %s" % timeout)
-
-        starttime = time.time()
-        check_url = "%s/%s/meta-data/instance-id" % (url, MD_VERSION)
-        url = util.wait_for_url(urls=[check_url], max_wait=max_wait,
-            timeout=timeout, status_cb=log.warn,
-            headers_cb=self.md_headers)
-
-        if url:
-            log.debug("Using metadata source: '%s'" % url)
-        else:
-            log.critical("giving up on md after %i seconds\n" %
-                         int(time.time() - starttime))
-
-        return (bool(url))
-
-
-def read_maas_seed_dir(seed_d):
-    """
-    Return user-data and metadata for a maas seed dir in seed_d.
-    Expected format of seed_d are the following files:
-      * instance-id
-      * local-hostname
-      * user-data
-    """
-    files = ('local-hostname', 'instance-id', 'user-data', 'public-keys')
-    md = {}
-
-    if not os.path.isdir(seed_d):
-        raise MAASSeedDirNone("%s: not a directory")
-
-    for fname in files:
-        try:
-            with open(os.path.join(seed_d, fname)) as fp:
-                md[fname] = fp.read()
-                fp.close()
-        except IOError as e:
-            if e.errno != errno.ENOENT:
-                raise
-
-    return(check_seed_contents(md, seed_d))
-
-
-def read_maas_seed_url(seed_url, header_cb=None, timeout=None,
-    version=MD_VERSION):
-    """
-    Read the maas datasource at seed_url.
-    header_cb is a method that should return a headers dictionary that will
-    be given to urllib2.Request()
-
-    Expected format of seed_url is are the following files:
-      * <seed_url>/<version>/meta-data/instance-id
-      * <seed_url>/<version>/meta-data/local-hostname
-      * <seed_url>/<version>/user-data
-    """
-    files = ('meta-data/local-hostname',
-             'meta-data/instance-id',
-             'meta-data/public-keys',
-             'user-data')
-
-    base_url = "%s/%s" % (seed_url, version)
-    md = {}
-    for fname in files:
-        url = "%s/%s" % (base_url, fname)
-        if header_cb:
-            headers = header_cb(url)
-        else:
-            headers = {}
-
-        try:
-            req = urllib2.Request(url, data=None, headers=headers)
-            resp = urllib2.urlopen(req, timeout=timeout)
-            md[os.path.basename(fname)] = resp.read()
-        except urllib2.HTTPError as e:
-            if e.code != 404:
-                raise
-
-    return(check_seed_contents(md, seed_url))
-
-
-def check_seed_contents(content, seed):
-    """Validate if content is Is the content a dict that is valid as a
-       return for a datasource.
-       Either return a (userdata, metadata) tuple or
-       Raise MAASSeedDirMalformed or MAASSeedDirNone
-    """
-    md_required = ('instance-id', 'local-hostname')
-    found = content.keys()
-
-    if len(content) == 0:
-        raise MAASSeedDirNone("%s: no data files found" % seed)
-
-    missing = [k for k in md_required if k not in found]
-    if len(missing):
-        raise MAASSeedDirMalformed("%s: missing files %s" % (seed, missing))
-
-    userdata = content.get('user-data', "")
-    md = {}
-    for (key, val) in content.iteritems():
-        if key == 'user-data':
-            continue
-        md[key] = val
-
-    return(userdata, md)
-
-
-def oauth_headers(url, consumer_key, token_key, token_secret, consumer_secret):
-    consumer = oauth.OAuthConsumer(consumer_key, consumer_secret)
-    token = oauth.OAuthToken(token_key, token_secret)
-    params = {
-        'oauth_version': "1.0",
-        'oauth_nonce': oauth.generate_nonce(),
-        'oauth_timestamp': int(time.time()),
-        'oauth_token': token.key,
-        'oauth_consumer_key': consumer.key,
-    }
-    req = oauth.OAuthRequest(http_url=url, parameters=params)
-    req.sign_request(oauth.OAuthSignatureMethod_PLAINTEXT(),
-        consumer, token)
-    return(req.to_header())
-
-
-class MAASSeedDirNone(Exception):
-    pass
-
-
-class MAASSeedDirMalformed(Exception):
-    pass
-
-
-datasources = [
-  (DataSourceMAAS, (DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK)),
-]
-
-
-# return a list of data sources that match this set of dependencies
-def get_datasource_list(depends):
-    return(DataSource.list_from_depends(depends, datasources))
-
-
-if __name__ == "__main__":
-    def main():
-        """
-        Call with single argument of directory or http or https url.
-        If url is given additional arguments are allowed, which will be
-        interpreted as consumer_key, token_key, token_secret, consumer_secret
-        """
-        import argparse
-        import pprint
-
-        parser = argparse.ArgumentParser(description='Interact with MAAS DS')
-        parser.add_argument("--config", metavar="file",
-            help="specify DS config file", default=None)
-        parser.add_argument("--ckey", metavar="key",
-            help="the consumer key to auth with", default=None)
-        parser.add_argument("--tkey", metavar="key",
-            help="the token key to auth with", default=None)
-        parser.add_argument("--csec", metavar="secret",
-            help="the consumer secret (likely '')", default="")
-        parser.add_argument("--tsec", metavar="secret",
-            help="the token secret to auth with", default=None)
-        parser.add_argument("--apiver", metavar="version",
-            help="the apiver to use ("" can be used)", default=MD_VERSION)
-
-        subcmds = parser.add_subparsers(title="subcommands", dest="subcmd")
-        subcmds.add_parser('crawl', help="crawl the datasource")
-        subcmds.add_parser('get', help="do a single GET of provided url")
-        subcmds.add_parser('check-seed', help="read andn verify seed at url")
-
-        parser.add_argument("url", help="the data source to query")
-
-        args = parser.parse_args()
-
-        creds = {'consumer_key': args.ckey, 'token_key': args.tkey,
-            'token_secret': args.tsec, 'consumer_secret': args.csec}
-
-        if args.config:
-            import yaml
-            with open(args.config) as fp:
-                cfg = yaml.safe_load(fp)
-            if 'datasource' in cfg:
-                cfg = cfg['datasource']['MAAS']
-            for key in creds.keys():
-                if key in cfg and creds[key] == None:
-                    creds[key] = cfg[key]
-
-        def geturl(url, headers_cb):
-            req = urllib2.Request(url, data=None, headers=headers_cb(url))
-            return(urllib2.urlopen(req).read())
-
-        def printurl(url, headers_cb):
-            print "== %s ==\n%s\n" % (url, geturl(url, headers_cb))
-
-        def crawl(url, headers_cb=None):
-            if url.endswith("/"):
-                for line in geturl(url, headers_cb).splitlines():
-                    if line.endswith("/"):
-                        crawl("%s%s" % (url, line), headers_cb)
-                    else:
-                        printurl("%s%s" % (url, line), headers_cb)
-            else:
-                printurl(url, headers_cb)
-
-        def my_headers(url):
-            headers = {}
-            if creds.get('consumer_key', None) != None:
-                headers = oauth_headers(url, **creds)
-            return headers
-
-        if args.subcmd == "check-seed":
-            if args.url.startswith("http"):
-                (userdata, metadata) = read_maas_seed_url(args.url,
-                    header_cb=my_headers, version=args.apiver)
-            else:
-                (userdata, metadata) = read_maas_seed_url(args.url)
-            print "=== userdata ==="
-            print userdata
-            print "=== metadata ==="
-            pprint.pprint(metadata)
-
-        elif args.subcmd == "get":
-            printurl(args.url, my_headers)
-
-        elif args.subcmd == "crawl":
-            if not args.url.endswith("/"):
-                args.url = "%s/" % args.url
-            crawl(args.url, my_headers)
-
-    main()
diff --git a/cloudinit/SshUtil.py b/cloudinit/SshUtil.py
deleted file mode 100644
index a081fbe8..00000000
--- a/cloudinit/SshUtil.py
+++ /dev/null
@@ -1,227 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2012 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import os.path
-import cloudinit.util as util
-
-
-class AuthKeyEntry():
-    # lines are options, keytype, base64-encoded key, comment
-    # man page says the following which I did not understand:
-    #   The options field is optional; its presence is determined by whether
-    #   the line starts with a number or not (the options field never starts
-    #   with a number)
-    options = None
-    keytype = None
-    base64 = None
-    comment = None
-    is_comment = False
-    line_in = ""
-
-    def __init__(self, line, def_opt=None):
-        line = line.rstrip("\n\r")
-        self.line_in = line
-        if line.startswith("#") or line.strip() == "":
-            self.is_comment = True
-        else:
-            ent = line.strip()
-            toks = ent.split(None, 3)
-            if len(toks) == 1:
-                self.base64 = toks[0]
-            elif len(toks) == 2:
-                (self.base64, self.comment) = toks
-            elif len(toks) == 3:
-                (self.keytype, self.base64, self.comment) = toks
-            elif len(toks) == 4:
-                i = 0
-                ent = line.strip()
-                quoted = False
-                # taken from auth_rsa_key_allowed in auth-rsa.c
-                try:
-                    while (i < len(ent) and
-                           ((quoted) or (ent[i] not in (" ", "\t")))):
-                        curc = ent[i]
-                        nextc = ent[i + 1]
-                        if curc == "\\" and nextc == '"':
-                            i = i + 1
-                        elif curc == '"':
-                            quoted = not quoted
-                        i = i + 1
-                except IndexError:
-                    self.is_comment = True
-                    return
-
-                try:
-                    self.options = ent[0:i]
-                    (self.keytype, self.base64, self.comment) = \
-                        ent[i + 1:].split(None, 3)
-                except ValueError:
-                    # we did not understand this line
-                    self.is_comment = True
-
-        if self.options == None and def_opt:
-            self.options = def_opt
-
-        return
-
-    def debug(self):
-        print("line_in=%s\ncomment: %s\noptions=%s\nkeytype=%s\nbase64=%s\n"
-              "comment=%s\n" % (self.line_in, self.is_comment, self.options,
-                                self.keytype, self.base64, self.comment)),
-
-    def __repr__(self):
-        if self.is_comment:
-            return(self.line_in)
-        else:
-            toks = []
-            for e in (self.options, self.keytype, self.base64, self.comment):
-                if e:
-                    toks.append(e)
-
-            return(' '.join(toks))
-
-
-def update_authorized_keys(fname, keys):
-    # keys is a list of AuthKeyEntries
-    # key_prefix is the prefix (options) to prepend
-    try:
-        fp = open(fname, "r")
-        lines = fp.readlines()  # lines have carriage return
-        fp.close()
-    except IOError:
-        lines = []
-
-    ka_stats = {}  # keys_added status
-    for k in keys:
-        ka_stats[k] = False
-
-    to_add = []
-    for key in keys:
-        to_add.append(key)
-
-    for i in range(0, len(lines)):
-        ent = AuthKeyEntry(lines[i])
-        for k in keys:
-            if k.base64 == ent.base64 and not k.is_comment:
-                ent = k
-                try:
-                    to_add.remove(k)
-                except ValueError:
-                    pass
-        lines[i] = str(ent)
-
-    # now append any entries we did not match above
-    for key in to_add:
-        lines.append(str(key))
-
-    if len(lines) == 0:
-        return("")
-    else:
-        return('\n'.join(lines) + "\n")
-
-
-def setup_user_keys(keys, user, key_prefix, log=None):
-    import pwd
-    saved_umask = os.umask(077)
-
-    pwent = pwd.getpwnam(user)
-
-    ssh_dir = '%s/.ssh' % pwent.pw_dir
-    if not os.path.exists(ssh_dir):
-        os.mkdir(ssh_dir)
-        os.chown(ssh_dir, pwent.pw_uid, pwent.pw_gid)
-
-    try:
-        ssh_cfg = parse_ssh_config()
-        akeys = ssh_cfg.get("AuthorizedKeysFile", "%h/.ssh/authorized_keys")
-        akeys = akeys.replace("%h", pwent.pw_dir)
-        akeys = akeys.replace("%u", user)
-        if not akeys.startswith('/'):
-            akeys = os.path.join(pwent.pw_dir, akeys)
-        authorized_keys = akeys
-    except Exception:
-        authorized_keys = '%s/.ssh/authorized_keys' % pwent.pw_dir
-        if log:
-            util.logexc(log)
-
-    key_entries = []
-    for k in keys:
-        ke = AuthKeyEntry(k, def_opt=key_prefix)
-        key_entries.append(ke)
-
-    content = update_authorized_keys(authorized_keys, key_entries)
-    util.write_file(authorized_keys, content, 0600)
-
-    os.chown(authorized_keys, pwent.pw_uid, pwent.pw_gid)
-    util.restorecon_if_possible(ssh_dir, recursive=True)
-
-    os.umask(saved_umask)
-
-
-def parse_ssh_config(fname="/etc/ssh/sshd_config"):
-    ret = {}
-    fp = open(fname)
-    for l in fp.readlines():
-        l = l.strip()
-        if not l or l.startswith("#"):
-            continue
-        key, val = l.split(None, 1)
-        ret[key] = val
-    fp.close()
-    return(ret)
-
-if __name__ == "__main__":
-    def main():
-        import sys
-        # usage: orig_file, new_keys, [key_prefix]
-        #   prints out merged, where 'new_keys' will trump old
-        ##  example
-        ## ### begin auth_keys ###
-        # ssh-rsa AAAAB3NzaC1xxxxxxxxxV3csgm8cJn7UveKHkYjJp8= smoser-work
-        # ssh-rsa AAAAB3NzaC1xxxxxxxxxCmXp5Kt5/82cD/VN3NtHw== smoser@brickies
-        # ### end authorized_keys ###
-        #
-        # ### begin new_keys ###
-        # ssh-rsa nonmatch smoser@newhost
-        # ssh-rsa AAAAB3NzaC1xxxxxxxxxV3csgm8cJn7UveKHkYjJp8= new_comment
-        # ### end new_keys ###
-        #
-        # Then run as:
-        #  program auth_keys new_keys \
-        #      'no-port-forwarding,command=\"echo hi world;\"'
-        def_prefix = None
-        orig_key_file = sys.argv[1]
-        new_key_file = sys.argv[2]
-        if len(sys.argv) > 3:
-            def_prefix = sys.argv[3]
-        fp = open(new_key_file)
-
-        newkeys = []
-        for line in fp.readlines():
-            newkeys.append(AuthKeyEntry(line, def_prefix))
-
-        fp.close()
-        print update_authorized_keys(orig_key_file, newkeys)
-
-    main()
-
-# vi: ts=4 expandtab
diff --git a/cloudinit/UserDataHandler.py b/cloudinit/UserDataHandler.py
deleted file mode 100644
index bf694a8e..00000000
--- a/cloudinit/UserDataHandler.py
+++ /dev/null
@@ -1,262 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import email
-
-from email.mime.multipart import MIMEMultipart
-from email.mime.text import MIMEText
-from email.mime.base import MIMEBase
-import yaml
-import cloudinit
-import cloudinit.util as util
-import hashlib
-import urllib
-
-
-starts_with_mappings = {
-    '#include': 'text/x-include-url',
-    '#include-once': 'text/x-include-once-url',
-    '#!': 'text/x-shellscript',
-    '#cloud-config': 'text/cloud-config',
-    '#upstart-job': 'text/upstart-job',
-    '#part-handler': 'text/part-handler',
-    '#cloud-boothook': 'text/cloud-boothook',
-    '#cloud-config-archive': 'text/cloud-config-archive',
-}
-
-
-# if 'string' is compressed return decompressed otherwise return it
-def decomp_str(string):
-    import StringIO
-    import gzip
-    try:
-        uncomp = gzip.GzipFile(None, "rb", 1, StringIO.StringIO(string)).read()
-        return(uncomp)
-    except:
-        return(string)
-
-
-def do_include(content, appendmsg):
-    import os
-    # is just a list of urls, one per line
-    # also support '#include <url here>'
-    includeonce = False
-    for line in content.splitlines():
-        if line == "#include":
-            continue
-        if line == "#include-once":
-            includeonce = True
-            continue
-        if line.startswith("#include-once"):
-            line = line[len("#include-once"):].lstrip()
-            includeonce = True
-        elif line.startswith("#include"):
-            line = line[len("#include"):].lstrip()
-        if line.startswith("#"):
-            continue
-        if line.strip() == "":
-            continue
-
-        # urls cannot not have leading or trailing white space
-        msum = hashlib.md5()  # pylint: disable=E1101
-        msum.update(line.strip())
-        includeonce_filename = "%s/urlcache/%s" % (
-            cloudinit.get_ipath_cur("data"), msum.hexdigest())
-        try:
-            if includeonce and os.path.isfile(includeonce_filename):
-                with open(includeonce_filename, "r") as fp:
-                    content = fp.read()
-            else:
-                content = urllib.urlopen(line).read()
-                if includeonce:
-                    util.write_file(includeonce_filename, content, mode=0600)
-        except Exception:
-            raise
-
-        process_includes(message_from_string(decomp_str(content)), appendmsg)
-
-
-def explode_cc_archive(archive, appendmsg):
-    for ent in yaml.safe_load(archive):
-        # ent can be one of:
-        #  dict { 'filename' : 'value', 'content' : 'value', 'type' : 'value' }
-        #    filename and type not be present
-        # or
-        #  scalar(payload)
-
-        def_type = "text/cloud-config"
-        if isinstance(ent, str):
-            ent = {'content': ent}
-
-        content = ent.get('content', '')
-        mtype = ent.get('type', None)
-        if mtype == None:
-            mtype = type_from_startswith(content, def_type)
-
-        maintype, subtype = mtype.split('/', 1)
-        if maintype == "text":
-            msg = MIMEText(content, _subtype=subtype)
-        else:
-            msg = MIMEBase(maintype, subtype)
-            msg.set_payload(content)
-
-        if 'filename' in ent:
-            msg.add_header('Content-Disposition', 'attachment',
-                           filename=ent['filename'])
-
-        for header in ent.keys():
-            if header in ('content', 'filename', 'type'):
-                continue
-            msg.add_header(header, ent['header'])
-
-        _attach_part(appendmsg, msg)
-
-
-def multi_part_count(outermsg, newcount=None):
-    """
-    Return the number of attachments to this MIMEMultipart by looking
-    at its 'Number-Attachments' header.
-    """
-    nfield = 'Number-Attachments'
-    if nfield not in outermsg:
-        outermsg[nfield] = "0"
-
-    if newcount != None:
-        outermsg.replace_header(nfield, str(newcount))
-
-    return(int(outermsg.get('Number-Attachments', 0)))
-
-
-def _attach_part(outermsg, part):
-    """
-    Attach an part to an outer message. outermsg must be a MIMEMultipart.
-    Modifies a header in outermsg to keep track of number of attachments.
-    """
-    cur = multi_part_count(outermsg)
-    if not part.get_filename(None):
-        part.add_header('Content-Disposition', 'attachment',
-            filename='part-%03d' % (cur + 1))
-    outermsg.attach(part)
-    multi_part_count(outermsg, cur + 1)
-
-
-def type_from_startswith(payload, default=None):
-    # slist is sorted longest first
-    slist = sorted(starts_with_mappings.keys(), key=lambda e: 0 - len(e))
-    for sstr in slist:
-        if payload.startswith(sstr):
-            return(starts_with_mappings[sstr])
-    return default
-
-
-def process_includes(msg, appendmsg=None):
-    if appendmsg == None:
-        appendmsg = MIMEMultipart()
-
-    for part in msg.walk():
-        # multipart/* are just containers
-        if part.get_content_maintype() == 'multipart':
-            continue
-
-        ctype = None
-        ctype_orig = part.get_content_type()
-
-        payload = part.get_payload(decode=True)
-
-        if ctype_orig in ("text/plain", "text/x-not-multipart"):
-            ctype = type_from_startswith(payload)
-
-        if ctype is None:
-            ctype = ctype_orig
-
-        if ctype in ('text/x-include-url', 'text/x-include-once-url'):
-            do_include(payload, appendmsg)
-            continue
-
-        if ctype == "text/cloud-config-archive":
-            explode_cc_archive(payload, appendmsg)
-            continue
-
-        if 'Content-Type' in msg:
-            msg.replace_header('Content-Type', ctype)
-        else:
-            msg['Content-Type'] = ctype
-
-        _attach_part(appendmsg, part)
-
-
-def message_from_string(data, headers=None):
-    if headers is None:
-        headers = {}
-    if "mime-version:" in data[0:4096].lower():
-        msg = email.message_from_string(data)
-        for (key, val) in headers.items():
-            if key in msg:
-                msg.replace_header(key, val)
-            else:
-                msg[key] = val
-    else:
-        mtype = headers.get("Content-Type", "text/x-not-multipart")
-        maintype, subtype = mtype.split("/", 1)
-        msg = MIMEBase(maintype, subtype, *headers)
-        msg.set_payload(data)
-
-    return(msg)
-
-
-# this is heavily wasteful, reads through userdata string input
-def preprocess_userdata(data):
-    newmsg = MIMEMultipart()
-    process_includes(message_from_string(decomp_str(data)), newmsg)
-    return(newmsg.as_string())
-
-
-# callback is a function that will be called with (data, content_type,
-# filename, payload)
-def walk_userdata(istr, callback, data=None):
-    partnum = 0
-    for part in message_from_string(istr).walk():
-        # multipart/* are just containers
-        if part.get_content_maintype() == 'multipart':
-            continue
-
-        ctype = part.get_content_type()
-        if ctype is None:
-            ctype = 'application/octet-stream'
-
-        filename = part.get_filename()
-        if not filename:
-            filename = 'part-%03d' % partnum
-
-        callback(data, ctype, filename, part.get_payload(decode=True))
-
-        partnum = partnum + 1
-
-
-if __name__ == "__main__":
-    def main():
-        import sys
-        data = decomp_str(file(sys.argv[1]).read())
-        newmsg = MIMEMultipart()
-        process_includes(message_from_string(data), newmsg)
-        print newmsg
-        print "#found %s parts" % multi_part_count(newmsg)
-
-    main()
diff --git a/cloudinit/__init__.py b/cloudinit/__init__.py
index dafb128a..da124641 100644
--- a/cloudinit/__init__.py
+++ b/cloudinit/__init__.py
@@ -1,11 +1,12 @@
 # vi: ts=4 expandtab
 #
-#    Common code for the EC2 initialisation scripts in Ubuntu
-#    Copyright (C) 2008-2009 Canonical Ltd
+#    Copyright (C) 2012 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
-#    Author: Soren Hansen <soren@canonical.com>
+#    Author: Scott Moser <scott.moser@canonical.com>
 #    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -18,650 +19,3 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-varlibdir = '/var/lib/cloud'
-cur_instance_link = varlibdir + "/instance"
-boot_finished = cur_instance_link + "/boot-finished"
-system_config = '/etc/cloud/cloud.cfg'
-seeddir = varlibdir + "/seed"
-cfg_env_name = "CLOUD_CFG"
-
-cfg_builtin = """
-log_cfgs: []
-datasource_list: ["NoCloud", "ConfigDrive", "OVF", "MAAS", "Ec2", "CloudStack"]
-def_log_file: /var/log/cloud-init.log
-syslog_fix_perms: syslog:adm
-"""
-logger_name = "cloudinit"
-
-pathmap = {
-   "handlers": "/handlers",
-   "scripts": "/scripts",
-   "sem": "/sem",
-   "boothooks": "/boothooks",
-   "userdata_raw": "/user-data.txt",
-   "userdata": "/user-data.txt.i",
-   "obj_pkl": "/obj.pkl",
-   "cloud_config": "/cloud-config.txt",
-   "data": "/data",
-   None: "",
-}
-
-per_instance = "once-per-instance"
-per_always = "always"
-per_once = "once"
-
-parsed_cfgs = {}
-
-import os
-
-import cPickle
-import sys
-import os.path
-import errno
-import subprocess
-import yaml
-import logging
-import logging.config
-import StringIO
-import glob
-import traceback
-
-import cloudinit.util as util
-
-
-class NullHandler(logging.Handler):
-    def emit(self, record):
-        pass
-
-
-log = logging.getLogger(logger_name)
-log.addHandler(NullHandler())
-
-
-def logging_set_from_cfg_file(cfg_file=system_config):
-    logging_set_from_cfg(util.get_base_cfg(cfg_file, cfg_builtin, parsed_cfgs))
-
-
-def logging_set_from_cfg(cfg):
-    log_cfgs = []
-    logcfg = util.get_cfg_option_str(cfg, "log_cfg", False)
-    if logcfg:
-        # if there is a 'logcfg' entry in the config, respect
-        # it, it is the old keyname
-        log_cfgs = [logcfg]
-    elif "log_cfgs" in cfg:
-        for cfg in cfg['log_cfgs']:
-            if isinstance(cfg, list):
-                log_cfgs.append('\n'.join(cfg))
-            else:
-                log_cfgs.append()
-
-    if not len(log_cfgs):
-        sys.stderr.write("Warning, no logging configured\n")
-        return
-
-    for logcfg in log_cfgs:
-        try:
-            logging.config.fileConfig(StringIO.StringIO(logcfg))
-            return
-        except:
-            pass
-
-    raise Exception("no valid logging found\n")
-
-
-import cloudinit.DataSource as DataSource
-import cloudinit.UserDataHandler as UserDataHandler
-
-
-class CloudInit:
-    cfg = None
-    part_handlers = {}
-    old_conffile = '/etc/ec2-init/ec2-config.cfg'
-    ds_deps = [DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK]
-    datasource = None
-    cloud_config_str = ''
-    datasource_name = ''
-
-    builtin_handlers = []
-
-    def __init__(self, ds_deps=None, sysconfig=system_config):
-        self.builtin_handlers = [
-            ['text/x-shellscript', self.handle_user_script, per_always],
-            ['text/cloud-config', self.handle_cloud_config, per_always],
-            ['text/upstart-job', self.handle_upstart_job, per_instance],
-            ['text/cloud-boothook', self.handle_cloud_boothook, per_always],
-        ]
-
-        if ds_deps != None:
-            self.ds_deps = ds_deps
-
-        self.sysconfig = sysconfig
-
-        self.cfg = self.read_cfg()
-
-    def read_cfg(self):
-        if self.cfg:
-            return(self.cfg)
-
-        try:
-            conf = util.get_base_cfg(self.sysconfig, cfg_builtin, parsed_cfgs)
-        except Exception:
-            conf = get_builtin_cfg()
-
-        # support reading the old ConfigObj format file and merging
-        # it into the yaml dictionary
-        try:
-            from configobj import ConfigObj
-            oldcfg = ConfigObj(self.old_conffile)
-            if oldcfg is None:
-                oldcfg = {}
-            conf = util.mergedict(conf, oldcfg)
-        except:
-            pass
-
-        return(conf)
-
-    def restore_from_cache(self):
-        try:
-            # we try to restore from a current link and static path
-            # by using the instance link, if purge_cache was called
-            # the file wont exist
-            cache = get_ipath_cur('obj_pkl')
-            f = open(cache, "rb")
-            data = cPickle.load(f)
-            f.close()
-            self.datasource = data
-            return True
-        except:
-            return False
-
-    def write_to_cache(self):
-        cache = self.get_ipath("obj_pkl")
-        try:
-            os.makedirs(os.path.dirname(cache))
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                return False
-
-        try:
-            f = open(cache, "wb")
-            cPickle.dump(self.datasource, f)
-            f.close()
-            os.chmod(cache, 0400)
-        except:
-            raise
-
-    def get_data_source(self):
-        if self.datasource is not None:
-            return True
-
-        if self.restore_from_cache():
-            log.debug("restored from cache type %s" % self.datasource)
-            return True
-
-        cfglist = self.cfg['datasource_list']
-        dslist = list_sources(cfglist, self.ds_deps)
-        dsnames = [f.__name__ for f in dslist]
-
-        log.debug("searching for data source in %s" % dsnames)
-        for cls in dslist:
-            ds = cls.__name__
-            try:
-                s = cls(sys_cfg=self.cfg)
-                if s.get_data():
-                    self.datasource = s
-                    self.datasource_name = ds
-                    log.debug("found data source %s" % ds)
-                    return True
-            except Exception as e:
-                log.warn("get_data of %s raised %s" % (ds, e))
-                util.logexc(log)
-        msg = "Did not find data source. searched classes: %s" % dsnames
-        log.debug(msg)
-        raise DataSourceNotFoundException(msg)
-
-    def set_cur_instance(self):
-        try:
-            os.unlink(cur_instance_link)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise
-
-        iid = self.get_instance_id()
-        os.symlink("./instances/%s" % iid, cur_instance_link)
-        idir = self.get_ipath()
-        dlist = []
-        for d in ["handlers", "scripts", "sem"]:
-            dlist.append("%s/%s" % (idir, d))
-
-        util.ensure_dirs(dlist)
-
-        ds = "%s: %s\n" % (self.datasource.__class__, str(self.datasource))
-        dp = self.get_cpath('data')
-        util.write_file("%s/%s" % (idir, 'datasource'), ds)
-        util.write_file("%s/%s" % (dp, 'previous-datasource'), ds)
-        util.write_file("%s/%s" % (dp, 'previous-instance-id'), "%s\n" % iid)
-
-    def get_userdata(self):
-        return(self.datasource.get_userdata())
-
-    def get_userdata_raw(self):
-        return(self.datasource.get_userdata_raw())
-
-    def get_instance_id(self):
-        return(self.datasource.get_instance_id())
-
-    def update_cache(self):
-        self.write_to_cache()
-        self.store_userdata()
-
-    def store_userdata(self):
-        util.write_file(self.get_ipath('userdata_raw'),
-            self.datasource.get_userdata_raw(), 0600)
-        util.write_file(self.get_ipath('userdata'),
-            self.datasource.get_userdata(), 0600)
-
-    def sem_getpath(self, name, freq):
-        if freq == 'once-per-instance':
-            return("%s/%s" % (self.get_ipath("sem"), name))
-        return("%s/%s.%s" % (get_cpath("sem"), name, freq))
-
-    def sem_has_run(self, name, freq):
-        if freq == per_always:
-            return False
-        semfile = self.sem_getpath(name, freq)
-        if os.path.exists(semfile):
-            return True
-        return False
-
-    def sem_acquire(self, name, freq):
-        from time import time
-        semfile = self.sem_getpath(name, freq)
-
-        try:
-            os.makedirs(os.path.dirname(semfile))
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                raise e
-
-        if os.path.exists(semfile) and freq != per_always:
-            return False
-
-        # race condition
-        try:
-            f = open(semfile, "w")
-            f.write("%s\n" % str(time()))
-            f.close()
-        except:
-            return(False)
-        return(True)
-
-    def sem_clear(self, name, freq):
-        semfile = self.sem_getpath(name, freq)
-        try:
-            os.unlink(semfile)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                return False
-
-        return True
-
-    # acquire lock on 'name' for given 'freq'
-    # if that does not exist, then call 'func' with given 'args'
-    # if 'clear_on_fail' is True and func throws an exception
-    #  then remove the lock (so it would run again)
-    def sem_and_run(self, semname, freq, func, args=None, clear_on_fail=False):
-        if args is None:
-            args = []
-        if self.sem_has_run(semname, freq):
-            log.debug("%s already ran %s", semname, freq)
-            return False
-        try:
-            if not self.sem_acquire(semname, freq):
-                raise Exception("Failed to acquire lock on %s" % semname)
-
-            func(*args)
-        except:
-            if clear_on_fail:
-                self.sem_clear(semname, freq)
-            raise
-
-        return True
-
-    # get_ipath : get the instance path for a name in pathmap
-    # (/var/lib/cloud/instances/<instance>/name)<name>)
-    def get_ipath(self, name=None):
-        return("%s/instances/%s%s"
-               % (varlibdir, self.get_instance_id(), pathmap[name]))
-
-    def consume_userdata(self, frequency=per_instance):
-        self.get_userdata()
-        data = self
-
-        cdir = get_cpath("handlers")
-        idir = self.get_ipath("handlers")
-
-        # add the path to the plugins dir to the top of our list for import
-        # instance dir should be read before cloud-dir
-        sys.path.insert(0, cdir)
-        sys.path.insert(0, idir)
-
-        part_handlers = {}
-        # add handlers in cdir
-        for fname in glob.glob("%s/*.py" % cdir):
-            if not os.path.isfile(fname):
-                continue
-            modname = os.path.basename(fname)[0:-3]
-            try:
-                mod = __import__(modname)
-                handler_register(mod, part_handlers, data, frequency)
-                log.debug("added handler for [%s] from %s" % (mod.list_types(),
-                                                              fname))
-            except:
-                log.warn("failed to initialize handler in %s" % fname)
-                util.logexc(log)
-
-        # add the internal handers if their type hasn't been already claimed
-        for (btype, bhand, bfreq) in self.builtin_handlers:
-            if btype in part_handlers:
-                continue
-            handler_register(InternalPartHandler(bhand, [btype], bfreq),
-                part_handlers, data, frequency)
-
-        # walk the data
-        pdata = {'handlers': part_handlers, 'handlerdir': idir,
-                 'data': data, 'frequency': frequency}
-        UserDataHandler.walk_userdata(self.get_userdata(),
-            partwalker_callback, data=pdata)
-
-        # give callbacks opportunity to finalize
-        called = []
-        for (_mtype, mod) in part_handlers.iteritems():
-            if mod in called:
-                continue
-            handler_call_end(mod, data, frequency)
-
-    def handle_user_script(self, _data, ctype, filename, payload, _frequency):
-        if ctype == "__end__":
-            return
-        if ctype == "__begin__":
-            # maybe delete existing things here
-            return
-
-        filename = filename.replace(os.sep, '_')
-        scriptsdir = get_ipath_cur('scripts')
-        util.write_file("%s/%s" %
-            (scriptsdir, filename), util.dos2unix(payload), 0700)
-
-    def handle_upstart_job(self, _data, ctype, filename, payload, frequency):
-        # upstart jobs are only written on the first boot
-        if frequency != per_instance:
-            return
-
-        if ctype == "__end__" or ctype == "__begin__":
-            return
-        if not filename.endswith(".conf"):
-            filename = filename + ".conf"
-
-        util.write_file("%s/%s" % ("/etc/init", filename),
-            util.dos2unix(payload), 0644)
-
-    def handle_cloud_config(self, _data, ctype, filename, payload, _frequency):
-        if ctype == "__begin__":
-            self.cloud_config_str = ""
-            return
-        if ctype == "__end__":
-            cloud_config = self.get_ipath("cloud_config")
-            util.write_file(cloud_config, self.cloud_config_str, 0600)
-
-            ## this could merge the cloud config with the system config
-            ## for now, not doing this as it seems somewhat circular
-            ## as CloudConfig does that also, merging it with this cfg
-            ##
-            # ccfg = yaml.safe_load(self.cloud_config_str)
-            # if ccfg is None: ccfg = {}
-            # self.cfg = util.mergedict(ccfg, self.cfg)
-
-            return
-
-        self.cloud_config_str += "\n#%s\n%s" % (filename, payload)
-
-    def handle_cloud_boothook(self, _data, ctype, filename, payload,
-                              _frequency):
-        if ctype == "__end__":
-            return
-        if ctype == "__begin__":
-            return
-
-        filename = filename.replace(os.sep, '_')
-        payload = util.dos2unix(payload)
-        prefix = "#cloud-boothook"
-        start = 0
-        if payload.startswith(prefix):
-            start = len(prefix) + 1
-
-        boothooks_dir = self.get_ipath("boothooks")
-        filepath = "%s/%s" % (boothooks_dir, filename)
-        util.write_file(filepath, payload[start:], 0700)
-        try:
-            env = os.environ.copy()
-            env['INSTANCE_ID'] = self.datasource.get_instance_id()
-            subprocess.check_call([filepath], env=env)
-        except subprocess.CalledProcessError as e:
-            log.error("boothooks script %s returned %i" %
-                (filepath, e.returncode))
-        except Exception as e:
-            log.error("boothooks unknown exception %s when running %s" %
-                (e, filepath))
-
-    def get_public_ssh_keys(self):
-        return(self.datasource.get_public_ssh_keys())
-
-    def get_locale(self):
-        return(self.datasource.get_locale())
-
-    def get_mirror(self):
-        return(self.datasource.get_local_mirror())
-
-    def get_hostname(self, fqdn=False):
-        return(self.datasource.get_hostname(fqdn=fqdn))
-
-    def device_name_to_device(self, name):
-        return(self.datasource.device_name_to_device(name))
-
-    # I really don't know if this should be here or not, but
-    # I needed it in cc_update_hostname, where that code had a valid 'cloud'
-    # reference, but did not have a cloudinit handle
-    # (ie, no cloudinit.get_cpath())
-    def get_cpath(self, name=None):
-        return(get_cpath(name))
-
-
-def initfs():
-    subds = ['scripts/per-instance', 'scripts/per-once', 'scripts/per-boot',
-             'seed', 'instances', 'handlers', 'sem', 'data']
-    dlist = []
-    for subd in subds:
-        dlist.append("%s/%s" % (varlibdir, subd))
-    util.ensure_dirs(dlist)
-
-    cfg = util.get_base_cfg(system_config, cfg_builtin, parsed_cfgs)
-    log_file = util.get_cfg_option_str(cfg, 'def_log_file', None)
-    perms = util.get_cfg_option_str(cfg, 'syslog_fix_perms', None)
-    if log_file:
-        fp = open(log_file, "ab")
-        fp.close()
-    if log_file and perms:
-        (u, g) = perms.split(':', 1)
-        if u == "-1" or u == "None":
-            u = None
-        if g == "-1" or g == "None":
-            g = None
-        util.chownbyname(log_file, u, g)
-
-
-def purge_cache(rmcur=True):
-    rmlist = [boot_finished]
-    if rmcur:
-        rmlist.append(cur_instance_link)
-    for f in rmlist:
-        try:
-            os.unlink(f)
-        except OSError as e:
-            if e.errno == errno.ENOENT:
-                continue
-            return(False)
-        except:
-            return(False)
-    return(True)
-
-
-# get_ipath_cur: get the current instance path for an item
-def get_ipath_cur(name=None):
-    return("%s/%s%s" % (varlibdir, "instance", pathmap[name]))
-
-
-# get_cpath : get the "clouddir" (/var/lib/cloud/<name>)
-# for a name in dirmap
-def get_cpath(name=None):
-    return("%s%s" % (varlibdir, pathmap[name]))
-
-
-def get_base_cfg(cfg_path=None):
-    if cfg_path is None:
-        cfg_path = system_config
-    return(util.get_base_cfg(cfg_path, cfg_builtin, parsed_cfgs))
-
-
-def get_builtin_cfg():
-    return(yaml.safe_load(cfg_builtin))
-
-
-class DataSourceNotFoundException(Exception):
-    pass
-
-
-def list_sources(cfg_list, depends):
-    return(DataSource.list_sources(cfg_list, depends, ["cloudinit", ""]))
-
-
-def handler_register(mod, part_handlers, data, frequency=per_instance):
-    if not hasattr(mod, "handler_version"):
-        setattr(mod, "handler_version", 1)
-
-    for mtype in mod.list_types():
-        part_handlers[mtype] = mod
-
-    handler_call_begin(mod, data, frequency)
-    return(mod)
-
-
-def handler_call_begin(mod, data, frequency):
-    handler_handle_part(mod, data, "__begin__", None, None, frequency)
-
-
-def handler_call_end(mod, data, frequency):
-    handler_handle_part(mod, data, "__end__", None, None, frequency)
-
-
-def handler_handle_part(mod, data, ctype, filename, payload, frequency):
-    # only add the handler if the module should run
-    modfreq = getattr(mod, "frequency", per_instance)
-    if not (modfreq == per_always or
-            (frequency == per_instance and modfreq == per_instance)):
-        return
-    try:
-        if mod.handler_version == 1:
-            mod.handle_part(data, ctype, filename, payload)
-        else:
-            mod.handle_part(data, ctype, filename, payload, frequency)
-    except:
-        util.logexc(log)
-        traceback.print_exc(file=sys.stderr)
-
-
-def partwalker_handle_handler(pdata, _ctype, _filename, payload):
-    curcount = pdata['handlercount']
-    modname = 'part-handler-%03d' % curcount
-    frequency = pdata['frequency']
-
-    modfname = modname + ".py"
-    util.write_file("%s/%s" % (pdata['handlerdir'], modfname), payload, 0600)
-
-    try:
-        mod = __import__(modname)
-        handler_register(mod, pdata['handlers'], pdata['data'], frequency)
-        pdata['handlercount'] = curcount + 1
-    except:
-        util.logexc(log)
-        traceback.print_exc(file=sys.stderr)
-
-
-def partwalker_callback(pdata, ctype, filename, payload):
-    # data here is the part_handlers array and then the data to pass through
-    if ctype == "text/part-handler":
-        if 'handlercount' not in pdata:
-            pdata['handlercount'] = 0
-        partwalker_handle_handler(pdata, ctype, filename, payload)
-        return
-    if ctype not in pdata['handlers'] and payload:
-        if ctype == "text/x-not-multipart":
-            # Extract the first line or 24 bytes for displaying in the log
-            start = payload.split("\n", 1)[0][:24]
-            if start < payload:
-                details = "starting '%s...'" % start.encode("string-escape")
-            else:
-                details = repr(payload)
-            log.warning("Unhandled non-multipart userdata %s", details)
-        return
-    handler_handle_part(pdata['handlers'][ctype], pdata['data'],
-        ctype, filename, payload, pdata['frequency'])
-
-
-class InternalPartHandler:
-    freq = per_instance
-    mtypes = []
-    handler_version = 1
-    handler = None
-
-    def __init__(self, handler, mtypes, frequency, version=2):
-        self.handler = handler
-        self.mtypes = mtypes
-        self.frequency = frequency
-        self.handler_version = version
-
-    def __repr__(self):
-        return("InternalPartHandler: [%s]" % self.mtypes)
-
-    def list_types(self):
-        return(self.mtypes)
-
-    def handle_part(self, data, ctype, filename, payload, frequency):
-        return(self.handler(data, ctype, filename, payload, frequency))
-
-
-def get_cmdline_url(names=('cloud-config-url', 'url'),
-                    starts="#cloud-config", cmdline=None):
-
-    if cmdline == None:
-        cmdline = util.get_cmdline()
-
-    data = util.keyval_str_to_dict(cmdline)
-    url = None
-    key = None
-    for key in names:
-        if key in data:
-            url = data[key]
-            break
-    if url == None:
-        return (None, None, None)
-
-    contents = util.readurl(url)
-
-    if contents.startswith(starts):
-        return (key, url, contents)
-
-    return (key, url, None)
diff --git a/cloudinit/cloud.py b/cloudinit/cloud.py
new file mode 100644
index 00000000..22d9167e
--- /dev/null
+++ b/cloudinit/cloud.py
@@ -0,0 +1,101 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import copy
+import os
+
+from cloudinit import log as logging
+
+LOG = logging.getLogger(__name__)
+
+# This class is the high level wrapper that provides
+# access to cloud-init objects without exposing the stage objects
+# to handler and or module manipulation. It allows for cloud
+# init to restrict what those types of user facing code may see
+# and or adjust (which helps avoid code messing with each other)
+#
+# It also provides util functions that avoid having to know
+# how to get a certain member from this submembers as well
+# as providing a backwards compatible object that can be maintained
+# while the stages/other objects can be worked on independently...
+
+
+class Cloud(object):
+    def __init__(self, datasource, paths, cfg, distro, runners):
+        self.datasource = datasource
+        self.paths = paths
+        self.distro = distro
+        self._cfg = cfg
+        self._runners = runners
+
+    # If a 'user' manipulates logging or logging services
+    # it is typically useful to cause the logging to be
+    # setup again.
+    def cycle_logging(self):
+        logging.resetLogging()
+        logging.setupLogging(self.cfg)
+
+    @property
+    def cfg(self):
+        # Ensure that not indirectly modified
+        return copy.deepcopy(self._cfg)
+
+    def run(self, name, functor, args, freq=None, clear_on_fail=False):
+        return self._runners.run(name, functor, args, freq, clear_on_fail)
+
+    def get_template_filename(self, name):
+        fn = self.paths.template_tpl % (name)
+        if not os.path.isfile(fn):
+            LOG.warn("No template found at %s for template named %s", fn, name)
+            return None
+        return fn
+
+    # The rest of thes are just useful proxies
+    def get_userdata(self):
+        return self.datasource.get_userdata()
+
+    def get_instance_id(self):
+        return self.datasource.get_instance_id()
+
+    def get_public_ssh_keys(self):
+        return self.datasource.get_public_ssh_keys()
+
+    def get_locale(self):
+        return self.datasource.get_locale()
+
+    def get_local_mirror(self):
+        return self.datasource.get_local_mirror()
+
+    def get_hostname(self, fqdn=False):
+        return self.datasource.get_hostname(fqdn=fqdn)
+
+    def device_name_to_device(self, name):
+        return self.datasource.device_name_to_device(name)
+
+    def get_ipath_cur(self, name=None):
+        return self.paths.get_ipath_cur(name)
+
+    def get_cpath(self, name=None):
+        return self.paths.get_cpath(name)
+
+    def get_ipath(self, name=None):
+        return self.paths.get_ipath(name)
diff --git a/cloudinit/config/__init__.py b/cloudinit/config/__init__.py
new file mode 100644
index 00000000..69a8cc68
--- /dev/null
+++ b/cloudinit/config/__init__.py
@@ -0,0 +1,56 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2008-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Chuck Short <chuck.short@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from cloudinit.settings import (PER_INSTANCE, FREQUENCIES)
+
+from cloudinit import log as logging
+
+LOG = logging.getLogger(__name__)
+
+# This prefix is used to make it less
+# of a chance that when importing
+# we will not find something else with the same
+# name in the lookup path...
+MOD_PREFIX = "cc_"
+
+
+def form_module_name(name):
+    canon_name = name.replace("-", "_")
+    if canon_name.lower().endswith(".py"):
+        canon_name = canon_name[0:(len(canon_name) - 3)]
+    canon_name = canon_name.strip()
+    if not canon_name:
+        return None
+    if not canon_name.startswith(MOD_PREFIX):
+        canon_name = '%s%s' % (MOD_PREFIX, canon_name)
+    return canon_name
+
+
+def fixup_module(mod, def_freq=PER_INSTANCE):
+    if not hasattr(mod, 'frequency'):
+        setattr(mod, 'frequency', def_freq)
+    else:
+        freq = mod.frequency
+        if freq and freq not in FREQUENCIES:
+            LOG.warn("Module %s has an unknown frequency %s", mod, freq)
+    if not hasattr(mod, 'distros'):
+        setattr(mod, 'distros', None)
+    return mod
diff --git a/cloudinit/config/cc_apt_pipelining.py b/cloudinit/config/cc_apt_pipelining.py
new file mode 100644
index 00000000..3426099e
--- /dev/null
+++ b/cloudinit/config/cc_apt_pipelining.py
@@ -0,0 +1,59 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#
+#    Author: Ben Howard <ben.howard@canonical.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import util
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+distros = ['ubuntu', 'debian']
+
+DEFAULT_FILE = "/etc/apt/apt.conf.d/90cloud-init-pipelining"
+
+APT_PIPE_TPL = ("//Written by cloud-init per 'apt_pipelining'\n"
+                'Acquire::http::Pipeline-Depth "%s";\n')
+
+# Acquire::http::Pipeline-Depth can be a value
+# from 0 to 5 indicating how many outstanding requests APT should send.
+# A value of zero MUST be specified if the remote host does not properly linger
+# on TCP connections - otherwise data corruption will occur.
+
+
+def handle(_name, cfg, cloud, log, _args):
+
+    apt_pipe_value = util.get_cfg_option_str(cfg, "apt_pipelining", False)
+    apt_pipe_value_s = str(apt_pipe_value).lower().strip()
+
+    if apt_pipe_value_s == "false":
+        write_apt_snippet(cloud, "0", log, DEFAULT_FILE)
+    elif apt_pipe_value_s in ("none", "unchanged", "os"):
+        return
+    elif apt_pipe_value_s in [str(b) for b in xrange(0, 6)]:
+        write_apt_snippet(cloud, apt_pipe_value_s, log, DEFAULT_FILE)
+    else:
+        log.warn("Invalid option for apt_pipeling: %s", apt_pipe_value)
+
+
+def write_apt_snippet(cloud, setting, log, f_name):
+    """ Writes f_name with apt pipeline depth 'setting' """
+
+    file_contents = APT_PIPE_TPL % (setting)
+
+    util.write_file(cloud.paths.join(False, f_name), file_contents)
+
+    log.debug("Wrote %s with apt pipeline depth setting %s", f_name, setting)
diff --git a/cloudinit/config/cc_apt_update_upgrade.py b/cloudinit/config/cc_apt_update_upgrade.py
new file mode 100644
index 00000000..5c5e510c
--- /dev/null
+++ b/cloudinit/config/cc_apt_update_upgrade.py
@@ -0,0 +1,272 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import glob
+import os
+
+from cloudinit import templater
+from cloudinit import util
+
+distros = ['ubuntu', 'debian']
+
+PROXY_TPL = "Acquire::HTTP::Proxy \"%s\";\n"
+PROXY_FN = "/etc/apt/apt.conf.d/95cloud-init-proxy"
+
+# A temporary shell program to get a given gpg key
+# from a given keyserver
+EXPORT_GPG_KEYID = """
+    k=${1} ks=${2};
+    exec 2>/dev/null
+    [ -n "$k" ] || exit 1;
+    armour=$(gpg --list-keys --armour "${k}")
+    if [ -z "${armour}" ]; then
+       gpg --keyserver ${ks} --recv $k >/dev/null &&
+          armour=$(gpg --export --armour "${k}") &&
+          gpg --batch --yes --delete-keys "${k}"
+    fi
+    [ -n "${armour}" ] && echo "${armour}"
+"""
+
+
+def handle(name, cfg, cloud, log, _args):
+    update = util.get_cfg_option_bool(cfg, 'apt_update', False)
+    upgrade = util.get_cfg_option_bool(cfg, 'apt_upgrade', False)
+
+    release = get_release()
+    mirror = find_apt_mirror(cloud, cfg)
+    if not mirror:
+        log.debug(("Skipping module named %s,"
+                   " no package 'mirror' located"), name)
+        return
+
+    log.debug("Selected mirror at: %s" % mirror)
+
+    if not util.get_cfg_option_bool(cfg,
+                                    'apt_preserve_sources_list', False):
+        generate_sources_list(release, mirror, cloud, log)
+        old_mir = util.get_cfg_option_str(cfg, 'apt_old_mirror',
+                                          "archive.ubuntu.com/ubuntu")
+        rename_apt_lists(old_mir, mirror)
+
+    # Set up any apt proxy
+    proxy = cfg.get("apt_proxy", None)
+    proxy_filename = PROXY_FN
+    if proxy:
+        try:
+            # See man 'apt.conf'
+            contents = PROXY_TPL % (proxy)
+            util.write_file(cloud.paths.join(False, proxy_filename),
+                            contents)
+        except Exception as e:
+            util.logexc(log, "Failed to write proxy to %s", proxy_filename)
+    elif os.path.isfile(proxy_filename):
+        util.del_file(proxy_filename)
+
+    # Process 'apt_sources'
+    if 'apt_sources' in cfg:
+        errors = add_sources(cloud, cfg['apt_sources'],
+                             {'MIRROR': mirror, 'RELEASE': release})
+        for e in errors:
+            log.warn("Source Error: %s", ':'.join(e))
+
+    dconf_sel = util.get_cfg_option_str(cfg, 'debconf_selections', False)
+    if dconf_sel:
+        log.debug("setting debconf selections per cloud config")
+        try:
+            util.subp(('debconf-set-selections', '-'), dconf_sel)
+        except:
+            util.logexc(log, "Failed to run debconf-set-selections")
+
+    pkglist = util.get_cfg_option_list(cfg, 'packages', [])
+
+    errors = []
+    if update or len(pkglist) or upgrade:
+        try:
+            cloud.distro.update_package_sources()
+        except Exception as e:
+            util.logexc(log, "Package update failed")
+            errors.append(e)
+
+    if upgrade:
+        try:
+            cloud.distro.package_command("upgrade")
+        except Exception as e:
+            util.logexc(log, "Package upgrade failed")
+            errors.append(e)
+
+    if len(pkglist):
+        try:
+            cloud.distro.install_packages(pkglist)
+        except Exception as e:
+            util.logexc(log, "Failed to install packages: %s ", pkglist)
+            errors.append(e)
+
+    if len(errors):
+        log.warn("%s failed with exceptions, re-raising the last one",
+                 len(errors))
+        raise errors[-1]
+
+
+# get gpg keyid from keyserver
+def getkeybyid(keyid, keyserver):
+    with util.ExtendedTemporaryFile(suffix='.sh') as fh:
+        fh.write(EXPORT_GPG_KEYID)
+        fh.flush()
+        cmd = ['/bin/sh', fh.name, keyid, keyserver]
+        (stdout, _stderr) = util.subp(cmd)
+        return stdout.strip()
+
+
+def mirror2lists_fileprefix(mirror):
+    string = mirror
+    # take off http:// or ftp://
+    if string.endswith("/"):
+        string = string[0:-1]
+    pos = string.find("://")
+    if pos >= 0:
+        string = string[pos + 3:]
+    string = string.replace("/", "_")
+    return string
+
+
+def rename_apt_lists(omirror, new_mirror, lists_d="/var/lib/apt/lists"):
+    oprefix = os.path.join(lists_d, mirror2lists_fileprefix(omirror))
+    nprefix = os.path.join(lists_d, mirror2lists_fileprefix(new_mirror))
+    if oprefix == nprefix:
+        return
+    olen = len(oprefix)
+    for filename in glob.glob("%s_*" % oprefix):
+        # TODO use the cloud.paths.join...
+        util.rename(filename, "%s%s" % (nprefix, filename[olen:]))
+
+
+def get_release():
+    (stdout, _stderr) = util.subp(['lsb_release', '-cs'])
+    return stdout.strip()
+
+
+def generate_sources_list(codename, mirror, cloud, log):
+    template_fn = cloud.get_template_filename('sources.list')
+    if template_fn:
+        params = {'mirror': mirror, 'codename': codename}
+        out_fn = cloud.paths.join(False, '/etc/apt/sources.list')
+        templater.render_to_file(template_fn, out_fn, params)
+    else:
+        log.warn("No template found, not rendering /etc/apt/sources.list")
+
+
+def add_sources(cloud, srclist, template_params=None):
+    """
+    add entries in /etc/apt/sources.list.d for each abbreviated
+    sources.list entry in 'srclist'.  When rendering template, also
+    include the values in dictionary searchList
+    """
+    if template_params is None:
+        template_params = {}
+
+    errorlist = []
+    for ent in srclist:
+        if 'source' not in ent:
+            errorlist.append(["", "missing source"])
+            continue
+
+        source = ent['source']
+        if source.startswith("ppa:"):
+            try:
+                util.subp(["add-apt-repository", source])
+            except:
+                errorlist.append([source, "add-apt-repository failed"])
+            continue
+
+        source = templater.render_string(source, template_params)
+
+        if 'filename' not in ent:
+            ent['filename'] = 'cloud_config_sources.list'
+
+        if not ent['filename'].startswith("/"):
+            ent['filename'] = os.path.join("/etc/apt/sources.list.d/",
+                                           ent['filename'])
+
+        if ('keyid' in ent and 'key' not in ent):
+            ks = "keyserver.ubuntu.com"
+            if 'keyserver' in ent:
+                ks = ent['keyserver']
+            try:
+                ent['key'] = getkeybyid(ent['keyid'], ks)
+            except:
+                errorlist.append([source, "failed to get key from %s" % ks])
+                continue
+
+        if 'key' in ent:
+            try:
+                util.subp(('apt-key', 'add', '-'), ent['key'])
+            except:
+                errorlist.append([source, "failed add key"])
+
+        try:
+            contents = "%s\n" % (source)
+            util.write_file(cloud.paths.join(False, ent['filename']),
+                            contents, omode="ab")
+        except:
+            errorlist.append([source,
+                             "failed write to file %s" % ent['filename']])
+
+    return errorlist
+
+
+def find_apt_mirror(cloud, cfg):
+    """ find an apt_mirror given the cloud and cfg provided """
+
+    mirror = None
+
+    cfg_mirror = cfg.get("apt_mirror", None)
+    if cfg_mirror:
+        mirror = cfg["apt_mirror"]
+    elif "apt_mirror_search" in cfg:
+        mirror = util.search_for_mirror(cfg['apt_mirror_search'])
+    else:
+        mirror = cloud.get_local_mirror()
+
+        mydom = ""
+
+        doms = []
+
+        if not mirror:
+            # if we have a fqdn, then search its domain portion first
+            (_hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
+            mydom = ".".join(fqdn.split(".")[1:])
+            if mydom:
+                doms.append(".%s" % mydom)
+
+        if not mirror:
+            doms.extend((".localdomain", "",))
+
+            mirror_list = []
+            distro = cloud.distro.name
+            mirrorfmt = "http://%s-mirror%s/%s" % (distro, "%s", distro)
+            for post in doms:
+                mirror_list.append(mirrorfmt % (post))
+
+            mirror = util.search_for_mirror(mirror_list)
+
+    if not mirror:
+        mirror = cloud.distro.get_package_mirror()
+
+    return mirror
diff --git a/cloudinit/config/cc_bootcmd.py b/cloudinit/config/cc_bootcmd.py
new file mode 100644
index 00000000..bae1ea54
--- /dev/null
+++ b/cloudinit/config/cc_bootcmd.py
@@ -0,0 +1,55 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import util
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+
+def handle(name, cfg, cloud, log, _args):
+
+    if "bootcmd" not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'bootcmd' key in configuration"), name)
+        return
+
+    with util.ExtendedTemporaryFile(suffix=".sh") as tmpf:
+        try:
+            content = util.shellify(cfg["bootcmd"])
+            tmpf.write(content)
+            tmpf.flush()
+        except:
+            util.logexc(log, "Failed to shellify bootcmd")
+            raise
+
+        try:
+            env = os.environ.copy()
+            iid = cloud.get_instance_id()
+            if iid:
+                env['INSTANCE_ID'] = str(iid)
+            cmd = ['/bin/sh', tmpf.name]
+            util.subp(cmd, env=env, capture=False)
+        except:
+            util.logexc(log,
+                        ("Failed to run bootcmd module %s"), name)
+            raise
diff --git a/cloudinit/CloudConfig/cc_byobu.py b/cloudinit/config/cc_byobu.py
index e821b261..4e2e06bb 100644
--- a/cloudinit/CloudConfig/cc_byobu.py
+++ b/cloudinit/config/cc_byobu.py
@@ -18,18 +18,19 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import subprocess
-import traceback
+from cloudinit import util
 
+distros = ['ubuntu', 'debian']
 
-def handle(_name, cfg, _cloud, log, args):
+
+def handle(name, cfg, _cloud, log, args):
     if len(args) != 0:
         value = args[0]
     else:
         value = util.get_cfg_option_str(cfg, "byobu_by_default", "")
 
     if not value:
+        log.debug("Skipping module named %s, no 'byobu' values found", name)
         return
 
     if value == "user" or value == "system":
@@ -38,7 +39,7 @@ def handle(_name, cfg, _cloud, log, args):
     valid = ("enable-user", "enable-system", "enable",
              "disable-user", "disable-system", "disable")
     if not value in valid:
-        log.warn("Unknown value %s for byobu_by_default" % value)
+        log.warn("Unknown value %s for byobu_by_default", value)
 
     mod_user = value.endswith("-user")
     mod_sys = value.endswith("-system")
@@ -65,13 +66,6 @@ def handle(_name, cfg, _cloud, log, args):
 
     cmd = ["/bin/sh", "-c", "%s %s %s" % ("X=0;", shcmd, "exit $X")]
 
-    log.debug("setting byobu to %s" % value)
+    log.debug("Setting byobu to %s", value)
 
-    try:
-        subprocess.check_call(cmd)
-    except subprocess.CalledProcessError as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("Cmd returned %s: %s" % (e.returncode, cmd))
-    except OSError as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("Cmd failed to execute: %s" % (cmd))
+    util.subp(cmd, capture=False)
diff --git a/cloudinit/CloudConfig/cc_ca_certs.py b/cloudinit/config/cc_ca_certs.py
index 3af6238a..dc046bda 100644
--- a/cloudinit/CloudConfig/cc_ca_certs.py
+++ b/cloudinit/config/cc_ca_certs.py
@@ -13,25 +13,27 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
-from subprocess import check_call
-from cloudinit.util import (write_file, get_cfg_option_list_or_str,
-                            delete_dir_contents, subp)
+
+from cloudinit import util
 
 CA_CERT_PATH = "/usr/share/ca-certificates/"
 CA_CERT_FILENAME = "cloud-init-ca-certs.crt"
 CA_CERT_CONFIG = "/etc/ca-certificates.conf"
 CA_CERT_SYSTEM_PATH = "/etc/ssl/certs/"
 
+distros = ['ubuntu', 'debian']
+
 
 def update_ca_certs():
     """
     Updates the CA certificate cache on the current machine.
     """
-    check_call(["update-ca-certificates"])
+    util.subp(["update-ca-certificates"], capture=False)
 
 
-def add_ca_certs(certs):
+def add_ca_certs(paths, certs):
     """
     Adds certificates to the system. To actually apply the new certificates
     you must also call L{update_ca_certs}.
@@ -39,26 +41,29 @@ def add_ca_certs(certs):
     @param certs: A list of certificate strings.
     """
     if certs:
-        cert_file_contents = "\n".join(certs)
+        # First ensure they are strings...
+        cert_file_contents = "\n".join([str(c) for c in certs])
         cert_file_fullpath = os.path.join(CA_CERT_PATH, CA_CERT_FILENAME)
-        write_file(cert_file_fullpath, cert_file_contents, mode=0644)
+        cert_file_fullpath = paths.join(False, cert_file_fullpath)
+        util.write_file(cert_file_fullpath, cert_file_contents, mode=0644)
         # Append cert filename to CA_CERT_CONFIG file.
-        write_file(CA_CERT_CONFIG, "\n%s" % CA_CERT_FILENAME, omode="a")
+        util.write_file(paths.join(False, CA_CERT_CONFIG),
+                        "\n%s" % CA_CERT_FILENAME, omode="ab")
 
 
-def remove_default_ca_certs():
+def remove_default_ca_certs(paths):
     """
     Removes all default trusted CA certificates from the system. To actually
     apply the change you must also call L{update_ca_certs}.
     """
-    delete_dir_contents(CA_CERT_PATH)
-    delete_dir_contents(CA_CERT_SYSTEM_PATH)
-    write_file(CA_CERT_CONFIG, "", mode=0644)
+    util.delete_dir_contents(paths.join(False, CA_CERT_PATH))
+    util.delete_dir_contents(paths.join(False, CA_CERT_SYSTEM_PATH))
+    util.write_file(paths.join(False, CA_CERT_CONFIG), "", mode=0644)
     debconf_sel = "ca-certificates ca-certificates/trust_new_crts select no"
-    subp(('debconf-set-selections', '-'), debconf_sel)
+    util.subp(('debconf-set-selections', '-'), debconf_sel)
 
 
-def handle(_name, cfg, _cloud, log, _args):
+def handle(name, cfg, cloud, log, _args):
     """
     Call to handle ca-cert sections in cloud-config file.
 
@@ -70,21 +75,25 @@ def handle(_name, cfg, _cloud, log, _args):
     """
     # If there isn't a ca-certs section in the configuration don't do anything
     if "ca-certs" not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'ca-certs' key in configuration"), name)
         return
+
     ca_cert_cfg = cfg['ca-certs']
 
     # If there is a remove-defaults option set to true, remove the system
     # default trusted CA certs first.
     if ca_cert_cfg.get("remove-defaults", False):
-        log.debug("removing default certificates")
-        remove_default_ca_certs()
+        log.debug("Removing default certificates")
+        remove_default_ca_certs(cloud.paths)
 
     # If we are given any new trusted CA certs to add, add them.
     if "trusted" in ca_cert_cfg:
-        trusted_certs = get_cfg_option_list_or_str(ca_cert_cfg, "trusted")
+        trusted_certs = util.get_cfg_option_list(ca_cert_cfg, "trusted")
         if trusted_certs:
-            log.debug("adding %d certificates" % len(trusted_certs))
-            add_ca_certs(trusted_certs)
+            log.debug("Adding %d certificates" % len(trusted_certs))
+            add_ca_certs(cloud.paths, trusted_certs)
 
     # Update the system with the new cert configuration.
+    log.debug("Updating certificates")
     update_ca_certs()
diff --git a/cloudinit/config/cc_chef.py b/cloudinit/config/cc_chef.py
new file mode 100644
index 00000000..6f568261
--- /dev/null
+++ b/cloudinit/config/cc_chef.py
@@ -0,0 +1,129 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Avishai Ish-Shalom <avishai@fewbytes.com>
+#    Author: Mike Moulton <mike@meltmedia.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import os
+
+from cloudinit import templater
+from cloudinit import util
+
+RUBY_VERSION_DEFAULT = "1.8"
+
+
+def handle(name, cfg, cloud, log, _args):
+
+    # If there isn't a chef key in the configuration don't do anything
+    if 'chef' not in cfg:
+        log.debug(("Skipping module named %s,"
+                  " no 'chef' key in configuration"), name)
+        return
+    chef_cfg = cfg['chef']
+
+    # Ensure the chef directories we use exist
+    c_dirs = [
+        '/etc/chef',
+        '/var/log/chef',
+        '/var/lib/chef',
+        '/var/cache/chef',
+        '/var/backups/chef',
+        '/var/run/chef',
+    ]
+    for d in c_dirs:
+        util.ensure_dir(cloud.paths.join(False, d))
+
+    # Set the validation key based on the presence of either 'validation_key'
+    # or 'validation_cert'. In the case where both exist, 'validation_key'
+    # takes precedence
+    for key in ('validation_key', 'validation_cert'):
+        if key in chef_cfg and chef_cfg[key]:
+            v_fn = cloud.paths.join(False, '/etc/chef/validation.pem')
+            util.write_file(v_fn, chef_cfg[key])
+            break
+
+    # Create the chef config from template
+    template_fn = cloud.get_template_filename('chef_client.rb')
+    if template_fn:
+        iid = str(cloud.datasource.get_instance_id())
+        params = {
+            'server_url': chef_cfg['server_url'],
+            'node_name': util.get_cfg_option_str(chef_cfg, 'node_name', iid),
+            'environment': util.get_cfg_option_str(chef_cfg, 'environment',
+                                                   '_default'),
+            'validation_name': chef_cfg['validation_name']
+        }
+        out_fn = cloud.paths.join(False, '/etc/chef/client.rb')
+        templater.render_to_file(template_fn, out_fn, params)
+    else:
+        log.warn("No template found, not rendering to /etc/chef/client.rb")
+
+    # set the firstboot json
+    initial_json = {}
+    if 'run_list' in chef_cfg:
+        initial_json['run_list'] = chef_cfg['run_list']
+    if 'initial_attributes' in chef_cfg:
+        initial_attributes = chef_cfg['initial_attributes']
+        for k in list(initial_attributes.keys()):
+            initial_json[k] = initial_attributes[k]
+    firstboot_fn = cloud.paths.join(False, '/etc/chef/firstboot.json')
+    util.write_file(firstboot_fn, json.dumps(initial_json))
+
+    # If chef is not installed, we install chef based on 'install_type'
+    if not os.path.isfile('/usr/bin/chef-client'):
+        install_type = util.get_cfg_option_str(chef_cfg, 'install_type',
+                                               'packages')
+        if install_type == "gems":
+            # this will install and run the chef-client from gems
+            chef_version = util.get_cfg_option_str(chef_cfg, 'version', None)
+            ruby_version = util.get_cfg_option_str(chef_cfg, 'ruby_version',
+                                                   RUBY_VERSION_DEFAULT)
+            install_chef_from_gems(cloud.distro, ruby_version, chef_version)
+            # and finally, run chef-client
+            log.debug('Running chef-client')
+            util.subp(['/usr/bin/chef-client',
+                       '-d', '-i', '1800', '-s', '20'], capture=False)
+        elif install_type == 'packages':
+            # this will install and run the chef-client from packages
+            cloud.distro.install_packages(('chef',))
+        else:
+            log.warn("Unknown chef install type %s", install_type)
+
+
+def get_ruby_packages(version):
+    # return a list of packages needed to install ruby at version
+    pkgs = ['ruby%s' % version, 'ruby%s-dev' % version]
+    if version == "1.8":
+        pkgs.extend(('libopenssl-ruby1.8', 'rubygems1.8'))
+    return pkgs
+
+
+def install_chef_from_gems(ruby_version, chef_version, distro):
+    distro.install_packages(get_ruby_packages(ruby_version))
+    if not os.path.exists('/usr/bin/gem'):
+        util.sym_link('/usr/bin/gem%s' % ruby_version, '/usr/bin/gem')
+    if not os.path.exists('/usr/bin/ruby'):
+        util.sym_link('/usr/bin/ruby%s' % ruby_version, '/usr/bin/ruby')
+    if chef_version:
+        util.subp(['/usr/bin/gem', 'install', 'chef',
+                  '-v %s' % chef_version, '--no-ri',
+                  '--no-rdoc', '--bindir', '/usr/bin', '-q'], capture=False)
+    else:
+        util.subp(['/usr/bin/gem', 'install', 'chef',
+                  '--no-ri', '--no-rdoc', '--bindir',
+                  '/usr/bin', '-q'], capture=False)
diff --git a/cloudinit/config/cc_disable_ec2_metadata.py b/cloudinit/config/cc_disable_ec2_metadata.py
new file mode 100644
index 00000000..3fd2c20f
--- /dev/null
+++ b/cloudinit/config/cc_disable_ec2_metadata.py
@@ -0,0 +1,36 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import util
+
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+REJECT_CMD = ['route', 'add', '-host', '169.254.169.254', 'reject']
+
+
+def handle(name, cfg, _cloud, log, _args):
+    disabled = util.get_cfg_option_bool(cfg, "disable_ec2_metadata", False)
+    if disabled:
+        util.subp(REJECT_CMD, capture=False)
+    else:
+        log.debug(("Skipping module named %s,"
+                   " disabling the ec2 route not enabled"), name)
diff --git a/cloudinit/config/cc_final_message.py b/cloudinit/config/cc_final_message.py
new file mode 100644
index 00000000..b1caca47
--- /dev/null
+++ b/cloudinit/config/cc_final_message.py
@@ -0,0 +1,68 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import templater
+from cloudinit import util
+from cloudinit import version
+
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+FINAL_MESSAGE_DEF = ("Cloud-init v. {{version}} finished at {{timestamp}}."
+                     " Up {{uptime}} seconds.")
+
+
+def handle(_name, cfg, cloud, log, args):
+
+    msg_in = None
+    if len(args) != 0:
+        msg_in = args[0]
+    else:
+        msg_in = util.get_cfg_option_str(cfg, "final_message")
+
+    if not msg_in:
+        template_fn = cloud.get_template_filename('final_message')
+        if template_fn:
+            msg_in = util.load_file(template_fn)
+
+    if not msg_in:
+        msg_in = FINAL_MESSAGE_DEF
+
+    uptime = util.uptime()
+    ts = util.time_rfc2822()
+    cver = version.version_string()
+    try:
+        subs = {
+            'uptime': uptime,
+            'timestamp': ts,
+            'version': cver,
+        }
+        util.multi_log("%s\n" % (templater.render_string(msg_in, subs)),
+                       console=False, stderr=True)
+    except Exception:
+        util.logexc(log, "Failed to render final message template")
+
+    boot_fin_fn = cloud.paths.boot_finished
+    try:
+        contents = "%s - %s - v. %s\n" % (uptime, ts, cver)
+        util.write_file(boot_fin_fn, contents)
+    except:
+        util.logexc(log, "Failed to write boot finished file %s", boot_fin_fn)
diff --git a/cloudinit/config/cc_foo.py b/cloudinit/config/cc_foo.py
new file mode 100644
index 00000000..95aab4dd
--- /dev/null
+++ b/cloudinit/config/cc_foo.py
@@ -0,0 +1,52 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit.settings import PER_INSTANCE
+
+# Modules are expected to have the following attributes.
+# 1. A required 'handle' method which takes the following params.
+#    a) The name will not be this files name, but instead
+#    the name specified in configuration (which is the name
+#    which will be used to find this module).
+#    b) A configuration object that is the result of the merging
+#    of cloud configs configuration with legacy configuration
+#    as well as any datasource provided configuration
+#    c) A cloud object that can be used to access various
+#    datasource and paths for the given distro and data provided
+#    by the various datasource instance types.
+#    d) A argument list that may or may not be empty to this module.
+#    Typically those are from module configuration where the module
+#    is defined with some extra configuration that will eventually
+#    be translated from yaml into arguments to this module.
+# 2. A optional 'frequency' that defines how often this module should be ran.
+#    Typically one of PER_INSTANCE, PER_ALWAYS, PER_ONCE. If not
+#    provided PER_INSTANCE will be assumed.
+#    See settings.py for these constants.
+# 3. A optional 'distros' array/set/tuple that defines the known distros
+#    this module will work with (if not all of them). This is used to write
+#    a warning out if a module is being ran on a untested distribution for
+#    informational purposes. If non existent all distros are assumed and
+#    no warning occurs.
+
+frequency = PER_INSTANCE
+
+
+def handle(name, _cfg, _cloud, log, _args):
+    log.debug("Hi from module %s", name)
diff --git a/cloudinit/CloudConfig/cc_grub_dpkg.py b/cloudinit/config/cc_grub_dpkg.py
index 9f3a7eaf..b3ce6fb6 100644
--- a/cloudinit/CloudConfig/cc_grub_dpkg.py
+++ b/cloudinit/config/cc_grub_dpkg.py
@@ -18,10 +18,12 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import traceback
 import os
 
+from cloudinit import util
+
+distros = ['ubuntu', 'debian']
+
 
 def handle(_name, cfg, _cloud, log, _args):
     idevs = None
@@ -35,14 +37,14 @@ def handle(_name, cfg, _cloud, log, _args):
 
     if ((os.path.exists("/dev/sda1") and not os.path.exists("/dev/sda")) or
         (os.path.exists("/dev/xvda1") and not os.path.exists("/dev/xvda"))):
-        if idevs == None:
+        if idevs is None:
             idevs = ""
-        if idevs_empty == None:
+        if idevs_empty is None:
             idevs_empty = "true"
     else:
-        if idevs_empty == None:
+        if idevs_empty is None:
             idevs_empty = "false"
-        if idevs == None:
+        if idevs is None:
             idevs = "/dev/sda"
             for dev in ("/dev/sda", "/dev/vda", "/dev/sda1", "/dev/vda1"):
                 if os.path.exists(dev):
@@ -52,13 +54,14 @@ def handle(_name, cfg, _cloud, log, _args):
     # now idevs and idevs_empty are set to determined values
     # or, those set by user
 
-    dconf_sel = "grub-pc grub-pc/install_devices string %s\n" % idevs + \
-        "grub-pc grub-pc/install_devices_empty boolean %s\n" % idevs_empty
-    log.debug("setting grub debconf-set-selections with '%s','%s'" %
+    dconf_sel = (("grub-pc grub-pc/install_devices string %s\n"
+                 "grub-pc grub-pc/install_devices_empty boolean %s\n") %
+                 (idevs, idevs_empty))
+
+    log.debug("Setting grub debconf-set-selections with '%s','%s'" %
         (idevs, idevs_empty))
 
     try:
-        util.subp(('debconf-set-selections'), dconf_sel)
+        util.subp(['debconf-set-selections'], dconf_sel)
     except:
-        log.error("Failed to run debconf-set-selections for grub-dpkg")
-        log.debug(traceback.format_exc())
+        util.logexc(log, "Failed to run debconf-set-selections for grub-dpkg")
diff --git a/cloudinit/config/cc_keys_to_console.py b/cloudinit/config/cc_keys_to_console.py
new file mode 100644
index 00000000..ed7af690
--- /dev/null
+++ b/cloudinit/config/cc_keys_to_console.py
@@ -0,0 +1,53 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit.settings import PER_INSTANCE
+from cloudinit import util
+
+frequency = PER_INSTANCE
+
+# This is a tool that cloud init provides
+HELPER_TOOL = '/usr/lib/cloud-init/write-ssh-key-fingerprints'
+
+
+def handle(name, cfg, _cloud, log, _args):
+    if not os.path.exists(HELPER_TOOL):
+        log.warn(("Unable to activate module %s,"
+                  " helper tool not found at %s"), name, HELPER_TOOL)
+        return
+
+    fp_blacklist = util.get_cfg_option_list(cfg,
+                                            "ssh_fp_console_blacklist", [])
+    key_blacklist = util.get_cfg_option_list(cfg,
+                                             "ssh_key_console_blacklist",
+                                              ["ssh-dss"])
+
+    try:
+        cmd = [HELPER_TOOL]
+        cmd.append(','.join(fp_blacklist))
+        cmd.append(','.join(key_blacklist))
+        (stdout, _stderr) = util.subp(cmd)
+        util.multi_log("%s\n" % (stdout.strip()),
+                       stderr=False, console=True)
+    except:
+        log.warn("Writing keys to the system console failed!")
+        raise
diff --git a/cloudinit/CloudConfig/cc_landscape.py b/cloudinit/config/cc_landscape.py
index a4113cbe..906a6ff7 100644
--- a/cloudinit/CloudConfig/cc_landscape.py
+++ b/cloudinit/config/cc_landscape.py
@@ -19,16 +19,23 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import os.path
-from cloudinit.CloudConfig import per_instance
+
+from StringIO import StringIO
+
 from configobj import ConfigObj
 
-frequency = per_instance
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+LSC_CLIENT_CFG_FILE = "/etc/landscape/client.conf"
 
-lsc_client_cfg_file = "/etc/landscape/client.conf"
+distros = ['ubuntu']
 
 # defaults taken from stock client.conf in landscape-client 11.07.1.1-0ubuntu2
-lsc_builtincfg = {
+LSC_BUILTIN_CFG = {
   'client': {
     'log_level': "info",
     'url': "https://landscape.canonical.com/message-system",
@@ -38,7 +45,7 @@ lsc_builtincfg = {
 }
 
 
-def handle(_name, cfg, _cloud, log, _args):
+def handle(_name, cfg, cloud, log, _args):
     """
     Basically turn a top level 'landscape' entry with a 'client' dict
     and render it to ConfigObj format under '[client]' section in
@@ -47,27 +54,40 @@ def handle(_name, cfg, _cloud, log, _args):
 
     ls_cloudcfg = cfg.get("landscape", {})
 
-    if not isinstance(ls_cloudcfg, dict):
-        raise(Exception("'landscape' existed in config, but not a dict"))
+    if not isinstance(ls_cloudcfg, (dict)):
+        raise RuntimeError(("'landscape' key existed in config,"
+                            " but not a dictionary type,"
+                            " is a %s instead"), util.obj_name(ls_cloudcfg))
 
-    merged = mergeTogether([lsc_builtincfg, lsc_client_cfg_file, ls_cloudcfg])
+    merge_data = [
+        LSC_BUILTIN_CFG,
+        cloud.paths.join(True, LSC_CLIENT_CFG_FILE),
+        ls_cloudcfg,
+    ]
+    merged = merge_together(merge_data)
 
-    if not os.path.isdir(os.path.dirname(lsc_client_cfg_file)):
-        os.makedirs(os.path.dirname(lsc_client_cfg_file))
+    lsc_client_fn = cloud.paths.join(False, LSC_CLIENT_CFG_FILE)
+    lsc_dir = cloud.paths.join(False, os.path.dirname(lsc_client_fn))
+    if not os.path.isdir(lsc_dir):
+        util.ensure_dir(lsc_dir)
 
-    with open(lsc_client_cfg_file, "w") as fp:
-        merged.write(fp)
+    contents = StringIO()
+    merged.write(contents)
+    contents.flush()
 
-    log.debug("updated %s" % lsc_client_cfg_file)
+    util.write_file(lsc_client_fn, contents.getvalue())
+    log.debug("Wrote landscape config file to %s", lsc_client_fn)
 
 
-def mergeTogether(objs):
+def merge_together(objs):
     """
     merge together ConfigObj objects or things that ConfigObj() will take in
     later entries override earlier
     """
     cfg = ConfigObj({})
     for obj in objs:
+        if not obj:
+            continue
         if isinstance(obj, ConfigObj):
             cfg.merge(obj)
         else:
diff --git a/cloudinit/config/cc_locale.py b/cloudinit/config/cc_locale.py
new file mode 100644
index 00000000..6feaae9d
--- /dev/null
+++ b/cloudinit/config/cc_locale.py
@@ -0,0 +1,37 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import util
+
+
+def handle(name, cfg, cloud, log, args):
+    if len(args) != 0:
+        locale = args[0]
+    else:
+        locale = util.get_cfg_option_str(cfg, "locale", cloud.get_locale())
+
+    if not locale:
+        log.debug(("Skipping module named %s, "
+                   "no 'locale' configuration found"), name)
+        return
+
+    log.debug("Setting locale to %s", locale)
+    locale_cfgfile = util.get_cfg_option_str(cfg, "locale_configfile")
+    cloud.distro.apply_locale(locale, locale_cfgfile)
diff --git a/cloudinit/config/cc_mcollective.py b/cloudinit/config/cc_mcollective.py
new file mode 100644
index 00000000..2acdbc6f
--- /dev/null
+++ b/cloudinit/config/cc_mcollective.py
@@ -0,0 +1,91 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Marc Cluet <marc.cluet@canonical.com>
+#    Based on code by Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from StringIO import StringIO
+
+# Used since this can maintain comments
+# and doesn't need a top level section
+from configobj import ConfigObj
+
+from cloudinit import util
+
+PUBCERT_FILE = "/etc/mcollective/ssl/server-public.pem"
+PRICERT_FILE = "/etc/mcollective/ssl/server-private.pem"
+
+
+def handle(name, cfg, cloud, log, _args):
+
+    # If there isn't a mcollective key in the configuration don't do anything
+    if 'mcollective' not in cfg:
+        log.debug(("Skipping module named %s, "
+                   "no 'mcollective' key in configuration"), name)
+        return
+
+    mcollective_cfg = cfg['mcollective']
+
+    # Start by installing the mcollective package ...
+    cloud.distro.install_packages(("mcollective",))
+
+    # ... and then update the mcollective configuration
+    if 'conf' in mcollective_cfg:
+        # Read server.cfg values from the
+        # original file in order to be able to mix the rest up
+        server_cfg_fn = cloud.paths.join(True, '/etc/mcollective/server.cfg')
+        mcollective_config = ConfigObj(server_cfg_fn)
+        # See: http://tiny.cc/jh9agw
+        for (cfg_name, cfg) in mcollective_cfg['conf'].iteritems():
+            if cfg_name == 'public-cert':
+                pubcert_fn = cloud.paths.join(True, PUBCERT_FILE)
+                util.write_file(pubcert_fn, cfg, mode=0644)
+                mcollective_config['plugin.ssl_server_public'] = pubcert_fn
+                mcollective_config['securityprovider'] = 'ssl'
+            elif cfg_name == 'private-cert':
+                pricert_fn = cloud.paths.join(True, PRICERT_FILE)
+                util.write_file(pricert_fn, cfg, mode=0600)
+                mcollective_config['plugin.ssl_server_private'] = pricert_fn
+                mcollective_config['securityprovider'] = 'ssl'
+            else:
+                if isinstance(cfg, (basestring, str)):
+                    # Just set it in the 'main' section
+                    mcollective_config[cfg_name] = cfg
+                elif isinstance(cfg, (dict)):
+                    # Iterate throug the config items, create a section
+                    # if it is needed and then add/or create items as needed
+                    if cfg_name not in mcollective_config.sections:
+                        mcollective_config[cfg_name] = {}
+                    for (o, v) in cfg.iteritems():
+                        mcollective_config[cfg_name][o] = v
+                else:
+                    # Otherwise just try to convert it to a string
+                    mcollective_config[cfg_name] = str(cfg)
+        # We got all our config as wanted we'll rename
+        # the previous server.cfg and create our new one
+        old_fn = cloud.paths.join(False, '/etc/mcollective/server.cfg.old')
+        util.rename(server_cfg_fn, old_fn)
+        # Now we got the whole file, write to disk...
+        contents = StringIO()
+        mcollective_config.write(contents)
+        contents = contents.getvalue()
+        server_cfg_rw = cloud.paths.join(False, '/etc/mcollective/server.cfg')
+        util.write_file(server_cfg_rw, contents, mode=0644)
+
+    # Start mcollective
+    util.subp(['service', 'mcollective', 'start'], capture=False)
diff --git a/cloudinit/CloudConfig/cc_mounts.py b/cloudinit/config/cc_mounts.py
index 6cdd74e8..d3dcf7af 100644
--- a/cloudinit/CloudConfig/cc_mounts.py
+++ b/cloudinit/config/cc_mounts.py
@@ -18,11 +18,17 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import os
-import re
 from string import whitespace  # pylint: disable=W0402
 
+import re
+
+from cloudinit import util
+
+# Shortname matches 'sda', 'sda1', 'xvda', 'hda', 'sdb', xvdb, vda, vdd1
+SHORTNAME_FILTER = r"^[x]{0,1}[shv]d[a-z][0-9]*$"
+SHORTNAME = re.compile(SHORTNAME_FILTER)
+WS = re.compile("[%s]+" % (whitespace))
+
 
 def is_mdname(name):
     # return true if this is a metadata service name
@@ -49,38 +55,46 @@ def handle(_name, cfg, cloud, log, _args):
     if "mounts" in cfg:
         cfgmnt = cfg["mounts"]
 
-    # shortname matches 'sda', 'sda1', 'xvda', 'hda', 'sdb', xvdb, vda, vdd1
-    shortname_filter = r"^[x]{0,1}[shv]d[a-z][0-9]*$"
-    shortname = re.compile(shortname_filter)
-
     for i in range(len(cfgmnt)):
         # skip something that wasn't a list
         if not isinstance(cfgmnt[i], list):
+            log.warn("Mount option %s not a list, got a %s instead",
+                     (i + 1), util.obj_name(cfgmnt[i]))
             continue
 
+        startname = str(cfgmnt[i][0])
+        log.debug("Attempting to determine the real name of %s", startname)
+
         # workaround, allow user to specify 'ephemeral'
         # rather than more ec2 correct 'ephemeral0'
-        if cfgmnt[i][0] == "ephemeral":
+        if startname == "ephemeral":
             cfgmnt[i][0] = "ephemeral0"
+            log.debug(("Adjusted mount option %s "
+                       "name from ephemeral to ephemeral0"), (i + 1))
 
-        if is_mdname(cfgmnt[i][0]):
-            newname = cloud.device_name_to_device(cfgmnt[i][0])
+        if is_mdname(startname):
+            newname = cloud.device_name_to_device(startname)
             if not newname:
-                log.debug("ignoring nonexistant named mount %s" % cfgmnt[i][0])
+                log.debug("Ignoring nonexistant named mount %s", startname)
                 cfgmnt[i][1] = None
             else:
-                if newname.startswith("/"):
-                    cfgmnt[i][0] = newname
-                else:
-                    cfgmnt[i][0] = "/dev/%s" % newname
+                renamed = newname
+                if not newname.startswith("/"):
+                    renamed = "/dev/%s" % newname
+                cfgmnt[i][0] = renamed
+                log.debug("Mapped metadata name %s to %s", startname, renamed)
         else:
-            if shortname.match(cfgmnt[i][0]):
-                cfgmnt[i][0] = "/dev/%s" % cfgmnt[i][0]
+            if SHORTNAME.match(startname):
+                renamed = "/dev/%s" % startname
+                log.debug("Mapped shortname name %s to %s", startname, renamed)
+                cfgmnt[i][0] = renamed
 
         # in case the user did not quote a field (likely fs-freq, fs_passno)
         # but do not convert None to 'None' (LP: #898365)
         for j in range(len(cfgmnt[i])):
-            if isinstance(cfgmnt[i][j], int):
+            if j is None:
+                continue
+            else:
                 cfgmnt[i][j] = str(cfgmnt[i][j])
 
     for i in range(len(cfgmnt)):
@@ -102,14 +116,18 @@ def handle(_name, cfg, cloud, log, _args):
     # for each of the "default" mounts, add them only if no other
     # entry has the same device name
     for defmnt in defmnts:
-        devname = cloud.device_name_to_device(defmnt[0])
+        startname = defmnt[0]
+        devname = cloud.device_name_to_device(startname)
         if devname is None:
+            log.debug("Ignoring nonexistant named default mount %s", startname)
             continue
         if devname.startswith("/"):
             defmnt[0] = devname
         else:
             defmnt[0] = "/dev/%s" % devname
 
+        log.debug("Mapped default device %s to %s", startname, defmnt[0])
+
         cfgmnt_has = False
         for cfgm in cfgmnt:
             if cfgm[0] == defmnt[0]:
@@ -117,14 +135,22 @@ def handle(_name, cfg, cloud, log, _args):
                 break
 
         if cfgmnt_has:
+            log.debug(("Not including %s, already"
+                       " previously included"), startname)
             continue
         cfgmnt.append(defmnt)
 
     # now, each entry in the cfgmnt list has all fstab values
     # if the second field is None (not the string, the value) we skip it
-    actlist = [x for x in cfgmnt if x[1] is not None]
+    actlist = []
+    for x in cfgmnt:
+        if x[1] is None:
+            log.debug("Skipping non-existent device named %s", x[0])
+        else:
+            actlist.append(x)
 
     if len(actlist) == 0:
+        log.debug("No modifications to fstab needed.")
         return
 
     comment = "comment=cloudconfig"
@@ -133,7 +159,7 @@ def handle(_name, cfg, cloud, log, _args):
     dirs = []
     for line in actlist:
         # write 'comment' in the fs_mntops, entry,  claiming this
-        line[3] = "%s,comment=cloudconfig" % line[3]
+        line[3] = "%s,%s" % (line[3], comment)
         if line[2] == "swap":
             needswap = True
         if line[1].startswith("/"):
@@ -141,11 +167,10 @@ def handle(_name, cfg, cloud, log, _args):
         cc_lines.append('\t'.join(line))
 
     fstab_lines = []
-    fstab = open("/etc/fstab", "r+")
-    ws = re.compile("[%s]+" % whitespace)
-    for line in fstab.read().splitlines():
+    fstab = util.load_file(cloud.paths.join(True, "/etc/fstab"))
+    for line in fstab.splitlines():
         try:
-            toks = ws.split(line)
+            toks = WS.split(line)
             if toks[3].find(comment) != -1:
                 continue
         except:
@@ -153,27 +178,23 @@ def handle(_name, cfg, cloud, log, _args):
         fstab_lines.append(line)
 
     fstab_lines.extend(cc_lines)
-
-    fstab.seek(0)
-    fstab.write("%s\n" % '\n'.join(fstab_lines))
-    fstab.truncate()
-    fstab.close()
+    contents = "%s\n" % ('\n'.join(fstab_lines))
+    util.write_file(cloud.paths.join(False, "/etc/fstab"), contents)
 
     if needswap:
         try:
             util.subp(("swapon", "-a"))
         except:
-            log.warn("Failed to enable swap")
+            util.logexc(log, "Activating swap via 'swapon -a' failed")
 
     for d in dirs:
-        if os.path.exists(d):
-            continue
+        real_dir = cloud.paths.join(False, d)
         try:
-            os.makedirs(d)
+            util.ensure_dir(real_dir)
         except:
-            log.warn("Failed to make '%s' config-mount\n", d)
+            util.logexc(log, "Failed to make '%s' config-mount", d)
 
     try:
         util.subp(("mount", "-a"))
     except:
-        log.warn("'mount -a' failed")
+        util.logexc(log, "Activating mounts via 'mount -a' failed")
diff --git a/cloudinit/CloudConfig/cc_phone_home.py b/cloudinit/config/cc_phone_home.py
index a7ff74e1..ae1349eb 100644
--- a/cloudinit/CloudConfig/cc_phone_home.py
+++ b/cloudinit/config/cc_phone_home.py
@@ -17,13 +17,22 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-from cloudinit.CloudConfig import per_instance
-import cloudinit.util as util
-from time import sleep
 
-frequency = per_instance
-post_list_all = ['pub_key_dsa', 'pub_key_rsa', 'pub_key_ecdsa', 'instance_id',
-                 'hostname']
+from cloudinit import templater
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+POST_LIST_ALL = [
+    'pub_key_dsa',
+    'pub_key_rsa',
+    'pub_key_ecdsa',
+    'instance_id',
+    'hostname'
+]
 
 
 # phone_home:
@@ -35,29 +44,33 @@ post_list_all = ['pub_key_dsa', 'pub_key_rsa', 'pub_key_ecdsa', 'instance_id',
 #  url: http://my.foo.bar/$INSTANCE_ID/
 #  post: [ pub_key_dsa, pub_key_rsa, pub_key_ecdsa, instance_id
 #
-def handle(_name, cfg, cloud, log, args):
+def handle(name, cfg, cloud, log, args):
     if len(args) != 0:
         ph_cfg = util.read_conf(args[0])
     else:
         if not 'phone_home' in cfg:
+            log.debug(("Skipping module named %s, "
+                       "no 'phone_home' configuration found"), name)
             return
         ph_cfg = cfg['phone_home']
 
     if 'url' not in ph_cfg:
-        log.warn("no 'url' token in phone_home")
+        log.warn(("Skipping module named %s, "
+                  "no 'url' found in 'phone_home' configuration"), name)
         return
 
     url = ph_cfg['url']
     post_list = ph_cfg.get('post', 'all')
-    tries = ph_cfg.get('tries', 10)
+    tries = ph_cfg.get('tries')
     try:
         tries = int(tries)
     except:
-        log.warn("tries is not an integer. using 10")
         tries = 10
+        util.logexc(log, ("Configuration entry 'tries'"
+                          " is not an integer, using %s instead"), tries)
 
     if post_list == "all":
-        post_list = post_list_all
+        post_list = POST_LIST_ALL
 
     all_keys = {}
     all_keys['instance_id'] = cloud.get_instance_id()
@@ -69,38 +82,37 @@ def handle(_name, cfg, cloud, log, args):
         'pub_key_ecdsa': '/etc/ssh/ssh_host_ecdsa_key.pub',
     }
 
-    for n, path in pubkeys.iteritems():
+    for (n, path) in pubkeys.iteritems():
         try:
-            fp = open(path, "rb")
-            all_keys[n] = fp.read()
-            fp.close()
+            all_keys[n] = util.load_file(cloud.paths.join(True, path))
         except:
-            log.warn("%s: failed to open in phone_home" % path)
+            util.logexc(log, ("%s: failed to open, can not"
+                              " phone home that data"), path)
 
     submit_keys = {}
     for k in post_list:
         if k in all_keys:
             submit_keys[k] = all_keys[k]
         else:
-            submit_keys[k] = "N/A"
-            log.warn("requested key %s from 'post' list not available")
+            submit_keys[k] = None
+            log.warn(("Requested key %s from 'post'"
+                      " configuration list not available"), k)
 
-    url = util.render_string(url, {'INSTANCE_ID': all_keys['instance_id']})
-
-    null_exc = object()
-    last_e = null_exc
-    for i in range(0, tries):
-        try:
-            util.readurl(url, submit_keys)
-            log.debug("succeeded submit to %s on try %i" % (url, i + 1))
-            return
-        except Exception as e:
-            log.debug("failed to post to %s on try %i" % (url, i + 1))
-            last_e = e
-        sleep(3)
-
-    log.warn("failed to post to %s in %i tries" % (url, tries))
-    if last_e is not null_exc:
-        raise(last_e)
+    # Get them read to be posted
+    real_submit_keys = {}
+    for (k, v) in submit_keys.iteritems():
+        if v is None:
+            real_submit_keys[k] = 'N/A'
+        else:
+            real_submit_keys[k] = str(v)
 
-    return
+    # Incase the url is parameterized
+    url_params = {
+        'INSTANCE_ID': all_keys['instance_id'],
+    }
+    url = templater.render_string(url, url_params)
+    try:
+        uhelp.readurl(url, data=real_submit_keys, retries=tries, sec_between=3)
+    except:
+        util.logexc(log, ("Failed to post phone home data to"
+                          " %s in %s tries"), url, tries)
diff --git a/cloudinit/config/cc_puppet.py b/cloudinit/config/cc_puppet.py
new file mode 100644
index 00000000..467c1496
--- /dev/null
+++ b/cloudinit/config/cc_puppet.py
@@ -0,0 +1,113 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from StringIO import StringIO
+
+import os
+import pwd
+import socket
+
+from cloudinit import helpers
+from cloudinit import util
+
+
+def handle(name, cfg, cloud, log, _args):
+    # If there isn't a puppet key in the configuration don't do anything
+    if 'puppet' not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'puppet' configuration found"), name)
+        return
+
+    puppet_cfg = cfg['puppet']
+
+    # Start by installing the puppet package ...
+    cloud.distro.install_packages(["puppet"])
+
+    # ... and then update the puppet configuration
+    if 'conf' in puppet_cfg:
+        # Add all sections from the conf object to puppet.conf
+        puppet_conf_fn = cloud.paths.join(True, '/etc/puppet/puppet.conf')
+        contents = util.load_file(puppet_conf_fn)
+        # Create object for reading puppet.conf values
+        puppet_config = helpers.DefaultingConfigParser()
+        # Read puppet.conf values from original file in order to be able to
+        # mix the rest up. First clean them up (TODO is this really needed??)
+        cleaned_lines = [i.lstrip() for i in contents.splitlines()]
+        cleaned_contents = '\n'.join(cleaned_lines)
+        puppet_config.readfp(StringIO(cleaned_contents),
+                             filename=puppet_conf_fn)
+        for (cfg_name, cfg) in puppet_cfg['conf'].iteritems():
+            # Cert configuration is a special case
+            # Dump the puppet master ca certificate in the correct place
+            if cfg_name == 'ca_cert':
+                # Puppet ssl sub-directory isn't created yet
+                # Create it with the proper permissions and ownership
+                pp_ssl_dir = cloud.paths.join(False, '/var/lib/puppet/ssl')
+                util.ensure_dir(pp_ssl_dir, 0771)
+                util.chownbyid(pp_ssl_dir,
+                               pwd.getpwnam('puppet').pw_uid, 0)
+                pp_ssl_certs = cloud.paths.join(False,
+                                                '/var/lib/puppet/ssl/certs/')
+                util.ensure_dir(pp_ssl_certs)
+                util.chownbyid(pp_ssl_certs,
+                               pwd.getpwnam('puppet').pw_uid, 0)
+                pp_ssl_ca_certs = cloud.paths.join(False,
+                                                   ('/var/lib/puppet/'
+                                                    'ssl/certs/ca.pem'))
+                util.write_file(pp_ssl_ca_certs, cfg)
+                util.chownbyid(pp_ssl_ca_certs,
+                               pwd.getpwnam('puppet').pw_uid, 0)
+            else:
+                # Iterate throug the config items, we'll use ConfigParser.set
+                # to overwrite or create new items as needed
+                for (o, v) in cfg.iteritems():
+                    if o == 'certname':
+                        # Expand %f as the fqdn
+                        # TODO should this use the cloud fqdn??
+                        v = v.replace("%f", socket.getfqdn())
+                        # Expand %i as the instance id
+                        v = v.replace("%i", cloud.get_instance_id())
+                        # certname needs to be downcased
+                        v = v.lower()
+                    puppet_config.set(cfg_name, o, v)
+            # We got all our config as wanted we'll rename
+            # the previous puppet.conf and create our new one
+            conf_old_fn = cloud.paths.join(False,
+                                           '/etc/puppet/puppet.conf.old')
+            util.rename(puppet_conf_fn, conf_old_fn)
+            puppet_conf_rw = cloud.paths.join(False, '/etc/puppet/puppet.conf')
+            util.write_file(puppet_conf_rw, puppet_config.stringify())
+
+    # Set puppet to automatically start
+    if os.path.exists('/etc/default/puppet'):
+        util.subp(['sed', '-i',
+                  '-e', 's/^START=.*/START=yes/',
+                  '/etc/default/puppet'], capture=False)
+    elif os.path.exists('/bin/systemctl'):
+        util.subp(['/bin/systemctl', 'enable', 'puppet.service'],
+                  capture=False)
+    elif os.path.exists('/sbin/chkconfig'):
+        util.subp(['/sbin/chkconfig', 'puppet', 'on'], capture=False)
+    else:
+        log.warn(("Sorry we do not know how to enable"
+                  " puppet services on this system"))
+
+    # Start puppetd
+    util.subp(['service', 'puppet', 'start'], capture=False)
diff --git a/cloudinit/config/cc_resizefs.py b/cloudinit/config/cc_resizefs.py
new file mode 100644
index 00000000..69cd8872
--- /dev/null
+++ b/cloudinit/config/cc_resizefs.py
@@ -0,0 +1,140 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import stat
+import time
+
+from cloudinit import util
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+RESIZE_FS_PREFIXES_CMDS = [
+    ('ext', 'resize2fs'),
+    ('xfs', 'xfs_growfs'),
+]
+
+
+def nodeify_path(devpth, where, log):
+    try:
+        st_dev = os.stat(where).st_dev
+        dev = os.makedev(os.major(st_dev), os.minor(st_dev))
+        os.mknod(devpth, 0400 | stat.S_IFBLK, dev)
+        return st_dev
+    except:
+        if util.is_container():
+            log.debug("Inside container, ignoring mknod failure in resizefs")
+            return
+        log.warn("Failed to make device node to resize %s at %s",
+                 where, devpth)
+        raise
+
+
+def get_fs_type(st_dev, path, log):
+    try:
+        dev_entries = util.find_devs_with(tag='TYPE', oformat='value',
+                                         no_cache=True, path=path)
+        if not dev_entries:
+            return None
+        return dev_entries[0].strip()
+    except util.ProcessExecutionError:
+        util.logexc(log, ("Failed to get filesystem type"
+                          " of maj=%s, min=%s for path %s"),
+                    os.major(st_dev), os.minor(st_dev), path)
+        raise
+
+
+def handle(name, cfg, cloud, log, args):
+    if len(args) != 0:
+        resize_root = args[0]
+    else:
+        resize_root = util.get_cfg_option_str(cfg, "resize_rootfs", True)
+
+    if not util.translate_bool(resize_root):
+        log.debug("Skipping module named %s, resizing disabled", name)
+        return
+
+    # TODO is the directory ok to be used??
+    resize_root_d = util.get_cfg_option_str(cfg, "resize_rootfs_tmp", "/run")
+    resize_root_d = cloud.paths.join(False, resize_root_d)
+    util.ensure_dir(resize_root_d)
+
+    # TODO: allow what is to be resized to be configurable??
+    resize_what = cloud.paths.join(False, "/")
+    with util.ExtendedTemporaryFile(prefix="cloudinit.resizefs.",
+                                    dir=resize_root_d, delete=True) as tfh:
+        devpth = tfh.name
+
+        # Delete the file so that mknod will work
+        # but don't change the file handle to know that its
+        # removed so that when a later call that recreates
+        # occurs this temporary file will still benefit from
+        # auto deletion
+        tfh.unlink_now()
+
+        st_dev = nodeify_path(devpth, resize_what, log)
+        fs_type = get_fs_type(st_dev, devpth, log)
+        if not fs_type:
+            log.warn("Could not determine filesystem type of %s", resize_what)
+            return
+
+        resizer = None
+        fstype_lc = fs_type.lower()
+        for (pfix, root_cmd) in RESIZE_FS_PREFIXES_CMDS:
+            if fstype_lc.startswith(pfix):
+                resizer = root_cmd
+                break
+
+        if not resizer:
+            log.warn("Not resizing unknown filesystem type %s for %s",
+                     fs_type, resize_what)
+            return
+
+        log.debug("Resizing %s (%s) using %s", resize_what, fs_type, resizer)
+        resize_cmd = [resizer, devpth]
+
+        if resize_root == "noblock":
+            # Fork to a child that will run
+            # the resize command
+            util.fork_cb(do_resize, resize_cmd, log)
+            # Don't delete the file now in the parent
+            tfh.delete = False
+        else:
+            do_resize(resize_cmd, log)
+
+    action = 'Resized'
+    if resize_root == "noblock":
+        action = 'Resizing (via forking)'
+    log.debug("%s root filesystem (type=%s, maj=%i, min=%i, val=%s)",
+              action, fs_type, os.major(st_dev), os.minor(st_dev), resize_root)
+
+
+def do_resize(resize_cmd, log):
+    start = time.time()
+    try:
+        util.subp(resize_cmd)
+    except util.ProcessExecutionError:
+        util.logexc(log, "Failed to resize filesystem (cmd=%s)", resize_cmd)
+        raise
+    tot_time = int(time.time() - start)
+    log.debug("Resizing took %s seconds", tot_time)
+    # TODO: Should we add a fsck check after this to make
+    # sure we didn't corrupt anything?
diff --git a/cloudinit/config/cc_rightscale_userdata.py b/cloudinit/config/cc_rightscale_userdata.py
new file mode 100644
index 00000000..7a134569
--- /dev/null
+++ b/cloudinit/config/cc_rightscale_userdata.py
@@ -0,0 +1,102 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+##
+## The purpose of this script is to allow cloud-init to consume
+## rightscale style userdata.  rightscale user data is key-value pairs
+## in a url-query-string like format.
+##
+## for cloud-init support, there will be a key named
+## 'CLOUD_INIT_REMOTE_HOOK'.
+##
+## This cloud-config module will
+## - read the blob of data from raw user data, and parse it as key/value
+## - for each key that is found, download the content to
+##   the local instance/scripts directory and set them executable.
+## - the files in that directory will be run by the user-scripts module
+##   Therefore, this must run before that.
+##
+##
+
+import os
+
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+from cloudinit.settings import PER_INSTANCE
+
+from urlparse import parse_qs
+
+frequency = PER_INSTANCE
+
+MY_NAME = "cc_rightscale_userdata"
+MY_HOOKNAME = 'CLOUD_INIT_REMOTE_HOOK'
+
+
+def handle(name, _cfg, cloud, log, _args):
+    try:
+        ud = cloud.get_userdata_raw()
+    except:
+        log.warn("Failed to get raw userdata in module %s", name)
+        return
+
+    try:
+        mdict = parse_qs(ud)
+        if not mdict or not MY_HOOKNAME in mdict:
+            log.debug(("Skipping module %s, "
+                       "did not find %s in parsed"
+                       " raw userdata"), name, MY_HOOKNAME)
+            return
+    except:
+        util.logexc(log, ("Failed to parse query string %s"
+                           " into a dictionary"), ud)
+        raise
+
+    wrote_fns = []
+    captured_excps = []
+
+    # These will eventually be then ran by the cc_scripts_user
+    # TODO: maybe this should just be a new user data handler??
+    # Instead of a late module that acts like a user data handler?
+    scripts_d = cloud.get_ipath_cur('scripts')
+    urls = mdict[MY_HOOKNAME]
+    for (i, url) in enumerate(urls):
+        fname = os.path.join(scripts_d, "rightscale-%02i" % (i))
+        try:
+            resp = uhelp.readurl(url)
+            # Ensure its a valid http response (and something gotten)
+            if resp.ok() and resp.contents:
+                util.write_file(fname, str(resp), mode=0700)
+                wrote_fns.append(fname)
+        except Exception as e:
+            captured_excps.append(e)
+            util.logexc(log, "%s failed to read %s and write %s",
+                        MY_NAME, url, fname)
+
+    if wrote_fns:
+        log.debug("Wrote out rightscale userdata to %s files", len(wrote_fns))
+
+    if len(wrote_fns) != len(urls):
+        skipped = len(urls) - len(wrote_fns)
+        log.debug("%s urls were skipped or failed", skipped)
+
+    if captured_excps:
+        log.warn("%s failed with exceptions, re-raising the last one",
+                 len(captured_excps))
+        raise captured_excps[-1]
diff --git a/cloudinit/CloudConfig/cc_rsyslog.py b/cloudinit/config/cc_rsyslog.py
index ac7f2c74..78327526 100644
--- a/cloudinit/CloudConfig/cc_rsyslog.py
+++ b/cloudinit/config/cc_rsyslog.py
@@ -18,16 +18,15 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit
-import logging
-import cloudinit.util as util
-import traceback
+import os
+
+from cloudinit import util
 
 DEF_FILENAME = "20-cloud-config.conf"
 DEF_DIR = "/etc/rsyslog.d"
 
 
-def handle(_name, cfg, _cloud, log, _args):
+def handle(name, cfg, cloud, log, _args):
     # rsyslog:
     #  - "*.* @@192.158.1.1"
     #  - content: "*.*   @@192.0.2.1:10514"
@@ -37,17 +36,18 @@ def handle(_name, cfg, _cloud, log, _args):
 
     # process 'rsyslog'
     if not 'rsyslog' in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'rsyslog' key in configuration"), name)
         return
 
     def_dir = cfg.get('rsyslog_dir', DEF_DIR)
     def_fname = cfg.get('rsyslog_filename', DEF_FILENAME)
 
     files = []
-    elst = []
-    for ent in cfg['rsyslog']:
+    for i, ent in enumerate(cfg['rsyslog']):
         if isinstance(ent, dict):
             if not "content" in ent:
-                elst.append((ent, "no 'content' entry"))
+                log.warn("No 'content' entry in config entry %s", i + 1)
                 continue
             content = ent['content']
             filename = ent.get("filename", def_fname)
@@ -55,47 +55,48 @@ def handle(_name, cfg, _cloud, log, _args):
             content = ent
             filename = def_fname
 
+        filename = filename.strip()
+        if not filename:
+            log.warn("Entry %s has an empty filename", i + 1)
+            continue
+
         if not filename.startswith("/"):
-            filename = "%s/%s" % (def_dir, filename)
+            filename = os.path.join(def_dir, filename)
 
+        # Truncate filename first time you see it
         omode = "ab"
-        # truncate filename first time you see it
         if filename not in files:
             omode = "wb"
             files.append(filename)
 
         try:
-            util.write_file(filename, content + "\n", omode=omode)
-        except Exception as e:
-            log.debug(traceback.format_exc(e))
-            elst.append((content, "failed to write to %s" % filename))
+            contents = "%s\n" % (content)
+            util.write_file(cloud.paths.join(False, filename),
+                            contents, omode=omode)
+        except Exception:
+            util.logexc(log, "Failed to write to %s", filename)
 
-    # need to restart syslogd
+    # Attempt to restart syslogd
     restarted = False
     try:
-        # if this config module is running at cloud-init time
+        # If this config module is running at cloud-init time
         # (before rsyslog is running) we don't actually have to
         # restart syslog.
         #
-        # upstart actually does what we want here, in that it doesn't
+        # Upstart actually does what we want here, in that it doesn't
         # start a service that wasn't running already on 'restart'
         # it will also return failure on the attempt, so 'restarted'
-        # won't get set
-        log.debug("restarting rsyslog")
+        # won't get set.
+        log.debug("Restarting rsyslog")
         util.subp(['service', 'rsyslog', 'restart'])
         restarted = True
-
-    except Exception as e:
-        elst.append(("restart", str(e)))
+    except Exception:
+        util.logexc(log, "Failed restarting rsyslog")
 
     if restarted:
-        # this only needs to run if we *actually* restarted
+        # This only needs to run if we *actually* restarted
         # syslog above.
-        cloudinit.logging_set_from_cfg_file()
-        log = logging.getLogger()
-        log.debug("rsyslog configured %s" % files)
-
-    for e in elst:
-        log.warn("rsyslog error: %s\n" % ':'.join(e))
-
-    return
+        cloud.cycle_logging()
+        # This should now use rsyslog if
+        # the logging was setup to use it...
+        log.debug("%s configured %s files", name, files)
diff --git a/cloudinit/CloudConfig/cc_runcmd.py b/cloudinit/config/cc_runcmd.py
index f7e8c671..65064cfb 100644
--- a/cloudinit/CloudConfig/cc_runcmd.py
+++ b/cloudinit/config/cc_runcmd.py
@@ -18,15 +18,21 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
+import os
 
+from cloudinit import util
 
-def handle(_name, cfg, cloud, log, _args):
+
+def handle(name, cfg, cloud, log, _args):
     if "runcmd" not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'runcmd' key in configuration"), name)
         return
-    outfile = "%s/runcmd" % cloud.get_ipath('scripts')
+
+    out_fn = os.path.join(cloud.get_ipath('scripts'), "runcmd")
+    cmd = cfg["runcmd"]
     try:
-        content = util.shellify(cfg["runcmd"])
-        util.write_file(outfile, content, 0700)
+        content = util.shellify(cmd)
+        util.write_file(cloud.paths.join(False, out_fn), content, 0700)
     except:
-        log.warn("failed to open %s for runcmd" % outfile)
+        util.logexc(log, "Failed to shellify %s into file %s", cmd, out_fn)
diff --git a/cloudinit/CloudConfig/cc_salt_minion.py b/cloudinit/config/cc_salt_minion.py
index 59522ab8..79ed8807 100644
--- a/cloudinit/CloudConfig/cc_salt_minion.py
+++ b/cloudinit/config/cc_salt_minion.py
@@ -15,42 +15,46 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import os.path
-import subprocess
-import cloudinit.CloudConfig as cc
-import yaml
 
+from cloudinit import util
 
-def handle(_name, cfg, _cloud, _log, _args):
+# Note: see http://saltstack.org/topics/installation/
+
+
+def handle(name, cfg, cloud, log, _args):
     # If there isn't a salt key in the configuration don't do anything
     if 'salt_minion' not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'salt_minion' key in configuration"), name)
         return
+
     salt_cfg = cfg['salt_minion']
+
     # Start by installing the salt package ...
-    cc.install_packages(("salt-minion",))
-    config_dir = '/etc/salt'
-    if not os.path.isdir(config_dir):
-        os.makedirs(config_dir)
+    cloud.distro.install_packages(["salt-minion"])
+
+    # Ensure we can configure files at the right dir
+    config_dir = cloud.paths.join(False, salt_cfg.get("config_dir",
+                                                      '/etc/salt'))
+    util.ensure_dir(config_dir)
+
     # ... and then update the salt configuration
     if 'conf' in salt_cfg:
         # Add all sections from the conf object to /etc/salt/minion
         minion_config = os.path.join(config_dir, 'minion')
-        yaml.dump(salt_cfg['conf'],
-                  file(minion_config, 'w'),
-                  default_flow_style=False)
+        minion_data = util.yaml_dumps(salt_cfg.get('conf'))
+        util.write_file(minion_config, minion_data)
+
     # ... copy the key pair if specified
     if 'public_key' in salt_cfg and 'private_key' in salt_cfg:
-        pki_dir = '/etc/salt/pki'
-        cumask = os.umask(077)
-        if not os.path.isdir(pki_dir):
-            os.makedirs(pki_dir)
-        pub_name = os.path.join(pki_dir, 'minion.pub')
-        pem_name = os.path.join(pki_dir, 'minion.pem')
-        with open(pub_name, 'w') as f:
-            f.write(salt_cfg['public_key'])
-        with open(pem_name, 'w') as f:
-            f.write(salt_cfg['private_key'])
-        os.umask(cumask)
+        pki_dir = cloud.paths.join(False, salt_cfg.get('pki_dir',
+                                                       '/etc/salt/pki'))
+        with util.umask(077):
+            util.ensure_dir(pki_dir)
+            pub_name = os.path.join(pki_dir, 'minion.pub')
+            pem_name = os.path.join(pki_dir, 'minion.pem')
+            util.write_file(pub_name, salt_cfg['public_key'])
+            util.write_file(pem_name, salt_cfg['private_key'])
 
     # Start salt-minion
-    subprocess.check_call(['service', 'salt-minion', 'start'])
+    util.subp(['service', 'salt-minion', 'start'], capture=False)
diff --git a/cloudinit/CloudConfig/cc_scripts_per_boot.py b/cloudinit/config/cc_scripts_per_boot.py
index 41a74754..42b987eb 100644
--- a/cloudinit/CloudConfig/cc_scripts_per_boot.py
+++ b/cloudinit/config/cc_scripts_per_boot.py
@@ -18,17 +18,24 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_always
-from cloudinit import get_cpath
+import os
 
-frequency = per_always
-runparts_path = "%s/%s" % (get_cpath(), "scripts/per-boot")
+from cloudinit import util
 
+from cloudinit.settings import PER_ALWAYS
 
-def handle(_name, _cfg, _cloud, log, _args):
+frequency = PER_ALWAYS
+
+SCRIPT_SUBDIR = 'per-boot'
+
+
+def handle(name, _cfg, cloud, log, _args):
+    # Comes from the following:
+    # https://forums.aws.amazon.com/thread.jspa?threadID=96918
+    runparts_path = os.path.join(cloud.get_cpath(), 'scripts', SCRIPT_SUBDIR)
     try:
         util.runparts(runparts_path)
     except:
-        log.warn("failed to run-parts in %s" % runparts_path)
+        log.warn("Failed to run module %s (%s in %s)",
+                 name, SCRIPT_SUBDIR, runparts_path)
         raise
diff --git a/cloudinit/CloudConfig/cc_scripts_per_instance.py b/cloudinit/config/cc_scripts_per_instance.py
index a2981eab..b5d71c13 100644
--- a/cloudinit/CloudConfig/cc_scripts_per_instance.py
+++ b/cloudinit/config/cc_scripts_per_instance.py
@@ -18,17 +18,24 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-from cloudinit import get_cpath
+import os
 
-frequency = per_instance
-runparts_path = "%s/%s" % (get_cpath(), "scripts/per-instance")
+from cloudinit import util
 
+from cloudinit.settings import PER_INSTANCE
 
-def handle(_name, _cfg, _cloud, log, _args):
+frequency = PER_INSTANCE
+
+SCRIPT_SUBDIR = 'per-instance'
+
+
+def handle(name, _cfg, cloud, log, _args):
+    # Comes from the following:
+    # https://forums.aws.amazon.com/thread.jspa?threadID=96918
+    runparts_path = os.path.join(cloud.get_cpath(), 'scripts', SCRIPT_SUBDIR)
     try:
         util.runparts(runparts_path)
     except:
-        log.warn("failed to run-parts in %s" % runparts_path)
+        log.warn("Failed to run module %s (%s in %s)",
+                 name, SCRIPT_SUBDIR, runparts_path)
         raise
diff --git a/cloudinit/CloudConfig/cc_scripts_per_once.py b/cloudinit/config/cc_scripts_per_once.py
index a69151da..d77d36d5 100644
--- a/cloudinit/CloudConfig/cc_scripts_per_once.py
+++ b/cloudinit/config/cc_scripts_per_once.py
@@ -18,17 +18,24 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_once
-from cloudinit import get_cpath
+import os
 
-frequency = per_once
-runparts_path = "%s/%s" % (get_cpath(), "scripts/per-once")
+from cloudinit import util
 
+from cloudinit.settings import PER_ONCE
 
-def handle(_name, _cfg, _cloud, log, _args):
+frequency = PER_ONCE
+
+SCRIPT_SUBDIR = 'per-once'
+
+
+def handle(name, _cfg, cloud, log, _args):
+    # Comes from the following:
+    # https://forums.aws.amazon.com/thread.jspa?threadID=96918
+    runparts_path = os.path.join(cloud.get_cpath(), 'scripts', SCRIPT_SUBDIR)
     try:
         util.runparts(runparts_path)
     except:
-        log.warn("failed to run-parts in %s" % runparts_path)
+        log.warn("Failed to run module %s (%s in %s)",
+                 name, SCRIPT_SUBDIR, runparts_path)
         raise
diff --git a/cloudinit/CloudConfig/cc_scripts_user.py b/cloudinit/config/cc_scripts_user.py
index 933aa4e0..5c53014f 100644
--- a/cloudinit/CloudConfig/cc_scripts_user.py
+++ b/cloudinit/config/cc_scripts_user.py
@@ -18,17 +18,25 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-from cloudinit import get_ipath_cur
+import os
 
-frequency = per_instance
-runparts_path = "%s/%s" % (get_ipath_cur(), "scripts")
+from cloudinit import util
 
+from cloudinit.settings import PER_INSTANCE
 
-def handle(_name, _cfg, _cloud, log, _args):
+frequency = PER_INSTANCE
+
+SCRIPT_SUBDIR = 'scripts'
+
+
+def handle(name, _cfg, cloud, log, _args):
+    # This is written to by the user data handlers
+    # Ie, any custom shell scripts that come down
+    # go here...
+    runparts_path = os.path.join(cloud.get_ipath_cur(), SCRIPT_SUBDIR)
     try:
         util.runparts(runparts_path)
     except:
-        log.warn("failed to run-parts in %s" % runparts_path)
+        log.warn("Failed to run module %s (%s in %s)",
+                 name, SCRIPT_SUBDIR, runparts_path)
         raise
diff --git a/cloudinit/CloudConfig/cc_set_hostname.py b/cloudinit/config/cc_set_hostname.py
index acea74d9..b0f27ebf 100644
--- a/cloudinit/CloudConfig/cc_set_hostname.py
+++ b/cloudinit/config/cc_set_hostname.py
@@ -18,25 +18,18 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
+from cloudinit import util
 
 
-def handle(_name, cfg, cloud, log, _args):
+def handle(name, cfg, cloud, log, _args):
     if util.get_cfg_option_bool(cfg, "preserve_hostname", False):
-        log.debug("preserve_hostname is set. not setting hostname")
-        return(True)
+        log.debug(("Configuration option 'preserve_hostname' is set,"
+                    " not setting the hostname in module %s"), name)
+        return
 
     (hostname, _fqdn) = util.get_hostname_fqdn(cfg, cloud)
     try:
-        set_hostname(hostname, log)
+        log.debug("Setting hostname to %s", hostname)
+        cloud.distro.set_hostname(hostname)
     except Exception:
-        util.logexc(log)
-        log.warn("failed to set hostname to %s\n", hostname)
-
-    return(True)
-
-
-def set_hostname(hostname, log):
-    util.subp(['hostname', hostname])
-    util.write_file("/etc/hostname", "%s\n" % hostname, 0644)
-    log.debug("populated /etc/hostname with %s on first boot", hostname)
+        util.logexc(log, "Failed to set hostname to %s", hostname)
diff --git a/cloudinit/CloudConfig/cc_set_passwords.py b/cloudinit/config/cc_set_passwords.py
index 9d0bbdb8..ab266741 100644
--- a/cloudinit/CloudConfig/cc_set_passwords.py
+++ b/cloudinit/config/cc_set_passwords.py
@@ -18,13 +18,19 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
 import sys
-import random
+
+from cloudinit import ssh_util
+from cloudinit import util
+
 from string import letters, digits  # pylint: disable=W0402
 
+# We are removing certain 'painful' letters/numbers
+PW_SET = (letters.translate(None, 'loLOI') +
+          digits.translate(None, '01'))
+
 
-def handle(_name, cfg, _cloud, log, args):
+def handle(_name, cfg, cloud, log, args):
     if len(args) != 0:
         # if run from command line, and give args, wipe the chpasswd['list']
         password = args[0]
@@ -62,68 +68,79 @@ def handle(_name, cfg, _cloud, log, args):
 
         ch_in = '\n'.join(plist_in)
         try:
+            log.debug("Changing password for %s:", users)
             util.subp(['chpasswd'], ch_in)
-            log.debug("changed password for %s:" % users)
         except Exception as e:
             errors.append(e)
-            log.warn("failed to set passwords with chpasswd: %s" % e)
+            util.logexc(log,
+                        "Failed to set passwords with chpasswd for %s", users)
 
         if len(randlist):
-            sys.stdout.write("%s\n%s\n" % ("Set the following passwords\n",
-                '\n'.join(randlist)))
+            blurb = ("Set the following 'random' passwords\n",
+                     '\n'.join(randlist))
+            sys.stderr.write("%s\n%s\n" % blurb)
 
         if expire:
-            enum = len(errors)
+            expired_users = []
             for u in users:
                 try:
                     util.subp(['passwd', '--expire', u])
+                    expired_users.append(u)
                 except Exception as e:
                     errors.append(e)
-                    log.warn("failed to expire account for %s" % u)
-            if enum == len(errors):
-                log.debug("expired passwords for: %s" % u)
+                    util.logexc(log, "Failed to set 'expire' for %s", u)
+            if expired_users:
+                log.debug("Expired passwords for: %s users", expired_users)
 
+    change_pwauth = False
+    pw_auth = None
     if 'ssh_pwauth' in cfg:
-        val = str(cfg['ssh_pwauth']).lower()
-        if val in ("true", "1", "yes"):
-            pw_auth = "yes"
-            change_pwauth = True
-        elif val in ("false", "0", "no"):
-            pw_auth = "no"
-            change_pwauth = True
-        else:
-            change_pwauth = False
+        change_pwauth = True
+        if util.is_true(cfg['ssh_pwauth']):
+            pw_auth = 'yes'
+        if util.is_false(cfg['ssh_pwauth']):
+            pw_auth = 'no'
 
     if change_pwauth:
-        pa_s = "\(#*\)\(PasswordAuthentication[[:space:]]\+\)\(yes\|no\)"
-        msg = "set PasswordAuthentication to '%s'" % pw_auth
-        try:
-            cmd = ['sed', '-i', 's,%s,\\2%s,' % (pa_s, pw_auth),
-                   '/etc/ssh/sshd_config']
-            util.subp(cmd)
-            log.debug(msg)
-        except Exception as e:
-            log.warn("failed %s" % msg)
-            errors.append(e)
+        replaced_auth = False
+
+        # See: man sshd_config
+        conf_fn = cloud.paths.join(True, ssh_util.DEF_SSHD_CFG)
+        old_lines = ssh_util.parse_ssh_config(conf_fn)
+        new_lines = []
+        i = 0
+        for (i, line) in enumerate(old_lines):
+            # Keywords are case-insensitive and arguments are case-sensitive
+            if line.key == 'passwordauthentication':
+                log.debug("Replacing auth line %s with %s", i + 1, pw_auth)
+                replaced_auth = True
+                line.value = pw_auth
+            new_lines.append(line)
+
+        if not replaced_auth:
+            log.debug("Adding new auth line %s", i + 1)
+            replaced_auth = True
+            new_lines.append(ssh_util.SshdConfigLine('',
+                                                     'PasswordAuthentication',
+                                                     pw_auth))
+
+        lines = [str(e) for e in new_lines]
+        ssh_rw_fn = cloud.paths.join(False, ssh_util.DEF_SSHD_CFG)
+        util.write_file(ssh_rw_fn, "\n".join(lines))
 
         try:
-            p = util.subp(['service', cfg.get('ssh_svcname', 'ssh'),
-                           'restart'])
-            log.debug("restarted sshd")
+            cmd = ['service']
+            cmd.append(cloud.distro.get_option('ssh_svcname', 'ssh'))
+            cmd.append('restart')
+            util.subp(cmd)
+            log.debug("Restarted the ssh daemon")
         except:
-            log.warn("restart of ssh failed")
+            util.logexc(log, "Restarting of the ssh daemon failed")
 
     if len(errors):
-        raise(errors[0])
-
-    return
-
-
-def rand_str(strlen=32, select_from=letters + digits):
-    return("".join([random.choice(select_from) for _x in range(0, strlen)]))
+        log.debug("%s errors occured, re-raising the last one", len(errors))
+        raise errors[-1]
 
 
 def rand_user_password(pwlen=9):
-    selfrom = (letters.translate(None, 'loLOI') +
-               digits.translate(None, '01'))
-    return(rand_str(pwlen, select_from=selfrom))
+    return util.rand_str(pwlen, select_from=PW_SET)
diff --git a/cloudinit/config/cc_ssh.py b/cloudinit/config/cc_ssh.py
new file mode 100644
index 00000000..4019ae90
--- /dev/null
+++ b/cloudinit/config/cc_ssh.py
@@ -0,0 +1,132 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import glob
+
+from cloudinit import util
+from cloudinit import ssh_util
+
+DISABLE_ROOT_OPTS = ("no-port-forwarding,no-agent-forwarding,"
+"no-X11-forwarding,command=\"echo \'Please login as the user \\\"$USER\\\" "
+"rather than the user \\\"root\\\".\';echo;sleep 10\"")
+
+KEY_2_FILE = {
+    "rsa_private": ("/etc/ssh/ssh_host_rsa_key", 0600),
+    "rsa_public": ("/etc/ssh/ssh_host_rsa_key.pub", 0644),
+    "dsa_private": ("/etc/ssh/ssh_host_dsa_key", 0600),
+    "dsa_public": ("/etc/ssh/ssh_host_dsa_key.pub", 0644),
+    "ecdsa_private": ("/etc/ssh/ssh_host_ecdsa_key", 0600),
+    "ecdsa_public": ("/etc/ssh/ssh_host_ecdsa_key.pub", 0644),
+}
+
+PRIV_2_PUB = {
+    'rsa_private': 'rsa_public',
+    'dsa_private': 'dsa_public',
+    'ecdsa_private': 'ecdsa_public',
+}
+
+KEY_GEN_TPL = 'o=$(ssh-keygen -yf "%s") && echo "$o" root@localhost > "%s"'
+
+GENERATE_KEY_NAMES = ['rsa', 'dsa', 'ecdsa']
+
+KEY_FILE_TPL = '/etc/ssh/ssh_host_%s_key'
+
+
+def handle(_name, cfg, cloud, log, _args):
+
+    # remove the static keys from the pristine image
+    if cfg.get("ssh_deletekeys", True):
+        key_pth = cloud.paths.join(False, "/etc/ssh/", "ssh_host_*key*")
+        for f in glob.glob(key_pth):
+            try:
+                util.del_file(f)
+            except:
+                util.logexc(log, "Failed deleting key file %s", f)
+
+    if "ssh_keys" in cfg:
+        # if there are keys in cloud-config, use them
+        for (key, val) in cfg["ssh_keys"].iteritems():
+            if key in KEY_2_FILE:
+                tgt_fn = KEY_2_FILE[key][0]
+                tgt_perms = KEY_2_FILE[key][1]
+                util.write_file(cloud.paths.join(False, tgt_fn),
+                                val, tgt_perms)
+
+        for (priv, pub) in PRIV_2_PUB.iteritems():
+            if pub in cfg['ssh_keys'] or not priv in cfg['ssh_keys']:
+                continue
+            pair = (KEY_2_FILE[priv][0], KEY_2_FILE[pub][0])
+            cmd = ['sh', '-xc', KEY_GEN_TPL % pair]
+            try:
+                # TODO: Is this guard needed?
+                with util.SeLinuxGuard("/etc/ssh", recursive=True):
+                    util.subp(cmd, capture=False)
+                log.debug("Generated a key for %s from %s", pair[0], pair[1])
+            except:
+                util.logexc(log, ("Failed generated a key"
+                                  " for %s from %s"), pair[0], pair[1])
+    else:
+        # if not, generate them
+        genkeys = util.get_cfg_option_list(cfg,
+                                           'ssh_genkeytypes',
+                                           GENERATE_KEY_NAMES)
+        for keytype in genkeys:
+            keyfile = cloud.paths.join(False, KEY_FILE_TPL % (keytype))
+            util.ensure_dir(os.path.dirname(keyfile))
+            if not os.path.exists(keyfile):
+                cmd = ['ssh-keygen', '-t', keytype, '-N', '', '-f', keyfile]
+                try:
+                    # TODO: Is this guard needed?
+                    with util.SeLinuxGuard("/etc/ssh", recursive=True):
+                        util.subp(cmd, capture=False)
+                except:
+                    util.logexc(log, ("Failed generating key type"
+                                      " %s to file %s"), keytype, keyfile)
+
+    try:
+        user = util.get_cfg_option_str(cfg, 'user')
+        disable_root = util.get_cfg_option_bool(cfg, "disable_root", True)
+        disable_root_opts = util.get_cfg_option_str(cfg, "disable_root_opts",
+                                                    DISABLE_ROOT_OPTS)
+
+        keys = cloud.get_public_ssh_keys() or []
+        if "ssh_authorized_keys" in cfg:
+            cfgkeys = cfg["ssh_authorized_keys"]
+            keys.extend(cfgkeys)
+
+        apply_credentials(keys, user, cloud.paths,
+                          disable_root, disable_root_opts)
+    except:
+        util.logexc(log, "Applying ssh credentials failed!")
+
+
+def apply_credentials(keys, user, paths, disable_root, disable_root_opts):
+
+    keys = set(keys)
+    if user:
+        ssh_util.setup_user_keys(keys, user, '', paths)
+
+    if disable_root and user:
+        key_prefix = disable_root_opts.replace('$USER', user)
+    else:
+        key_prefix = ''
+
+    ssh_util.setup_user_keys(keys, 'root', key_prefix, paths)
diff --git a/cloudinit/CloudConfig/cc_ssh_import_id.py b/cloudinit/config/cc_ssh_import_id.py
index bbf5bd83..c58b28ec 100644
--- a/cloudinit/CloudConfig/cc_ssh_import_id.py
+++ b/cloudinit/config/cc_ssh_import_id.py
@@ -18,12 +18,14 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import subprocess
-import traceback
+from cloudinit import util
 
+# The ssh-import-id only seems to exist on ubuntu (for now)
+# https://launchpad.net/ssh-import-id
+distros = ['ubuntu']
 
-def handle(_name, cfg, _cloud, log, args):
+
+def handle(name, cfg, _cloud, log, args):
     if len(args) != 0:
         user = args[0]
         ids = []
@@ -31,20 +33,21 @@ def handle(_name, cfg, _cloud, log, args):
             ids = args[1:]
     else:
         user = util.get_cfg_option_str(cfg, "user", "ubuntu")
-        ids = util.get_cfg_option_list_or_str(cfg, "ssh_import_id", [])
+        ids = util.get_cfg_option_list(cfg, "ssh_import_id", [])
 
     if len(ids) == 0:
+        log.debug("Skipping module named %s, no ids found to import", name)
         return
 
-    cmd = ["sudo", "-Hu", user, "ssh-import-id"] + ids
+    if not user:
+        log.debug("Skipping module named %s, no user found to import", name)
+        return
 
-    log.debug("importing ssh ids. cmd = %s" % cmd)
+    cmd = ["sudo", "-Hu", user, "ssh-import-id"] + ids
+    log.debug("Importing ssh ids for user %s.", user)
 
     try:
-        subprocess.check_call(cmd)
-    except subprocess.CalledProcessError as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("Cmd returned %s: %s" % (e.returncode, cmd))
-    except OSError as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("Cmd failed to execute: %s" % (cmd))
+        util.subp(cmd, capture=False)
+    except util.ProcessExecutionError as e:
+        util.logexc(log, "Failed to run command to import %s ssh ids", user)
+        raise e
diff --git a/cloudinit/config/cc_timezone.py b/cloudinit/config/cc_timezone.py
new file mode 100644
index 00000000..b9eb85b2
--- /dev/null
+++ b/cloudinit/config/cc_timezone.py
@@ -0,0 +1,39 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+
+def handle(name, cfg, cloud, log, args):
+    if len(args) != 0:
+        timezone = args[0]
+    else:
+        timezone = util.get_cfg_option_str(cfg, "timezone", False)
+
+    if not timezone:
+        log.debug("Skipping module named %s, no 'timezone' specified", name)
+        return
+
+    # Let the distro handle settings its timezone
+    cloud.distro.set_timezone(timezone)
diff --git a/cloudinit/config/cc_update_etc_hosts.py b/cloudinit/config/cc_update_etc_hosts.py
new file mode 100644
index 00000000..c148b12e
--- /dev/null
+++ b/cloudinit/config/cc_update_etc_hosts.py
@@ -0,0 +1,60 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import util
+from cloudinit import templater
+
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+
+def handle(name, cfg, cloud, log, _args):
+    manage_hosts = util.get_cfg_option_str(cfg, "manage_etc_hosts", False)
+    if util.translate_bool(manage_hosts, addons=['template']):
+        (hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
+        if not hostname:
+            log.warn(("Option 'manage_etc_hosts' was set,"
+                     " but no hostname was found"))
+            return
+
+        # Render from a template file
+        distro_n = cloud.distro.name
+        tpl_fn_name = cloud.get_template_filename("hosts.%s" % (distro_n))
+        if not tpl_fn_name:
+            raise RuntimeError(("No hosts template could be"
+                                " found for distro %s") % (distro_n))
+
+        out_fn = cloud.paths.join(False, '/etc/hosts')
+        templater.render_to_file(tpl_fn_name, out_fn,
+                                {'hostname': hostname, 'fqdn': fqdn})
+
+    elif manage_hosts == "localhost":
+        (hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
+        if not hostname:
+            log.warn(("Option 'manage_etc_hosts' was set,"
+                     " but no hostname was found"))
+            return
+
+        log.debug("Managing localhost in /etc/hosts")
+        cloud.distro.update_etc_hosts(hostname, fqdn)
+    else:
+        log.debug(("Configuration option 'manage_etc_hosts' is not set,"
+                    " not managing /etc/hosts in module %s"), name)
diff --git a/cloudinit/config/cc_update_hostname.py b/cloudinit/config/cc_update_hostname.py
new file mode 100644
index 00000000..b84a1a06
--- /dev/null
+++ b/cloudinit/config/cc_update_hostname.py
@@ -0,0 +1,41 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2011 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import util
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+
+def handle(name, cfg, cloud, log, _args):
+    if util.get_cfg_option_bool(cfg, "preserve_hostname", False):
+        log.debug(("Configuration option 'preserve_hostname' is set,"
+                    " not updating the hostname in module %s"), name)
+        return
+
+    (hostname, _fqdn) = util.get_hostname_fqdn(cfg, cloud)
+    try:
+        prev_fn = os.path.join(cloud.get_cpath('data'), "previous-hostname")
+        cloud.distro.update_hostname(hostname, prev_fn)
+    except Exception:
+        util.logexc(log, "Failed to set the hostname to %s", hostname)
+        raise
diff --git a/cloudinit/distros/__init__.py b/cloudinit/distros/__init__.py
new file mode 100644
index 00000000..da4d0180
--- /dev/null
+++ b/cloudinit/distros/__init__.py
@@ -0,0 +1,163 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from StringIO import StringIO
+
+import abc
+
+from cloudinit import importer
+from cloudinit import log as logging
+from cloudinit import util
+
+# TODO: Make this via config??
+IFACE_ACTIONS = {
+    'up': ['ifup', '--all'],
+    'down': ['ifdown', '--all'],
+}
+
+LOG = logging.getLogger(__name__)
+
+
+class Distro(object):
+
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self, name, cfg, paths):
+        self._paths = paths
+        self._cfg = cfg
+        self.name = name
+
+    @abc.abstractmethod
+    def install_packages(self, pkglist):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def _write_network(self, settings):
+        # In the future use the http://fedorahosted.org/netcf/
+        # to write this blob out in a distro format
+        raise NotImplementedError()
+
+    def get_option(self, opt_name, default=None):
+        return self._cfg.get(opt_name, default)
+
+    @abc.abstractmethod
+    def set_hostname(self, hostname):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def update_hostname(self, hostname, prev_hostname_fn):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def package_command(self, cmd, args=None):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def update_package_sources(self):
+        raise NotImplementedError()
+
+    def get_package_mirror(self):
+        return self.get_option('package_mirror')
+
+    def apply_network(self, settings, bring_up=True):
+        # Write it out
+        self._write_network(settings)
+        # Now try to bring them up
+        if bring_up:
+            return self._interface_action('up')
+        return False
+
+    @abc.abstractmethod
+    def apply_locale(self, locale, out_fn=None):
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def set_timezone(self, tz):
+        raise NotImplementedError()
+
+    def _get_localhost_ip(self):
+        return "127.0.0.1"
+
+    def update_etc_hosts(self, hostname, fqdn):
+        # Format defined at
+        # http://unixhelp.ed.ac.uk/CGI/man-cgi?hosts
+        header = "# Added by cloud-init"
+        real_header = "%s on %s" % (header, util.time_rfc2822())
+        local_ip = self._get_localhost_ip()
+        hosts_line = "%s\t%s %s" % (local_ip, fqdn, hostname)
+        new_etchosts = StringIO()
+        need_write = False
+        need_change = True
+        hosts_ro_fn = self._paths.join(True, "/etc/hosts")
+        for line in util.load_file(hosts_ro_fn).splitlines():
+            if line.strip().startswith(header):
+                continue
+            if not line.strip() or line.strip().startswith("#"):
+                new_etchosts.write("%s\n" % (line))
+                continue
+            split_line = [s.strip() for s in line.split()]
+            if len(split_line) < 2:
+                new_etchosts.write("%s\n" % (line))
+                continue
+            (ip, hosts) = split_line[0], split_line[1:]
+            if ip == local_ip:
+                if sorted([hostname, fqdn]) == sorted(hosts):
+                    need_change = False
+                if need_change:
+                    line = "%s\n%s" % (real_header, hosts_line)
+                    need_change = False
+                    need_write = True
+            new_etchosts.write("%s\n" % (line))
+        if need_change:
+            new_etchosts.write("%s\n%s\n" % (real_header, hosts_line))
+            need_write = True
+        if need_write:
+            contents = new_etchosts.getvalue()
+            util.write_file(self._paths.join(False, "/etc/hosts"),
+                            contents, mode=0644)
+
+    def _interface_action(self, action):
+        if action not in IFACE_ACTIONS:
+            raise NotImplementedError("Unknown interface action %s" % (action))
+        cmd = IFACE_ACTIONS[action]
+        try:
+            LOG.debug("Attempting to run %s interface action using command %s",
+                      action, cmd)
+            (_out, err) = util.subp(cmd)
+            if len(err):
+                LOG.warn("Running %s resulted in stderr output: %s", cmd, err)
+            return True
+        except util.ProcessExecutionError:
+            util.logexc(LOG, "Running interface command %s failed", cmd)
+            return False
+
+
+def fetch(name):
+    locs = importer.find_module(name,
+                                ['', __name__],
+                                ['Distro'])
+    if not locs:
+        raise ImportError("No distribution found for distro %s"
+                           % (name))
+    mod = importer.import_module(locs[0])
+    cls = getattr(mod, 'Distro')
+    return cls
diff --git a/cloudinit/distros/debian.py b/cloudinit/distros/debian.py
new file mode 100644
index 00000000..3247d7ce
--- /dev/null
+++ b/cloudinit/distros/debian.py
@@ -0,0 +1,149 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import distros
+from cloudinit import helpers
+from cloudinit import log as logging
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+LOG = logging.getLogger(__name__)
+
+
+class Distro(distros.Distro):
+
+    def __init__(self, name, cfg, paths):
+        distros.Distro.__init__(self, name, cfg, paths)
+        # This will be used to restrict certain
+        # calls from repeatly happening (when they
+        # should only happen say once per instance...)
+        self._runner = helpers.Runners(paths)
+
+    def apply_locale(self, locale, out_fn=None):
+        if not out_fn:
+            out_fn = self._paths.join(False, '/etc/default/locale')
+        util.subp(['locale-gen', locale], capture=False)
+        util.subp(['update-locale', locale], capture=False)
+        contents = [
+            "# Created by cloud-init",
+            'LANG="%s"' % (locale),
+        ]
+        util.write_file(out_fn, "\n".join(contents))
+
+    def install_packages(self, pkglist):
+        self.update_package_sources()
+        self.package_command('install', pkglist)
+
+    def _write_network(self, settings):
+        net_fn = self._paths.join(False, "/etc/network/interfaces")
+        util.write_file(net_fn, settings)
+
+    def set_hostname(self, hostname):
+        out_fn = self._paths.join(False, "/etc/hostname")
+        self._write_hostname(hostname, out_fn)
+        if out_fn == '/etc/hostname':
+            # Only do this if we are running in non-adjusted root mode
+            LOG.debug("Setting hostname to %s", hostname)
+            util.subp(['hostname', hostname])
+
+    def _write_hostname(self, hostname, out_fn):
+        lines = []
+        lines.append("# Created by cloud-init")
+        lines.append(str(hostname))
+        contents = "\n".join(lines)
+        util.write_file(out_fn, contents, 0644)
+
+    def update_hostname(self, hostname, prev_fn):
+        hostname_prev = self._read_hostname(prev_fn)
+        read_fn = self._paths.join(True, "/etc/hostname")
+        hostname_in_etc = self._read_hostname(read_fn)
+        update_files = []
+        if not hostname_prev or hostname_prev != hostname:
+            update_files.append(prev_fn)
+        if (not hostname_in_etc or
+            (hostname_in_etc == hostname_prev and
+             hostname_in_etc != hostname)):
+            write_fn = self._paths.join(False, "/etc/hostname")
+            update_files.append(write_fn)
+        for fn in update_files:
+            try:
+                self._write_hostname(hostname, fn)
+            except:
+                util.logexc(LOG, "Failed to write hostname %s to %s",
+                            hostname, fn)
+        if (hostname_in_etc and hostname_prev and
+            hostname_in_etc != hostname_prev):
+            LOG.debug(("%s differs from /etc/hostname."
+                        " Assuming user maintained hostname."), prev_fn)
+        if "/etc/hostname" in update_files:
+            # Only do this if we are running in non-adjusted root mode
+            LOG.debug("Setting hostname to %s", hostname)
+            util.subp(['hostname', hostname])
+
+    def _read_hostname(self, filename, default=None):
+        contents = util.load_file(filename, quiet=True)
+        for line in contents.splitlines():
+            c_pos = line.find("#")
+            # Handle inline comments
+            if c_pos != -1:
+                line = line[0:c_pos]
+            line_c = line.strip()
+            if line_c:
+                return line_c
+        return default
+
+    def _get_localhost_ip(self):
+        # Note: http://www.leonardoborda.com/blog/127-0-1-1-ubuntu-debian/
+        return "127.0.1.1"
+
+    def set_timezone(self, tz):
+        tz_file = os.path.join("/usr/share/zoneinfo", tz)
+        if not os.path.isfile(tz_file):
+            raise RuntimeError(("Invalid timezone %s,"
+                                " no file found at %s") % (tz, tz_file))
+        tz_lines = [
+            "# Created by cloud-init",
+            str(tz),
+        ]
+        tz_contents = "\n".join(tz_lines)
+        tz_fn = self._paths.join(False, "/etc/timezone")
+        util.write_file(tz_fn, tz_contents)
+        util.copy(tz_file, self._paths.join(False, "/etc/localtime"))
+
+    def package_command(self, command, args=None):
+        e = os.environ.copy()
+        # See: http://tiny.cc/kg91fw
+        # Or: http://tiny.cc/mh91fw
+        e['DEBIAN_FRONTEND'] = 'noninteractive'
+        cmd = ['apt-get', '--option', 'Dpkg::Options::=--force-confold',
+               '--assume-yes', '--quiet', command]
+        if args:
+            cmd.extend(args)
+        # Allow the output of this to flow outwards (ie not be captured)
+        util.subp(cmd, env=e, capture=False)
+
+    def update_package_sources(self):
+        self._runner.run("update-sources", self.package_command,
+                         ["update"], freq=PER_INSTANCE)
diff --git a/cloudinit/CloudConfig/cc_foo.py b/cloudinit/distros/fedora.py
index 35ec3fa7..c777845d 100644
--- a/cloudinit/CloudConfig/cc_foo.py
+++ b/cloudinit/distros/fedora.py
@@ -1,10 +1,12 @@
 # vi: ts=4 expandtab
 #
-#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Scott Moser <scott.moser@canonical.com>
 #    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -18,12 +20,12 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-#import cloudinit
-#import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
+from cloudinit.distros import rhel
 
-frequency = per_instance
+from cloudinit import log as logging
 
+LOG = logging.getLogger(__name__)
 
-def handle(_name, _cfg, _cloud, _log, _args):
-    print "hi"
+
+class Distro(rhel.Distro):
+    pass
diff --git a/cloudinit/distros/rhel.py b/cloudinit/distros/rhel.py
new file mode 100644
index 00000000..7fa69f03
--- /dev/null
+++ b/cloudinit/distros/rhel.py
@@ -0,0 +1,337 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import distros
+from cloudinit import helpers
+from cloudinit import log as logging
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+LOG = logging.getLogger(__name__)
+
+NETWORK_FN_TPL = '/etc/sysconfig/network-scripts/ifcfg-%s'
+
+# See: http://tiny.cc/6r99fw
+# For what alot of these files that are being written
+# are and the format of them
+
+# This library is used to parse/write
+# out the various sysconfig files edited
+#
+# It has to be slightly modified though
+# to ensure that all values are quoted
+# since these configs are usually sourced into
+# bash scripts...
+from configobj import ConfigObj
+
+# See: http://tiny.cc/oezbgw
+D_QUOTE_CHARS = {
+    "\"": "\\\"",
+    "(": "\\(",
+    ")": "\\)",
+    "$": '\$',
+    '`': '\`',
+}
+
+
+class Distro(distros.Distro):
+
+    def __init__(self, name, cfg, paths):
+        distros.Distro.__init__(self, name, cfg, paths)
+        # This will be used to restrict certain
+        # calls from repeatly happening (when they
+        # should only happen say once per instance...)
+        self._runner = helpers.Runners(paths)
+
+    def install_packages(self, pkglist):
+        self.package_command('install', pkglist)
+
+    def _write_network(self, settings):
+        # TODO fix this... since this is the ubuntu format
+        entries = translate_network(settings)
+        LOG.debug("Translated ubuntu style network settings %s into %s",
+                  settings, entries)
+        # Make the intermediate format as the rhel format...
+        for (dev, info) in entries.iteritems():
+            net_fn = NETWORK_FN_TPL % (dev)
+            net_ro_fn = self._paths.join(True, net_fn)
+            (prev_exist, net_cfg) = self._read_conf(net_ro_fn)
+            net_cfg['DEVICE'] = dev
+            boot_proto = info.get('bootproto')
+            if boot_proto:
+                net_cfg['BOOTPROTO'] = boot_proto
+            net_mask = info.get('netmask')
+            if net_mask:
+                net_cfg["NETMASK"] = net_mask
+            addr = info.get('address')
+            if addr:
+                net_cfg["IPADDR"] = addr
+            if info.get('auto'):
+                net_cfg['ONBOOT'] = 'yes'
+            else:
+                net_cfg['ONBOOT'] = 'no'
+            gtway = info.get('gateway')
+            if gtway:
+                net_cfg["GATEWAY"] = gtway
+            bcast = info.get('broadcast')
+            if bcast:
+                net_cfg["BROADCAST"] = bcast
+            mac_addr = info.get('hwaddress')
+            if mac_addr:
+                net_cfg["MACADDR"] = mac_addr
+            lines = net_cfg.write()
+            if not prev_exist:
+                lines.insert(0, '# Created by cloud-init')
+            w_contents = "\n".join(lines)
+            net_rw_fn = self._paths.join(False, net_fn)
+            util.write_file(net_rw_fn, w_contents, 0644)
+
+    def set_hostname(self, hostname):
+        out_fn = self._paths.join(False, '/etc/sysconfig/network')
+        self._write_hostname(hostname, out_fn)
+        if out_fn == '/etc/sysconfig/network':
+            # Only do this if we are running in non-adjusted root mode
+            LOG.debug("Setting hostname to %s", hostname)
+            util.subp(['hostname', hostname])
+
+    def apply_locale(self, locale, out_fn=None):
+        if not out_fn:
+            out_fn = self._paths.join(False, '/etc/sysconfig/i18n')
+        ro_fn = self._paths.join(True, '/etc/sysconfig/i18n')
+        (_exists, contents) = self._read_conf(ro_fn)
+        contents['LANG'] = locale
+        w_contents = "\n".join(contents.write())
+        util.write_file(out_fn, w_contents, 0644)
+
+    def _write_hostname(self, hostname, out_fn):
+        (_exists, contents) = self._read_conf(out_fn)
+        contents['HOSTNAME'] = hostname
+        w_contents = "\n".join(contents.write())
+        util.write_file(out_fn, w_contents, 0644)
+
+    def update_hostname(self, hostname, prev_file):
+        hostname_prev = self._read_hostname(prev_file)
+        read_fn = self._paths.join(True, "/etc/sysconfig/network")
+        hostname_in_sys = self._read_hostname(read_fn)
+        update_files = []
+        if not hostname_prev or hostname_prev != hostname:
+            update_files.append(prev_file)
+        if (not hostname_in_sys or
+            (hostname_in_sys == hostname_prev
+             and hostname_in_sys != hostname)):
+            write_fn = self._paths.join(False, "/etc/sysconfig/network")
+            update_files.append(write_fn)
+        for fn in update_files:
+            try:
+                self._write_hostname(hostname, fn)
+            except:
+                util.logexc(LOG, "Failed to write hostname %s to %s",
+                            hostname, fn)
+        if (hostname_in_sys and hostname_prev and
+            hostname_in_sys != hostname_prev):
+            LOG.debug(("%s differs from /etc/sysconfig/network."
+                        " Assuming user maintained hostname."), prev_file)
+        if "/etc/sysconfig/network" in update_files:
+            # Only do this if we are running in non-adjusted root mode
+            LOG.debug("Setting hostname to %s", hostname)
+            util.subp(['hostname', hostname])
+
+    def _read_hostname(self, filename, default=None):
+        (_exists, contents) = self._read_conf(filename)
+        if 'HOSTNAME' in contents:
+            return contents['HOSTNAME']
+        else:
+            return default
+
+    def _read_conf(self, fn):
+        exists = False
+        if os.path.isfile(fn):
+            contents = util.load_file(fn).splitlines()
+            exists = True
+        else:
+            contents = []
+        return (exists, QuotingConfigObj(contents))
+
+    def set_timezone(self, tz):
+        tz_file = os.path.join("/usr/share/zoneinfo", tz)
+        if not os.path.isfile(tz_file):
+            raise RuntimeError(("Invalid timezone %s,"
+                                " no file found at %s") % (tz, tz_file))
+        # Adjust the sysconfig clock zone setting
+        read_fn = self._paths.join(True, "/etc/sysconfig/clock")
+        (_exists, contents) = self._read_conf(read_fn)
+        contents['ZONE'] = tz
+        tz_contents = "\n".join(contents.write())
+        write_fn = self._paths.join(False, "/etc/sysconfig/clock")
+        util.write_file(write_fn, tz_contents)
+        # This ensures that the correct tz will be used for the system
+        util.copy(tz_file, self._paths.join(False, "/etc/localtime"))
+
+    def package_command(self, command, args=None):
+        cmd = ['yum']
+        # If enabled, then yum will be tolerant of errors on the command line
+        # with regard to packages.
+        # For example: if you request to install foo, bar and baz and baz is
+        # installed; yum won't error out complaining that baz is already
+        # installed.
+        cmd.append("-t")
+        # Determines whether or not yum prompts for confirmation
+        # of critical actions. We don't want to prompt...
+        cmd.append("-y")
+        cmd.append(command)
+        if args:
+            cmd.extend(args)
+        # Allow the output of this to flow outwards (ie not be captured)
+        util.subp(cmd, capture=False)
+
+    def update_package_sources(self):
+        self._runner.run("update-sources", self.package_command,
+                         ["update"], freq=PER_INSTANCE)
+
+
+# This class helps adjust the configobj
+# writing to ensure that when writing a k/v
+# on a line, that they are properly quoted
+# and have no spaces between the '=' sign.
+# - This is mainly due to the fact that
+# the sysconfig scripts are often sourced
+# directly into bash/shell scripts so ensure
+# that it works for those types of use cases.
+class QuotingConfigObj(ConfigObj):
+    def __init__(self, lines):
+        ConfigObj.__init__(self, lines,
+                           interpolation=False,
+                           write_empty_values=True)
+
+    def _quote_posix(self, text):
+        if not text:
+            return ''
+        for (k, v) in D_QUOTE_CHARS.iteritems():
+            text = text.replace(k, v)
+        return '"%s"' % (text)
+
+    def _quote_special(self, text):
+        if text.lower() in ['yes', 'no', 'true', 'false']:
+            return text
+        else:
+            return self._quote_posix(text)
+
+    def _write_line(self, indent_string, entry, this_entry, comment):
+        # Ensure it is formatted fine for
+        # how these sysconfig scripts are used
+        val = self._decode_element(self._quote(this_entry))
+        # Single quoted strings should
+        # always work.
+        if not val.startswith("'"):
+            # Perform any special quoting
+            val = self._quote_special(val)
+        key = self._decode_element(self._quote(entry, multiline=False))
+        cmnt = self._decode_element(comment)
+        return '%s%s%s%s%s' % (indent_string,
+                               key,
+                               "=",
+                               val,
+                               cmnt)
+
+
+# This is a util function to translate a ubuntu /etc/network/interfaces 'blob'
+# to a rhel equiv. that can then be written to /etc/sysconfig/network-scripts/
+# TODO remove when we have python-netcf active...
+def translate_network(settings):
+    # Get the standard cmd, args from the ubuntu format
+    entries = []
+    for line in settings.splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        split_up = line.split(None, 1)
+        if len(split_up) <= 1:
+            continue
+        entries.append(split_up)
+    # Figure out where each iface section is
+    ifaces = []
+    consume = {}
+    for (cmd, args) in entries:
+        if cmd == 'iface':
+            if consume:
+                ifaces.append(consume)
+                consume = {}
+            consume[cmd] = args
+        else:
+            consume[cmd] = args
+    # Check if anything left over to consume
+    absorb = False
+    for (cmd, args) in consume.iteritems():
+        if cmd == 'iface':
+            absorb = True
+    if absorb:
+        ifaces.append(consume)
+    # Now translate
+    real_ifaces = {}
+    for info in ifaces:
+        if 'iface' not in info:
+            continue
+        iface_details = info['iface'].split(None)
+        dev_name = None
+        if len(iface_details) >= 1:
+            dev = iface_details[0].strip().lower()
+            if dev:
+                dev_name = dev
+        if not dev_name:
+            continue
+        iface_info = {}
+        if len(iface_details) >= 3:
+            proto_type = iface_details[2].strip().lower()
+            # Seems like this can be 'loopback' which we don't
+            # really care about
+            if proto_type in ['dhcp', 'static']:
+                iface_info['bootproto'] = proto_type
+        # These can just be copied over
+        for k in ['netmask', 'address', 'gateway', 'broadcast']:
+            if k in info:
+                val = info[k].strip().lower()
+                if val:
+                    iface_info[k] = val
+        # Is any mac address spoofing going on??
+        if 'hwaddress' in info:
+            hw_info = info['hwaddress'].lower().strip()
+            hw_split = hw_info.split(None, 1)
+            if len(hw_split) == 2 and hw_split[0].startswith('ether'):
+                hw_addr = hw_split[1]
+                if hw_addr:
+                    iface_info['hwaddress'] = hw_addr
+        real_ifaces[dev_name] = iface_info
+    # Check for those that should be started on boot via 'auto'
+    for (cmd, args) in entries:
+        if cmd == 'auto':
+            # Seems like auto can be like 'auto eth0 eth0:1' so just get the
+            # first part out as the device name
+            args = args.split(None)
+            if not args:
+                continue
+            dev_name = args[0].strip().lower()
+            if dev_name in real_ifaces:
+                real_ifaces[dev_name]['auto'] = True
+    return real_ifaces
diff --git a/cloudinit/CloudConfig/cc_disable_ec2_metadata.py b/cloudinit/distros/ubuntu.py
index 6b31ea8e..77c2aff4 100644
--- a/cloudinit/CloudConfig/cc_disable_ec2_metadata.py
+++ b/cloudinit/distros/ubuntu.py
@@ -1,10 +1,12 @@
 # vi: ts=4 expandtab
 #
-#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Scott Moser <scott.moser@canonical.com>
 #    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -17,14 +19,13 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-import cloudinit.util as util
-import subprocess
-from cloudinit.CloudConfig import per_always
 
-frequency = per_always
+from cloudinit.distros import debian
 
+from cloudinit import log as logging
 
-def handle(_name, cfg, _cloud, _log, _args):
-    if util.get_cfg_option_bool(cfg, "disable_ec2_metadata", False):
-        fwall = "route add -host 169.254.169.254 reject"
-        subprocess.call(fwall.split(' '))
+LOG = logging.getLogger(__name__)
+
+
+class Distro(debian.Distro):
+    pass
diff --git a/cloudinit/handlers/__init__.py b/cloudinit/handlers/__init__.py
new file mode 100644
index 00000000..dce2abef
--- /dev/null
+++ b/cloudinit/handlers/__init__.py
@@ -0,0 +1,222 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+import os
+
+from cloudinit.settings import (PER_ALWAYS, PER_INSTANCE, FREQUENCIES)
+
+from cloudinit import importer
+from cloudinit import log as logging
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+# Used as the content type when a message is not multipart
+# and it doesn't contain its own content-type
+NOT_MULTIPART_TYPE = "text/x-not-multipart"
+
+# When none is assigned this gets used
+OCTET_TYPE = 'application/octet-stream'
+
+# Special content types that signal the start and end of processing
+CONTENT_END = "__end__"
+CONTENT_START = "__begin__"
+CONTENT_SIGNALS = [CONTENT_START, CONTENT_END]
+
+# Used when a part-handler type is encountered
+# to allow for registration of new types.
+PART_CONTENT_TYPES = ["text/part-handler"]
+PART_HANDLER_FN_TMPL = 'part-handler-%03d'
+
+# For parts without filenames
+PART_FN_TPL = 'part-%03d'
+
+# Different file beginnings to there content type
+INCLUSION_TYPES_MAP = {
+    '#include': 'text/x-include-url',
+    '#include-once': 'text/x-include-once-url',
+    '#!': 'text/x-shellscript',
+    '#cloud-config': 'text/cloud-config',
+    '#upstart-job': 'text/upstart-job',
+    '#part-handler': 'text/part-handler',
+    '#cloud-boothook': 'text/cloud-boothook',
+    '#cloud-config-archive': 'text/cloud-config-archive',
+}
+
+# Sorted longest first
+INCLUSION_SRCH = sorted(list(INCLUSION_TYPES_MAP.keys()),
+                        key=(lambda e: 0 - len(e)))
+
+
+class Handler(object):
+
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self, frequency, version=2):
+        self.handler_version = version
+        self.frequency = frequency
+
+    def __repr__(self):
+        return "%s: [%s]" % (util.obj_name(self), self.list_types())
+
+    @abc.abstractmethod
+    def list_types(self):
+        raise NotImplementedError()
+
+    def handle_part(self, data, ctype, filename, payload, frequency):
+        return self._handle_part(data, ctype, filename, payload, frequency)
+
+    @abc.abstractmethod
+    def _handle_part(self, data, ctype, filename, payload, frequency):
+        raise NotImplementedError()
+
+
+def run_part(mod, data, ctype, filename, payload, frequency):
+    mod_freq = mod.frequency
+    if not (mod_freq == PER_ALWAYS or
+            (frequency == PER_INSTANCE and mod_freq == PER_INSTANCE)):
+        return
+    mod_ver = mod.handler_version
+    # Sanity checks on version (should be an int convertable)
+    try:
+        mod_ver = int(mod_ver)
+    except:
+        mod_ver = 1
+    try:
+        LOG.debug("Calling handler %s (%s, %s, %s) with frequency %s",
+                  mod, ctype, filename, mod_ver, frequency)
+        if mod_ver >= 2:
+            # Treat as v. 2 which does get a frequency
+            mod.handle_part(data, ctype, filename, payload, frequency)
+        else:
+            # Treat as v. 1 which gets no frequency
+            mod.handle_part(data, ctype, filename, payload)
+    except:
+        util.logexc(LOG, ("Failed calling handler %s (%s, %s, %s)"
+                         " with frequency %s"),
+                    mod, ctype, filename,
+                    mod_ver, frequency)
+
+
+def call_begin(mod, data, frequency):
+    run_part(mod, data, CONTENT_START, None, None, frequency)
+
+
+def call_end(mod, data, frequency):
+    run_part(mod, data, CONTENT_END, None, None, frequency)
+
+
+def walker_handle_handler(pdata, _ctype, _filename, payload):
+    curcount = pdata['handlercount']
+    modname = PART_HANDLER_FN_TMPL % (curcount)
+    frequency = pdata['frequency']
+    modfname = os.path.join(pdata['handlerdir'], "%s" % (modname))
+    if not modfname.endswith(".py"):
+        modfname = "%s.py" % (modfname)
+    # TODO: Check if path exists??
+    util.write_file(modfname, payload, 0600)
+    handlers = pdata['handlers']
+    try:
+        mod = fixup_handler(importer.import_module(modname))
+        call_begin(mod, pdata['data'], frequency)
+        # Only register and increment
+        # after the above have worked (so we don't if it
+        # fails)
+        handlers.register(mod)
+        pdata['handlercount'] = curcount + 1
+    except:
+        util.logexc(LOG, ("Failed at registering python file: %s"
+                          " (part handler %s)"), modfname, curcount)
+
+
+def _extract_first_or_bytes(blob, size):
+    # Extract the first line upto X bytes or X bytes from more than the
+    # first line if the first line does not contain enough bytes
+    first_line = blob.split("\n", 1)[0]
+    if len(first_line) >= size:
+        start = first_line[:size]
+    else:
+        start = blob[0:size]
+    return start
+
+
+def walker_callback(pdata, ctype, filename, payload):
+    if ctype in PART_CONTENT_TYPES:
+        walker_handle_handler(pdata, ctype, filename, payload)
+        return
+    handlers = pdata['handlers']
+    if ctype not in pdata['handlers'] and payload:
+        # Extract the first line or 24 bytes for displaying in the log
+        start = _extract_first_or_bytes(payload, 24)
+        details = "'%s...'" % (start.encode("string-escape"))
+        if ctype == NOT_MULTIPART_TYPE:
+            LOG.warning("Unhandled non-multipart (%s) userdata: %s",
+                        ctype, details)
+        else:
+            LOG.warning("Unhandled unknown content-type (%s) userdata: %s",
+                        ctype, details)
+    else:
+        run_part(handlers[ctype], pdata['data'], ctype, filename,
+                 payload, pdata['frequency'])
+
+
+# Callback is a function that will be called with
+# (data, content_type, filename, payload)
+def walk(msg, callback, data):
+    partnum = 0
+    for part in msg.walk():
+        # multipart/* are just containers
+        if part.get_content_maintype() == 'multipart':
+            continue
+
+        ctype = part.get_content_type()
+        if ctype is None:
+            ctype = OCTET_TYPE
+
+        filename = part.get_filename()
+        if not filename:
+            filename = PART_FN_TPL % (partnum)
+
+        callback(data, ctype, filename, part.get_payload(decode=True))
+        partnum = partnum + 1
+
+
+def fixup_handler(mod, def_freq=PER_INSTANCE):
+    if not hasattr(mod, "handler_version"):
+        setattr(mod, "handler_version", 1)
+    if not hasattr(mod, 'frequency'):
+        setattr(mod, 'frequency', def_freq)
+    else:
+        freq = mod.frequency
+        if freq and freq not in FREQUENCIES:
+            LOG.warn("Handler %s has an unknown frequency %s", mod, freq)
+    return mod
+
+
+def type_from_starts_with(payload, default=None):
+    payload_lc = payload.lower()
+    payload_lc = payload_lc.lstrip()
+    for text in INCLUSION_SRCH:
+        if payload_lc.startswith(text):
+            return INCLUSION_TYPES_MAP[text]
+    return default
diff --git a/cloudinit/handlers/boot_hook.py b/cloudinit/handlers/boot_hook.py
new file mode 100644
index 00000000..456b8020
--- /dev/null
+++ b/cloudinit/handlers/boot_hook.py
@@ -0,0 +1,73 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import handlers
+from cloudinit import log as logging
+from cloudinit import util
+
+from cloudinit.settings import (PER_ALWAYS)
+
+LOG = logging.getLogger(__name__)
+
+
+class BootHookPartHandler(handlers.Handler):
+    def __init__(self, paths, datasource, **_kwargs):
+        handlers.Handler.__init__(self, PER_ALWAYS)
+        self.boothook_dir = paths.get_ipath("boothooks")
+        self.instance_id = None
+        if datasource:
+            self.instance_id = datasource.get_instance_id()
+
+    def list_types(self):
+        return [
+            handlers.type_from_starts_with("#cloud-boothook"),
+        ]
+
+    def _write_part(self, payload, filename):
+        filename = util.clean_filename(filename)
+        payload = util.dos2unix(payload)
+        prefix = "#cloud-boothook"
+        start = 0
+        if payload.startswith(prefix):
+            start = len(prefix) + 1
+        filepath = os.path.join(self.boothook_dir, filename)
+        contents = payload[start:]
+        util.write_file(filepath, contents, 0700)
+        return filepath
+
+    def _handle_part(self, _data, ctype, filename, payload, _frequency):
+        if ctype in handlers.CONTENT_SIGNALS:
+            return
+
+        filepath = self._write_part(payload, filename)
+        try:
+            env = os.environ.copy()
+            if self.instance_id is not None:
+                env['INSTANCE_ID'] = str(self.instance_id)
+            util.subp([filepath], env=env)
+        except util.ProcessExecutionError:
+            util.logexc(LOG, "Boothooks script %s execution error", filepath)
+        except Exception:
+            util.logexc(LOG, ("Boothooks unknown "
+                              "error when running %s"), filepath)
diff --git a/cloudinit/handlers/cloud_config.py b/cloudinit/handlers/cloud_config.py
new file mode 100644
index 00000000..f6d95244
--- /dev/null
+++ b/cloudinit/handlers/cloud_config.py
@@ -0,0 +1,62 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit import handlers
+from cloudinit import log as logging
+from cloudinit import util
+
+from cloudinit.settings import (PER_ALWAYS)
+
+LOG = logging.getLogger(__name__)
+
+
+class CloudConfigPartHandler(handlers.Handler):
+    def __init__(self, paths, **_kwargs):
+        handlers.Handler.__init__(self, PER_ALWAYS)
+        self.cloud_buf = []
+        self.cloud_fn = paths.get_ipath("cloud_config")
+
+    def list_types(self):
+        return [
+            handlers.type_from_starts_with("#cloud-config"),
+        ]
+
+    def _write_cloud_config(self, buf):
+        if not self.cloud_fn:
+            return
+        lines = [str(b) for b in buf]
+        payload = "\n".join(lines)
+        util.write_file(self.cloud_fn, payload, 0600)
+
+    def _handle_part(self, _data, ctype, filename, payload, _frequency):
+        if ctype == handlers.CONTENT_START:
+            self.cloud_buf = []
+            return
+        if ctype == handlers.CONTENT_END:
+            self._write_cloud_config(self.cloud_buf)
+            self.cloud_buf = []
+            return
+
+        filename = util.clean_filename(filename)
+        if not filename:
+            filename = '??'
+        self.cloud_buf.extend(["#%s" % (filename), str(payload)])
diff --git a/cloudinit/handlers/shell_script.py b/cloudinit/handlers/shell_script.py
new file mode 100644
index 00000000..a9d8e544
--- /dev/null
+++ b/cloudinit/handlers/shell_script.py
@@ -0,0 +1,52 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import handlers
+from cloudinit import log as logging
+from cloudinit import util
+
+from cloudinit.settings import (PER_ALWAYS)
+
+LOG = logging.getLogger(__name__)
+
+
+class ShellScriptPartHandler(handlers.Handler):
+    def __init__(self, paths, **_kwargs):
+        handlers.Handler.__init__(self, PER_ALWAYS)
+        self.script_dir = paths.get_ipath_cur('scripts')
+
+    def list_types(self):
+        return [
+            handlers.type_from_starts_with("#!"),
+        ]
+
+    def _handle_part(self, _data, ctype, filename, payload, _frequency):
+        if ctype in handlers.CONTENT_SIGNALS:
+            # TODO: maybe delete existing things here
+            return
+
+        filename = util.clean_filename(filename)
+        payload = util.dos2unix(payload)
+        path = os.path.join(self.script_dir, filename)
+        util.write_file(path, payload, 0700)
diff --git a/cloudinit/handlers/upstart_job.py b/cloudinit/handlers/upstart_job.py
new file mode 100644
index 00000000..99e0afde
--- /dev/null
+++ b/cloudinit/handlers/upstart_job.py
@@ -0,0 +1,66 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import os
+
+from cloudinit import handlers
+from cloudinit import log as logging
+from cloudinit import util
+
+from cloudinit.settings import (PER_INSTANCE)
+
+LOG = logging.getLogger(__name__)
+
+
+class UpstartJobPartHandler(handlers.Handler):
+    def __init__(self, paths, **_kwargs):
+        handlers.Handler.__init__(self, PER_INSTANCE)
+        self.upstart_dir = paths.upstart_conf_d
+
+    def list_types(self):
+        return [
+            handlers.type_from_starts_with("#upstart-job"),
+        ]
+
+    def _handle_part(self, _data, ctype, filename, payload, frequency):
+        if ctype in handlers.CONTENT_SIGNALS:
+            return
+
+        # See: https://bugs.launchpad.net/bugs/819507
+        if frequency != PER_INSTANCE:
+            return
+
+        if not self.upstart_dir:
+            return
+
+        filename = util.clean_filename(filename)
+        (_name, ext) = os.path.splitext(filename)
+        if not ext:
+            ext = ''
+        ext = ext.lower()
+        if ext != ".conf":
+            filename = filename + ".conf"
+
+        payload = util.dos2unix(payload)
+        path = os.path.join(self.upstart_dir, filename)
+        util.write_file(path, payload, 0644)
diff --git a/cloudinit/helpers.py b/cloudinit/helpers.py
new file mode 100644
index 00000000..15036a50
--- /dev/null
+++ b/cloudinit/helpers.py
@@ -0,0 +1,452 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from time import time
+
+import contextlib
+import io
+import os
+
+from ConfigParser import (NoSectionError, NoOptionError, RawConfigParser)
+
+from cloudinit.settings import (PER_INSTANCE, PER_ALWAYS, PER_ONCE,
+                                CFG_ENV_NAME)
+
+from cloudinit import log as logging
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+
+class LockFailure(Exception):
+    pass
+
+
+class DummyLock(object):
+    pass
+
+
+class DummySemaphores(object):
+    def __init__(self):
+        pass
+
+    @contextlib.contextmanager
+    def lock(self, _name, _freq, _clear_on_fail=False):
+        yield DummyLock()
+
+    def has_run(self, _name, _freq):
+        return False
+
+    def clear(self, _name, _freq):
+        return True
+
+    def clear_all(self):
+        pass
+
+
+class FileLock(object):
+    def __init__(self, fn):
+        self.fn = fn
+
+
+class FileSemaphores(object):
+    def __init__(self, sem_path):
+        self.sem_path = sem_path
+
+    @contextlib.contextmanager
+    def lock(self, name, freq, clear_on_fail=False):
+        try:
+            yield self._acquire(name, freq)
+        except:
+            if clear_on_fail:
+                self.clear(name, freq)
+            raise
+
+    def clear(self, name, freq):
+        sem_file = self._get_path(name, freq)
+        try:
+            util.del_file(sem_file)
+        except (IOError, OSError):
+            util.logexc(LOG, "Failed deleting semaphore %s", sem_file)
+            return False
+        return True
+
+    def clear_all(self):
+        try:
+            util.del_dir(self.sem_path)
+        except (IOError, OSError):
+            util.logexc(LOG, "Failed deleting semaphore directory %s",
+                        self.sem_path)
+
+    def _acquire(self, name, freq):
+        # Check again if its been already gotten
+        if self.has_run(name, freq):
+            return None
+        # This is a race condition since nothing atomic is happening
+        # here, but this should be ok due to the nature of when
+        # and where cloud-init runs... (file writing is not a lock...)
+        sem_file = self._get_path(name, freq)
+        contents = "%s: %s\n" % (os.getpid(), time())
+        try:
+            util.write_file(sem_file, contents)
+        except (IOError, OSError):
+            util.logexc(LOG, "Failed writing semaphore file %s", sem_file)
+            return None
+        return FileLock(sem_file)
+
+    def has_run(self, name, freq):
+        if not freq or freq == PER_ALWAYS:
+            return False
+        sem_file = self._get_path(name, freq)
+        # This isn't really a good atomic check
+        # but it suffices for where and when cloudinit runs
+        if os.path.exists(sem_file):
+            return True
+        return False
+
+    def _get_path(self, name, freq):
+        sem_path = self.sem_path
+        if not freq or freq == PER_INSTANCE:
+            return os.path.join(sem_path, name)
+        else:
+            return os.path.join(sem_path, "%s.%s" % (name, freq))
+
+
+class Runners(object):
+    def __init__(self, paths):
+        self.paths = paths
+        self.sems = {}
+
+    def _get_sem(self, freq):
+        if freq == PER_ALWAYS or not freq:
+            return None
+        sem_path = None
+        if freq == PER_INSTANCE:
+            # This may not exist,
+            # so thats why we still check for none
+            # below if say the paths object
+            # doesn't have a datasource that can
+            # provide this instance path...
+            sem_path = self.paths.get_ipath("sem")
+        elif freq == PER_ONCE:
+            sem_path = self.paths.get_cpath("sem")
+        if not sem_path:
+            return None
+        if sem_path not in self.sems:
+            self.sems[sem_path] = FileSemaphores(sem_path)
+        return self.sems[sem_path]
+
+    def run(self, name, functor, args, freq=None, clear_on_fail=False):
+        sem = self._get_sem(freq)
+        if not sem:
+            sem = DummySemaphores()
+        if not args:
+            args = []
+        if sem.has_run(name, freq):
+            LOG.debug("%s already ran (freq=%s)", name, freq)
+            return (False, None)
+        with sem.lock(name, freq, clear_on_fail) as lk:
+            if not lk:
+                raise LockFailure("Failed to acquire lock for %s" % name)
+            else:
+                LOG.debug("Running %s using lock (%s)", name, lk)
+                if isinstance(args, (dict)):
+                    results = functor(**args)
+                else:
+                    results = functor(*args)
+                return (True, results)
+
+
+class ConfigMerger(object):
+    def __init__(self, paths=None, datasource=None,
+                 additional_fns=None, base_cfg=None):
+        self._paths = paths
+        self._ds = datasource
+        self._fns = additional_fns
+        self._base_cfg = base_cfg
+        # Created on first use
+        self._cfg = None
+
+    def _get_datasource_configs(self):
+        d_cfgs = []
+        if self._ds:
+            try:
+                ds_cfg = self._ds.get_config_obj()
+                if ds_cfg and isinstance(ds_cfg, (dict)):
+                    d_cfgs.append(ds_cfg)
+            except:
+                util.logexc(LOG, ("Failed loading of datasource"
+                                  " config object from %s"), self._ds)
+        return d_cfgs
+
+    def _get_env_configs(self):
+        e_cfgs = []
+        if CFG_ENV_NAME in os.environ:
+            e_fn = os.environ[CFG_ENV_NAME]
+            try:
+                e_cfgs.append(util.read_conf(e_fn))
+            except:
+                util.logexc(LOG, ('Failed loading of env. config'
+                                  ' from %s'), e_fn)
+        return e_cfgs
+
+    def _get_instance_configs(self):
+        i_cfgs = []
+        # If cloud-config was written, pick it up as
+        # a configuration file to use when running...
+        if not self._paths:
+            return i_cfgs
+        cc_fn = self._paths.get_ipath_cur('cloud_config')
+        if cc_fn and os.path.isfile(cc_fn):
+            try:
+                i_cfgs.append(util.read_conf(cc_fn))
+            except:
+                util.logexc(LOG, ('Failed loading of cloud-config'
+                                      ' from %s'), cc_fn)
+        return i_cfgs
+
+    def _read_cfg(self):
+        # Input config files override
+        # env config files which
+        # override instance configs
+        # which override datasource
+        # configs which override
+        # base configuration
+        cfgs = []
+        if self._fns:
+            for c_fn in self._fns:
+                try:
+                    cfgs.append(util.read_conf(c_fn))
+                except:
+                    util.logexc(LOG, ("Failed loading of configuration"
+                                       " from %s"), c_fn)
+
+        cfgs.extend(self._get_env_configs())
+        cfgs.extend(self._get_instance_configs())
+        cfgs.extend(self._get_datasource_configs())
+        if self._base_cfg:
+            cfgs.append(self._base_cfg)
+        return util.mergemanydict(cfgs)
+
+    @property
+    def cfg(self):
+        # None check to avoid empty case causing re-reading
+        if self._cfg is None:
+            self._cfg = self._read_cfg()
+        return self._cfg
+
+
+class ContentHandlers(object):
+
+    def __init__(self):
+        self.registered = {}
+
+    def __contains__(self, item):
+        return self.is_registered(item)
+
+    def __getitem__(self, key):
+        return self._get_handler(key)
+
+    def is_registered(self, content_type):
+        return content_type in self.registered
+
+    def register(self, mod):
+        types = set()
+        for t in mod.list_types():
+            self.registered[t] = mod
+            types.add(t)
+        return types
+
+    def _get_handler(self, content_type):
+        return self.registered[content_type]
+
+    def items(self):
+        return self.registered.items()
+
+    def iteritems(self):
+        return self.registered.iteritems()
+
+    def register_defaults(self, defs):
+        registered = set()
+        for mod in defs:
+            for t in mod.list_types():
+                if not self.is_registered(t):
+                    self.registered[t] = mod
+                    registered.add(t)
+        return registered
+
+
+class Paths(object):
+    def __init__(self, path_cfgs, ds=None):
+        self.cfgs = path_cfgs
+        # Populate all the initial paths
+        self.cloud_dir = self.join(False,
+                                   path_cfgs.get('cloud_dir',
+                                                 '/var/lib/cloud'))
+        self.instance_link = os.path.join(self.cloud_dir, 'instance')
+        self.boot_finished = os.path.join(self.instance_link, "boot-finished")
+        self.upstart_conf_d = path_cfgs.get('upstart_dir')
+        if self.upstart_conf_d:
+            self.upstart_conf_d = self.join(False, self.upstart_conf_d)
+        self.seed_dir = os.path.join(self.cloud_dir, 'seed')
+        # This one isn't joined, since it should just be read-only
+        template_dir = path_cfgs.get('templates_dir', '/etc/cloud/templates/')
+        self.template_tpl = os.path.join(template_dir, '%s.tmpl')
+        self.lookups = {
+           "handlers": "handlers",
+           "scripts": "scripts",
+           "sem": "sem",
+           "boothooks": "boothooks",
+           "userdata_raw": "user-data.txt",
+           "userdata": "user-data.txt.i",
+           "obj_pkl": "obj.pkl",
+           "cloud_config": "cloud-config.txt",
+           "data": "data",
+        }
+        # Set when a datasource becomes active
+        self.datasource = ds
+
+    # joins the paths but also appends a read
+    # or write root if available
+    def join(self, read_only, *paths):
+        if read_only:
+            root = self.cfgs.get('read_root')
+        else:
+            root = self.cfgs.get('write_root')
+        if not paths:
+            return root
+        if len(paths) > 1:
+            joined = os.path.join(*paths)
+        else:
+            joined = paths[0]
+        if root:
+            pre_joined = joined
+            # Need to remove any starting '/' since this
+            # will confuse os.path.join
+            joined = joined.lstrip("/")
+            joined = os.path.join(root, joined)
+            LOG.debug("Translated %s to adjusted path %s (read-only=%s)",
+                      pre_joined, joined, read_only)
+        return joined
+
+    # get_ipath_cur: get the current instance path for an item
+    def get_ipath_cur(self, name=None):
+        ipath = self.instance_link
+        add_on = self.lookups.get(name)
+        if add_on:
+            ipath = os.path.join(ipath, add_on)
+        return ipath
+
+    # get_cpath : get the "clouddir" (/var/lib/cloud/<name>)
+    # for a name in dirmap
+    def get_cpath(self, name=None):
+        cpath = self.cloud_dir
+        add_on = self.lookups.get(name)
+        if add_on:
+            cpath = os.path.join(cpath, add_on)
+        return cpath
+
+    # _get_ipath : get the instance path for a name in pathmap
+    # (/var/lib/cloud/instances/<instance>/<name>)
+    def _get_ipath(self, name=None):
+        if not self.datasource:
+            return None
+        iid = self.datasource.get_instance_id()
+        if iid is None:
+            return None
+        ipath = os.path.join(self.cloud_dir, 'instances', str(iid))
+        add_on = self.lookups.get(name)
+        if add_on:
+            ipath = os.path.join(ipath, add_on)
+        return ipath
+
+    # get_ipath : get the instance path for a name in pathmap
+    # (/var/lib/cloud/instances/<instance>/<name>)
+    # returns None + warns if no active datasource....
+    def get_ipath(self, name=None):
+        ipath = self._get_ipath(name)
+        if not ipath:
+            LOG.warn(("No per instance data available, "
+                      "is there an datasource/iid set?"))
+            return None
+        else:
+            return ipath
+
+
+# This config parser will not throw when sections don't exist
+# and you are setting values on those sections which is useful
+# when writing to new options that may not have corresponding
+# sections. Also it can default other values when doing gets
+# so that if those sections/options do not exist you will
+# get a default instead of an error. Another useful case where
+# you can avoid catching exceptions that you typically don't
+# care about...
+
+class DefaultingConfigParser(RawConfigParser):
+    DEF_INT = 0
+    DEF_FLOAT = 0.0
+    DEF_BOOLEAN = False
+    DEF_BASE = None
+
+    def get(self, section, option):
+        value = self.DEF_BASE
+        try:
+            value = RawConfigParser.get(self, section, option)
+        except NoSectionError:
+            pass
+        except NoOptionError:
+            pass
+        return value
+
+    def set(self, section, option, value=None):
+        if not self.has_section(section) and section.lower() != 'default':
+            self.add_section(section)
+        RawConfigParser.set(self, section, option, value)
+
+    def remove_option(self, section, option):
+        if self.has_option(section, option):
+            RawConfigParser.remove_option(self, section, option)
+
+    def getboolean(self, section, option):
+        if not self.has_option(section, option):
+            return self.DEF_BOOLEAN
+        return RawConfigParser.getboolean(self, section, option)
+
+    def getfloat(self, section, option):
+        if not self.has_option(section, option):
+            return self.DEF_FLOAT
+        return RawConfigParser.getfloat(self, section, option)
+
+    def getint(self, section, option):
+        if not self.has_option(section, option):
+            return self.DEF_INT
+        return RawConfigParser.getint(self, section, option)
+
+    def stringify(self, header=None):
+        contents = ''
+        with io.BytesIO() as outputstream:
+            self.write(outputstream)
+            outputstream.flush()
+            contents = outputstream.getvalue()
+            if header:
+                contents = "\n".join([header, contents])
+        return contents
diff --git a/cloudinit/importer.py b/cloudinit/importer.py
new file mode 100644
index 00000000..71cf2726
--- /dev/null
+++ b/cloudinit/importer.py
@@ -0,0 +1,65 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+
+from cloudinit import log as logging
+
+LOG = logging.getLogger(__name__)
+
+
+def import_module(module_name):
+    __import__(module_name)
+    return sys.modules[module_name]
+
+
+def find_module(base_name, search_paths, required_attrs=None):
+    found_places = []
+    if not required_attrs:
+        required_attrs = []
+    real_paths = []
+    for path in search_paths:
+        real_path = []
+        if path:
+            real_path.extend(path.split("."))
+        real_path.append(base_name)
+        full_path = '.'.join(real_path)
+        real_paths.append(full_path)
+    LOG.debug("Looking for modules %s that have attributes %s",
+              real_paths, required_attrs)
+    for full_path in real_paths:
+        mod = None
+        try:
+            mod = import_module(full_path)
+        except ImportError:
+            pass
+        if not mod:
+            continue
+        found_attrs = 0
+        for attr in required_attrs:
+            if hasattr(mod, attr):
+                found_attrs += 1
+        if found_attrs == len(required_attrs):
+            found_places.append(full_path)
+    LOG.debug("Found %s with attributes %s in %s", base_name,
+              required_attrs, found_places)
+    return found_places
diff --git a/cloudinit/log.py b/cloudinit/log.py
new file mode 100644
index 00000000..fc1428a2
--- /dev/null
+++ b/cloudinit/log.py
@@ -0,0 +1,133 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import logging.handlers
+import logging.config
+
+import os
+import sys
+
+from StringIO import StringIO
+
+# Logging levels for easy access
+CRITICAL = logging.CRITICAL
+FATAL = logging.FATAL
+ERROR = logging.ERROR
+WARNING = logging.WARNING
+WARN = logging.WARN
+INFO = logging.INFO
+DEBUG = logging.DEBUG
+NOTSET = logging.NOTSET
+
+# Default basic format
+DEF_CON_FORMAT = '%(asctime)s - %(filename)s[%(levelname)s]: %(message)s'
+
+
+def setupBasicLogging():
+    root = logging.getLogger()
+    console = logging.StreamHandler(sys.stderr)
+    console.setFormatter(logging.Formatter(DEF_CON_FORMAT))
+    console.setLevel(DEBUG)
+    root.addHandler(console)
+    root.setLevel(DEBUG)
+
+
+def setupLogging(cfg=None):
+    # See if the config provides any logging conf...
+    if not cfg:
+        cfg = {}
+
+    log_cfgs = []
+    log_cfg = cfg.get('logcfg')
+    if log_cfg and isinstance(log_cfg, (str, basestring)):
+        # If there is a 'logcfg' entry in the config,
+        # respect it, it is the old keyname
+        log_cfgs.append(str(log_cfg))
+    elif "log_cfgs" in cfg and isinstance(cfg['log_cfgs'], (set, list)):
+        for a_cfg in cfg['log_cfgs']:
+            if isinstance(a_cfg, (list, set, dict)):
+                cfg_str = [str(c) for c in a_cfg]
+                log_cfgs.append('\n'.join(cfg_str))
+            else:
+                log_cfgs.append(str(a_cfg))
+
+    # See if any of them actually load...
+    am_tried = 0
+    am_worked = 0
+    for i, log_cfg in enumerate(log_cfgs):
+        try:
+            am_tried += 1
+            # Assume its just a string if not a filename
+            if log_cfg.startswith("/") and os.path.isfile(log_cfg):
+                pass
+            else:
+                log_cfg = StringIO(log_cfg)
+            # Attempt to load its config
+            logging.config.fileConfig(log_cfg)
+            am_worked += 1
+        except Exception as e:
+            sys.stderr.write(("WARN: Setup of logging config %s"
+                              " failed due to: %s\n") % (i + 1, e))
+
+    # If it didn't work, at least setup a basic logger (if desired)
+    basic_enabled = cfg.get('log_basic', True)
+    if not am_worked:
+        sys.stderr.write(("WARN: no logging configured!"
+                          " (tried %s configs)\n") % (am_tried))
+        if basic_enabled:
+            sys.stderr.write("Setting up basic logging...\n")
+            setupBasicLogging()
+
+
+def getLogger(name='cloudinit'):
+    return logging.getLogger(name)
+
+
+# Fixes this annoyance...
+# No handlers could be found for logger XXX annoying output...
+try:
+    from logging import NullHandler
+except ImportError:
+    class NullHandler(logging.Handler):
+        def emit(self, record):
+            pass
+
+
+def _resetLogger(log):
+    if not log:
+        return
+    handlers = list(log.handlers)
+    for h in handlers:
+        h.flush()
+        h.close()
+        log.removeHandler(h)
+    log.setLevel(NOTSET)
+    log.addHandler(NullHandler())
+
+
+def resetLogging():
+    _resetLogger(logging.getLogger())
+    _resetLogger(getLogger())
+
+
+resetLogging()
diff --git a/cloudinit/netinfo.py b/cloudinit/netinfo.py
index aac4af04..feba5a62 100644
--- a/cloudinit/netinfo.py
+++ b/cloudinit/netinfo.py
@@ -1,11 +1,12 @@
-#!/usr/bin/python
 # vi: ts=4 expandtab
 #
 #    Copyright (C) 2012 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Scott Moser <scott.moser@canonical.com>
 #    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -21,6 +22,8 @@
 
 import cloudinit.util as util
 
+from prettytable import PrettyTable
+
 
 def netdev_info(empty=""):
     fields = ("hwaddr", "addr", "bcast", "mask")
@@ -66,51 +69,99 @@ def netdev_info(empty=""):
                 if dev[field] == "":
                     dev[field] = empty
 
-    return(devs)
+    return devs
 
 
 def route_info():
     (route_out, _err) = util.subp(["route", "-n"])
     routes = []
-    for line in str(route_out).splitlines()[1:]:
+    entries = route_out.splitlines()[1:]
+    for line in entries:
         if not line:
             continue
         toks = line.split()
-        if toks[0] == "Kernel" or toks[0] == "Destination":
+        if len(toks) < 8 or toks[0] == "Kernel" or toks[0] == "Destination":
             continue
-        routes.append(toks)
-    return(routes)
+        entry = {
+            'destination': toks[0],
+            'gateway': toks[1],
+            'genmask': toks[2],
+            'flags': toks[3],
+            'metric': toks[4],
+            'ref': toks[5],
+            'use': toks[6],
+            'iface': toks[7],
+        }
+        routes.append(entry)
+    return routes
 
 
 def getgateway():
-    for r in route_info():
-        if r[3].find("G") >= 0:
-            return("%s[%s]" % (r[1], r[7]))
-    return(None)
+    routes = []
+    try:
+        routes = route_info()
+    except:
+        pass
+    for r in routes:
+        if r['flags'].find("G") >= 0:
+            return "%s[%s]" % (r['gateway'], r['iface'])
+    return None
 
 
-def debug_info(pre="ci-info: "):
+def netdev_pformat():
     lines = []
     try:
         netdev = netdev_info(empty=".")
     except Exception:
-        lines.append("netdev_info failed!")
-        netdev = {}
-    for (dev, d) in netdev.iteritems():
-        lines.append("%s%-6s: %i %-15s %-15s %s" %
-            (pre, dev, d["up"], d["addr"], d["mask"], d["hwaddr"]))
+        lines.append(util.center("Net device info failed", '!', 80))
+        netdev = None
+    if netdev is not None:
+        fields = ['Device', 'Up', 'Address', 'Mask', 'Hw-Address']
+        tbl = PrettyTable(fields)
+        for (dev, d) in netdev.iteritems():
+            tbl.add_row([dev, d["up"], d["addr"], d["mask"], d["hwaddr"]])
+        netdev_s = tbl.get_string()
+        max_len = len(max(netdev_s.splitlines(), key=len))
+        header = util.center("Net device info", "+", max_len)
+        lines.extend([header, netdev_s])
+    return "\n".join(lines)
+
+
+def route_pformat():
+    lines = []
     try:
         routes = route_info()
     except Exception:
-        lines.append("route_info failed")
-        routes = []
-    n = 0
-    for r in routes:
-        lines.append("%sroute-%d: %-15s %-15s %-15s %-6s %s" %
-            (pre, n, r[0], r[1], r[2], r[7], r[3]))
-        n = n + 1
-    return('\n'.join(lines))
+        lines.append(util.center('Route info failed', '!', 80))
+        routes = None
+    if routes is not None:
+        fields = ['Route', 'Destination', 'Gateway',
+                  'Genmask', 'Interface', 'Flags']
+        tbl = PrettyTable(fields)
+        for (n, r) in enumerate(routes):
+            route_id = str(n)
+            tbl.add_row([route_id, r['destination'],
+                        r['gateway'], r['genmask'],
+                        r['iface'], r['flags']])
+        route_s = tbl.get_string()
+        max_len = len(max(route_s.splitlines(), key=len))
+        header = util.center("Route info", "+", max_len)
+        lines.extend([header, route_s])
+    return "\n".join(lines)
 
 
-if __name__ == '__main__':
-    print debug_info()
+def debug_info(prefix='ci-info: '):
+    lines = []
+    netdev_lines = netdev_pformat().splitlines()
+    if prefix:
+        for line in netdev_lines:
+            lines.append("%s%s" % (prefix, line))
+    else:
+        lines.extend(netdev_lines)
+    route_lines = route_pformat().splitlines()
+    if prefix:
+        for line in route_lines:
+            lines.append("%s%s" % (prefix, line))
+    else:
+        lines.extend(route_lines)
+    return "\n".join(lines)
diff --git a/cloudinit/settings.py b/cloudinit/settings.py
new file mode 100644
index 00000000..2083cf60
--- /dev/null
+++ b/cloudinit/settings.py
@@ -0,0 +1,57 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Set and read for determining the cloud config file location
+CFG_ENV_NAME = "CLOUD_CFG"
+
+# This is expected to be a yaml formatted file
+CLOUD_CONFIG = '/etc/cloud/cloud.cfg'
+
+# What u get if no config is provided
+CFG_BUILTIN = {
+    'datasource_list': [
+        'NoCloud',
+        'ConfigDrive',
+        'OVF',
+        'MAAS',
+        'Ec2',
+        'CloudStack'
+    ],
+    'def_log_file': '/var/log/cloud-init.log',
+    'log_cfgs': [],
+    'syslog_fix_perms': 'syslog:adm',
+    'system_info': {
+        'paths': {
+            'cloud_dir': '/var/lib/cloud',
+            'templates_dir': '/etc/cloud/templates/',
+        },
+        'distro': 'ubuntu',
+    },
+}
+
+# Valid frequencies of handlers/modules
+PER_INSTANCE = "once-per-instance"
+PER_ALWAYS = "always"
+PER_ONCE = "once"
+
+# Used to sanity check incoming handlers/modules frequencies
+FREQUENCIES = [PER_INSTANCE, PER_ALWAYS, PER_ONCE]
diff --git a/cloudinit/sources/DataSourceCloudStack.py b/cloudinit/sources/DataSourceCloudStack.py
new file mode 100644
index 00000000..751bef4f
--- /dev/null
+++ b/cloudinit/sources/DataSourceCloudStack.py
@@ -0,0 +1,147 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Cosmin Luta
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Cosmin Luta <q4break@gmail.com>
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from socket import inet_ntoa
+from struct import pack
+
+import os
+import time
+
+import boto.utils as boto_utils
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+
+class DataSourceCloudStack(sources.DataSource):
+    def __init__(self, sys_cfg, distro, paths):
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+        self.seed_dir = os.path.join(paths.seed_dir, 'cs')
+        # Cloudstack has its metadata/userdata URLs located at
+        # http://<default-gateway-ip>/latest/
+        self.api_ver = 'latest'
+        gw_addr = self.get_default_gateway()
+        if not gw_addr:
+            raise RuntimeError("No default gateway found!")
+        self.metadata_address = "http://%s/" % (gw_addr)
+
+    def get_default_gateway(self):
+        """ Returns the default gateway ip address in the dotted format
+        """
+        lines = util.load_file("/proc/net/route").splitlines()
+        for line in lines:
+            items = line.split("\t")
+            if items[1] == "00000000":
+                # Found the default route, get the gateway
+                gw = inet_ntoa(pack("<L", int(items[2], 16)))
+                LOG.debug("Found default route, gateway is %s", gw)
+                return gw
+        return None
+
+    def __str__(self):
+        return util.obj_name(self)
+
+    def _get_url_settings(self):
+        mcfg = self.ds_cfg
+        if not mcfg:
+            mcfg = {}
+        max_wait = 120
+        try:
+            max_wait = int(mcfg.get("max_wait", max_wait))
+        except Exception:
+            util.logexc(LOG, "Failed to get max wait. using %s", max_wait)
+
+        if max_wait == 0:
+            return False
+
+        timeout = 50
+        try:
+            timeout = int(mcfg.get("timeout", timeout))
+        except Exception:
+            util.logexc(LOG, "Failed to get timeout, using %s", timeout)
+
+        return (max_wait, timeout)
+
+    def wait_for_metadata_service(self):
+        mcfg = self.ds_cfg
+        if not mcfg:
+            mcfg = {}
+
+        (max_wait, timeout) = self._get_url_settings()
+
+        urls = [self.metadata_address]
+        start_time = time.time()
+        url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
+                                timeout=timeout, status_cb=LOG.warn)
+
+        if url:
+            LOG.debug("Using metadata source: '%s'", url)
+        else:
+            LOG.critical(("Giving up on waiting for the metadata from %s"
+                          " after %s seconds"),
+                          urls, int(time.time() - start_time))
+
+        return bool(url)
+
+    def get_data(self):
+        seed_ret = {}
+        if util.read_optional_seed(seed_ret, base=(self.seed_dir + "/")):
+            self.userdata_raw = seed_ret['user-data']
+            self.metadata = seed_ret['meta-data']
+            LOG.debug("Using seeded cloudstack data from: %s", self.seed_dir)
+            return True
+        try:
+            if not self.wait_for_metadata_service():
+                return False
+            start_time = time.time()
+            self.userdata_raw = boto_utils.get_instance_userdata(self.api_ver,
+                None, self.metadata_address)
+            self.metadata = boto_utils.get_instance_metadata(self.api_ver,
+                self.metadata_address)
+            LOG.debug("Crawl of metadata service took %s seconds",
+                      int(time.time() - start_time))
+            return True
+        except Exception:
+            util.logexc(LOG, ('Failed fetching from metadata '
+                              'service %s'), self.metadata_address)
+            return False
+
+    def get_instance_id(self):
+        return self.metadata['instance-id']
+
+    def get_availability_zone(self):
+        return self.metadata['availability-zone']
+
+
+# Used to match classes to dependencies
+datasources = [
+  (DataSourceCloudStack, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py
new file mode 100644
index 00000000..320dd1d1
--- /dev/null
+++ b/cloudinit/sources/DataSourceConfigDrive.py
@@ -0,0 +1,226 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import os
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+# Various defaults/constants...
+DEFAULT_IID = "iid-dsconfigdrive"
+DEFAULT_MODE = 'pass'
+CFG_DRIVE_FILES = [
+    "etc/network/interfaces",
+    "root/.ssh/authorized_keys",
+    "meta.js",
+]
+DEFAULT_METADATA = {
+    "instance-id": DEFAULT_IID,
+    "dsmode": DEFAULT_MODE,
+}
+CFG_DRIVE_DEV_ENV = 'CLOUD_INIT_CONFIG_DRIVE_DEVICE'
+
+
+class DataSourceConfigDrive(sources.DataSource):
+    def __init__(self, sys_cfg, distro, paths):
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+        self.seed = None
+        self.cfg = {}
+        self.dsmode = 'local'
+        self.seed_dir = os.path.join(paths.seed_dir, 'config_drive')
+
+    def __str__(self):
+        mstr = "%s [%s]" % (util.obj_name(self), self.dsmode)
+        mstr += "[seed=%s]" % (self.seed)
+        return mstr
+
+    def get_data(self):
+        found = None
+        md = {}
+        ud = ""
+
+        if os.path.isdir(self.seed_dir):
+            try:
+                (md, ud) = read_config_drive_dir(self.seed_dir)
+                found = self.seed_dir
+            except NonConfigDriveDir:
+                util.logexc(LOG, "Failed reading config drive from %s",
+                            self.seed_dir)
+        if not found:
+            dev = find_cfg_drive_device()
+            if dev:
+                try:
+                    (md, ud) = util.mount_cb(dev, read_config_drive_dir)
+                    found = dev
+                except (NonConfigDriveDir, util.MountFailedError):
+                    pass
+
+        if not found:
+            return False
+
+        if 'dsconfig' in md:
+            self.cfg = md['dscfg']
+
+        md = util.mergedict(md, DEFAULT_METADATA)
+
+        # Update interfaces and ifup only on the local datasource
+        # this way the DataSourceConfigDriveNet doesn't do it also.
+        if 'network-interfaces' in md and self.dsmode == "local":
+            LOG.debug("Updating network interfaces from config drive (%s)",
+                     md['dsmode'])
+            self.distro.apply_network(md['network-interfaces'])
+
+        self.seed = found
+        self.metadata = md
+        self.userdata_raw = ud
+
+        if md['dsmode'] == self.dsmode:
+            return True
+
+        LOG.debug("%s: not claiming datasource, dsmode=%s", self, md['dsmode'])
+        return False
+
+    def get_public_ssh_keys(self):
+        if not 'public-keys' in self.metadata:
+            return []
+        return self.metadata['public-keys']
+
+    # The data sources' config_obj is a cloud-config formated
+    # object that came to it from ways other than cloud-config
+    # because cloud-config content would be handled elsewhere
+    def get_config_obj(self):
+        return self.cfg
+
+
+class DataSourceConfigDriveNet(DataSourceConfigDrive):
+    def __init__(self, sys_cfg, distro, paths):
+        DataSourceConfigDrive.__init__(self, sys_cfg, distro, paths)
+        self.dsmode = 'net'
+
+
+class NonConfigDriveDir(Exception):
+    pass
+
+
+def find_cfg_drive_device():
+    """ Get the config drive device.  Return a string like '/dev/vdb'
+        or None (if there is no non-root device attached). This does not
+        check the contents, only reports that if there *were* a config_drive
+        attached, it would be this device.
+        Note: per config_drive documentation, this is
+        "associated as the last available disk on the instance"
+    """
+
+    # This seems to be for debugging??
+    if CFG_DRIVE_DEV_ENV in os.environ:
+        return os.environ[CFG_DRIVE_DEV_ENV]
+
+    # We are looking for a raw block device (sda, not sda1) with a vfat
+    # filesystem on it....
+    letters = "abcdefghijklmnopqrstuvwxyz"
+    devs = util.find_devs_with("TYPE=vfat")
+
+    # Filter out anything not ending in a letter (ignore partitions)
+    devs = [f for f in devs if f[-1] in letters]
+
+    # Sort them in reverse so "last" device is first
+    devs.sort(reverse=True)
+
+    if devs:
+        return devs[0]
+
+    return None
+
+
+def read_config_drive_dir(source_dir):
+    """
+    read_config_drive_dir(source_dir):
+       read source_dir, and return a tuple with metadata dict and user-data
+       string populated.  If not a valid dir, raise a NonConfigDriveDir
+    """
+
+    # TODO: fix this for other operating systems...
+    # Ie: this is where https://fedorahosted.org/netcf/ or similar should
+    # be hooked in... (or could be)
+    found = {}
+    for af in CFG_DRIVE_FILES:
+        fn = os.path.join(source_dir, af)
+        if os.path.isfile(fn):
+            found[af] = fn
+
+    if len(found) == 0:
+        raise NonConfigDriveDir("%s: %s" % (source_dir, "no files found"))
+
+    md = {}
+    ud = ""
+    keydata = ""
+    if "etc/network/interfaces" in found:
+        fn = found["etc/network/interfaces"]
+        md['network-interfaces'] = util.load_file(fn)
+
+    if "root/.ssh/authorized_keys" in found:
+        fn = found["root/.ssh/authorized_keys"]
+        keydata = util.load_file(fn)
+
+    meta_js = {}
+    if "meta.js" in found:
+        fn = found['meta.js']
+        content = util.load_file(fn)
+        try:
+            # Just check if its really json...
+            meta_js = json.loads(content)
+            if not isinstance(meta_js, (dict)):
+                raise TypeError("Dict expected for meta.js root node")
+        except (ValueError, TypeError) as e:
+            raise NonConfigDriveDir("%s: %s, %s" %
+                (source_dir, "invalid json in meta.js", e))
+        md['meta_js'] = content
+
+    # Key data override??
+    keydata = meta_js.get('public-keys', keydata)
+    if keydata:
+        lines = keydata.splitlines()
+        md['public-keys'] = [l for l in lines
+            if len(l) and not l.startswith("#")]
+
+    for copy in ('dsmode', 'instance-id', 'dscfg'):
+        if copy in meta_js:
+            md[copy] = meta_js[copy]
+
+    if 'user-data' in meta_js:
+        ud = meta_js['user-data']
+
+    return (md, ud)
+
+
+# Used to match classes to dependencies
+datasources = [
+  (DataSourceConfigDrive, (sources.DEP_FILESYSTEM, )),
+  (DataSourceConfigDriveNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
new file mode 100644
index 00000000..cb460de1
--- /dev/null
+++ b/cloudinit/sources/DataSourceEc2.py
@@ -0,0 +1,265 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import time
+
+import boto.utils as boto_utils
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+DEF_MD_URL = "http://169.254.169.254"
+
+# Which version we are requesting of the ec2 metadata apis
+DEF_MD_VERSION = '2009-04-04'
+
+# Default metadata urls that will be used if none are provided
+# They will be checked for 'resolveability' and some of the
+# following may be discarded if they do not resolve
+DEF_MD_URLS = [DEF_MD_URL, "http://instance-data:8773"]
+
+
+class DataSourceEc2(sources.DataSource):
+    def __init__(self, sys_cfg, distro, paths):
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+        self.metadata_address = DEF_MD_URL
+        self.seed_dir = os.path.join(paths.seed_dir, "ec2")
+        self.api_ver = DEF_MD_VERSION
+
+    def __str__(self):
+        return util.obj_name(self)
+
+    def get_data(self):
+        seed_ret = {}
+        if util.read_optional_seed(seed_ret, base=(self.seed_dir + "/")):
+            self.userdata_raw = seed_ret['user-data']
+            self.metadata = seed_ret['meta-data']
+            LOG.debug("Using seeded ec2 data from %s", self.seed_dir)
+            return True
+
+        try:
+            if not self.wait_for_metadata_service():
+                return False
+            start_time = time.time()
+            self.userdata_raw = boto_utils.get_instance_userdata(self.api_ver,
+                None, self.metadata_address)
+            self.metadata = boto_utils.get_instance_metadata(self.api_ver,
+                self.metadata_address)
+            LOG.debug("Crawl of metadata service took %s seconds",
+                       int(time.time() - start_time))
+            return True
+        except Exception:
+            util.logexc(LOG, "Failed reading from metadata address %s",
+                        self.metadata_address)
+            return False
+
+    def get_instance_id(self):
+        return self.metadata['instance-id']
+
+    def get_availability_zone(self):
+        return self.metadata['placement']['availability-zone']
+
+    def get_local_mirror(self):
+        return self.get_mirror_from_availability_zone()
+
+    def get_mirror_from_availability_zone(self, availability_zone=None):
+        # Availability is like 'us-west-1b' or 'eu-west-1a'
+        if availability_zone is None:
+            availability_zone = self.get_availability_zone()
+
+        if self.is_vpc():
+            return None
+
+        # Use the distro to get the mirror
+        if not availability_zone:
+            return None
+
+        mirror_tpl = self.distro.get_option('availability_zone_template')
+        if not mirror_tpl:
+            return None
+
+        tpl_params = {
+            'zone': availability_zone.strip(),
+        }
+        mirror_url = mirror_tpl % (tpl_params)
+
+        (max_wait, timeout) = self._get_url_settings()
+        worked = uhelp.wait_for_url([mirror_url], max_wait=max_wait,
+                                timeout=timeout, status_cb=LOG.warn)
+        if not worked:
+            return None
+
+        return mirror_url
+
+    def _get_url_settings(self):
+        mcfg = self.ds_cfg
+        if not mcfg:
+            mcfg = {}
+        max_wait = 120
+        try:
+            max_wait = int(mcfg.get("max_wait", max_wait))
+        except Exception:
+            util.logexc(LOG, "Failed to get max wait. using %s", max_wait)
+
+        if max_wait == 0:
+            return False
+
+        timeout = 50
+        try:
+            timeout = int(mcfg.get("timeout", timeout))
+        except Exception:
+            util.logexc(LOG, "Failed to get timeout, using %s", timeout)
+
+        return (max_wait, timeout)
+
+    def wait_for_metadata_service(self):
+        mcfg = self.ds_cfg
+        if not mcfg:
+            mcfg = {}
+
+        (max_wait, timeout) = self._get_url_settings()
+
+        # Remove addresses from the list that wont resolve.
+        mdurls = mcfg.get("metadata_urls", DEF_MD_URLS)
+        filtered = [x for x in mdurls if util.is_resolvable_url(x)]
+
+        if set(filtered) != set(mdurls):
+            LOG.debug("Removed the following from metadata urls: %s",
+                      list((set(mdurls) - set(filtered))))
+
+        if len(filtered):
+            mdurls = filtered
+        else:
+            LOG.warn("Empty metadata url list! using default list")
+            mdurls = DEF_MD_URLS
+
+        urls = []
+        url2base = {}
+        for url in mdurls:
+            cur = "%s/%s/meta-data/instance-id" % (url, self.api_ver)
+            urls.append(cur)
+            url2base[cur] = url
+
+        start_time = time.time()
+        url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
+                                timeout=timeout, status_cb=LOG.warn)
+
+        if url:
+            LOG.debug("Using metadata source: '%s'", url2base[url])
+        else:
+            LOG.critical("Giving up on md from %s after %s seconds",
+                            urls, int(time.time() - start_time))
+
+        self.metadata_address = url2base.get(url)
+        return bool(url)
+
+    def _remap_device(self, short_name):
+        # LP: #611137
+        # the metadata service may believe that devices are named 'sda'
+        # when the kernel named them 'vda' or 'xvda'
+        # we want to return the correct value for what will actually
+        # exist in this instance
+        mappings = {"sd": ("vd", "xvd")}
+        for (nfrom, tlist) in mappings.iteritems():
+            if not short_name.startswith(nfrom):
+                continue
+            for nto in tlist:
+                cand = "/dev/%s%s" % (nto, short_name[len(nfrom):])
+                if os.path.exists(cand):
+                    return cand
+        return None
+
+    def device_name_to_device(self, name):
+        # Consult metadata service, that has
+        #  ephemeral0: sdb
+        # and return 'sdb' for input 'ephemeral0'
+        if 'block-device-mapping' not in self.metadata:
+            return None
+
+        # Example:
+        # 'block-device-mapping':
+        # {'ami': '/dev/sda1',
+        # 'ephemeral0': '/dev/sdb',
+        # 'root': '/dev/sda1'}
+        found = None
+        bdm_items = self.metadata['block-device-mapping'].iteritems()
+        for (entname, device) in bdm_items:
+            if entname == name:
+                found = device
+                break
+            # LP: #513842 mapping in Euca has 'ephemeral' not 'ephemeral0'
+            if entname == "ephemeral" and name == "ephemeral0":
+                found = device
+
+        if found is None:
+            LOG.debug("Unable to convert %s to a device", name)
+            return None
+
+        ofound = found
+        if not found.startswith("/"):
+            found = "/dev/%s" % found
+
+        if os.path.exists(found):
+            return found
+
+        remapped = self._remap_device(os.path.basename(found))
+        if remapped:
+            LOG.debug("Remapped device name %s => %s", (found, remapped))
+            return remapped
+
+        # On t1.micro, ephemeral0 will appear in block-device-mapping from
+        # metadata, but it will not exist on disk (and never will)
+        # at this point, we've verified that the path did not exist
+        # in the special case of 'ephemeral0' return None to avoid bogus
+        # fstab entry (LP: #744019)
+        if name == "ephemeral0":
+            return None
+        return ofound
+
+    def is_vpc(self):
+        # See: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/615545
+        # Detect that the machine was launched in a VPC.
+        # But I did notice that when in a VPC, meta-data
+        # does not have public-ipv4 and public-hostname
+        # listed as a possibility.
+        ph = "public-hostname"
+        p4 = "public-ipv4"
+        if ((ph not in self.metadata or self.metadata[ph] == "") and
+            (p4 not in self.metadata or self.metadata[p4] == "")):
+            return True
+        return False
+
+
+# Used to match classes to dependencies
+datasources = [
+  (DataSourceEc2, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/sources/DataSourceMAAS.py b/cloudinit/sources/DataSourceMAAS.py
new file mode 100644
index 00000000..f16d5c21
--- /dev/null
+++ b/cloudinit/sources/DataSourceMAAS.py
@@ -0,0 +1,264 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import errno
+import oauth.oauth as oauth
+import os
+import time
+import urllib2
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+MD_VERSION = "2012-03-01"
+
+
+class DataSourceMAAS(sources.DataSource):
+    """
+    DataSourceMAAS reads instance information from MAAS.
+    Given a config metadata_url, and oauth tokens, it expects to find
+    files under the root named:
+      instance-id
+      user-data
+      hostname
+    """
+    def __init__(self, sys_cfg, distro, paths):
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+        self.base_url = None
+        self.seed_dir = os.path.join(paths.seed_dir, 'maas')
+
+    def __str__(self):
+        return "%s [%s]" % (util.obj_name(self), self.base_url)
+
+    def get_data(self):
+        mcfg = self.ds_cfg
+
+        try:
+            (userdata, metadata) = read_maas_seed_dir(self.seed_dir)
+            self.userdata_raw = userdata
+            self.metadata = metadata
+            self.base_url = self.seed_dir
+            return True
+        except MAASSeedDirNone:
+            pass
+        except MAASSeedDirMalformed as exc:
+            LOG.warn("%s was malformed: %s" % (self.seed_dir, exc))
+            raise
+
+        # If there is no metadata_url, then we're not configured
+        url = mcfg.get('metadata_url', None)
+        if not url:
+            return False
+
+        try:
+            if not self.wait_for_metadata_service(url):
+                return False
+
+            self.base_url = url
+
+            (userdata, metadata) = read_maas_seed_url(self.base_url,
+                                                      self.md_headers)
+            self.userdata_raw = userdata
+            self.metadata = metadata
+            return True
+        except Exception:
+            util.logexc(LOG, "Failed fetching metadata from url %s", url)
+            return False
+
+    def md_headers(self, url):
+        mcfg = self.ds_cfg
+
+        # If we are missing token_key, token_secret or consumer_key
+        # then just do non-authed requests
+        for required in ('token_key', 'token_secret', 'consumer_key'):
+            if required not in mcfg:
+                return {}
+
+        consumer_secret = mcfg.get('consumer_secret', "")
+        return oauth_headers(url=url,
+                             consumer_key=mcfg['consumer_key'],
+                             token_key=mcfg['token_key'],
+                             token_secret=mcfg['token_secret'],
+                             consumer_secret=consumer_secret)
+
+    def wait_for_metadata_service(self, url):
+        mcfg = self.ds_cfg
+
+        max_wait = 120
+        try:
+            max_wait = int(mcfg.get("max_wait", max_wait))
+        except Exception:
+            util.logexc(LOG, "Failed to get max wait. using %s", max_wait)
+
+        if max_wait == 0:
+            return False
+
+        timeout = 50
+        try:
+            if timeout in mcfg:
+                timeout = int(mcfg.get("timeout", timeout))
+        except Exception:
+            LOG.warn("Failed to get timeout, using %s" % timeout)
+
+        starttime = time.time()
+        check_url = "%s/%s/meta-data/instance-id" % (url, MD_VERSION)
+        urls = [check_url]
+        url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
+                                 timeout=timeout, status_cb=LOG.warn,
+                                 headers_cb=self.md_headers)
+
+        if url:
+            LOG.debug("Using metadata source: '%s'", url)
+        else:
+            LOG.critical("Giving up on md from %s after %i seconds",
+                            urls, int(time.time() - starttime))
+
+        return bool(url)
+
+
+def read_maas_seed_dir(seed_d):
+    """
+    Return user-data and metadata for a maas seed dir in seed_d.
+    Expected format of seed_d are the following files:
+      * instance-id
+      * local-hostname
+      * user-data
+    """
+    if not os.path.isdir(seed_d):
+        raise MAASSeedDirNone("%s: not a directory")
+
+    files = ('local-hostname', 'instance-id', 'user-data', 'public-keys')
+    md = {}
+    for fname in files:
+        try:
+            md[fname] = util.load_file(os.path.join(seed_d, fname))
+        except IOError as e:
+            if e.errno != errno.ENOENT:
+                raise
+
+    return check_seed_contents(md, seed_d)
+
+
+def read_maas_seed_url(seed_url, header_cb=None, timeout=None,
+    version=MD_VERSION):
+    """
+    Read the maas datasource at seed_url.
+    header_cb is a method that should return a headers dictionary that will
+    be given to urllib2.Request()
+
+    Expected format of seed_url is are the following files:
+      * <seed_url>/<version>/meta-data/instance-id
+      * <seed_url>/<version>/meta-data/local-hostname
+      * <seed_url>/<version>/user-data
+    """
+    base_url = "%s/%s" % (seed_url, version)
+    file_order = [
+        'local-hostname',
+        'instance-id',
+        'public-keys',
+        'user-data',
+    ]
+    files = {
+        'local-hostname': "%s/%s" % (base_url, 'meta-data/local-hostname'),
+        'instance-id': "%s/%s" % (base_url, 'meta-data/instance-id'),
+        'public-keys': "%s/%s" % (base_url, 'meta-data/public-keys'),
+        'user-data': "%s/%s" % (base_url, 'user-data'),
+    }
+    md = {}
+    for name in file_order:
+        url = files.get(name)
+        if header_cb:
+            headers = header_cb(url)
+        else:
+            headers = {}
+        try:
+            resp = uhelp.readurl(url, headers=headers, timeout=timeout)
+            if resp.ok():
+                md[name] = str(resp)
+            else:
+                LOG.warn(("Fetching from %s resulted in"
+                          " an invalid http code %s"), url, resp.code)
+        except urllib2.HTTPError as e:
+            if e.code != 404:
+                raise
+    return check_seed_contents(md, seed_url)
+
+
+def check_seed_contents(content, seed):
+    """Validate if content is Is the content a dict that is valid as a
+       return for a datasource.
+       Either return a (userdata, metadata) tuple or
+       Raise MAASSeedDirMalformed or MAASSeedDirNone
+    """
+    md_required = ('instance-id', 'local-hostname')
+    if len(content) == 0:
+        raise MAASSeedDirNone("%s: no data files found" % seed)
+
+    found = list(content.keys())
+    missing = [k for k in md_required if k not in found]
+    if len(missing):
+        raise MAASSeedDirMalformed("%s: missing files %s" % (seed, missing))
+
+    userdata = content.get('user-data', "")
+    md = {}
+    for (key, val) in content.iteritems():
+        if key == 'user-data':
+            continue
+        md[key] = val
+
+    return (userdata, md)
+
+
+def oauth_headers(url, consumer_key, token_key, token_secret, consumer_secret):
+    consumer = oauth.OAuthConsumer(consumer_key, consumer_secret)
+    token = oauth.OAuthToken(token_key, token_secret)
+    params = {
+        'oauth_version': "1.0",
+        'oauth_nonce': oauth.generate_nonce(),
+        'oauth_timestamp': int(time.time()),
+        'oauth_token': token.key,
+        'oauth_consumer_key': consumer.key,
+    }
+    req = oauth.OAuthRequest(http_url=url, parameters=params)
+    req.sign_request(oauth.OAuthSignatureMethod_PLAINTEXT(),
+                     consumer, token)
+    return req.to_header()
+
+
+class MAASSeedDirNone(Exception):
+    pass
+
+
+class MAASSeedDirMalformed(Exception):
+    pass
+
+
+# Used to match classes to dependencies
+datasources = [
+  (DataSourceMAAS, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/DataSourceNoCloud.py b/cloudinit/sources/DataSourceNoCloud.py
index e8c56b8f..bed500a2 100644
--- a/cloudinit/DataSourceNoCloud.py
+++ b/cloudinit/sources/DataSourceNoCloud.py
@@ -2,9 +2,11 @@
 #
 #    Copyright (C) 2009-2010 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Scott Moser <scott.moser@canonical.com>
 #    Author: Juerg Hafliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -18,33 +20,34 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.DataSource as DataSource
-
-from cloudinit import seeddir as base_seeddir
-from cloudinit import log
-import cloudinit.util as util
 import errno
-import subprocess
+import os
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import util
 
+LOG = logging.getLogger(__name__)
 
-class DataSourceNoCloud(DataSource.DataSource):
-    metadata = None
-    userdata = None
-    userdata_raw = None
-    supported_seed_starts = ("/", "file://")
-    dsmode = "local"
-    seed = None
-    cmdline_id = "ds=nocloud"
-    seeddir = base_seeddir + '/nocloud'
+
+class DataSourceNoCloud(sources.DataSource):
+    def __init__(self, sys_cfg, distro, paths):
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+        self.dsmode = 'local'
+        self.seed = None
+        self.cmdline_id = "ds=nocloud"
+        self.seed_dir = os.path.join(paths.seed_dir, 'nocloud')
+        self.supported_seed_starts = ("/", "file://")
 
     def __str__(self):
-        mstr = "DataSourceNoCloud"
-        mstr = mstr + " [seed=%s]" % self.seed
-        return(mstr)
+        mstr = "%s [seed=%s][dsmode=%s]" % (util.obj_name(self),
+                                            self.seed, self.dsmode)
+        return mstr
 
     def get_data(self):
         defaults = {
-            "instance-id": "nocloud", "dsmode": self.dsmode
+            "instance-id": "nocloud",
+            "dsmode": self.dsmode,
         }
 
         found = []
@@ -52,24 +55,24 @@ class DataSourceNoCloud(DataSource.DataSource):
         ud = ""
 
         try:
-            # parse the kernel command line, getting data passed in
+            # Parse the kernel command line, getting data passed in
             if parse_cmdline_data(self.cmdline_id, md):
                 found.append("cmdline")
         except:
-            util.logexc(log)
+            util.logexc(LOG, "Unable to parse command line data")
             return False
 
-        # check to see if the seeddir has data.
+        # Check to see if the seed dir has data.
         seedret = {}
-        if util.read_optional_seed(seedret, base=self.seeddir + "/"):
+        if util.read_optional_seed(seedret, base=self.seed_dir + "/"):
             md = util.mergedict(md, seedret['meta-data'])
             ud = seedret['user-data']
-            found.append(self.seeddir)
-            log.debug("using seeded cache data in %s" % self.seeddir)
+            found.append(self.seed_dir)
+            LOG.debug("Using seeded cache data from %s", self.seed_dir)
 
-        # if the datasource config had a 'seedfrom' entry, then that takes
+        # If the datasource config had a 'seedfrom' entry, then that takes
         # precedence over a 'seedfrom' that was found in a filesystem
-        # but not over external medi
+        # but not over external media
         if 'seedfrom' in self.ds_cfg and self.ds_cfg['seedfrom']:
             found.append("ds_config")
             md["seedfrom"] = self.ds_cfg['seedfrom']
@@ -83,35 +86,37 @@ class DataSourceNoCloud(DataSource.DataSource):
 
         for dev in devlist:
             try:
-                (newmd, newud) = util.mount_callback_umount(dev,
-                    util.read_seeded)
+                LOG.debug("Attempting to use data from %s", dev)
+
+                (newmd, newud) = util.mount_cb(dev, util.read_seeded)
                 md = util.mergedict(newmd, md)
                 ud = newud
 
-                # for seed from a device, the default mode is 'net'.
+                # For seed from a device, the default mode is 'net'.
                 # that is more likely to be what is desired.
                 # If they want dsmode of local, then they must
                 # specify that.
                 if 'dsmode' not in md:
                     md['dsmode'] = "net"
 
-                log.debug("using data from %s" % dev)
+                LOG.debug("Using data from %s", dev)
                 found.append(dev)
                 break
-            except OSError, e:
+            except OSError as e:
                 if e.errno != errno.ENOENT:
                     raise
-            except util.mountFailedError:
-                log.warn("Failed to mount %s when looking for seed" % dev)
+            except util.MountFailedError:
+                util.logexc(LOG, ("Failed to mount %s"
+                                  " when looking for data"), dev)
 
-        # there was no indication on kernel cmdline or data
+        # There was no indication on kernel cmdline or data
         # in the seeddir suggesting this handler should be used.
         if len(found) == 0:
             return False
 
         seeded_interfaces = None
 
-        # the special argument "seedfrom" indicates we should
+        # The special argument "seedfrom" indicates we should
         # attempt to seed the userdata / metadata from its value
         # its primarily value is in allowing the user to type less
         # on the command line, ie: ds=nocloud;s=http://bit.ly/abcdefg
@@ -123,57 +128,46 @@ class DataSourceNoCloud(DataSource.DataSource):
                     seedfound = proto
                     break
             if not seedfound:
-                log.debug("seed from %s not supported by %s" %
-                    (seedfrom, self.__class__))
+                LOG.debug("Seed from %s not supported by %s", seedfrom, self)
                 return False
 
             if 'network-interfaces' in md:
                 seeded_interfaces = self.dsmode
 
-            # this could throw errors, but the user told us to do it
+            # This could throw errors, but the user told us to do it
             # so if errors are raised, let them raise
             (md_seed, ud) = util.read_seeded(seedfrom, timeout=None)
-            log.debug("using seeded cache data from %s" % seedfrom)
+            LOG.debug("Using seeded cache data from %s", seedfrom)
 
-            # values in the command line override those from the seed
+            # Values in the command line override those from the seed
             md = util.mergedict(md, md_seed)
             found.append(seedfrom)
 
+        # Now that we have exhausted any other places merge in the defaults
         md = util.mergedict(md, defaults)
 
-        # update the network-interfaces if metadata had 'network-interfaces'
+        # Update the network-interfaces if metadata had 'network-interfaces'
         # entry and this is the local datasource, or 'seedfrom' was used
         # and the source of the seed was self.dsmode
         # ('local' for NoCloud, 'net' for NoCloudNet')
         if ('network-interfaces' in md and
             (self.dsmode in ("local", seeded_interfaces))):
-            log.info("updating network interfaces from nocloud")
-
-            util.write_file("/etc/network/interfaces",
-                md['network-interfaces'])
-            try:
-                (out, err) = util.subp(['ifup', '--all'])
-                if len(out) or len(err):
-                    log.warn("ifup --all had stderr: %s" % err)
-
-            except subprocess.CalledProcessError as exc:
-                log.warn("ifup --all failed: %s" % (exc.output[1]))
-
-        self.seed = ",".join(found)
-        self.metadata = md
-        self.userdata_raw = ud
+            LOG.debug("Updating network interfaces from %s", self)
+            self.distro.apply_network(md['network-interfaces'])
 
         if md['dsmode'] == self.dsmode:
+            self.seed = ",".join(found)
+            self.metadata = md
+            self.userdata_raw = ud
             return True
 
-        log.debug("%s: not claiming datasource, dsmode=%s" %
-            (self, md['dsmode']))
+        LOG.debug("%s: not claiming datasource, dsmode=%s", self, md['dsmode'])
         return False
 
 
-# returns true or false indicating if cmdline indicated
+# Returns true or false indicating if cmdline indicated
 # that this module should be used
-# example cmdline:
+# Example cmdline:
 #  root=LABEL=uec-rootfs ro ds=nocloud
 def parse_cmdline_data(ds_id, fill, cmdline=None):
     if cmdline is None:
@@ -210,23 +204,25 @@ def parse_cmdline_data(ds_id, fill, cmdline=None):
             k = s2l[k]
         fill[k] = v
 
-    return(True)
+    return True
 
 
 class DataSourceNoCloudNet(DataSourceNoCloud):
-    cmdline_id = "ds=nocloud-net"
-    supported_seed_starts = ("http://", "https://", "ftp://")
-    seeddir = base_seeddir + '/nocloud-net'
-    dsmode = "net"
+    def __init__(self, sys_cfg, distro, paths):
+        DataSourceNoCloud.__init__(self, sys_cfg, distro, paths)
+        self.cmdline_id = "ds=nocloud-net"
+        self.supported_seed_starts = ("http://", "https://", "ftp://")
+        self.seed_dir = os.path.join(paths.seed_dir, 'nocloud-net')
+        self.dsmode = "net"
 
 
-datasources = (
-  (DataSourceNoCloud, (DataSource.DEP_FILESYSTEM, )),
-  (DataSourceNoCloudNet,
-    (DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK)),
-)
+# Used to match classes to dependencies
+datasources = [
+  (DataSourceNoCloud, (sources.DEP_FILESYSTEM, )),
+  (DataSourceNoCloudNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
 
 
-# return a list of data sources that match this set of dependencies
+# Return a list of data sources that match this set of dependencies
 def get_datasource_list(depends):
-    return(DataSource.list_from_depends(depends, datasources))
+    return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py
index a0b1b518..7728b36f 100644
--- a/cloudinit/DataSourceOVF.py
+++ b/cloudinit/sources/DataSourceOVF.py
@@ -2,9 +2,11 @@
 #
 #    Copyright (C) 2011 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Scott Moser <scott.moser@canonical.com>
 #    Author: Juerg Hafliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -18,33 +20,30 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.DataSource as DataSource
-
-from cloudinit import seeddir as base_seeddir
-from cloudinit import log
-import cloudinit.util as util
-import os.path
-import os
 from xml.dom import minidom
+
 import base64
+import os
 import re
-import tempfile
-import subprocess
 
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
 
-class DataSourceOVF(DataSource.DataSource):
-    seed = None
-    seeddir = base_seeddir + '/ovf'
-    environment = None
-    cfg = {}
-    userdata_raw = None
-    metadata = None
-    supported_seed_starts = ("/", "file://")
+
+class DataSourceOVF(sources.DataSource):
+    def __init__(self, sys_cfg, distro, paths):
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+        self.seed = None
+        self.seed_dir = os.path.join(paths.seed_dir, 'ovf')
+        self.environment = None
+        self.cfg = {}
+        self.supported_seed_starts = ("/", "file://")
 
     def __str__(self):
-        mstr = "DataSourceOVF"
-        mstr = mstr + " [seed=%s]" % self.seed
-        return(mstr)
+        return "%s [seed=%s]" % (util.obj_name(self), self.seed)
 
     def get_data(self):
         found = []
@@ -52,26 +51,24 @@ class DataSourceOVF(DataSource.DataSource):
         ud = ""
 
         defaults = {
-            "instance-id": "iid-dsovf"
+            "instance-id": "iid-dsovf",
         }
 
-        (seedfile, contents) = get_ovf_env(base_seeddir)
+        (seedfile, contents) = get_ovf_env(self.paths.seed_dir)
         if seedfile:
-            # found a seed dir
-            seed = "%s/%s" % (base_seeddir, seedfile)
+            # Found a seed dir
+            seed = os.path.join(self.paths.seed_dir, seedfile)
             (md, ud, cfg) = read_ovf_environment(contents)
             self.environment = contents
-
             found.append(seed)
         else:
             np = {'iso': transport_iso9660,
                   'vmware-guestd': transport_vmware_guestd, }
             name = None
-            for name, transfunc in np.iteritems():
+            for (name, transfunc) in np.iteritems():
                 (contents, _dev, _fname) = transfunc()
                 if contents:
                     break
-
             if contents:
                 (md, ud, cfg) = read_ovf_environment(contents)
                 self.environment = contents
@@ -89,17 +86,19 @@ class DataSourceOVF(DataSource.DataSource):
                     seedfound = proto
                     break
             if not seedfound:
-                log.debug("seed from %s not supported by %s" %
-                    (seedfrom, self.__class__))
+                LOG.debug("Seed from %s not supported by %s",
+                          seedfrom, self)
                 return False
 
             (md_seed, ud) = util.read_seeded(seedfrom, timeout=None)
-            log.debug("using seeded cache data from %s" % seedfrom)
+            LOG.debug("Using seeded cache data from %s", seedfrom)
 
             md = util.mergedict(md, md_seed)
             found.append(seedfrom)
 
+        # Now that we have exhausted any other places merge in the defaults
         md = util.mergedict(md, defaults)
+
         self.seed = ",".join(found)
         self.metadata = md
         self.userdata_raw = ud
@@ -108,31 +107,37 @@ class DataSourceOVF(DataSource.DataSource):
 
     def get_public_ssh_keys(self):
         if not 'public-keys' in self.metadata:
-            return([])
-        return([self.metadata['public-keys'], ])
+            return []
+        pks = self.metadata['public-keys']
+        if isinstance(pks, (list)):
+            return pks
+        else:
+            return [pks]
 
-    # the data sources' config_obj is a cloud-config formated
+    # The data sources' config_obj is a cloud-config formatted
     # object that came to it from ways other than cloud-config
     # because cloud-config content would be handled elsewhere
     def get_config_obj(self):
-        return(self.cfg)
+        return self.cfg
 
 
 class DataSourceOVFNet(DataSourceOVF):
-    seeddir = base_seeddir + '/ovf-net'
-    supported_seed_starts = ("http://", "https://", "ftp://")
+    def __init__(self, sys_cfg, distro, paths):
+        DataSourceOVF.__init__(self, sys_cfg, distro, paths)
+        self.seed_dir = os.path.join(paths.seed_dir, 'ovf-net')
+        self.supported_seed_starts = ("http://", "https://", "ftp://")
 
 
-# this will return a dict with some content
-#  meta-data, user-data
+# This will return a dict with some content
+#  meta-data, user-data, some config
 def read_ovf_environment(contents):
-    props = getProperties(contents)
+    props = get_properties(contents)
     md = {}
     cfg = {}
     ud = ""
-    cfg_props = ['password', ]
+    cfg_props = ['password']
     md_props = ['seedfrom', 'local-hostname', 'public-keys', 'instance-id']
-    for prop, val in props.iteritems():
+    for (prop, val) in props.iteritems():
         if prop == 'hostname':
             prop = "local-hostname"
         if prop in md_props:
@@ -144,23 +149,25 @@ def read_ovf_environment(contents):
                 ud = base64.decodestring(val)
             except:
                 ud = val
-    return(md, ud, cfg)
+    return (md, ud, cfg)
 
 
-# returns tuple of filename (in 'dirname', and the contents of the file)
+# Returns tuple of filename (in 'dirname', and the contents of the file)
 # on "not found", returns 'None' for filename and False for contents
 def get_ovf_env(dirname):
     env_names = ("ovf-env.xml", "ovf_env.xml", "OVF_ENV.XML", "OVF-ENV.XML")
     for fname in env_names:
-        if os.path.isfile("%s/%s" % (dirname, fname)):
-            fp = open("%s/%s" % (dirname, fname))
-            contents = fp.read()
-            fp.close()
-            return(fname, contents)
-    return(None, False)
+        full_fn = os.path.join(dirname, fname)
+        if os.path.isfile(full_fn):
+            try:
+                contents = util.load_file(full_fn)
+                return (fname, contents)
+            except:
+                util.logexc(LOG, "Failed loading ovf file %s", full_fn)
+    return (None, False)
 
 
-# transport functions take no input and return
+# Transport functions take no input and return
 # a 3 tuple of content, path, filename
 def transport_iso9660(require_iso=True):
 
@@ -173,79 +180,46 @@ def transport_iso9660(require_iso=True):
     devname_regex = os.environ.get(envname, default_regex)
     cdmatch = re.compile(devname_regex)
 
-    # go through mounts to see if it was already mounted
-    fp = open("/proc/mounts")
-    mounts = fp.readlines()
-    fp.close()
-
-    mounted = {}
-    for mpline in mounts:
-        (dev, mp, fstype, _opts, _freq, _passno) = mpline.split()
-        mounted[dev] = (dev, fstype, mp, False)
-        mp = mp.replace("\\040", " ")
+    # Go through mounts to see if it was already mounted
+    mounts = util.mounts()
+    for (dev, info) in mounts.iteritems():
+        fstype = info['fstype']
         if fstype != "iso9660" and require_iso:
             continue
-
-        if cdmatch.match(dev[5:]) == None:  # take off '/dev/'
+        if cdmatch.match(dev[5:]) is None:  # take off '/dev/'
             continue
-
+        mp = info['mountpoint']
         (fname, contents) = get_ovf_env(mp)
         if contents is not False:
-            return(contents, dev, fname)
-
-    tmpd = None
-    dvnull = None
+            return (contents, dev, fname)
 
     devs = os.listdir("/dev/")
     devs.sort()
-
     for dev in devs:
-        fullp = "/dev/%s" % dev
+        fullp = os.path.join("/dev/", dev)
 
-        if fullp in mounted or not cdmatch.match(dev) or os.path.isdir(fullp):
+        if (fullp in mounts or
+            not cdmatch.match(dev) or os.path.isdir(fullp)):
             continue
 
-        fp = None
         try:
-            fp = open(fullp, "rb")
-            fp.read(512)
-            fp.close()
+            # See if we can read anything at all...??
+            with open(fullp, 'rb') as fp:
+                fp.read(512)
         except:
-            if fp:
-                fp.close()
             continue
 
-        if tmpd is None:
-            tmpd = tempfile.mkdtemp()
-        if dvnull is None:
-            try:
-                dvnull = open("/dev/null")
-            except:
-                pass
-
-        cmd = ["mount", "-o", "ro", fullp, tmpd]
-        if require_iso:
-            cmd.extend(('-t', 'iso9660'))
-
-        rc = subprocess.call(cmd, stderr=dvnull, stdout=dvnull, stdin=dvnull)
-        if rc:
+        try:
+            (fname, contents) = util.mount_cb(fullp,
+                                               get_ovf_env, mtype="iso9660")
+        except util.MountFailedError:
+            util.logexc(LOG, "Failed mounting %s", fullp)
             continue
 
-        (fname, contents) = get_ovf_env(tmpd)
-
-        subprocess.call(["umount", tmpd])
-
         if contents is not False:
-            os.rmdir(tmpd)
-            return(contents, fullp, fname)
-
-    if tmpd:
-        os.rmdir(tmpd)
-
-    if dvnull:
-        dvnull.close()
+            return (contents, fullp, fname)
 
-    return(False, None, None)
+    return (False, None, None)
 
 
 def transport_vmware_guestd():
@@ -259,74 +233,61 @@ def transport_vmware_guestd():
     #     # would need to error check here and see why this failed
     #     # to know if log/error should be raised
     #     return(False, None, None)
-    return(False, None, None)
+    return (False, None, None)
 
 
-def findChild(node, filter_func):
+def find_child(node, filter_func):
     ret = []
     if not node.hasChildNodes():
         return ret
     for child in node.childNodes:
         if filter_func(child):
             ret.append(child)
-    return(ret)
+    return ret
 
 
-def getProperties(environString):
-    dom = minidom.parseString(environString)
+def get_properties(contents):
+
+    dom = minidom.parseString(contents)
     if dom.documentElement.localName != "Environment":
-        raise Exception("No Environment Node")
+        raise XmlError("No Environment Node")
 
     if not dom.documentElement.hasChildNodes():
-        raise Exception("No Child Nodes")
+        raise XmlError("No Child Nodes")
 
     envNsURI = "http://schemas.dmtf.org/ovf/environment/1"
 
     # could also check here that elem.namespaceURI ==
     #   "http://schemas.dmtf.org/ovf/environment/1"
-    propSections = findChild(dom.documentElement,
+    propSections = find_child(dom.documentElement,
         lambda n: n.localName == "PropertySection")
 
     if len(propSections) == 0:
-        raise Exception("No 'PropertySection's")
+        raise XmlError("No 'PropertySection's")
 
     props = {}
-    propElems = findChild(propSections[0], lambda n: n.localName == "Property")
+    propElems = find_child(propSections[0],
+                            (lambda n: n.localName == "Property"))
 
     for elem in propElems:
         key = elem.attributes.getNamedItemNS(envNsURI, "key").value
         val = elem.attributes.getNamedItemNS(envNsURI, "value").value
         props[key] = val
 
-    return(props)
+    return props
+
+
+class XmlError(Exception):
+    pass
 
 
+# Used to match classes to dependencies
 datasources = (
-  (DataSourceOVF, (DataSource.DEP_FILESYSTEM, )),
-  (DataSourceOVFNet,
-    (DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK)),
+  (DataSourceOVF, (sources.DEP_FILESYSTEM, )),
+  (DataSourceOVFNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
 )
 
 
-# return a list of data sources that match this set of dependencies
+# Return a list of data sources that match this set of dependencies
 def get_datasource_list(depends):
-    return(DataSource.list_from_depends(depends, datasources))
-
-
-if __name__ == "__main__":
-    def main():
-        import sys
-        envStr = open(sys.argv[1]).read()
-        props = getProperties(envStr)
-        import pprint
-        pprint.pprint(props)
-
-        md, ud, cfg = read_ovf_environment(envStr)
-        print "=== md ==="
-        pprint.pprint(md)
-        print "=== ud ==="
-        pprint.pprint(ud)
-        print "=== cfg ==="
-        pprint.pprint(cfg)
-
-    main()
+    return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
new file mode 100644
index 00000000..b25724a5
--- /dev/null
+++ b/cloudinit/sources/__init__.py
@@ -0,0 +1,223 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import abc
+
+from cloudinit import importer
+from cloudinit import log as logging
+from cloudinit import user_data as ud
+from cloudinit import util
+
+DEP_FILESYSTEM = "FILESYSTEM"
+DEP_NETWORK = "NETWORK"
+DS_PREFIX = 'DataSource'
+
+LOG = logging.getLogger(__name__)
+
+
+class DataSourceNotFoundException(Exception):
+    pass
+
+
+class DataSource(object):
+
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self, sys_cfg, distro, paths, ud_proc=None):
+        self.sys_cfg = sys_cfg
+        self.distro = distro
+        self.paths = paths
+        self.userdata = None
+        self.metadata = None
+        self.userdata_raw = None
+        name = util.obj_name(self)
+        if name.startswith(DS_PREFIX):
+            name = name[len(DS_PREFIX):]
+        self.ds_cfg = util.get_cfg_by_path(self.sys_cfg,
+                                          ("datasource", name), {})
+        if not ud_proc:
+            self.ud_proc = ud.UserDataProcessor(self.paths)
+        else:
+            self.ud_proc = ud_proc
+
+    def get_userdata(self):
+        if self.userdata is None:
+            raw_data = self.get_userdata_raw()
+            self.userdata = self.ud_proc.process(raw_data)
+        return self.userdata
+
+    def get_userdata_raw(self):
+        return self.userdata_raw
+
+    # the data sources' config_obj is a cloud-config formated
+    # object that came to it from ways other than cloud-config
+    # because cloud-config content would be handled elsewhere
+    def get_config_obj(self):
+        return {}
+
+    def get_public_ssh_keys(self):
+        keys = []
+
+        if not self.metadata or 'public-keys' not in self.metadata:
+            return keys
+
+        if isinstance(self.metadata['public-keys'], (basestring, str)):
+            return str(self.metadata['public-keys']).splitlines()
+
+        if isinstance(self.metadata['public-keys'], (list, set)):
+            return list(self.metadata['public-keys'])
+
+        if isinstance(self.metadata['public-keys'], (dict)):
+            for (_keyname, klist) in self.metadata['public-keys'].iteritems():
+                # lp:506332 uec metadata service responds with
+                # data that makes boto populate a string for 'klist' rather
+                # than a list.
+                if isinstance(klist, (str, basestring)):
+                    klist = [klist]
+                if isinstance(klist, (list, set)):
+                    for pkey in klist:
+                        # There is an empty string at
+                        # the end of the keylist, trim it
+                        if pkey:
+                            keys.append(pkey)
+
+        return keys
+
+    def device_name_to_device(self, _name):
+        # translate a 'name' to a device
+        # the primary function at this point is on ec2
+        # to consult metadata service, that has
+        #  ephemeral0: sdb
+        # and return 'sdb' for input 'ephemeral0'
+        return None
+
+    def get_locale(self):
+        return 'en_US.UTF-8'
+
+    def get_local_mirror(self):
+        # ??
+        return None
+
+    def get_instance_id(self):
+        if not self.metadata or 'instance-id' not in self.metadata:
+            # Return a magic not really instance id string
+            return "iid-datasource"
+        return str(self.metadata['instance-id'])
+
+    def get_hostname(self, fqdn=False):
+        defdomain = "localdomain"
+        defhost = "localhost"
+        domain = defdomain
+
+        if not self.metadata or not 'local-hostname' in self.metadata:
+            # this is somewhat questionable really.
+            # the cloud datasource was asked for a hostname
+            # and didn't have one. raising error might be more appropriate
+            # but instead, basically look up the existing hostname
+            toks = []
+            hostname = util.get_hostname()
+            fqdn = util.get_fqdn_from_hosts(hostname)
+            if fqdn and fqdn.find(".") > 0:
+                toks = str(fqdn).split(".")
+            elif hostname:
+                toks = [hostname, defdomain]
+            else:
+                toks = [defhost, defdomain]
+        else:
+            # if there is an ipv4 address in 'local-hostname', then
+            # make up a hostname (LP: #475354) in format ip-xx.xx.xx.xx
+            lhost = self.metadata['local-hostname']
+            if util.is_ipv4(lhost):
+                toks = "ip-%s" % lhost.replace(".", "-")
+            else:
+                toks = lhost.split(".")
+
+        if len(toks) > 1:
+            hostname = toks[0]
+            domain = '.'.join(toks[1:])
+        else:
+            hostname = toks[0]
+
+        if fqdn:
+            return "%s.%s" % (hostname, domain)
+        else:
+            return hostname
+
+
+def find_source(sys_cfg, distro, paths, ds_deps, cfg_list, pkg_list):
+    ds_list = list_sources(cfg_list, ds_deps, pkg_list)
+    ds_names = [util.obj_name(f) for f in ds_list]
+    LOG.debug("Searching for data source in: %s", ds_names)
+
+    for cls in ds_list:
+        try:
+            LOG.debug("Seeing if we can get any data from %s", cls)
+            s = cls(sys_cfg, distro, paths)
+            if s.get_data():
+                return (s, util.obj_name(cls))
+        except Exception:
+            util.logexc(LOG, "Getting data from %s failed", cls)
+
+    msg = ("Did not find any data source,"
+           " searched classes: (%s)") % (", ".join(ds_names))
+    raise DataSourceNotFoundException(msg)
+
+
+# Return a list of classes that have the same depends as 'depends'
+# iterate through cfg_list, loading "DataSource*" modules
+# and calling their "get_datasource_list".
+# Return an ordered list of classes that match (if any)
+def list_sources(cfg_list, depends, pkg_list):
+    src_list = []
+    LOG.debug(("Looking for for data source in: %s,"
+               " via packages %s that matches dependencies %s"),
+              cfg_list, pkg_list, depends)
+    for ds_name in cfg_list:
+        if not ds_name.startswith(DS_PREFIX):
+            ds_name = '%s%s' % (DS_PREFIX, ds_name)
+        m_locs = importer.find_module(ds_name,
+                                      pkg_list,
+                                      ['get_datasource_list'])
+        for m_loc in m_locs:
+            mod = importer.import_module(m_loc)
+            lister = getattr(mod, "get_datasource_list")
+            matches = lister(depends)
+            if matches:
+                src_list.extend(matches)
+                break
+    return src_list
+
+
+# 'depends' is a list of dependencies (DEP_FILESYSTEM)
+# ds_list is a list of 2 item lists
+# ds_list = [
+#   ( class, ( depends-that-this-class-needs ) )
+# }
+# It returns a list of 'class' that matched these deps exactly
+# It mainly is a helper function for DataSourceCollections
+def list_from_depends(depends, ds_list):
+    ret_list = []
+    depset = set(depends)
+    for (cls, deps) in ds_list:
+        if depset == set(deps):
+            ret_list.append(cls)
+    return ret_list
diff --git a/cloudinit/ssh_util.py b/cloudinit/ssh_util.py
new file mode 100644
index 00000000..e0a2f0ca
--- /dev/null
+++ b/cloudinit/ssh_util.py
@@ -0,0 +1,314 @@
+#!/usr/bin/python
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from StringIO import StringIO
+
+import csv
+import os
+import pwd
+
+from cloudinit import log as logging
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+# See: man sshd_config
+DEF_SSHD_CFG = "/etc/ssh/sshd_config"
+
+
+class AuthKeyLine(object):
+    def __init__(self, source, keytype=None, base64=None,
+                 comment=None, options=None):
+        self.base64 = base64
+        self.comment = comment
+        self.options = options
+        self.keytype = keytype
+        self.source = source
+
+    def empty(self):
+        if (not self.base64 and
+            not self.comment and not self.keytype and not self.options):
+            return True
+        return False
+
+    def __str__(self):
+        toks = []
+        if self.options:
+            toks.append(self.options)
+        if self.keytype:
+            toks.append(self.keytype)
+        if self.base64:
+            toks.append(self.base64)
+        if self.comment:
+            toks.append(self.comment)
+        if not toks:
+            return self.source
+        else:
+            return ' '.join(toks)
+
+
+class AuthKeyLineParser(object):
+    """
+    AUTHORIZED_KEYS FILE FORMAT
+     AuthorizedKeysFile specifies the file containing public keys for public
+     key authentication; if none is specified, the default is
+     ~/.ssh/authorized_keys.  Each line of the file contains one key (empty
+     (because of the size of the public key encoding) up to a limit of 8 kilo-
+     bytes, which permits DSA keys up to 8 kilobits and RSA keys up to 16
+     kilobits.  You don't want to type them in; instead, copy the
+     identity.pub, id_dsa.pub, or the id_rsa.pub file and edit it.
+
+     sshd enforces a minimum RSA key modulus size for protocol 1 and protocol
+     2 keys of 768 bits.
+
+     The options (if present) consist of comma-separated option specifica-
+     tions.  No spaces are permitted, except within double quotes.  The fol-
+     lowing option specifications are supported (note that option keywords are
+     case-insensitive):
+    """
+
+    def _extract_options(self, ent):
+        """
+        The options (if present) consist of comma-separated option specifica-
+         tions.  No spaces are permitted, except within double quotes.
+         Note that option keywords are case-insensitive.
+        """
+        quoted = False
+        i = 0
+        while (i < len(ent) and
+               ((quoted) or (ent[i] not in (" ", "\t")))):
+            curc = ent[i]
+            if i + 1 >= len(ent):
+                i = i + 1
+                break
+            nextc = ent[i + 1]
+            if curc == "\\" and nextc == '"':
+                i = i + 1
+            elif curc == '"':
+                quoted = not quoted
+            i = i + 1
+
+        options = ent[0:i]
+        options_lst = []
+
+        # Now use a csv parser to pull the options
+        # out of the above string that we just found an endpoint for.
+        #
+        # No quoting so we don't mess up any of the quoting that
+        # is already there.
+        reader = csv.reader(StringIO(options), quoting=csv.QUOTE_NONE)
+        for row in reader:
+            for e in row:
+                # Only keep non-empty csv options
+                e = e.strip()
+                if e:
+                    options_lst.append(e)
+
+        # Now take the rest of the items before the string
+        # as long as there is room to do this...
+        toks = []
+        if i + 1 < len(ent):
+            rest = ent[i + 1:]
+            toks = rest.split(None, 2)
+        return (options_lst, toks)
+
+    def _form_components(self, src_line, toks, options=None):
+        components = {}
+        if len(toks) == 1:
+            components['base64'] = toks[0]
+        elif len(toks) == 2:
+            components['base64'] = toks[0]
+            components['comment'] = toks[1]
+        elif len(toks) == 3:
+            components['keytype'] = toks[0]
+            components['base64'] = toks[1]
+            components['comment'] = toks[2]
+        components['options'] = options
+        if not components:
+            return AuthKeyLine(src_line)
+        else:
+            return AuthKeyLine(src_line, **components)
+
+    def parse(self, src_line, def_opt=None):
+        line = src_line.rstrip("\r\n")
+        if line.startswith("#") or line.strip() == '':
+            return AuthKeyLine(src_line)
+        else:
+            ent = line.strip()
+            toks = ent.split(None, 3)
+            if len(toks) < 4:
+                return self._form_components(src_line, toks, def_opt)
+            else:
+                (options, toks) = self._extract_options(ent)
+                if options:
+                    options = ",".join(options)
+                else:
+                    options = def_opt
+                return self._form_components(src_line, toks, options)
+
+
+def parse_authorized_keys(fname):
+    lines = []
+    try:
+        if os.path.isfile(fname):
+            lines = util.load_file(fname).splitlines()
+    except (IOError, OSError):
+        util.logexc(LOG, "Error reading lines from %s", fname)
+        lines = []
+
+    parser = AuthKeyLineParser()
+    contents = []
+    for line in lines:
+        contents.append(parser.parse(line))
+    return contents
+
+
+def update_authorized_keys(fname, keys):
+    entries = parse_authorized_keys(fname)
+    to_add = list(keys)
+
+    for i in range(0, len(entries)):
+        ent = entries[i]
+        if ent.empty() or not ent.base64:
+            continue
+        # Replace those with the same base64
+        for k in keys:
+            if k.empty() or not k.base64:
+                continue
+            if k.base64 == ent.base64:
+                # Replace it with our better one
+                ent = k
+                # Don't add it later
+                if k in to_add:
+                    to_add.remove(k)
+        entries[i] = ent
+
+    # Now append any entries we did not match above
+    for key in to_add:
+        entries.append(key)
+
+    # Now format them back to strings...
+    lines = [str(b) for b in entries]
+
+    # Ensure it ends with a newline
+    lines.append('')
+    return '\n'.join(lines)
+
+
+def setup_user_keys(keys, user, key_prefix, paths):
+    # Make sure the users .ssh dir is setup accordingly
+    pwent = pwd.getpwnam(user)
+    ssh_dir = os.path.join(pwent.pw_dir, '.ssh')
+    ssh_dir = paths.join(False, ssh_dir)
+    if not os.path.exists(ssh_dir):
+        util.ensure_dir(ssh_dir, mode=0700)
+        util.chownbyid(ssh_dir, pwent.pw_uid, pwent.pw_gid)
+
+    # Turn the keys given into actual entries
+    parser = AuthKeyLineParser()
+    key_entries = []
+    for k in keys:
+        key_entries.append(parser.parse(str(k), def_opt=key_prefix))
+
+    sshd_conf_fn = paths.join(True, DEF_SSHD_CFG)
+    with util.SeLinuxGuard(ssh_dir, recursive=True):
+        try:
+            # AuthorizedKeysFile may contain tokens
+            # of the form %T which are substituted during connection set-up.
+            # The following tokens are defined: %% is replaced by a literal
+            # '%', %h is replaced by the home directory of the user being
+            # authenticated and %u is replaced by the username of that user.
+            ssh_cfg = parse_ssh_config_map(sshd_conf_fn)
+            akeys = ssh_cfg.get("authorizedkeysfile", '')
+            akeys = akeys.strip()
+            if not akeys:
+                akeys = "%h/.ssh/authorized_keys"
+            akeys = akeys.replace("%h", pwent.pw_dir)
+            akeys = akeys.replace("%u", user)
+            akeys = akeys.replace("%%", '%')
+            if not akeys.startswith('/'):
+                akeys = os.path.join(pwent.pw_dir, akeys)
+            authorized_keys = paths.join(False, akeys)
+        except (IOError, OSError):
+            authorized_keys = os.path.join(ssh_dir, 'authorized_keys')
+            util.logexc(LOG, ("Failed extracting 'AuthorizedKeysFile'"
+                              " in ssh config"
+                              " from %s, using 'AuthorizedKeysFile' file"
+                              " %s instead"),
+                        sshd_conf_fn, authorized_keys)
+
+        content = update_authorized_keys(authorized_keys, key_entries)
+        util.ensure_dir(os.path.dirname(authorized_keys), mode=0700)
+        util.write_file(authorized_keys, content, mode=0600)
+        util.chownbyid(authorized_keys, pwent.pw_uid, pwent.pw_gid)
+
+
+class SshdConfigLine(object):
+    def __init__(self, line, k=None, v=None):
+        self.line = line
+        self._key = k
+        self.value = v
+
+    @property
+    def key(self):
+        if self._key is None:
+            return None
+        # Keywords are case-insensitive
+        return self._key.lower()
+
+    def __str__(self):
+        if self._key is None:
+            return str(self.line)
+        else:
+            v = str(self._key)
+            if self.value:
+                v += " " + str(self.value)
+            return v
+
+
+def parse_ssh_config(fname):
+    # See: man sshd_config
+    # The file contains keyword-argument pairs, one per line.
+    # Lines starting with '#' and empty lines are interpreted as comments.
+    # Note: key-words are case-insensitive and arguments are case-sensitive
+    lines = []
+    if not os.path.isfile(fname):
+        return lines
+    for line in util.load_file(fname).splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            lines.append(SshdConfigLine(line))
+            continue
+        (key, val) = line.split(None, 1)
+        lines.append(SshdConfigLine(line, key, val))
+    return lines
+
+
+def parse_ssh_config_map(fname):
+    lines = parse_ssh_config(fname)
+    if not lines:
+        return {}
+    ret = {}
+    for line in lines:
+        if not line.key:
+            continue
+        ret[line.key] = line.value
+    return ret
diff --git a/cloudinit/stages.py b/cloudinit/stages.py
new file mode 100644
index 00000000..8fd6aa5d
--- /dev/null
+++ b/cloudinit/stages.py
@@ -0,0 +1,551 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import cPickle as pickle
+
+import copy
+import os
+import sys
+
+from cloudinit.settings import (PER_INSTANCE, FREQUENCIES, CLOUD_CONFIG)
+
+from cloudinit import handlers
+
+# Default handlers (used if not overridden)
+from cloudinit.handlers import boot_hook as bh_part
+from cloudinit.handlers import cloud_config as cc_part
+from cloudinit.handlers import shell_script as ss_part
+from cloudinit.handlers import upstart_job as up_part
+
+from cloudinit import cloud
+from cloudinit import config
+from cloudinit import distros
+from cloudinit import helpers
+from cloudinit import importer
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+
+class Init(object):
+    def __init__(self, ds_deps=None):
+        if ds_deps is not None:
+            self.ds_deps = ds_deps
+        else:
+            self.ds_deps = [sources.DEP_FILESYSTEM, sources.DEP_NETWORK]
+        # Created on first use
+        self._cfg = None
+        self._paths = None
+        self._distro = None
+        # Created only when a fetch occurs
+        self.datasource = None
+
+    @property
+    def distro(self):
+        if not self._distro:
+            # Try to find the right class to use
+            scfg = self._extract_cfg('system')
+            name = scfg.pop('distro', 'ubuntu')
+            cls = distros.fetch(name)
+            LOG.debug("Using distro class %s", cls)
+            self._distro = cls(name, scfg, self.paths)
+        return self._distro
+
+    @property
+    def cfg(self):
+        return self._extract_cfg('restricted')
+
+    def _extract_cfg(self, restriction):
+        # Ensure actually read
+        self.read_cfg()
+        # Nobody gets the real config
+        ocfg = copy.deepcopy(self._cfg)
+        if restriction == 'restricted':
+            ocfg.pop('system_info', None)
+        elif restriction == 'system':
+            ocfg = util.get_cfg_by_path(ocfg, ('system_info',), {})
+        elif restriction == 'paths':
+            ocfg = util.get_cfg_by_path(ocfg, ('system_info', 'paths'), {})
+        if not isinstance(ocfg, (dict)):
+            ocfg = {}
+        return ocfg
+
+    @property
+    def paths(self):
+        if not self._paths:
+            path_info = self._extract_cfg('paths')
+            self._paths = helpers.Paths(path_info, self.datasource)
+        return self._paths
+
+    def _initial_subdirs(self):
+        c_dir = self.paths.cloud_dir
+        initial_dirs = [
+            c_dir,
+            os.path.join(c_dir, 'scripts'),
+            os.path.join(c_dir, 'scripts', 'per-instance'),
+            os.path.join(c_dir, 'scripts', 'per-once'),
+            os.path.join(c_dir, 'scripts', 'per-boot'),
+            os.path.join(c_dir, 'seed'),
+            os.path.join(c_dir, 'instances'),
+            os.path.join(c_dir, 'handlers'),
+            os.path.join(c_dir, 'sem'),
+            os.path.join(c_dir, 'data'),
+        ]
+        return initial_dirs
+
+    def purge_cache(self, rm_instance_lnk=True):
+        rm_list = []
+        rm_list.append(self.paths.boot_finished)
+        if rm_instance_lnk:
+            rm_list.append(self.paths.instance_link)
+        for f in rm_list:
+            util.del_file(f)
+        return len(rm_list)
+
+    def initialize(self):
+        self._initialize_filesystem()
+
+    def _initialize_filesystem(self):
+        util.ensure_dirs(self._initial_subdirs())
+        log_file = util.get_cfg_option_str(self.cfg, 'def_log_file')
+        perms = util.get_cfg_option_str(self.cfg, 'syslog_fix_perms')
+        if log_file:
+            util.ensure_file(log_file)
+            if perms:
+                (u, g) = perms.split(':', 1)
+                if u == "-1" or u == "None":
+                    u = None
+                if g == "-1" or g == "None":
+                    g = None
+                util.chownbyname(log_file, u, g)
+
+    def read_cfg(self, extra_fns=None):
+        # None check so that we don't keep on re-loading if empty
+        if self._cfg is None:
+            self._cfg = self._read_cfg(extra_fns)
+            # LOG.debug("Loaded 'init' config %s", self._cfg)
+
+    def _read_base_cfg(self):
+        base_cfgs = []
+        default_cfg = util.get_builtin_cfg()
+        kern_contents = util.read_cc_from_cmdline()
+        # Kernel/cmdline parameters override system config
+        if kern_contents:
+            base_cfgs.append(util.load_yaml(kern_contents, default={}))
+        # Anything in your conf.d location??
+        # or the 'default' cloud.cfg location???
+        base_cfgs.append(util.read_conf_with_confd(CLOUD_CONFIG))
+        # And finally the default gets to play
+        if default_cfg:
+            base_cfgs.append(default_cfg)
+        return util.mergemanydict(base_cfgs)
+
+    def _read_cfg(self, extra_fns):
+        no_cfg_paths = helpers.Paths({}, self.datasource)
+        merger = helpers.ConfigMerger(paths=no_cfg_paths,
+                                      datasource=self.datasource,
+                                      additional_fns=extra_fns,
+                                      base_cfg=self._read_base_cfg())
+        return merger.cfg
+
+    def _restore_from_cache(self):
+        # We try to restore from a current link and static path
+        # by using the instance link, if purge_cache was called
+        # the file wont exist.
+        pickled_fn = self.paths.get_ipath_cur('obj_pkl')
+        pickle_contents = None
+        try:
+            pickle_contents = util.load_file(pickled_fn)
+        except Exception:
+            pass
+        # This is expected so just return nothing
+        # successfully loaded...
+        if not pickle_contents:
+            return None
+        try:
+            return pickle.loads(pickle_contents)
+        except Exception:
+            util.logexc(LOG, "Failed loading pickled blob from %s", pickled_fn)
+            return None
+
+    def _write_to_cache(self):
+        if not self.datasource:
+            return False
+        pickled_fn = self.paths.get_ipath_cur("obj_pkl")
+        try:
+            pk_contents = pickle.dumps(self.datasource)
+        except Exception:
+            util.logexc(LOG, "Failed pickling datasource %s", self.datasource)
+            return False
+        try:
+            util.write_file(pickled_fn, pk_contents, mode=0400)
+        except Exception:
+            util.logexc(LOG, "Failed pickling datasource to %s", pickled_fn)
+            return False
+        return True
+
+    def _get_datasources(self):
+        # Any config provided???
+        pkg_list = self.cfg.get('datasource_pkg_list') or []
+        # Add the defaults at the end
+        for n in ['', util.obj_name(sources)]:
+            if n not in pkg_list:
+                pkg_list.append(n)
+        cfg_list = self.cfg.get('datasource_list') or []
+        return (cfg_list, pkg_list)
+
+    def _get_data_source(self):
+        if self.datasource:
+            return self.datasource
+        ds = self._restore_from_cache()
+        if ds:
+            LOG.debug("Restored from cache, datasource: %s", ds)
+        if not ds:
+            (cfg_list, pkg_list) = self._get_datasources()
+            # Deep copy so that user-data handlers can not modify
+            # (which will affect user-data handlers down the line...)
+            (ds, dsname) = sources.find_source(self.cfg,
+                                               self.distro,
+                                               self.paths,
+                                               copy.deepcopy(self.ds_deps),
+                                               cfg_list,
+                                               pkg_list)
+            LOG.debug("Loaded datasource %s - %s", dsname, ds)
+        self.datasource = ds
+        # Ensure we adjust our path members datasource
+        # now that we have one (thus allowing ipath to be used)
+        self.paths.datasource = ds
+        return ds
+
+    def _get_instance_subdirs(self):
+        return ['handlers', 'scripts', 'sems']
+
+    def _get_ipath(self, subname=None):
+        # Force a check to see if anything
+        # actually comes back, if not
+        # then a datasource has not been assigned...
+        instance_dir = self.paths.get_ipath(subname)
+        if not instance_dir:
+            raise RuntimeError(("No instance directory is available."
+                                " Has a datasource been fetched??"))
+        return instance_dir
+
+    def _reflect_cur_instance(self):
+        # Remove the old symlink and attach a new one so
+        # that further reads/writes connect into the right location
+        idir = self._get_ipath()
+        util.del_file(self.paths.instance_link)
+        util.sym_link(idir, self.paths.instance_link)
+
+        # Ensures these dirs exist
+        dir_list = []
+        for d in self._get_instance_subdirs():
+            dir_list.append(os.path.join(idir, d))
+        util.ensure_dirs(dir_list)
+
+        # Write out information on what is being used for the current instance
+        # and what may have been used for a previous instance...
+        dp = self.paths.get_cpath('data')
+
+        # Write what the datasource was and is..
+        ds = "%s: %s" % (util.obj_name(self.datasource), self.datasource)
+        previous_ds = None
+        ds_fn = os.path.join(idir, 'datasource')
+        try:
+            previous_ds = util.load_file(ds_fn).strip()
+        except Exception:
+            pass
+        if not previous_ds:
+            previous_ds = ds
+        util.write_file(ds_fn, "%s\n" % ds)
+        util.write_file(os.path.join(dp, 'previous-datasource'),
+                        "%s\n" % (previous_ds))
+
+        # What the instance id was and is...
+        iid = self.datasource.get_instance_id()
+        previous_iid = None
+        iid_fn = os.path.join(dp, 'instance-id')
+        try:
+            previous_iid = util.load_file(iid_fn).strip()
+        except Exception:
+            pass
+        if not previous_iid:
+            previous_iid = iid
+        util.write_file(iid_fn, "%s\n" % iid)
+        util.write_file(os.path.join(dp, 'previous-instance-id'),
+                        "%s\n" % (previous_iid))
+        return iid
+
+    def fetch(self):
+        return self._get_data_source()
+
+    def instancify(self):
+        return self._reflect_cur_instance()
+
+    def cloudify(self):
+        # Form the needed options to cloudify our members
+        return cloud.Cloud(self.datasource,
+                           self.paths, self.cfg,
+                           self.distro, helpers.Runners(self.paths))
+
+    def update(self):
+        if not self._write_to_cache():
+            return
+        self._store_userdata()
+
+    def _store_userdata(self):
+        raw_ud = "%s" % (self.datasource.get_userdata_raw())
+        util.write_file(self._get_ipath('userdata_raw'), raw_ud, 0600)
+        processed_ud = "%s" % (self.datasource.get_userdata())
+        util.write_file(self._get_ipath('userdata'), processed_ud, 0600)
+
+    def _default_userdata_handlers(self):
+        opts = {
+            'paths': self.paths,
+            'datasource': self.datasource,
+        }
+        # TODO Hmmm, should we dynamically import these??
+        def_handlers = [
+            cc_part.CloudConfigPartHandler(**opts),
+            ss_part.ShellScriptPartHandler(**opts),
+            bh_part.BootHookPartHandler(**opts),
+            up_part.UpstartJobPartHandler(**opts),
+        ]
+        return def_handlers
+
+    def consume_userdata(self, frequency=PER_INSTANCE):
+        cdir = self.paths.get_cpath("handlers")
+        idir = self._get_ipath("handlers")
+
+        # Add the path to the plugins dir to the top of our list for import
+        # instance dir should be read before cloud-dir
+        if cdir and cdir not in sys.path:
+            sys.path.insert(0, cdir)
+        if idir and idir not in sys.path:
+            sys.path.insert(0, idir)
+
+        # Ensure datasource fetched before activation (just incase)
+        user_data_msg = self.datasource.get_userdata()
+
+        # This keeps track of all the active handlers
+        c_handlers = helpers.ContentHandlers()
+
+        # Add handlers in cdir
+        potential_handlers = util.find_modules(cdir)
+        for (fname, mod_name) in potential_handlers.iteritems():
+            try:
+                mod_locs = importer.find_module(mod_name, [''],
+                                                ['list_types',
+                                                 'handle_part'])
+                if not mod_locs:
+                    LOG.warn(("Could not find a valid user-data handler"
+                              " named %s in file %s"), mod_name, fname)
+                    continue
+                mod = importer.import_module(mod_locs[0])
+                mod = handlers.fixup_handler(mod)
+                types = c_handlers.register(mod)
+                LOG.debug("Added handler for %s from %s", types, fname)
+            except:
+                util.logexc(LOG, "Failed to register handler from %s", fname)
+
+        def_handlers = self._default_userdata_handlers()
+        applied_def_handlers = c_handlers.register_defaults(def_handlers)
+        if applied_def_handlers:
+            LOG.debug("Registered default handlers: %s", applied_def_handlers)
+
+        # Form our cloud interface
+        data = self.cloudify()
+
+        # Init the handlers first
+        called = []
+        for (_ctype, mod) in c_handlers.iteritems():
+            if mod in called:
+                continue
+            handlers.call_begin(mod, data, frequency)
+            called.append(mod)
+
+        # Walk the user data
+        part_data = {
+            'handlers': c_handlers,
+            # Any new handlers that are encountered get writen here
+            'handlerdir': idir,
+            'data': data,
+            # The default frequency if handlers don't have one
+            'frequency': frequency,
+            # This will be used when new handlers are found
+            # to help write there contents to files with numbered
+            # names...
+            'handlercount': 0,
+        }
+        handlers.walk(user_data_msg, handlers.walker_callback, data=part_data)
+
+        # Give callbacks opportunity to finalize
+        called = []
+        for (_ctype, mod) in c_handlers.iteritems():
+            if mod in called:
+                continue
+            handlers.call_end(mod, data, frequency)
+            called.append(mod)
+
+
+class Modules(object):
+    def __init__(self, init, cfg_files=None):
+        self.init = init
+        self.cfg_files = cfg_files
+        # Created on first use
+        self._cached_cfg = None
+
+    @property
+    def cfg(self):
+        # None check to avoid empty case causing re-reading
+        if self._cached_cfg is None:
+            merger = helpers.ConfigMerger(paths=self.init.paths,
+                                          datasource=self.init.datasource,
+                                          additional_fns=self.cfg_files,
+                                          base_cfg=self.init.cfg)
+            self._cached_cfg = merger.cfg
+            # LOG.debug("Loading 'module' config %s", self._cached_cfg)
+        # Only give out a copy so that others can't modify this...
+        return copy.deepcopy(self._cached_cfg)
+
+    def _read_modules(self, name):
+        module_list = []
+        if name not in self.cfg:
+            return module_list
+        cfg_mods = self.cfg[name]
+        # Create 'module_list', an array of hashes
+        # Where hash['mod'] = module name
+        #       hash['freq'] = frequency
+        #       hash['args'] = arguments
+        for item in cfg_mods:
+            if not item:
+                continue
+            if isinstance(item, (str, basestring)):
+                module_list.append({
+                    'mod': item.strip(),
+                })
+            elif isinstance(item, (list)):
+                contents = {}
+                # Meant to fall through...
+                if len(item) >= 1:
+                    contents['mod'] = item[0].strip()
+                if len(item) >= 2:
+                    contents['freq'] = item[1].strip()
+                if len(item) >= 3:
+                    contents['args'] = item[2:]
+                if contents:
+                    module_list.append(contents)
+            elif isinstance(item, (dict)):
+                contents = {}
+                valid = False
+                if 'name' in item:
+                    contents['mod'] = item['name'].strip()
+                    valid = True
+                if 'frequency' in item:
+                    contents['freq'] = item['frequency'].strip()
+                if 'args' in item:
+                    contents['args'] = item['args'] or []
+                if contents and valid:
+                    module_list.append(contents)
+            else:
+                raise TypeError(("Failed to read '%s' item in config,"
+                                 " unknown type %s") %
+                                 (item, util.obj_name(item)))
+        return module_list
+
+    def _fixup_modules(self, raw_mods):
+        mostly_mods = []
+        for raw_mod in raw_mods:
+            raw_name = raw_mod['mod']
+            freq = raw_mod.get('freq')
+            run_args = raw_mod.get('args') or []
+            mod_name = config.form_module_name(raw_name)
+            if not mod_name:
+                continue
+            if freq and freq not in FREQUENCIES:
+                LOG.warn(("Config specified module %s"
+                          " has an unknown frequency %s"), raw_name, freq)
+                # Reset it so when ran it will get set to a known value
+                freq = None
+            mod_locs = importer.find_module(mod_name,
+                                            ['', util.obj_name(config)],
+                                            ['handle'])
+            if not mod_locs:
+                LOG.warn("Could not find module named %s", mod_name)
+                continue
+            mod = config.fixup_module(importer.import_module(mod_locs[0]))
+            mostly_mods.append([mod, raw_name, freq, run_args])
+        return mostly_mods
+
+    def _run_modules(self, mostly_mods):
+        d_name = self.init.distro.name
+        cc = self.init.cloudify()
+        # Return which ones ran
+        # and which ones failed + the exception of why it failed
+        failures = []
+        which_ran = []
+        for (mod, name, freq, args) in mostly_mods:
+            try:
+                # Try the modules frequency, otherwise fallback to a known one
+                if not freq:
+                    freq = mod.frequency
+                if not freq in FREQUENCIES:
+                    freq = PER_INSTANCE
+                worked_distros = mod.distros
+                if (worked_distros and d_name not in worked_distros):
+                    LOG.warn(("Module %s is verified on %s distros"
+                              " but not on %s distro. It may or may not work"
+                              " correctly."), name, worked_distros, d_name)
+                # Use the configs logger and not our own
+                # TODO: possibly check the module
+                # for having a LOG attr and just give it back
+                # its own logger?
+                func_args = [name, self.cfg,
+                             cc, config.LOG, args]
+                # Mark it as having started running
+                which_ran.append(name)
+                # This name will affect the semaphore name created
+                run_name = "config-%s" % (name)
+                cc.run(run_name, mod.handle, func_args, freq=freq)
+            except Exception as e:
+                util.logexc(LOG, "Running %s (%s) failed", name, mod)
+                failures.append((name, e))
+        return (which_ran, failures)
+
+    def run_single(self, mod_name, args=None, freq=None):
+        # Form the users module 'specs'
+        mod_to_be = {
+            'mod': mod_name,
+            'args': args,
+            'freq': freq,
+        }
+        # Now resume doing the normal fixups and running
+        raw_mods = [mod_to_be]
+        mostly_mods = self._fixup_modules(raw_mods)
+        return self._run_modules(mostly_mods)
+
+    def run_section(self, section_name):
+        raw_mods = self._read_modules(section_name)
+        mostly_mods = self._fixup_modules(raw_mods)
+        return self._run_modules(mostly_mods)
diff --git a/cloudinit/templater.py b/cloudinit/templater.py
new file mode 100644
index 00000000..c4259fa0
--- /dev/null
+++ b/cloudinit/templater.py
@@ -0,0 +1,41 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from tempita import Template
+
+from cloudinit import util
+
+
+def render_from_file(fn, params):
+    return render_string(util.load_file(fn), params, name=fn)
+
+
+def render_to_file(fn, outfn, params, mode=0644):
+    contents = render_from_file(fn, params)
+    util.write_file(outfn, contents, mode=mode)
+
+
+def render_string(content, params, name=None):
+    tpl = Template(content, name=name)
+    if not params:
+        params = dict()
+    return tpl.substitute(params)
diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py
new file mode 100644
index 00000000..dbf72392
--- /dev/null
+++ b/cloudinit/url_helper.py
@@ -0,0 +1,226 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from contextlib import closing
+
+import errno
+import socket
+import time
+import urllib
+import urllib2
+
+from cloudinit import log as logging
+from cloudinit import version
+
+LOG = logging.getLogger(__name__)
+
+
+class UrlResponse(object):
+    def __init__(self, status_code, contents=None, headers=None):
+        self._status_code = status_code
+        self._contents = contents
+        self._headers = headers
+
+    @property
+    def code(self):
+        return self._status_code
+
+    @property
+    def contents(self):
+        return self._contents
+
+    @property
+    def headers(self):
+        return self._headers
+
+    def __str__(self):
+        if not self.contents:
+            return ''
+        else:
+            return str(self.contents)
+
+    def ok(self, redirects_ok=False):
+        upper = 300
+        if redirects_ok:
+            upper = 400
+        if self.code >= 200 and self.code < upper:
+            return True
+        else:
+            return False
+
+
+def readurl(url, data=None, timeout=None,
+            retries=0, sec_between=1, headers=None):
+
+    req_args = {}
+    req_args['url'] = url
+    if data is not None:
+        req_args['data'] = urllib.urlencode(data)
+
+    if not headers:
+        headers = {
+            'User-Agent': 'Cloud-Init/%s' % (version.version_string()),
+        }
+
+    req_args['headers'] = headers
+    req = urllib2.Request(**req_args)
+
+    retries = max(retries, 0)
+    attempts = retries + 1
+
+    excepts = []
+    LOG.debug(("Attempting to open '%s' with %s attempts"
+               " (%s retries, timeout=%s) to be performed"),
+              url, attempts, retries, timeout)
+    open_args = {}
+    if timeout is not None:
+        open_args['timeout'] = int(timeout)
+    for i in range(0, attempts):
+        try:
+            with closing(urllib2.urlopen(req, **open_args)) as rh:
+                content = rh.read()
+                status = rh.getcode()
+                if status is None:
+                    # This seems to happen when files are read...
+                    status = 200
+                headers = {}
+                if rh.headers:
+                    headers = dict(rh.headers)
+                LOG.debug("Read from %s (%s, %sb) after %s attempts",
+                          url, status, len(content), (i + 1))
+                return UrlResponse(status, content, headers)
+        except urllib2.HTTPError as e:
+            excepts.append(e)
+        except urllib2.URLError as e:
+            # This can be a message string or
+            # another exception instance
+            # (socket.error for remote URLs, OSError for local URLs).
+            if (isinstance(e.reason, (OSError)) and
+                e.reason.errno == errno.ENOENT):
+                excepts.append(e.reason)
+            else:
+                excepts.append(e)
+        except Exception as e:
+            excepts.append(e)
+        if i + 1 < attempts:
+            LOG.debug("Please wait %s seconds while we wait to try again",
+                     sec_between)
+            time.sleep(sec_between)
+
+    # Didn't work out
+    LOG.warn("Failed reading from %s after %s attempts", url, attempts)
+
+    # It must of errored at least once for code
+    # to get here so re-raise the last error
+    LOG.debug("%s errors occured, re-raising the last one", len(excepts))
+    raise excepts[-1]
+
+
+def wait_for_url(urls, max_wait=None, timeout=None,
+                 status_cb=None, headers_cb=None, sleep_time=1):
+    """
+    urls:      a list of urls to try
+    max_wait:  roughly the maximum time to wait before giving up
+               The max time is *actually* len(urls)*timeout as each url will
+               be tried once and given the timeout provided.
+    timeout:   the timeout provided to urllib2.urlopen
+    status_cb: call method with string message when a url is not available
+    headers_cb: call method with single argument of url to get headers
+                for request.
+
+    the idea of this routine is to wait for the EC2 metdata service to
+    come up.  On both Eucalyptus and EC2 we have seen the case where
+    the instance hit the MD before the MD service was up.  EC2 seems
+    to have permenantely fixed this, though.
+
+    In openstack, the metadata service might be painfully slow, and
+    unable to avoid hitting a timeout of even up to 10 seconds or more
+    (LP: #894279) for a simple GET.
+
+    Offset those needs with the need to not hang forever (and block boot)
+    on a system where cloud-init is configured to look for EC2 Metadata
+    service but is not going to find one.  It is possible that the instance
+    data host (169.254.169.254) may be firewalled off Entirely for a sytem,
+    meaning that the connection will block forever unless a timeout is set.
+    """
+    start_time = time.time()
+
+    def log_status_cb(msg):
+        LOG.debug(msg)
+
+    if status_cb is None:
+        status_cb = log_status_cb
+
+    def timeup(max_wait, start_time):
+        return ((max_wait <= 0 or max_wait is None) or
+                (time.time() - start_time > max_wait))
+
+    loop_n = 0
+    while True:
+        sleep_time = int(loop_n / 5) + 1
+        for url in urls:
+            now = time.time()
+            if loop_n != 0:
+                if timeup(max_wait, start_time):
+                    break
+                if timeout and (now + timeout > (start_time + max_wait)):
+                    # shorten timeout to not run way over max_time
+                    timeout = int((start_time + max_wait) - now)
+
+            reason = ""
+            try:
+                if headers_cb is not None:
+                    headers = headers_cb(url)
+                else:
+                    headers = {}
+
+                resp = readurl(url, headers=headers, timeout=timeout)
+                if not resp.contents:
+                    reason = "empty response [%s]" % (resp.code)
+                elif not resp.ok():
+                    reason = "bad status code [%s]" % (resp.code)
+                else:
+                    return url
+            except urllib2.HTTPError as e:
+                reason = "http error [%s]" % e.code
+            except urllib2.URLError as e:
+                reason = "url error [%s]" % e.reason
+            except socket.timeout as e:
+                reason = "socket timeout [%s]" % e
+            except Exception as e:
+                reason = "unexpected error [%s]" % e
+
+            time_taken = int(time.time() - start_time)
+            status_msg = "Calling '%s' failed [%s/%ss]: %s" % (url,
+                                                             time_taken,
+                                                             max_wait, reason)
+            status_cb(status_msg)
+
+        if timeup(max_wait, start_time):
+            break
+
+        loop_n = loop_n + 1
+        LOG.debug("Please wait %s seconds while we wait to try again",
+                  sleep_time)
+        time.sleep(sleep_time)
+
+    return False
diff --git a/cloudinit/user_data.py b/cloudinit/user_data.py
new file mode 100644
index 00000000..0842594d
--- /dev/null
+++ b/cloudinit/user_data.py
@@ -0,0 +1,243 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <scott.moser@canonical.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+import email
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.mime.base import MIMEBase
+
+from cloudinit import handlers
+from cloudinit import log as logging
+from cloudinit import url_helper
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+# Constants copied in from the handler module
+NOT_MULTIPART_TYPE = handlers.NOT_MULTIPART_TYPE
+PART_FN_TPL = handlers.PART_FN_TPL
+OCTET_TYPE = handlers.OCTET_TYPE
+
+# Saves typing errors
+CONTENT_TYPE = 'Content-Type'
+
+# Various special content types that cause special actions
+TYPE_NEEDED = ["text/plain", "text/x-not-multipart"]
+INCLUDE_TYPES = ['text/x-include-url', 'text/x-include-once-url']
+ARCHIVE_TYPES = ["text/cloud-config-archive"]
+UNDEF_TYPE = "text/plain"
+ARCHIVE_UNDEF_TYPE = "text/cloud-config"
+
+# Msg header used to track attachments
+ATTACHMENT_FIELD = 'Number-Attachments'
+
+
+class UserDataProcessor(object):
+    def __init__(self, paths):
+        self.paths = paths
+
+    def process(self, blob):
+        base_msg = convert_string(blob)
+        process_msg = MIMEMultipart()
+        self._process_msg(base_msg, process_msg)
+        return process_msg
+
+    def _process_msg(self, base_msg, append_msg):
+        for part in base_msg.walk():
+            # multipart/* are just containers
+            if part.get_content_maintype() == 'multipart':
+                continue
+
+            ctype = None
+            ctype_orig = part.get_content_type()
+            payload = part.get_payload(decode=True)
+
+            if not ctype_orig:
+                ctype_orig = UNDEF_TYPE
+
+            if ctype_orig in TYPE_NEEDED:
+                ctype = handlers.type_from_starts_with(payload)
+
+            if ctype is None:
+                ctype = ctype_orig
+
+            if ctype in INCLUDE_TYPES:
+                self._do_include(payload, append_msg)
+                continue
+
+            if ctype in ARCHIVE_TYPES:
+                self._explode_archive(payload, append_msg)
+                continue
+
+            if CONTENT_TYPE in base_msg:
+                base_msg.replace_header(CONTENT_TYPE, ctype)
+            else:
+                base_msg[CONTENT_TYPE] = ctype
+
+            self._attach_part(append_msg, part)
+
+    def _get_include_once_filename(self, entry):
+        entry_fn = util.hash_blob(entry, 'md5', 64)
+        return os.path.join(self.paths.get_ipath_cur('data'),
+                            'urlcache', entry_fn)
+
+    def _do_include(self, content, append_msg):
+        # Include a list of urls, one per line
+        # also support '#include <url here>'
+        # or #include-once '<url here>'
+        include_once_on = False
+        for line in content.splitlines():
+            lc_line = line.lower()
+            if lc_line.startswith("#include-once"):
+                line = line[len("#include-once"):].lstrip()
+                # Every following include will now
+                # not be refetched.... but will be
+                # re-read from a local urlcache (if it worked)
+                include_once_on = True
+            elif lc_line.startswith("#include"):
+                line = line[len("#include"):].lstrip()
+                # Disable the include once if it was on
+                # if it wasn't, then this has no effect.
+                include_once_on = False
+            if line.startswith("#"):
+                continue
+            include_url = line.strip()
+            if not include_url:
+                continue
+
+            include_once_fn = None
+            content = None
+            if include_once_on:
+                include_once_fn = self._get_include_once_filename(include_url)
+            if include_once_on and os.path.isfile(include_once_fn):
+                content = util.load_file(include_once_fn)
+            else:
+                resp = url_helper.readurl(include_url)
+                if include_once_on and resp.ok():
+                    util.write_file(include_once_fn, str(resp), mode=0600)
+                if resp.ok():
+                    content = str(resp)
+                else:
+                    LOG.warn(("Fetching from %s resulted in"
+                              " a invalid http code of %s"),
+                             include_url, resp.code)
+
+            if content is not None:
+                new_msg = convert_string(content)
+                self._process_msg(new_msg, append_msg)
+
+    def _explode_archive(self, archive, append_msg):
+        entries = util.load_yaml(archive, default=[], allowed=[list, set])
+        for ent in entries:
+            # ent can be one of:
+            #  dict { 'filename' : 'value', 'content' :
+            #       'value', 'type' : 'value' }
+            #    filename and type not be present
+            # or
+            #  scalar(payload)
+            if isinstance(ent, (str, basestring)):
+                ent = {'content': ent}
+            if not isinstance(ent, (dict)):
+                # TODO raise?
+                continue
+
+            content = ent.get('content', '')
+            mtype = ent.get('type')
+            if not mtype:
+                mtype = handlers.type_from_starts_with(content,
+                                                       ARCHIVE_UNDEF_TYPE)
+
+            maintype, subtype = mtype.split('/', 1)
+            if maintype == "text":
+                msg = MIMEText(content, _subtype=subtype)
+            else:
+                msg = MIMEBase(maintype, subtype)
+                msg.set_payload(content)
+
+            if 'filename' in ent:
+                msg.add_header('Content-Disposition',
+                               'attachment', filename=ent['filename'])
+
+            for header in list(ent.keys()):
+                if header in ('content', 'filename', 'type'):
+                    continue
+                msg.add_header(header, ent['header'])
+
+            self._attach_part(append_msg, msg)
+
+    def _multi_part_count(self, outer_msg, new_count=None):
+        """
+        Return the number of attachments to this MIMEMultipart by looking
+        at its 'Number-Attachments' header.
+        """
+        if ATTACHMENT_FIELD not in outer_msg:
+            outer_msg[ATTACHMENT_FIELD] = '0'
+
+        if new_count is not None:
+            outer_msg.replace_header(ATTACHMENT_FIELD, str(new_count))
+
+        fetched_count = 0
+        try:
+            fetched_count = int(outer_msg.get(ATTACHMENT_FIELD))
+        except (ValueError, TypeError):
+            outer_msg.replace_header(ATTACHMENT_FIELD, str(fetched_count))
+        return fetched_count
+
+    def _part_filename(self, _unnamed_part, count):
+        return PART_FN_TPL % (count + 1)
+
+    def _attach_part(self, outer_msg, part):
+        """
+        Attach an part to an outer message. outermsg must be a MIMEMultipart.
+        Modifies a header in the message to keep track of number of attachments.
+        """
+        cur_c = self._multi_part_count(outer_msg)
+        if not part.get_filename():
+            fn = self._part_filename(part, cur_c)
+            part.add_header('Content-Disposition',
+                            'attachment', filename=fn)
+        outer_msg.attach(part)
+        self._multi_part_count(outer_msg, cur_c + 1)
+
+
+# Coverts a raw string into a mime message
+def convert_string(raw_data, headers=None):
+    if not raw_data:
+        raw_data = ''
+    if not headers:
+        headers = {}
+    data = util.decomp_str(raw_data)
+    if "mime-version:" in data[0:4096].lower():
+        msg = email.message_from_string(data)
+        for (key, val) in headers.iteritems():
+            if key in msg:
+                msg.replace_header(key, val)
+            else:
+                msg[key] = val
+    else:
+        mtype = headers.get(CONTENT_TYPE, NOT_MULTIPART_TYPE)
+        maintype, subtype = mtype.split("/", 1)
+        msg = MIMEBase(maintype, subtype, *headers)
+        msg.set_payload(data)
+    return msg
diff --git a/cloudinit/util.py b/cloudinit/util.py
index 84aae3ea..d7dd20b5 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -1,10 +1,12 @@
 # vi: ts=4 expandtab
 #
-#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Scott Moser <scott.moser@canonical.com>
-#    Author: Juerg Hafliger <juerg.haefliger@hp.com>
+#    Author: Juerg Haefliger <juerg.haefliger@hp.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -18,87 +20,314 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import yaml
+from StringIO import StringIO
+
+import copy as obj_copy
+import contextlib
+import errno
+import glob
+import grp
+import gzip
+import hashlib
 import os
-import os.path
+import platform
+import pwd
+import random
 import shutil
-import errno
-import subprocess
-from Cheetah.Template import Template
-import urllib2
-import urllib
-import logging
-import re
 import socket
+import stat
+import string  # pylint: disable=W0402
+import subprocess
 import sys
-import time
 import tempfile
-import traceback
+import time
+import types
 import urlparse
 
-try:
-    import selinux
-    HAVE_LIBSELINUX = True
-except ImportError:
-    HAVE_LIBSELINUX = False
+import yaml
+
+from cloudinit import importer
+from cloudinit import log as logging
+from cloudinit import url_helper as uhelp
+
+from cloudinit.settings import (CFG_BUILTIN)
+
+
+LOG = logging.getLogger(__name__)
+
+# Helps cleanup filenames to ensure they aren't FS incompatible
+FN_REPLACEMENTS = {
+    os.sep: '_',
+}
+FN_ALLOWED = ('_-.()' + string.digits + string.ascii_letters)
+
+# Helper utils to see if running in a container
+CONTAINER_TESTS = ['running-in-container', 'lxc-is-container']
+
+
+class ProcessExecutionError(IOError):
+
+    MESSAGE_TMPL = ('%(description)s\n'
+                    'Command: %(cmd)s\n'
+                    'Exit code: %(exit_code)s\n'
+                    'Reason: %(reason)s\n'
+                    'Stdout: %(stdout)r\n'
+                    'Stderr: %(stderr)r')
+
+    def __init__(self, stdout=None, stderr=None,
+                 exit_code=None, cmd=None,
+                 description=None, reason=None):
+        if not cmd:
+            self.cmd = '-'
+        else:
+            self.cmd = cmd
+
+        if not description:
+            self.description = 'Unexpected error while running command.'
+        else:
+            self.description = description
+
+        if not isinstance(exit_code, (long, int)):
+            self.exit_code = '-'
+        else:
+            self.exit_code = exit_code
+
+        if not stderr:
+            self.stderr = ''
+        else:
+            self.stderr = stderr
+
+        if not stdout:
+            self.stdout = ''
+        else:
+            self.stdout = stdout
+
+        if reason:
+            self.reason = reason
+        else:
+            self.reason = '-'
+
+        message = self.MESSAGE_TMPL % {
+            'description': self.description,
+            'cmd': self.cmd,
+            'exit_code': self.exit_code,
+            'stdout': self.stdout,
+            'stderr': self.stderr,
+            'reason': self.reason,
+        }
+        IOError.__init__(self, message)
+
+
+class SeLinuxGuard(object):
+    def __init__(self, path, recursive=False):
+        # Late import since it might not always
+        # be possible to use this
+        try:
+            self.selinux = importer.import_module('selinux')
+        except ImportError:
+            self.selinux = None
+        self.path = path
+        self.recursive = recursive
+
+    def __enter__(self):
+        if self.selinux:
+            return True
+        else:
+            return False
+
+    def __exit__(self, excp_type, excp_value, excp_traceback):
+        if self.selinux:
+            path = os.path.realpath(os.path.expanduser(self.path))
+            do_restore = False
+            try:
+                # See if even worth restoring??
+                stats = os.lstat(path)
+                if stat.ST_MODE in stats:
+                    self.selinux.matchpathcon(path, stats[stat.ST_MODE])
+                    do_restore = True
+            except OSError:
+                pass
+            if do_restore:
+                LOG.debug("Restoring selinux mode for %s (recursive=%s)",
+                          path, self.recursive)
+                self.selinux.restorecon(path, recursive=self.recursive)
+
+
+class MountFailedError(Exception):
+    pass
+
+
+def ExtendedTemporaryFile(**kwargs):
+    fh = tempfile.NamedTemporaryFile(**kwargs)
+    # Replace its unlink with a quiet version
+    # that does not raise errors when the
+    # file to unlink has been unlinked elsewhere..
+    LOG.debug("Created temporary file %s", fh.name)
+    fh.unlink = del_file
+
+    # Add a new method that will unlink
+    # right 'now' but still lets the exit
+    # method attempt to remove it (which will
+    # not throw due to our del file being quiet
+    # about files that are not there)
+    def unlink_now():
+        fh.unlink(fh.name)
+
+    setattr(fh, 'unlink_now', unlink_now)
+    return fh
+
+
+def fork_cb(child_cb, *args):
+    fid = os.fork()
+    if fid == 0:
+        try:
+            child_cb(*args)
+            os._exit(0)  # pylint: disable=W0212
+        except:
+            logexc(LOG, ("Failed forking and"
+                         " calling callback %s"), obj_name(child_cb))
+            os._exit(1)  # pylint: disable=W0212
+    else:
+        LOG.debug("Forked child %s who will run callback %s",
+                  fid, obj_name(child_cb))
+
+
+def is_true(val, addons=None):
+    if isinstance(val, (bool)):
+        return val is True
+    check_set = ['true', '1', 'on', 'yes']
+    if addons:
+        check_set = check_set + addons
+    if str(val).lower().strip() in check_set:
+        return True
+    return False
+
+
+def is_false(val, addons=None):
+    if isinstance(val, (bool)):
+        return val is False
+    check_set = ['off', '0', 'no', 'false']
+    if addons:
+        check_set = check_set + addons
+    if str(val).lower().strip() in check_set:
+        return True
+    return False
+
+
+def translate_bool(val, addons=None):
+    if not val:
+        # This handles empty lists and false and
+        # other things that python believes are false
+        return False
+    # If its already a boolean skip
+    if isinstance(val, (bool)):
+        return val
+    return is_true(val, addons)
+
+
+def rand_str(strlen=32, select_from=None):
+    if not select_from:
+        select_from = string.letters + string.digits
+    return "".join([random.choice(select_from) for _x in range(0, strlen)])
 
 
 def read_conf(fname):
     try:
-        stream = open(fname, "r")
-        conf = yaml.safe_load(stream)
-        stream.close()
-        return conf
+        return load_yaml(load_file(fname), default={})
     except IOError as e:
         if e.errno == errno.ENOENT:
             return {}
-        raise
-
+        else:
+            raise
 
-def get_base_cfg(cfgfile, cfg_builtin="", parsed_cfgs=None):
-    kerncfg = {}
-    syscfg = {}
-    if parsed_cfgs and cfgfile in parsed_cfgs:
-        return(parsed_cfgs[cfgfile])
 
-    syscfg = read_conf_with_confd(cfgfile)
+def clean_filename(fn):
+    for (k, v) in FN_REPLACEMENTS.iteritems():
+        fn = fn.replace(k, v)
+    removals = []
+    for k in fn:
+        if k not in FN_ALLOWED:
+            removals.append(k)
+    for k in removals:
+        fn = fn.replace(k, '')
+    fn = fn.strip()
+    return fn
 
-    kern_contents = read_cc_from_cmdline()
-    if kern_contents:
-        kerncfg = yaml.safe_load(kern_contents)
 
-    # kernel parameters override system config
-    combined = mergedict(kerncfg, syscfg)
+def decomp_str(data):
+    try:
+        buf = StringIO(str(data))
+        with contextlib.closing(gzip.GzipFile(None, "rb", 1, buf)) as gh:
+            return gh.read()
+    except:
+        return data
+
+
+def find_modules(root_dir):
+    entries = dict()
+    for fname in glob.glob(os.path.join(root_dir, "*.py")):
+        if not os.path.isfile(fname):
+            continue
+        modname = os.path.basename(fname)[0:-3]
+        modname = modname.strip()
+        if modname and modname.find(".") == -1:
+            entries[fname] = modname
+    return entries
+
+
+def multi_log(text, console=True, stderr=True,
+              log=None, log_level=logging.DEBUG):
+    if stderr:
+        sys.stderr.write(text)
+    if console:
+        # Don't use the write_file since
+        # this might be 'sensitive' info (not debug worthy?)
+        with open('/dev/console', 'wb') as wfh:
+            wfh.write(text)
+            wfh.flush()
+    if log:
+        log.log(log_level, text)
+
+
+def is_ipv4(instr):
+    """ determine if input string is a ipv4 address. return boolean"""
+    toks = instr.split('.')
+    if len(toks) != 4:
+        return False
 
-    if cfg_builtin:
-        builtin = yaml.safe_load(cfg_builtin)
-        fin = mergedict(combined, builtin)
-    else:
-        fin = combined
+    try:
+        toks = [x for x in toks if (int(x) < 256 and int(x) > 0)]
+    except:
+        return False
 
-    if parsed_cfgs != None:
-        parsed_cfgs[cfgfile] = fin
-    return(fin)
+    return (len(toks) == 4)
 
 
 def get_cfg_option_bool(yobj, key, default=False):
     if key not in yobj:
         return default
-    val = yobj[key]
-    if val is True:
-        return True
-    if str(val).lower() in ['true', '1', 'on', 'yes']:
-        return True
-    return False
+    return translate_bool(yobj[key])
 
 
 def get_cfg_option_str(yobj, key, default=None):
     if key not in yobj:
         return default
-    return yobj[key]
+    val = yobj[key]
+    if not isinstance(val, (str, basestring)):
+        val = str(val)
+    return val
 
 
-def get_cfg_option_list_or_str(yobj, key, default=None):
+def system_info():
+    return {
+        'platform': platform.platform(),
+        'release': platform.release(),
+        'python': platform.python_version(),
+        'uname': platform.uname(),
+    }
+
+
+def get_cfg_option_list(yobj, key, default=None):
     """
     Gets the C{key} config option from C{yobj} as a list of strings. If the
     key is present as a single string it will be returned as a list with one
@@ -114,9 +343,14 @@ def get_cfg_option_list_or_str(yobj, key, default=None):
         return default
     if yobj[key] is None:
         return []
-    if isinstance(yobj[key], list):
-        return yobj[key]
-    return [yobj[key]]
+    val = yobj[key]
+    if isinstance(val, (list)):
+        # Should we ensure they are all strings??
+        cval = [str(v) for v in val]
+        return cval
+    if not isinstance(val, (str, basestring)):
+        val = str(val)
+    return [val]
 
 
 # get a cfg entry by its path array
@@ -125,18 +359,121 @@ def get_cfg_by_path(yobj, keyp, default=None):
     cur = yobj
     for tok in keyp:
         if tok not in cur:
-            return(default)
+            return default
         cur = cur[tok]
-    return(cur)
+    return cur
+
+
+def fixup_output(cfg, mode):
+    (outfmt, errfmt) = get_output_cfg(cfg, mode)
+    redirect_output(outfmt, errfmt)
+    return (outfmt, errfmt)
+
+
+# redirect_output(outfmt, errfmt, orig_out, orig_err)
+#  replace orig_out and orig_err with filehandles specified in outfmt or errfmt
+#  fmt can be:
+#   > FILEPATH
+#   >> FILEPATH
+#   | program [ arg1 [ arg2 [ ... ] ] ]
+#
+#   with a '|', arguments are passed to shell, so one level of
+#   shell escape is required.
+def redirect_output(outfmt, errfmt, o_out=None, o_err=None):
+    if not o_out:
+        o_out = sys.stdout
+    if not o_err:
+        o_err = sys.stderr
+
+    if outfmt:
+        LOG.debug("Redirecting %s to %s", o_out, outfmt)
+        (mode, arg) = outfmt.split(" ", 1)
+        if mode == ">" or mode == ">>":
+            owith = "ab"
+            if mode == ">":
+                owith = "wb"
+            new_fp = open(arg, owith)
+        elif mode == "|":
+            proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
+            new_fp = proc.stdin
+        else:
+            raise TypeError("Invalid type for output format: %s" % outfmt)
+
+        if o_out:
+            os.dup2(new_fp.fileno(), o_out.fileno())
+
+        if errfmt == outfmt:
+            LOG.debug("Redirecting %s to %s", o_err, outfmt)
+            os.dup2(new_fp.fileno(), o_err.fileno())
+            return
+
+    if errfmt:
+        LOG.debug("Redirecting %s to %s", o_err, errfmt)
+        (mode, arg) = errfmt.split(" ", 1)
+        if mode == ">" or mode == ">>":
+            owith = "ab"
+            if mode == ">":
+                owith = "wb"
+            new_fp = open(arg, owith)
+        elif mode == "|":
+            proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
+            new_fp = proc.stdin
+        else:
+            raise TypeError("Invalid type for error format: %s" % errfmt)
+
+        if o_err:
+            os.dup2(new_fp.fileno(), o_err.fileno())
+
+
+def make_url(scheme, host, port=None,
+                path='', params='', query='', fragment=''):
+
+    pieces = []
+    pieces.append(scheme or '')
+
+    netloc = ''
+    if host:
+        netloc = str(host)
+
+    if port is not None:
+        netloc += ":" + "%s" % (port)
+
+    pieces.append(netloc or '')
+    pieces.append(path or '')
+    pieces.append(params or '')
+    pieces.append(query or '')
+    pieces.append(fragment or '')
+
+    return urlparse.urlunparse(pieces)
+
+
+def obj_name(obj):
+    if isinstance(obj, (types.TypeType,
+                        types.ModuleType,
+                        types.FunctionType,
+                        types.LambdaType)):
+        return str(obj.__name__)
+    return obj_name(obj.__class__)
+
+
+def mergemanydict(srcs, reverse=False):
+    if reverse:
+        srcs = reversed(srcs)
+    m_cfg = {}
+    for a_cfg in srcs:
+        if a_cfg:
+            m_cfg = mergedict(m_cfg, a_cfg)
+    return m_cfg
 
 
 def mergedict(src, cand):
     """
-    Merge values from C{cand} into C{src}. If C{src} has a key C{cand} will
-    not override. Nested dictionaries are merged recursively.
+    Merge values from C{cand} into C{src}.
+    If C{src} has a key C{cand} will not override.
+    Nested dictionaries are merged recursively.
     """
     if isinstance(src, dict) and isinstance(cand, dict):
-        for k, v in cand.iteritems():
+        for (k, v) in cand.iteritems():
             if k not in src:
                 src[k] = v
             else:
@@ -144,104 +481,66 @@ def mergedict(src, cand):
     return src
 
 
-def delete_dir_contents(dirname):
-    """
-    Deletes all contents of a directory without deleting the directory itself.
+@contextlib.contextmanager
+def chdir(ndir):
+    curr = os.getcwd()
+    try:
+        os.chdir(ndir)
+        yield ndir
+    finally:
+        os.chdir(curr)
 
-    @param dirname: The directory whose contents should be deleted.
-    """
-    for node in os.listdir(dirname):
-        node_fullpath = os.path.join(dirname, node)
-        if os.path.isdir(node_fullpath):
-            shutil.rmtree(node_fullpath)
-        else:
-            os.unlink(node_fullpath)
 
+@contextlib.contextmanager
+def umask(n_msk):
+    old = os.umask(n_msk)
+    try:
+        yield old
+    finally:
+        os.umask(old)
 
-def write_file(filename, content, mode=0644, omode="wb"):
-    """
-    Writes a file with the given content and sets the file mode as specified.
-    Resotres the SELinux context if possible.
 
-    @param filename: The full path of the file to write.
-    @param content: The content to write to the file.
-    @param mode: The filesystem mode to set on the file.
-    @param omode: The open mode used when opening the file (r, rb, a, etc.)
-    """
+@contextlib.contextmanager
+def tempdir(**kwargs):
+    # This seems like it was only added in python 3.2
+    # Make it since its useful...
+    # See: http://bugs.python.org/file12970/tempdir.patch
+    tdir = tempfile.mkdtemp(**kwargs)
     try:
-        os.makedirs(os.path.dirname(filename))
-    except OSError as e:
-        if e.errno != errno.EEXIST:
-            raise e
+        yield tdir
+    finally:
+        del_dir(tdir)
 
-    f = open(filename, omode)
-    if mode is not None:
-        os.chmod(filename, mode)
-    f.write(content)
-    f.close()
-    restorecon_if_possible(filename)
-
-
-def restorecon_if_possible(path, recursive=False):
-    if HAVE_LIBSELINUX and selinux.is_selinux_enabled():
-        selinux.restorecon(path, recursive=recursive)
-
-
-# get keyid from keyserver
-def getkeybyid(keyid, keyserver):
-    shcmd = """
-    k=${1} ks=${2};
-    exec 2>/dev/null
-    [ -n "$k" ] || exit 1;
-    armour=$(gpg --list-keys --armour "${k}")
-    if [ -z "${armour}" ]; then
-       gpg --keyserver ${ks} --recv $k >/dev/null &&
-          armour=$(gpg --export --armour "${k}") &&
-          gpg --batch --yes --delete-keys "${k}"
-    fi
-    [ -n "${armour}" ] && echo "${armour}"
-    """
-    args = ['sh', '-c', shcmd, "export-gpg-keyid", keyid, keyserver]
-    return(subp(args)[0])
+
+def center(text, fill, max_len):
+    return '{0:{fill}{align}{size}}'.format(text, fill=fill,
+                                            align="^", size=max_len)
+
+
+def del_dir(path):
+    LOG.debug("Recursively deleting %s", path)
+    shutil.rmtree(path)
 
 
 def runparts(dirp, skip_no_exist=True):
     if skip_no_exist and not os.path.isdir(dirp):
         return
 
-    failed = 0
+    failed = []
+    attempted = []
     for exe_name in sorted(os.listdir(dirp)):
         exe_path = os.path.join(dirp, exe_name)
         if os.path.isfile(exe_path) and os.access(exe_path, os.X_OK):
-            popen = subprocess.Popen([exe_path])
-            popen.communicate()
-            if popen.returncode is not 0:
-                failed += 1
-                sys.stderr.write("failed: %s [%i]\n" %
-                    (exe_path, popen.returncode))
-    if failed:
-        raise RuntimeError('runparts: %i failures' % failed)
-
-
-def subp(args, input_=None):
-    sp = subprocess.Popen(args, stdout=subprocess.PIPE,
-        stderr=subprocess.PIPE, stdin=subprocess.PIPE)
-    out, err = sp.communicate(input_)
-    if sp.returncode is not 0:
-        raise subprocess.CalledProcessError(sp.returncode, args, (out, err))
-    return(out, err)
-
-
-def render_to_file(template, outfile, searchList):
-    t = Template(file='/etc/cloud/templates/%s.tmpl' % template,
-                 searchList=[searchList])
-    f = open(outfile, 'w')
-    f.write(t.respond())
-    f.close()
-
+            attempted.append(exe_path)
+            try:
+                subp([exe_path])
+            except ProcessExecutionError as e:
+                logexc(LOG, "Failed running %s [%s]", exe_path, e.exit_code)
+                failed.append(e)
 
-def render_string(template, searchList):
-    return(Template(template, searchList=[searchList]).respond())
+    if failed and attempted:
+        raise RuntimeError('Runparts: %s failures in %s attempted commands'
+                           % (len(failed), len(attempted)))
 
 
 # read_optional_seed
@@ -254,13 +553,39 @@ def read_optional_seed(fill, base="", ext="", timeout=5):
         fill['user-data'] = ud
         fill['meta-data'] = md
         return True
-    except OSError, e:
+    except OSError as e:
         if e.errno == errno.ENOENT:
             return False
         raise
 
 
-# raise OSError with enoent if not found
+def read_file_or_url(url, timeout=5, retries=10, file_retries=0):
+    if url.startswith("/"):
+        url = "file://%s" % url
+    if url.startswith("file://"):
+        retries = file_retries
+    return uhelp.readurl(url, timeout=timeout, retries=retries)
+
+
+def load_yaml(blob, default=None, allowed=(dict,)):
+    loaded = default
+    try:
+        blob = str(blob)
+        LOG.debug(("Attempting to load yaml from string "
+                 "of length %s with allowed root types %s"),
+                 len(blob), allowed)
+        converted = yaml.safe_load(blob)
+        if not isinstance(converted, allowed):
+            # Yes this will just be caught, but thats ok for now...
+            raise TypeError(("Yaml load allows %s root types,"
+                             " but got %s instead") %
+                            (allowed, obj_name(converted)))
+        loaded = converted
+    except (yaml.YAMLError, TypeError, ValueError):
+        logexc(LOG, "Failed loading yaml blob")
+    return loaded
+
+
 def read_seeded(base="", ext="", timeout=5, retries=10, file_retries=0):
     if base.startswith("/"):
         base = "file://%s" % base
@@ -276,139 +601,62 @@ def read_seeded(base="", ext="", timeout=5, retries=10, file_retries=0):
         ud_url = "%s%s%s" % (base, "user-data", ext)
         md_url = "%s%s%s" % (base, "meta-data", ext)
 
-    no_exc = object()
-    raise_err = no_exc
-    for attempt in range(0, retries + 1):
-        try:
-            md_str = readurl(md_url, timeout=timeout)
-            ud = readurl(ud_url, timeout=timeout)
-            md = yaml.safe_load(md_str)
-
-            return(md, ud)
-        except urllib2.HTTPError as e:
-            raise_err = e
-        except urllib2.URLError as e:
-            raise_err = e
-            if (isinstance(e.reason, OSError) and
-                e.reason.errno == errno.ENOENT):
-                raise_err = e.reason
-
-        if attempt == retries:
-            break
-
-        #print "%s failed, sleeping" % attempt
-        time.sleep(1)
-
-    raise(raise_err)
-
+    md_resp = read_file_or_url(md_url, timeout, retries, file_retries)
+    md = None
+    if md_resp.ok():
+        md_str = str(md_resp)
+        md = load_yaml(md_str, default={})
 
-def logexc(log, lvl=logging.DEBUG):
-    log.log(lvl, traceback.format_exc())
+    ud_resp = read_file_or_url(ud_url, timeout, retries, file_retries)
+    ud = None
+    if ud_resp.ok():
+        ud_str = str(ud_resp)
+        ud = ud_str
 
-
-class RecursiveInclude(Exception):
-    pass
-
-
-def read_file_with_includes(fname, rel=".", stack=None, patt=None):
-    if stack is None:
-        stack = []
-    if not fname.startswith("/"):
-        fname = os.sep.join((rel, fname))
-
-    fname = os.path.realpath(fname)
-
-    if fname in stack:
-        raise(RecursiveInclude("%s recursively included" % fname))
-    if len(stack) > 10:
-        raise(RecursiveInclude("%s included, stack size = %i" %
-                               (fname, len(stack))))
-
-    if patt == None:
-        patt = re.compile("^#(opt_include|include)[ \t].*$", re.MULTILINE)
-
-    try:
-        fp = open(fname)
-        contents = fp.read()
-        fp.close()
-    except:
-        raise
-
-    rel = os.path.dirname(fname)
-    stack.append(fname)
-
-    cur = 0
-    while True:
-        match = patt.search(contents[cur:])
-        if not match:
-            break
-        loc = match.start() + cur
-        endl = match.end() + cur
-
-        (key, cur_fname) = contents[loc:endl].split(None, 2)
-        cur_fname = cur_fname.strip()
-
-        try:
-            inc_contents = read_file_with_includes(cur_fname, rel, stack, patt)
-        except IOError, e:
-            if e.errno == errno.ENOENT and key == "#opt_include":
-                inc_contents = ""
-            else:
-                raise
-        contents = contents[0:loc] + inc_contents + contents[endl + 1:]
-        cur = loc + len(inc_contents)
-    stack.pop()
-    return(contents)
+    return (md, ud)
 
 
 def read_conf_d(confd):
-    # get reverse sorted list (later trumps newer)
+    # Get reverse sorted list (later trumps newer)
     confs = sorted(os.listdir(confd), reverse=True)
 
-    # remove anything not ending in '.cfg'
+    # Remove anything not ending in '.cfg'
     confs = [f for f in confs if f.endswith(".cfg")]
 
-    # remove anything not a file
-    confs = [f for f in confs if os.path.isfile("%s/%s" % (confd, f))]
+    # Remove anything not a file
+    confs = [f for f in confs
+             if os.path.isfile(os.path.join(confd, f))]
 
-    cfg = {}
-    for conf in confs:
-        cfg = mergedict(cfg, read_conf("%s/%s" % (confd, conf)))
+    # Load them all so that they can be merged
+    cfgs = []
+    for fn in confs:
+        cfgs.append(read_conf(os.path.join(confd, fn)))
 
-    return(cfg)
+    return mergemanydict(cfgs)
 
 
 def read_conf_with_confd(cfgfile):
     cfg = read_conf(cfgfile)
+
     confd = False
     if "conf_d" in cfg:
-        if cfg['conf_d'] is not None:
-            confd = cfg['conf_d']
-            if not isinstance(confd, str):
-                raise Exception("cfgfile %s contains 'conf_d' "
-                                "with non-string" % cfgfile)
+        confd = cfg['conf_d']
+        if confd:
+            if not isinstance(confd, (str, basestring)):
+                raise TypeError(("Config file %s contains 'conf_d' "
+                                 "with non-string type %s") %
+                                 (cfgfile, obj_name(confd)))
+            else:
+                confd = str(confd).strip()
     elif os.path.isdir("%s.d" % cfgfile):
         confd = "%s.d" % cfgfile
 
-    if not confd:
-        return(cfg)
+    if not confd or not os.path.isdir(confd):
+        return cfg
 
+    # Conf.d settings override input configuration
     confd_cfg = read_conf_d(confd)
-
-    return(mergedict(confd_cfg, cfg))
-
-
-def get_cmdline():
-    if 'DEBUG_PROC_CMDLINE' in os.environ:
-        cmdline = os.environ["DEBUG_PROC_CMDLINE"]
-    else:
-        try:
-            cmdfp = open("/proc/cmdline")
-            cmdline = cmdfp.read().strip()
-            cmdfp.close()
-        except:
-            cmdline = ""
-    return(cmdline)
+    return mergedict(confd_cfg, cfg)
 
 
 def read_cc_from_cmdline(cmdline=None):
@@ -439,147 +687,15 @@ def read_cc_from_cmdline(cmdline=None):
 
         begin = cmdline.find(tag_begin, end + end_l)
 
-    return('\n'.join(tokens))
-
-
-def ensure_dirs(dirlist, mode=0755):
-    fixmodes = []
-    for d in dirlist:
-        try:
-            if mode != None:
-                os.makedirs(d)
-            else:
-                os.makedirs(d, mode)
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                raise
-            if mode != None:
-                fixmodes.append(d)
-
-    for d in fixmodes:
-        os.chmod(d, mode)
-
-
-def chownbyname(fname, user=None, group=None):
-    uid = -1
-    gid = -1
-    if user == None and group == None:
-        return
-    if user:
-        import pwd
-        uid = pwd.getpwnam(user).pw_uid
-    if group:
-        import grp
-        gid = grp.getgrnam(group).gr_gid
-
-    os.chown(fname, uid, gid)
-
+    return '\n'.join(tokens)
 
-def readurl(url, data=None, timeout=None):
-    openargs = {}
-    if timeout != None:
-        openargs['timeout'] = timeout
 
-    if data is None:
-        req = urllib2.Request(url)
-    else:
-        encoded = urllib.urlencode(data)
-        req = urllib2.Request(url, encoded)
-
-    response = urllib2.urlopen(req, **openargs)
-    return(response.read())
-
-
-# shellify, takes a list of commands
-#  for each entry in the list
-#    if it is an array, shell protect it (with single ticks)
-#    if it is a string, do nothing
-def shellify(cmdlist):
-    content = "#!/bin/sh\n"
-    escaped = "%s%s%s%s" % ("'", '\\', "'", "'")
-    for args in cmdlist:
-        # if the item is a list, wrap all items in single tick
-        # if its not, then just write it directly
-        if isinstance(args, list):
-            fixed = []
-            for f in args:
-                fixed.append("'%s'" % str(f).replace("'", escaped))
-            content = "%s%s\n" % (content, ' '.join(fixed))
-        else:
-            content = "%s%s\n" % (content, str(args))
-    return content
-
-
-def dos2unix(string):
+def dos2unix(contents):
     # find first end of line
-    pos = string.find('\n')
-    if pos <= 0 or string[pos - 1] != '\r':
-        return(string)
-    return(string.replace('\r\n', '\n'))
-
-
-def is_container():
-    # is this code running in a container of some sort
-
-    for helper in ('running-in-container', 'lxc-is-container'):
-        try:
-            # try to run a helper program. if it returns true
-            # then we're inside a container. otherwise, no
-            sp = subprocess.Popen(helper, stdout=subprocess.PIPE,
-                                  stderr=subprocess.PIPE)
-            sp.communicate(None)
-            return(sp.returncode == 0)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise
-
-    # this code is largely from the logic in
-    # ubuntu's /etc/init/container-detect.conf
-    try:
-        # Detect old-style libvirt
-        # Detect OpenVZ containers
-        pid1env = get_proc_env(1)
-        if "container" in pid1env:
-            return True
-
-        if "LIBVIRT_LXC_UUID" in pid1env:
-            return True
-
-    except IOError as e:
-        if e.errno != errno.ENOENT:
-            pass
-
-    # Detect OpenVZ containers
-    if os.path.isdir("/proc/vz") and not os.path.isdir("/proc/bc"):
-        return True
-
-    try:
-        # Detect Vserver containers
-        with open("/proc/self/status") as fp:
-            lines = fp.read().splitlines()
-            for line in lines:
-                if line.startswith("VxID:"):
-                    (_key, val) = line.strip().split(":", 1)
-                    if val != "0":
-                        return True
-    except IOError as e:
-        if e.errno != errno.ENOENT:
-            pass
-
-    return False
-
-
-def get_proc_env(pid):
-    # return the environment in a dict that a given process id was started with
-    env = {}
-    with open("/proc/%s/environ" % pid) as fp:
-        toks = fp.read().split("\0")
-        for tok in toks:
-            if tok == "":
-                continue
-            (name, val) = tok.split("=", 1)
-            env[name] = val
-    return env
+    pos = contents.find('\n')
+    if pos <= 0 or contents[pos - 1] != '\r':
+        return contents
+    return contents.replace('\r\n', '\n')
 
 
 def get_hostname_fqdn(cfg, cloud):
@@ -603,38 +719,72 @@ def get_hostname_fqdn(cfg, cloud):
                 hostname = cfg['hostname']
             else:
                 hostname = cloud.get_hostname()
-    return(hostname, fqdn)
+    return (hostname, fqdn)
 
 
 def get_fqdn_from_hosts(hostname, filename="/etc/hosts"):
-    # this parses /etc/hosts to get a fqdn.  It should return the same
-    # result as 'hostname -f <hostname>' if /etc/hosts.conf
-    # did not have did not have 'bind' in the order attribute
+    """
+    For each host a single line should be present with
+      the following information:
+
+        IP_address canonical_hostname [aliases...]
+
+      Fields of the entry are separated by any number of  blanks  and/or  tab
+      characters.  Text  from	a "#" character until the end of the line is a
+      comment, and is ignored.	 Host  names  may  contain  only  alphanumeric
+      characters, minus signs ("-"), and periods (".").  They must begin with
+      an  alphabetic  character  and  end  with  an  alphanumeric  character.
+      Optional aliases provide for name changes, alternate spellings, shorter
+      hostnames, or generic hostnames (for example, localhost).
+    """
     fqdn = None
     try:
-        with open(filename, "r") as hfp:
-            for line in hfp.readlines():
-                hashpos = line.find("#")
-                if hashpos >= 0:
-                    line = line[0:hashpos]
-                toks = line.split()
-
-                # if there there is less than 3 entries (ip, canonical, alias)
-                # then ignore this line
-                if len(toks) < 3:
-                    continue
-
-                if hostname in toks[2:]:
-                    fqdn = toks[1]
-                    break
-            hfp.close()
-    except IOError as e:
-        if e.errno == errno.ENOENT:
-            pass
+        for line in load_file(filename).splitlines():
+            hashpos = line.find("#")
+            if hashpos >= 0:
+                line = line[0:hashpos]
+            line = line.strip()
+            if not line:
+                continue
 
+            # If there there is less than 3 entries
+            # (IP_address, canonical_hostname, alias)
+            # then ignore this line
+            toks = line.split()
+            if len(toks) < 3:
+                continue
+
+            if hostname in toks[2:]:
+                fqdn = toks[1]
+                break
+    except IOError:
+        pass
     return fqdn
 
 
+def get_cmdline_url(names=('cloud-config-url', 'url'),
+                    starts="#cloud-config", cmdline=None):
+    if cmdline is None:
+        cmdline = get_cmdline()
+
+    data = keyval_str_to_dict(cmdline)
+    url = None
+    key = None
+    for key in names:
+        if key in data:
+            url = data[key]
+            break
+
+    if not url:
+        return (None, None, None)
+
+    resp = uhelp.readurl(url)
+    if resp.contents.startswith(starts) and resp.ok():
+        return (key, url, str(resp))
+
+    return (key, url, None)
+
+
 def is_resolvable(name):
     """ determine if a url is resolvable, return a boolean """
     try:
@@ -644,9 +794,14 @@ def is_resolvable(name):
         return False
 
 
+def get_hostname():
+    hostname = socket.gethostname()
+    return hostname
+
+
 def is_resolvable_url(url):
     """ determine if this url is resolvable (existing or ip) """
-    return(is_resolvable(urlparse.urlparse(url).hostname))
+    return (is_resolvable(urlparse.urlparse(url).hostname))
 
 
 def search_for_mirror(candidates):
@@ -656,8 +811,7 @@ def search_for_mirror(candidates):
             if is_resolvable_url(cand):
                 return cand
         except Exception:
-            raise
-
+            pass
     return None
 
 
@@ -669,13 +823,14 @@ def close_stdin():
     if _CLOUD_INIT_SAVE_STDIN is set in environment to a non empty or '0' value
     then input will not be closed (only useful potentially for debugging).
     """
-    if os.environ.get("_CLOUD_INIT_SAVE_STDIN") in ("", "0", False):
+    if os.environ.get("_CLOUD_INIT_SAVE_STDIN") in ("", "0", 'False'):
         return
     with open(os.devnull) as fp:
         os.dup2(fp.fileno(), sys.stdin.fileno())
 
 
-def find_devs_with(criteria):
+def find_devs_with(criteria=None, oformat='device',
+                    tag=None, no_cache=False, path=None):
     """
     find devices matching given criteria (via blkid)
     criteria can be *one* of:
@@ -683,165 +838,555 @@ def find_devs_with(criteria):
       LABEL=<label>
       UUID=<uuid>
     """
+    blk_id_cmd = ['blkid']
+    options = []
+    if criteria:
+        # Search for block devices with tokens named NAME that
+        # have the value 'value' and display any devices which are found.
+        # Common values for NAME include  TYPE, LABEL, and UUID.
+        # If there are no devices specified on the command line,
+        # all block devices will be searched; otherwise,
+        # only search the devices specified by the user.
+        options.append("-t%s" % (criteria))
+    if tag:
+        # For each (specified) device, show only the tags that match tag.
+        options.append("-s%s" % (tag))
+    if no_cache:
+        # If you want to start with a clean cache
+        # (i.e. don't report devices previously scanned
+        # but not necessarily available at this time), specify /dev/null.
+        options.extend(["-c", "/dev/null"])
+    if oformat:
+        # Display blkid's output using the specified format.
+        # The format parameter may be:
+        # full, value, list, device, udev, export
+        options.append('-o%s' % (oformat))
+    if path:
+        options.append(path)
+    cmd = blk_id_cmd + options
+    # See man blkid for why 2 is added
+    (out, _err) = subp(cmd, rcs=[0, 2])
+    entries = []
+    for line in out.splitlines():
+        line = line.strip()
+        if line:
+            entries.append(line)
+    return entries
+
+
+def load_file(fname, read_cb=None, quiet=False):
+    LOG.debug("Reading from %s (quiet=%s)", fname, quiet)
+    ofh = StringIO()
     try:
-        (out, _err) = subp(['blkid', '-t%s' % criteria, '-odevice'])
-    except subprocess.CalledProcessError:
-        return([])
-    return(str(out).splitlines())
+        with open(fname, 'rb') as ifh:
+            pipe_in_out(ifh, ofh, chunk_cb=read_cb)
+    except IOError as e:
+        if not quiet:
+            raise
+        if e.errno != errno.ENOENT:
+            raise
+    contents = ofh.getvalue()
+    LOG.debug("Read %s bytes from %s", len(contents), fname)
+    return contents
 
 
-class mountFailedError(Exception):
-    pass
+def get_cmdline():
+    if 'DEBUG_PROC_CMDLINE' in os.environ:
+        cmdline = os.environ["DEBUG_PROC_CMDLINE"]
+    else:
+        try:
+            cmdline = load_file("/proc/cmdline").strip()
+        except:
+            cmdline = ""
+    return cmdline
 
 
-def mount_callback_umount(device, callback, data=None):
+def pipe_in_out(in_fh, out_fh, chunk_size=1024, chunk_cb=None):
+    bytes_piped = 0
+    while True:
+        data = in_fh.read(chunk_size)
+        if data == '':
+            break
+        else:
+            out_fh.write(data)
+            bytes_piped += len(data)
+            if chunk_cb:
+                chunk_cb(bytes_piped)
+    out_fh.flush()
+    return bytes_piped
+
+
+def chownbyid(fname, uid=None, gid=None):
+    if uid in [None, -1] and gid in [None, -1]:
+        # Nothing to do
+        return
+    LOG.debug("Changing the ownership of %s to %s:%s", fname, uid, gid)
+    os.chown(fname, uid, gid)
+
+
+def chownbyname(fname, user=None, group=None):
+    uid = -1
+    gid = -1
+    try:
+        if user:
+            uid = pwd.getpwnam(user).pw_uid
+        if group:
+            gid = grp.getgrnam(group).gr_gid
+    except KeyError:
+        logexc(LOG, ("Failed changing the ownership of %s using username %s and"
+                     " groupname %s (do they exist?)"), fname, user, group)
+        return False
+    chownbyid(fname, uid, gid)
+    return True
+
+
+# Always returns well formated values
+# cfg is expected to have an entry 'output' in it, which is a dictionary
+# that includes entries for 'init', 'config', 'final' or 'all'
+#   init: /var/log/cloud.out
+#   config: [ ">> /var/log/cloud-config.out", /var/log/cloud-config.err ]
+#   final:
+#     output: "| logger -p"
+#     error: "> /dev/null"
+# this returns the specific 'mode' entry, cleanly formatted, with value
+def get_output_cfg(cfg, mode):
+    ret = [None, None]
+    if not cfg or not 'output' in cfg:
+        return ret
+
+    outcfg = cfg['output']
+    if mode in outcfg:
+        modecfg = outcfg[mode]
+    else:
+        if 'all' not in outcfg:
+            return ret
+        # if there is a 'all' item in the output list
+        # then it applies to all users of this (init, config, final)
+        modecfg = outcfg['all']
+
+    # if value is a string, it specifies stdout and stderr
+    if isinstance(modecfg, str):
+        ret = [modecfg, modecfg]
+
+    # if its a list, then we expect (stdout, stderr)
+    if isinstance(modecfg, list):
+        if len(modecfg) > 0:
+            ret[0] = modecfg[0]
+        if len(modecfg) > 1:
+            ret[1] = modecfg[1]
+
+    # if it is a dictionary, expect 'out' and 'error'
+    # items, which indicate out and error
+    if isinstance(modecfg, dict):
+        if 'output' in modecfg:
+            ret[0] = modecfg['output']
+        if 'error' in modecfg:
+            ret[1] = modecfg['error']
+
+    # if err's entry == "&1", then make it same as stdout
+    # as in shell syntax of "echo foo >/dev/null 2>&1"
+    if ret[1] == "&1":
+        ret[1] = ret[0]
+
+    swlist = [">>", ">", "|"]
+    for i in range(len(ret)):
+        if not ret[i]:
+            continue
+        val = ret[i].lstrip()
+        found = False
+        for s in swlist:
+            if val.startswith(s):
+                val = "%s %s" % (s, val[len(s):].strip())
+                found = True
+                break
+        if not found:
+            # default behavior is append
+            val = "%s %s" % (">>", val.strip())
+        ret[i] = val
+
+    return ret
+
+
+def logexc(log, msg, *args):
+    # Setting this here allows this to change
+    # levels easily (not always error level)
+    # or even desirable to have that much junk
+    # coming out to a non-debug stream
+    if msg:
+        log.warn(msg, *args)
+    # Debug gets the full trace
+    log.debug(msg, exc_info=1, *args)
+
+
+def hash_blob(blob, routine, mlen=None):
+    hasher = hashlib.new(routine)
+    hasher.update(blob)
+    digest = hasher.hexdigest()
+    # Don't get to long now
+    if mlen is not None:
+        return digest[0:mlen]
+    else:
+        return digest
+
+
+def rename(src, dest):
+    LOG.debug("Renaming %s to %s", src, dest)
+    # TODO use a se guard here??
+    os.rename(src, dest)
+
+
+def ensure_dirs(dirlist, mode=0755):
+    for d in dirlist:
+        ensure_dir(d, mode)
+
+
+def read_write_cmdline_url(target_fn):
+    if not os.path.exists(target_fn):
+        try:
+            (key, url, content) = get_cmdline_url()
+        except:
+            logexc(LOG, "Failed fetching command line url")
+            return
+        try:
+            if key and content:
+                write_file(target_fn, content, mode=0600)
+                LOG.debug(("Wrote to %s with contents of command line"
+                          " url %s (len=%s)"), target_fn, url, len(content))
+            elif key and not content:
+                LOG.debug(("Command line key %s with url"
+                          " %s had no contents"), key, url)
+        except:
+            logexc(LOG, "Failed writing url content to %s", target_fn)
+
+
+def yaml_dumps(obj):
+    formatted = yaml.dump(obj,
+                    line_break="\n",
+                    indent=4,
+                    explicit_start=True,
+                    explicit_end=True,
+                    default_flow_style=False,
+                    )
+    return formatted
+
+
+def ensure_dir(path, mode=None):
+    if not os.path.isdir(path):
+        # Make the dir and adjust the mode
+        with SeLinuxGuard(os.path.dirname(path), recursive=True):
+            os.makedirs(path)
+        chmod(path, mode)
+    else:
+        # Just adjust the mode
+        chmod(path, mode)
+
+
+@contextlib.contextmanager
+def unmounter(umount):
+    try:
+        yield umount
+    finally:
+        if umount:
+            umount_cmd = ["umount", '-l', umount]
+            subp(umount_cmd)
+
+
+def mounts():
+    mounted = {}
+    try:
+        # Go through mounts to see what is already mounted
+        mount_locs = load_file("/proc/mounts").splitlines()
+        for mpline in mount_locs:
+            # Format at: man fstab
+            try:
+                (dev, mp, fstype, opts, _freq, _passno) = mpline.split()
+            except:
+                continue
+            # If the name of the mount point contains spaces these
+            # can be escaped as '\040', so undo that..
+            mp = mp.replace("\\040", " ")
+            mounted[dev] = {
+                'fstype': fstype,
+                'mountpoint': mp,
+                'opts': opts,
+            }
+        LOG.debug("Fetched %s mounts from %s", mounted, "/proc/mounts")
+    except (IOError, OSError):
+        logexc(LOG, "Failed fetching mount points from /proc/mounts")
+    return mounted
+
+
+def mount_cb(device, callback, data=None, rw=False, mtype=None, sync=True):
     """
-    mount the device, call method 'callback' passing the directory
+    Mount the device, call method 'callback' passing the directory
     in which it was mounted, then unmount.  Return whatever 'callback'
     returned.  If data != None, also pass data to callback.
     """
-
-    def _cleanup(umount, tmpd):
-        if umount:
+    mounted = mounts()
+    with tempdir() as tmpd:
+        umount = False
+        if device in mounted:
+            mountpoint = mounted[device]['mountpoint']
+        else:
             try:
-                subp(["umount", '-l', umount])
-            except subprocess.CalledProcessError:
-                raise
-        if tmpd:
-            os.rmdir(tmpd)
+                mountcmd = ['mount']
+                mountopts = []
+                if rw:
+                    mountopts.append('rw')
+                else:
+                    mountopts.append('ro')
+                if sync:
+                    # This seems like the safe approach to do
+                    # (ie where this is on by default)
+                    mountopts.append("sync")
+                if mountopts:
+                    mountcmd.extend(["-o", ",".join(mountopts)])
+                if mtype:
+                    mountcmd.extend(['-t', mtype])
+                mountcmd.append(device)
+                mountcmd.append(tmpd)
+                subp(mountcmd)
+                umount = tmpd  # This forces it to be unmounted (when set)
+                mountpoint = tmpd
+            except (IOError, OSError) as exc:
+                raise MountFailedError(("Failed mounting %s "
+                                        "to %s due to: %s") %
+                                       (device, tmpd, exc))
+        # Be nice and ensure it ends with a slash
+        if not mountpoint.endswith("/"):
+            mountpoint += "/"
+        with unmounter(umount):
+            if data is None:
+                ret = callback(mountpoint)
+            else:
+                ret = callback(mountpoint, data)
+            return ret
 
-    # go through mounts to see if it was already mounted
-    fp = open("/proc/mounts")
-    mounts = fp.readlines()
-    fp.close()
 
-    tmpd = None
+def get_builtin_cfg():
+    # Deep copy so that others can't modify
+    return obj_copy.deepcopy(CFG_BUILTIN)
 
-    mounted = {}
-    for mpline in mounts:
-        (dev, mp, fstype, _opts, _freq, _passno) = mpline.split()
-        mp = mp.replace("\\040", " ")
-        mounted[dev] = (dev, fstype, mp, False)
-
-    umount = False
-    if device in mounted:
-        mountpoint = "%s/" % mounted[device][2]
-    else:
-        tmpd = tempfile.mkdtemp()
 
-        mountcmd = ["mount", "-o", "ro", device, tmpd]
+def sym_link(source, link):
+    LOG.debug("Creating symbolic link from %r => %r" % (link, source))
+    os.symlink(source, link)
 
-        try:
-            (_out, _err) = subp(mountcmd)
-            umount = tmpd
-        except subprocess.CalledProcessError as exc:
-            _cleanup(umount, tmpd)
-            raise mountFailedError(exc.output[1])
 
-        mountpoint = "%s/" % tmpd
+def del_file(path):
+    LOG.debug("Attempting to remove %s", path)
+    try:
+        os.unlink(path)
+    except OSError as e:
+        if e.errno != errno.ENOENT:
+            raise e
+
 
+def copy(src, dest):
+    LOG.debug("Copying %s to %s", src, dest)
+    shutil.copy(src, dest)
+
+
+def time_rfc2822():
     try:
-        if data == None:
-            ret = callback(mountpoint)
-        else:
-            ret = callback(mountpoint, data)
+        ts = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.gmtime())
+    except:
+        ts = "??"
+    return ts
 
-    except Exception as exc:
-        _cleanup(umount, tmpd)
-        raise exc
 
-    _cleanup(umount, tmpd)
+def uptime():
+    uptime_str = '??'
+    try:
+        contents = load_file("/proc/uptime").strip()
+        if contents:
+            uptime_str = contents.split()[0]
+    except:
+        logexc(LOG, "Unable to read uptime from /proc/uptime")
+    return uptime_str
+
 
-    return(ret)
+def ensure_file(path, mode=0644):
+    write_file(path, content='', omode="ab", mode=mode)
 
 
-def wait_for_url(urls, max_wait=None, timeout=None,
-                 status_cb=None, headers_cb=None):
+def chmod(path, mode):
+    real_mode = None
+    try:
+        real_mode = int(mode)
+    except (ValueError, TypeError):
+        pass
+    if path and real_mode:
+        with SeLinuxGuard(path):
+            os.chmod(path, real_mode)
+
+
+def write_file(filename, content, mode=0644, omode="wb"):
     """
-    urls:      a list of urls to try
-    max_wait:  roughly the maximum time to wait before giving up
-               The max time is *actually* len(urls)*timeout as each url will
-               be tried once and given the timeout provided.
-    timeout:   the timeout provided to urllib2.urlopen
-    status_cb: call method with string message when a url is not available
-    headers_cb: call method with single argument of url to get headers
-                for request.
-
-    the idea of this routine is to wait for the EC2 metdata service to
-    come up.  On both Eucalyptus and EC2 we have seen the case where
-    the instance hit the MD before the MD service was up.  EC2 seems
-    to have permenantely fixed this, though.
-
-    In openstack, the metadata service might be painfully slow, and
-    unable to avoid hitting a timeout of even up to 10 seconds or more
-    (LP: #894279) for a simple GET.
-
-    Offset those needs with the need to not hang forever (and block boot)
-    on a system where cloud-init is configured to look for EC2 Metadata
-    service but is not going to find one.  It is possible that the instance
-    data host (169.254.169.254) may be firewalled off Entirely for a sytem,
-    meaning that the connection will block forever unless a timeout is set.
+    Writes a file with the given content and sets the file mode as specified.
+    Resotres the SELinux context if possible.
+
+    @param filename: The full path of the file to write.
+    @param content: The content to write to the file.
+    @param mode: The filesystem mode to set on the file.
+    @param omode: The open mode used when opening the file (r, rb, a, etc.)
     """
-    starttime = time.time()
+    ensure_dir(os.path.dirname(filename))
+    LOG.debug("Writing to %s - %s: [%s] %s bytes",
+               filename, omode, mode, len(content))
+    with SeLinuxGuard(path=filename):
+        with open(filename, omode) as fh:
+            fh.write(content)
+            fh.flush()
+    chmod(filename, mode)
 
-    sleeptime = 1
 
-    def nullstatus_cb(msg):
-        return
+def delete_dir_contents(dirname):
+    """
+    Deletes all contents of a directory without deleting the directory itself.
+
+    @param dirname: The directory whose contents should be deleted.
+    """
+    for node in os.listdir(dirname):
+        node_fullpath = os.path.join(dirname, node)
+        if os.path.isdir(node_fullpath):
+            del_dir(node_fullpath)
+        else:
+            del_file(node_fullpath)
 
-    if status_cb == None:
-        status_cb = nullstatus_cb
 
-    def timeup(max_wait, starttime):
-        return((max_wait <= 0 or max_wait == None) or
-               (time.time() - starttime > max_wait))
+def subp(args, data=None, rcs=None, env=None, capture=True, shell=False):
+    if rcs is None:
+        rcs = [0]
+    try:
+        LOG.debug(("Running command %s with allowed return codes %s"
+                   " (shell=%s, capture=%s)"), args, rcs, shell, capture)
+        if not capture:
+            stdout = None
+            stderr = None
+        else:
+            stdout = subprocess.PIPE
+            stderr = subprocess.PIPE
+        stdin = subprocess.PIPE
+        sp = subprocess.Popen(args, stdout=stdout,
+                        stderr=stderr, stdin=stdin,
+                        env=env, shell=shell)
+        (out, err) = sp.communicate(data)
+    except OSError as e:
+        raise ProcessExecutionError(cmd=args, reason=e)
+    rc = sp.returncode
+    if rc not in rcs:
+        raise ProcessExecutionError(stdout=out, stderr=err,
+                                    exit_code=rc,
+                                    cmd=args)
+    # Just ensure blank instead of none?? (iff capturing)
+    if not out and capture:
+        out = ''
+    if not err and capture:
+        err = ''
+    return (out, err)
 
-    loop_n = 0
-    while True:
-        sleeptime = int(loop_n / 5) + 1
-        for url in urls:
-            now = time.time()
-            if loop_n != 0:
-                if timeup(max_wait, starttime):
-                    break
-                if timeout and (now + timeout > (starttime + max_wait)):
-                    # shorten timeout to not run way over max_time
-                    timeout = int((starttime + max_wait) - now)
-
-            reason = ""
-            try:
-                if headers_cb != None:
-                    headers = headers_cb(url)
-                else:
-                    headers = {}
-
-                req = urllib2.Request(url, data=None, headers=headers)
-                resp = urllib2.urlopen(req, timeout=timeout)
-                if resp.read() != "":
-                    return url
-                reason = "empty data [%s]" % resp.getcode()
-            except urllib2.HTTPError as e:
-                reason = "http error [%s]" % e.code
-            except urllib2.URLError as e:
-                reason = "url error [%s]" % e.reason
-            except socket.timeout as e:
-                reason = "socket timeout [%s]" % e
-            except Exception as e:
-                reason = "unexpected error [%s]" % e
-
-            status_cb("'%s' failed [%s/%ss]: %s" %
-                      (url, int(time.time() - starttime), max_wait,
-                       reason))
-
-        if timeup(max_wait, starttime):
-            break
 
-        loop_n = loop_n + 1
-        time.sleep(sleeptime)
+def abs_join(*paths):
+    return os.path.abspath(os.path.join(*paths))
+
+
+# shellify, takes a list of commands
+#  for each entry in the list
+#    if it is an array, shell protect it (with single ticks)
+#    if it is a string, do nothing
+def shellify(cmdlist, add_header=True):
+    content = ''
+    if add_header:
+        content += "#!/bin/sh\n"
+    escaped = "%s%s%s%s" % ("'", '\\', "'", "'")
+    cmds_made = 0
+    for args in cmdlist:
+        # If the item is a list, wrap all items in single tick.
+        # If its not, then just write it directly.
+        if isinstance(args, list):
+            fixed = []
+            for f in args:
+                fixed.append("'%s'" % (str(f).replace("'", escaped)))
+            content = "%s%s\n" % (content, ' '.join(fixed))
+            cmds_made += 1
+        elif isinstance(args, (str, basestring)):
+            content = "%s%s\n" % (content, args)
+            cmds_made += 1
+        else:
+            raise RuntimeError(("Unable to shellify type %s"
+                                " which is not a list or string")
+                               % (obj_name(args)))
+    LOG.debug("Shellified %s commands.", cmds_made)
+    return content
+
+
+def is_container():
+    """
+    Checks to see if this code running in a container of some sort
+    """
+
+    for helper in CONTAINER_TESTS:
+        try:
+            # try to run a helper program. if it returns true/zero
+            # then we're inside a container. otherwise, no
+            subp([helper])
+            return True
+        except (IOError, OSError):
+            pass
+
+    # this code is largely from the logic in
+    # ubuntu's /etc/init/container-detect.conf
+    try:
+        # Detect old-style libvirt
+        # Detect OpenVZ containers
+        pid1env = get_proc_env(1)
+        if "container" in pid1env:
+            return True
+        if "LIBVIRT_LXC_UUID" in pid1env:
+            return True
+    except (IOError, OSError):
+        pass
+
+    # Detect OpenVZ containers
+    if os.path.isdir("/proc/vz") and not os.path.isdir("/proc/bc"):
+        return True
+
+    try:
+        # Detect Vserver containers
+        lines = load_file("/proc/self/status").splitlines()
+        for line in lines:
+            if line.startswith("VxID:"):
+                (_key, val) = line.strip().split(":", 1)
+                if val != "0":
+                    return True
+    except (IOError, OSError):
+        pass
 
     return False
 
 
+def get_proc_env(pid):
+    """
+    Return the environment in a dict that a given process id was started with.
+    """
+
+    env = {}
+    fn = os.path.join("/proc/", str(pid), "environ")
+    try:
+        contents = load_file(fn)
+        toks = contents.split("\0")
+        for tok in toks:
+            if tok == "":
+                continue
+            (name, val) = tok.split("=", 1)
+            if name:
+                env[name] = val
+    except (IOError, OSError):
+        pass
+    return env
+
+
 def keyval_str_to_dict(kvstring):
     ret = {}
     for tok in kvstring.split():
@@ -851,5 +1396,4 @@ def keyval_str_to_dict(kvstring):
             key = tok
             val = True
         ret[key] = val
-
-    return(ret)
+    return ret
diff --git a/cloudinit/version.py b/cloudinit/version.py
new file mode 100644
index 00000000..4599910c
--- /dev/null
+++ b/cloudinit/version.py
@@ -0,0 +1,27 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from distutils import version as vr
+
+
+def version():
+    return vr.StrictVersion("0.7.0")
+
+
+def version_string():
+    return str(version())
diff --git a/config/cloud.cfg b/config/cloud.cfg
index 3688f88d..5dae4047 100644
--- a/config/cloud.cfg
+++ b/config/cloud.cfg
@@ -1,8 +1,24 @@
+# The top level settings are used as module
+# and system configuration.
+
+# This user will have its password adjusted
 user: ubuntu
-disable_root: 1
-preserve_hostname: False
-# datasource_list: ["NoCloud", "ConfigDrive", "OVF", "MAAS", "Ec2", "CloudStack"]
 
+# If this is set, 'root' will not be able to ssh in and they 
+# will get a message to login instead as the above $user (ubuntu)
+disable_root: true
+
+# This will cause the set+update hostname module to not operate (if true)
+preserve_hostname: false
+
+# Example datasource config
+# datasource: 
+#    Ec2: 
+#      metadata_urls: [ 'blah.com' ]
+#      timeout: 5 # (defaults to 50 seconds)
+#      max_wait: 10 # (defaults to 120 seconds)
+
+# The modules that run in the 'init' stage
 cloud_init_modules:
  - bootcmd
  - resizefs
@@ -13,6 +29,7 @@ cloud_init_modules:
  - rsyslog
  - ssh
 
+# The modules that run in the 'config' stage
 cloud_config_modules:
  - mounts
  - ssh-import-id
@@ -31,6 +48,7 @@ cloud_config_modules:
  - runcmd
  - byobu
 
+# The modules that run in the 'final' stage
 cloud_final_modules:
  - rightscale_userdata
  - scripts-per-once
@@ -40,3 +58,17 @@ cloud_final_modules:
  - keys-to-console
  - phone-home
  - final-message
+
+# System and/or distro specific settings
+# (not accessible to handlers/transforms)
+system_info:
+   # This will affect which distro class gets used
+   distro: ubuntu
+   # Other config here will be given to the distro class and/or path classes
+   paths:
+      cloud_dir: /var/lib/cloud/
+      templates_dir: /etc/cloud/templates/
+      upstart_dir: /etc/init/
+   package_mirror: http://archive.ubuntu.com/ubuntu
+   availability_zone_template: http://%(zone)s.ec2.archive.ubuntu.com/ubuntu/
+   ssh_svcname: ssh
diff --git a/config/cloud.cfg.d/05_logging.cfg b/config/cloud.cfg.d/05_logging.cfg
index 2e7ac2ed..410a0650 100644
--- a/config/cloud.cfg.d/05_logging.cfg
+++ b/config/cloud.cfg.d/05_logging.cfg
@@ -1,4 +1,4 @@
-## this yaml formated config file handles setting
+## This yaml formated config file handles setting
 ## logger information.  The values that are necessary to be set
 ## are seen at the bottom.  The top '_log' are only used to remove
 ## redundency in a syslog and fallback-to-file case.
@@ -53,5 +53,9 @@ _log:
    args=("/dev/log", handlers.SysLogHandler.LOG_USER)
 
 log_cfgs:
+# These will be joined into a string that defines the configuration
  - [ *log_base, *log_syslog ]
+# These will be joined into a string that defines the configuration
  - [ *log_base, *log_file ]
+# A file path can also be used
+# - /etc/log.conf
diff --git a/debian.trunk/rules b/debian.trunk/rules
deleted file mode 100755
index 19384687..00000000
--- a/debian.trunk/rules
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/usr/bin/make -f
-
-DEB_PYTHON2_MODULE_PACKAGES = cloud-init
-
-binary-install/cloud-init::cloud-init-fixups
-
-include /usr/share/cdbs/1/rules/debhelper.mk
-include /usr/share/cdbs/1/class/python-distutils.mk
-
-DEB_DH_INSTALL_SOURCEDIR := debian/tmp
-
-cloud-init-fixups:
-	for x in $(DEB_DESTDIR)/usr/bin/*.py; do mv "$$x" "$${x%.py}"; done
-	install -d $(DEB_DESTDIR)/etc/rsyslog.d
-	cp tools/21-cloudinit.conf $(DEB_DESTDIR)/etc/rsyslog.d/21-cloudinit.conf
-	ln -sf cloud-init-per $(DEB_DESTDIR)/usr/bin/cloud-init-run-module
-
-# You only need to run this immediately after checking out the package from
-# revision control.
-# http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572204
-quilt-setup:
-	@[ ! -d .pc ] || { echo ".pc exists. remove it and re-run to start fresh"; exit 1; }
-	set -e; for patch in $$(quilt series | tac); do \
-		patch -p1 -R --no-backup-if-mismatch <"debian/patches/$$patch"; \
-	done
-	quilt push -a
-
-.PHONY: quilt-setup
-
diff --git a/install.sh b/install.sh
deleted file mode 100755
index e7521bfa..00000000
--- a/install.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/sh
-
-#                    cd $(DEB_SRCDIR) && $(call cdbs_python_binary,python$(cdbs_python_compile_version)) $(DEB_PYTHON_SETUP_CMD) install --root=$(cdbs_python_destdir) $(DEB_PYTHON_INSTALL_ARGS_ALL)
-#                    for ddir in $(cdbs_python_destdir)/usr/lib/python?.?/dist-packages; do \
-#                      [ -d $$ddir ] || continue; \
-#                      sdir=$$(dirname $$ddir)/site-packages; \
-#                      mkdir -p $$sdir; \
-#                      tar -c -f - -C $$ddir . | tar -x -f - -C $$sdir; \
-#                      rm -rf $$ddir; \
-#                    done
-
-DEB_PYTHON_INSTALL_ARGS_ALL="-O0 --install-layout=deb"
-rm -Rf build
-
-destdir=$(readlink -f ${1})
-[ -z "${destdir}" ] && { echo "give destdir"; exit 1; }
-cd $(dirname ${0})
-./setup.py install --root=${destdir} ${DEB_PYTHON_INSTALL_ARGS_ALL}
-
-#mkdir -p ${destdir}/usr/share/pyshared
-#for x in ${destdir}/usr/lib/python2.6/dist-packages/*; do
-#   [ -d "$x" ] || continue
-#   [ ! -d "${destdir}/usr/share/pyshared/${x##*/}" ] ||
-#      rm -Rf "${destdir}/usr/share/pyshared/${x##*/}"
-#   mv $x ${destdir}/usr/share/pyshared
-#done
-#rm -Rf ${destdir}/usr/lib/python2.6
-
-for x in "${destdir}/usr/bin/"*.py; do
-   [ -f "${x}" ] && mv "${x}" "${x%.py}"
-done
diff --git a/packages/bddeb b/packages/bddeb
new file mode 100755
index 00000000..10ad08b3
--- /dev/null
+++ b/packages/bddeb
@@ -0,0 +1,172 @@
+#!/usr/bin/python
+
+import os
+import shutil
+import sys
+import glob
+
+# Use the util functions from cloudinit
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(
+        sys.argv[0]), os.pardir, os.pardir))
+if os.path.exists(os.path.join(possible_topdir, "cloudinit", "__init__.py")):
+    sys.path.insert(0, possible_topdir)
+
+from cloudinit import templater
+from cloudinit import util
+
+import argparse
+
+# Package names that will showup in requires to what we can actually
+# use in our debian 'control' file
+PKG_MP = {
+    'tempita': 'python-tempita',
+    'boto': 'python-boto',
+    'configobj': 'python-configobj',
+    'oauth': 'python-oauth',
+    'yaml': 'python-yaml',
+    'prettytable': 'python-prettytable',
+    'argparse': 'python-argparse',
+}
+
+
+def write_debian_folder(root, version, revno, init_sys):
+    deb_dir = util.abs_join(root, 'debian')
+    os.makedirs(deb_dir)
+    
+    # Fill in the change log template
+    templater.render_to_file(util.abs_join('debian', 'changelog'),
+                             util.abs_join(deb_dir, 'changelog'),
+                             params={
+                                 'version': version,
+                                 'revision': revno,
+                             })
+
+    # Write out the control file template
+    cmd = [sys.executable,
+            util.abs_join(os.pardir, 'tools', 'read-dependencies')]
+    (stdout, _stderr) = util.subp(cmd)
+    
+    # Map to known packages
+    pkgs = [p.lower().strip() for p in stdout.splitlines()]
+    requires = []
+    for p in pkgs:
+        tgt_pkg = None
+        for name in PKG_MP.keys():
+            if p.find(name) != -1:
+                tgt_pkg = PKG_MP.get(name)
+                break
+        if not tgt_pkg:
+            raise RuntimeError(("Do not know how to translate %s to "
+                                " a known package") % (p))
+        else:
+            requires.append(tgt_pkg)
+
+    templater.render_to_file(util.abs_join('debian', 'control'),
+                             util.abs_join(deb_dir, 'control'),
+                             params={'requires': requires})
+    
+    templater.render_to_file(util.abs_join('debian', 'rules'),
+                             util.abs_join(deb_dir, 'rules'),
+                             params={'init_sys': init_sys})
+    
+    # Just copy the following directly
+    for base_fn in ['dirs', 'copyright', 'compat', 'pycompat']:
+        shutil.copy(util.abs_join('debian', base_fn),
+                    util.abs_join(deb_dir, base_fn))
+
+
+def main():
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-n", "--no-sign", dest="sign",
+                        help=("attempt to sign "
+                              "the package (default: %(default)s)"),
+                        default=True,
+                        action='store_false')
+    parser.add_argument("-v", "--verbose", dest="verbose",
+                        help=("run verbosely"
+                              " (default: %(default)s)"),
+                        default=False,
+                        action='store_true')
+    parser.add_argument("-b", "--boot", dest="boot",
+                        help="select boot type (default: %(default)s)", 
+                        metavar="TYPE", default='upstart',
+                        choices=('upstart', 'upstart-local'))
+    args = parser.parse_args()
+
+    capture = True
+    if args.verbose:
+        capture = False
+
+    with util.tempdir() as tdir:
+
+        cmd = [sys.executable,
+                 util.abs_join(os.pardir, 'tools', 'read-version')]
+        (sysout, _stderr) = util.subp(cmd)
+        version = sysout.strip()
+
+        cmd = ['bzr', 'revno']
+        (sysout, _stderr) = util.subp(cmd)
+        revno = sysout.strip()
+        
+        # This is really only a temporary archive
+        # since we will extract it then add in the debian
+        # folder, then re-archive it for debian happiness
+        print("Creating a temporary tarball using the 'make-tarball' helper")
+        cmd = [sys.executable,
+                util.abs_join(os.getcwd(), 'make-tarball')]
+        (sysout, _stderr) = util.subp(cmd)
+        arch_fn = sysout.strip()
+        tmp_arch_fn = util.abs_join(tdir, os.path.basename(arch_fn))
+        shutil.move(arch_fn, tmp_arch_fn)
+
+        print("Extracting temporary tarball %r" % (tmp_arch_fn))
+        cmd = ['tar', '-xvzf', tmp_arch_fn, '-C', tdir]
+        util.subp(cmd, capture=capture)
+        base_name = os.path.basename(arch_fn)[:-len(".tar.gz")]
+        shutil.move(util.abs_join(tdir, base_name), 
+                    util.abs_join(tdir, 'cloud-init'))
+
+        print("Creating a debian/ folder in %r" % 
+             (util.abs_join(tdir, 'cloud-init')))
+        write_debian_folder(util.abs_join(tdir, 'cloud-init'),
+                            version, revno, args.boot)
+
+        # The naming here seems to follow some debian standard
+        # so it will whine if it is changed...
+        tar_fn = "cloud-init_%s~%s.orig.tar.gz" % (version, revno)
+        print("Archiving that new folder into %r" % (tar_fn))
+        cmd = ['tar', '-czvf', 
+               util.abs_join(tdir, tar_fn), 
+               '-C', util.abs_join(tdir, 'cloud-init')]
+        cmd.extend(os.listdir(util.abs_join(tdir, 'cloud-init')))
+        util.subp(cmd, capture=capture)
+        shutil.copy(util.abs_join(tdir, tar_fn), tar_fn)
+        print("Wrote out archive %r" % (util.abs_join(tar_fn)))
+
+        print("Running 'debuild' in %r" % (util.abs_join(tdir, 'cloud-init')))
+        with util.chdir(util.abs_join(tdir, 'cloud-init')):
+            cmd = ['debuild']
+            if not args.sign:
+                cmd.extend(['-us', '-uc'])
+            util.subp(cmd, capture=capture)
+
+        globs = [] 
+        globs.extend(glob.glob("%s/*.deb" %
+                     (os.path.join(tdir))))
+        link_fn = os.path.join(os.getcwd(), 'cloud-init_all.deb')
+        for fn in globs:
+            base_fn = os.path.basename(fn)
+            shutil.move(fn, base_fn)
+            print("Wrote out debian package %r" % (base_fn))
+            if fn.endswith('_all.deb'):
+                # Add in the local link
+                util.del_file(link_fn)
+                os.symlink(base_fn, link_fn)
+                print("Linked %r to %r" % (base_fn, link_fn))
+
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/packages/brpm b/packages/brpm
new file mode 100755
index 00000000..1d05bd2a
--- /dev/null
+++ b/packages/brpm
@@ -0,0 +1,216 @@
+#!/usr/bin/python
+
+import contextlib
+import glob
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import re
+
+import argparse
+
+# Use the util functions from cloudinit
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(
+        sys.argv[0]), os.pardir, os.pardir))
+if os.path.exists(os.path.join(possible_topdir, "cloudinit", "__init__.py")):
+    sys.path.insert(0, possible_topdir)
+
+from cloudinit import templater
+from cloudinit import util
+
+from datetime import datetime
+
+
+# Mapping of expected packages to there full name...
+PKG_MP = {
+    'boto': 'python-boto',
+    'tempita': 'python-tempita',
+    'prettytable': 'python-prettytable',
+    'oauth': 'python-oauth',
+    'configobj': 'python-configobj',
+    'yaml': 'PyYAML',
+    'argparse': 'python-argparse'
+}
+
+
+def get_log_header(version):
+    # Try to find the version in the tags output
+    cmd = ['bzr', 'tags']
+    (stdout, _stderr) = util.subp(cmd)
+    a_rev = None
+    for t in stdout.splitlines():
+        ver, rev = t.split(None)
+        if ver == version:
+            a_rev = rev
+            break
+    if not a_rev:
+        return format_change_line(datetime.now(),
+                                  '??', version)
+    
+    # Extract who made that tag as the header
+    cmd = ['bzr', 'log', '-r%s' % (a_rev), '--timezone=utc']
+    (stdout, _stderr) = util.subp(cmd)
+    kvs = {
+        'comment': version,
+    }
+
+    for line in stdout.splitlines():
+        if line.startswith('committer:'):
+            kvs['who'] = line[len('committer:'):].strip()
+        if line.startswith('timestamp:'):
+            ts = line[len('timestamp:'):]
+            ts = ts.strip()
+            # http://bugs.python.org/issue6641
+            ts = ts.replace("+0000", '').strip()
+            ds = datetime.strptime(ts, '%a %Y-%m-%d %H:%M:%S')
+            kvs['ds'] = ds
+
+    return format_change_line(**kvs)
+
+
+def format_change_line(ds, who, comment=None):
+    # Rpmbuild seems to be pretty strict about the date format
+    d = ds.strftime("%a %b %d %Y")
+    d += " - %s" % (who)
+    if comment:
+        d += " - %s" % (comment)
+    return "* %s" % (d)
+
+
+def generate_spec_contents(args, tmpl_fn):
+
+    # Figure out the version and revno
+    cmd = [sys.executable,
+            util.abs_join(os.pardir, 'tools', 'read-version')]
+    (stdout, _stderr) = util.subp(cmd)
+    version = stdout.strip()
+    
+    cmd = ['bzr', 'revno']
+    (stdout, _stderr) = util.subp(cmd)
+    revno = stdout.strip()
+
+    # Tmpl params
+    subs = {}
+    subs['version'] = version
+    subs['revno'] = revno
+    subs['release'] = revno
+    subs['archive_name'] = '%{name}-%{version}-' + revno + '.tar.gz'
+    subs['bd_requires'] = ['python-devel', 'python-setuptools']
+
+    cmd = [sys.executable, 
+            util.abs_join(os.pardir, 'tools', 'read-dependencies')]
+    (stdout, _stderr) = util.subp(cmd)
+
+    # Map to known packages
+    pkgs = [p.lower().strip() for p in stdout.splitlines()]
+
+    # Map to known packages
+    requires = []
+    for p in pkgs:
+        tgt_pkg = None
+        for name in PKG_MP.keys():
+            if p.find(name) != -1:
+                tgt_pkg = PKG_MP.get(name)
+                break
+        if not tgt_pkg:
+            raise RuntimeError(("Do not know how to translate %s to "
+                                " a known package") % (p))
+        else:
+            requires.append(tgt_pkg)
+    subs['requires'] = requires
+
+    # Format a nice changelog (as best as we can)
+    changelog = util.load_file(util.abs_join(os.pardir, 'ChangeLog'))
+    changelog_lines = []
+    for line in changelog.splitlines():
+        if not line.strip():
+            continue
+        if re.match(r"^\s*[\d][.][\d][.][\d]:\s*", line):
+            line = line.strip(":")
+            header = get_log_header(line)
+            changelog_lines.append(header)
+        else:
+            changelog_lines.append(line)
+    subs['changelog'] = "\n".join(changelog_lines)
+
+    if args.boot == 'initd':
+        subs['init_d'] = True
+        subs['init_d_local'] = False
+    elif args.boot == 'initd-local':
+        subs['init_d'] = True
+        subs['init_d_local'] = True
+    else:
+        subs['init_d'] = False
+        subs['init_d_local'] = False
+
+    if args.boot == 'systemd':
+        subs['systemd'] = True
+    else:
+        subs['systemd'] = False
+
+    subs['init_sys'] = args.boot
+    return templater.render_from_file(tmpl_fn, params=subs)
+
+
+def main():
+    
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-b", "--boot", dest="boot",
+                        help="select boot type (default: %(default)s)", 
+                        metavar="TYPE", default='initd',
+                        choices=('initd', 'systemd', 'initd-local'))
+    parser.add_argument("-v", "--verbose", dest="verbose",
+                        help=("run verbosely"
+                              " (default: %(default)s)"),
+                        default=False,
+                        action='store_true')
+    args = parser.parse_args()
+    capture = True
+    if args.verbose:
+        capture = False
+
+    # Clean out the root dir and make sure the dirs we want are in place
+    root_dir = os.path.expanduser("~/rpmbuild")
+    if os.path.isdir(root_dir):
+        shutil.rmtree(root_dir)
+    arc_dir = util.abs_join(root_dir, 'SOURCES')
+    util.ensure_dirs([root_dir, arc_dir])
+
+    # Archive the code
+    cmd = [sys.executable,
+            util.abs_join(os.getcwd(), 'make-tarball')]
+    (stdout, _stderr) = util.subp(cmd)
+    archive_fn = stdout.strip()
+    real_archive_fn = os.path.join(arc_dir, os.path.basename(archive_fn))
+    shutil.move(archive_fn, real_archive_fn)
+
+    # Form the spec file to be used
+    tmpl_fn = util.abs_join(os.getcwd(), 'redhat', 'cloud-init.spec')
+    contents = generate_spec_contents(args, tmpl_fn)
+    spec_fn = os.path.join(root_dir, 'cloud-init.spec')
+    util.write_file(spec_fn, contents)
+
+    # Now build it!
+    cmd = ['rpmbuild', '-ba', spec_fn]
+    util.subp(cmd, capture=capture)
+
+    # Copy the items built to our local dir
+    globs = [] 
+    globs.extend(glob.glob("%s/*.rpm" %
+                           (os.path.join(root_dir, 'RPMS', 'noarch'))))
+    globs.extend(glob.glob("%s/*.rpm" %
+                           (os.path.join(root_dir, 'RPMS'))))
+    globs.extend(glob.glob("%s/*.rpm" %
+                           (os.path.join(root_dir, 'SRPMS'))))
+    for rpm_fn in globs:
+        tgt_fn = util.abs_join(os.getcwd(), os.path.basename(rpm_fn))
+        shutil.move(rpm_fn, tgt_fn)
+        print(tgt_fn)
+
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/debian.trunk/changelog b/packages/debian/changelog
index 53e3678c..ac5bcf98 100644
--- a/debian.trunk/changelog
+++ b/packages/debian/changelog
@@ -1,4 +1,4 @@
-cloud-init (VERSION~REVNO-0) UNRELEASED; urgency=low
+cloud-init ({{version}}~{{revision}}-1) UNRELEASED; urgency=low
 
   * build
 
diff --git a/debian.trunk/compat b/packages/debian/compat
index 7ed6ff82..7ed6ff82 100644
--- a/debian.trunk/compat
+++ b/packages/debian/compat
diff --git a/debian.trunk/control b/packages/debian/control
index f2eec1e4..e00901af 100644
--- a/debian.trunk/control
+++ b/packages/debian/control
@@ -10,19 +10,17 @@ Build-Depends: cdbs,
                pylint,
                python-mocker,
 XS-Python-Version: all
-Standards-Version: 3.9.1
+Standards-Version: 3.9.3
 
 Package: cloud-init
 Architecture: all
 Depends: cloud-utils,
          procps,
          python,
-         python-boto (>=2.0),
-         python-cheetah,
-         python-configobj,
-         python-oauth,
+{{for r in requires}}
+         {{r}},
+{{endfor}}
          python-software-properties,
-         python-yaml,
          ${misc:Depends},
          ${python:Depends}
 XB-Python-Version: ${python:Versions}
diff --git a/debian.trunk/copyright b/packages/debian/copyright
index dc993525..dc993525 100644
--- a/debian.trunk/copyright
+++ b/packages/debian/copyright
diff --git a/debian.trunk/dirs b/packages/debian/dirs
index f3de468d..f3de468d 100644
--- a/debian.trunk/dirs
+++ b/packages/debian/dirs
diff --git a/debian.trunk/pycompat b/packages/debian/pycompat
index 0cfbf088..0cfbf088 100644
--- a/debian.trunk/pycompat
+++ b/packages/debian/pycompat
diff --git a/packages/debian/rules b/packages/debian/rules
new file mode 100755
index 00000000..87cd6538
--- /dev/null
+++ b/packages/debian/rules
@@ -0,0 +1,17 @@
+#!/usr/bin/make -f
+
+DEB_PYTHON2_MODULE_PACKAGES = cloud-init
+
+binary-install/cloud-init::cloud-init-fixups
+
+include /usr/share/cdbs/1/rules/debhelper.mk
+include /usr/share/cdbs/1/class/python-distutils.mk
+
+DEB_PYTHON_INSTALL_ARGS_ALL += --init-system={{init_sys}}
+
+DEB_DH_INSTALL_SOURCEDIR := debian/tmp
+
+cloud-init-fixups:
+	install -d $(DEB_DESTDIR)/etc/rsyslog.d
+	cp tools/21-cloudinit.conf $(DEB_DESTDIR)/etc/rsyslog.d/21-cloudinit.conf
+	
diff --git a/tools/make-dist-tarball b/packages/make-dist-tarball
index d6d53aa7..622283bd 100755
--- a/tools/make-dist-tarball
+++ b/packages/make-dist-tarball
@@ -9,7 +9,7 @@ Usage: ${0##*/} version
 EOF
 }
 
-topdir=$PWD
+topdir="../$PWD"
 tag=${1}
 
 [ -n "$tag" ] || { Usage 1>&2 ; exit 1; }
@@ -22,4 +22,4 @@ out=${topdir}/cloud-init-${tag}.tar.gz
 cd ${tmpd} && 
    bzr branch -r "tag:${tag}" "${topdir}" ./cloud-init-${tag} &&
    tar czf "${out}" cloud-init-${tag}/ --exclude cloud-init-${tag}/.bzr &&
-   echo "wrote ${out}"
+   echo "Wrote ${out}"
diff --git a/packages/make-tarball b/packages/make-tarball
new file mode 100755
index 00000000..43a6fc33
--- /dev/null
+++ b/packages/make-tarball
@@ -0,0 +1,89 @@
+#!/usr/bin/python
+
+import contextlib
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import optparse
+
+
+# Use the util functions from cloudinit
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(
+        sys.argv[0]), os.pardir, os.pardir))
+if os.path.exists(os.path.join(possible_topdir, "cloudinit", "__init__.py")):
+    sys.path.insert(0, possible_topdir)
+
+from cloudinit import util
+
+
+def find_versioned_files():
+    (stdout, _stderr) = util.subp(['bzr', 'ls', '--versioned', '--recursive'])
+    fns = [fn for fn in stdout.splitlines() 
+                  if fn and not fn.startswith('.')]
+    fns.sort()
+    return fns
+
+
+def copy(fn, where_to, verbose):
+    if verbose:
+        print("Copying %r --> %r" % (fn, where_to))
+    if os.path.isfile(fn):
+        shutil.copy(fn, where_to)
+    elif os.path.isdir(fn) and not os.path.isdir(where_to):
+        os.makedirs(where_to)
+    else:
+        raise RuntimeError("Do not know how to copy %s" % (fn))
+
+
+def main():
+
+    parser = optparse.OptionParser()
+    parser.add_option("-f", "--file", dest="filename",
+                      help="write archive to FILE", metavar="FILE")
+    parser.add_option("-v", "--verbose",
+                  action="store_true", dest="verbose", default=False,
+                  help="show verbose messaging")
+
+    (options, args) = parser.parse_args()
+
+    base_fn = options.filename
+    if not base_fn:
+        (stdout, _stderr) = util.subp(['bzr', 'revno'])
+        revno = stdout.strip()
+        cmd = [sys.executable,
+               util.abs_join(os.pardir, 'tools', 'read-version')]
+        (stdout, _stderr) = util.subp(cmd)
+        version = stdout.strip()
+        base_fn = 'cloud-init-%s-%s' % (version, revno)
+
+    with util.tempdir() as tdir:
+        util.ensure_dir(util.abs_join(tdir, base_fn))
+        arch_fn = '%s.tar.gz' % (base_fn)
+
+        with util.chdir(os.pardir):
+            fns =  find_versioned_files()
+            for fn in fns:
+                copy(fn, util.abs_join(tdir, base_fn, fn),
+                     verbose=options.verbose)
+
+        arch_full_fn = util.abs_join(tdir, arch_fn)
+        cmd = ['tar', '-czvf', arch_full_fn, '-C', tdir, base_fn]
+        if options.verbose:
+            print("Creating an archive from directory %r to %r" %
+                  (util.abs_join(tdir, base_fn), arch_full_fn))
+
+        util.subp(cmd, capture=(not options.verbose))
+        shutil.move(util.abs_join(tdir, arch_fn),
+                    util.abs_join(os.getcwd(), arch_fn))
+
+        print(os.path.abspath(arch_fn))
+        
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())
+
diff --git a/packages/redhat/cloud-init.spec b/packages/redhat/cloud-init.spec
new file mode 100644
index 00000000..d0f83a4b
--- /dev/null
+++ b/packages/redhat/cloud-init.spec
@@ -0,0 +1,183 @@
+%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")}
+
+# See: http://www.zarb.org/~jasonc/macros.php
+# Or: http://fedoraproject.org/wiki/Packaging:ScriptletSnippets
+# Or: http://www.rpm.org/max-rpm/ch-rpm-inside.html
+
+Name:           cloud-init
+Version:        {{version}}
+Release:        {{release}}%{?dist}
+Summary:        Cloud instance init scripts
+
+Group:          System Environment/Base
+License:        GPLv3
+URL:            http://launchpad.net/cloud-init
+
+Source0:        {{archive_name}}
+BuildArch:      noarch
+BuildRoot:      %{_tmppath}
+
+BuildRequires:        python-devel
+BuildRequires:        python-setuptools
+
+# System util packages needed
+Requires:       shadow-utils
+Requires:       rsyslog
+Requires:       iproute
+Requires:       e2fsprogs
+Requires:       net-tools
+Requires:       procps
+Requires:       shadow-utils
+
+# Install pypi 'dynamic' requirements
+{{for r in requires}}
+Requires:       {{r}}
+{{endfor}}
+
+{{if init_d}}
+Requires(post):       chkconfig
+Requires(postun):     initscripts
+Requires(preun):      chkconfig
+Requires(preun):      initscripts
+{{endif}}
+
+{{if systemd}}
+BuildRequires:  systemd-units
+Requires(post):   systemd-units
+Requires(postun): systemd-units
+Requires(preun):  systemd-units
+{{endif}}
+
+%description
+Cloud-init is a set of init scripts for cloud instances.  Cloud instances
+need special scripts to run during initialization to retrieve and install
+ssh keys and to let the user run various scripts.
+
+%prep
+%setup -q -n %{name}-%{version}-{{revno}}
+
+%build
+%{__python} setup.py build
+
+%install
+rm -rf $RPM_BUILD_ROOT
+%{__python} setup.py install -O1 \
+            --skip-build --root $RPM_BUILD_ROOT \
+            --init-system={{init_sys}}
+
+# Note that /etc/rsyslog.d didn't exist by default until F15.
+# el6 request: https://bugzilla.redhat.com/show_bug.cgi?id=740420
+mkdir -p $RPM_BUILD_ROOT/%{_sysconfdir}/rsyslog.d
+cp -p tools/21-cloudinit.conf \
+                    $RPM_BUILD_ROOT/%{_sysconfdir}/rsyslog.d/21-cloudinit.conf
+
+%clean
+rm -rf $RPM_BUILD_ROOT
+
+%post
+
+{{if systemd}}
+if [ $1 -eq 1 ]
+then
+    /bin/systemctl enable cloud-config.service     >/dev/null 2>&1 || :
+    /bin/systemctl enable cloud-final.service      >/dev/null 2>&1 || :
+    /bin/systemctl enable cloud-init.service       >/dev/null 2>&1 || :
+    /bin/systemctl enable cloud-init-local.service >/dev/null 2>&1 || :
+fi
+{{endif}}
+
+{{if init_d_local}}
+/sbin/chkconfig --add %{_initrddir}/cloud-init-local
+{{elif init_d}}
+/sbin/chkconfig --add %{_initrddir}/cloud-init
+{{endif}}
+{{if init_d}}
+/sbin/chkconfig --add %{_initrddir}/cloud-config
+/sbin/chkconfig --add %{_initrddir}/cloud-final
+{{endif}}
+
+%preun
+
+{{if init_d_local}}
+if [ $1 -eq 0 ]
+then
+    /sbin/service cloud-init-local stop >/dev/null 2>&1
+    /sbin/chkconfig --del cloud-init-local
+fi
+{{elif init_d}}
+if [ $1 -eq 0 ]
+then
+    /sbin/service cloud-init stop >/dev/null 2>&1
+    /sbin/chkconfig --del cloud-init
+fi
+{{endif}}
+{{if init_d}}
+if [ $1 -eq 0 ]
+then
+    /sbin/service cloud-config stop >/dev/null 2>&1
+    /sbin/chkconfig --del cloud-config
+    /sbin/service cloud-final stop >/dev/null 2>&1
+    /sbin/chkconfig --del cloud-final
+fi
+{{endif}}
+
+{{if systemd}}
+if [ $1 -eq 0 ]
+then
+    /bin/systemctl --no-reload disable cloud-config.service >/dev/null 2>&1 || :
+    /bin/systemctl --no-reload disable cloud-final.service  >/dev/null 2>&1 || :
+    /bin/systemctl --no-reload disable cloud-init.service   >/dev/null 2>&1 || :
+    /bin/systemctl --no-reload disable cloud-init-local.service >/dev/null 2>&1 || :
+fi
+{{endif}}
+
+%postun
+
+{{if systemd}}
+/bin/systemctl daemon-reload >/dev/null 2>&1 || :
+{{endif}}
+
+%files
+
+{{if init_d}}
+%attr(0755, root, root) %{_initddir}/cloud-config
+%attr(0755, root, root) %{_initddir}/cloud-final
+{{endif}}
+{{if init_d_local}}
+%attr(0755, root, root) %{_initddir}/cloud-init-local
+{{elif init_d}}
+%attr(0755, root, root) %{_initddir}/cloud-init
+{{endif}}
+
+{{if systemd}}
+%{_unitdir}/cloud-*
+{{endif}}
+
+# Program binaries
+%{_bindir}/cloud-init*
+
+# There doesn't seem to be an agreed upon place for these
+# although it appears the standard says /usr/lib but rpmbuild
+# will try /usr/lib64 ??
+/usr/lib/%{name}/uncloud-init
+/usr/lib/%{name}/write-ssh-key-fingerprints
+
+# Docs
+%doc TODO LICENSE ChangeLog Requires
+%doc %{_defaultdocdir}/cloud-init/*
+
+# Configs
+%config(noreplace)      %{_sysconfdir}/cloud/cloud.cfg
+%dir                    %{_sysconfdir}/cloud/cloud.cfg.d
+%config(noreplace)      %{_sysconfdir}/cloud/cloud.cfg.d/*.cfg
+%config(noreplace)      %{_sysconfdir}/cloud/cloud.cfg.d/README
+%dir                    %{_sysconfdir}/cloud/templates
+%config(noreplace)      %{_sysconfdir}/cloud/templates/*
+%config(noreplace) %{_sysconfdir}/rsyslog.d/21-cloudinit.conf
+
+# Python code is here...
+%{python_sitelib}/*
+
+%changelog
+
+{{changelog}}
diff --git a/setup.py b/setup.py
index f32662b8..06b897a5 100755
--- a/setup.py
+++ b/setup.py
@@ -1,10 +1,12 @@
-#!/usr/bin/python
 # vi: ts=4 expandtab
 #
 #    Distutils magic for ec2-init
+#
 #    Copyright (C) 2009 Canonical Ltd.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
 #    Author: Soren Hansen <soren@canonical.com>
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -17,36 +19,119 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-#
-from distutils.core import setup
+
 from glob import glob
-import os.path
+
+import os
+import re
+
+import setuptools
+from setuptools.command.install import install
+
+from distutils.command.install_data import install_data
+from distutils.errors import DistutilsArgError
+
 import subprocess
 
+
 def is_f(p):
-    return(os.path.isfile(p))
+    return os.path.isfile(p)
+
+
+INITSYS_FILES = {
+    'sysvinit': filter((lambda x: is_f(x)), glob('sysvinit/*')),
+    'systemd': filter((lambda x: is_f(x)), glob('systemd/*')),
+    'upstart': filter((lambda x: is_f(x)), glob('upstart/*')),
+}
+INITSYS_ROOTS = {
+    'sysvinit': '/etc/rc.d/init.d',
+    'systemd': '/etc/systemd/system/',
+    'upstart': '/etc/init/',
+}
+INITSYS_TYPES = sorted(list(INITSYS_ROOTS.keys()))
+
+
+def tiny_p(cmd, capture=True):
+    # Darn python 2.6 doesn't have check_output (argggg)
+    stdout = subprocess.PIPE
+    stderr = subprocess.PIPE
+    if not capture:
+        stdout = None
+        stderr = None
+    sp = subprocess.Popen(cmd, stdout=stdout,
+                    stderr=stderr, stdin=None)
+    (out, err) = sp.communicate()
+    if sp.returncode not in [0]:
+        raise RuntimeError("Failed running %s [rc=%s] (%s, %s)" 
+                            % (cmd, sp.returncode, out, err))
+    return (out, err)
+
+
+def get_version():
+    cmd = ['tools/read-version']
+    (ver, _e) = tiny_p(cmd)
+    return ver.strip()
+
+
+def read_requires():
+    cmd = ['tools/read-dependencies']
+    (deps, _e) = tiny_p(cmd)
+    return deps.splitlines()
+
+
+# TODO: Is there a better way to do this??
+class InitsysInstallData(install):
+    user_options = install.user_options + [
+        # This will magically show up in member variable 'init_sys'
+        ('init-system=', None,
+            ('init system to configure (%s) [default: None]') %
+                (", ".join(INITSYS_TYPES))
+        ),
+    ]
+
+    def initialize_options(self):
+        install.initialize_options(self)
+        self.init_system = None
+
+    def finalize_options(self):
+        install.finalize_options(self)
+        if self.init_system and self.init_system not in INITSYS_TYPES:
+                raise DistutilsArgError(
+                    ("You must specify one of (%s) when"
+                     " specifying a init system!") % (", ".join(INITSYS_TYPES))
+                )
+        elif self.init_system:
+            self.distribution.data_files.append((INITSYS_ROOTS[self.init_system], 
+                                                 INITSYS_FILES[self.init_system]))
+            # Force that command to reinitalize (with new file list)
+            self.distribution.reinitialize_command('install_data', True)
+
 
-setup(name='cloud-init',
-      version='0.6.3',
+setuptools.setup(name='cloud-init',
+      version=get_version(),
       description='EC2 initialisation magic',
       author='Scott Moser',
       author_email='scott.moser@canonical.com',
       url='http://launchpad.net/cloud-init/',
-      packages=['cloudinit', 'cloudinit.CloudConfig' ],
-      scripts=['cloud-init.py',
-               'cloud-init-cfg.py',
+      packages=setuptools.find_packages(exclude=['tests']),
+      scripts=['bin/cloud-init',
                'tools/cloud-init-per',
                ],
+      license='GPLv3',
       data_files=[('/etc/cloud', glob('config/*.cfg')),
                   ('/etc/cloud/cloud.cfg.d', glob('config/cloud.cfg.d/*')),
                   ('/etc/cloud/templates', glob('templates/*')),
-                  ('/etc/init', glob('upstart/*.conf')),
                   ('/usr/share/cloud-init', []),
-                  ('/usr/lib/cloud-init', 
+                  ('/usr/lib/cloud-init',
                     ['tools/uncloud-init', 'tools/write-ssh-key-fingerprints']),
-                  ('/usr/share/doc/cloud-init', filter(is_f,glob('doc/*'))),
-                  ('/usr/share/doc/cloud-init/examples', filter(is_f,glob('doc/examples/*'))),
-                  ('/usr/share/doc/cloud-init/examples/seed', filter(is_f,glob('doc/examples/seed/*'))),
-                  ('/etc/profile.d', ['tools/Z99-cloud-locale-test.sh']),
+                  ('/usr/share/doc/cloud-init', filter(is_f, glob('doc/*'))),
+                  ('/usr/share/doc/cloud-init/examples', filter(is_f, glob('doc/examples/*'))),
+                  ('/usr/share/doc/cloud-init/examples/seed', filter(is_f, glob('doc/examples/seed/*'))),
                   ],
+      install_requires=read_requires(),
+      cmdclass = {
+          # Use a subclass for install that handles
+          # adding on the right init system configuration files
+          'install': InitsysInstallData,
+      },
       )
diff --git a/sysvinit/cloud-config b/sysvinit/cloud-config
new file mode 100755
index 00000000..dd0bca8b
--- /dev/null
+++ b/sysvinit/cloud-config
@@ -0,0 +1,124 @@
+#!/bin/sh
+
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# See: http://wiki.debian.org/LSBInitScripts
+# See: http://tiny.cc/czvbgw
+# See: http://www.novell.com/coolsolutions/feature/15380.html
+# Also based on dhcpd in RHEL (for comparison)
+
+### BEGIN INIT INFO
+# Provides:          cloud-config
+# Required-Start:    cloud-init
+# Should-Start:      $time
+# Required-Stop:
+# Should-Stop:
+# Default-Start:     3 5
+# Default-Stop:
+# Short-Description: The config cloud-init job
+# Description:       Start cloud-init and runs the config phase
+#	and any associated config modules as desired.
+### END INIT INFO
+
+. /etc/init.d/functions
+
+# Return values acc. to LSB for all commands but status:
+# 0	  - success
+# 1       - generic or unspecified error
+# 2       - invalid or excess argument(s)
+# 3       - unimplemented feature (e.g. "reload")
+# 4       - user had insufficient privileges
+# 5       - program is not installed
+# 6       - program is not configured
+# 7       - program is not running
+# 8--199  - reserved (8--99 LSB, 100--149 distrib, 150--199 appl)
+# 
+# Note that starting an already running service, stopping
+# or restarting a not-running service as well as the restart
+# with force-reload (in case signaling is not supported) are
+# considered a success.
+
+RETVAL=0
+
+prog="cloud-init"
+cloud_init="/usr/bin/cloud-init"
+conf="/etc/cloud/cloud.cfg"
+
+# If there exists a sysconfig variable override file use it...
+[ -f /etc/sysconfig/cloud-init ] && . /etc/sysconfig/cloud-init
+
+start() {
+    [ -x $cloud_init ] || return 5
+    [ -f $conf ] || return 6
+
+    echo -n $"Starting $prog: "
+    $cloud_init $CLOUDINITARGS modules --mode config
+    RETVAL=$?
+    return $RETVAL
+}
+
+stop() {
+    echo -n $"Shutting down $prog: "
+    # No-op
+    RETVAL=7
+    return $RETVAL
+}
+
+. /etc/init.d/functions
+
+case "$1" in
+    start)
+        start
+        RETVAL=$?
+	;;
+    stop)
+        stop
+        RETVAL=$?
+	;;
+    restart|try-restart|condrestart)
+        ## Stop the service and regardless of whether it was
+        ## running or not, start it again.
+        # 
+        ## Note: try-restart is now part of LSB (as of 1.9).
+        ## RH has a similar command named condrestart.
+        start
+        RETVAL=$?
+	;;
+    reload|force-reload)
+        # It does not support reload
+        RETVAL=3
+	;;
+    status)
+        echo -n $"Checking for service $prog:"
+        # Return value is slightly different for the status command:
+        # 0 - service up and running
+        # 1 - service dead, but /var/run/  pid  file exists
+        # 2 - service dead, but /var/lock/ lock file exists
+        # 3 - service not running (unused)
+        # 4 - service status unknown :-(
+        # 5--199 reserved (5--99 LSB, 100--149 distro, 150--199 appl.)
+        RETVAL=3
+	;;
+    *)
+        echo "Usage: $0 {start|stop|status|try-restart|condrestart|restart|force-reload|reload}"
+        RETVAL=3
+	;;
+esac
+
+exit $RETVAL
diff --git a/sysvinit/cloud-final b/sysvinit/cloud-final
new file mode 100755
index 00000000..2e462c17
--- /dev/null
+++ b/sysvinit/cloud-final
@@ -0,0 +1,124 @@
+#!/bin/sh
+
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# See: http://wiki.debian.org/LSBInitScripts
+# See: http://tiny.cc/czvbgw
+# See: http://www.novell.com/coolsolutions/feature/15380.html
+# Also based on dhcpd in RHEL (for comparison)
+
+### BEGIN INIT INFO
+# Provides:          cloud-final
+# Required-Start:    $all cloud-init cloud-config
+# Should-Start:      $time
+# Required-Stop:
+# Should-Stop:
+# Default-Start:     3 5
+# Default-Stop:
+# Short-Description: The final cloud-init job
+# Description:       Start cloud-init and runs the final phase
+#	and any associated final modules as desired.
+### END INIT INFO
+
+. /etc/init.d/functions
+
+# Return values acc. to LSB for all commands but status:
+# 0	  - success
+# 1       - generic or unspecified error
+# 2       - invalid or excess argument(s)
+# 3       - unimplemented feature (e.g. "reload")
+# 4       - user had insufficient privileges
+# 5       - program is not installed
+# 6       - program is not configured
+# 7       - program is not running
+# 8--199  - reserved (8--99 LSB, 100--149 distrib, 150--199 appl)
+# 
+# Note that starting an already running service, stopping
+# or restarting a not-running service as well as the restart
+# with force-reload (in case signaling is not supported) are
+# considered a success.
+
+RETVAL=0
+
+prog="cloud-init"
+cloud_init="/usr/bin/cloud-init"
+conf="/etc/cloud/cloud.cfg"
+
+# If there exists a sysconfig variable override file use it...
+[ -f /etc/sysconfig/cloud-init ] && . /etc/sysconfig/cloud-init
+
+start() {
+    [ -x $cloud_init ] || return 5
+    [ -f $conf ] || return 6
+
+    echo -n $"Starting $prog: "
+    $cloud_init $CLOUDINITARGS modules --mode final
+    RETVAL=$?
+    return $RETVAL
+}
+
+stop() {
+    echo -n $"Shutting down $prog: "
+    # No-op
+    RETVAL=7
+    return $RETVAL
+}
+
+. /etc/init.d/functions
+
+case "$1" in
+    start)
+        start
+        RETVAL=$?
+	;;
+    stop)
+        stop
+        RETVAL=$?
+	;;
+    restart|try-restart|condrestart)
+        ## Stop the service and regardless of whether it was
+        ## running or not, start it again.
+        # 
+        ## Note: try-restart is now part of LSB (as of 1.9).
+        ## RH has a similar command named condrestart.
+        start
+        RETVAL=$?
+	;;
+    reload|force-reload)
+        # It does not support reload
+        RETVAL=3
+	;;
+    status)
+        echo -n $"Checking for service $prog:"
+        # Return value is slightly different for the status command:
+        # 0 - service up and running
+        # 1 - service dead, but /var/run/  pid  file exists
+        # 2 - service dead, but /var/lock/ lock file exists
+        # 3 - service not running (unused)
+        # 4 - service status unknown :-(
+        # 5--199 reserved (5--99 LSB, 100--149 distro, 150--199 appl.)
+        RETVAL=3
+	;;
+    *)
+        echo "Usage: $0 {start|stop|status|try-restart|condrestart|restart|force-reload|reload}"
+        RETVAL=3
+	;;
+esac
+
+exit $RETVAL
diff --git a/sysvinit/cloud-init b/sysvinit/cloud-init
new file mode 100755
index 00000000..7726c452
--- /dev/null
+++ b/sysvinit/cloud-init
@@ -0,0 +1,124 @@
+#!/bin/sh
+
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# See: http://wiki.debian.org/LSBInitScripts
+# See: http://tiny.cc/czvbgw
+# See: http://www.novell.com/coolsolutions/feature/15380.html
+# Also based on dhcpd in RHEL (for comparison)
+
+### BEGIN INIT INFO
+# Provides:          cloud-init
+# Required-Start:    $local_fs $network $named $remote_fs
+# Should-Start:      $time
+# Required-Stop:
+# Should-Stop:
+# Default-Start:     3 5
+# Default-Stop:
+# Short-Description: The initial cloud-init job (net and fs contingent)
+# Description:       Start cloud-init and runs the initialization phase
+#	and any associated initial modules as desired.
+### END INIT INFO
+
+. /etc/init.d/functions
+
+# Return values acc. to LSB for all commands but status:
+# 0	  - success
+# 1       - generic or unspecified error
+# 2       - invalid or excess argument(s)
+# 3       - unimplemented feature (e.g. "reload")
+# 4       - user had insufficient privileges
+# 5       - program is not installed
+# 6       - program is not configured
+# 7       - program is not running
+# 8--199  - reserved (8--99 LSB, 100--149 distrib, 150--199 appl)
+# 
+# Note that starting an already running service, stopping
+# or restarting a not-running service as well as the restart
+# with force-reload (in case signaling is not supported) are
+# considered a success.
+
+RETVAL=0
+
+prog="cloud-init"
+cloud_init="/usr/bin/cloud-init"
+conf="/etc/cloud/cloud.cfg"
+
+# If there exists a sysconfig variable override file use it...
+[ -f /etc/sysconfig/cloud-init ] && . /etc/sysconfig/cloud-init
+
+start() {
+    [ -x $cloud_init ] || return 5
+    [ -f $conf ] || return 6
+
+    echo -n $"Starting $prog: "
+    $cloud_init $CLOUDINITARGS init
+    RETVAL=$?
+    return $RETVAL
+}
+
+stop() {
+    echo -n $"Shutting down $prog: "
+    # No-op
+    RETVAL=7
+    return $RETVAL
+}
+
+. /etc/init.d/functions
+
+case "$1" in
+    start)
+        start
+        RETVAL=$?
+	;;
+    stop)
+        stop
+        RETVAL=$?
+	;;
+    restart|try-restart|condrestart)
+        ## Stop the service and regardless of whether it was
+        ## running or not, start it again.
+        # 
+        ## Note: try-restart is now part of LSB (as of 1.9).
+        ## RH has a similar command named condrestart.
+        start
+        RETVAL=$?
+	;;
+    reload|force-reload)
+        # It does not support reload
+        RETVAL=3
+	;;
+    status)
+        echo -n $"Checking for service $prog:"
+        # Return value is slightly different for the status command:
+        # 0 - service up and running
+        # 1 - service dead, but /var/run/  pid  file exists
+        # 2 - service dead, but /var/lock/ lock file exists
+        # 3 - service not running (unused)
+        # 4 - service status unknown :-(
+        # 5--199 reserved (5--99 LSB, 100--149 distro, 150--199 appl.)
+        RETVAL=3
+	;;
+    *)
+        echo "Usage: $0 {start|stop|status|try-restart|condrestart|restart|force-reload|reload}"
+        RETVAL=3
+	;;
+esac
+
+exit $RETVAL
diff --git a/sysvinit/cloud-init-local b/sysvinit/cloud-init-local
new file mode 100755
index 00000000..bf5d409a
--- /dev/null
+++ b/sysvinit/cloud-init-local
@@ -0,0 +1,124 @@
+#!/bin/sh
+
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <harlowja@yahoo-inc.com>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# See: http://wiki.debian.org/LSBInitScripts
+# See: http://tiny.cc/czvbgw
+# See: http://www.novell.com/coolsolutions/feature/15380.html
+# Also based on dhcpd in RHEL (for comparison)
+
+### BEGIN INIT INFO
+# Provides:          cloud-init
+# Required-Start:    $local_fs $remote_fs
+# Should-Start:      $time
+# Required-Stop:
+# Should-Stop:
+# Default-Start:     3 5
+# Default-Stop:
+# Short-Description: The initial cloud-init job (local fs contingent)
+# Description:       Start cloud-init and runs the initialization phases
+#	and any associated initial modules as desired.
+### END INIT INFO
+
+. /etc/init.d/functions
+
+# Return values acc. to LSB for all commands but status:
+# 0	  - success
+# 1       - generic or unspecified error
+# 2       - invalid or excess argument(s)
+# 3       - unimplemented feature (e.g. "reload")
+# 4       - user had insufficient privileges
+# 5       - program is not installed
+# 6       - program is not configured
+# 7       - program is not running
+# 8--199  - reserved (8--99 LSB, 100--149 distrib, 150--199 appl)
+# 
+# Note that starting an already running service, stopping
+# or restarting a not-running service as well as the restart
+# with force-reload (in case signaling is not supported) are
+# considered a success.
+
+RETVAL=0
+
+prog="cloud-init"
+cloud_init="/usr/bin/cloud-init"
+conf="/etc/cloud/cloud.cfg"
+
+# If there exists a sysconfig variable override file use it...
+[ -f /etc/sysconfig/cloud-init ] && . /etc/sysconfig/cloud-init
+
+start() {
+    [ -x $cloud_init ] || return 5
+    [ -f $conf ] || return 6
+
+    echo -n $"Starting $prog: "
+    $cloud_init $CLOUDINITARGS init --local
+    RETVAL=$?
+    return $RETVAL
+}
+
+stop() {
+    echo -n $"Shutting down $prog: "
+    # No-op
+    RETVAL=7
+    return $RETVAL
+}
+
+. /etc/init.d/functions
+
+case "$1" in
+    start)
+        start
+        RETVAL=$?
+	;;
+    stop)
+        stop
+        RETVAL=$?
+	;;
+    restart|try-restart|condrestart)
+        ## Stop the service and regardless of whether it was
+        ## running or not, start it again.
+        # 
+        ## Note: try-restart is now part of LSB (as of 1.9).
+        ## RH has a similar command named condrestart.
+        start
+        RETVAL=$?
+	;;
+    reload|force-reload)
+        # It does not support reload
+        RETVAL=3
+	;;
+    status)
+        echo -n $"Checking for service $prog:"
+        # Return value is slightly different for the status command:
+        # 0 - service up and running
+        # 1 - service dead, but /var/run/  pid  file exists
+        # 2 - service dead, but /var/lock/ lock file exists
+        # 3 - service not running (unused)
+        # 4 - service status unknown :-(
+        # 5--199 reserved (5--99 LSB, 100--149 distro, 150--199 appl.)
+        RETVAL=3
+	;;
+    *)
+        echo "Usage: $0 {start|stop|status|try-restart|condrestart|restart|force-reload|reload}"
+        RETVAL=3
+	;;
+esac
+
+exit $RETVAL
diff --git a/templates/chef_client.rb.tmpl b/templates/chef_client.rb.tmpl
index d3d9a922..35123ced 100644
--- a/templates/chef_client.rb.tmpl
+++ b/templates/chef_client.rb.tmpl
@@ -1,12 +1,12 @@
 log_level              :info
 log_location           "/var/log/chef/client.log"
 ssl_verify_mode        :verify_none
-validation_client_name "$validation_name"
+validation_client_name "{{validation_name}}"
 validation_key         "/etc/chef/validation.pem"
 client_key             "/etc/chef/client.pem"
-chef_server_url        "$server_url"
-environment            "$environment"
-node_name              "$node_name"
+chef_server_url        "{{server_url}}"
+environment            "{{environment}}"
+node_name              "{{node_name}}"
 json_attribs           "/etc/chef/firstboot.json"
 file_cache_path        "/var/cache/chef"
 file_backup_path       "/var/backups/chef"
diff --git a/templates/default-locale.tmpl b/templates/default-locale.tmpl
deleted file mode 100644
index 7940672b..00000000
--- a/templates/default-locale.tmpl
+++ /dev/null
@@ -1 +0,0 @@
-LANG="$locale"
diff --git a/templates/hosts.redhat.tmpl b/templates/hosts.redhat.tmpl
new file mode 100644
index 00000000..cfc40668
--- /dev/null
+++ b/templates/hosts.redhat.tmpl
@@ -0,0 +1,22 @@
+{{# This file /etc/cloud/templates/hosts.tmpl is only utilized
+    if enabled in cloud-config.  Specifically, in order to enable it
+    you need to add the following to config:
+      manage_etc_hosts: True}}
+# 
+# Your system has configured 'manage_etc_hosts' as True.
+# As a result, if you wish for changes to this file to persist
+# then you will need to either
+# a.) make changes to the master file in /etc/cloud/templates/hosts.tmpl
+# b.) change or remove the value of 'manage_etc_hosts' in
+#     /etc/cloud/cloud.cfg or cloud-config from user-data
+# 
+# The following lines are desirable for IPv4 capable hosts
+127.0.0.1 {{fqdn}} {{hostname}} 
+127.0.0.1 localhost.localdomain localhost
+127.0.0.1 localhost4.localdomain4 localhost4
+
+# The following lines are desirable for IPv6 capable hosts
+::1 {{fqdn}} {{hostname}}
+::1 localhost.localdomain localhost
+::1 localhost6.localdomain6 localhost6
+
diff --git a/templates/hosts.tmpl b/templates/hosts.ubuntu.tmpl
index ae120b02..9eebe971 100644
--- a/templates/hosts.tmpl
+++ b/templates/hosts.ubuntu.tmpl
@@ -1,9 +1,7 @@
-## This file (/etc/cloud/templates/hosts.tmpl) is only utilized
-## if enabled in cloud-config.  Specifically, in order to enable it
-## you need to add the following to config:
-##    manage_etc_hosts: True
-##
-## Note, double-hash commented lines will not appear in /etc/hosts
+{{# This file /etc/cloud/templates/hosts.tmpl is only utilized
+    if enabled in cloud-config.  Specifically, in order to enable it
+    you need to add the following to config:
+      manage_etc_hosts: True}}
 # 
 # Your system has configured 'manage_etc_hosts' as True.
 # As a result, if you wish for changes to this file to persist
@@ -12,8 +10,8 @@
 # b.) change or remove the value of 'manage_etc_hosts' in
 #     /etc/cloud/cloud.cfg or cloud-config from user-data
 # 
-## The value '$hostname' will be replaced with the local-hostname
-127.0.1.1 $fqdn $hostname
+# The following lines are desirable for IPv4 capable hosts
+127.0.1.1 {{fqdn}} {{hostname}}
 127.0.0.1 localhost
 
 # The following lines are desirable for IPv6 capable hosts
@@ -23,3 +21,4 @@ ff00::0 ip6-mcastprefix
 ff02::1 ip6-allnodes
 ff02::2 ip6-allrouters
 ff02::3 ip6-allhosts
+
diff --git a/templates/sources.list.tmpl b/templates/sources.list.tmpl
index f702025f..8acbd7d5 100644
--- a/templates/sources.list.tmpl
+++ b/templates/sources.list.tmpl
@@ -1,60 +1,59 @@
-\## Note, this file is written by cloud-init on first boot of an instance
-\## modifications made here will not survive a re-bundle.
-\## if you wish to make changes you can:
-\## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
-\##     or do the same in user-data
-\## b.) add sources in /etc/apt/sources.list.d
-\## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl
-\###
+# Note, this file is written by cloud-init on first boot of an instance
+# modifications made here will not survive a re-bundle.
+# if you wish to make changes you can:
+# a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
+#     or do the same in user-data
+# b.) add sources in /etc/apt/sources.list.d
+# c.) make changes to template file /etc/cloud/templates/sources.list.tmpl
 
 # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
 # newer versions of the distribution.
-deb $mirror $codename main
-deb-src $mirror $codename main
+deb {{mirror}} {{codename}} main
+deb-src {{mirror}} {{codename}} main
 
-\## Major bug fix updates produced after the final release of the
-\## distribution.
-deb $mirror $codename-updates main
-deb-src $mirror $codename-updates main
+# Major bug fix updates produced after the final release of the
+# distribution.
+deb {{mirror}} {{codename}}-updates main
+deb-src {{mirror}} {{codename}}-updates main
 
-\## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
-\## team. Also, please note that software in universe WILL NOT receive any
-\## review or updates from the Ubuntu security team.
-deb $mirror $codename universe
-deb-src $mirror $codename universe
-deb $mirror $codename-updates universe
-deb-src $mirror $codename-updates universe
+# N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
+# team. Also, please note that software in universe WILL NOT receive any
+# review or updates from the Ubuntu security team.
+deb {{mirror}} {{codename}} universe
+deb-src {{mirror}} {{codename}} universe
+deb {{mirror}} {{codename}}-updates universe
+deb-src {{mirror}} {{codename}}-updates universe
 
-\## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
-\## team, and may not be under a free licence. Please satisfy yourself as to
-\## your rights to use the software. Also, please note that software in 
-\## multiverse WILL NOT receive any review or updates from the Ubuntu
-\## security team.
-# deb $mirror $codename multiverse
-# deb-src $mirror $codename multiverse
-# deb $mirror $codename-updates multiverse
-# deb-src $mirror $codename-updates multiverse
+# N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
+# team, and may not be under a free licence. Please satisfy yourself as to
+# your rights to use the software. Also, please note that software in 
+# multiverse WILL NOT receive any review or updates from the Ubuntu
+# security team.
+# deb {{mirror}} {{codename}} multiverse
+# deb-src {{mirror}} {{codename}} multiverse
+# deb {{mirror}} {{codename}}-updates multiverse
+# deb-src {{mirror}} {{codename}}-updates multiverse
 
-\## Uncomment the following two lines to add software from the 'backports'
-\## repository.
-\## N.B. software from this repository may not have been tested as
-\## extensively as that contained in the main release, although it includes
-\## newer versions of some applications which may provide useful features.
-\## Also, please note that software in backports WILL NOT receive any review
-\## or updates from the Ubuntu security team.
-# deb $mirror $codename-backports main restricted universe multiverse
-# deb-src $mirror $codename-backports main restricted universe multiverse
+# Uncomment the following two lines to add software from the 'backports'
+# repository.
+# N.B. software from this repository may not have been tested as
+# extensively as that contained in the main release, although it includes
+# newer versions of some applications which may provide useful features.
+# Also, please note that software in backports WILL NOT receive any review
+# or updates from the Ubuntu security team.
+# deb {{mirror}} {{codename}}-backports main restricted universe multiverse
+# deb-src {{mirror}} {{codename}}-backports main restricted universe multiverse
 
-\## Uncomment the following two lines to add software from Canonical's
-\## 'partner' repository.
-\## This software is not part of Ubuntu, but is offered by Canonical and the
-\## respective vendors as a service to Ubuntu users.
-# deb http://archive.canonical.com/ubuntu $codename partner
-# deb-src http://archive.canonical.com/ubuntu $codename partner
+# Uncomment the following two lines to add software from Canonical's
+# 'partner' repository.
+# This software is not part of Ubuntu, but is offered by Canonical and the
+# respective vendors as a service to Ubuntu users.
+# deb http://archive.canonical.com/ubuntu {{codename}} partner
+# deb-src http://archive.canonical.com/ubuntu {{codename}} partner
 
-deb http://security.ubuntu.com/ubuntu $codename-security main
-deb-src http://security.ubuntu.com/ubuntu $codename-security main
-deb http://security.ubuntu.com/ubuntu $codename-security universe
-deb-src http://security.ubuntu.com/ubuntu $codename-security universe
-# deb http://security.ubuntu.com/ubuntu $codename-security multiverse
-# deb-src http://security.ubuntu.com/ubuntu $codename-security multiverse
+deb http://security.ubuntu.com/ubuntu {{codename}}-security main
+deb-src http://security.ubuntu.com/ubuntu {{codename}}-security main
+deb http://security.ubuntu.com/ubuntu {{codename}}-security universe
+deb-src http://security.ubuntu.com/ubuntu {{codename}}-security universe
+# deb http://security.ubuntu.com/ubuntu {{codename}}-security multiverse
+# deb-src http://security.ubuntu.com/ubuntu {{codename}}-security multiverse
diff --git a/tests/configs/sample1.yaml b/tests/configs/sample1.yaml
new file mode 100644
index 00000000..24e874ee
--- /dev/null
+++ b/tests/configs/sample1.yaml
@@ -0,0 +1,53 @@
+#cloud-config
+#apt_update: false
+#apt_upgrade: true
+packages: [ bzr, pastebinit, ubuntu-dev-tools, ccache, bzr-builddeb, vim-nox, git-core, lftp ]
+
+#apt_sources:
+# - source: ppa:smoser/ppa
+
+#disable_root: False
+
+# mounts:
+#  - [ ephemeral0, /mnt ]
+#  - [ swap, none, swap, sw, 0, 0 ]
+
+ssh_import_id: [smoser ]
+
+#!/bin/sh
+
+output: {all: '| tee -a /var/log/cloud-init-output.log'}
+
+sm_misc:
+ - &user_setup |
+   set -x; exec > ~/user_setup.log 2>&1
+   echo "starting at $(date -R)"
+   echo "set -o vi" >> ~/.bashrc
+   cat >> ~/.profile <<"EOF"
+   export EDITOR=vi
+   export DEB_BUILD_OPTIONS=parallel=4
+   export PATH=/usr/lib/ccache:$PATH
+   EOF
+
+   mkdir ~/bin
+   chmod 755 ~/bin
+   cat > ~/bin/mdebuild <<"EOF"
+   #!/bin/sh
+   exec debuild --prepend-path /usr/lib/ccache "$@"
+   EOF
+   chmod 755 ~/bin/*
+
+   #byobu-launcher-install
+   byobu-ctrl-a screen 2>&1 || :
+
+   echo "pinging 8.8.8.8"
+   ping -c 4 8.8.8.8
+
+runcmd:
+ - [ sudo, -Hu, ubuntu, sh, -c, '[ -e /var/log/cloud-init.log ] || exit 0; grep "cloud-init.*running" /var/log/cloud-init.log > ~/runcmd.log' ]
+ - [ sudo, -Hu, ubuntu, sh, -c, 'read up sleep < /proc/uptime; echo $(date): runcmd up at $up | tee -a ~/runcmd.log' ]
+ - [ sudo, -Hu, ubuntu, sh, -c, *user_setup ]
+ 
+
+byobu_by_default: user
+output: {all: '| tee -a /var/log/cloud-init-output.log'}
diff --git a/tests/unittests/test__init__.py b/tests/unittests/test__init__.py
index 4f60f0ea..af18955d 100644
--- a/tests/unittests/test__init__.py
+++ b/tests/unittests/test__init__.py
@@ -1,18 +1,42 @@
-from mocker import MockerTestCase, ANY, ARGS, KWARGS
+import StringIO
+import logging
 import os
+import sys
+
+from mocker import MockerTestCase, ANY, ARGS, KWARGS
+
+from cloudinit import handlers
+from cloudinit import helpers
+from cloudinit import importer
+from cloudinit import log
+from cloudinit import settings
+from cloudinit import url_helper
+from cloudinit import util
+
+
+class FakeModule(handlers.Handler):
+    def __init__(self):
+        handlers.Handler.__init__(self, settings.PER_ALWAYS)
+        self.types = []
+
+    def list_types(self):
+        return self.types
+
+    def _handle_part(self, data, ctype, filename, payload, frequency):
+        pass
 
-from cloudinit import (partwalker_handle_handler, handler_handle_part,
-                       handler_register, get_cmdline_url)
-from cloudinit.util import write_file, logexc, readurl
 
+class TestWalkerHandleHandler(MockerTestCase):
 
-class TestPartwalkerHandleHandler(MockerTestCase):
     def setUp(self):
+
+        MockerTestCase.setUp(self)
+
         self.data = {
             "handlercount": 0,
-            "frequency": "?",
-            "handlerdir": "?",
-            "handlers": [],
+            "frequency": "",
+            "handlerdir": self.makeDir(),
+            "handlers": helpers.ContentHandlers(),
             "data": None}
 
         self.expected_module_name = "part-handler-%03d" % (
@@ -20,179 +44,138 @@ class TestPartwalkerHandleHandler(MockerTestCase):
         expected_file_name = "%s.py" % self.expected_module_name
         expected_file_fullname = os.path.join(self.data["handlerdir"],
                                               expected_file_name)
-        self.module_fake = "fake module handle"
+        self.module_fake = FakeModule()
         self.ctype = None
         self.filename = None
         self.payload = "dummy payload"
 
         # Mock the write_file function
-        write_file_mock = self.mocker.replace(write_file, passthrough=False)
+        write_file_mock = self.mocker.replace(util.write_file, passthrough=False)
         write_file_mock(expected_file_fullname, self.payload, 0600)
 
     def test_no_errors(self):
         """Payload gets written to file and added to C{pdata}."""
-        # Mock the __import__ builtin
-        import_mock = self.mocker.replace("__builtin__.__import__")
+        import_mock = self.mocker.replace(importer.import_module, passthrough=False)
         import_mock(self.expected_module_name)
         self.mocker.result(self.module_fake)
-        # Mock the handle_register function
-        handle_reg_mock = self.mocker.replace(handler_register,
-                                              passthrough=False)
-        handle_reg_mock(self.module_fake, self.data["handlers"],
-                        self.data["data"], self.data["frequency"])
-        # Activate mocks
         self.mocker.replay()
-
-        partwalker_handle_handler(self.data, self.ctype, self.filename,
-                                  self.payload)
-
+ 
+        handlers.walker_handle_handler(self.data, self.ctype, self.filename,
+                                       self.payload)
+ 
         self.assertEqual(1, self.data["handlercount"])
-
+ 
     def test_import_error(self):
         """Module import errors are logged. No handler added to C{pdata}"""
-        # Mock the __import__ builtin
-        import_mock = self.mocker.replace("__builtin__.__import__")
+        import_mock = self.mocker.replace(importer.import_module, passthrough=False)
         import_mock(self.expected_module_name)
         self.mocker.throw(ImportError())
-        # Mock log function
-        logexc_mock = self.mocker.replace(logexc, passthrough=False)
-        logexc_mock(ANY)
-        # Mock the print_exc function
-        print_exc_mock = self.mocker.replace("traceback.print_exc",
-                                             passthrough=False)
-        print_exc_mock(ARGS, KWARGS)
-        # Activate mocks
         self.mocker.replay()
 
-        partwalker_handle_handler(self.data, self.ctype, self.filename,
-                                  self.payload)
+        handlers.walker_handle_handler(self.data, self.ctype, self.filename,
+                                       self.payload)
 
         self.assertEqual(0, self.data["handlercount"])
 
     def test_attribute_error(self):
         """Attribute errors are logged. No handler added to C{pdata}"""
-        # Mock the __import__ builtin
-        import_mock = self.mocker.replace("__builtin__.__import__")
+        import_mock = self.mocker.replace(importer.import_module, passthrough=False)
         import_mock(self.expected_module_name)
         self.mocker.result(self.module_fake)
-        # Mock the handle_register function
-        handle_reg_mock = self.mocker.replace(handler_register,
-                                              passthrough=False)
-        handle_reg_mock(self.module_fake, self.data["handlers"],
-                        self.data["data"], self.data["frequency"])
         self.mocker.throw(AttributeError())
-        # Mock log function
-        logexc_mock = self.mocker.replace(logexc, passthrough=False)
-        logexc_mock(ANY)
-        # Mock the print_exc function
-        print_exc_mock = self.mocker.replace("traceback.print_exc",
-                                             passthrough=False)
-        print_exc_mock(ARGS, KWARGS)
-        # Activate mocks
         self.mocker.replay()
 
-        partwalker_handle_handler(self.data, self.ctype, self.filename,
-                                  self.payload)
+        handlers.walker_handle_handler(self.data, self.ctype, self.filename,
+                                       self.payload)
 
         self.assertEqual(0, self.data["handlercount"])
 
 
 class TestHandlerHandlePart(MockerTestCase):
+
     def setUp(self):
         self.data = "fake data"
         self.ctype = "fake ctype"
         self.filename = "fake filename"
         self.payload = "fake payload"
-        self.frequency = "once-per-instance"
+        self.frequency = settings.PER_INSTANCE
 
     def test_normal_version_1(self):
         """
         C{handle_part} is called without C{frequency} for
         C{handler_version} == 1.
         """
-        # Build a mock part-handler module
         mod_mock = self.mocker.mock()
         getattr(mod_mock, "frequency")
-        self.mocker.result("once-per-instance")
+        self.mocker.result(settings.PER_INSTANCE)
         getattr(mod_mock, "handler_version")
         self.mocker.result(1)
         mod_mock.handle_part(self.data, self.ctype, self.filename,
                              self.payload)
         self.mocker.replay()
 
-        handler_handle_part(mod_mock, self.data, self.ctype, self.filename,
-                            self.payload, self.frequency)
+        handlers.run_part(mod_mock, self.data, self.ctype, self.filename,
+                          self.payload, self.frequency)
 
     def test_normal_version_2(self):
         """
         C{handle_part} is called with C{frequency} for
         C{handler_version} == 2.
         """
-        # Build a mock part-handler module
         mod_mock = self.mocker.mock()
         getattr(mod_mock, "frequency")
-        self.mocker.result("once-per-instance")
+        self.mocker.result(settings.PER_INSTANCE)
         getattr(mod_mock, "handler_version")
         self.mocker.result(2)
         mod_mock.handle_part(self.data, self.ctype, self.filename,
                              self.payload, self.frequency)
         self.mocker.replay()
 
-        handler_handle_part(mod_mock, self.data, self.ctype, self.filename,
-                            self.payload, self.frequency)
+        handlers.run_part(mod_mock, self.data, self.ctype, self.filename,
+                          self.payload, self.frequency)
 
     def test_modfreq_per_always(self):
         """
         C{handle_part} is called regardless of frequency if nofreq is always.
         """
         self.frequency = "once"
-        # Build a mock part-handler module
         mod_mock = self.mocker.mock()
         getattr(mod_mock, "frequency")
-        self.mocker.result("always")
+        self.mocker.result(settings.PER_ALWAYS)
         getattr(mod_mock, "handler_version")
         self.mocker.result(1)
         mod_mock.handle_part(self.data, self.ctype, self.filename,
                              self.payload)
         self.mocker.replay()
 
-        handler_handle_part(mod_mock, self.data, self.ctype, self.filename,
-                            self.payload, self.frequency)
+        handlers.run_part(mod_mock, self.data, self.ctype, self.filename,
+                          self.payload, self.frequency)
 
     def test_no_handle_when_modfreq_once(self):
         """C{handle_part} is not called if frequency is once"""
         self.frequency = "once"
-        # Build a mock part-handler module
         mod_mock = self.mocker.mock()
         getattr(mod_mock, "frequency")
-        self.mocker.result("once-per-instance")
+        self.mocker.result(settings.PER_ONCE)
         self.mocker.replay()
 
-        handler_handle_part(mod_mock, self.data, self.ctype, self.filename,
-                            self.payload, self.frequency)
+        handlers.run_part(mod_mock, self.data, self.ctype, self.filename,
+                          self.payload, self.frequency)
 
     def test_exception_is_caught(self):
         """Exceptions within C{handle_part} are caught and logged."""
-        # Build a mock part-handler module
         mod_mock = self.mocker.mock()
         getattr(mod_mock, "frequency")
-        self.mocker.result("once-per-instance")
+        self.mocker.result(settings.PER_INSTANCE)
         getattr(mod_mock, "handler_version")
         self.mocker.result(1)
         mod_mock.handle_part(self.data, self.ctype, self.filename,
                              self.payload)
         self.mocker.throw(Exception())
-        # Mock log function
-        logexc_mock = self.mocker.replace(logexc, passthrough=False)
-        logexc_mock(ANY)
-        # Mock the print_exc function
-        print_exc_mock = self.mocker.replace("traceback.print_exc",
-                                             passthrough=False)
-        print_exc_mock(ARGS, KWARGS)
         self.mocker.replay()
 
-        handler_handle_part(mod_mock, self.data, self.ctype, self.filename,
-                            self.payload, self.frequency)
+        handlers.run_part(mod_mock, self.data, self.ctype, self.filename,
+                          self.payload, self.frequency)
 
 
 class TestCmdlineUrl(MockerTestCase):
@@ -202,14 +185,13 @@ class TestCmdlineUrl(MockerTestCase):
         payload = "0"
         cmdline = "ro %s=%s bar=1" % (key, url)
 
-        mock_readurl = self.mocker.replace(readurl, passthrough=False)
+        mock_readurl = self.mocker.replace(url_helper.readurl, passthrough=False)
         mock_readurl(url)
-        self.mocker.result(payload)
-
+        self.mocker.result(url_helper.UrlResponse(200, payload))
         self.mocker.replay()
 
         self.assertEqual((key, url, None),
-            get_cmdline_url(names=[key], starts="xxxxxx", cmdline=cmdline))
+            util.get_cmdline_url(names=[key], starts="xxxxxx", cmdline=cmdline))
 
     def test_valid_content(self):
         url = "http://example.com/foo"
@@ -217,14 +199,13 @@ class TestCmdlineUrl(MockerTestCase):
         payload = "xcloud-config\nmydata: foo\nbar: wark\n"
         cmdline = "ro %s=%s bar=1" % (key, url)
 
-        mock_readurl = self.mocker.replace(readurl, passthrough=False)
+        mock_readurl = self.mocker.replace(url_helper.readurl, passthrough=False)
         mock_readurl(url)
-        self.mocker.result(payload)
-
+        self.mocker.result(url_helper.UrlResponse(200, payload))
         self.mocker.replay()
 
         self.assertEqual((key, url, payload),
-            get_cmdline_url(names=[key], starts="xcloud-config",
+            util.get_cmdline_url(names=[key], starts="xcloud-config",
                             cmdline=cmdline))
 
     def test_no_key_found(self):
@@ -232,11 +213,12 @@ class TestCmdlineUrl(MockerTestCase):
         key = "mykey"
         cmdline = "ro %s=%s bar=1" % (key, url)
 
-        self.mocker.replace(readurl, passthrough=False)
+        self.mocker.replace(url_helper.readurl, passthrough=False)
+        self.mocker.result(url_helper.UrlResponse(400))
         self.mocker.replay()
 
         self.assertEqual((None, None, None),
-            get_cmdline_url(names=["does-not-appear"],
+            util.get_cmdline_url(names=["does-not-appear"],
                 starts="#cloud-config", cmdline=cmdline))
 
 # vi: ts=4 expandtab
diff --git a/tests/unittests/test_builtin_handlers.py b/tests/unittests/test_builtin_handlers.py
new file mode 100644
index 00000000..84d85d4d
--- /dev/null
+++ b/tests/unittests/test_builtin_handlers.py
@@ -0,0 +1,54 @@
+"""Tests of the built-in user data handlers"""
+
+import os
+
+from mocker import MockerTestCase
+
+from cloudinit import handlers
+from cloudinit import helpers
+from cloudinit import util
+
+from cloudinit.handlers import upstart_job
+
+from cloudinit.settings import (PER_ALWAYS, PER_INSTANCE)
+
+
+class TestBuiltins(MockerTestCase):
+
+    def test_upstart_frequency_no_out(self):
+        c_root = self.makeDir()
+        up_root = self.makeDir()
+        paths = helpers.Paths({
+            'cloud_dir': c_root,
+            'upstart_dir': up_root,
+        })
+        freq = PER_ALWAYS
+        h = upstart_job.UpstartJobPartHandler(paths)
+        # No files should be written out when
+        # the frequency is ! per-instance
+        h.handle_part('', handlers.CONTENT_START,
+                      None, None, None)
+        h.handle_part('blah', 'text/upstart-job',
+                      'test.conf', 'blah', freq)
+        h.handle_part('', handlers.CONTENT_END,
+                      None, None, None)
+        self.assertEquals(0, len(os.listdir(up_root)))
+
+    def test_upstart_frequency_single(self): 
+        c_root = self.makeDir()
+        up_root = self.makeDir()
+        paths = helpers.Paths({
+            'cloud_dir': c_root,
+            'upstart_dir': up_root,
+        })
+        freq = PER_INSTANCE
+        h = upstart_job.UpstartJobPartHandler(paths)
+        # No files should be written out when
+        # the frequency is ! per-instance
+        h.handle_part('', handlers.CONTENT_START,
+                      None, None, None)
+        h.handle_part('blah', 'text/upstart-job',
+                      'test.conf', 'blah', freq)
+        h.handle_part('', handlers.CONTENT_END,
+                      None, None, None)
+        self.assertEquals(1, len(os.listdir(up_root)))
diff --git a/tests/unittests/test_datasource/test_maas.py b/tests/unittests/test_datasource/test_maas.py
index 7659dd03..261c410a 100644
--- a/tests/unittests/test_datasource/test_maas.py
+++ b/tests/unittests/test_datasource/test_maas.py
@@ -1,14 +1,11 @@
-from tempfile import mkdtemp
-from shutil import rmtree
 import os
 from StringIO import StringIO
 from copy import copy
-from cloudinit.DataSourceMAAS import (
-    MAASSeedDirNone,
-    MAASSeedDirMalformed,
-    read_maas_seed_dir,
-    read_maas_seed_url,
-)
+
+from cloudinit import util
+from cloudinit import url_helper
+from cloudinit.sources import DataSourceMAAS
+
 from mocker import MockerTestCase
 
 
@@ -17,12 +14,7 @@ class TestMAASDataSource(MockerTestCase):
     def setUp(self):
         super(TestMAASDataSource, self).setUp()
         # Make a temp directoy for tests to use.
-        self.tmp = mkdtemp(prefix="unittest_")
-
-    def tearDown(self):
-        super(TestMAASDataSource, self).tearDown()
-        # Clean up temp directory
-        rmtree(self.tmp)
+        self.tmp = self.makeDir()
 
     def test_seed_dir_valid(self):
         """Verify a valid seeddir is read as such"""
@@ -35,7 +27,7 @@ class TestMAASDataSource(MockerTestCase):
         my_d = os.path.join(self.tmp, "valid")
         populate_dir(my_d, data)
 
-        (userdata, metadata) = read_maas_seed_dir(my_d)
+        (userdata, metadata) = DataSourceMAAS.read_maas_seed_dir(my_d)
 
         self.assertEqual(userdata, data['user-data'])
         for key in ('instance-id', 'local-hostname'):
@@ -54,7 +46,7 @@ class TestMAASDataSource(MockerTestCase):
         my_d = os.path.join(self.tmp, "valid_extra")
         populate_dir(my_d, data)
 
-        (userdata, metadata) = read_maas_seed_dir(my_d)
+        (userdata, metadata) = DataSourceMAAS.read_maas_seed_dir(my_d)
 
         self.assertEqual(userdata, data['user-data'])
         for key in ('instance-id', 'local-hostname'):
@@ -76,24 +68,28 @@ class TestMAASDataSource(MockerTestCase):
         invalid_data = copy(valid)
         del invalid_data['local-hostname']
         populate_dir(my_d, invalid_data)
-        self.assertRaises(MAASSeedDirMalformed, read_maas_seed_dir, my_d)
+        self.assertRaises(DataSourceMAAS.MAASSeedDirMalformed,
+                          DataSourceMAAS.read_maas_seed_dir, my_d)
 
         # missing 'instance-id'
         my_d = "%s-02" % my_based
         invalid_data = copy(valid)
         del invalid_data['instance-id']
         populate_dir(my_d, invalid_data)
-        self.assertRaises(MAASSeedDirMalformed, read_maas_seed_dir, my_d)
+        self.assertRaises(DataSourceMAAS.MAASSeedDirMalformed,
+                          DataSourceMAAS.read_maas_seed_dir, my_d)
 
     def test_seed_dir_none(self):
         """Verify that empty seed_dir raises MAASSeedDirNone"""
 
         my_d = os.path.join(self.tmp, "valid_empty")
-        self.assertRaises(MAASSeedDirNone, read_maas_seed_dir, my_d)
+        self.assertRaises(DataSourceMAAS.MAASSeedDirNone,
+                          DataSourceMAAS.read_maas_seed_dir, my_d)
 
     def test_seed_dir_missing(self):
         """Verify that missing seed_dir raises MAASSeedDirNone"""
-        self.assertRaises(MAASSeedDirNone, read_maas_seed_dir,
+        self.assertRaises(DataSourceMAAS.MAASSeedDirNone, 
+            DataSourceMAAS.read_maas_seed_dir,
             os.path.join(self.tmp, "nonexistantdirectory"))
 
     def test_seed_url_valid(self):
@@ -102,30 +98,30 @@ class TestMAASDataSource(MockerTestCase):
             'meta-data/local-hostname': 'test-hostname',
             'meta-data/public-keys': 'test-hostname',
             'user-data': 'foodata'}
-
+        valid_order = [
+            'meta-data/local-hostname',
+            'meta-data/instance-id',
+            'meta-data/public-keys',
+            'user-data',
+        ]
         my_seed = "http://example.com/xmeta"
         my_ver = "1999-99-99"
         my_headers = {'header1': 'value1', 'header2': 'value2'}
 
         def my_headers_cb(url):
-            return(my_headers)
+            return my_headers
 
-        mock_request = self.mocker.replace("urllib2.Request",
-            passthrough=False)
-        mock_urlopen = self.mocker.replace("urllib2.urlopen",
+        mock_request = self.mocker.replace(url_helper.readurl,
             passthrough=False)
 
-        for (key, val) in valid.iteritems():
-            mock_request("%s/%s/%s" % (my_seed, my_ver, key),
-                data=None, headers=my_headers)
-            self.mocker.nospec()
-            self.mocker.result("fake-request-%s" % key)
-            mock_urlopen("fake-request-%s" % key, timeout=None)
-            self.mocker.result(StringIO(val))
-
+        for key in valid_order:
+            url = "%s/%s/%s" % (my_seed, my_ver, key)
+            mock_request(url, headers=my_headers, timeout=None)
+            resp = valid.get(key)
+            self.mocker.result(url_helper.UrlResponse(200, resp))
         self.mocker.replay()
 
-        (userdata, metadata) = read_maas_seed_url(my_seed,
+        (userdata, metadata) = DataSourceMAAS.read_maas_seed_url(my_seed,
             header_cb=my_headers_cb, version=my_ver)
 
         self.assertEqual("foodata", userdata)
diff --git a/tests/unittests/test_handler/test_handler_ca_certs.py b/tests/unittests/test_handler/test_handler_ca_certs.py
index 21d2442f..1f96e992 100644
--- a/tests/unittests/test_handler/test_handler_ca_certs.py
+++ b/tests/unittests/test_handler/test_handler_ca_certs.py
@@ -1,9 +1,12 @@
 from mocker import MockerTestCase
 
-from cloudinit.util import write_file, delete_dir_contents
-from cloudinit.CloudConfig.cc_ca_certs import (
-    handle, update_ca_certs, add_ca_certs, remove_default_ca_certs)
-from logging import getLogger
+from cloudinit import util
+from cloudinit import cloud
+from cloudinit import helpers
+
+from cloudinit.config import cc_ca_certs
+
+import logging
 
 
 class TestNoConfig(MockerTestCase):
@@ -11,36 +14,37 @@ class TestNoConfig(MockerTestCase):
         super(TestNoConfig, self).setUp()
         self.name = "ca-certs"
         self.cloud_init = None
-        self.log = getLogger("TestNoConfig")
+        self.log = logging.getLogger("TestNoConfig")
         self.args = []
 
     def test_no_config(self):
         """
         Test that nothing is done if no ca-certs configuration is provided.
         """
-        config = {"unknown-key": "value"}
-
-        self.mocker.replace(write_file, passthrough=False)
-        self.mocker.replace(update_ca_certs, passthrough=False)
+        config = util.get_builtin_cfg()
+        self.mocker.replace(util.write_file, passthrough=False)
+        self.mocker.replace(cc_ca_certs.update_ca_certs, passthrough=False)
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud_init, self.log, self.args)
 
 
 class TestConfig(MockerTestCase):
     def setUp(self):
         super(TestConfig, self).setUp()
         self.name = "ca-certs"
-        self.cloud_init = None
-        self.log = getLogger("TestNoConfig")
+        self.paths = None
+        self.cloud = cloud.Cloud(None, self.paths, None, None, None)
+        self.log = logging.getLogger("TestNoConfig")
         self.args = []
 
         # Mock out the functions that actually modify the system
-        self.mock_add = self.mocker.replace(add_ca_certs, passthrough=False)
-        self.mock_update = self.mocker.replace(update_ca_certs,
+        self.mock_add = self.mocker.replace(cc_ca_certs.add_ca_certs, passthrough=False)
+        self.mock_update = self.mocker.replace(cc_ca_certs.update_ca_certs,
                                                passthrough=False)
-        self.mock_remove = self.mocker.replace(remove_default_ca_certs,
+        self.mock_remove = self.mocker.replace(cc_ca_certs.remove_default_ca_certs,
                                                passthrough=False)
+
         # Order must be correct
         self.mocker.order()
 
@@ -55,7 +59,7 @@ class TestConfig(MockerTestCase):
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
     def test_empty_trusted_list(self):
         """Test that no certificate are written if 'trusted' list is empty"""
@@ -65,37 +69,37 @@ class TestConfig(MockerTestCase):
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
     def test_single_trusted(self):
         """Test that a single cert gets passed to add_ca_certs"""
         config = {"ca-certs": {"trusted": ["CERT1"]}}
 
-        self.mock_add(["CERT1"])
+        self.mock_add(self.paths, ["CERT1"])
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
     def test_multiple_trusted(self):
         """Test that multiple certs get passed to add_ca_certs"""
         config = {"ca-certs": {"trusted": ["CERT1", "CERT2"]}}
 
-        self.mock_add(["CERT1", "CERT2"])
+        self.mock_add(self.paths, ["CERT1", "CERT2"])
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
     def test_remove_default_ca_certs(self):
         """Test remove_defaults works as expected"""
         config = {"ca-certs": {"remove-defaults": True}}
 
-        self.mock_remove()
+        self.mock_remove(self.paths)
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
     def test_no_remove_defaults_if_false(self):
         """Test remove_defaults is not called when config value is False"""
@@ -104,72 +108,85 @@ class TestConfig(MockerTestCase):
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
     def test_correct_order_for_remove_then_add(self):
         """Test remove_defaults is not called when config value is False"""
         config = {"ca-certs": {"remove-defaults": True, "trusted": ["CERT1"]}}
 
-        self.mock_remove()
-        self.mock_add(["CERT1"])
+        self.mock_remove(self.paths)
+        self.mock_add(self.paths, ["CERT1"])
         self.mock_update()
         self.mocker.replay()
 
-        handle(self.name, config, self.cloud_init, self.log, self.args)
+        cc_ca_certs.handle(self.name, config, self.cloud, self.log, self.args)
 
 
 class TestAddCaCerts(MockerTestCase):
+
+    def setUp(self):
+        super(TestAddCaCerts, self).setUp()
+        self.paths = helpers.Paths({
+            'cloud_dir': self.makeDir()
+        })
+
     def test_no_certs_in_list(self):
         """Test that no certificate are written if not provided."""
-        self.mocker.replace(write_file, passthrough=False)
+        self.mocker.replace(util.write_file, passthrough=False)
         self.mocker.replay()
-
-        add_ca_certs([])
+        cc_ca_certs.add_ca_certs(self.paths, [])
 
     def test_single_cert(self):
         """Test adding a single certificate to the trusted CAs"""
         cert = "CERT1\nLINE2\nLINE3"
 
-        mock_write = self.mocker.replace(write_file, passthrough=False)
+        mock_write = self.mocker.replace(util.write_file, passthrough=False)
         mock_write("/usr/share/ca-certificates/cloud-init-ca-certs.crt",
                    cert, mode=0644)
         mock_write("/etc/ca-certificates.conf",
-                   "\ncloud-init-ca-certs.crt", omode="a")
+                   "\ncloud-init-ca-certs.crt", omode="ab")
         self.mocker.replay()
 
-        add_ca_certs([cert])
+        cc_ca_certs.add_ca_certs(self.paths, [cert])
 
     def test_multiple_certs(self):
         """Test adding multiple certificates to the trusted CAs"""
         certs = ["CERT1\nLINE2\nLINE3", "CERT2\nLINE2\nLINE3"]
         expected_cert_file = "\n".join(certs)
 
-        mock_write = self.mocker.replace(write_file, passthrough=False)
+        mock_write = self.mocker.replace(util.write_file, passthrough=False)
         mock_write("/usr/share/ca-certificates/cloud-init-ca-certs.crt",
                    expected_cert_file, mode=0644)
         mock_write("/etc/ca-certificates.conf",
-                   "\ncloud-init-ca-certs.crt", omode="a")
+                   "\ncloud-init-ca-certs.crt", omode="ab")
         self.mocker.replay()
 
-        add_ca_certs(certs)
+        cc_ca_certs.add_ca_certs(self.paths, certs)
 
 
 class TestUpdateCaCerts(MockerTestCase):
     def test_commands(self):
-        mock_check_call = self.mocker.replace("subprocess.check_call",
+        mock_check_call = self.mocker.replace(util.subp,
                                               passthrough=False)
-        mock_check_call(["update-ca-certificates"])
+        mock_check_call(["update-ca-certificates"], capture=False)
         self.mocker.replay()
 
-        update_ca_certs()
+        cc_ca_certs.update_ca_certs()
 
 
 class TestRemoveDefaultCaCerts(MockerTestCase):
+
+    def setUp(self):
+        super(TestRemoveDefaultCaCerts, self).setUp()
+        self.paths = helpers.Paths({
+            'cloud_dir': self.makeDir()
+        })
+
     def test_commands(self):
-        mock_delete_dir_contents = self.mocker.replace(delete_dir_contents,
+        mock_delete_dir_contents = self.mocker.replace(util.delete_dir_contents,
                                                        passthrough=False)
-        mock_write = self.mocker.replace(write_file, passthrough=False)
-        mock_subp = self.mocker.replace("cloudinit.util.subp",
+        mock_write = self.mocker.replace(util.write_file, passthrough=False)
+        mock_subp = self.mocker.replace(util.subp,
                                         passthrough=False)
 
         mock_delete_dir_contents("/usr/share/ca-certificates/")
@@ -179,4 +196,4 @@ class TestRemoveDefaultCaCerts(MockerTestCase):
                   "ca-certificates ca-certificates/trust_new_crts select no")
         self.mocker.replay()
 
-        remove_default_ca_certs()
+        cc_ca_certs.remove_default_ca_certs(self.paths)
diff --git a/tests/unittests/test_userdata.py b/tests/unittests/test_userdata.py
index 8eb7b259..861642b6 100644
--- a/tests/unittests/test_userdata.py
+++ b/tests/unittests/test_userdata.py
@@ -1,107 +1,144 @@
 """Tests for handling of userdata within cloud init"""
 
-import logging
 import StringIO
 
+import logging
+import os
+import shutil
+import tempfile
+
 from email.mime.base import MIMEBase
 
 from mocker import MockerTestCase
 
-import cloudinit
-from cloudinit.DataSource import DataSource
-
+from cloudinit import helpers
+from cloudinit import log
+from cloudinit import sources
+from cloudinit import stages
+from cloudinit import util
 
-instance_id = "i-testing"
+INSTANCE_ID = "i-testing"
 
 
-class FakeDataSource(DataSource):
+class FakeDataSource(sources.DataSource):
 
     def __init__(self, userdata):
-        DataSource.__init__(self)
-        self.metadata = {'instance-id': instance_id}
+        sources.DataSource.__init__(self, {}, None, None)
+        self.metadata = {'instance-id': INSTANCE_ID}
         self.userdata_raw = userdata
 
 
-class TestConsumeUserData(MockerTestCase):
+# FIXME: these tests shouldn't be checking log output??
+# Weirddddd...
+
 
-    _log_handler = None
-    _log = None
-    log_file = None
+class TestConsumeUserData(MockerTestCase):
 
     def setUp(self):
+        MockerTestCase.setUp(self)
+        # Replace the write so no actual files
+        # get written out...
         self.mock_write = self.mocker.replace("cloudinit.util.write_file",
             passthrough=False)
-        self.mock_write(self.get_ipath("cloud_config"), "", 0600)
-        self.capture_log()
+        self._log = None
+        self._log_file = None
+        self._log_handler = None
 
     def tearDown(self):
-        self._log.removeHandler(self._log_handler)
-
-    @staticmethod
-    def get_ipath(name):
-        return "%s/instances/%s%s" % (cloudinit.varlibdir, instance_id,
-            cloudinit.pathmap[name])
-
-    def capture_log(self):
-        self.log_file = StringIO.StringIO()
-        self._log_handler = logging.StreamHandler(self.log_file)
-        self._log_handler.setLevel(logging.DEBUG)
-        self._log = logging.getLogger(cloudinit.logger_name)
+        MockerTestCase.tearDown(self)
+        if self._log_handler and self._log:
+            self._log.removeHandler(self._log_handler)
+
+    def capture_log(self, lvl=logging.DEBUG):
+        log_file = StringIO.StringIO()
+        self._log_handler = logging.StreamHandler(log_file)
+        self._log_handler.setLevel(lvl)
+        self._log = log.getLogger()
         self._log.addHandler(self._log_handler)
+        return log_file
 
     def test_unhandled_type_warning(self):
         """Raw text without magic is ignored but shows warning"""
+        ci = stages.Init()
+        data = "arbitrary text\n"
+        ci.datasource = FakeDataSource(data)
+
+        self.mock_write(ci.paths.get_ipath("cloud_config"), "", 0600)
         self.mocker.replay()
-        ci = cloudinit.CloudInit()
-        ci.datasource = FakeDataSource("arbitrary text\n")
+
+        log_file = self.capture_log(logging.WARNING)
+        ci.fetch()
         ci.consume_userdata()
-        self.assertEqual(
-            "Unhandled non-multipart userdata starting 'arbitrary text...'\n",
-            self.log_file.getvalue())
+        self.assertIn(
+            "Unhandled non-multipart (text/x-not-multipart) userdata:",
+            log_file.getvalue())
 
     def test_mime_text_plain(self):
-        """Mime message of type text/plain is ignored without warning"""
-        self.mocker.replay()
-        ci = cloudinit.CloudInit()
+        """Mime message of type text/plain is ignored but shows warning"""
+        ci = stages.Init()
         message = MIMEBase("text", "plain")
         message.set_payload("Just text")
         ci.datasource = FakeDataSource(message.as_string())
+
+        self.mock_write(ci.paths.get_ipath("cloud_config"), "", 0600)
+        self.mocker.replay()
+
+        log_file = self.capture_log(logging.WARNING)
+        ci.fetch()
         ci.consume_userdata()
-        self.assertEqual("", self.log_file.getvalue())
+        self.assertIn(
+            "Unhandled unknown content-type (text/plain)",
+            log_file.getvalue())
 
+    
     def test_shellscript(self):
         """Raw text starting #!/bin/sh is treated as script"""
+        ci = stages.Init()
         script = "#!/bin/sh\necho hello\n"
-        outpath = cloudinit.get_ipath_cur("scripts") + "/part-001"
+        ci.datasource = FakeDataSource(script)
+
+        outpath = os.path.join(ci.paths.get_ipath_cur("scripts"), "part-001")
+        self.mock_write(ci.paths.get_ipath("cloud_config"), "", 0600)
         self.mock_write(outpath, script, 0700)
         self.mocker.replay()
-        ci = cloudinit.CloudInit()
-        ci.datasource = FakeDataSource(script)
+
+        log_file = self.capture_log(logging.WARNING)
+        ci.fetch()
         ci.consume_userdata()
-        self.assertEqual("", self.log_file.getvalue())
+        self.assertEqual("", log_file.getvalue())
 
     def test_mime_text_x_shellscript(self):
         """Mime message of type text/x-shellscript is treated as script"""
+        ci = stages.Init()
         script = "#!/bin/sh\necho hello\n"
-        outpath = cloudinit.get_ipath_cur("scripts") + "/part-001"
-        self.mock_write(outpath, script, 0700)
-        self.mocker.replay()
-        ci = cloudinit.CloudInit()
         message = MIMEBase("text", "x-shellscript")
         message.set_payload(script)
         ci.datasource = FakeDataSource(message.as_string())
+
+        outpath = os.path.join(ci.paths.get_ipath_cur("scripts"), "part-001")
+        self.mock_write(ci.paths.get_ipath("cloud_config"), "", 0600)
+        self.mock_write(outpath, script, 0700)
+        self.mocker.replay()
+
+        log_file = self.capture_log(logging.WARNING)
+        ci.fetch()
         ci.consume_userdata()
-        self.assertEqual("", self.log_file.getvalue())
+        self.assertEqual("", log_file.getvalue())
 
     def test_mime_text_plain_shell(self):
         """Mime type text/plain starting #!/bin/sh is treated as script"""
+        ci = stages.Init()
         script = "#!/bin/sh\necho hello\n"
-        outpath = cloudinit.get_ipath_cur("scripts") + "/part-001"
-        self.mock_write(outpath, script, 0700)
-        self.mocker.replay()
-        ci = cloudinit.CloudInit()
         message = MIMEBase("text", "plain")
         message.set_payload(script)
         ci.datasource = FakeDataSource(message.as_string())
+
+        outpath = os.path.join(ci.paths.get_ipath_cur("scripts"), "part-001")
+        self.mock_write(outpath, script, 0700)
+        self.mock_write(ci.paths.get_ipath("cloud_config"), "", 0600)
+        self.mocker.replay()
+
+        log_file = self.capture_log(logging.WARNING)
+        ci.fetch()
         ci.consume_userdata()
-        self.assertEqual("", self.log_file.getvalue())
+        self.assertEqual("", log_file.getvalue())
diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
index e8f5885c..93979f06 100644
--- a/tests/unittests/test_util.py
+++ b/tests/unittests/test_util.py
@@ -1,28 +1,45 @@
-from unittest import TestCase
-from mocker import MockerTestCase
-from tempfile import mkdtemp
-from shutil import rmtree
 import os
 import stat
 
-from cloudinit.util import (mergedict, get_cfg_option_list_or_str, write_file,
-                            delete_dir_contents, get_cmdline,
-                            keyval_str_to_dict)
+from unittest import TestCase
+from mocker import MockerTestCase
+
+from cloudinit import util
+from cloudinit import importer
+
+
+class FakeSelinux(object):
+
+    def __init__(self, match_what):
+        self.match_what = match_what
+        self.restored = []
+
+    def matchpathcon(self, path, mode):
+        if path == self.match_what:
+            return
+        else:
+            raise OSError("No match!")
 
+    def is_selinux_enabled(self):
+        return True
 
-class TestMergeDict(TestCase):
+    def restorecon(self, path, recursive):
+        self.restored.append(path)
+
+
+class TestMergeDict(MockerTestCase):
     def test_simple_merge(self):
         """Test simple non-conflict merge."""
         source = {"key1": "value1"}
         candidate = {"key2": "value2"}
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual({"key1": "value1", "key2": "value2"}, result)
 
     def test_nested_merge(self):
         """Test nested merge."""
         source = {"key1": {"key1.1": "value1.1"}}
         candidate = {"key1": {"key1.2": "value1.2"}}
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(
             {"key1": {"key1.1": "value1.1", "key1.2": "value1.2"}}, result)
 
@@ -30,42 +47,42 @@ class TestMergeDict(TestCase):
         """Test that candidate doesn't override source."""
         source = {"key1": "value1", "key2": "value2"}
         candidate = {"key1": "value2", "key2": "NEW VALUE"}
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(source, result)
 
     def test_empty_candidate(self):
         """Test empty candidate doesn't change source."""
         source = {"key": "value"}
         candidate = {}
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(source, result)
 
     def test_empty_source(self):
         """Test empty source is replaced by candidate."""
         source = {}
         candidate = {"key": "value"}
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(candidate, result)
 
     def test_non_dict_candidate(self):
         """Test non-dict candidate is discarded."""
         source = {"key": "value"}
         candidate = "not a dict"
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(source, result)
 
     def test_non_dict_source(self):
         """Test non-dict source is not modified with a dict candidate."""
         source = "not a dict"
         candidate = {"key": "value"}
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(source, result)
 
     def test_neither_dict(self):
         """Test if neither candidate or source is dict source wins."""
         source = "source"
         candidate = "candidate"
-        result = mergedict(source, candidate)
+        result = util.mergedict(source, candidate)
         self.assertEqual(source, result)
 
 
@@ -73,51 +90,45 @@ class TestGetCfgOptionListOrStr(TestCase):
     def test_not_found_no_default(self):
         """None is returned if key is not found and no default given."""
         config = {}
-        result = get_cfg_option_list_or_str(config, "key")
-        self.assertIsNone(result)
+        result = util.get_cfg_option_list(config, "key")
+        self.assertEqual(None, result)
 
     def test_not_found_with_default(self):
         """Default is returned if key is not found."""
         config = {}
-        result = get_cfg_option_list_or_str(config, "key", default=["DEFAULT"])
+        result = util.get_cfg_option_list(config, "key", default=["DEFAULT"])
         self.assertEqual(["DEFAULT"], result)
 
     def test_found_with_default(self):
         """Default is not returned if key is found."""
         config = {"key": ["value1"]}
-        result = get_cfg_option_list_or_str(config, "key", default=["DEFAULT"])
+        result = util.get_cfg_option_list(config, "key", default=["DEFAULT"])
         self.assertEqual(["value1"], result)
 
     def test_found_convert_to_list(self):
         """Single string is converted to one element list."""
         config = {"key": "value1"}
-        result = get_cfg_option_list_or_str(config, "key")
+        result = util.get_cfg_option_list(config, "key")
         self.assertEqual(["value1"], result)
 
     def test_value_is_none(self):
         """If value is None empty list is returned."""
         config = {"key": None}
-        result = get_cfg_option_list_or_str(config, "key")
+        result = util.get_cfg_option_list(config, "key")
         self.assertEqual([], result)
 
 
 class TestWriteFile(MockerTestCase):
     def setUp(self):
         super(TestWriteFile, self).setUp()
-        # Make a temp directoy for tests to use.
-        self.tmp = mkdtemp(prefix="unittest_")
-
-    def tearDown(self):
-        super(TestWriteFile, self).tearDown()
-        # Clean up temp directory
-        rmtree(self.tmp)
+        self.tmp = self.makeDir(prefix="unittest_")
 
     def test_basic_usage(self):
         """Verify basic usage with default args."""
         path = os.path.join(self.tmp, "NewFile.txt")
         contents = "Hey there"
 
-        write_file(path, contents)
+        util.write_file(path, contents)
 
         self.assertTrue(os.path.exists(path))
         self.assertTrue(os.path.isfile(path))
@@ -133,7 +144,7 @@ class TestWriteFile(MockerTestCase):
         path = os.path.join(dirname, "NewFile.txt")
         contents = "Hey there"
 
-        write_file(path, contents)
+        util.write_file(path, contents)
 
         self.assertTrue(os.path.isdir(dirname))
         self.assertTrue(os.path.isfile(path))
@@ -143,7 +154,7 @@ class TestWriteFile(MockerTestCase):
         path = os.path.join(self.tmp, "NewFile.txt")
         contents = "Hey there"
 
-        write_file(path, contents, mode=0666)
+        util.write_file(path, contents, mode=0666)
 
         self.assertTrue(os.path.exists(path))
         self.assertTrue(os.path.isfile(path))
@@ -158,7 +169,7 @@ class TestWriteFile(MockerTestCase):
         # Create file first with basic content
         with open(path, "wb") as f:
             f.write("LINE1\n")
-        write_file(path, contents, omode="a")
+        util.write_file(path, contents, omode="a")
 
         self.assertTrue(os.path.exists(path))
         self.assertTrue(os.path.isfile(path))
@@ -167,36 +178,30 @@ class TestWriteFile(MockerTestCase):
             self.assertEqual("LINE1\nHey there", create_contents)
 
     def test_restorecon_if_possible_is_called(self):
-        """Make sure the restorecon_if_possible is called correctly."""
-        path = os.path.join(self.tmp, "NewFile.txt")
-        contents = "Hey there"
-
-        # Mock out the restorecon_if_possible call to test if it's called.
-        mock_restorecon = self.mocker.replace(
-            "cloudinit.util.restorecon_if_possible", passthrough=False)
-        mock_restorecon(path)
+        """Make sure the selinux guard is called correctly."""
+        import_mock = self.mocker.replace(importer.import_module,
+                                          passthrough=False)
+        import_mock('selinux')
+        fake_se = FakeSelinux('/etc/hosts')
+        self.mocker.result(fake_se)
         self.mocker.replay()
+        with util.SeLinuxGuard("/etc/hosts") as is_on:
+            self.assertTrue(is_on)
+        self.assertEqual(1, len(fake_se.restored))
+        self.assertEqual('/etc/hosts', fake_se.restored[0])
 
-        write_file(path, contents)
 
-
-class TestDeleteDirContents(TestCase):
+class TestDeleteDirContents(MockerTestCase):
     def setUp(self):
         super(TestDeleteDirContents, self).setUp()
-        # Make a temp directoy for tests to use.
-        self.tmp = mkdtemp(prefix="unittest_")
-
-    def tearDown(self):
-        super(TestDeleteDirContents, self).tearDown()
-        # Clean up temp directory
-        rmtree(self.tmp)
+        self.tmp = self.makeDir(prefix="unittest_")
 
     def assertDirEmpty(self, dirname):
         self.assertEqual([], os.listdir(dirname))
 
     def test_does_not_delete_dir(self):
         """Ensure directory itself is not deleted."""
-        delete_dir_contents(self.tmp)
+        util.delete_dir_contents(self.tmp)
 
         self.assertTrue(os.path.isdir(self.tmp))
         self.assertDirEmpty(self.tmp)
@@ -206,7 +211,7 @@ class TestDeleteDirContents(TestCase):
         with open(os.path.join(self.tmp, "new_file.txt"), "wb") as f:
             f.write("DELETE ME")
 
-        delete_dir_contents(self.tmp)
+        util.delete_dir_contents(self.tmp)
 
         self.assertDirEmpty(self.tmp)
 
@@ -214,7 +219,7 @@ class TestDeleteDirContents(TestCase):
         """Empty directories should be deleted."""
         os.mkdir(os.path.join(self.tmp, "new_dir"))
 
-        delete_dir_contents(self.tmp)
+        util.delete_dir_contents(self.tmp)
 
         self.assertDirEmpty(self.tmp)
 
@@ -223,7 +228,7 @@ class TestDeleteDirContents(TestCase):
         os.mkdir(os.path.join(self.tmp, "new_dir"))
         os.mkdir(os.path.join(self.tmp, "new_dir", "new_subdir"))
 
-        delete_dir_contents(self.tmp)
+        util.delete_dir_contents(self.tmp)
 
         self.assertDirEmpty(self.tmp)
 
@@ -234,7 +239,7 @@ class TestDeleteDirContents(TestCase):
         with open(f_name, "wb") as f:
             f.write("DELETE ME")
 
-        delete_dir_contents(self.tmp)
+        util.delete_dir_contents(self.tmp)
 
         self.assertDirEmpty(self.tmp)
 
@@ -246,7 +251,7 @@ class TestDeleteDirContents(TestCase):
             f.write("DELETE ME")
         os.symlink(file_name, link_name)
 
-        delete_dir_contents(self.tmp)
+        util.delete_dir_contents(self.tmp)
 
         self.assertDirEmpty(self.tmp)
 
@@ -255,12 +260,12 @@ class TestKeyValStrings(TestCase):
     def test_keyval_str_to_dict(self):
         expected = {'1': 'one', '2': 'one+one', 'ro': True}
         cmdline = "1=one ro 2=one+one"
-        self.assertEqual(expected, keyval_str_to_dict(cmdline))
+        self.assertEqual(expected, util.keyval_str_to_dict(cmdline))
 
 
 class TestGetCmdline(TestCase):
     def test_cmdline_reads_debug_env(self):
         os.environ['DEBUG_PROC_CMDLINE'] = 'abcd 123'
-        self.assertEqual(os.environ['DEBUG_PROC_CMDLINE'], get_cmdline())
+        self.assertEqual(os.environ['DEBUG_PROC_CMDLINE'], util.get_cmdline())
 
 # vi: ts=4 expandtab
diff --git a/tools/bddeb b/tools/bddeb
deleted file mode 100755
index 598f71bb..00000000
--- a/tools/bddeb
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/bin/sh
-
-TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXXX")
-#TEMP_D=/tmp/my.d
-start=${PWD}
-rm -Rf "${TEMP_D}"; mkdir "${TEMP_D}"
-set -e
-trap "rm -Rf '${TEMP_D}'" exit
-files=$(bzr ls --versioned)
-revno=$(bzr revno)
-version=$(awk \
-  -F= '$1 ~ /version$/ { gsub("[^0-9.]","",$2); print $2; }' setup.py)
-mkdir "${TEMP_D}/cloud-init"
-otar="$TEMP_D/cloud-init_$version~bzr${revno}.orig.tar.gz"
-tar -czf - ${files} > "$otar"
-tar -C "${TEMP_D}/cloud-init" -xzf - <"$otar"
-
-if [ ! -d "${TEMP_D}/cloud-init/debian" ]; then
-   rsync -a debian.trunk/ "${TEMP_D}/cloud-init/debian"
-fi
-sed -i -e "s,VERSION,$version," -e "s,REVNO,bzr$revno," \
-   "$TEMP_D/cloud-init/debian/changelog"
-cd "${TEMP_D}/cloud-init"
-debuild "$@"
-#for x in ../*.deb; do
-#   echo wrote ${x##*/}
-#done
-debname="cloud-init_${version}~bzr${revno}-0_all.deb"
-mv "../$debname" "$start"
-link="$start/cloud-init_all.deb"
-echo "wrote $debname"
-[ ! -e "$link" -o -L "$link" ] 
-   { ln -sf "$debname" "$link" && echo "linked ${link##*/}"; }
diff --git a/tools/hacking.py b/tools/hacking.py
new file mode 100755
index 00000000..d0c27d25
--- /dev/null
+++ b/tools/hacking.py
@@ -0,0 +1,175 @@
+#!/usr/bin/env python
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2012, Cloudscaling
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""cloudinit HACKING file compliance testing (based off of nova hacking.py)
+
+built on top of pep8.py
+"""
+
+import inspect
+import logging
+import os
+import re
+import sys
+import tokenize
+import warnings
+
+import pep8
+
+# Don't need this for testing
+logging.disable('LOG')
+
+# N1xx comments
+# N2xx except
+# N3xx imports
+# N4xx docstrings
+# N[5-9]XX (future use)
+
+DOCSTRING_TRIPLE = ['"""', "'''"]
+VERBOSE_MISSING_IMPORT = False
+_missingImport = set([])
+
+
+def import_normalize(line):
+    # convert "from x import y" to "import x.y"
+    # handle "from x import y as z" to "import x.y as z"
+    split_line = line.split()
+    if (line.startswith("from ") and "," not in line and
+           split_line[2] == "import" and split_line[3] != "*" and
+           split_line[1] != "__future__" and
+           (len(split_line) == 4 or
+           (len(split_line) == 6 and split_line[4] == "as"))):
+        return "import %s.%s" % (split_line[1], split_line[3])
+    else:
+        return line
+
+
+def cloud_import_alphabetical(physical_line, line_number, lines):
+    """Check for imports in alphabetical order.
+
+    HACKING guide recommendation for imports:
+    imports in human alphabetical order
+    N306
+    """
+    # handle import x
+    # use .lower since capitalization shouldn't dictate order
+    split_line = import_normalize(physical_line.strip()).lower().split()
+    split_previous = import_normalize(lines[line_number - 2]
+            ).strip().lower().split()
+    # with or without "as y"
+    length = [2, 4]
+    if (len(split_line) in length and len(split_previous) in length and
+        split_line[0] == "import" and split_previous[0] == "import"):
+        if split_line[1] < split_previous[1]:
+            return (0, "N306: imports not in alphabetical order (%s, %s)"
+                % (split_previous[1], split_line[1]))
+
+
+def cloud_docstring_start_space(physical_line):
+    """Check for docstring not start with space.
+
+    HACKING guide recommendation for docstring:
+    Docstring should not start with space
+    N401
+    """
+    pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE])  # start
+    if (pos != -1 and len(physical_line) > pos + 1):
+        if (physical_line[pos + 3] == ' '):
+            return (pos, "N401: one line docstring should not start with"
+                " a space")
+
+
+def cloud_todo_format(physical_line):
+    """Check for 'TODO()'.
+
+    HACKING guide recommendation for TODO:
+    Include your name with TODOs as in "#TODO(termie)"
+    N101
+    """
+    pos = physical_line.find('TODO')
+    pos1 = physical_line.find('TODO(')
+    pos2 = physical_line.find('#') # make sure it's a comment
+    if (pos != pos1 and pos2 >= 0 and pos2 < pos):
+        return pos, "N101: Use TODO(NAME)"
+
+
+def cloud_docstring_one_line(physical_line):
+    """Check one line docstring end.
+
+    HACKING guide recommendation for one line docstring:
+    A one line docstring looks like this and ends in a period.
+    N402
+    """
+    pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE])  # start
+    end = max([physical_line[-4:-1] == i for i in DOCSTRING_TRIPLE])  # end
+    if (pos != -1 and end and len(physical_line) > pos + 4):
+        if (physical_line[-5] != '.'):
+            return pos, "N402: one line docstring needs a period"
+
+
+def cloud_docstring_multiline_end(physical_line):
+    """Check multi line docstring end.
+
+    HACKING guide recommendation for docstring:
+    Docstring should end on a new line
+    N403
+    """
+    pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE])  # start
+    if (pos != -1 and len(physical_line) == pos):
+        print physical_line
+        if (physical_line[pos + 3] == ' '):
+            return (pos, "N403: multi line docstring end on new line")
+
+
+
+current_file = ""
+
+
+def readlines(filename):
+    """Record the current file being tested."""
+    pep8.current_file = filename
+    return open(filename).readlines()
+
+
+def add_cloud():
+    """Monkey patch pep8 for cloud-init guidelines.
+
+    Look for functions that start with cloud_
+    and add them to pep8 module.
+
+    Assumes you know how to write pep8.py checks
+    """
+    for name, function in globals().items():
+        if not inspect.isfunction(function):
+            continue
+        if name.startswith("cloud_"):
+            exec("pep8.%s = %s" % (name, name))
+
+if __name__ == "__main__":
+    # NOVA based 'hacking.py' error codes start with an N
+    pep8.ERRORCODE_REGEX = re.compile(r'[EWN]\d{3}')
+    add_cloud()
+    pep8.current_file = current_file
+    pep8.readlines = readlines
+    try:
+        pep8._main()
+    finally:
+        if len(_missingImport) > 0:
+            print >> sys.stderr, ("%i imports missing in this test environment"
+                    % len(_missingImport))
+
diff --git a/tools/mock-meta.py b/tools/mock-meta.py
new file mode 100755
index 00000000..4548e4ae
--- /dev/null
+++ b/tools/mock-meta.py
@@ -0,0 +1,444 @@
+#!/usr/bin/python
+
+# Provides a somewhat random, somewhat compat, somewhat useful mock version of
+#
+# http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/AESDG-chapter-instancedata.html
+
+"""
+To use this to mimic the EC2 metadata service entirely, run it like:
+  # Where 'eth0' is *some* interface.  
+  sudo ifconfig eth0:0 169.254.169.254 netmask 255.255.255.255
+
+  sudo ./mock-meta -a 169.254.169.254 -p 80
+
+Then:
+  wget -q http://169.254.169.254/latest/meta-data/instance-id -O -; echo
+  curl --silent http://169.254.169.254/latest/meta-data/instance-id ; echo
+  ec2metadata --instance-id
+"""
+
+import functools
+import httplib
+import json
+import logging
+import os
+import random
+import string
+import sys
+import yaml
+
+from optparse import OptionParser
+
+from BaseHTTPServer import (HTTPServer, BaseHTTPRequestHandler)
+
+log = logging.getLogger('meta-server')
+
+EC2_VERSIONS = [
+    '1.0',
+    '2007-01-19',
+    '2007-03-01',
+    '2007-08-29',
+    '2007-10-10',
+    '2007-12-15',
+    '2008-02-01',
+    '2008-09-01',
+    '2009-04-04',
+]
+
+BLOCK_DEVS = [
+    'ami',
+    'ephemeral0',
+    'root',
+]
+
+DEV_PREFIX = 'v'  # This seems to vary alot depending on images...
+DEV_MAPPINGS = {
+    'ephemeral0': '%sda2' % (DEV_PREFIX),
+    'root': '/dev/%sda1' % (DEV_PREFIX),
+    'ami': '%sda1' % (DEV_PREFIX),
+    'swap': '%sda3' % (DEV_PREFIX),
+}
+
+META_CAPABILITIES = [
+    'aki-id',
+    'ami-id',
+    'ami-launch-index',
+    'ami-manifest-path',
+    'ari-id',
+    'block-device-mapping/',
+    'hostname',
+    'instance-action',
+    'instance-id',
+    'instance-type',
+    'local-hostname',
+    'local-ipv4',
+    'placement/',
+    'product-codes',
+    'public-hostname',
+    'public-ipv4',
+    'public-keys/',
+    'reservation-id',
+    'security-groups'
+]
+
+PUB_KEYS = {
+    'brickies': [
+        ('ssh-rsa '
+         'AAAAB3NzaC1yc2EAAAABIwAAAQEA3I7VUf2l5gSn5uavROsc5HRDpZdQueUq5ozemNSj8T'
+         '7enqKHOEaFoU2VoPgGEWC9RyzSQVeyD6s7APMcE82EtmW4skVEgEGSbDc1pvxzxtchBj78'
+         'hJP6Cf5TCMFSXw+Fz5rF1dR23QDbN1mkHs7adr8GW4kSWqU7Q7NDwfIrJJtO7Hi42GyXtv'
+         'EONHbiRPOe8stqUly7MvUoN+5kfjBM8Qqpfl2+FNhTYWpMfYdPUnE7u536WqzFmsaqJctz'
+         '3gBxH9Ex7dFtrxR4qiqEr9Qtlu3xGn7Bw07/+i1D+ey3ONkZLN+LQ714cgj8fRS4Hj29SC'
+         'mXp5Kt5/82cD/VN3NtHw== brickies'),
+        '',
+    ],
+}
+
+INSTANCE_TYPES = [
+    'm1.large',
+    'm1.medium',
+    'm1.small',
+    'm1.xlarge',
+]
+
+AVAILABILITY_ZONES = [
+    "us-east-1a",
+    "us-east-1b",
+    "us-east-1c",
+    "us-east-1d",
+    'eu-west-1a',
+    'eu-west-1b',
+    'us-west-1',
+]
+
+PLACEMENT_CAPABILITIES = {
+    'availability-zone': AVAILABILITY_ZONES,
+}
+
+NOT_IMPL_RESPONSE = json.dumps({})
+
+
+class WebException(Exception):
+    def __init__(self, code, msg):
+        Exception.__init__(self, msg)
+        self.code = code
+
+
+def yamlify(data):
+    formatted = yaml.dump(data,
+        line_break="\n",
+        indent=4,
+        explicit_start=True,
+        explicit_end=True,
+        default_flow_style=False)
+    return formatted
+
+
+def format_text(text):
+    if not len(text):
+        return "<<"
+    lines = text.splitlines()
+    nlines = []
+    for line in lines:
+        nlines.append("<< %s" % line)
+    return "\n".join(nlines)
+
+
+def traverse(keys, mp):
+    result = dict(mp)
+    for k in keys:
+        try:
+            result = result.get(k)
+        except (AttributeError, TypeError):
+            result = None
+            break
+    return result
+
+
+ID_CHARS = [c for c in (string.ascii_uppercase + string.digits)]
+def id_generator(size=6, lower=False):
+    txt = ''.join(random.choice(ID_CHARS) for x in range(size))
+    if lower:
+        return txt.lower()
+    else:
+        return txt
+
+
+def get_ssh_keys():
+    keys = {}
+    keys.update(PUB_KEYS)
+
+    # Nice helper to add in the 'running' users key (if they have one)
+    key_pth = os.path.expanduser('~/.ssh/id_rsa.pub')
+    if not os.path.isfile(key_pth):
+        key_pth = os.path.expanduser('~/.ssh/id_dsa.pub')
+
+    if os.path.isfile(key_pth):
+        with open(key_pth, 'rb') as fh:
+            contents = fh.read()
+        keys[os.getlogin()] = [contents, '']
+
+    return keys
+
+
+class MetaDataHandler(object):
+
+    def __init__(self, opts):
+        self.opts = opts
+        self.instances = {}
+
+    def get_data(self, params, who, **kwargs):
+        if not params:
+            # Show the root level capabilities when
+            # no params are passed...
+            caps = sorted(META_CAPABILITIES)
+            return "\n".join(caps)
+        action = params[0]
+        action = action.lower()
+        if action == 'instance-id':
+            return 'i-%s' % (id_generator(lower=True))
+        elif action == 'ami-launch-index':
+            return "%s" % random.choice([0, 1, 2, 3])
+        elif action == 'aki-id':
+            return 'aki-%s' % (id_generator(lower=True))
+        elif action == 'ami-id':
+            return 'ami-%s' % (id_generator(lower=True))
+        elif action == 'ari-id':
+            return 'ari-%s' % (id_generator(lower=True))
+        elif action == 'block-device-mapping':
+            nparams = params[1:]
+            if not nparams:
+                return "\n".join(BLOCK_DEVS)
+            else:
+                subvalue = traverse(nparams, DEV_MAPPINGS)
+                if not subvalue:
+                    return "\n".join(sorted(list(DEV_MAPPINGS.keys())))
+                else:
+                    return str(subvalue)
+        elif action in ['hostname', 'local-hostname', 'public-hostname']:
+            # Just echo back there own hostname that they called in on..
+            return "%s" % (who)
+        elif action == 'instance-type':
+            return random.choice(INSTANCE_TYPES)
+        elif action == 'ami-manifest-path':
+            return 'my-amis/spamd-image.manifest.xml'
+        elif action == 'security-groups':
+            return 'default'
+        elif action in ['local-ipv4', 'public-ipv4']:
+            # Just echo back there own ip that they called in on...
+            return "%s" % (kwargs.get('client_ip', '10.0.0.1'))
+        elif action == 'reservation-id':
+            return "r-%s" % (id_generator(lower=True))
+        elif action == 'product-codes':
+            return "%s" % (id_generator(size=8))
+        elif action == 'public-keys':
+            nparams = params[1:]
+            # This is a weird kludge, why amazon why!!!
+            # public-keys is messed up, a list of /latest/meta-data/public-keys/
+	        # shows something like: '0=brickies'
+	        # but a GET to /latest/meta-data/public-keys/0=brickies will fail
+	        # you have to know to get '/latest/meta-data/public-keys/0', then
+	        # from there you get a 'openssh-key', which you can get.
+	        # this hunk of code just re-works the object for that.
+            avail_keys = get_ssh_keys()
+            key_ids = sorted(list(avail_keys.keys()))
+            if nparams:
+                mybe_key = nparams[0]
+                try:
+                    key_id = int(mybe_key)
+                    key_name = key_ids[key_id]
+                except:
+                    raise WebException(httplib.BAD_REQUEST, "Unknown key id %r" % mybe_key)
+                # Extract the possible sub-params
+                result = traverse(nparams[1:], {
+                    "openssh-key": "\n".join(avail_keys[key_name]),
+                })
+                if isinstance(result, (dict)):
+                    # TODO: This might not be right??
+                    result = "\n".join(sorted(result.keys()))
+                if not result:
+                    result = ''
+                return result
+            else:
+                contents = []
+                for (i, key_id) in enumerate(key_ids):
+                    contents.append("%s=%s" % (i, key_id))
+                return "\n".join(contents)
+        elif action == 'placement':
+            nparams = params[1:]
+            if not nparams:
+                pcaps = sorted(PLACEMENT_CAPABILITIES.keys())
+                return "\n".join(pcaps)
+            else:
+                pentry = nparams[0].strip().lower()
+                if pentry == 'availability-zone':
+                    zones = PLACEMENT_CAPABILITIES[pentry]
+                    return "%s" % random.choice(zones)
+                else:
+                    return "%s" % (PLACEMENT_CAPABILITIES.get(pentry, ''))
+        else:
+            log.warn(("Did not implement action %s, "
+                      "returning empty response: %r"),
+                      action, NOT_IMPL_RESPONSE)
+            return NOT_IMPL_RESPONSE
+
+
+class UserDataHandler(object):
+
+    def __init__(self, opts):
+        self.opts = opts
+
+    def _get_user_blob(self, **kwargs):
+        blob = None
+        if self.opts['user_data_file'] is not None:
+            blob = self.opts['user_data_file']
+        if not blob:
+            blob_mp = {
+                'hostname': kwargs.get('who', 'localhost'),
+            }
+            lines = [
+                "#cloud-config",
+                yamlify(blob_mp),
+            ]
+            blob = "\n".join(lines)
+        return blob.strip()
+
+    def get_data(self, params, who, **kwargs):
+        if not params:
+            return self._get_user_blob(who=who)
+        return NOT_IMPL_RESPONSE
+
+
+# Seem to need to use globals since can't pass 
+# data into the request handlers instances...
+# Puke!
+meta_fetcher = None
+user_fetcher = None
+
+
+class Ec2Handler(BaseHTTPRequestHandler):
+
+    def _get_versions(self):
+        versions = ['latest'] + EC2_VERSIONS
+        versions = sorted(versions)
+        return "\n".join(versions)
+
+    def log_message(self, format, *args):
+        msg = "%s - %s" % (self.address_string(), format % (args))
+        log.info(msg)
+
+    def _find_method(self, path):
+        # Puke! (globals)
+        global meta_fetcher
+        global user_fetcher
+        func_mapping = {
+            'user-data': user_fetcher.get_data,
+            'meta-data': meta_fetcher.get_data,
+        }
+        segments = [piece for piece in path.split('/') if len(piece)]
+        log.info("Received segments %s", segments)
+        if not segments:
+            return self._get_versions
+        date = segments[0].strip().lower()
+        if date not in self._get_versions():
+            raise WebException(httplib.BAD_REQUEST, "Unknown version format %r" % date)
+        if len(segments) < 2:
+            raise WebException(httplib.BAD_REQUEST, "No action provided")
+        look_name = segments[1].lower()
+        if look_name not in func_mapping:
+            raise WebException(httplib.BAD_REQUEST, "Unknown requested data %r" % look_name)
+        base_func = func_mapping[look_name]
+        who = self.address_string()
+        ip_from = self.client_address[0]
+        if who == ip_from:
+            # Nothing resolved, so just use 'localhost'
+            who = 'localhost'
+        kwargs = {
+            'params': list(segments[2:]),
+            'who': who,
+            'client_ip': ip_from,
+        }
+        return functools.partial(base_func, **kwargs)
+
+    def _do_response(self):
+        who = self.client_address
+        log.info("Got a call from %s for path %s", who, self.path)
+        try:
+            func = self._find_method(self.path)
+            data = func()
+            if not data:
+                data = ''
+            self.send_response(httplib.OK)
+            self.send_header("Content-Type", "binary/octet-stream")
+            self.send_header("Content-Length", len(data))
+            log.info("Sending data (len=%s):\n%s", len(data), format_text(data))
+            self.end_headers()
+            self.wfile.write(data)
+        except RuntimeError as e:
+            log.exception("Error somewhere in the server.")
+            self.send_error(httplib.INTERNAL_SERVER_ERROR, message=str(e))
+        except WebException as e:
+            code = e.code
+            log.exception(str(e))
+            self.send_error(code, message=str(e))
+
+    def do_GET(self):
+        self._do_response()
+
+    def do_POST(self):
+        self._do_response()
+
+
+def setup_logging(log_level, format='%(levelname)s: @%(name)s : %(message)s'):
+    root_logger = logging.getLogger()
+    console_logger = logging.StreamHandler(sys.stdout)
+    console_logger.setFormatter(logging.Formatter(format))
+    root_logger.addHandler(console_logger)
+    root_logger.setLevel(log_level)
+
+
+def extract_opts():
+    parser = OptionParser()
+    parser.add_option("-p", "--port", dest="port", action="store", type=int, default=80,
+                  help="port from which to serve traffic (default: %default)", metavar="PORT")
+    parser.add_option("-a", "--addr", dest="address", action="store", type=str, default='0.0.0.0',
+                  help="address from which to serve traffic (default: %default)", metavar="ADDRESS")
+    parser.add_option("-f", '--user-data-file', dest='user_data_file', action='store',
+                      help="user data filename to serve back to incoming requests", metavar='FILE')
+    (options, args) = parser.parse_args()
+    out = dict()
+    out['extra'] = args
+    out['port'] = options.port
+    out['user_data_file'] = None
+    out['address'] = options.address
+    if options.user_data_file:
+        if not os.path.isfile(options.user_data_file):
+            parser.error("Option -f specified a non-existent file")
+        with open(options.user_data_file, 'rb') as fh:
+            out['user_data_file'] = fh.read()
+    return out
+
+
+def setup_fetchers(opts):
+    global meta_fetcher
+    global user_fetcher
+    meta_fetcher = MetaDataHandler(opts)
+    user_fetcher = UserDataHandler(opts)
+
+
+def run_server():
+    # Using global here since it doesn't seem like we 
+    # can pass opts into a request handler constructor...
+    opts = extract_opts()
+    setup_logging(logging.DEBUG)
+    setup_fetchers(opts)
+    log.info("CLI opts: %s", opts)
+    server_address = (opts['address'], opts['port'])
+    server = HTTPServer(server_address, Ec2Handler)
+    sa = server.socket.getsockname()
+    log.info("Serving ec2 metadata on %s using port %s ...", sa[0], sa[1])
+    server.serve_forever()
+
+
+if __name__ == '__main__':
+    run_server()
diff --git a/tools/read-dependencies b/tools/read-dependencies
new file mode 100755
index 00000000..72e1e095
--- /dev/null
+++ b/tools/read-dependencies
@@ -0,0 +1,45 @@
+#!/usr/bin/python
+# vi: ts=4 expandtab
+
+import os
+import sys
+import re
+
+
+def parse_requires(fn):
+    requires = []
+    with open(fn, 'r') as fh:
+        lines = fh.read().splitlines()
+    for line in lines:
+        line = line.strip()
+        if not line or line[0] == '#':
+            continue
+        else:
+            requires.append(line)
+    return requires
+
+
+def find_requires(args):
+    p_files = []
+    if args:
+        p_files.append(args[0])
+    p_files.append(os.path.join(os.pardir, "Requires"))
+    p_files.append(os.path.join(os.getcwd(), 'Requires'))
+    found = None
+    for fn in p_files:
+        if os.path.isfile(fn):
+            found = fn
+            break
+    return found
+
+
+if __name__ == '__main__':
+    run_args = sys.argv[1:]
+    fn = find_requires(run_args)
+    if not fn:
+        sys.stderr.write("'Requires' file not found!\n")
+        sys.exit(1)
+    else:
+        deps = parse_requires(fn)
+        for entry in deps:
+            print entry
diff --git a/tools/read-version b/tools/read-version
new file mode 100755
index 00000000..e6167a2c
--- /dev/null
+++ b/tools/read-version
@@ -0,0 +1,70 @@
+#!/usr/bin/python
+# vi: ts=4 expandtab
+
+import os
+import sys
+import re
+
+from distutils import version as ver
+
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(
+        sys.argv[0]), os.pardir, os.pardir))
+if os.path.exists(os.path.join(possible_topdir, "cloudinit", "__init__.py")):
+    sys.path.insert(0, possible_topdir)
+
+from cloudinit import version as cver
+
+def parse_versions(fn):
+    with open(fn, 'r') as fh:
+        lines = fh.read().splitlines()
+    versions = []
+    for line in lines:
+        line = line.strip()
+        if line.startswith("-") or not line:
+            continue
+        if not re.match(r"[\d]", line):
+            continue
+        line = line.strip(":")
+        if (re.match(r"^[\d+]\.[\d+]\.[\d+]$", line) or
+            re.match(r"^[\d+]\.[\d+]$", line)):
+            versions.append(line)
+    return versions
+
+def find_changelog(args):
+    p_files = []
+    if args:
+        p_files.append(args[0])
+    p_files.append(os.path.join(os.pardir, "ChangeLog"))
+    p_files.append(os.path.join(os.getcwd(), 'ChangeLog'))
+    found = None
+    for fn in p_files:
+        if os.path.isfile(fn):
+            found = fn
+            break
+    return found
+
+
+if __name__ == '__main__':
+    run_args = sys.argv[1:]
+    fn = find_changelog(run_args)
+    if not fn:
+        sys.stderr.write("'ChangeLog' file not found!\n")
+        sys.exit(1)
+    else:
+        versions = parse_versions(fn)
+        if not versions:
+            sys.stderr.write("No versions found in %s!\n" % (fn))
+            sys.exit(1)
+        else:
+            # Check that the code version is the same 
+            # as the version we found!
+            ch_ver = versions[0].strip()
+            code_ver = cver.version()
+            ch_ver_obj = ver.StrictVersion(ch_ver)
+            if ch_ver_obj != code_ver:
+                sys.stderr.write(("Code version %s does not match"
+                                  " changelog version %s\n") % 
+                                  (code_ver, ch_ver_obj))
+                sys.exit(1)
+            sys.stdout.write(ch_ver)
+            sys.exit(0)
diff --git a/tools/run-pep8 b/tools/run-pep8
new file mode 100755
index 00000000..ea46c117
--- /dev/null
+++ b/tools/run-pep8
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+ci_files='cloud*.py cloudinit/*.py cloudinit/config/*.py'
+test_files=$(find tests -name "*.py")
+def_files="$ci_files $test_files"
+
+if [ $# -eq 0 ]; then
+   files=( )
+   for f in $def_files; do
+      [ -f "$f" ] || { echo "failed, $f not a file" 1>&2; exit 1; }
+      files[${#files[@]}]=${f}
+   done
+else
+   files=( "$@" );
+fi
+
+if [ -f 'hacking.py' ]
+then
+    base=`pwd`
+else
+    base=`pwd`/tools/
+fi
+
+cmd=(
+    ${base}/hacking.py
+
+    --ignore=E501 # Line too long (these are caught by pylint)
+
+    "${files[@]}"
+)
+
+echo -e "\nRunning 'cloudinit' pep8:"
+echo "${cmd[@]}"
+"${cmd[@]}"
+
diff --git a/tools/run-pylint b/tools/run-pylint
index 46748ffb..dd6369aa 100755
--- a/tools/run-pylint
+++ b/tools/run-pylint
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-ci_files='cloud*.py cloudinit/*.py cloudinit/CloudConfig/*.py'
+ci_files='cloud*.py cloudinit/*.py cloudinit/config/*.py'
 test_files=$(find tests -name "*.py")
 def_files="$ci_files $test_files"
 
@@ -38,14 +38,3 @@ echo -e "\nRunning pylint:"
 echo "${cmd[@]}"
 "${cmd[@]}"
 
-cmd=(
-    pep8
-
-    --ignore=E501 # Line too long (these are caught by pylint above)
-
-    "${files[@]}"
-)
-
-echo -e "\nRunning pep8:"
-echo "${cmd[@]}"
-"${cmd[@]}"
diff --git a/upstart/cloud-config.conf b/upstart/cloud-config.conf
index 5edc58b9..3ac113f3 100644
--- a/upstart/cloud-config.conf
+++ b/upstart/cloud-config.conf
@@ -5,4 +5,4 @@ start on (filesystem and started rsyslog)
 console output
 task
 
-exec cloud-init-cfg all config
+exec cloud-init modules --mode=config
diff --git a/upstart/cloud-final.conf b/upstart/cloud-final.conf
index a04105a1..72ae5052 100644
--- a/upstart/cloud-final.conf
+++ b/upstart/cloud-final.conf
@@ -7,4 +7,4 @@ start on (stopped rc RUNLEVEL=[2345] and stopped cloud-config)
 console output
 task
 
-exec cloud-init-cfg all final
+exec cloud-init modules --mode=final
diff --git a/upstart/cloud-init-local.conf b/upstart/cloud-init-local.conf
index b6eb21b4..061fe406 100644
--- a/upstart/cloud-init-local.conf
+++ b/upstart/cloud-init-local.conf
@@ -6,4 +6,4 @@ task
 
 console output
 
-exec /usr/bin/cloud-init start-local
+exec /usr/bin/cloud-init init --local  
diff --git a/upstart/cloud-init.conf b/upstart/cloud-init.conf
index b9be5981..41ddd284 100644
--- a/upstart/cloud-init.conf
+++ b/upstart/cloud-init.conf
@@ -6,4 +6,4 @@ task
 
 console output
 
-exec /usr/bin/cloud-init start
+exec /usr/bin/cloud-init init