Cloud-init: T2117: Updated to 20.1

  - Merge 20.1 version from the Canonical repository
  - Removed unneeded changes in datasources (now only OVF datasource is not equal to upstream's version)
  - Adapted cc_vyos module to new Cloud-init version
  - Changed Jenkinsfile to use build scripts, provided by upstream

38 files changed, 2170 insertions, 744 deletions

diff --git a/doc/rtd/conf.py b/doc/rtd/conf.py
index 50eb05cf..86441986 100644
--- a/doc/rtd/conf.py
+++ b/doc/rtd/conf.py
@@ -17,7 +17,8 @@ from cloudinit.config.schema import get_schema_doc
 # ]
 
 # General information about the project.
-project = 'Cloud-Init'
+project = 'cloud-init'
+copyright = '2019, Canonical Ltd.'
 
 # -- General configuration ----------------------------------------------------
 
@@ -27,16 +28,12 @@ project = 'Cloud-Init'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
-    'sphinx.ext.intersphinx',
+    'm2r',
     'sphinx.ext.autodoc',
     'sphinx.ext.autosectionlabel',
     'sphinx.ext.viewcode',
 ]
 
-intersphinx_mapping = {
-    'sphinx': ('http://sphinx.pocoo.org', None)
-}
-
 # The suffix of source filenames.
 source_suffix = '.rst'
 
@@ -64,15 +61,7 @@ show_authors = False
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'default'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-html_theme_options = {
-    "bodyfont": "Ubuntu, Arial, sans-serif",
-    "headfont": "Ubuntu, Arial, sans-serif"
-}
+html_theme = 'sphinx_rtd_theme'
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
diff --git a/doc/rtd/index.rst b/doc/rtd/index.rst
index 20a99a30..5d90c131 100644
--- a/doc/rtd/index.rst
+++ b/doc/rtd/index.rst
@@ -1,50 +1,79 @@
 .. _index:
 
-.. http://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html
-.. As suggested at link above for headings use:
-..   # with overline, for parts
-..   * with overline, for chapters
-..   =, for sections
-..   -, for subsections
-..   ^, for subsubsections
-..   “, for paragraphs
+cloud-init Documentation
+########################
 
-#############
-Documentation
-#############
+Cloud-init is the *industry standard* multi-distribution method for
+cross-platform cloud instance initialization. It is supported across all
+major public cloud providers, provisioning systems for private cloud
+infrastructure, and bare-metal installations.
 
-.. rubric:: Everything about cloud-init, a set of **python** scripts and
-            utilities to make your cloud images be all they can be!
+Cloud instances are initialized from a disk image and instance data:
 
-*******
-Summary
-*******
+- Cloud metadata
+- User data (optional)
+- Vendor data (optional)
 
-`Cloud-init`_ is the *defacto* multi-distribution package that handles early
-initialization of a cloud instance.
+Cloud-init will identify the cloud it is running on during boot, read any
+provided metadata from the cloud and initialize the system accordingly. This
+may involve setting up the network and storage devices to configuring SSH
+access key and many other aspects of a system. Later on the cloud-init will
+also parse and process any optional user or vendor data that was passed to the
+instance.
 
-----
+Getting help
+************
+
+Having trouble? We would like to help!
+
+- Try the :ref:`FAQ` – its got answers to some common questions
+- Ask a question in the ``#cloud-init`` IRC channel on Freenode
+- Join and ask questions on the `cloud-init mailing list <https://launchpad.net/~cloud-init>`_
+- Find a bug? `Report bugs on Launchpad <https://bugs.launchpad.net/cloud-init/+filebug>`_
 
 .. toctree::
-   :maxdepth: 2
+   :hidden:
+   :titlesonly:
+   :caption: Getting Started
 
-   topics/capabilities.rst
    topics/availability.rst
+   topics/boot.rst
+   topics/cli.rst
+   topics/faq.rst
+   topics/bugs.rst
+
+.. toctree::
+   :hidden:
+   :titlesonly:
+   :caption: User Data
+
    topics/format.rst
-   topics/instancedata.rst
-   topics/dir_layout.rst
    topics/examples.rst
-   topics/boot.rst
-   topics/datasources.rst
-   topics/logging.rst
    topics/modules.rst
    topics/merging.rst
-   topics/network-config.rst
+
+.. toctree::
+   :hidden:
+   :titlesonly:
+   :caption: Instance Data
+
+   topics/instancedata.rst
+   topics/datasources.rst
    topics/vendordata.rst
-   topics/debugging.rst
-   topics/moreinfo.rst
+   topics/network-config.rst
+
+.. toctree::
+   :hidden:
+   :titlesonly:
+   :caption: Development
+
    topics/hacking.rst
+   topics/security.rst
+   topics/debugging.rst
+   topics/logging.rst
+   topics/dir_layout.rst
+   topics/analyze.rst
+   topics/docs.rst
    topics/tests.rst
 
-.. _Cloud-init: https://launchpad.net/cloud-init
-.. vi: textwidth=78
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/analyze.rst b/doc/rtd/topics/analyze.rst
new file mode 100644
index 00000000..709131b8
--- /dev/null
+++ b/doc/rtd/topics/analyze.rst
@@ -0,0 +1,318 @@
+.. _analyze:
+
+Analyze
+*******
+
+The analyze subcommand was added to cloud-init in order to help analyze
+cloud-init boot time performance. It is loosely based on systemd-analyze where
+there are four subcommands:
+
+- blame
+- show
+- dump
+- boot
+
+Usage
+=====
+
+The analyze command requires one of the four subcommands:
+
+.. code-block:: shell-session
+
+  $ cloud-init analyze blame
+  $ cloud-init analyze show
+  $ cloud-init analyze dump
+  $ cloud-init analyze boot
+
+Availability
+============
+
+The analyze subcommand is generally available across all distributions with the
+exception of Gentoo and FreeBSD.
+
+Subcommands
+===========
+
+Blame
+-----
+
+The ``blame`` action matches ``systemd-analyze blame`` where it prints, in
+descending order, the units that took the longest to run.  This output is
+highly useful for examining where cloud-init is spending its time during
+execution.
+
+.. code-block:: shell-session
+
+  $ cloud-init analyze blame
+  -- Boot Record 01 --
+      00.80300s (init-network/config-growpart)
+      00.64300s (init-network/config-resizefs)
+      00.62100s (init-network/config-ssh)
+      00.57300s (modules-config/config-grub-dpkg)
+      00.40300s (init-local/search-NoCloud)
+      00.38200s (init-network/config-users-groups)
+      00.19800s (modules-config/config-apt-configure)
+      00.03700s (modules-final/config-keys-to-console)
+      00.02100s (init-network/config-update_etc_hosts)
+      00.02100s (init-network/check-cache)
+      00.00800s (modules-final/config-ssh-authkey-fingerprints)
+      00.00800s (init-network/consume-vendor-data)
+      00.00600s (modules-config/config-timezone)
+      00.00500s (modules-final/config-final-message)
+      00.00400s (init-network/consume-user-data)
+      00.00400s (init-network/config-mounts)
+      00.00400s (init-network/config-disk_setup)
+      00.00400s (init-network/config-bootcmd)
+      00.00400s (init-network/activate-datasource)
+      00.00300s (init-network/config-update_hostname)
+      00.00300s (init-network/config-set_hostname)
+      00.00200s (modules-final/config-snappy)
+      00.00200s (init-network/config-rsyslog)
+      00.00200s (init-network/config-ca-certs)
+      00.00200s (init-local/check-cache)
+      00.00100s (modules-final/config-scripts-vendor)
+      00.00100s (modules-final/config-scripts-per-once)
+      00.00100s (modules-final/config-salt-minion)
+      00.00100s (modules-final/config-rightscale_userdata)
+      00.00100s (modules-final/config-phone-home)
+      00.00100s (modules-final/config-package-update-upgrade-install)
+      00.00100s (modules-final/config-fan)
+      00.00100s (modules-config/config-ubuntu-advantage)
+      00.00100s (modules-config/config-ssh-import-id)
+      00.00100s (modules-config/config-snap)
+      00.00100s (modules-config/config-set-passwords)
+      00.00100s (modules-config/config-runcmd)
+      00.00100s (modules-config/config-locale)
+      00.00100s (modules-config/config-byobu)
+      00.00100s (modules-config/config-apt-pipelining)
+      00.00100s (init-network/config-write-files)
+      00.00100s (init-network/config-seed_random)
+      00.00100s (init-network/config-migrator)
+      00.00000s (modules-final/config-ubuntu-drivers)
+      00.00000s (modules-final/config-scripts-user)
+      00.00000s (modules-final/config-scripts-per-instance)
+      00.00000s (modules-final/config-scripts-per-boot)
+      00.00000s (modules-final/config-puppet)
+      00.00000s (modules-final/config-power-state-change)
+      00.00000s (modules-final/config-mcollective)
+      00.00000s (modules-final/config-lxd)
+      00.00000s (modules-final/config-landscape)
+      00.00000s (modules-final/config-chef)
+      00.00000s (modules-config/config-snap_config)
+      00.00000s (modules-config/config-ntp)
+      00.00000s (modules-config/config-emit_upstart)
+      00.00000s (modules-config/config-disable-ec2-metadata)
+      00.00000s (init-network/setup-datasource)
+
+  1 boot records analyzed
+
+Show
+----
+
+The ``show`` action is similar to ``systemd-analyze critical-chain`` which
+prints a list of units, the time they started and how long they took.
+Cloud-init has four stages and within each stage a number of modules may run
+depending on configuration. ``cloudinit-analyze show`` will, for each boot,
+print this information and a summary total time, per boot.
+
+The following is an abbreviated example of the show output:
+
+.. code-block:: shell-session
+
+  $ cloud-init analyze show
+  -- Boot Record 01 --
+  The total time elapsed since completing an event is printed after the "@" character.
+  The time the event takes is printed after the "+" character.
+
+  Starting stage: init-local
+  |``->no cache found @00.01700s +00.00200s
+  |`->found local data from DataSourceNoCloud @00.11000s +00.40300s
+  Finished stage: (init-local) 00.94200 seconds
+
+  Starting stage: init-network
+  |`->restored from cache with run check: DataSourceNoCloud [seed=/dev/sr0][dsmode=net] @04.79500s +00.02100s
+  |`->setting up datasource @04.88900s +00.00000s
+  |`->reading and applying user-data @04.90100s +00.00400s
+  |`->reading and applying vendor-data @04.90500s +00.00800s
+  |`->activating datasource @04.95200s +00.00400s
+  Finished stage: (init-network) 02.72100 seconds
+
+  Starting stage: modules-config
+  |`->config-emit_upstart ran successfully @15.43100s +00.00000s
+  |`->config-snap ran successfully @15.43100s +00.00100s
+  ...
+  |`->config-runcmd ran successfully @16.22300s +00.00100s
+  |`->config-byobu ran successfully @16.23400s +00.00100s
+  Finished stage: (modules-config) 00.83500 seconds
+
+  Starting stage: modules-final
+  |`->config-snappy ran successfully @16.87400s +00.00200s
+  |`->config-package-update-upgrade-install ran successfully @16.87600s +00.00100s
+  ...
+  |`->config-final-message ran successfully @16.93700s +00.00500s
+  |`->config-power-state-change ran successfully @16.94300s +00.00000s
+  Finished stage: (modules-final) 00.10300 seconds
+
+  Total Time: 4.60100 seconds
+
+  1 boot records analyzed
+
+If additional boot records are detected then they are printed out from oldest
+to newest.
+
+Dump
+----
+
+The ``dump`` action simply dumps the cloud-init logs that the analyze module
+is performing the analysis on and returns a list of dictionaries that can be
+consumed for other reporting needs. Each element in the list is a boot entry.
+
+.. code-block:: shell-session
+
+  $ cloud-init analyze dump
+  [
+  {
+    "description": "starting search for local datasources",
+    "event_type": "start",
+    "name": "init-local",
+    "origin": "cloudinit",
+    "timestamp": 1567057578.037
+  },
+  {
+    "description": "attempting to read from cache [check]",
+    "event_type": "start",
+    "name": "init-local/check-cache",
+    "origin": "cloudinit",
+    "timestamp": 1567057578.054
+  },
+  {
+    "description": "no cache found",
+    "event_type": "finish",
+    "name": "init-local/check-cache",
+    "origin": "cloudinit",
+    "result": "SUCCESS",
+    "timestamp": 1567057578.056
+  },
+  {
+    "description": "searching for local data from DataSourceNoCloud",
+    "event_type": "start",
+    "name": "init-local/search-NoCloud",
+    "origin": "cloudinit",
+    "timestamp": 1567057578.147
+  },
+  {
+    "description": "found local data from DataSourceNoCloud",
+    "event_type": "finish",
+    "name": "init-local/search-NoCloud",
+    "origin": "cloudinit",
+    "result": "SUCCESS",
+    "timestamp": 1567057578.55
+  },
+  {
+    "description": "searching for local datasources",
+    "event_type": "finish",
+    "name": "init-local",
+    "origin": "cloudinit",
+    "result": "SUCCESS",
+    "timestamp": 1567057578.979
+  },
+  {
+    "description": "searching for network datasources",
+    "event_type": "start",
+    "name": "init-network",
+    "origin": "cloudinit",
+    "timestamp": 1567057582.814
+  },
+  {
+    "description": "attempting to read from cache [trust]",
+    "event_type": "start",
+    "name": "init-network/check-cache",
+    "origin": "cloudinit",
+    "timestamp": 1567057582.832
+  },
+  ...
+  {
+    "description": "config-power-state-change ran successfully",
+    "event_type": "finish",
+    "name": "modules-final/config-power-state-change",
+    "origin": "cloudinit",
+    "result": "SUCCESS",
+    "timestamp": 1567057594.98
+  },
+  {
+    "description": "running modules for final",
+    "event_type": "finish",
+    "name": "modules-final",
+    "origin": "cloudinit",
+    "result": "SUCCESS",
+    "timestamp": 1567057594.982
+  }
+  ]
+
+
+Boot
+----
+
+The ``boot`` action prints out kernel related timestamps that are not included
+in any of the cloud-init logs. There are three different timestamps that are
+presented to the user:
+
+- kernel start
+- kernel finish boot
+- cloud-init start
+
+This was added for additional clarity into the boot process that cloud-init
+does not have control over, to aid in debugging of performance issues related
+to cloud-init startup, and tracking regression.
+
+.. code-block:: shell-session
+
+  $ cloud-init analyze boot
+  -- Most Recent Boot Record --
+      Kernel Started at: 2019-08-29 01:35:37.753790
+      Kernel ended boot at: 2019-08-29 01:35:38.807407
+      Kernel time to boot (seconds): 1.053617000579834
+      Cloud-init activated by systemd at: 2019-08-29 01:35:43.992460
+      Time between Kernel end boot and Cloud-init activation (seconds): 5.185053110122681
+      Cloud-init start: 2019-08-29 08:35:45.867000
+  successful
+
+Timestamp Gathering
+^^^^^^^^^^^^^^^^^^^
+
+The following boot related timestamps are gathered on demand when cloud-init
+analyze boot runs:
+
+- Kernel startup gathered from system uptime
+- Kernel finishes initialization from systemd
+  UserSpaceMonotonicTimestamp property
+- Cloud-init activation from the property InactiveExitTimestamp of the
+  cloud-init local systemd unit
+
+In order to gather the necessary timestamps using systemd, running the
+commands below will gather the UserspaceTimestamp and InactiveExitTimestamp:
+
+.. code-block:: shell-session
+
+  $ systemctl show -p UserspaceTimestampMonotonic
+  UserspaceTimestampMonotonic=989279
+  $ systemctl show cloud-init-local -p InactiveExitTimestampMonotonic
+  InactiveExitTimestampMonotonic=4493126
+
+The UserspaceTimestamp tracks when the init system starts, which is used as
+an indicator of kernel finishing initialization. The InactiveExitTimestamp
+tracks when a particular systemd unit transitions from the Inactive to Active
+state, which can be used to mark the beginning of systemd's activation of
+cloud-init.
+
+Currently this only works for distros that use systemd as the init process.
+We will be expanding support for other distros in the future and this document
+will be updated accordingly.
+
+If systemd is not present on the system, dmesg is used to attempt to find an
+event that logs the beginning of the init system. However, with this method
+only the first two timestamps are able to be found; dmesg does not monitor
+userspace processes, so no cloud-init start timestamps are emitted like when
+using systemd.
+
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/availability.rst b/doc/rtd/topics/availability.rst
index ef5ae7bf..3f215b1b 100644
--- a/doc/rtd/topics/availability.rst
+++ b/doc/rtd/topics/availability.rst
@@ -1,21 +1,64 @@
-************
+.. _availability:
+
 Availability
 ************
 
-It is currently installed in the `Ubuntu Cloud Images`_ and also in the official `Ubuntu`_ images available on EC2, Azure, GCE and many other clouds.
+Below outlines the current availability of cloud-init across
+distributions and clouds, both public and private.
+
+.. note::
+
+    If a distribution or cloud does not show up in the list below contact
+    them and ask for images to be generated using cloud-init!
 
-Versions for other systems can be (or have been) created for the following distributions:
+Distributions
+=============
+
+Cloud-init has support across all major Linux distributions and
+FreeBSD:
 
 - Ubuntu
+- SLES/openSUSE
+- RHEL/CentOS
 - Fedora
+- Gentoo Linux
 - Debian
-- RHEL
-- CentOS
-- *and more...*
+- ArchLinux
+- FreeBSD
+
+Clouds
+======
+
+Cloud-init provides support across a wide ranging list of execution
+environments in the public cloud:
+
+- Amazon Web Services
+- Microsoft Azure
+- Google Cloud Platform
+- Oracle Cloud Infrastructure
+- Softlayer
+- Rackspace Public Cloud
+- IBM Cloud
+- Digital Ocean
+- Bigstep
+- Hetzner
+- Joyent
+- CloudSigma
+- Alibaba Cloud
+- OVH
+- OpenNebula
+- Exoscale
+- Scaleway
+- CloudStack
+- AltCloud
+- SmartOS
 
-So ask your distribution provider where you can obtain an image with it built-in if one is not already available ☺
+Additionally, cloud-init is supported on these private clouds:
 
+- Bare metal installs
+- OpenStack
+- LXD
+- KVM
+- Metal-as-a-Service (MAAS)
 
-.. _Ubuntu Cloud Images: http://cloud-images.ubuntu.com/
-.. _Ubuntu: http://www.ubuntu.com/
-.. vi: textwidth=78
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/boot.rst b/doc/rtd/topics/boot.rst
index f2976fdf..d846867b 100644
--- a/doc/rtd/topics/boot.rst
+++ b/doc/rtd/topics/boot.rst
@@ -1,43 +1,51 @@
 .. _boot_stages:
 
-***********
 Boot Stages
 ***********
-In order to be able to provide the functionality that it does, cloud-init
-must be integrated into the boot in fairly controlled way.
 
-There are 5 stages.
+In order to be able to provide the functionality that it does, cloud-init
+must be integrated into the boot in fairly controlled way. There are five
+stages to boot:
 
-1. **Generator**
-2. **Local**
-3. **Network**
-4. **Config**
-5. **Final**
+1. Generator
+2. Local
+3. Network
+4. Config
+5. Final
 
 Generator
 =========
+
 When booting under systemd, a
 `generator <https://www.freedesktop.org/software/systemd/man/systemd.generator.html>`_
-will run that determines if cloud-init.target should be included in the
-boot goals.  By default, this generator will enable cloud-init.  It will
-not enable cloud-init if either:
+will run that determines if cloud-init.target should be included in the boot
+goals.  By default, this generator will enable cloud-init.  It will not enable
+cloud-init if either:
 
- * A file exists: ``/etc/cloud/cloud-init.disabled``
- * The kernel command line as found in /proc/cmdline contains ``cloud-init=disabled``.
-   When running in a container, the kernel command line is not honored, but
-   cloud-init will read an environment variable named ``KERNEL_CMDLINE`` in
-   its place.
+ * The file ``/etc/cloud/cloud-init.disabled`` exists
+ * The kernel command line as found in ``/proc/cmdline`` contains
+   ``cloud-init=disabled``. When running in a container, the kernel command
+   line is not honored, but cloud-init will read an environment variable named
+   ``KERNEL_CMDLINE`` in its place.
 
-This mechanism for disabling at runtime currently only exists in systemd.
+Again, these mechanisms for disabling cloud-init at runtime currently only
+exist in systemd.
 
 Local
 =====
- * **systemd service**: ``cloud-init-local.service``
- * **runs**: As soon as possible with / mounted read-write.
- * **blocks**: as much of boot as possible, *must* block network bringup.
- * **modules**: none
 
-The purpose of the local stage is:
++------------------+----------------------------------------------------------+
+| systemd service  | ``cloud-init-local.service``                             |
++---------+--------+----------------------------------------------------------+
+| runs             | as soon as possible with ``/`` mounted read-write        |
++---------+--------+----------------------------------------------------------+
+| blocks           | as much of boot as possible, *must* block network        |
++---------+--------+----------------------------------------------------------+
+| modules          | none                                                     |
++---------+--------+----------------------------------------------------------+
+
+The purpose of the local stage is to:
+
  * locate "local" data sources.
  * apply networking configuration to the system (including "Fallback")
 
@@ -45,13 +53,13 @@ In most cases, this stage does not do much more than that.  It finds the
 datasource and determines the network configuration to be used.  That
 network configuration can come from:
 
- * the datasource
- * fallback: Cloud-init's fallback networking consists of rendering the
+ * **datasource**: cloud provided network configuration via metadata
+ * **fallback**: cloud-init's fallback networking consists of rendering the
    equivalent to "dhcp on eth0", which was historically the most popular
-   mechanism for network configuration of a guest.
- * none.  network configuration can be disabled entirely with 
-   config like the following in /etc/cloud/cloud.cfg: 
-   '``network: {config: disabled}``'.
+   mechanism for network configuration of a guest
+ * **none**: network configuration can be disabled by writing the file
+   ``/etc/cloud/cloud.cfg`` with the content:
+   ``network: {config: disabled}``
 
 If this is an instance's first boot, then the selected network configuration
 is rendered.  This includes clearing of all previous (stale) configuration
@@ -73,20 +81,26 @@ that require a network can operate at this stage.
 
 Network
 =======
- * **systemd service**: ``cloud-init.service``
- * **runs**: After local stage and configured networking is up.
- * **blocks**: As much of remaining boot as possible.
- * **modules**: ``cloud_init_modules`` in **/etc/cloud/cloud.cfg**
+
++------------------+----------------------------------------------------------+
+| systemd service  | ``cloud-init.service``                                   |
++---------+--------+----------------------------------------------------------+
+| runs             | after local stage and configured networking is up        |
++---------+--------+----------------------------------------------------------+
+| blocks           | as much of remaining boot as possible                    |
++---------+--------+----------------------------------------------------------+
+| modules          | *cloud_init_modules* in ``/etc/cloud/cloud.cfg``         |
++---------+--------+----------------------------------------------------------+
 
 This stage requires all configured networking to be online, as it will fully
 process any user-data that is found.  Here, processing means:
 
- * retrive any ``#include`` or ``#include-once`` (recursively) including http
- * uncompress any compressed content
+ * retrieve any ``#include`` or ``#include-once`` (recursively) including http
+ * decompress any compressed content
  * run any part-handler found.
 
 This stage runs the ``disk_setup`` and ``mounts`` modules which may partition
-and format disks and configure mount points (such as in /etc/fstab).
+and format disks and configure mount points (such as in ``/etc/fstab``).
 Those modules cannot run earlier as they may receive configuration input
 from sources only available via network.  For example, a user may have
 provided user-data in a network resource that describes how local mounts
@@ -94,30 +108,41 @@ should be done.
 
 On some clouds such as Azure, this stage will create filesystems to be
 mounted, including ones that have stale (previous instance) references in
-/etc/fstab. As such, entries /etc/fstab other than those necessary for
+``/etc/fstab``. As such, entries ``/etc/fstab`` other than those necessary for
 cloud-init to run should not be done until after this stage.
 
-A part-handler will run at this stage, as will boothooks including
+A part-handler will run at this stage, as will boot-hooks including
 cloud-config ``bootcmd``.  The user of this functionality has to be aware
 that the system is in the process of booting when their code runs.
 
 Config
 ======
- * **systemd service**: ``cloud-config.service``
- * **runs**: After network stage.
- * **blocks**: None.
- * **modules**: ``cloud_config_modules`` in **/etc/cloud/cloud.cfg**
+
++------------------+----------------------------------------------------------+
+| systemd service  | ``cloud-config.service``                                 |
++---------+--------+----------------------------------------------------------+
+| runs             | after network                                            |
++---------+--------+----------------------------------------------------------+
+| blocks           | nothing                                                  |
++---------+--------+----------------------------------------------------------+
+| modules          | *cloud_config_modules* in ``/etc/cloud/cloud.cfg``       |
++---------+--------+----------------------------------------------------------+
 
 This stage runs config modules only.  Modules that do not really have an
 effect on other stages of boot are run here.
 
-
 Final
 =====
- * **systemd service**: ``cloud-final.service``
- * **runs**: As final part of boot (traditional "rc.local")
- * **blocks**: None.
- * **modules**: ``cloud_final_modules`` in **/etc/cloud/cloud.cfg**
+
++------------------+----------------------------------------------------------+
+| systemd service  | ``cloud-final.service``                                  |
++---------+--------+----------------------------------------------------------+
+| runs             | as final part of boot (traditional "rc.local")           |
++---------+--------+----------------------------------------------------------+
+| blocks           | nothing                                                  |
++---------+--------+----------------------------------------------------------+
+| modules          | *cloud_final_modules* in ``/etc/cloud/cloud.cfg``        |
++---------+--------+----------------------------------------------------------+
 
 This stage runs as late in boot as possible.  Any scripts that a user is
 accustomed to running after logging into a system should run correctly here.
@@ -127,9 +152,9 @@ Things that run here include
  * configuration management plugins (puppet, chef, salt-minion)
  * user-scripts (including ``runcmd``).
 
-For scripts external to cloud-init looking to wait until cloud-init
+For scripts external to cloud-init looking to wait until cloud-init is
 finished, the ``cloud-init status`` subcommand can help block external
 scripts until cloud-init is done without having to write your own systemd
 units dependency chains. See :ref:`cli_status` for more info.
 
-.. vi: textwidth=78
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/bugs.rst b/doc/rtd/topics/bugs.rst
new file mode 100644
index 00000000..4b60776b
--- /dev/null
+++ b/doc/rtd/topics/bugs.rst
@@ -0,0 +1,108 @@
+.. _reporting_bugs:
+
+Reporting Bugs
+**************
+
+The following documents:
+
+1) How to collect information for reporting bugs
+2) How to file bugs to the upstream cloud-init project or for distro specific
+   packages
+
+Collect Logs
+============
+
+To aid in debugging, please collect the necessary logs. To do so, run the
+`collect-logs` subcommand to produce a tarfile that you can easily upload:
+
+.. code-block:: shell-session
+
+  $ cloud-init collect-logs
+  Wrote /home/ubuntu/cloud-init.tar.gz
+
+If your version of cloud-init does not have the  `collect-logs` subcommand,
+then please manually collect the base log files by doing the following:
+
+.. code-block:: shell-session
+
+  $ dmesg > dmesg.txt
+  $ sudo journalctl -o short-precise > journal.txt
+  $ sudo tar -cvf cloud-init.tar dmesg.txt journal.txt /run/cloud-init \
+      /var/log/cloud-init.log /var/log/cloud-init-output.log
+
+Report Upstream Bug
+===================
+
+Bugs for upstream cloud-init are tracked using Launchpad. To file a bug:
+
+1. Collect the necessary debug logs as described above
+2. `Create a Launchpad account`_ or login to your existing account
+3. `Report an upstream cloud-init bug`_
+
+If debug logs are not provided, you will be asked for them before any
+further time is spent debugging. If you are unable to obtain the required
+logs please explain why in the bug.
+
+If your bug is for a specific distro using cloud-init, please first consider
+reporting it with the upstream distro or confirm that it still occurs
+with the latest upstream cloud-init code. See below for details on specific
+distro reporting.
+
+Distro Specific Issues
+======================
+
+For issues specific to your distro please use one of the following distro
+specific reporting mechanisms:
+
+Ubuntu
+------
+
+To report a bug on Ubuntu use the `ubuntu-bug` command on the affected
+system to automatically collect the necessary logs and file a bug on
+Launchpad:
+
+.. code-block:: shell-session
+
+  $ ubuntu-bug cloud-init
+
+If that does not work or is not an option, please collect the logs using the
+commands in the above Collect Logs section and then report the bug on the
+`Ubuntu bug tracker`_. Make sure to attach your collected logs!
+
+Debian
+------
+
+To file a bug against the Debian package fo cloud-init please use the
+`Debian bug tracker`_ to file against 'Package: cloud-init'. See the
+`Debian bug reporting wiki`_ wiki page for more details.
+
+Red Hat, CentOS, & Fedora
+-------------------------
+
+To file a bug against the Red Hat or Fedora packages of cloud-init please use
+the `Red Hat bugzilla`_.
+
+SUSE & openSUSE
+---------------
+
+To file a bug against the SuSE packages of cloud-init please use the
+`SUSE bugzilla`_.
+
+Arch
+----
+
+To file a bug against the Arch package of cloud-init please use the
+`Arch Linux Bugtracker`_. See the `Arch bug reporting wiki`_ for more
+details.
+
+.. _Create a Launchpad account: https://help.launchpad.net/YourAccount/NewAccount
+.. _Report an upstream cloud-init bug: https://bugs.launchpad.net/cloud-init/+filebug
+.. _Ubuntu bug tracker: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+filebug
+.. _Debian bug tracker: https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=cloud-init;dist=unstable
+.. _Debian bug reporting wiki: https://www.debian.org/Bugs/Reporting
+.. _Red Hat bugzilla: https://bugzilla.redhat.com/
+.. _SUSE bugzilla: https://bugzilla.suse.com/index.cgi
+.. _Arch Linux Bugtracker: https://bugs.archlinux.org/
+.. _Arch bug reporting wiki: https://wiki.archlinux.org/index.php/Bug_reporting_guidelines
+
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/capabilities.rst b/doc/rtd/topics/capabilities.rst
deleted file mode 100644
index 0d8b8947..00000000
--- a/doc/rtd/topics/capabilities.rst
+++ /dev/null
@@ -1,299 +0,0 @@
-.. _capabilities:
-
-************
-Capabilities
-************
-
-- Setting a default locale
-- Setting an instance hostname
-- Generating instance SSH private keys
-- Adding SSH keys to a user's ``.ssh/authorized_keys`` so they can log in
-- Setting up ephemeral mount points
-- Configuring network devices
-
-User configurability
-====================
-
-`Cloud-init`_ 's behavior can be configured via user-data.
-
-    User-data can be given by the user at instance launch time. See
-    :ref:`user_data_formats` for acceptable user-data content.
-
-
-This is done via the ``--user-data`` or ``--user-data-file`` argument to
-ec2-run-instances for example.
-
-* Check your local client's documentation for how to provide a `user-data`
-  string or `user-data` file to cloud-init on instance creation.
-
-
-Feature detection
-=================
-
-Newer versions of cloud-init may have a list of additional features that they
-support. This allows other applications to detect what features the installed
-cloud-init supports without having to parse its version number. If present,
-this list of features will be located at ``cloudinit.version.FEATURES``.
-
-Currently defined feature names include:
-
- - ``NETWORK_CONFIG_V1`` support for v1 networking configuration,
-   see :ref:`network_config_v1` documentation for examples.
- - ``NETWORK_CONFIG_V2`` support for v2 networking configuration,
-   see :ref:`network_config_v2` documentation for examples.
-
-
-CLI Interface
-=============
-
-The command line documentation is accessible on any cloud-init installed
-system:
-
-.. code-block:: shell-session
-
-  % cloud-init --help
-  usage: cloud-init [-h] [--version] [--file FILES]
-                    [--debug] [--force]
-                    {init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status}
-                                                         ...
-
-  optional arguments:
-    -h, --help            show this help message and exit
-    --version, -v         show program's version number and exit
-    --file FILES, -f FILES
-                          additional yaml configuration files to use
-    --debug, -d           show additional pre-action logging (default: False)
-    --force               force running even if no datasource is found (use at
-                          your own risk)
-
-  Subcommands:
-    {init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status}
-      init                initializes cloud-init and performs initial modules
-      modules             activates modules using a given configuration key
-      single              run a single module
-      query               Query instance metadata from the command line
-      dhclient-hook       run the dhclient hookto record network info
-      features            list defined features
-      analyze             Devel tool: Analyze cloud-init logs and data
-      devel               Run development tools
-      collect-logs        Collect and tar all cloud-init debug info
-      clean               Remove logs and artifacts so cloud-init can re-run
-      status              Report cloud-init status or wait on completion
-
-
-CLI Subcommand details
-======================
-
-.. _cli_features:
-
-cloud-init features
--------------------
-Print out each feature supported.  If cloud-init does not have the
-features subcommand, it also does not support any features described in
-this document.
-
-.. code-block:: shell-session
-
-  % cloud-init features
-  NETWORK_CONFIG_V1
-  NETWORK_CONFIG_V2
-
-.. _cli_status:
-
-cloud-init status
------------------
-Report whether cloud-init is running, done, disabled or errored. Exits
-non-zero if an error is detected in cloud-init.
-
-* **--long**: Detailed status information.
-* **--wait**: Block until cloud-init completes.
-
-.. code-block:: shell-session
-
-  % cloud-init status --long
-  status: done
-  time: Wed, 17 Jan 2018 20:41:59 +0000
-  detail:
-  DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]
-
-  # Cloud-init running still short versus long options
-  % cloud-init status
-  status: running
-  % cloud-init status --long
-  status: running
-  time: Fri, 26 Jan 2018 21:39:43 +0000
-  detail:
-  Running in stage: init-local
-
-.. _cli_collect_logs:
-
-cloud-init collect-logs
------------------------
-Collect and tar cloud-init generated logs, data files and system
-information for triage. This subcommand is integrated with apport. 
-
-**Note**: Ubuntu users can file bugs with `ubuntu-bug cloud-init` to
-automaticaly attach these logs to a bug report.
-
-Logs collected are:
-
- * /var/log/cloud-init*log
- * /run/cloud-init
- * cloud-init package version
- * dmesg output
- * journalctl output
- * /var/lib/cloud/instance/user-data.txt
-
-.. _cli_query:
-
-cloud-init query
-------------------
-Query standardized cloud instance metadata crawled by cloud-init and stored
-in ``/run/cloud-init/instance-data.json``. This is a convenience command-line
-interface to reference any cached configuration metadata that cloud-init
-crawls when booting the instance. See :ref:`instance_metadata` for more info.
-
-* **--all**: Dump all available instance data as json which can be queried.
-* **--instance-data**: Optional path to a different instance-data.json file to
-  source for queries.
-* **--list-keys**: List available query keys from cached instance data.
-
-.. code-block:: shell-session
-
-  # List all top-level query keys available (includes standardized aliases)
-  % cloud-init query --list-keys
-  availability_zone
-  base64_encoded_keys
-  cloud_name
-  ds
-  instance_id
-  local_hostname
-  region
-  v1
-
-* **<varname>**: A dot-delimited variable path into the instance-data.json
-   object.
-
-.. code-block:: shell-session
-
-  # Query cloud-init standardized metadata on any cloud
-  % cloud-init query v1.cloud_name
-  aws  # or openstack, azure, gce etc.
-
-  # Any standardized instance-data under a <v#> key is aliased as a top-level
-  # key for convenience.
-  % cloud-init query cloud_name
-  aws  # or openstack, azure, gce etc.
-
-  # Query datasource-specific metadata on EC2
-  % cloud-init query ds.meta_data.public_ipv4
-
-* **--format** A string that will use jinja-template syntax to render a string
-   replacing
-
-.. code-block:: shell-session
-
-  # Generate a custom hostname fqdn based on instance-id, cloud and region
-  % cloud-init query --format 'custom-{{instance_id}}.{{region}}.{{v1.cloud_name}}.com'
-  custom-i-0e91f69987f37ec74.us-east-2.aws.com
-
-
-.. note::
-  The standardized instance data keys under **v#** are guaranteed not to change
-  behavior or format. If using top-level convenience aliases for any
-  standardized instance data keys, the most value (highest **v#**) of that key
-  name is what is reported as the top-level value. So these aliases act as a
-  'latest'.
-
-
-.. _cli_analyze:
-
-cloud-init analyze
-------------------
-Get detailed reports of where cloud-init spends most of its time. See
-:ref:`boot_time_analysis` for more info.
-
-* **blame** Report ordered by most costly operations.
-* **dump** Machine-readable JSON dump of all cloud-init tracked events.
-* **show** show time-ordered report of the cost of operations during each
-  boot stage.
-
-.. _cli_devel:
-
-cloud-init devel
-----------------
-Collection of development tools under active development. These tools will
-likely be promoted to top-level subcommands when stable.
-
- * ``cloud-init devel schema``: A **#cloud-config** format and schema
-   validator. It accepts a cloud-config yaml file and annotates potential
-   schema errors locally without the need for deployment. Schema
-   validation is work in progress and supports a subset of cloud-config
-   modules.
-
- * ``cloud-init devel render``: Use cloud-init's jinja template render to
-   process  **#cloud-config** or **custom-scripts**, injecting any variables
-   from ``/run/cloud-init/instance-data.json``. It accepts a user-data file
-   containing  the jinja template header ``## template: jinja`` and renders
-   that content with any instance-data.json variables present.
-
-
-.. _cli_clean:
-
-cloud-init clean
-----------------
-Remove cloud-init artifacts from /var/lib/cloud and optionally reboot the
-machine to so cloud-init re-runs all stages as it did on first boot.
-
-* **--logs**: Optionally remove /var/log/cloud-init*log files.
-* **--reboot**: Reboot the system after removing artifacts.
-
-.. _cli_init:
-
-cloud-init init
----------------
-Generally run by OS init systems to execute cloud-init's stages
-*init* and *init-local*. See :ref:`boot_stages` for more info.
-Can be run on the commandline, but is generally gated to run only once
-due to semaphores in **/var/lib/cloud/instance/sem/** and
-**/var/lib/cloud/sem**.
-
-* **--local**: Run *init-local* stage instead of *init*.
-
-.. _cli_modules:
-
-cloud-init modules
-------------------
-Generally run by OS init systems to execute *modules:config* and
-*modules:final* boot stages. This executes cloud config :ref:`modules`
-configured to run in the init, config and final stages. The modules are
-declared to run in various boot stages in the file
-**/etc/cloud/cloud.cfg** under keys **cloud_init_modules**,
-**cloud_init_modules** and **cloud_init_modules**. Can be run on the
-commandline, but each module is gated to run only once due to semaphores
-in ``/var/lib/cloud/``.
-
-* **--mode (init|config|final)**: Run *modules:init*, *modules:config* or
-  *modules:final* cloud-init stages. See :ref:`boot_stages` for more info.
-
-.. _cli_single:
-
-cloud-init single
------------------
-Attempt to run a single named cloud config module.  The following example
-re-runs the cc_set_hostname module ignoring the module default frequency
-of once-per-instance:
-
-* **--name**: The cloud-config module name to run
-* **--frequency**: Optionally override the declared module frequency
-  with one of (always|once-per-instance|once)
-
-.. code-block:: shell-session
-
-  % cloud-init single --name set_hostname --frequency always
-
-**Note**: Mileage may vary trying to re-run each cloud-config module, as
-some are not idempotent.
-
-.. _Cloud-init: https://launchpad.net/cloud-init
-.. vi: textwidth=78
diff --git a/doc/rtd/topics/cli.rst b/doc/rtd/topics/cli.rst
new file mode 100644
index 00000000..b32677b0
--- /dev/null
+++ b/doc/rtd/topics/cli.rst
@@ -0,0 +1,304 @@
+.. _cli:
+
+CLI Interface
+*************
+
+For the latest list of subcommands and arguments use cloud-init's ``--help``
+option. This can be used against cloud-init itself or any of its subcommands.
+
+.. code-block:: shell-session
+
+  $ cloud-init --help
+    usage: /usr/bin/cloud-init [-h] [--version] [--file FILES] [--debug] [--force]
+                            {init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status}
+                            ...
+
+    optional arguments:
+    -h, --help            show this help message and exit
+    --version, -v         show program's version number and exit
+    --file FILES, -f FILES
+                            additional yaml configuration files to use
+    --debug, -d           show additional pre-action logging (default: False)
+    --force               force running even if no datasource is found (use at
+                            your own risk)
+
+    Subcommands:
+    {init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status}
+        init                initializes cloud-init and performs initial modules
+        modules             activates modules using a given configuration key
+        single              run a single module
+        query               Query standardized instance metadata from the command
+                            line.
+        dhclient-hook       Run the dhclient hook to record network info.
+        features            list defined features
+        analyze             Devel tool: Analyze cloud-init logs and data
+        devel               Run development tools
+        collect-logs        Collect and tar all cloud-init debug info
+        clean               Remove logs and artifacts so cloud-init can re-run.
+        status              Report cloud-init status or wait on completion.
+
+The rest of this document will give an overview of each of the subcommands.
+
+
+.. _cli_analyze:
+
+analyze
+=======
+
+Get detailed reports of where cloud-init spends its time during the boot
+process. For more complete reference see :ref:`analyze`.
+
+Possible subcommands include:
+
+* *blame*: report ordered by most costly operations
+* *dump*: machine-readable JSON dump of all cloud-init tracked events
+* *show*: show time-ordered report of the cost of operations during each
+  boot stage
+* *boot*: show timestamps from kernel initialization, kernel finish
+  initialization, and cloud-init start
+
+
+.. _cli_clean:
+
+clean
+=====
+
+Remove cloud-init artifacts from ``/var/lib/cloud`` to simulate a clean
+instance. On reboot, cloud-init will re-run all stages as it did on first boot.
+
+* *\\-\\-logs*: optionally remove all cloud-init log files in ``/var/log/``
+* *\\-\\-reboot*: reboot the system after removing artifacts
+
+
+.. _cli_collect_logs:
+
+collect-logs
+============
+
+Collect and tar cloud-init generated logs, data files, and system
+information for triage. This subcommand is integrated with apport.
+
+Logs collected include:
+
+ * ``/var/log/cloud-init.log``
+ * ``/var/log/cloud-init-output.log``
+ * ``/run/cloud-init``
+ * ``/var/lib/cloud/instance/user-data.txt``
+ * cloud-init package version
+ * ``dmesg`` output
+ * journalctl output
+
+.. note::
+
+  Ubuntu users can file bugs with ``ubuntu-bug cloud-init`` to
+  automatically attach these logs to a bug report
+
+
+.. _cli_devel:
+
+devel
+=====
+
+Collection of development tools under active development. These tools will
+likely be promoted to top-level subcommands when stable.
+
+Do **NOT** rely on the output of these commands as they can and will change.
+
+Current subcommands:
+
+ * ``schema``: a **#cloud-config** format and schema
+   validator. It accepts a cloud-config yaml file and annotates potential
+   schema errors locally without the need for deployment. Schema
+   validation is work in progress and supports a subset of cloud-config
+   modules.
+
+ * ``render``: use cloud-init's jinja template render to
+   process  **#cloud-config** or **custom-scripts**, injecting any variables
+   from ``/run/cloud-init/instance-data.json``. It accepts a user-data file
+   containing  the jinja template header ``## template: jinja`` and renders
+   that content with any instance-data.json variables present.
+
+
+.. _cli_features:
+
+features
+========
+
+Print out each feature supported.  If cloud-init does not have the
+features subcommand, it also does not support any features described in
+this document.
+
+.. code-block:: shell-session
+
+  $ cloud-init features
+  NETWORK_CONFIG_V1
+  NETWORK_CONFIG_V2
+
+
+.. _cli_init:
+
+init
+====
+
+Generally run by OS init systems to execute cloud-init's stages
+*init* and *init-local*. See :ref:`boot_stages` for more info.
+Can be run on the commandline, but is generally gated to run only once
+due to semaphores in ``/var/lib/cloud/instance/sem/`` and
+``/var/lib/cloud/sem``.
+
+* *\\-\\-local*: run *init-local* stage instead of *init*
+
+
+.. _cli_modules:
+
+modules
+=======
+
+Generally run by OS init systems to execute *modules:config* and
+*modules:final* boot stages. This executes cloud config :ref:`modules`
+configured to run in the init, config and final stages. The modules are
+declared to run in various boot stages in the file
+``/etc/cloud/cloud.cfg`` under keys:
+
+* *cloud_init_modules*
+* *cloud_config_modules*
+* *cloud_init_modules*
+
+Can be run on the command line, but each module is gated to run only once due
+to semaphores in ``/var/lib/cloud/``.
+
+* *\\-\\-mode [init|config|final]*: run *modules:init*, *modules:config* or
+  *modules:final* cloud-init stages. See :ref:`boot_stages` for more info.
+
+
+.. _cli_query:
+
+query
+=====
+
+Query standardized cloud instance metadata crawled by cloud-init and stored
+in ``/run/cloud-init/instance-data.json``. This is a convenience command-line
+interface to reference any cached configuration metadata that cloud-init
+crawls when booting the instance. See :ref:`instance_metadata` for more info.
+
+* *\\-\\-all*: dump all available instance data as json which can be queried
+* *\\-\\-instance-data*: optional path to a different instance-data.json file
+  to source for queries
+* *\\-\\-list-keys*: list available query keys from cached instance data
+* *\\-\\-format*: a string that will use jinja-template syntax to render a
+  string replacing
+* *<varname>*: a dot-delimited variable path into the instance-data.json
+  object
+
+Below demonstrates how to list all top-level query keys that are standardized
+aliases:
+
+.. code-block:: shell-session
+
+    $ cloud-init query --list-keys
+    _beta_keys
+    availability_zone
+    base64_encoded_keys
+    cloud_name
+    ds
+    instance_id
+    local_hostname
+    platform
+    public_ssh_keys
+    region
+    sensitive_keys
+    subplatform
+    userdata
+    v1
+    vendordata
+
+Below demonstrates how to query standardized metadata from clouds:
+
+.. code-block:: shell-session
+
+  % cloud-init query v1.cloud_name
+  aws  # or openstack, azure, gce etc.
+
+  # Any standardized instance-data under a <v#> key is aliased as a top-level key for convenience.
+  % cloud-init query cloud_name
+  aws  # or openstack, azure, gce etc.
+
+  # Query datasource-specific metadata on EC2
+  % cloud-init query ds.meta_data.public_ipv4
+
+.. note::
+
+  The standardized instance data keys under **v#** are guaranteed not to change
+  behavior or format. If using top-level convenience aliases for any
+  standardized instance data keys, the most value (highest **v#**) of that key
+  name is what is reported as the top-level value. So these aliases act as a
+  'latest'.
+
+This data can then be formatted to generate custom strings or data:
+
+.. code-block:: shell-session
+
+  # Generate a custom hostname fqdn based on instance-id, cloud and region
+  % cloud-init query --format 'custom-{{instance_id}}.{{region}}.{{v1.cloud_name}}.com'
+  custom-i-0e91f69987f37ec74.us-east-2.aws.com
+
+
+.. _cli_single:
+
+single
+======
+
+Attempt to run a single named cloud config module.
+
+* *\\-\\-name*: the cloud-config module name to run
+* *\\-\\-frequency*: optionally override the declared module frequency
+  with one of (always|once-per-instance|once)
+
+The following example re-runs the cc_set_hostname module ignoring the module
+default frequency of once-per-instance:
+
+.. code-block:: shell-session
+
+  $ cloud-init single --name set_hostname --frequency always
+
+.. note::
+
+  Mileage may vary trying to re-run each cloud-config module, as
+  some are not idempotent.
+
+
+.. _cli_status:
+
+status
+======
+
+Report whether cloud-init is running, done, disabled or errored. Exits
+non-zero if an error is detected in cloud-init.
+
+* *\\-\\-long*: detailed status information
+* *\\-\\-wait*: block until cloud-init completes
+
+Below are examples of output when cloud-init is running, showing status and
+the currently running modules, as well as when it is done.
+
+.. code-block:: shell-session
+
+  $ cloud-init status
+  status: running
+
+  $ cloud-init status --long
+  status: running
+  time: Fri, 26 Jan 2018 21:39:43 +0000
+  detail:
+  Running in stage: init-local
+
+  $ cloud-init status
+  status: done
+
+  $ cloud-init status --long
+  status: done
+  time: Wed, 17 Jan 2018 20:41:59 +0000
+  detail:
+  DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]
+
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst
index e34f145c..3d026143 100644
--- a/doc/rtd/topics/datasources.rst
+++ b/doc/rtd/topics/datasources.rst
@@ -1,37 +1,125 @@
 .. _datasources:
 
-***********
 Datasources
 ***********
 
-What is a datasource?
-=====================
-
 Datasources are sources of configuration data for cloud-init that typically
-come from the user (aka userdata) or come from the stack that created the
-configuration drive (aka metadata). Typical userdata would include files,
+come from the user (e.g. userdata) or come from the cloud that created the
+configuration drive (e.g. metadata). Typical userdata would include files,
 yaml, and shell scripts while typical metadata would include server name,
-instance id, display name and other cloud specific details. Since there are
-multiple ways to provide this data (each cloud solution seems to prefer its
-own way) internally a datasource abstract class was created to allow for a
-single way to access the different cloud systems methods to provide this data
-through the typical usage of subclasses.
+instance id, display name and other cloud specific details.
+
+Since there are multiple ways to provide this data (each cloud solution seems
+to prefer its own way) internally a datasource abstract class was created to
+allow for a single way to access the different cloud systems methods to provide
+this data through the typical usage of subclasses.
 
 Any metadata processed by cloud-init's datasources is persisted as
-``/run/cloud0-init/instance-data.json``. Cloud-init provides tooling
-to quickly introspect some of that data. See :ref:`instance_metadata` for
-more information.
+``/run/cloud-init/instance-data.json``. Cloud-init provides tooling to quickly
+introspect some of that data. See :ref:`instance_metadata` for more
+information.
+
+Known Sources
+=============
+
+The following is a list of documents for each supported datasource:
+
+.. toctree::
+   :titlesonly:
+
+   datasources/aliyun.rst
+   datasources/altcloud.rst
+   datasources/azure.rst
+   datasources/cloudsigma.rst
+   datasources/cloudstack.rst
+   datasources/configdrive.rst
+   datasources/digitalocean.rst
+   datasources/e24cloud.rst
+   datasources/ec2.rst
+   datasources/exoscale.rst
+   datasources/fallback.rst
+   datasources/gce.rst
+   datasources/maas.rst
+   datasources/nocloud.rst
+   datasources/opennebula.rst
+   datasources/openstack.rst
+   datasources/oracle.rst
+   datasources/ovf.rst
+   datasources/rbxcloud.rst
+   datasources/smartos.rst
+   datasources/zstack.rst
+
+
+Creation
+========
+
+The datasource objects have a few touch points with cloud-init.  If you
+are interested in adding a new datasource for your cloud platform you will
+need to take care of the following items:
+
+* **Identify a mechanism for positive identification of the platform**:
+  It is good practice for a cloud platform to positively identify itself
+  to the guest.  This allows the guest to make educated decisions based
+  on the platform on which it is running. On the x86 and arm64 architectures,
+  many clouds identify themselves through DMI data.  For example,
+  Oracle's public cloud provides the string 'OracleCloud.com' in the
+  DMI chassis-asset field.
+
+  cloud-init enabled images produce a log file with details about the
+  platform.  Reading through this log in ``/run/cloud-init/ds-identify.log``
+  may provide the information needed to uniquely identify the platform.
+  If the log is not present, you can generate it by running from source
+  ``./tools/ds-identify`` or the installed location
+  ``/usr/lib/cloud-init/ds-identify``.
+
+  The mechanism used to identify the platform will be required for the
+  ds-identify and datasource module sections below.
 
+* **Add datasource module ``cloudinit/sources/DataSource<CloudPlatform>.py``**:
+  It is suggested that you start by copying one of the simpler datasources
+  such as DataSourceHetzner.
+
+* **Add tests for datasource module**:
+  Add a new file with some tests for the module to
+  ``cloudinit/sources/test_<yourplatform>.py``.  For example see
+  ``cloudinit/sources/tests/test_oracle.py``
+
+* **Update ds-identify**:  In systemd systems, ds-identify is used to detect
+  which datasource should be enabled or if cloud-init should run at all.
+  You'll need to make changes to ``tools/ds-identify``.
+
+* **Add tests for ds-identify**: Add relevant tests in a new class to
+  ``tests/unittests/test_ds_identify.py``.  You can use ``TestOracle`` as an
+  example.
+
+* **Add your datasource name to the builtin list of datasources:** Add
+  your datasource module name to the end of the ``datasource_list``
+  entry in ``cloudinit/settings.py``.
+
+* **Add your your cloud platform to apport collection prompts:** Update the
+  list of cloud platforms in ``cloudinit/apport.py``.  This list will be
+  provided to the user who invokes ``ubuntu-bug cloud-init``.
+
+* **Enable datasource by default in ubuntu packaging branches:**
+  Ubuntu packaging branches contain a template file
+  ``debian/cloud-init.templates`` that ultimately sets the default
+  datasource_list when installed via package.  This file needs updating when
+  the commit gets into a package.
+
+* **Add documentation for your datasource**: You should add a new
+  file in ``doc/datasources/<cloudplatform>.rst``
+
+
+API
+===
 
-Datasource API
---------------
 The current interface that a datasource object must provide is the following:
 
 .. sourcecode:: python
 
     # returns a mime multipart message that contains
     # all the various fully-expanded components that
-    # were found from processing the raw userdata string
+    # were found from processing the raw user data string
     # - when filtering only the mime messages targeting
     #   this instance id will be returned (or messages with
     #   no instance id)
@@ -52,7 +140,7 @@ The current interface that a datasource object must provide is the following:
     # because cloud-config content would be handled elsewhere
     def get_config_obj(self)
 
-    #returns a list of public ssh keys
+    # returns a list of public SSH keys
     def get_public_ssh_keys(self)
 
     # translates a device 'short' name into the actual physical device
@@ -73,37 +161,10 @@ The current interface that a datasource object must provide is the following:
     def get_instance_id(self)
 
     # gets the fully qualified domain name that this host should  be using
-    # when configuring network or hostname releated settings, typically
+    # when configuring network or hostname related settings, typically
     # assigned either by the cloud provider or the user creating the vm
     def get_hostname(self, fqdn=False)
 
     def get_package_mirror_info(self)
 
-
-Datasource Documentation
-========================
-The following is a list of the implemented datasources.
-Follow for more information.
-
-.. toctree::
-   :maxdepth: 2
-
-   datasources/aliyun.rst
-   datasources/altcloud.rst
-   datasources/azure.rst
-   datasources/cloudsigma.rst
-   datasources/cloudstack.rst
-   datasources/configdrive.rst
-   datasources/digitalocean.rst
-   datasources/ec2.rst
-   datasources/maas.rst
-   datasources/nocloud.rst
-   datasources/opennebula.rst
-   datasources/openstack.rst
-   datasources/oracle.rst
-   datasources/ovf.rst
-   datasources/smartos.rst
-   datasources/fallback.rst
-   datasources/gce.rst
-
-.. vi: textwidth=78
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/datasources/altcloud.rst b/doc/rtd/topics/datasources/altcloud.rst
index eeb197f2..9d7e3de1 100644
--- a/doc/rtd/topics/datasources/altcloud.rst
+++ b/doc/rtd/topics/datasources/altcloud.rst
@@ -3,24 +3,25 @@
 Alt Cloud
 =========
 
-The datasource altcloud will be used to pick up user data on `RHEVm`_ and `vSphere`_.
+The datasource altcloud will be used to pick up user data on `RHEVm`_ and
+`vSphere`_.
 
 RHEVm
 -----
 
 For `RHEVm`_ v3.0 the userdata is injected into the VM using floppy
-injection via the `RHEVm`_ dashboard "Custom Properties". 
+injection via the `RHEVm`_ dashboard "Custom Properties".
 
 The format of the Custom Properties entry must be:
 
 ::
-    
+
     floppyinject=user-data.txt:<base64 encoded data>
 
 For example to pass a simple bash script:
 
 .. sourcecode:: sh
-    
+
     % cat simple_script.bash
     #!/bin/bash
     echo "Hello Joe!" >> /tmp/JJV_Joe_out.txt
@@ -38,7 +39,7 @@ set the "Custom Properties" when creating the RHEMv v3.0 VM to:
 **NOTE:** The prefix with file name must be: ``floppyinject=user-data.txt:``
 
 It is also possible to launch a `RHEVm`_ v3.0 VM and pass optional user
-data to it using the Delta Cloud. 
+data to it using the Delta Cloud.
 
 For more information on Delta Cloud see: http://deltacloud.apache.org
 
@@ -46,12 +47,12 @@ vSphere
 -------
 
 For VMWare's `vSphere`_ the userdata is injected into the VM as an ISO
-via the cdrom. This can be done using the `vSphere`_ dashboard 
+via the cdrom. This can be done using the `vSphere`_ dashboard
 by connecting an ISO image to the CD/DVD drive.
 
 To pass this example script to cloud-init running in a `vSphere`_ VM
 set the CD/DVD drive when creating the vSphere VM to point to an
-ISO on the data store. 
+ISO on the data store.
 
 **Note:** The ISO must contain the user data.
 
@@ -61,13 +62,13 @@ Create the ISO
 ^^^^^^^^^^^^^^
 
 .. sourcecode:: sh
-    
+
     % mkdir my-iso
 
 NOTE: The file name on the ISO must be: ``user-data.txt``
 
 .. sourcecode:: sh
-    
+
     % cp simple_script.bash my-iso/user-data.txt
     % genisoimage -o user-data.iso -r my-iso
 
@@ -75,7 +76,7 @@ Verify the ISO
 ^^^^^^^^^^^^^^
 
 .. sourcecode:: sh
-    
+
     % sudo mkdir /media/vsphere_iso
     % sudo mount -o loop user-data.iso /media/vsphere_iso
     % cat /media/vsphere_iso/user-data.txt
@@ -84,7 +85,7 @@ Verify the ISO
 Then, launch the `vSphere`_ VM the ISO user-data.iso attached as a CDROM.
 
 It is also possible to launch a `vSphere`_ VM and pass optional user
-data to it using the Delta Cloud. 
+data to it using the Delta Cloud.
 
 For more information on Delta Cloud see: http://deltacloud.apache.org
 
diff --git a/doc/rtd/topics/datasources/azure.rst b/doc/rtd/topics/datasources/azure.rst
index f73c3694..1427fb3d 100644
--- a/doc/rtd/topics/datasources/azure.rst
+++ b/doc/rtd/topics/datasources/azure.rst
@@ -5,9 +5,30 @@ Azure
 
 This datasource finds metadata and user-data from the Azure cloud platform.
 
-Azure Platform
---------------
-The azure cloud-platform provides initial data to an instance via an attached
+walinuxagent
+------------
+walinuxagent has several functions within images.  For cloud-init
+specifically, the relevant functionality it performs is to register the
+instance with the Azure cloud platform at boot so networking will be
+permitted.  For more information about the other functionality of
+walinuxagent, see `Azure's documentation
+<https://github.com/Azure/WALinuxAgent#introduction>`_ for more details.
+(Note, however, that only one of walinuxagent's provisioning and cloud-init
+should be used to perform instance customisation.)
+
+If you are configuring walinuxagent yourself, you will want to ensure that you
+have `Provisioning.UseCloudInit
+<https://github.com/Azure/WALinuxAgent#provisioningusecloudinit>`_ set to
+``y``.
+
+
+Builtin Agent
+-------------
+An alternative to using walinuxagent to register to the Azure cloud platform
+is to use the ``__builtin__`` agent command.  This section contains more
+background on what that code path does, and how to enable it.
+
+The Azure cloud platform provides initial data to an instance via an attached
 CD formatted in UDF.  That CD contains a 'ovf-env.xml' file that provides some
 information.  Additional information is obtained via interaction with the
 "endpoint".
@@ -23,44 +44,36 @@ information in json format to /run/cloud-init/dhclient.hook/<interface>.json.
 In order for cloud-init to leverage this method to find the endpoint, the
 cloud.cfg file must contain:
 
-datasource:
-  Azure:
-    set_hostname: False
-    agent_command: __builtin__
+.. sourcecode:: yaml
+
+  datasource:
+    Azure:
+      set_hostname: False
+      agent_command: __builtin__
 
 If those files are not available, the fallback is to check the leases file
 for the endpoint server (again option 245).
 
 You can define the path to the lease file with the 'dhclient_lease_file'
-configuration.  The default value is /var/lib/dhcp/dhclient.eth0.leases.
+configuration.
 
-    dhclient_lease_file: /var/lib/dhcp/dhclient.eth0.leases
-
-walinuxagent
-------------
-In order to operate correctly, cloud-init needs walinuxagent to provide much
-of the interaction with azure.  In addition to "provisioning" code, walinux
-does the following on the agent is a long running daemon that handles the
-following things:
-- generate a x509 certificate and send that to the endpoint
 
-waagent.conf config
-^^^^^^^^^^^^^^^^^^^
-in order to use waagent.conf with cloud-init, the following settings are recommended.  Other values can be changed or set to the defaults.
+IMDS
+----
+Azure provides the `instance metadata service (IMDS)
+<https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service>`_
+which is a REST service on ``169.254.169.254`` providing additional
+configuration information to the instance. Cloud-init uses the IMDS for:
 
-  ::
-
-   # disabling provisioning turns off all 'Provisioning.*' function
-   Provisioning.Enabled=n
-   # this is currently not handled by cloud-init, so let walinuxagent do it.
-   ResourceDisk.Format=y
-   ResourceDisk.MountPoint=/mnt
+- network configuration for the instance which is applied per boot
+- a preprovisioing gate which blocks instance configuration until Azure fabric
+  is ready to provision
 
 
 Configuration
 -------------
 The following configuration can be set for the datasource in system
-configuration (in `/etc/cloud/cloud.cfg` or `/etc/cloud/cloud.cfg.d/`).
+configuration (in ``/etc/cloud/cloud.cfg`` or ``/etc/cloud/cloud.cfg.d/``).
 
 The settings that may be configured are:
 
@@ -69,20 +82,33 @@ The settings that may be configured are:
    provided command to obtain metadata.
  * **apply_network_config**: Boolean set to True to use network configuration
    described by Azure's IMDS endpoint instead of fallback network config of
-   dhcp on eth0. Default is True. For Ubuntu 16.04 or earlier, default is False.
+   dhcp on eth0. Default is True. For Ubuntu 16.04 or earlier, default is
+   False.
  * **data_dir**: Path used to read metadata files and write crawled data.
  * **dhclient_lease_file**: The fallback lease file to source when looking for
    custom DHCP option 245 from Azure fabric.
  * **disk_aliases**: A dictionary defining which device paths should be
    interpreted as ephemeral images. See cc_disk_setup module for more info.
  * **hostname_bounce**: A dictionary Azure hostname bounce behavior to react to
-   metadata changes.
+   metadata changes.  The '``hostname_bounce: command``' entry can be either
+   the literal string 'builtin' or a command to execute.  The command will be
+   invoked after the hostname is set, and will have the 'interface' in its
+   environment.  If ``set_hostname`` is not true, then ``hostname_bounce``
+   will be ignored.  An example might be:
+
+     ``command:  ["sh", "-c", "killall dhclient; dhclient $interface"]``
+
  * **hostname_bounce**: A dictionary Azure hostname bounce behavior to react to
    metadata changes. Azure will throttle ifup/down in some cases after metadata
    has been updated to inform dhcp server about updated hostnames.
  * **set_hostname**: Boolean set to True when we want Azure to set the hostname
    based on metadata.
 
+Configuration for the datasource can also be read from a
+``dscfg`` entry in the ``LinuxProvisioningConfigurationSet``.  Content in
+dscfg node is expected to be base64 encoded yaml content, and it will be
+merged into the 'datasource: Azure' entry.
+
 An example configuration with the default values is provided below:
 
 .. sourcecode:: yaml
@@ -143,37 +169,6 @@ Example:
    </LinuxProvisioningConfigurationSet>
  </wa:ProvisioningSection>
 
-Configuration
--------------
-Configuration for the datasource can be read from the system config's or set
-via the `dscfg` entry in the `LinuxProvisioningConfigurationSet`.  Content in
-dscfg node is expected to be base64 encoded yaml content, and it will be
-merged into the 'datasource: Azure' entry.
-
-The '``hostname_bounce: command``' entry can be either the literal string
-'builtin' or a command to execute.  The command will be invoked after the
-hostname is set, and will have the 'interface' in its environment.  If
-``set_hostname`` is not true, then ``hostname_bounce`` will be ignored.
-
-An example might be:
-  command:  ["sh", "-c", "killall dhclient; dhclient $interface"]
-
-.. code:: yaml
-
-  datasource:
-   agent_command
-   Azure:
-    agent_command: [service, walinuxagent, start]
-    set_hostname: True
-    hostname_bounce:
-     # the name of the interface to bounce
-     interface: eth0
-     # policy can be 'on', 'off' or 'force'
-     policy: on
-     # the method 'bounce' command.
-     command: "builtin"
-     hostname_command: "hostname"
-
 hostname
 --------
 When the user launches an instance, they provide a hostname for that instance.
diff --git a/doc/rtd/topics/datasources/cloudstack.rst b/doc/rtd/topics/datasources/cloudstack.rst
index a3101ed7..da183226 100644
--- a/doc/rtd/topics/datasources/cloudstack.rst
+++ b/doc/rtd/topics/datasources/cloudstack.rst
@@ -4,10 +4,10 @@ CloudStack
 ==========
 
 `Apache CloudStack`_ expose user-data, meta-data, user password and account
-sshkey thru the Virtual-Router. The datasource obtains the VR address via
+SSH key thru the Virtual-Router. The datasource obtains the VR address via
 dhcp lease information given to the instance.
 For more details on meta-data and user-data,
-refer the `CloudStack Administrator Guide`_. 
+refer the `CloudStack Administrator Guide`_.
 
 URLs to access user-data and meta-data from the Virtual Machine. Here 10.1.1.1
 is the Virtual Router IP:
diff --git a/doc/rtd/topics/datasources/configdrive.rst b/doc/rtd/topics/datasources/configdrive.rst
index f1a488a2..4fcbccee 100644
--- a/doc/rtd/topics/datasources/configdrive.rst
+++ b/doc/rtd/topics/datasources/configdrive.rst
@@ -6,7 +6,7 @@ Config Drive
 The configuration drive datasource supports the `OpenStack`_ configuration
 drive disk.
 
-  See `the config drive extension`_ and `introduction`_ in the public
+  See `the config drive extension`_ and `metadata introduction`_ in the public
   documentation for more information.
 
 By default, cloud-init does *always* consider this source to be a full-fledged
@@ -64,7 +64,7 @@ The following criteria are required to as a config drive:
 ::
 
   openstack/
-    - 2012-08-10/ or latest/ 
+    - 2012-08-10/ or latest/
       - meta_data.json
       - user_data (not mandatory)
     - content/
@@ -83,7 +83,7 @@ only) file in the following ways.
 
 ::
 
-   dsmode:  
+   dsmode:
      values: local, net, pass
      default: pass
 
@@ -97,10 +97,10 @@ The difference between 'local' and 'net' is that local will not require
 networking to be up before user-data actions (or boothooks) are run.
 
 ::
-    
+
    instance-id:
      default: iid-dsconfigdrive
-     
+
 This is utilized as the metadata's instance-id.  It should generally
 be unique, as it is what is used to determine "is this a new instance".
 
@@ -108,24 +108,24 @@ be unique, as it is what is used to determine "is this a new instance".
 
    public-keys:
      default: None
-  
+
 If present, these keys will be used as the public keys for the
 instance.  This value overrides the content in authorized_keys.
 
 Note: it is likely preferable to provide keys via user-data
 
 ::
-    
+
    user-data:
      default: None
-     
-This provides cloud-init user-data. See :ref:`examples <yaml_examples>` for 
+
+This provides cloud-init user-data. See :ref:`examples <yaml_examples>` for
 what all can be present here.
 
 .. _OpenStack: http://www.openstack.org/
-.. _introduction: http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html
+.. _metadata introduction: https://docs.openstack.org/nova/latest/user/metadata.html#config-drives
 .. _python-novaclient: https://github.com/openstack/python-novaclient
 .. _iso9660: https://en.wikipedia.org/wiki/ISO_9660
 .. _vfat: https://en.wikipedia.org/wiki/File_Allocation_Table
-.. _the config drive extension: http://docs.openstack.org/user-guide/content/config-drive.html
+.. _the config drive extension: https://docs.openstack.org/nova/latest/admin/config-drive.html
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/datasources/digitalocean.rst b/doc/rtd/topics/datasources/digitalocean.rst
index 938ede89..88f1e5f5 100644
--- a/doc/rtd/topics/datasources/digitalocean.rst
+++ b/doc/rtd/topics/datasources/digitalocean.rst
@@ -20,8 +20,10 @@ DigitalOcean's datasource can be configured as follows:
       retries: 3
       timeout: 2
 
-- *retries*: Determines the number of times to attempt to connect to the metadata service
-- *timeout*: Determines the timeout in seconds to wait for a response from the metadata service
+- *retries*: Determines the number of times to attempt to connect to the
+  metadata service
+- *timeout*: Determines the timeout in seconds to wait for a response from the
+  metadata service
 
 .. _DigitalOcean: http://digitalocean.com/
 .. _metadata service: https://developers.digitalocean.com/metadata/
diff --git a/doc/rtd/topics/datasources/e24cloud.rst b/doc/rtd/topics/datasources/e24cloud.rst
new file mode 100644
index 00000000..de9a4127
--- /dev/null
+++ b/doc/rtd/topics/datasources/e24cloud.rst
@@ -0,0 +1,9 @@
+.. _datasource_e24cloud:
+
+E24Cloud
+========
+`E24Cloud <https://www.e24cloud.com/en/>` platform provides an AWS Ec2 metadata
+service clone.  It identifies itself to guests using the dmi
+system-manufacturer (/sys/class/dmi/id/sys_vendor).
+
+.. vi: textwidth=78
diff --git a/doc/rtd/topics/datasources/ec2.rst b/doc/rtd/topics/datasources/ec2.rst
index 64c325d8..a90f3779 100644
--- a/doc/rtd/topics/datasources/ec2.rst
+++ b/doc/rtd/topics/datasources/ec2.rst
@@ -13,7 +13,7 @@ instance metadata.
 Metadata is accessible via the following URL:
 
 ::
-    
+
     GET http://169.254.169.254/2009-04-04/meta-data/
     ami-id
     ami-launch-index
@@ -34,19 +34,20 @@ Metadata is accessible via the following URL:
 Userdata is accessible via the following URL:
 
 ::
-    
+
     GET http://169.254.169.254/2009-04-04/user-data
     1234,fred,reboot,true | 4512,jimbo, | 173,,,
 
 Note that there are multiple versions of this data provided, cloud-init
 by default uses **2009-04-04** but newer versions can be supported with
 relative ease (newer versions have more data exposed, while maintaining
-backward compatibility with the previous versions). 
+backward compatibility with the previous versions).
 
-To see which versions are supported from your cloud provider use the following URL:
+To see which versions are supported from your cloud provider use the following
+URL:
 
 ::
-    
+
     GET http://169.254.169.254/
     1.0
     2007-01-19
@@ -90,4 +91,15 @@ An example configuration with the default values is provided below:
     max_wait: 120
     timeout: 50
 
+Notes
+-----
+ * There are 2 types of EC2 instances network-wise: VPC ones (Virtual Private
+   Cloud) and Classic ones (also known as non-VPC). One major difference
+   between them is that Classic instances have their MAC address changed on
+   stop/restart operations, so cloud-init will recreate the network config
+   file for EC2 Classic instances every boot. On VPC instances this file is
+   generated only in the first boot of the instance.
+   The check for the instance type is performed by is_classic_instance()
+   method.
+
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/datasources/exoscale.rst b/doc/rtd/topics/datasources/exoscale.rst
new file mode 100644
index 00000000..9074edc6
--- /dev/null
+++ b/doc/rtd/topics/datasources/exoscale.rst
@@ -0,0 +1,68 @@
+.. _datasource_exoscale:
+
+Exoscale
+========
+
+This datasource supports reading from the metadata server used on the
+`Exoscale platform <https://exoscale.com>`_.
+
+Use of the Exoscale datasource is recommended to benefit from new features of
+the Exoscale platform.
+
+The datasource relies on the availability of a compatible metadata server
+(``http://169.254.169.254`` is used by default) and its companion password
+server, reachable at the same address (by default on port 8080).
+
+Crawling of metadata
+--------------------
+
+The metadata service and password server are crawled slightly differently:
+
+ * The "metadata service" is crawled every boot.
+ * The password server is also crawled every boot (the Exoscale datasource
+   forces the password module to run with "frequency always").
+
+In the password server case, the following rules apply in order to enable the
+"restore instance password" functionality:
+
+ * If a password is returned by the password server, it is then marked "saved"
+   by the cloud-init datasource. Subsequent boots will skip setting the
+   password (the password server will return "saved_password").
+ * When the instance password is reset (via the Exoscale UI), the password
+   server will return the non-empty password at next boot, therefore causing
+   cloud-init to reset the instance's password.
+
+Configuration
+-------------
+
+Users of this datasource are discouraged from changing the default settings
+unless instructed to by Exoscale support.
+
+The following settings are available and can be set for the datasource in
+system configuration (in `/etc/cloud/cloud.cfg.d/`).
+
+The settings available are:
+
+ * **metadata_url**: The URL for the metadata service (defaults to
+   ``http://169.254.169.254``)
+ * **api_version**: The API version path on which to query the instance
+   metadata (defaults to ``1.0``)
+ * **password_server_port**: The port (on the metadata server) on which the
+   password server listens (defaults to ``8080``).
+ * **timeout**: the timeout value provided to urlopen for each individual http
+   request. (defaults to ``10``)
+ * **retries**: The number of retries that should be done for an http request
+   (defaults to ``6``)
+
+
+An example configuration with the default values is provided below:
+
+.. sourcecode:: yaml
+
+   datasource:
+     Exoscale:
+       metadata_url: "http://169.254.169.254"
+       api_version: "1.0"
+       password_server_port: 8080
+       timeout: 10
+       retries: 6
diff --git a/doc/rtd/topics/datasources/nocloud.rst b/doc/rtd/topics/datasources/nocloud.rst
index 08578e86..bc96f7fe 100644
--- a/doc/rtd/topics/datasources/nocloud.rst
+++ b/doc/rtd/topics/datasources/nocloud.rst
@@ -9,7 +9,7 @@ network at all).
 
 You can provide meta-data and user-data to a local vm boot via files on a
 `vfat`_ or `iso9660`_ filesystem. The filesystem volume label must be
-``cidata``.
+``cidata`` or ``CIDATA``.
 
 Alternatively, you can provide meta-data via kernel command line or SMBIOS
 "serial number" option. The data must be passed in the form of a string:
@@ -57,24 +57,24 @@ Given a disk ubuntu 12.04 cloud image in 'disk.img', you can create a
 sufficient disk by following the example below.
 
 ::
-    
+
     ## create user-data and meta-data files that will be used
     ## to modify image on first boot
     $ { echo instance-id: iid-local01; echo local-hostname: cloudimg; } > meta-data
-    
+
     $ printf "#cloud-config\npassword: passw0rd\nchpasswd: { expire: False }\nssh_pwauth: True\n" > user-data
-    
+
     ## create a disk to attach with some user-data and meta-data
     $ genisoimage  -output seed.iso -volid cidata -joliet -rock user-data meta-data
-    
+
     ## alternatively, create a vfat filesystem with same files
     ## $ truncate --size 2M seed.img
     ## $ mkfs.vfat -n cidata seed.img
     ## $ mcopy -oi seed.img user-data meta-data ::
-    
+
     ## create a new qcow image to boot, backed by your original image
     $ qemu-img create -f qcow2 -b disk.img boot-disk.img
-    
+
     ## boot the image and login as 'ubuntu' with password 'passw0rd'
     ## note, passw0rd was set as password through the user-data above,
     ## there is no password set on these images.
@@ -88,12 +88,12 @@ to determine if this is "first boot".  So if you are making updates to
 user-data you will also have to change that, or start the disk fresh.
 
 Also, you can inject an ``/etc/network/interfaces`` file by providing the
-content for that file in the ``network-interfaces`` field of metadata.  
+content for that file in the ``network-interfaces`` field of metadata.
 
 Example metadata:
 
 ::
-    
+
     instance-id: iid-abcdefg
     network-interfaces: |
       iface eth0 inet static
diff --git a/doc/rtd/topics/datasources/opennebula.rst b/doc/rtd/topics/datasources/opennebula.rst
index 7c0367c4..8e7c2558 100644
--- a/doc/rtd/topics/datasources/opennebula.rst
+++ b/doc/rtd/topics/datasources/opennebula.rst
@@ -21,7 +21,7 @@ Datasource configuration
 Datasource accepts following configuration options.
 
 ::
-    
+
     dsmode:
       values: local, net, disabled
       default: net
@@ -30,7 +30,7 @@ Tells if this datasource will be processed in 'local' (pre-networking) or
 'net' (post-networking) stage or even completely 'disabled'.
 
 ::
-    
+
     parseuser:
       default: nobody
 
@@ -46,7 +46,7 @@ The following criteria are required:
    or have a *filesystem* label of **CONTEXT** or **CDROM**
 2. Must contain file *context.sh* with contextualization variables.
    File is generated by OpenNebula, it has a KEY='VALUE' format and
-   can be easily read by bash 
+   can be easily read by bash
 
 Contextualization variables
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -57,7 +57,7 @@ the OpenNebula documentation. Where multiple similar variables are
 specified, only first found is taken.
 
 ::
-    
+
     DSMODE
 
 Datasource mode configuration override. Values: local, net, disabled.
@@ -75,30 +75,30 @@ Datasource mode configuration override. Values: local, net, disabled.
 Static `network configuration`_.
 
 ::
-    
+
     HOSTNAME
 
 Instance hostname.
 
 ::
-    
+
     PUBLIC_IP
     IP_PUBLIC
     ETH0_IP
 
 If no hostname has been specified, cloud-init will try to create hostname
-from instance's IP address in 'local' dsmode. In 'net' dsmode, cloud-init 
+from instance's IP address in 'local' dsmode. In 'net' dsmode, cloud-init
 tries to resolve one of its IP addresses to get hostname.
 
 ::
-    
+
     SSH_KEY
     SSH_PUBLIC_KEY
 
 One or multiple SSH keys (separated by newlines) can be specified.
 
 ::
-    
+
     USER_DATA
     USERDATA
 
@@ -111,7 +111,7 @@ This example cloud-init configuration (*cloud.cfg*) enables
 OpenNebula datasource only in 'net' mode.
 
 ::
-    
+
     disable_ec2_metadata: True
     datasource_list: ['OpenNebula']
     datasource:
@@ -123,17 +123,17 @@ Example VM's context section
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
-    
+
     CONTEXT=[
       PUBLIC_IP="$NIC[IP]",
-      SSH_KEY="$USER[SSH_KEY] 
-    $USER[SSH_KEY1] 
+      SSH_KEY="$USER[SSH_KEY]
+    $USER[SSH_KEY1]
     $USER[SSH_KEY2] ",
       USER_DATA="#cloud-config
     # see https://help.ubuntu.com/community/CloudInit
-    
+
     packages: []
-    
+
     mounts:
     - [vdc,none,swap,sw,0,0]
     runcmd:
diff --git a/doc/rtd/topics/datasources/openstack.rst b/doc/rtd/topics/datasources/openstack.rst
index 421da08f..8ce2a53d 100644
--- a/doc/rtd/topics/datasources/openstack.rst
+++ b/doc/rtd/topics/datasources/openstack.rst
@@ -78,6 +78,7 @@ upgrade packages and install ``htop`` on all instances:
   {"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}
 
 For more general information about how cloud-init handles vendor data,
-including how it can be disabled by users on instances, see :doc:`/topics/vendordata`.
+including how it can be disabled by users on instances, see
+:doc:`/topics/vendordata`.
 
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/datasources/oracle.rst b/doc/rtd/topics/datasources/oracle.rst
index f2383cee..98c4657c 100644
--- a/doc/rtd/topics/datasources/oracle.rst
+++ b/doc/rtd/topics/datasources/oracle.rst
@@ -8,7 +8,7 @@ This datasource reads metadata, vendor-data and user-data from
 
 Oracle Platform
 ---------------
-OCI provides bare metal and virtual machines.  In both cases, 
+OCI provides bare metal and virtual machines.  In both cases,
 the platform identifies itself via DMI data in the chassis asset tag
 with the string 'OracleCloud.com'.
 
@@ -22,5 +22,28 @@ Cloud-init has a specific datasource for Oracle in order to:
     implementation.
 
 
+Configuration
+-------------
+
+The following configuration can be set for the datasource in system
+configuration (in ``/etc/cloud/cloud.cfg`` or ``/etc/cloud/cloud.cfg.d/``).
+
+The settings that may be configured are:
+
+* **configure_secondary_nics**: A boolean, defaulting to False.  If set
+  to True on an OCI Virtual Machine, cloud-init will fetch networking
+  metadata from Oracle's IMDS and use it to configure the non-primary
+  network interface controllers in the system.  If set to True on an
+  OCI Bare Metal Machine, it will have no effect (though this may
+  change in the future).
+
+An example configuration with the default values is provided below:
+
+.. sourcecode:: yaml
+
+  datasource:
+   Oracle:
+    configure_secondary_nics: false
+
 .. _Oracle Compute Infrastructure: https://cloud.oracle.com/
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/datasources/rbxcloud.rst b/doc/rtd/topics/datasources/rbxcloud.rst
new file mode 100644
index 00000000..52ec02ff
--- /dev/null
+++ b/doc/rtd/topics/datasources/rbxcloud.rst
@@ -0,0 +1,25 @@
+.. _datasource_rbx:
+
+Rbx Cloud
+=========
+
+The Rbx datasource consumes the metadata drive available on platform
+`HyperOne`_ and `Rootbox`_ platform.
+
+Datasource supports, in particular, network configurations, hostname,
+user accounts and user metadata.
+
+Metadata drive
+--------------
+
+Drive metadata is a `FAT`_-formatted partition with the ```CLOUDMD``` label on
+the system disk. Its contents are refreshed each time the virtual machine
+is restarted, if the partition exists. For more information see
+`HyperOne Virtual Machine docs`_.
+
+.. _HyperOne: http://www.hyperone.com/
+.. _Rootbox: https://rootbox.com/
+.. _HyperOne Virtual Machine docs: http://www.hyperone.com/
+.. _FAT: https://en.wikipedia.org/wiki/File_Allocation_Table
+
+.. vi: textwidth=78
diff --git a/doc/rtd/topics/datasources/smartos.rst b/doc/rtd/topics/datasources/smartos.rst
index cb9a128e..be11dfbb 100644
--- a/doc/rtd/topics/datasources/smartos.rst
+++ b/doc/rtd/topics/datasources/smartos.rst
@@ -15,7 +15,8 @@ second serial console. On Linux, this is /dev/ttyS1. The data is a provided
 via a simple protocol: something queries for the data, the console responds
 responds with the status and if "SUCCESS" returns until a single ".\n".
 
-New versions of the SmartOS tooling will include support for base64 encoded data.
+New versions of the SmartOS tooling will include support for base64 encoded
+data.
 
 Meta-data channels
 ------------------
@@ -27,7 +28,7 @@ channels of SmartOS.
 
   - per the spec, user-data is for consumption by the end-user, not
     provisioning tools
-  - cloud-init entirely ignores this channel other than writting it to disk
+  - cloud-init entirely ignores this channel other than writing it to disk
   - removal of the meta-data key means that /var/db/user-data gets removed
   - a backup of previous meta-data is maintained as
     /var/db/user-data.<timestamp>. <timestamp> is the epoch time when
@@ -42,8 +43,9 @@ channels of SmartOS.
     - <timestamp> is the epoch time when cloud-init ran.
   - when the 'user-script' meta-data key goes missing, the user-script is
     removed from the file system, although a backup is maintained.
-  - if the script is not shebanged (i.e. starts with #!<executable>), then
-    or is not an executable, cloud-init will add a shebang of "#!/bin/bash"
+  - if the script does not start with a shebang (i.e. starts with
+    #!<executable>), then or is not an executable, cloud-init will add a
+    shebang of "#!/bin/bash"
 
 * cloud-init:user-data is treated like on other Clouds.
 
@@ -133,7 +135,7 @@ or not to base64 decode something:
   * base64_all: Except for excluded keys, attempt to base64 decode
     the values. If the value fails to decode properly, it will be
     returned in its text
-  * base64_keys: A comma deliminated list of which keys are base64 encoded.
+  * base64_keys: A comma delimited list of which keys are base64 encoded.
   * b64-<key>:
     for any key, if there exists an entry in the metadata for 'b64-<key>'
     Then 'b64-<key>' is expected to be a plaintext boolean indicating whether
diff --git a/doc/rtd/topics/datasources/zstack.rst b/doc/rtd/topics/datasources/zstack.rst
new file mode 100644
index 00000000..93a2791c
--- /dev/null
+++ b/doc/rtd/topics/datasources/zstack.rst
@@ -0,0 +1,37 @@
+.. _datasource_zstack:
+
+ZStack
+======
+ZStack platform provides a AWS Ec2 metadata service, but with different
+datasource identity.
+More information about ZStack can be found at `ZStack <https://www.zstack.io>`__.
+
+Discovery
+---------
+To determine whether a vm running on ZStack platform, cloud-init checks DMI
+information by 'dmidecode -s chassis-asset-tag', if the output ends with
+'.zstack.io', it's running on ZStack platform:
+
+
+Metadata
+^^^^^^^^
+Same as EC2, instance metadata can be queried at
+
+::
+
+    GET http://169.254.169.254/2009-04-04/meta-data/
+    instance-id
+    local-hostname
+
+Userdata
+^^^^^^^^
+Same as EC2, instance userdata can be queried at
+
+::
+
+    GET http://169.254.169.254/2009-04-04/user-data/
+    meta_data.json
+    user_data
+    password
+
+.. vi: textwidth=78
diff --git a/doc/rtd/topics/debugging.rst b/doc/rtd/topics/debugging.rst
index 51363ea5..0d416f32 100644
--- a/doc/rtd/topics/debugging.rst
+++ b/doc/rtd/topics/debugging.rst
@@ -68,6 +68,19 @@ subcommands default to reading /var/log/cloud-init.log.
          00.00100s (modules-final/config-rightscale_userdata)
          ...
 
+* ``analyze boot`` Make subprocess calls to the kernel in order to get relevant
+  pre-cloud-init timestamps, such as the kernel start, kernel finish boot, and cloud-init start.
+
+.. code-block:: shell-session
+
+    $ cloud-init analyze boot
+    -- Most Recent Boot Record --
+        Kernel Started at: 2019-06-13 15:59:55.809385
+        Kernel ended boot at: 2019-06-13 16:00:00.944740
+        Kernel time to boot (seconds): 5.135355
+        Cloud-init start: 2019-06-13 16:00:05.738396
+        Time between Kernel boot and Cloud-init start (seconds): 4.793656
+
 
 Analyze quickstart - LXC
 ---------------------------
@@ -150,3 +163,104 @@ commandline:
 
 Inspect cloud-init.log for output of what operations were performed as a
 result.
+
+.. _proposed_sru_testing:
+
+Stable Release Updates (SRU) testing for cloud-init
+===================================================
+Once an Ubuntu release is stable (i.e. after it is released), updates for it
+must follow a special procedure called a "stable release update" (or `SRU`_).
+
+The cloud-init project has a specific process it follows when validating
+a cloud-init SRU, documented in the `CloudinitUpdates`_ wiki page.
+
+Generally an SRU test of cloud-init performs the following:
+
+ * Install a pre-release version of cloud-init from the
+   **-proposed** APT pocket (e.g. **bionic-proposed**)
+ * Upgrade cloud-init and attempt a clean run of cloud-init to assert the new
+   version of cloud-init works properly the specific platform and Ubuntu series
+ * Check for tracebacks or errors in behavior
+
+
+Manual SRU verification procedure
+---------------------------------
+Below are steps to manually test a pre-release version of cloud-init
+from **-proposed**
+
+.. note::
+    For each Ubuntu SRU, the Ubuntu Server team manually validates the new version of cloud-init
+    on these platforms: **Amazon EC2, Azure, GCE, OpenStack, Oracle,
+    Softlayer (IBM), LXD, KVM**
+
+1. Launch a VM on your favorite platform, providing this cloud-config
+   user-data and replacing `<YOUR_LAUNCHPAD_USERNAME>` with your username:
+
+.. code-block:: yaml
+
+    ## template: jinja
+    #cloud-config
+    ssh_import_id: [<YOUR_LAUNCHPAD_USERNAME>]
+    hostname: SRU-worked-{{v1.cloud_name}}
+
+2. Wait for current cloud-init to complete, replace `<YOUR_VM_IP>` with the IP
+   address of the VM that you launched in step 1:
+
+.. code-block:: bash
+
+    CI_VM_IP=<YOUR_VM_IP>
+    # Make note of the datasource cloud-init detected in --long output.
+    # In step 5, you will use this to confirm the same datasource is detected after upgrade.
+    ssh ubuntu@$CI_VM_IP -- cloud-init status --wait --long
+
+3. Set up the **-proposed** pocket on your VM and upgrade to the **-proposed**
+   cloud-init:
+
+.. code-block:: bash
+
+    # Create a script that will add the -proposed pocket to APT's sources
+    # and install cloud-init from that pocket
+    cat > setup_proposed.sh <<EOF
+    #/bin/bash
+    mirror=http://archive.ubuntu.com/ubuntu
+    echo deb \$mirror \$(lsb_release -sc)-proposed main | tee \
+        /etc/apt/sources.list.d/proposed.list
+    apt-get update -q
+    apt-get install -qy cloud-init
+    EOF
+
+    scp setup_proposed.sh ubuntu@$CI_VM_IP:.
+    ssh ubuntu@$CI_VM_IP -- sudo bash setup_proposed.sh
+
+4. Change hostname, clean cloud-init's state, and reboot to run cloud-init
+   from scratch:
+
+.. code-block:: bash
+
+    ssh ubuntu@$CI_VM_IP -- sudo hostname something-else
+    ssh ubuntu@$CI_VM_IP -- sudo cloud-init clean --logs --reboot
+
+5. Validate **-proposed** cloud-init came up without error
+
+.. code-block:: bash
+
+    # Block until cloud-init completes and verify from --long the datasource
+    # from step 1. Errors would show up in --long
+
+    ssh ubuntu@$CI_VM_IP -- cloud-init status --wait --long
+    # Make sure hostname was set properly to SRU-worked-<cloud name>
+    ssh ubuntu@$CI_VM_IP -- hostname
+    # Check for any errors or warnings in cloud-init logs.
+    # (This should produce no output if successful.)
+    ssh ubuntu@$CI_VM_IP -- grep Trace "/var/log/cloud-init*"
+
+6. If you encounter an error during SRU testing:
+
+   * Create a `new cloud-init bug`_ reporting the version of cloud-init
+     affected
+   * Ping upstream cloud-init on Freenode's `#cloud-init IRC channel`_
+
+.. _SRU: https://wiki.ubuntu.com/StableReleaseUpdates
+.. _CloudinitUpdates: https://wiki.ubuntu.com/CloudinitUpdates
+.. _new cloud-init bug: https://bugs.launchpad.net/cloud-init/+filebug
+.. _#cloud-init IRC channel: https://webchat.freenode.net/?channel=#cloud-init
diff --git a/doc/rtd/topics/dir_layout.rst b/doc/rtd/topics/dir_layout.rst
index 7a6265eb..ebd63ae7 100644
--- a/doc/rtd/topics/dir_layout.rst
+++ b/doc/rtd/topics/dir_layout.rst
@@ -2,11 +2,12 @@
 Directory layout
 ****************
 
-Cloudinits's directory structure is somewhat different from a regular application::
+Cloud-init's directory structure is somewhat different from a regular
+application::
 
   /var/lib/cloud/
       - data/
-         - instance-id  
+         - instance-id
          - previous-instance-id
          - datasource
          - previous-datasource
@@ -35,38 +36,41 @@ Cloudinits's directory structure is somewhat different from a regular applicatio
 
   The main directory containing the cloud-init specific subdirectories.
   It is typically located at ``/var/lib`` but there are certain configuration
-  scenarios where this can be altered. 
+  scenarios where this can be altered.
 
   TBD, describe this overriding more.
 
 ``data/``
 
-  Contains information related to instance ids, datasources and hostnames of the previous
-  and current instance if they are different. These can be examined as needed to
-  determine any information related to a previous boot (if applicable).
+  Contains information related to instance ids, datasources and hostnames of
+  the previous and current instance if they are different. These can be
+  examined as needed to determine any information related to a previous boot
+  (if applicable).
 
 ``handlers/``
 
-  Custom ``part-handlers`` code is written out here. Files that end up here are written
-  out with in the scheme of ``part-handler-XYZ`` where ``XYZ`` is the handler number (the
-  first handler found starts at 0).
+  Custom ``part-handlers`` code is written out here. Files that end up here are
+  written out with in the scheme of ``part-handler-XYZ`` where ``XYZ`` is the
+  handler number (the first handler found starts at 0).
 
 
 ``instance``
 
-  A symlink to the current ``instances/`` subdirectory that points to the currently
-  active instance (which is active is dependent on the datasource loaded).
+  A symlink to the current ``instances/`` subdirectory that points to the
+  currently active instance (which is active is dependent on the datasource
+  loaded).
 
 ``instances/``
 
-  All instances that were created using this image end up with instance identifier
-  subdirectories (and corresponding data for each instance). The currently active
-  instance will be symlinked the ``instance`` symlink file defined previously.
+  All instances that were created using this image end up with instance
+  identifier subdirectories (and corresponding data for each instance). The
+  currently active instance will be symlinked the ``instance`` symlink file
+  defined previously.
 
 ``scripts/``
 
-  Scripts that are downloaded/created by the corresponding ``part-handler`` will end up
-  in one of these subdirectories.
+  Scripts that are downloaded/created by the corresponding ``part-handler``
+  will end up in one of these subdirectories.
 
 ``seed/``
 
@@ -77,6 +81,7 @@ Cloudinits's directory structure is somewhat different from a regular applicatio
   Cloud-init has a concept of a module semaphore, which basically consists
   of the module name and its frequency. These files are used to ensure a module
   is only ran `per-once`, `per-instance`, `per-always`. This folder contains
-  semaphore `files` which are only supposed to run `per-once` (not tied to the instance id).
+  semaphore `files` which are only supposed to run `per-once` (not tied to the
+  instance id).
 
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/docs.rst b/doc/rtd/topics/docs.rst
new file mode 100644
index 00000000..1b15377e
--- /dev/null
+++ b/doc/rtd/topics/docs.rst
@@ -0,0 +1,84 @@
+.. _docs:
+
+Docs
+****
+
+These docs are hosted on Read the Docs. The following will explain how to
+contribute to and build these docs locally.
+
+The documentation is primarily written in reStructuredText.
+
+
+Building
+========
+
+There is a makefile target to build the documentation for you:
+
+.. code-block:: shell-session
+
+    $ tox -e doc
+
+This will do two things:
+
+- Build the documentation using sphinx
+- Run doc8 against the documentation source code
+
+Once build the HTML files will be viewable in ``doc/rtd_html``. Use your
+web browser to open ``index.html`` to view and navigate the site.
+
+Style Guide
+===========
+
+Headings
+--------
+The headings used across the documentation use the following hierarchy:
+
+- ``*****``: used once atop of a new page
+- ``=====``: each sections on the page
+- ``-----``: subsections
+- ``^^^^^``: sub-subsections
+- ``"""""``: paragraphs
+
+The top level header ``######`` is reserved for the first page.
+
+If under and overline are used, their length must be identical. The length of
+the underline must be at least as long as the title itself
+
+Line Length
+-----------
+Please keep the line lengths to a maximum of **79** characters. This ensures
+that the pages and tables do not get too wide that side scrolling is required.
+
+Header
+------
+Adding a link at the top of the page allows for the page to be referenced by
+other pages. For example for the FAQ page this would be:
+
+.. code-block:: rst
+
+    .. _faq:
+
+Footer
+------
+The footer should include the textwidth
+
+.. code-block:: rst
+
+    .. vi: textwidth=79
+
+Vertical Whitespace
+-------------------
+One newline between each section helps ensure readability of the documentation
+source code.
+
+Common Words
+------------
+There are some common words that should follow specific usage:
+
+- ``cloud-init``: always lower case with a hyphen, unless starting a sentence
+  in which case only the 'C' is capitalized (e.g. ``Cloud-init``).
+- ``metadata``: one word
+- ``user data``: two words, not to be combined
+- ``vendor data``: like user data, it is two words
+
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/examples.rst b/doc/rtd/topics/examples.rst
index c30d2263..81860f85 100644
--- a/doc/rtd/topics/examples.rst
+++ b/doc/rtd/topics/examples.rst
@@ -60,8 +60,8 @@ Setup and run `puppet`_
    :language: yaml
    :linenos:
 
-Add apt repositories
-====================
+Add primary apt repositories
+============================
 
 .. literalinclude:: ../../examples/cloud-config-add-apt-repos.txt
    :language: yaml
@@ -128,15 +128,15 @@ Reboot/poweroff when finished
    :language: yaml
    :linenos:
 
-Configure instances ssh-keys
+Configure instances SSH keys
 ============================
 
 .. literalinclude:: ../../examples/cloud-config-ssh-keys.txt
    :language: yaml
    :linenos:
-   
-Additional apt configuration
-============================
+
+Additional apt configuration and repositories
+=============================================
 
 .. literalinclude:: ../../examples/cloud-config-apt.txt
     :language: yaml
diff --git a/doc/rtd/topics/faq.rst b/doc/rtd/topics/faq.rst
new file mode 100644
index 00000000..98c0cfaa
--- /dev/null
+++ b/doc/rtd/topics/faq.rst
@@ -0,0 +1,238 @@
+.. _faq:
+
+FAQ
+***
+
+How do I get help?
+==================
+
+Having trouble? We would like to help!
+
+- First go through this page with answers to common questions
+- Use the search bar at the upper left to search these docs
+- Ask a question in the ``#cloud-init`` IRC channel on Freenode
+- Join and ask questions on the `cloud-init mailing list <https://launchpad.net/~cloud-init>`_
+- Find a bug? Check out the :ref:`reporting_bugs` topic for
+  how to report one
+
+Where are the logs?
+===================
+
+Cloud-init uses two files to log to:
+
+- `/var/log/cloud-init-output.log`: captures the output from each stage of
+  cloud-init when it runs
+- `/var/log/cloud-init.log`: very detailed log with debugging output,
+  detailing each action taken
+- `/run/cloud-init`: contains logs about how cloud-init decided to enable or
+  disable itself, as well as what platforms/datasources were detected. These
+  logs are most useful when trying to determine what cloud-init ran or did not
+  run.
+
+Be aware that each time a system boots, new logs are appended to the files in
+`/var/log`. Therefore, the files may have more than one boot worth of
+information present.
+
+When reviewing these logs look for any errors or Python tracebacks to check
+for any errors.
+
+Where are the configuration files?
+==================================
+
+Cloud-init config is provided in two places:
+
+- `/etc/cloud/cloud.cfg`
+- `/etc/cloud/cloud.cfg.d/*.cfg`
+
+These files can define the modules that run during instance initialization,
+the datasources to evaluate on boot, and other settings.
+
+Where are the data files?
+=========================
+
+Inside the `/var/lib/cloud/` directory there are two important subdirectories:
+
+instance
+--------
+
+The `/var/lib/cloud/instance` directory is a symbolic link that points
+to the most recenlty used instance-id directory. This folder contains the
+information cloud-init received from datasources, including vendor and user
+data. This can be helpful to review to ensure the correct data was passed.
+
+It also contains the `datasource` file that containers the full information
+about what datasource was identified and used to setup the system.
+
+Finally, the `boot-finished` file is the last thing that cloud-init does.
+
+data
+----
+
+The `/var/lib/cloud/data` directory contain information related to the
+previous boot:
+
+* `instance-id`: id of the instance as discovered by cloud-init. Changing
+  this file has no effect.
+* `result.json`: json file will show both the datasource used to setup
+  the instance, and if any errors occured
+* `status.json`: json file shows the datasource used and a break down
+  of all four modules if any errors occured and the start and stop times.
+
+What datasource am I using?
+===========================
+
+To correctly setup an instance, cloud-init must correctly identify the
+cloud that it is on. Therefore knowing what datasource is used on an
+instance launch can help aid in debugging.
+
+To find what datasource is getting used run the `cloud-id` command:
+
+.. code-block:: shell-session
+
+    $ cloud-id
+    nocloud
+
+If the cloud-id is not what is expected, then running the `ds-identify`
+script in debug mode and providing that in a bug can help aid in resolving
+any issues:
+
+.. code-block:: shell-session
+
+    $ sudo DEBUG_LEVEL=2 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
+
+The force parameter allows the command to be run again since the instance has
+already launched. The other options increase the verbosity of logging and
+put the logs to STDERR.
+
+How can I debug my user data?
+=============================
+
+Two of the most common issues with user data, that also happens to be
+cloud-config is:
+
+1. Incorrectly formatted YAML
+2. First line does not contain `#cloud-config`
+
+To verify your YAML, we do have a short script called `validate-yaml.py`_
+that can validate your user data offline.
+
+.. _validate-yaml.py: https://github.com/canonical/cloud-init/blob/master/tools/validate-yaml.py
+
+Another option is to run the following on an instance when debugging:
+
+.. code-block:: shell-session
+
+    $ sudo cloud-init query userdata > user-data.yaml
+    $ cloud-init devel schema -c user-data.yaml --annotate
+
+As launching instances in the cloud can cost money and take a bit longer,
+sometimes it is easier to launch instances locally using Multipass or LXD:
+
+Multipass
+---------
+
+`Multipass`_ is a cross-platform tool to launch Ubuntu VMs across Linux,
+Windows, and macOS.
+
+When a user launches a Multipass VM, user data can be passed by adding the
+`--cloud-init` flag and the appropriate YAML file containing user data:
+
+.. code-block:: shell-session
+
+    $ multipass launch bionic --name test-vm --cloud-init userdata.yaml
+
+Multipass will validate the YAML syntax of the cloud-config file before
+attempting to start the VM! A nice addition to help save time when
+experimenting with launching instances with various cloud-configs.
+
+Multipass only supports passing user-data and only as YAML cloud-config
+files. Passing a script, a MIME archive, or any of the other user-data
+formats cloud-init supports will result in an error from the YAML syntax
+validator.
+
+.. _Multipass: https://multipass.run/
+
+LXD
+---
+
+`LXD`_ offers a streamlined user experience for using linux system
+containers. With LXD, a user can pass:
+
+* user data
+* vendor data
+* metadata
+* network configuration
+
+The following initializes a container with user data:
+
+.. code-block:: shell-session
+
+    $ lxc init ubuntu-daily:bionic test-container
+    $ lxc config set test-container user.user-data - < userdata.yaml
+    $ lxc start test-container
+
+To avoid the extra commands this can also be done at launch:
+
+.. code-block:: shell-session
+
+    $ lxc launch ubuntu-daily:bionic test-container --config=user.user-data="$(cat userdata.yaml)"
+
+Finally, a profile can be setup with the specific data if a user needs to
+launch this multiple times:
+
+.. code-block:: shell-session
+
+    $ lxc profile create dev-user-data
+    $ lxc profile set dev-user-data user.user-data - < cloud-init-config.yaml
+    $ lxc launch ubuntu-daily:bionic test-container -p default -p dev-user-data
+
+The above examples all show how to pass user data. To pass other types of
+configuration data use the config option specified below:
+
++----------------+---------------------+
+| Data           | Config Option       |
++================+=====================+
+| user data      | user.user-data      |
++----------------+---------------------+
+| vendor data    | user.vendor-data    |
++----------------+---------------------+
+| metadata       | user.meta-data      |
++----------------+---------------------+
+| network config | user.network-config |
++----------------+---------------------+
+
+See the LXD `Instance Configuration`_ docs for more info about configuration
+values or the LXD `Custom Network Configuration`_ document for more about
+custom network config.
+
+.. _LXD: https://linuxcontainers.org/
+.. _Instance Configuration: https://lxd.readthedocs.io/en/latest/instances/
+.. _Custom Network Configuration: https://lxd.readthedocs.io/en/latest/cloud-init/
+
+Where can I learn more?
+========================================
+
+Below are some videos, blog posts, and white papers about cloud-init from a
+variety of sources.
+
+- `Cloud Instance Initialization with cloud-init (Whitepaper)`_
+- `cloud-init Summit 2018`_
+- `cloud-init - The cross-cloud Magic Sauce (PDF)`_
+- `cloud-init Summit 2017`_
+- `cloud-init - Building clouds one Linux box at a time (Video)`_
+- `cloud-init - Building clouds one Linux box at a time (PDF)`_
+- `Metadata and cloud-init`_
+- `The beauty of cloud-init`_
+- `Introduction to cloud-init`_
+
+.. _Cloud Instance Initialization with cloud-init (Whitepaper): https://ubuntu.com/blog/cloud-instance-initialisation-with-cloud-init
+.. _cloud-init Summit 2018: https://powersj.io/post/cloud-init-summit18/
+.. _cloud-init - The cross-cloud Magic Sauce (PDF): https://events.linuxfoundation.org/wp-content/uploads/2017/12/cloud-init-The-cross-cloud-Magic-Sauce-Scott-Moser-Chad-Smith-Canonical.pdf
+.. _cloud-init Summit 2017: https://powersj.io/post/cloud-init-summit17/
+.. _cloud-init - Building clouds one Linux box at a time (Video): https://www.youtube.com/watch?v=1joQfUZQcPg
+.. _cloud-init - Building clouds one Linux box at a time (PDF): https://annex.debconf.org/debconf-share/debconf17/slides/164-cloud-init_Building_clouds_one_Linux_box_at_a_time.pdf
+.. _Metadata and cloud-init: https://www.youtube.com/watch?v=RHVhIWifVqU
+.. _The beauty of cloud-init: http://brandon.fuller.name/archives/2011/05/02/06.40.57/
+.. _Introduction to cloud-init: http://www.youtube.com/watch?v=-zL3BdbKyGY
+
+.. vi: textwidth=79
diff --git a/doc/rtd/topics/format.rst b/doc/rtd/topics/format.rst
index 15234d21..2b60bdd3 100644
--- a/doc/rtd/topics/format.rst
+++ b/doc/rtd/topics/format.rst
@@ -4,33 +4,36 @@
 User-Data Formats
 *****************
 
-User data that will be acted upon by cloud-init must be in one of the following types.
+User data that will be acted upon by cloud-init must be in one of the following
+types.
 
 Gzip Compressed Content
 =======================
 
 Content found to be gzip compressed will be uncompressed.
-The uncompressed data will then be used as if it were not compressed. 
+The uncompressed data will then be used as if it were not compressed.
 This is typically useful because user-data is limited to ~16384 [#]_ bytes.
 
 Mime Multi Part Archive
 =======================
 
-This list of rules is applied to each part of this multi-part file. 
+This list of rules is applied to each part of this multi-part file.
 Using a mime-multi part file, the user can specify more than one type of data.
 
-For example, both a user data script and a cloud-config type could be specified.
+For example, both a user data script and a cloud-config type could be
+specified.
 
 Supported content-types:
 
-- text/x-include-once-url
-- text/x-include-url
-- text/cloud-config-archive
-- text/upstart-job
+- text/cloud-boothook
 - text/cloud-config
+- text/cloud-config-archive
+- text/jinja2
 - text/part-handler
+- text/upstart-job
+- text/x-include-once-url
+- text/x-include-url
 - text/x-shellscript
-- text/cloud-boothook
 
 Helper script to generate mime messages
 ---------------------------------------
@@ -38,16 +41,16 @@ Helper script to generate mime messages
 .. code-block:: python
 
    #!/usr/bin/python
-   
+
    import sys
-   
+
    from email.mime.multipart import MIMEMultipart
    from email.mime.text import MIMEText
-   
+
    if len(sys.argv) == 1:
        print("%s input-file:type ..." % (sys.argv[0]))
        sys.exit(1)
-   
+
    combined_message = MIMEMultipart()
    for i in sys.argv[1:]:
        (filename, format_type) = i.split(":", 1)
@@ -56,7 +59,7 @@ Helper script to generate mime messages
        sub_message = MIMEText(contents, format_type, sys.getdefaultencoding())
        sub_message.add_header('Content-Disposition', 'attachment; filename="%s"' % (filename))
        combined_message.attach(sub_message)
-   
+
    print(combined_message)
 
 
@@ -65,7 +68,8 @@ User-Data Script
 
 Typically used by those who just want to execute a shell script.
 
-Begins with: ``#!`` or ``Content-Type: text/x-shellscript`` when using a MIME archive.
+Begins with: ``#!`` or ``Content-Type: text/x-shellscript`` when using a MIME
+archive.
 
 .. note::
    New in cloud-init v. 18.4: User-data scripts can also render cloud instance
@@ -78,44 +82,48 @@ Example
 ::
 
   $ cat myscript.sh
-  
+
   #!/bin/sh
   echo "Hello World.  The time is now $(date -R)!" | tee /root/output.txt
-  
-  $ euca-run-instances --key mykey --user-data-file myscript.sh ami-a07d95c9 
+
+  $ euca-run-instances --key mykey --user-data-file myscript.sh ami-a07d95c9
 
 Include File
 ============
 
 This content is a ``include`` file.
 
-The file contains a list of urls, one per line.
-Each of the URLs will be read, and their content will be passed through this same set of rules.
-Ie, the content read from the URL can be gzipped, mime-multi-part, or plain text.
-If an error occurs reading a file the remaining files will not be read.
+The file contains a list of urls, one per line. Each of the URLs will be read,
+and their content will be passed through this same set of rules. Ie, the
+content read from the URL can be gzipped, mime-multi-part, or plain text. If
+an error occurs reading a file the remaining files will not be read.
 
-Begins with: ``#include`` or ``Content-Type: text/x-include-url``  when using a MIME archive.
+Begins with: ``#include`` or ``Content-Type: text/x-include-url``  when using
+a MIME archive.
 
 Cloud Config Data
 =================
 
-Cloud-config is the simplest way to accomplish some things
-via user-data. Using cloud-config syntax, the user can specify certain things in a human friendly format. 
+Cloud-config is the simplest way to accomplish some things via user-data. Using
+cloud-config syntax, the user can specify certain things in a human friendly
+format.
 
 These things include:
 
 - apt upgrade should be run on first boot
 - a different apt mirror should be used
 - additional apt sources should be added
-- certain ssh keys should be imported
+- certain SSH keys should be imported
 - *and many more...*
 
 .. note::
    This file must be valid yaml syntax.
 
-See the :ref:`yaml_examples` section for a commented set of examples of supported cloud config formats.
+See the :ref:`yaml_examples` section for a commented set of examples of
+supported cloud config formats.
 
-Begins with: ``#cloud-config`` or ``Content-Type: text/cloud-config`` when using a MIME archive.
+Begins with: ``#cloud-config`` or ``Content-Type: text/cloud-config`` when
+using a MIME archive.
 
 .. note::
    New in cloud-init v. 18.4: Cloud config dta can also render cloud instance
@@ -125,25 +133,41 @@ Begins with: ``#cloud-config`` or ``Content-Type: text/cloud-config`` when using
 Upstart Job
 ===========
 
-Content is placed into a file in ``/etc/init``, and will be consumed by upstart as any other upstart job.
+Content is placed into a file in ``/etc/init``, and will be consumed by upstart
+as any other upstart job.
 
-Begins with: ``#upstart-job`` or ``Content-Type: text/upstart-job`` when using a MIME archive.
+Begins with: ``#upstart-job`` or ``Content-Type: text/upstart-job`` when using
+a MIME archive.
 
 Cloud Boothook
 ==============
 
-This content is ``boothook`` data. It is stored in a file under ``/var/lib/cloud`` and then executed immediately.
-This is the earliest ``hook`` available. Note, that there is no mechanism provided for running only once. The boothook must take care of this itself.
-It is provided with the instance id in the environment variable ``INSTANCE_ID``. This could be made use of to provide a 'once-per-instance' type of functionality.
+This content is ``boothook`` data. It is stored in a file under
+``/var/lib/cloud`` and then executed immediately. This is the earliest ``hook``
+available. Note, that there is no mechanism provided for running only once. The
+boothook must take care of this itself.
+
+It is provided with the instance id in the environment variable
+``INSTANCE_ID``. This could be made use of to provide a 'once-per-instance'
+type of functionality.
 
-Begins with: ``#cloud-boothook`` or ``Content-Type: text/cloud-boothook`` when using a MIME archive.
+Begins with: ``#cloud-boothook`` or ``Content-Type: text/cloud-boothook`` when
+using a MIME archive.
 
 Part Handler
 ============
 
-This is a ``part-handler``: It contains custom code for either supporting new mime-types in multi-part user data, or overriding the existing handlers for supported mime-types.  It will be written to a file in ``/var/lib/cloud/data`` based on its filename (which is generated).
-This must be python code that contains a ``list_types`` function and a ``handle_part`` function. 
-Once the section is read the ``list_types`` method will be called. It must return a list of mime-types that this part-handler handles.  Because mime parts are processed in order, a ``part-handler`` part must precede any parts with mime-types it is expected to handle in the same user data.
+This is a ``part-handler``: It contains custom code for either supporting new
+mime-types in multi-part user data, or overriding the existing handlers for
+supported mime-types.  It will be written to a file in ``/var/lib/cloud/data``
+based on its filename (which is generated).
+
+This must be python code that contains a ``list_types`` function and a
+``handle_part`` function. Once the section is read the ``list_types`` method
+will be called. It must return a list of mime-types that this part-handler
+handles.  Because mime parts are processed in order, a ``part-handler`` part
+must precede any parts with mime-types it is expected to handle in the same
+user data.
 
 The ``handle_part`` function must be defined like:
 
@@ -155,11 +179,13 @@ The ``handle_part`` function must be defined like:
       # filename = the filename of the part (or a generated filename if none is present in mime data)
       # payload = the parts' content
 
-Cloud-init will then call the ``handle_part`` function once before it handles any parts, once per part received, and once after all parts have been handled.
-The ``'__begin__'`` and ``'__end__'`` sentinels allow the part handler to do initialization or teardown before or after
-receiving any parts.
+Cloud-init will then call the ``handle_part`` function once before it handles
+any parts, once per part received, and once after all parts have been handled.
+The ``'__begin__'`` and ``'__end__'`` sentinels allow the part handler to do
+initialization or teardown before or after receiving any parts.
 
-Begins with: ``#part-handler`` or ``Content-Type: text/part-handler`` when using a MIME archive.
+Begins with: ``#part-handler`` or ``Content-Type: text/part-handler`` when
+using a MIME archive.
 
 Example
 -------
@@ -170,6 +196,22 @@ Example
 
 Also this `blog`_ post offers another example for more advanced usage.
 
+Kernel Command Line
+===================
+
+When using the :ref:`datasource_nocloud` datasource, users can pass user data
+via the kernel command line parameters. See the :ref:`datasource_nocloud`
+datasource documentation for more details.
+
+Disabling User-Data
+===================
+
+Cloud-init can be configured to ignore any user-data provided to instance.
+This allows custom images to prevent users from accidentally breaking closed
+appliances. Setting ``allow_userdata: false`` in the configuration will disable
+cloud-init from processing user-data.
+
 .. [#] See your cloud provider for applicable user-data size limitations...
 .. _blog: http://foss-boss.blogspot.com/2011/01/advanced-cloud-init-custom-handlers.html
+
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/instancedata.rst b/doc/rtd/topics/instancedata.rst
index 5d2dc948..e7dd0d62 100644
--- a/doc/rtd/topics/instancedata.rst
+++ b/doc/rtd/topics/instancedata.rst
@@ -4,7 +4,7 @@
 Instance Metadata
 *****************
 
-What is a instance data?
+What is instance data?
 ========================
 
 Instance data is the collection of all configuration data that cloud-init
@@ -90,47 +90,107 @@ There are three basic top-level keys:
 
 The standardized keys present:
 
-+----------------------+-----------------------------------------------+-----------------------------------+
-|  Key path            | Description                                   | Examples                          |
-+======================+===============================================+===================================+
-| v1._beta_keys        | List of standardized keys still in 'beta'.    | [subplatform]                     |
-|                      | The format, intent or presence of these keys  |                                   |
-|                      | can change. Do not consider them              |                                   |
-|                      | production-ready.                             |                                   |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.cloud_name        | Where possible this will indicate the 'name'  | aws, openstack, azure,            |
-|                      | of the cloud this system is running on.  This | configdrive, nocloud,             |
-|                      | is specifically different than the 'platform' | ovf, etc.                         |
-|                      | below.  As an example, the name of Amazon Web |                                   |
-|                      | Services is 'aws' while the platform is 'ec2'.|                                   |
-|                      |                                               |                                   |
-|                      | If no specific name is determinable or        |                                   |
-|                      | provided in meta-data, then this field may    |                                   |
-|                      | contain the same content as 'platform'.       |                                   |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.instance_id       | Unique instance_id allocated by the cloud     | i-<somehash>                      |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.local_hostname    | The internal or local hostname of the system  | ip-10-41-41-70,                   |
-|                      |                                               | <user-provided-hostname>          |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.platform          | An attempt to identify the cloud platform     | ec2, openstack, lxd, gce          |
-|                      | instance that the system is running on.       | nocloud, ovf                      |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.subplatform       | Additional platform details describing the    | metadata (http://168.254.169.254),|
-|                      | specific source or type of metadata used.     | seed-dir (/path/to/seed-dir/),    |
-|                      | The format of subplatform will be:            | config-disk (/dev/cd0),           |
-|                      | <subplatform_type> (<url_file_or_dev_path>)   | configdrive (/dev/sr0)            |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.public_ssh_keys   | A list of  ssh keys provided to the instance  | ['ssh-rsa AA...', ...]            |
-|                      | by the datasource metadata.                   |                                   |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.region            | The physical region/datacenter in which the   | us-east-2                         |
-|                      | instance is deployed                          |                                   |
-+----------------------+-----------------------------------------------+-----------------------------------+
-| v1.availability_zone | The physical availability zone in which the   | us-east-2b, nova, null            |
-|                      | instance is deployed                          |                                   |
-+----------------------+-----------------------------------------------+-----------------------------------+
+v1._beta_keys
+-------------
+List of standardized keys still in 'beta'. The format, intent or presence of
+these keys can change. Do not consider them production-ready.
 
+Example output:
+
+- [subplatform]
+
+v1.cloud_name
+-------------
+Where possible this will indicate the 'name' of the cloud the system is running
+on. This is different than the 'platform' item. For example, the cloud name of
+Amazone Web Services is 'aws', while the platform is 'ec2'.
+
+If determining a specific name is not possible or provided in meta-data, then
+this filed may contain the same content as 'platform'.
+
+Example output:
+
+- aws
+- openstack
+- azure
+- configdrive
+- nocloud
+- ovf
+
+
+v1.instance_id
+--------------
+Unique instance_id allocated by the cloud.
+
+Examples output:
+
+- i-<hash>
+
+v1.local_hostname
+-----------------
+The internal or local hostname of the system.
+
+Examples output:
+
+- ip-10-41-41-70
+- <user-provided-hostname>
+
+v1.platform
+-------------
+An attempt to identify the cloud platfrom instance that the system is running
+on.
+
+Examples output:
+
+- ec2
+- openstack
+- lxd
+- gce
+- nocloud
+- ovf
+
+v1.subplatform
+--------------
+Additional platform details describing the specific source or type of metadata
+used. The format of subplatform will be:
+
+``<subplatform_type> (<url_file_or_dev_path>``
+
+Examples output:
+
+- metadata (http://168.254.169.254)
+- seed-dir (/path/to/seed-dir/)
+- config-disk (/dev/cd0)
+- configdrive (/dev/sr0)
+
+v1.public_ssh_keys
+------------------
+A list of SSH keys provided to the instance by the datasource metadata.
+
+Examples output:
+
+- ['ssh-rsa AA...', ...]
+
+v1.region
+---------
+The physical region/data center in which the instance is deployed.
+
+Examples output:
+
+- us-east-2
+
+v1.availability_zone
+--------------------
+The physical availability zone in which the instance is deployed.
+
+Examples output:
+
+- us-east-2b
+- nova
+- null
+
+Example Output
+--------------
 
 Below is an example of ``/run/cloud-init/instance_data.json`` on an EC2
 instance:
@@ -229,28 +289,28 @@ instance:
      "network": {
       "interfaces": {
        "macs": {
-	"06:74:8f:39:cd:a6": {
-	 "device-number": "0",
-	 "interface-id": "eni-052058bbd7831eaae",
-	 "ipv4-associations": {
-	  "18.218.221.122": "10.41.41.95"
-	 },
-	 "local-hostname": "ip-10-41-41-95.us-east-2.compute.internal",
-	 "local-ipv4s": "10.41.41.95",
-	 "mac": "06:74:8f:39:cd:a6",
-	 "owner-id": "437526006925",
-	 "public-hostname": "ec2-18-218-221-122.us-east-2.compute.amazonaws.com",
-	 "public-ipv4s": "18.218.221.122",
-	 "security-group-ids": "sg-828247e9",
-	 "security-groups": "Cloud-init integration test secgroup",
-	 "subnet-id": "subnet-282f3053",
-	 "subnet-ipv4-cidr-block": "10.41.41.0/24",
-	 "subnet-ipv6-cidr-blocks": "2600:1f16:b80:ad00::/64",
-	 "vpc-id": "vpc-252ef24d",
-	 "vpc-ipv4-cidr-block": "10.41.0.0/16",
-	 "vpc-ipv4-cidr-blocks": "10.41.0.0/16",
-	 "vpc-ipv6-cidr-blocks": "2600:1f16:b80:ad00::/56"
-	}
+       "06:74:8f:39:cd:a6": {
+        "device-number": "0",
+        "interface-id": "eni-052058bbd7831eaae",
+        "ipv4-associations": {
+         "18.218.221.122": "10.41.41.95"
+        },
+        "local-hostname": "ip-10-41-41-95.us-east-2.compute.internal",
+        "local-ipv4s": "10.41.41.95",
+        "mac": "06:74:8f:39:cd:a6",
+        "owner-id": "437526006925",
+        "public-hostname": "ec2-18-218-221-122.us-east-2.compute.amazonaws.com",
+        "public-ipv4s": "18.218.221.122",
+        "security-group-ids": "sg-828247e9",
+        "security-groups": "Cloud-init integration test secgroup",
+        "subnet-id": "subnet-282f3053",
+        "subnet-ipv4-cidr-block": "10.41.41.0/24",
+        "subnet-ipv6-cidr-blocks": "2600:1f16:b80:ad00::/64",
+        "vpc-id": "vpc-252ef24d",
+        "vpc-ipv4-cidr-block": "10.41.0.0/16",
+        "vpc-ipv4-cidr-blocks": "10.41.0.0/16",
+        "vpc-ipv6-cidr-blocks": "2600:1f16:b80:ad00::/56"
+       }
        }
       }
      },
@@ -319,16 +379,16 @@ user.
 
 Below are some examples of providing these types of user-data:
 
-* Cloud config calling home with the ec2 public hostname and avaliability-zone
+* Cloud config calling home with the ec2 public hostname and availability-zone
 
-.. code-block:: shell-session
+.. code-block:: yaml
 
   ## template: jinja
   #cloud-config
   runcmd:
       - echo 'EC2 public hostname allocated to instance: {{
         ds.meta_data.public_hostname }}' > /tmp/instance_metadata
-      - echo 'EC2 avaiability zone: {{ v1.availability_zone }}' >>
+      - echo 'EC2 availability zone: {{ v1.availability_zone }}' >>
         /tmp/instance_metadata
       - curl -X POST -d '{"hostname": "{{ds.meta_data.public_hostname }}",
         "availability-zone": "{{ v1.availability_zone }}"}'
@@ -336,7 +396,7 @@ Below are some examples of providing these types of user-data:
 
 * Custom user-data script performing different operations based on region
 
-.. code-block:: shell-session
+.. code-block:: jinja
 
    ## template: jinja
    #!/bin/bash
@@ -352,7 +412,7 @@ Below are some examples of providing these types of user-data:
   and the following string in your rendered user-data:
   ``CI_MISSING_JINJA_VAR/<your_varname>``.
 
-Cloud-init also surfaces a commandline tool **cloud-init query** which can
+Cloud-init also surfaces a command line tool **cloud-init query** which can
 assist developers or scripts with obtaining instance metadata easily. See
 :ref:`cli_query` for more information.
 
diff --git a/doc/rtd/topics/merging.rst b/doc/rtd/topics/merging.rst
index c75ca59c..2b5e5dad 100644
--- a/doc/rtd/topics/merging.rst
+++ b/doc/rtd/topics/merging.rst
@@ -21,12 +21,12 @@ For example.
 .. code-block:: yaml
 
    #cloud-config (1)
-   run_cmd:
+   runcmd:
      - bash1
      - bash2
 
    #cloud-config (2)
-   run_cmd:
+   runcmd:
      - bash3
      - bash4
 
@@ -36,7 +36,7 @@ cloud-config object that contains the following.
 .. code-block:: yaml
 
    #cloud-config (merged)
-   run_cmd:
+   runcmd:
      - bash3
      - bash4
 
@@ -45,7 +45,7 @@ Typically this is not what users want; instead they would likely prefer:
 .. code-block:: yaml
 
    #cloud-config (merged)
-   run_cmd:
+   runcmd:
      - bash1
      - bash2
      - bash3
@@ -55,6 +55,51 @@ This way makes it easier to combine the various cloud-config objects you have
 into a more useful list, thus reducing duplication necessary to accomplish the
 same result with the previous method.
 
+
+Built-in Mergers
+================
+
+Cloud-init provides merging for the following built-in types:
+
+- Dict
+- List
+- String
+
+The ``Dict`` merger has the following options which control what is done with
+values contained within the config.
+
+- ``allow_delete``: Existing values not present in the new value can be
+  deleted, defaults to False
+- ``no_replace``: Do not replace an existing value if one is already present,
+  enabled by default.
+- ``replace``: Overwrite existing values with new ones.
+
+The ``List`` merger has the following options which control what is done with
+the values contained within the config.
+
+- ``append``:  Add new value to the end of the list, defaults to False.
+- ``prepend``:  Add new values to the start of the list, defaults to False.
+- ``no_replace``: Do not replace an existing value if one is already present,
+  enabled by default.
+- ``replace``: Overwrite existing values with new ones.
+
+The ``Str`` merger has the following options which control what is done with
+the values contained within the config.
+
+- ``append``:  Add new value to the end of the string, defaults to False.
+
+Common options for all merge types which control how recursive merging is
+done on other types.
+
+- ``recurse_dict``: If True merge the new values of the dictionary, defaults to
+  True.
+- ``recurse_list``: If True merge the new values of the list, defaults to
+  False.
+- ``recurse_array``: Alias for ``recurse_list``.
+- ``recurse_str``: If True merge the new values of the string, defaults to
+  False.
+
+
 Customizability
 ===============
 
@@ -164,8 +209,8 @@ string format (i.e. the second option above), for example:
 
 .. code-block:: python
 
-   {'merge_how': [{'name': 'list', 'settings': ['extend']},
-                  {'name': 'dict', 'settings': []},
+   {'merge_how': [{'name': 'list', 'settings': ['append']},
+                  {'name': 'dict', 'settings': ['no_replace', 'recurse_list']},
                   {'name': 'str', 'settings': ['append']}]}
 
 This would be the equivalent format for default string format but in dictionary
@@ -201,4 +246,43 @@ Note, however, that merge algorithms are not used *across* types of
 configuration.  As was the case before merging was implemented,
 user-data will overwrite conf.d configuration without merging.
 
+Example cloud-config
+====================
+
+A common request is to include multiple ``runcmd`` directives in different
+files and merge all of the commands together.  To achieve this, we must modify
+the default merging to allow for dictionaries to join list values.
+
+
+The first config
+
+.. code-block:: yaml
+
+   #cloud-config
+   merge_how:
+    - name: list
+      settings: [append]
+    - name: dict
+      settings: [no_replace, recurse_list]
+
+   runcmd:
+     - bash1
+     - bash2
+
+The second config
+
+.. code-block:: yaml
+
+   #cloud-config
+   merge_how:
+    - name: list
+      settings: [append]
+    - name: dict
+      settings: [no_replace, recurse_list]
+
+   runcmd:
+     - bash3
+     - bash4
+
+
 .. vi: textwidth=78
diff --git a/doc/rtd/topics/modules.rst b/doc/rtd/topics/modules.rst
index d9720f6a..9c9be804 100644
--- a/doc/rtd/topics/modules.rst
+++ b/doc/rtd/topics/modules.rst
@@ -1,8 +1,11 @@
 .. _modules:
 
+
 *******
 Modules
 *******
+.. contents:: Table of Contents
+
 .. automodule:: cloudinit.config.cc_apt_configure
 .. automodule:: cloudinit.config.cc_apt_pipelining
 .. automodule:: cloudinit.config.cc_bootcmd
@@ -46,14 +49,13 @@ Modules
 .. automodule:: cloudinit.config.cc_set_hostname
 .. automodule:: cloudinit.config.cc_set_passwords
 .. automodule:: cloudinit.config.cc_snap
-.. automodule:: cloudinit.config.cc_snappy
-.. automodule:: cloudinit.config.cc_snap_config
 .. automodule:: cloudinit.config.cc_spacewalk
 .. automodule:: cloudinit.config.cc_ssh
 .. automodule:: cloudinit.config.cc_ssh_authkey_fingerprints
 .. automodule:: cloudinit.config.cc_ssh_import_id
 .. automodule:: cloudinit.config.cc_timezone
 .. automodule:: cloudinit.config.cc_ubuntu_advantage
+.. automodule:: cloudinit.config.cc_ubuntu_drivers
 .. automodule:: cloudinit.config.cc_update_etc_hosts
 .. automodule:: cloudinit.config.cc_update_hostname
 .. automodule:: cloudinit.config.cc_users_groups
diff --git a/doc/rtd/topics/moreinfo.rst b/doc/rtd/topics/moreinfo.rst
deleted file mode 100644
index 9c3b7fba..00000000
--- a/doc/rtd/topics/moreinfo.rst
+++ /dev/null
@@ -1,13 +0,0 @@
-****************
-More information
-****************
-
-Useful external references
-==========================
-
-- `The beauty of cloudinit`_
-- `Introduction to cloud-init`_ (video)
-
-.. _Introduction to cloud-init: http://www.youtube.com/watch?v=-zL3BdbKyGY
-.. _The beauty of cloudinit: http://brandon.fuller.name/archives/2011/05/02/06.40.57/
-.. vi: textwidth=78
diff --git a/doc/rtd/topics/network-config-format-v2.rst b/doc/rtd/topics/network-config-format-v2.rst
index ea370ef5..7f857550 100644
--- a/doc/rtd/topics/network-config-format-v2.rst
+++ b/doc/rtd/topics/network-config-format-v2.rst
@@ -14,7 +14,7 @@ it must include ``version: 2``  and one or more of possible device
 
 Cloud-init will read this format from system config.
 For example the following could be present in
-``/etc/cloud/cloud.cfg.d/custom-networking.cfg``:
+``/etc/cloud/cloud.cfg.d/custom-networking.cfg``::
 
   network:
     version: 2
@@ -54,11 +54,11 @@ Physical devices
 
 :   (Examples: ethernet, wifi) These can dynamically come and go between
     reboots and even during runtime (hotplugging). In the generic case, they
-    can be selected by ``match:`` rules on desired properties, such as name/name
-    pattern, MAC address, driver, or device paths. In general these will match
-    any number of devices (unless they refer to properties which are unique
-    such as the full path or MAC address), so without further knowledge about
-    the hardware these will always be considered as a group.
+    can be selected by ``match:`` rules on desired properties, such as
+    name/name pattern, MAC address, driver, or device paths. In general these
+    will match any number of devices (unless they refer to properties which are
+    unique such as the full path or MAC address), so without further knowledge
+    about the hardware these will always be considered as a group.
 
     It is valid to specify no match rules at all, in which case the ID field is
     simply the interface name to be matched. This is mostly useful if you want
@@ -228,8 +228,8 @@ Example: ::
 
 **parameters**: *<(mapping)>*
 
-Customization parameters for special bonding options.  Time values are specified
-in seconds unless otherwise specified.
+Customization parameters for special bonding options.  Time values are
+specified in seconds unless otherwise specified.
 
 **mode**: *<(scalar)>*
 
@@ -367,8 +367,8 @@ Example: ::
 
 **parameters**: <*(mapping)>*
 
-Customization parameters for special bridging options.  Time values are specified
-in seconds unless otherwise specified.
+Customization parameters for special bridging options.  Time values are
+specified in seconds unless otherwise specified.
 
 **ageing-time**: <*(scalar)>*
 
diff --git a/doc/rtd/topics/network-config.rst b/doc/rtd/topics/network-config.rst
index 1e994551..1520ba9a 100644
--- a/doc/rtd/topics/network-config.rst
+++ b/doc/rtd/topics/network-config.rst
@@ -163,10 +163,11 @@ found in Ubuntu and Debian.
 
 - **Netplan**
 
-Since Ubuntu 16.10, codename Yakkety, the ``netplan`` project has been an
-optional network configuration tool which consumes :ref:`network_config_v2`
-input and renders network configuration for supported backends such as
-``systemd-networkd`` and ``NetworkManager``.
+Introduced in Ubuntu 16.10 (Yakkety Yak), `netplan <https://netplan.io/>`_ has
+been the default network configuration tool in Ubuntu since 17.10 (Artful
+Aardvark).  netplan consumes :ref:`network_config_v2` input and renders
+network configuration for supported backends such as ``systemd-networkd`` and
+``NetworkManager``.
 
 - **Sysconfig**
 
@@ -190,7 +191,7 @@ supplying an updated configuration in cloud-config. ::
 
   system_info:
     network:
-      renderers: ['netplan', 'eni', 'sysconfig']
+      renderers: ['netplan', 'eni', 'sysconfig', 'freebsd']
 
 
 Network Configuration Tools
diff --git a/doc/rtd/topics/security.rst b/doc/rtd/topics/security.rst
new file mode 100644
index 00000000..b8386843
--- /dev/null
+++ b/doc/rtd/topics/security.rst
@@ -0,0 +1,5 @@
+.. _security:
+
+.. mdinclude:: ../../../SECURITY.md
+
+.. vi: textwidth=78
diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
index b83bd899..aee3d7fc 100644
--- a/doc/rtd/topics/tests.rst
+++ b/doc/rtd/topics/tests.rst
@@ -23,7 +23,7 @@ configuration users can run the integration tests via tox:
 
 .. code-block:: shell-session
 
-    $ git clone https://git.launchpad.net/cloud-init
+    $ git clone https://github.com/canonical/cloud-init
     $ cd cloud-init
     $ tox -e citest -- -h
 
@@ -53,7 +53,7 @@ explaining how to run one or the other independently.
 
 .. code-block:: shell-session
 
-    $ git clone https://git.launchpad.net/cloud-init
+    $ git clone https://github.com/canonical/cloud-init
     $ cd cloud-init
     $ tox -e citest -- run --verbose \
         --os-name stretch --os-name xenial \
@@ -423,6 +423,57 @@ generated when running ``aws configure``:
     region = us-west-2
 
 
+Azure Cloud
+-----------
+
+To run on Azure Cloud platform users login with Service Principal and export
+credentials file. Region is defaulted and can be set in
+``tests/cloud_tests/platforms.yaml``. The Service Principal credentials are
+the standard authentication for Azure SDK to interact with Azure Services:
+
+Create Service Principal account or login
+
+.. code-block:: shell-session
+
+    $ az ad sp create-for-rbac --name "APP_ID" --password "STRONG-SECRET-PASSWORD"
+
+.. code-block:: shell-session
+
+    $ az login --service-principal --username "APP_ID" --password "STRONG-SECRET-PASSWORD"
+
+Export credentials
+
+.. code-block:: shell-session
+
+    $ az ad sp create-for-rbac --sdk-auth > $HOME/.azure/credentials.json
+
+.. code-block:: json
+
+    {
+        "clientId": "<Service principal ID>",
+        "clientSecret": "<Service principal secret/password>",
+        "subscriptionId": "<Subscription associated with the service principal>",
+        "tenantId": "<The service principal's tenant>",
+        "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
+        "resourceManagerEndpointUrl": "https://management.azure.com/",
+        "activeDirectoryGraphResourceId": "https://graph.windows.net/",
+        "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
+        "galleryEndpointUrl": "https://gallery.azure.com/",
+        "managementEndpointUrl": "https://management.core.windows.net/"
+    }
+
+Set region in platforms.yaml
+
+.. code-block:: yaml
+
+    azurecloud:
+        enabled: true
+        region: West US 2
+        vm_size: Standard_DS1_v2
+        storage_sku: standard_lrs
+        tag: ci
+
+
 Architecture
 ============