Integration Testing: tox env, pyxld 2.2.3, and revamp framework

Massive update to clean up and greatly enhance the integration testing
framework developed by Wesley Wiedenmeier.

 - Updated tox environment to run integration test 'citest' to utilize
   pylxd 2.2.3
 - Add support for distro feature flags
   - add framework for feature flags to release config with feature groups
     and overrides allowed in any release conf override level
   - add support for feature flags in platform and config handling
   - during collect, skip testcases that require features not supported by
     the image with a warning message
 - Enable additional distros (i.e. centos, debian)
 - Add 'bddeb' command to build a deb from the current working tree
   cleanly in a container, so deps do not have to be installed on host
 - Adds a command line option '--preserve-data' that ensures that
   collected data will be left after tests run. This also allows the
   directory to store collected data in during the run command to be
   specified using '--data-dir'.
 - Updated Read the Docs testing page and doc strings for pep 257
   compliance

1 files changed, 509 insertions, 122 deletions

diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
index 60c63bce..d668e3f4 100644
--- a/doc/rtd/topics/tests.rst
+++ b/doc/rtd/topics/tests.rst
@@ -1,14 +1,186 @@
-****************
-Test Development
-****************
-
+*******************
+Integration Testing
+*******************
 
 Overview
 ========
 
-The purpose of this page is to describe how to write integration tests for
-cloud-init. As a test writer you need to develop a test configuration and
-a verification file:
+This page describes the execution, development, and architecture of the
+cloud-init integration tests:
+
+* Execution explains the options available and running of tests
+* Development shows how to write test cases
+* Architecture explains the internal processes
+
+Execution
+=========
+
+Overview
+--------
+
+In order to avoid the need for dependencies and ease the setup and
+configuration users can run the integration tests via tox:
+
+.. code-block:: bash
+
+    $ git clone https://git.launchpad.net/cloud-init
+    $ cd cloud-init
+    $ tox -e citest -- -h
+
+Everything after the double dash will be passed to the integration tests.
+Executing tests has several options:
+
+* ``run`` an alias to run both ``collect`` and ``verify``. The ``tree_run``
+  command does the same thing, except uses a deb built from the current
+  working tree.
+
+* ``collect`` deploys on the specified platform and distro, patches with the
+  requested deb or rpm, and finally collects output of the arbitrary
+  commands. Similarly, ```tree_collect`` will collect output using a deb
+  built from the current working tree.
+
+* ``verify`` given a directory of test data, run the Python unit tests on
+  it to generate results.
+
+* ``bddeb`` will build a deb of the current working tree.
+
+Run
+---
+
+The first example will provide a complete end-to-end run of data
+collection and verification. There are additional examples below
+explaining how to run one or the other independently.
+
+.. code-block:: bash
+
+    $ git clone https://git.launchpad.net/cloud-init
+    $ cd cloud-init
+    $ tox -e citest -- run --verbose \
+        --os-name stretch --os-name xenial \
+        --deb cloud-init_0.7.8~my_patch_all.deb \
+        --preserve-data --data-dir ~/collection
+
+The above command will do the following:
+
+* ``run`` both collect output and run tests the output
+
+* ``--verbose`` verbose output
+
+* ``--os-name stretch`` on the Debian Stretch release
+
+* ``--os-name xenial`` on the Ubuntu Xenial release
+
+* ``--deb cloud-init_0.7.8~patch_all.deb`` use this deb as the version of
+  cloud-init to run with
+
+* ``--preserve-data`` always preserve collected data, do not remove data
+  after successful test run
+
+* ``--data-dir ~/collection`` write collected data into `~/collection`,
+  rather than using a temporary directory
+
+For a more detailed explanation of each option see below.
+
+.. note::
+    By default, data collected by the run command will be written into a
+    temporary directory and deleted after a successful. If you would
+    like to preserve this data, please use the option ``--preserve-data``.
+
+Collect
+-------
+
+If developing tests it may be necessary to see if cloud-config works as
+expected and the correct files are pulled down. In this case only a
+collect can be ran by running:
+
+.. code-block:: bash
+
+    $ tox -e citest -- collect -n xenial --data-dir /tmp/collection
+
+The above command will run the collection tests on xenial and place
+all results into `/tmp/collection`.
+
+Verify
+------
+
+When developing tests it is much easier to simply rerun the verify scripts
+without the more lengthy collect process. This can be done by running:
+
+.. code-block:: bash
+
+    $ tox -e citest -- verify --data-dir /tmp/collection
+
+The above command will run the verify scripts on the data discovered in
+`/tmp/collection`.
+
+TreeRun and TreeCollect
+-----------------------
+
+If working on a cloud-init feature or resolving a bug, it may be useful to
+run the current copy of cloud-init in the integration testing environment.
+The integration testing suite can automatically build a deb based on the 
+current working tree of cloud-init and run the test suite using this deb.
+
+The ``tree_run`` and ``tree_collect`` commands take the same arguments as
+the ``run`` and ``collect`` commands. These commands will build a deb and 
+write it into a temporary file, then start the test suite and pass that deb 
+in. To build a deb only, and not run the test suite, the ``bddeb`` command
+can be used.
+
+Note that code in the cloud-init working tree that has not been committed
+when the cloud-init deb is built will still be included. To build a
+cloud-init deb from or use the ``tree_run`` command using a copy of
+cloud-init located in a different directory, use the option ``--cloud-init 
+/path/to/cloud-init``.
+
+.. code-block:: bash
+
+    $ tox -e citest -- tree_run --verbose \
+        --os-name xenial --os-name stretch \
+        --test modules/final_message --test modules/write_files \
+        --result /tmp/result.yaml
+
+Bddeb
+-----
+
+The ``bddeb`` command can be used to generate a deb file. This is used by
+the tree_run and tree_collect commands to build a deb of the current
+working tree. It can also be used a user to generate a deb for use in other
+situations and avoid needing to have all the build and test dependencies
+installed locally.
+
+* ``--bddeb-args``: arguments to pass through to bddeb
+* ``--build-os``: distribution to use as build system (default is xenial)
+* ``--build-platform``: platform to use for build system (default is lxd)
+* ``--cloud-init``: path to base of cloud-init tree (default is '.')
+* ``--deb``: path to write output deb to (default is '.')
+
+Setup Image
+-----------
+
+By default an image that is used will remain unmodified, but certain
+scenarios may require image modification. For example, many images may use
+a much older cloud-init. As a result tests looking at newer functionality
+will fail because a newer version of cloud-init may be required. The
+following options can be used for further customization:
+
+* ``--deb``: install the specified deb into the image
+* ``--rpm``: install the specified rpm into the image
+* ``--repo``: enable a repository and upgrade cloud-init afterwards
+* ``--ppa``: enable a ppa and upgrade cloud-init afterwards
+* ``--upgrade``: upgrade cloud-init from repos
+* ``--upgrade-full``: run a full system upgrade
+* ``--script``: execute a script in the image. This can perform any setup
+  required that is not covered by the other options
+
+Test Case Development
+=====================
+
+Overview
+--------
+
+As a test writer you need to develop a test configuration and a
+verification file:
 
  * The test configuration specifies a specific cloud-config to be used by
    cloud-init and a list of arbitrary commands to capture the output of
@@ -21,20 +193,28 @@ The names must match, however the extensions will of course be different,
 yaml vs py.
 
 Configuration
-=============
+-------------
 
 The test configuration is a YAML file such as *ntp_server.yaml* below:
 
 .. code-block:: yaml
 
     #
-    # NTP config using specific servers (ntp_server.yaml)
+    # Empty NTP config to setup using defaults
     #
+    # NOTE: this should not require apt feature, use 'which' rather than 'dpkg -l'
+    # NOTE: this should not require no_ntpdate feature, use 'which' to check for
+    #       installation rather than 'dpkg -l', as 'grep ntp' matches 'ntpdate'
+    # NOTE: the verifier should check for any ntp server not 'ubuntu.pool.ntp.org'
     cloud_config: |
       #cloud-config
       ntp:
         servers:
           - pool.ntp.org
+    required_features:
+      - apt
+      - no_ntpdate
+      - ubuntu_ntp
     collect_scripts:
       ntp_installed_servers: |
         #!/bin/bash
@@ -46,21 +226,30 @@ The test configuration is a YAML file such as *ntp_server.yaml* below:
         #!/bin/bash
         cat /etc/ntp.conf | grep '^server'
 
-
-There are two keys, 1 required and 1 optional, in the YAML file:
+There are several keys, 1 required and some optional, in the YAML file:
 
 1. The required key is ``cloud_config``. This should be a string of valid
-   YAML that is exactly what would normally be placed in a cloud-config file,
-   including the cloud-config header. This essentially sets up the scenario
-   under test.
+   YAML that is exactly what would normally be placed in a cloud-config
+   file, including the cloud-config header. This essentially sets up the
+   scenario under test.
 
-2. The optional key is ``collect_scripts``. This key has one or more
+2. One optional key is ``collect_scripts``. This key has one or more
    sub-keys containing strings of arbitrary commands to execute (e.g.
    ```cat /var/log/cloud-config-output.log```). In the example above the
    output of dpkg is captured, grep for ntp, and the number of lines
    reported. The name of the sub-key is important. The sub-key is used by
    the verification script to recall the output of the commands ran.
 
+3. The optional ``enabled`` key enables or disables the test case. By
+   default the test case will be enabled.
+
+4. The optional ``required_features`` key may be used to specify a list
+   of features flags that an image must have to be able to run the test
+   case. For example, if a test case relies on an image supporting apt,
+   then the config for the test case should include ``required_features:
+   [ apt ]``.
+
+
 Default Collect Scripts
 -----------------------
 
@@ -75,51 +264,68 @@ no need to specify these items:
 * ```dpkg-query -W -f='${Version}' cloud-init```
 
 Verification
-============
+------------
 
 The verification script is a Python file with unit tests like the one,
 `ntp_server.py`, below:
 
 .. code-block:: python
 
-    """cloud-init Integration Test Verify Script (ntp_server.yaml)"""
+    # This file is part of cloud-init. See LICENSE file for license information.
+
+    """cloud-init Integration Test Verify Script"""
     from tests.cloud_tests.testcases import base
 
 
-    class TestNtpServers(base.CloudTestCase):
+    class TestNtp(base.CloudTestCase):
         """Test ntp module"""
 
         def test_ntp_installed(self):
             """Test ntp installed"""
-            out = self.get_data_file('ntp_installed_servers')
+            out = self.get_data_file('ntp_installed_empty')
             self.assertEqual(1, int(out))
 
         def test_ntp_dist_entries(self):
             """Test dist config file has one entry"""
-            out = self.get_data_file('ntp_conf_dist_servers')
+            out = self.get_data_file('ntp_conf_dist_empty')
             self.assertEqual(1, int(out))
 
         def test_ntp_entires(self):
             """Test config entries"""
-            out = self.get_data_file('ntp_conf_servers')
-            self.assertIn('server pool.ntp.org iburst', out)
+            out = self.get_data_file('ntp_conf_empty')
+            self.assertIn('pool 0.ubuntu.pool.ntp.org iburst', out)
+            self.assertIn('pool 1.ubuntu.pool.ntp.org iburst', out)
+            self.assertIn('pool 2.ubuntu.pool.ntp.org iburst', out)
+            self.assertIn('pool 3.ubuntu.pool.ntp.org iburst', out)
+
+    # vi: ts=4 expandtab
 
 
 Here is a breakdown of the unit test file:
 
 * The import statement allows access to the output files.
 
-* The class can be named anything, but must import the ``base.CloudTestCase``
+* The class can be named anything, but must import the
+  ``base.CloudTestCase``, either directly or via another test class.
 
 * There can be 1 to N number of functions with any name, however only
-  tests starting with ``test_*`` will be executed.
+  functions starting with ``test_*`` will be executed.
+
+* There can be 1 to N number of classes in a test module, however only
+  classes inheriting from ``base.CloudTestCase`` will be loaded.
 
 * Output from the commands can be accessed via
   ``self.get_data_file('key')`` where key is the sub-key of
   ``collect_scripts`` above.
 
+* The cloud config that the test ran with can be accessed via
+  ``self.cloud_config``, or any entry from the cloud config can be accessed
+  via ``self.get_config_entry('key')``.
+
+* See the base ``CloudTestCase`` for additional helper functions.
+
 Layout
-======
+------
 
 Integration tests are located under the `tests/cloud_tests` directory.
 Test configurations are placed under `configs` and the test verification
@@ -144,126 +350,65 @@ The sub-folders of bugs, examples, main, and modules help organize the
 tests. View the README.md in each to understand in more detail each
 directory.
 
+Test Creation Helper
+--------------------
+
+The integration testing suite has a built in helper to aid in test
+development. Help can be invoked via ``tox -e citest -- create --help``. It
+can create a template test case config file with user data passed in from
+the command line, as well as a template test case verifier module.
+
+The following would create a test case named ``example`` under the
+``modules`` category with the given description, and cloud config data read
+in from ``/tmp/user_data``.
+
+.. code-block:: bash
+
+    $ tox -e citest -- create modules/example \
+        -d "a simple example test case" -c "$(< /tmp/user_data)"
+
 
 Development Checklist
-=====================
+---------------------
 
 * Configuration File
-    * Named 'your_test_here.yaml'
+    * Named 'your_test.yaml'
     * Contains at least a valid cloud-config
     * Optionally, commands to capture additional output
     * Valid YAML
     * Placed in the appropriate sub-folder in the configs directory
+    * Any image features required for the test are specified
 * Verification File
-    * Named 'your_test_here.py'
+    * Named 'your_test.py'
     * Valid unit tests validating output collected
     * Passes pylint & pep8 checks
-    * Placed in the appropriate sub-folder in the testcases directory
+    * Placed in the appropriate sub-folder in the test cases directory
 * Tested by running the test: 
 
    .. code-block:: bash
 
-       $ python3 -m tests.cloud_tests run -v -n <release of choice> \
-           --deb <build of cloud-init> \
-           -t tests/cloud_tests/configs/<dir>/your_test_here.yaml
-
-
-Execution
-=========
-
-Executing tests has three options:
-
-* ``run`` an alias to run both ``collect`` and ``verify``
-
-* ``collect`` deploys on the specified platform and os, patches with the
-  requested deb or rpm, and finally collects output of the arbitrary
-  commands.
-
-* ``verify`` given a directory of test data, run the Python unit tests on
-  it to generate results.
-
-Run
----
-The first example will provide a complete end-to-end run of data
-collection and verification. There are additional examples below
-explaining how to run one or the other independently.
-
-.. code-block:: bash
-
-    $ git clone https://git.launchpad.net/cloud-init
-    $ cd cloud-init
-    $ python3 -m tests.cloud_tests run -v -n trusty -n xenial \
-        --deb cloud-init_0.7.8~my_patch_all.deb
-
-The above command will do the following:
-
-* ``-v`` verbose output
-
-* ``run`` both collect output and run tests the output
-
-* ``-n trusty`` on the Ubuntu Trusty release
-
-* ``-n xenial`` on the Ubuntu Xenial release
-
-* ``--deb cloud-init_0.7.8~patch_all.deb`` use this deb as the version of
-  cloud-init to run with
-
-For a more detailed explanation of each option see below.
-
-Collect
--------
-
-If developing tests it may be necessary to see if cloud-config works as
-expected and the correct files are pulled down. In this case only a
-collect can be ran by running:
-
-.. code-block:: bash
-
-    $ python3 -m tests.cloud_tests collect -n xenial -d /tmp/collection \
-        --deb cloud-init_0.7.8~my_patch_all.deb 
-
-The above command will run the collection tests on Xenial with the
-provided deb and place all results into `/tmp/collection`.
-
-Verify
-------
-
-When developing tests it is much easier to simply rerun the verify scripts
-without the more lengthy collect process. This can be done by running:
-
-.. code-block:: bash
-
-    $ python3 -m tests.cloud_tests verify -d /tmp/collection
-
-The above command will run the verify scripts on the data discovered in
-`/tmp/collection`.
-
-Run via tox
------------
-In order to avoid the need for dependencies and ease the setup and
-configuration users can run the integration tests via tox:
-
-.. code-block:: bash
-
-    $ tox -e citest -- run [integration test arguments]
-    $ tox -e citest -- run -v -n zesty --deb=cloud-init_all.deb
-    $ tox -e citest -- run -t module/user_groups.yaml
-
-Users need to invoke the citest environment and then pass any additional
-arguments.
-
+       $ tox -e citest -- run -verbose \
+           --os-name <release target> \
+           --test modules/your_test.yaml \
+           [--deb <build of cloud-init>]
 
 Architecture
 ============
 
-The following outlines the process flow during a complete end-to-end LXD-backed test.
+The following section outlines the high-level architecture of the
+integration process.
+
+Overview
+--------
+The process flow during a complete end-to-end LXD-backed test.
 
 1. Configuration
-    * The back end and specific OS releases are verified as supported
-    * The test or tests that need to be run are determined either by directory or by individual yaml
+    * The back end and specific distro releases are verified as supported
+    * The test or tests that need to be run are determined either by
+      directory or by individual yaml
 
 2. Image Creation
-    * Acquire the daily LXD image
+    * Acquire the request LXD image
     * Install the specified cloud-init package
     * Clean the image so that it does not appear to have been booted
     * A snapshot of the image is created and reused by all tests
@@ -285,5 +430,247 @@ The following outlines the process flow during a complete end-to-end LXD-backed
 
 5. Results
     * If any failures were detected the test suite returns a failure
+    * Results can be dumped in yaml format to a specified file using the
+      ``-r <result_file_name>.yaml`` option
+
+Configuring the Test Suite
+--------------------------
+
+Most of the behavior of the test suite is configurable through several yaml
+files. These control the behavior of the test suite's platforms, images, and
+tests. The main config files for platforms, images and test cases are
+``platforms.yaml``, ``releases.yaml`` and ``testcases.yaml``.
 
+Config handling
+^^^^^^^^^^^^^^^
+
+All configurable parts of the test suite use a defaults + overrides system
+for managing config entries. All base config items are dictionaries.
+
+Merging is done on a key-by-key basis, with all keys in the default and
+override represented in the final result. If a key exists both in
+the defaults and the overrides, then the behavior depends on the type of data
+the key refers to. If it is atomic data or a list, then the overrides will
+replace the default. If the data is a dictionary then the value will be the
+result of merging that dictionary from the default config and that
+dictionary from the overrides.
+
+Merging is done using the function 
+``tests.cloud_tests.config.merge_config``, which can be examined for more
+detail on config merging behavior.
+
+The following demonstrates merge behavior:
+
+.. code-block:: yaml
+
+    defaults:
+        list_item:
+         - list_entry_1
+         - list_entry_2
+        int_item_1: 123
+        int_item_2: 234
+        dict_item:
+            subkey_1: 1
+            subkey_2: 2
+            subkey_dict:
+                subsubkey_1: a
+                subsubkey_2: b
+
+    overrides:
+        list_item:
+         - overridden_list_entry
+        int_item_1: 0
+        dict_item:
+            subkey_2: false
+            subkey_dict:
+                subsubkey_2: 'new value'
+
+    result:
+        list_item:
+         - overridden_list_entry
+        int_item_1: 0
+        int_item_2: 234
+        dict_item:
+            subkey_1: 1
+            subkey_2: false
+            subkey_dict:
+                subsubkey_1: a
+                subsubkey_2: 'new value'
+
+
+Image Config
+------------
+
+Image configuration is handled in ``releases.yaml``. The image configuration
+controls how platforms locate and acquire images, how the platforms should
+interact with the images, how platforms should detect when an image has
+fully booted, any options that are required to set the image up, and
+features that the image supports.
+
+Since settings for locating an image and interacting with it differ from
+platform to platform, there are 4 levels of settings available for images on
+top of the default image settings. The structure of the image config file
+is:
+
+.. code-block:: yaml
+
+    default_release_config:
+        default:
+            ...
+        <platform>:
+            ...
+        <platform>:
+            ...
+
+    releases:
+        <release name>:
+            <default>:
+                ...
+            <platform>:
+                ...
+            <platform>:
+                ...
+
+
+The base config is created from the overall defaults and the overrides for
+the platform. The overrides are created from the default config for the
+image and the platform specific overrides for the image.
+
+System Boot
+^^^^^^^^^^^
+
+The test suite must be able to test if a system has fully booted and if
+cloud-init has finished running, so that running collect scripts does not
+race against the target image booting. This is done using the
+``system_ready_script`` and ``cloud_init_ready_script`` image config keys.
+
+Each of these keys accepts a small bash test statement as a string that must
+return 0 or 1. Since this test statement will be added into a larger bash
+statement it must be a single statement using the ``[`` test syntax.
+
+The default image config provides a system ready script that works for any
+systemd based image. If the image is not systemd based, then a different
+test statement must be provided. The default config also provides a test
+for whether or not cloud-init has finished which checks for the file
+``/run/cloud-init/result.json``. This should be sufficient for most systems
+as writing this file is one of the last things cloud-init does.
+
+The setting ``boot_timeout`` controls how long, in seconds, the platform
+should wait for an image to boot. If the system ready script has not
+indicated that the system is fully booted within this time an error will be
+raised.
+
+Feature Flags
+^^^^^^^^^^^^^
+
+Not all test cases can work on all images due to features the test case
+requires not being present on that image. If a test case requires features
+in an image that are not likely to be present across all distros and
+platforms that the test suite supports, then the test can be skipped
+everywhere it is not supported.
+
+Feature flags, which are names for features supported on some images, but
+not all that may be required by test cases. Configuration for feature flags
+is provided in ``releases.yaml`` under the ``features`` top level key. The
+features config includes a list of all currently defined feature flags,
+their meanings, and a list of feature groups.
+
+Feature groups are groups of features that many images have in common. For
+example, the ``Ubuntu_specific`` feature group includes features that
+should be present across most Ubuntu releases, but may or may not be for
+other distros. Feature groups are specified for an image as a list under
+the key ``feature_groups``.
+
+An image's feature flags are derived from the features groups that that
+image has and any feature overrides provided. Feature overrides can be
+specified under the ``features`` key which accepts a dictionary of
+``{<feature_name>: true/false}`` mappings. If a feature is omitted from an
+image's feature flags or set to false in the overrides then the test suite
+will skip any tests that require that feature when using that image.
+
+Feature flags may be overridden at run time using the ``--feature-override``
+command line argument. It accepts a feature flag and value to set in the
+format ``<feature name>=true/false``. Multiple ``--feature-override``
+flags can be used, and will all be applied to all feature flags for images
+used during a test.
+
+Setup Overrides
+^^^^^^^^^^^^^^^
+
+If an image requires some of the options for image setup to be used, then it
+may specify overrides for the command line arguments passed into setup
+image. These may be specified as a dictionary under the ``setup_overrides``
+key. When an image is set up, the arguments that control how it is set up
+will be the arguments from the command line, with any entries in
+``setup_overrides`` used to override these arguments.
+
+For example, images that do not come with cloud-init already installed
+should have ``setup_overrides: {upgrade: true}`` specified so that in the
+event that no additional setup options are given, cloud-init will be
+installed from the image's repos before running tests. Note that if other
+options such as ``--deb`` are passed in on the command line, these will
+still work as expected, since apt's policy for cloud-init would prefer the
+locally installed deb over an older version from the repos.
+
+Platform Specific Options
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are many platform specific options in image configuration that allow
+platforms to locate images and that control additional setup that the
+platform may have to do to make the image usable. For information on how
+these work, please consult the documentation for that platform in the
+integration testing suite and the ``releases.yaml`` file for examples.
+
+Error Handling
+--------------
+
+The test suite makes an attempt to run as many tests as possible even in the
+event of some failing so that automated runs collect as much data as
+possible. In the event that something goes wrong while setting up for or
+running a test, the test suite will attempt to continue running any tests
+which have not been affected by the error.
+
+For example, if the test suite was told to run tests on one platform for two
+releases and an error occurred setting up the first image, all tests for
+that image would be skipped, and the test suite would continue to set up
+the second image and run tests on it. Or, if the system does not start
+properly for one test case out of many to run on that image, that test case
+will be skipped and the next one will be run.
+
+Note that if any errors occur, the test suite will record the failure and
+where it occurred in the result data and write it out to the specified
+result file.
+
+Results
+-------
 
+The test suite generates result data that includes how long each stage of
+the test suite took and which parts were and were not successful. This data
+is dumped to the log after the collect and verify stages, and may also be
+written out in yaml format to a file. If part of the setup failed, the
+traceback for the failure and the error message will be included in the
+result file. If a test verifier finds a problem with the collected data
+from a test run, the class, test function and test will be recorded in the
+result data.
+
+Exit Codes
+^^^^^^^^^^
+
+The test suite counts how many errors occur throughout a run. The exit code
+after a run is the number of errors that occurred. If the exit code is
+non-zero then something is wrong either with the test suite, the
+configuration for an image, a test case, or cloud-init itself.
+
+Note that the exit code does not always directly correspond to the number
+of failed test cases, since in some cases, a single error during image setup
+can mean that several test cases are not run. If run is used, then the exit
+code will be the sum of the number of errors in the collect and verify
+stages.
+
+Data Dir
+^^^^^^^^
+
+When using run, the collected data is written into a temporary directory. In
+the event that all tests pass, this directory is deleted, but if a test
+fails or an error occurs, this data will be left in place, and a message
+will be written to the log giving the location of the data.