tests: Enable AWS EC2 Integration Testing

This enables integration tests to utilize AWS EC2 as a testing platform by
utilizing the boto3 Python library.

Usage will create and delete a custom VPC for every run. All resources
will be tagged with the ec2 tag, 'cii', and the date (e.g.
cii-20171220-102452).  The VPC is setup with both IPv4 and IPv6
capabilities, but will only hand out IPv4 addresses by default. Instances
will have complete Internet access and have full ingress and egress access
(i.e. no firewall).

SSH keys are generated with each run of the integration tests with the key
getting uploaded to AWS at the start of tests and deleted on exit.  To
enable creation when the platform is setup the SSH generation code is
moved to be completed by the platform setup and not during image setup.
The nocloud-kvm platform was updated with this change.

Creating a custom image will utilize the same clean script,
boot_clean_script, that the LXD platform uses as well. The custom AMI is
generated, used, and de-registered after a test run.

The default instance type is set to t2.micro. This is one of the smallest
instance types and is free tier eligible.

The default timeout for ec2 was increased to 300 from 120 as many tests
hit up against the 2 minute timeout and depending on region load can
go over.

Documentation for the AWS platform was added with the expected
configuration files for the platform to be used. There are some
additional whitespace changes included as well.

pylint exception was added for paramiko and simplestreams. In the past
these were not already flagged due to no __init__.py in the subdirectories
of files that used these. boto3 was added to the list of dependencies in
the tox ci-test runner.

In order to grab console logs on EC2 the harness will now shut down an
instance before terminating and before collecting the console log.  This
is to address a behavior of EC2 where the console log is refreshed very
infrequently, but one point when it is refreshed is after shutdown.

1 files changed, 32 insertions, 6 deletions

diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
index d668e3f4..bf04bb3c 100644
--- a/doc/rtd/topics/tests.rst
+++ b/doc/rtd/topics/tests.rst
@@ -118,19 +118,19 @@ TreeRun and TreeCollect
 
 If working on a cloud-init feature or resolving a bug, it may be useful to
 run the current copy of cloud-init in the integration testing environment.
-The integration testing suite can automatically build a deb based on the 
+The integration testing suite can automatically build a deb based on the
 current working tree of cloud-init and run the test suite using this deb.
 
 The ``tree_run`` and ``tree_collect`` commands take the same arguments as
-the ``run`` and ``collect`` commands. These commands will build a deb and 
-write it into a temporary file, then start the test suite and pass that deb 
+the ``run`` and ``collect`` commands. These commands will build a deb and
+write it into a temporary file, then start the test suite and pass that deb
 in. To build a deb only, and not run the test suite, the ``bddeb`` command
 can be used.
 
 Note that code in the cloud-init working tree that has not been committed
 when the cloud-init deb is built will still be included. To build a
 cloud-init deb from or use the ``tree_run`` command using a copy of
-cloud-init located in a different directory, use the option ``--cloud-init 
+cloud-init located in a different directory, use the option ``--cloud-init
 /path/to/cloud-init``.
 
 .. code-block:: bash
@@ -383,7 +383,7 @@ Development Checklist
     * Valid unit tests validating output collected
     * Passes pylint & pep8 checks
     * Placed in the appropriate sub-folder in the test cases directory
-* Tested by running the test: 
+* Tested by running the test:
 
    .. code-block:: bash
 
@@ -392,6 +392,32 @@ Development Checklist
            --test modules/your_test.yaml \
            [--deb <build of cloud-init>]
 
+
+Platforms
+=========
+
+EC2
+---
+To run on the EC2 platform it is required that the user has an AWS credentials
+configuration file specifying his or her access keys and a default region.
+These configuration files are the standard that the AWS cli and other AWS
+tools utilize for interacting directly with AWS itself and are normally
+generated when running ``aws configure``:
+
+.. code-block:: bash
+
+    $ cat $HOME/.aws/credentials
+    [default]
+    aws_access_key_id = <KEY HERE>
+    aws_secret_access_key = <KEY HERE>
+
+.. code-block:: bash
+
+    $ cat $HOME/.aws/config
+    [default]
+    region = us-west-2
+
+
 Architecture
 ============
 
@@ -455,7 +481,7 @@ replace the default. If the data is a dictionary then the value will be the
 result of merging that dictionary from the default config and that
 dictionary from the overrides.
 
-Merging is done using the function 
+Merging is done using the function
 ``tests.cloud_tests.config.merge_config``, which can be examined for more
 detail on config merging behavior.