summaryrefslogtreecommitdiff
path: root/doc/rtd
diff options
context:
space:
mode:
authorWesley Wiedenmeier <wesley.wiedenmeier@gmail.com>2016-12-22 17:27:37 -0500
committerScott Moser <smoser@brickies.net>2016-12-22 17:41:39 -0500
commitf53fc46aa732e3b29991b3e5e39da31a722945ee (patch)
treea301733aa9991b58b218f61b187240d275e44968 /doc/rtd
parentb2a9f33616c806ae6e052520a8589113308f567c (diff)
downloadvyos-cloud-init-f53fc46aa732e3b29991b3e5e39da31a722945ee.tar.gz
vyos-cloud-init-f53fc46aa732e3b29991b3e5e39da31a722945ee.zip
integration test: initial commit of integration test framework
The adds in end-to-end testing of cloud-init. The framework utilizes LXD and cloud images as a backend to test user-data passed in. Arbitrary data is then captured from predefined commands specified by the user. After collection, data verification is completed by running a series of Python unit tests against the collected data. Currently only the Ubuntu Trusty, Xenial, Yakkety, and Zesty releases are supported. Test cases for 50% of the modules is complete and available. Additionally a Read the Docs file was created to guide test writing and execution.
Diffstat (limited to 'doc/rtd')
-rw-r--r--doc/rtd/index.rst1
-rw-r--r--doc/rtd/topics/tests.rst281
2 files changed, 282 insertions, 0 deletions
diff --git a/doc/rtd/index.rst b/doc/rtd/index.rst
index 3caf33f3..8f163b6a 100644
--- a/doc/rtd/index.rst
+++ b/doc/rtd/index.rst
@@ -41,6 +41,7 @@ initialization of a cloud instance.
topics/vendordata.rst
topics/moreinfo.rst
topics/hacking.rst
+ topics/tests.rst
.. _Cloud-init: https://launchpad.net/cloud-init
.. vi: textwidth=78
diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
new file mode 100644
index 00000000..56126aee
--- /dev/null
+++ b/doc/rtd/topics/tests.rst
@@ -0,0 +1,281 @@
+.. contents:: Table of Contents
+ :depth: 2
+
+============================
+Test Development
+============================
+
+
+Overview
+--------
+
+The purpose of this page is to describe how to write integration tests for
+cloud-init. As a test writer you need to develop a test configuration and
+a verification file:
+
+ * The test configuration specifies a specific cloud-config to be used by
+ cloud-init and a list of arbitrary commands to capture the output of
+ (e.g my_test.yaml)
+
+ * The verification file runs tests on the collected output to determine
+ the result of the test (e.g. my_test.py)
+
+The names must match, however the extensions will of course be different,
+yaml vs py.
+
+Configuration
+-------------
+
+The test configuration is a YAML file such as *ntp_server.yaml* below:
+
+.. code-block:: yaml
+
+ #
+ # NTP config using specific servers (ntp_server.yaml)
+ #
+ cloud_config: |
+ #cloud-config
+ ntp:
+ servers:
+ - pool.ntp.org
+ collect_scripts:
+ ntp_installed_servers: |
+ #!/bin/bash
+ dpkg -l | grep ntp | wc -l
+ ntp_conf_dist_servers: |
+ #!/bin/bash
+ ls /etc/ntp.conf.dist | wc -l
+ ntp_conf_servers: |
+ #!/bin/bash
+ cat /etc/ntp.conf | grep '^server'
+
+
+There are two keys, 1 required and 1 optional, in the YAML file:
+
+1. The required key is ``cloud_config``. This should be a string of valid
+ YAML that is exactly what would normally be placed in a cloud-config file,
+ including the cloud-config header. This essentially sets up the scenario
+ under test.
+
+2. The optional key is ``collect_scripts``. This key has one or more
+ sub-keys containing strings of arbitrary commands to execute (e.g.
+ ```cat /var/log/cloud-config-output.log```). In the example above the
+ output of dpkg is captured, grep for ntp, and the number of lines
+ reported. The name of the sub-key is important. The sub-key is used by
+ the verification script to recall the output of the commands ran.
+
+Default Collect Scripts
+~~~~~~~~~~~~~~~~~~~~~~~
+
+By default the following files will be collected for every test. There is
+no need to specify these items:
+
+* ``/var/log/cloud-init.log``
+* ``/var/log/cloud-init-output.log``
+* ``/run/cloud-init/.instance-id``
+* ``/run/cloud-init/result.json``
+* ``/run/cloud-init/status.json``
+* ```dpkg-query -W -f='${Version}' cloud-init```
+
+Verification
+------------
+
+The verification script is a Python file with unit tests like the one,
+`ntp_server.py`, below:
+
+.. code-block:: python
+
+ """cloud-init Integration Test Verify Script (ntp_server.yaml)"""
+ from tests.cloud_tests.testcases import base
+
+
+ class TestNtpServers(base.CloudTestCase):
+ """Test ntp module"""
+
+ def test_ntp_installed(self):
+ """Test ntp installed"""
+ out = self.get_data_file('ntp_installed_servers')
+ self.assertEqual(1, int(out))
+
+ def test_ntp_dist_entries(self):
+ """Test dist config file has one entry"""
+ out = self.get_data_file('ntp_conf_dist_servers')
+ self.assertEqual(1, int(out))
+
+ def test_ntp_entires(self):
+ """Test config entries"""
+ out = self.get_data_file('ntp_conf_servers')
+ self.assertIn('server pool.ntp.org iburst', out)
+
+
+Here is a breakdown of the unit test file:
+
+* The import statement allows access to the output files.
+
+* The class can be named anything, but must import the ``base.CloudTestCase``
+
+* There can be 1 to N number of functions with any name, however only
+ tests starting with ``test_*`` will be executed.
+
+* Output from the commands can be accessed via
+ ``self.get_data_file('key')`` where key is the sub-key of
+ ``collect_scripts`` above.
+
+Layout
+------
+
+Integration tests are located under the `tests/cloud_tests` directory.
+Test configurations are placed under `configs` and the test verification
+scripts under `testcases`:
+
+.. code-block:: bash
+
+ cloud-init$ tree -d tests/cloud_tests/
+ tests/cloud_tests/
+ ├── configs
+ │   ├── bugs
+ │   ├── examples
+ │   ├── main
+ │   └── modules
+ └── testcases
+ ├── bugs
+ ├── examples
+ ├── main
+ └── modules
+
+The sub-folders of bugs, examples, main, and modules help organize the
+tests. View the README.md in each to understand in more detail each
+directory.
+
+
+=====================
+Development Checklist
+=====================
+
+* Configuration File
+ * Named 'your_test_here.yaml'
+ * Contains at least a valid cloud-config
+ * Optionally, commands to capture additional output
+ * Valid YAML
+ * Placed in the appropriate sub-folder in the configs directory
+* Verification File
+ * Named 'your_test_here.py'
+ * Valid unit tests validating output collected
+ * Passes pylint & pep8 checks
+ * Placed in the appropriate sub-folder in the testcsaes directory
+* Tested by running the test:
+
+ .. code-block:: bash
+
+ $ python3 -m tests.cloud_tests run -v -n <release of choice> \
+ --deb <build of cloud-init> \
+ -t tests/cloud_tests/configs/<dir>/your_test_here.yaml
+
+
+=========
+Execution
+=========
+
+Executing tests has three options:
+
+* ``run`` an alias to run both ``collect`` and ``verify``
+
+* ``collect`` deploys on the specified platform and os, patches with the
+ requested deb or rpm, and finally collects output of the arbitrary
+ commands.
+
+* ``verify`` given a directory of test data, run the Python unit tests on
+ it to generate results.
+
+Run
+---
+The first example will provide a complete end-to-end run of data
+collection and verification. There are additional examples below
+explaining how to run one or the other independently.
+
+.. code-block:: bash
+
+ $ git clone https://git.launchpad.net/cloud-init
+ $ cd cloud-init
+ $ python3 -m tests.cloud_tests run -v -n trusty -n xenial \
+ --deb cloud-init_0.7.8~my_patch_all.deb
+
+The above command will do the following:
+
+* ``-v`` verbose output
+
+* ``run`` both collect output and run tests the output
+
+* ``-n trusty`` on the Ubuntu Trusty release
+
+* ``-n xenial`` on the Ubuntu Xenial release
+
+* ``--deb cloud-init_0.7.8~patch_all.deb`` use this deb as the version of
+ cloud-init to run with
+
+For a more detailed explanation of each option see below.
+
+Collect
+-------
+
+If developing tests it may be necessary to see if cloud-config works as
+expected and the correct files are pulled down. In this case only a
+collect can be ran by running:
+
+.. code-block:: bash
+
+ $ python3 -m tests.cloud_tests collect -n xenial -d /tmp/collection \
+ --deb cloud-init_0.7.8~my_patch_all.deb
+
+The above command will run the collection tests on xenial with the
+provided deb and place all results into `/tmp/collection`.
+
+Verify
+------
+
+When developing tests it is much easier to simply rerun the verify scripts
+without the more lengthy collect process. This can be done by running:
+
+.. code-block:: bash
+
+ $ python3 -m tests.cloud_tests verify -d /tmp/collection
+
+The above command will run the verify scripts on the data discovered in
+`/tmp/collection`.
+
+
+============
+Architecture
+============
+
+The following outlines the process flow during a complete end-to-end LXD-backed test.
+
+1. Configuration
+ * The back end and specific OS releases are verified as supported
+ * The test or tests that need to be run are determined either by directory or by individual yaml
+
+2. Image Creation
+ * Acquire the daily LXD image
+ * Install the specified cloud-init package
+ * Clean the image so that it does not appear to have been booted
+ * A snapshot of the image is created and reused by all tests
+
+3. Configuration
+ * For each test, the cloud-config is injected into a copy of the
+ snapshot and booted
+ * The framework waits for ``/var/lib/cloud/instance/boot-finished``
+ (up to 120 seconds)
+ * All default commands are ran and output collected
+ * Any commands the user specified are executed and output collected
+
+4. Verification
+ * The default commands are checked for any failures, errors, and
+ warnings to validate basic functionality of cloud-init completed
+ successfully
+ * The user generated unit tests are then ran validating against the
+ collected output
+
+5. Results
+ * If any failures were detected the test suite returns a failure
+
+