Integration testing docs and refactor (#1231)

* Include CI and Fixtures sections in integration test docs
* Incorporate additional variable annotations
* Remove unnecessary IntegrationInstance subclasses
* Move setup_image teardown into its fixture

1 files changed, 97 insertions, 17 deletions

diff --git a/doc/rtd/topics/integration_tests.rst b/doc/rtd/topics/integration_tests.rst
index 2677a213..1e910af6 100644
--- a/doc/rtd/topics/integration_tests.rst
+++ b/doc/rtd/topics/integration_tests.rst
@@ -22,30 +22,22 @@ marks can be used to limit tests to particular platforms. The
 client fixture can be used to interact with the launched
 test instance.
 
-A basic example:
-
-.. code-block:: python
-
-    USER_DATA = """#cloud-config
-    bootcmd:
-    - echo 'hello config!' > /tmp/user_data.txt"""
-
-
-    class TestSimple:
-        @pytest.mark.user_data(USER_DATA)
-        @pytest.mark.ec2
-        def test_simple(self, client):
-            print(client.exec('cloud-init -v'))
+See :ref:`Examples` section for examples.
 
 Test Execution
 ==============
-Test execution happens via pytest. To run all integration tests,
-you would run:
+Test execution happens via pytest. A tox definition exists to run integration
+tests. To run all integration tests, you would run:
 
 .. code-block:: bash
 
-    pytest tests/integration_tests/
+    $ tox -e integration-tests
 
+Pytest arguments may also be passed. For example:
+
+.. code-block:: bash
+
+    $ tox -e integration-tests tests/integration_tests/modules/test_combined.py
 
 Configuration
 =============
@@ -126,3 +118,91 @@ Test setup occurs between image setup and test execution. Test setup
 is implemented via one of the ``client`` fixtures. When a client fixture
 is used, a test instance from which to run tests is launched prior to
 test execution and torn down after.
+
+Continuous Integration
+======================
+A subset of the integration tests are run when a pull request
+is submitted on Github. The tests run on these continuous
+integration (CI) runs are given a pytest mark:
+
+.. code-block:: python
+
+    @pytest.mark.ci
+
+Most new tests should *not* use this mark, so be aware that having a
+successful CI run does not necessarily mean that your test passed
+successfully.
+
+Fixtures
+========
+Integration tests rely heavily on fixtures to do initial test setup.
+One or more of these fixtures will be used in almost every integration test.
+
+Details such as the cloud platform or initial image to use are determined
+via what is specified in the :ref:`Configuration`.
+
+client
+------
+The ``client`` fixture should be used for most test cases. It ensures:
+
+- All setup performed by :ref:`session_cloud` and :ref:`setup_image`
+- `Pytest marks <https://github.com/canonical/cloud-init/blob/af7eb1deab12c7208853c5d18b55228e0ba29c4d/tests/integration_tests/conftest.py#L220-L224>`_
+  used during instance creation are obtained and applied
+- The test instance is launched
+- Test failure status is determined after test execution
+- Logs are collected (if configured) after test execution
+- The test instance is torn down after test execution
+
+``module_client`` and ``class_client`` fixtures also exist for the
+purpose of running multiple tests against a single launched instance.
+They provide the exact same functionality as ``client``, but are
+scoped to the module or class respectively.
+
+session_cloud
+-------------
+The ``session_cloud`` session-scoped fixture will provide an
+`IntegrationCloud <https://github.com/canonical/cloud-init/blob/af7eb1deab12c7208853c5d18b55228e0ba29c4d/tests/integration_tests/clouds.py#L102>`_
+instance for the currently configured cloud. The fixture also
+ensures that any custom cloud session cleanup is performed.
+
+setup_image
+-----------
+The ``setup_image`` session-scope fixture will
+create a new image to launch all further cloud instances
+during this test run. It ensures:
+
+- A cloud instance is launched on the configured platform
+- The version of cloud-init under test is installed on the instance
+- ``cloud-init clean --logs`` is run on the instance
+- A snapshot of the instance is taken to be used as the basis for
+  future instance launches
+- The originally launched instance is torn down
+- The custom created image is torn down after all tests finish
+
+Examples
+--------
+A simple test case using the ``client`` fixture:
+
+.. code-block:: python
+
+    USER_DATA = """\
+    #cloud-config
+    bootcmd:
+    - echo 'hello!' > /var/tmp/hello.txt
+    """
+
+
+    @pytest.mark.user_data(USER_DATA)
+    def test_bootcmd(client):
+        log = client.read_from_file("/var/log/cloud-init.log")
+        assert "Shellified 1 commands." in log
+        assert client.execute('cat /var/tmp/hello.txt').strip() == "hello!"
+
+Customizing the launch arguments before launching an instance manually:
+
+.. code-block:: python
+
+    def test_launch(session_cloud: IntegrationCloud, setup_image):
+        with session_cloud.launch(launch_kwargs={"wait": False}) as client:
+            client.instance.wait()
+            assert client.execute("echo hello world").strip() == "hello world"