diff options
| -rw-r--r-- | HACKING.rst | 160 | ||||
| -rw-r--r-- | doc/rtd/index.rst | 1 | ||||
| -rw-r--r-- | doc/rtd/topics/debugging.rst | 6 | ||||
| -rw-r--r-- | doc/rtd/topics/integration_tests.rst | 2 | ||||
| -rw-r--r-- | doc/rtd/topics/testing.rst | 173 |
5 files changed, 183 insertions, 159 deletions
diff --git a/HACKING.rst b/HACKING.rst index 8a12e3e3..6ce4397d 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -173,154 +173,11 @@ Cloud Config Modules * Any new modules should use underscores in any new config options and not hyphens (e.g. `new_option` and *not* `new-option`). -.. _unit_testing: - -Testing ------------- - -cloud-init has both unit tests and integration tests. Unit tests can -be found in-tree alongside the source code, as well as -at ``tests/unittests``. Integration tests can be found at -``tests/integration_tests``. Documentation specifically for integration -tests can be found on the :ref:`integration_tests` page, but -the guidelines specified below apply to both types of tests. - -cloud-init uses `pytest`_ to run its tests, and has tests written both -as ``unittest.TestCase`` sub-classes and as un-subclassed pytest tests. -The following guidelines should be followed: - -* For ease of organisation and greater accessibility for developers not - familiar with pytest, all cloud-init unit tests must be contained - within test classes - - * Put another way, module-level test functions should not be used - -* pytest test classes should use `pytest fixtures`_ to share - functionality instead of inheritance - -* As all tests are contained within classes, it is acceptable to mix - ``TestCase`` test classes and pytest test classes within the same - test file - - * These can be easily distinguished by their definition: pytest - classes will not use inheritance at all (e.g. - `TestGetPackageMirrorInfo`_), whereas ``TestCase`` classes will - subclass (indirectly) from ``TestCase`` (e.g. - `TestPrependBaseCommands`_) - -* pytest tests should use bare ``assert`` statements, to take advantage - of pytest's `assertion introspection`_ - - * For ``==`` and other commutative assertions, the expected value - should be placed before the value under test: - ``assert expected_value == function_under_test()`` - -* As we still support Ubuntu 16.04 (Xenial Xerus), we can only use - pytest features that are available in v2.8.7. This is an - inexhaustive list of ways in which this may catch you out: - - * Support for using ``yield`` in ``pytest.fixture`` functions was - only introduced in `pytest 3.0`_. Such functions must instead use - the ``pytest.yield_fixture`` decorator. - - * Only the following built-in fixtures are available - [#fixture-list]_: - - * ``cache`` - * ``capfd`` - * ``caplog`` (provided by ``python3-pytest-catchlog`` on xenial) - * ``capsys`` - * ``monkeypatch`` - * ``pytestconfig`` - * ``record_xml_property`` - * ``recwarn`` - * ``tmpdir_factory`` - * ``tmpdir`` - - * On xenial, the objects returned by the ``tmpdir`` fixture cannot be - used where paths are required; they are rejected as invalid paths. - You must instead use their ``.strpath`` attribute. - - * For example, instead of - ``util.write_file(tmpdir.join("some_file"), ...)``, you should - write ``util.write_file(tmpdir.join("some_file").strpath, ...)``. - - * The `pytest.param`_ function cannot be used. It was introduced in - pytest 3.1, which means it is not available on xenial. The more - limited mechanism it replaced was removed in pytest 4.0, so is not - available in focal or later. The only available alternatives are - to write mark-requiring test instances as completely separate - tests, without utilising parameterisation, or to apply the mark to - the entire parameterized test (and therefore every test instance). - -* Variables/parameter names for ``Mock`` or ``MagicMock`` instances - should start with ``m_`` to clearly distinguish them from non-mock - variables - - * For example, ``m_readurl`` (which would be a mock for ``readurl``) - -* The ``assert_*`` methods that are available on ``Mock`` and - ``MagicMock`` objects should be avoided, as typos in these method - names may not raise ``AttributeError`` (and so can cause tests to - silently pass). An important exception: if a ``Mock`` is - `autospecced`_ then misspelled assertion methods *will* raise an - ``AttributeError``, so these assertion methods may be used on - autospecced ``Mock`` objects. - - For non-autospecced ``Mock`` s, these substitutions can be used - (``m`` is assumed to be a ``Mock``): - - * ``m.assert_any_call(*args, **kwargs)`` => ``assert - mock.call(*args, **kwargs) in m.call_args_list`` - * ``m.assert_called()`` => ``assert 0 != m.call_count`` - * ``m.assert_called_once()`` => ``assert 1 == m.call_count`` - * ``m.assert_called_once_with(*args, **kwargs)`` => ``assert - [mock.call(*args, **kwargs)] == m.call_args_list`` - * ``m.assert_called_with(*args, **kwargs)`` => ``assert - mock.call(*args, **kwargs) == m.call_args_list[-1]`` - * ``m.assert_has_calls(call_list, any_order=True)`` => ``for call in - call_list: assert call in m.call_args_list`` - - * ``m.assert_has_calls(...)`` and ``m.assert_has_calls(..., - any_order=False)`` are not easily replicated in a single - statement, so their use when appropriate is acceptable. - - * ``m.assert_not_called()`` => ``assert 0 == m.call_count`` - -* Test arguments should be ordered as follows: - - * ``mock.patch`` arguments. When used as a decorator, ``mock.patch`` - partially applies its generated ``Mock`` object as the first - argument, so these arguments must go first. - * ``pytest.mark.parametrize`` arguments, in the order specified to - the ``parametrize`` decorator. These arguments are also provided - by a decorator, so it's natural that they sit next to the - ``mock.patch`` arguments. - * Fixture arguments, alphabetically. These are not provided by a - decorator, so they are last, and their order has no defined - meaning, so we default to alphabetical. - -* It follows from this ordering of test arguments (so that we retain - the property that arguments left-to-right correspond to decorators - bottom-to-top) that test decorators should be ordered as follows: - - * ``pytest.mark.parametrize`` - * ``mock.patch`` - -* When there are multiple patch calls in a test file for the module it - is testing, it may be desirable to capture the shared string prefix - for these patch calls in a module-level variable. If used, such - variables should be named ``M_PATH`` or, for datasource tests, - ``DS_PATH``. - -.. _pytest: https://docs.pytest.org/ -.. _pytest fixtures: https://docs.pytest.org/en/latest/fixture.html -.. _TestGetPackageMirrorInfo: https://github.com/canonical/cloud-init/blob/42f69f410ab8850c02b1f53dd67c132aa8ef64f5/cloudinit/distros/tests/test_init.py\#L15 -.. _TestPrependBaseCommands: https://github.com/canonical/cloud-init/blob/master/cloudinit/tests/test_subp.py#L9 -.. _assertion introspection: https://docs.pytest.org/en/latest/assert.html -.. _pytest 3.0: https://docs.pytest.org/en/latest/changelog.html#id1093 -.. _pytest.param: https://docs.pytest.org/en/latest/reference.html#pytest-param -.. _autospecced: https://docs.python.org/3.8/library/unittest.mock.html#autospeccing +Tests +----- + +Submissions to cloud-init must include testing. See :ref:`testing` for +details on these requirements. Type Annotations ---------------- @@ -344,13 +201,6 @@ variable annotations specified in `PEP-526`_ were introduced in Python .. _PEP-484: https://www.python.org/dev/peps/pep-0484/ .. _PEP-526: https://www.python.org/dev/peps/pep-0526/ -.. [#fixture-list] This list of fixtures (with markup) can be - reproduced by running:: - - py.test-3 --fixtures -q | grep "^[^ -]" | grep -v '\(no\|capturelog\)' | sort | sed 's/.*/* ``\0``/' - - in a xenial lxd container with python3-pytest-catchlog installed. - Feature Flags ------------- diff --git a/doc/rtd/index.rst b/doc/rtd/index.rst index ddcb0b31..10e8228f 100644 --- a/doc/rtd/index.rst +++ b/doc/rtd/index.rst @@ -75,6 +75,7 @@ Having trouble? We would like to help! topics/dir_layout.rst topics/analyze.rst topics/docs.rst + topics/testing.rst topics/integration_tests.rst topics/cloud_tests.rst diff --git a/doc/rtd/topics/debugging.rst b/doc/rtd/topics/debugging.rst index 0d416f32..fb3006fe 100644 --- a/doc/rtd/topics/debugging.rst +++ b/doc/rtd/topics/debugging.rst @@ -1,6 +1,6 @@ -******************************** -Testing and debugging cloud-init -******************************** +******************** +Debugging cloud-init +******************** Overview ======== diff --git a/doc/rtd/topics/integration_tests.rst b/doc/rtd/topics/integration_tests.rst index 3cfca31e..6c124ad9 100644 --- a/doc/rtd/topics/integration_tests.rst +++ b/doc/rtd/topics/integration_tests.rst @@ -9,7 +9,7 @@ Overview Integration tests are written using pytest and are located at ``tests/integration_tests``. General design principles -laid out in :ref:`unit_testing` should be followed for integration tests. +laid out in :ref:`testing` should be followed for integration tests. Setup is accomplished via a set of fixtures located in ``tests/integration_tests/conftest.py``. diff --git a/doc/rtd/topics/testing.rst b/doc/rtd/topics/testing.rst new file mode 100644 index 00000000..5b702bd2 --- /dev/null +++ b/doc/rtd/topics/testing.rst @@ -0,0 +1,173 @@ +******* +Testing +******* + +cloud-init has both unit tests and integration tests. Unit tests can +be found in-tree alongside the source code, as well as +at ``tests/unittests``. Integration tests can be found at +``tests/integration_tests``. Documentation specifically for integration +tests can be found on the :ref:`integration_tests` page, but +the guidelines specified below apply to both types of tests. + +cloud-init uses `pytest`_ to run its tests, and has tests written both +as ``unittest.TestCase`` sub-classes and as un-subclassed pytest tests. + +Guidelines +========== + +The following guidelines should be followed. + +Test Layout +----------- + +* For ease of organisation and greater accessibility for developers not + familiar with pytest, all cloud-init unit tests must be contained + within test classes + + * Put another way, module-level test functions should not be used + +* As all tests are contained within classes, it is acceptable to mix + ``TestCase`` test classes and pytest test classes within the same + test file + + * These can be easily distinguished by their definition: pytest + classes will not use inheritance at all (e.g. + `TestGetPackageMirrorInfo`_), whereas ``TestCase`` classes will + subclass (indirectly) from ``TestCase`` (e.g. + `TestPrependBaseCommands`_) + +``pytest`` Tests +---------------- + +* pytest test classes should use `pytest fixtures`_ to share + functionality instead of inheritance + +* pytest tests should use bare ``assert`` statements, to take advantage + of pytest's `assertion introspection`_ + + * For ``==`` and other commutative assertions, the expected value + should be placed before the value under test: + ``assert expected_value == function_under_test()`` + + +``pytest`` Version Gotchas +-------------------------- + +As we still support Ubuntu 16.04 (Xenial Xerus), we can only use pytest +features that are available in v2.8.7. This is an inexhaustive list of +ways in which this may catch you out: + +* Support for using ``yield`` in ``pytest.fixture`` functions was only + introduced in `pytest 3.0`_. Such functions must instead use the + ``pytest.yield_fixture`` decorator. + +* Only the following built-in fixtures are available [#fixture-list]_: + + * ``cache`` + * ``capfd`` + * ``caplog`` (provided by ``python3-pytest-catchlog`` on xenial) + * ``capsys`` + * ``monkeypatch`` + * ``pytestconfig`` + * ``record_xml_property`` + * ``recwarn`` + * ``tmpdir_factory`` + * ``tmpdir`` + +* On xenial, the objects returned by the ``tmpdir`` fixture cannot be + used where paths are required; they are rejected as invalid paths. + You must instead use their ``.strpath`` attribute. + + * For example, instead of ``util.write_file(tmpdir.join("some_file"), + ...)``, you should write + ``util.write_file(tmpdir.join("some_file").strpath, ...)``. + +* The `pytest.param`_ function cannot be used. It was introduced in + pytest 3.1, which means it is not available on xenial. The more + limited mechanism it replaced was removed in pytest 4.0, so is not + available in focal or later. The only available alternatives are to + write mark-requiring test instances as completely separate tests, + without utilising parameterisation, or to apply the mark to the + entire parameterized test (and therefore every test instance). + +Mocking and Assertions +---------------------- + +* Variables/parameter names for ``Mock`` or ``MagicMock`` instances + should start with ``m_`` to clearly distinguish them from non-mock + variables + + * For example, ``m_readurl`` (which would be a mock for ``readurl``) + +* The ``assert_*`` methods that are available on ``Mock`` and + ``MagicMock`` objects should be avoided, as typos in these method + names may not raise ``AttributeError`` (and so can cause tests to + silently pass). An important exception: if a ``Mock`` is + `autospecced`_ then misspelled assertion methods *will* raise an + ``AttributeError``, so these assertion methods may be used on + autospecced ``Mock`` objects. + + For non-autospecced ``Mock`` s, these substitutions can be used + (``m`` is assumed to be a ``Mock``): + + * ``m.assert_any_call(*args, **kwargs)`` => ``assert + mock.call(*args, **kwargs) in m.call_args_list`` + * ``m.assert_called()`` => ``assert 0 != m.call_count`` + * ``m.assert_called_once()`` => ``assert 1 == m.call_count`` + * ``m.assert_called_once_with(*args, **kwargs)`` => ``assert + [mock.call(*args, **kwargs)] == m.call_args_list`` + * ``m.assert_called_with(*args, **kwargs)`` => ``assert + mock.call(*args, **kwargs) == m.call_args_list[-1]`` + * ``m.assert_has_calls(call_list, any_order=True)`` => ``for call in + call_list: assert call in m.call_args_list`` + + * ``m.assert_has_calls(...)`` and ``m.assert_has_calls(..., + any_order=False)`` are not easily replicated in a single + statement, so their use when appropriate is acceptable. + + * ``m.assert_not_called()`` => ``assert 0 == m.call_count`` + +* When there are multiple patch calls in a test file for the module it + is testing, it may be desirable to capture the shared string prefix + for these patch calls in a module-level variable. If used, such + variables should be named ``M_PATH`` or, for datasource tests, + ``DS_PATH``. + +Test Argument Ordering +---------------------- + +* Test arguments should be ordered as follows: + + * ``mock.patch`` arguments. When used as a decorator, ``mock.patch`` + partially applies its generated ``Mock`` object as the first + argument, so these arguments must go first. + * ``pytest.mark.parametrize`` arguments, in the order specified to + the ``parametrize`` decorator. These arguments are also provided + by a decorator, so it's natural that they sit next to the + ``mock.patch`` arguments. + * Fixture arguments, alphabetically. These are not provided by a + decorator, so they are last, and their order has no defined + meaning, so we default to alphabetical. + +* It follows from this ordering of test arguments (so that we retain + the property that arguments left-to-right correspond to decorators + bottom-to-top) that test decorators should be ordered as follows: + + * ``pytest.mark.parametrize`` + * ``mock.patch`` + +.. [#fixture-list] This list of fixtures (with markup) can be + reproduced by running:: + + py.test-3 --fixtures -q | grep "^[^ -]" | grep -v '\(no\|capturelog\)' | sort | sed 's/.*/* ``\0``/' + + in a xenial lxd container with python3-pytest-catchlog installed. + +.. _pytest: https://docs.pytest.org/ +.. _pytest fixtures: https://docs.pytest.org/en/latest/fixture.html +.. _TestGetPackageMirrorInfo: https://github.com/canonical/cloud-init/blob/42f69f410ab8850c02b1f53dd67c132aa8ef64f5/cloudinit/distros/tests/test_init.py\#L15 +.. _TestPrependBaseCommands: https://github.com/canonical/cloud-init/blob/master/cloudinit/tests/test_subp.py#L9 +.. _assertion introspection: https://docs.pytest.org/en/latest/assert.html +.. _pytest 3.0: https://docs.pytest.org/en/latest/changelog.html#id1093 +.. _pytest.param: https://docs.pytest.org/en/latest/reference.html#pytest-param +.. _autospecced: https://docs.python.org/3.8/library/unittest.mock.html#autospeccing |