Integration test upgrades for the 21.3-1 SRU (#1001)

* Update test_combined.py to allow either valid LXD subplatform
* Split jinja templated tests into separate module as they can be more
  fragile
* Move checks for warnings and tracebacks into dedicated utility
  function. This allows us to work around persistent and expected
  tracebacks/warnings on particular clouds.
* Update test_upgrade.py to allow either valid Azure datasource.
  /var/lib/waagent or a mounted device are both valid.
* Add specificity to test_ntp_servers.py
  Clouds will often specify their own ntp servers in the ntp
  configuration files, so make the tests manually specify their own.
* Account for additional keys on system in test_ssh_keysfiles.py
* Update tests to account for invalid cache
  test_user_events.py and test_version_change.py both have tests that
  assume we will have valid ds cache when rebooting.
  In test_user_events.py, subsequent boots should block applying
  network on boot if boot event is denied. However, if the cache is
  invalid, it is valid to apply networking config that boot.
  In test_version_change.py no cache found won't trigger the expected
  debug log. Additionally, the pickle used for that test on an older
  release triggered an unexpected issue that took a different error
  path.
* Ignore bionic in hotplug tests (LP: #1942247)
  On Bionic, we traceback when attempting to detect the hotplugged
  device in the updated metadata. This is because Bionic is
  specifically configured not to provide network metadata.
  See LP: #1942247 for more details.
* Fix date used in test_final_message.
  In test_final_message, we ensured the variable substitution works as
  expected. For $timestamp, we compared against the current date. It's
  possible for the host date to be massively different from the client
  date, so obtain date on client rather than host.
* Remove module success from lp1813396 test. Module may fail
  unrelatedly (in this case apt-get update is failing), but the test
  should still pass.
* Skip testing events if network is disabled
* Ensure we install expected version of cloud-init
  As part of test setup, we can install cloud-init from various
  sources, including PROPOSED, PPAs, etc. We were never checking that
  this install completes successfully, and on OCI, it wasn't
  completing successfully because of apt locking issues. Code has
  been updated to retry, and then fail loudly if we can't complete the
  install.
* Remove ubuntu-azure-fips metapkg which mandates FIPS-flavour kernel
  In test_lp1835584.py
* Update test_user_events.py to account for Azure behavior
  since Azure has a separate service to clear the pickled metadata
  every boot
* Change failure to warning in test_upgrade.py if initial boot errors
  If there's already a pre-existing cause for warnings or tracebacks,
  that shouldn't cause the new version to fail.
* Add retry to test_random_passwords_emitted_to_serial_console
  It's possible we haven't retrieved the entire log when the call returns,
  so retry a few times if the output isn't empty.

1 files changed, 19 insertions, 6 deletions

diff --git a/tests/integration_tests/test_upgrade.py b/tests/integration_tests/test_upgrade.py
index 376fcc96..e90a5f9d 100644
--- a/tests/integration_tests/test_upgrade.py
+++ b/tests/integration_tests/test_upgrade.py
@@ -5,6 +5,7 @@ import pytest
 
 from tests.integration_tests.clouds import ImageSpecification, IntegrationCloud
 from tests.integration_tests.conftest import get_validated_source
+from tests.integration_tests.util import verify_clean_log
 
 
 LOG = logging.getLogger('integration_testing.test_upgrade')
@@ -73,11 +74,15 @@ def test_clean_boot_of_upgraded_package(session_cloud: IntegrationCloud):
         pre_cloud_blame = instance.execute('cloud-init analyze blame')
 
         # Ensure no issues pre-upgrade
+        log = instance.read_from_file('/var/log/cloud-init.log')
         assert not json.loads(pre_result)['v1']['errors']
 
-        log = instance.read_from_file('/var/log/cloud-init.log')
-        assert 'Traceback' not in log
-        assert 'WARN' not in log
+        try:
+            verify_clean_log(log)
+        except AssertionError:
+            LOG.warning(
+                'There were errors/warnings/tracebacks pre-upgrade. '
+                'Any failures may be due to pre-upgrade problem')
 
         # Upgrade and reboot
         instance.install_new_cloud_init(source, take_snapshot=False)
@@ -105,13 +110,21 @@ def test_clean_boot_of_upgraded_package(session_cloud: IntegrationCloud):
         assert not json.loads(pre_result)['v1']['errors']
 
         log = instance.read_from_file('/var/log/cloud-init.log')
-        assert 'Traceback' not in log
-        assert 'WARN' not in log
+        verify_clean_log(log)
 
         # Ensure important things stayed the same
         assert pre_hostname == post_hostname
         assert pre_cloud_id == post_cloud_id
-        assert pre_result == post_result
+        try:
+            assert pre_result == post_result
+        except AssertionError:
+            if instance.settings.PLATFORM == 'azure':
+                pre_json = json.loads(pre_result)
+                post_json = json.loads(post_result)
+                assert pre_json['v1']['datasource'].startswith(
+                    'DataSourceAzure')
+                assert post_json['v1']['datasource'].startswith(
+                    'DataSourceAzure')
         assert pre_network == post_network
 
         # Calculate and log all the boot numbers