Integration test upgrades for the 21.3-1 SRU (#1001)

* Update test_combined.py to allow either valid LXD subplatform
* Split jinja templated tests into separate module as they can be more
  fragile
* Move checks for warnings and tracebacks into dedicated utility
  function. This allows us to work around persistent and expected
  tracebacks/warnings on particular clouds.
* Update test_upgrade.py to allow either valid Azure datasource.
  /var/lib/waagent or a mounted device are both valid.
* Add specificity to test_ntp_servers.py
  Clouds will often specify their own ntp servers in the ntp
  configuration files, so make the tests manually specify their own.
* Account for additional keys on system in test_ssh_keysfiles.py
* Update tests to account for invalid cache
  test_user_events.py and test_version_change.py both have tests that
  assume we will have valid ds cache when rebooting.
  In test_user_events.py, subsequent boots should block applying
  network on boot if boot event is denied. However, if the cache is
  invalid, it is valid to apply networking config that boot.
  In test_version_change.py no cache found won't trigger the expected
  debug log. Additionally, the pickle used for that test on an older
  release triggered an unexpected issue that took a different error
  path.
* Ignore bionic in hotplug tests (LP: #1942247)
  On Bionic, we traceback when attempting to detect the hotplugged
  device in the updated metadata. This is because Bionic is
  specifically configured not to provide network metadata.
  See LP: #1942247 for more details.
* Fix date used in test_final_message.
  In test_final_message, we ensured the variable substitution works as
  expected. For $timestamp, we compared against the current date. It's
  possible for the host date to be massively different from the client
  date, so obtain date on client rather than host.
* Remove module success from lp1813396 test. Module may fail
  unrelatedly (in this case apt-get update is failing), but the test
  should still pass.
* Skip testing events if network is disabled
* Ensure we install expected version of cloud-init
  As part of test setup, we can install cloud-init from various
  sources, including PROPOSED, PPAs, etc. We were never checking that
  this install completes successfully, and on OCI, it wasn't
  completing successfully because of apt locking issues. Code has
  been updated to retry, and then fail loudly if we can't complete the
  install.
* Remove ubuntu-azure-fips metapkg which mandates FIPS-flavour kernel
  In test_lp1835584.py
* Update test_user_events.py to account for Azure behavior
  since Azure has a separate service to clear the pickled metadata
  every boot
* Change failure to warning in test_upgrade.py if initial boot errors
  If there's already a pre-existing cause for warnings or tracebacks,
  that shouldn't cause the new version to fail.
* Add retry to test_random_passwords_emitted_to_serial_console
  It's possible we haven't retrieved the entire log when the call returns,
  so retry a few times if the output isn't empty.

1 files changed, 23 insertions, 17 deletions

diff --git a/tests/integration_tests/instances.py b/tests/integration_tests/instances.py
index 63e0e630..8f66bf43 100644
--- a/tests/integration_tests/instances.py
+++ b/tests/integration_tests/instances.py
@@ -9,6 +9,7 @@ from pycloudlib.instance import BaseInstance
 from pycloudlib.result import Result
 
 from tests.integration_tests import integration_settings
+from tests.integration_tests.util import retry
 
 try:
     from typing import TYPE_CHECKING
@@ -142,26 +143,31 @@ class IntegrationInstance:
             snapshot_id = self.snapshot()
             self.cloud.snapshot_id = snapshot_id
 
+    # assert with retry because we can compete with apt already running in the
+    # background and get: E: Could not get lock /var/lib/apt/lists/lock - open
+    # (11: Resource temporarily unavailable)
+
+    @retry(tries=30, delay=1)
     def install_proposed_image(self):
         log.info('Installing proposed image')
-        remote_script = (
+        assert self.execute(
             'echo deb "http://archive.ubuntu.com/ubuntu '
-            '$(lsb_release -sc)-proposed main" | '
-            'tee /etc/apt/sources.list.d/proposed.list\n'
-            'apt-get update -q\n'
-            'apt-get install -qy cloud-init'
-        )
-        self.execute(remote_script)
+            '$(lsb_release -sc)-proposed main" >> '
+            '/etc/apt/sources.list.d/proposed.list'
+        ).ok
+        assert self.execute('apt-get update -q').ok
+        assert self.execute('apt-get install -qy cloud-init').ok
 
+    @retry(tries=30, delay=1)
     def install_ppa(self):
         log.info('Installing PPA')
-        remote_script = (
-            'add-apt-repository {repo} -y && '
-            'apt-get update -q && '
-            'apt-get install -qy cloud-init'
-        ).format(repo=self.settings.CLOUD_INIT_SOURCE)
-        self.execute(remote_script)
+        assert self.execute('add-apt-repository {} -y'.format(
+            self.settings.CLOUD_INIT_SOURCE)
+        ).ok
+        assert self.execute('apt-get update -q').ok
+        assert self.execute('apt-get install -qy cloud-init').ok
 
+    @retry(tries=30, delay=1)
     def install_deb(self):
         log.info('Installing deb package')
         deb_path = integration_settings.CLOUD_INIT_SOURCE
@@ -170,13 +176,13 @@ class IntegrationInstance:
         self.push_file(
             local_path=integration_settings.CLOUD_INIT_SOURCE,
             remote_path=remote_path)
-        remote_script = 'dpkg -i {path}'.format(path=remote_path)
-        self.execute(remote_script)
+        assert self.execute('dpkg -i {path}'.format(path=remote_path)).ok
 
+    @retry(tries=30, delay=1)
     def upgrade_cloud_init(self):
         log.info('Upgrading cloud-init to latest version in archive')
-        self.execute("apt-get update -q")
-        self.execute("apt-get install -qy cloud-init")
+        assert self.execute("apt-get update -q").ok
+        assert self.execute("apt-get install -qy cloud-init").ok
 
     def __enter__(self):
         return self