Integration test upgrades for the 21.3-1 SRU (#1001)

* Update test_combined.py to allow either valid LXD subplatform
* Split jinja templated tests into separate module as they can be more
  fragile
* Move checks for warnings and tracebacks into dedicated utility
  function. This allows us to work around persistent and expected
  tracebacks/warnings on particular clouds.
* Update test_upgrade.py to allow either valid Azure datasource.
  /var/lib/waagent or a mounted device are both valid.
* Add specificity to test_ntp_servers.py
  Clouds will often specify their own ntp servers in the ntp
  configuration files, so make the tests manually specify their own.
* Account for additional keys on system in test_ssh_keysfiles.py
* Update tests to account for invalid cache
  test_user_events.py and test_version_change.py both have tests that
  assume we will have valid ds cache when rebooting.
  In test_user_events.py, subsequent boots should block applying
  network on boot if boot event is denied. However, if the cache is
  invalid, it is valid to apply networking config that boot.
  In test_version_change.py no cache found won't trigger the expected
  debug log. Additionally, the pickle used for that test on an older
  release triggered an unexpected issue that took a different error
  path.
* Ignore bionic in hotplug tests (LP: #1942247)
  On Bionic, we traceback when attempting to detect the hotplugged
  device in the updated metadata. This is because Bionic is
  specifically configured not to provide network metadata.
  See LP: #1942247 for more details.
* Fix date used in test_final_message.
  In test_final_message, we ensured the variable substitution works as
  expected. For $timestamp, we compared against the current date. It's
  possible for the host date to be massively different from the client
  date, so obtain date on client rather than host.
* Remove module success from lp1813396 test. Module may fail
  unrelatedly (in this case apt-get update is failing), but the test
  should still pass.
* Skip testing events if network is disabled
* Ensure we install expected version of cloud-init
  As part of test setup, we can install cloud-init from various
  sources, including PROPOSED, PPAs, etc. We were never checking that
  this install completes successfully, and on OCI, it wasn't
  completing successfully because of apt locking issues. Code has
  been updated to retry, and then fail loudly if we can't complete the
  install.
* Remove ubuntu-azure-fips metapkg which mandates FIPS-flavour kernel
  In test_lp1835584.py
* Update test_user_events.py to account for Azure behavior
  since Azure has a separate service to clear the pickled metadata
  every boot
* Change failure to warning in test_upgrade.py if initial boot errors
  If there's already a pre-existing cause for warnings or tracebacks,
  that shouldn't cause the new version to fail.
* Add retry to test_random_passwords_emitted_to_serial_console
  It's possible we haven't retrieved the entire log when the call returns,
  so retry a few times if the output isn't empty.

1 files changed, 34 insertions, 0 deletions

diff --git a/tests/integration_tests/util.py b/tests/integration_tests/util.py
index 80430eab..407096cd 100644
--- a/tests/integration_tests/util.py
+++ b/tests/integration_tests/util.py
@@ -28,6 +28,40 @@ def verify_ordered_items_in_text(to_verify: list, text: str):
         assert index > -1, "Expected item not found: '{}'".format(item)
 
 
+def verify_clean_log(log):
+    """Assert no unexpected tracebacks or warnings in logs"""
+    warning_count = log.count('WARN')
+    expected_warnings = 0
+    traceback_count = log.count('Traceback')
+    expected_tracebacks = 0
+
+    warning_texts = [
+        # Consistently on all Azure launches:
+        # azure.py[WARNING]: No lease found; using default endpoint
+        'No lease found; using default endpoint'
+    ]
+    traceback_texts = []
+    if 'oracle' in log:
+        # LP: #1842752
+        lease_exists_text = 'Stderr: RTNETLINK answers: File exists'
+        warning_texts.append(lease_exists_text)
+        traceback_texts.append(lease_exists_text)
+        # LP: #1833446
+        fetch_error_text = (
+            'UrlError: 404 Client Error: Not Found for url: '
+            'http://169.254.169.254/latest/meta-data/')
+        warning_texts.append(fetch_error_text)
+        traceback_texts.append(fetch_error_text)
+
+    for warning_text in warning_texts:
+        expected_warnings += log.count(warning_text)
+    for traceback_text in traceback_texts:
+        expected_tracebacks += log.count(traceback_text)
+
+    assert warning_count == expected_warnings
+    assert traceback_count == expected_tracebacks
+
+
 @contextmanager
 def emit_dots_on_travis():
     """emit a dot every 60 seconds if running on Travis.