Single JSON schema validation in early boot (#1175)

Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.

Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.

This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.

The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.

Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:

- get_schema: source base schema file from cloud-init-schema.json
  and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
  no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
  on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param


Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
  if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI

1 files changed, 41 insertions, 2 deletions

diff --git a/tests/integration_tests/modules/test_cli.py b/tests/integration_tests/modules/test_cli.py
index 97bfe52d..81a5e7a8 100644
--- a/tests/integration_tests/modules/test_cli.py
+++ b/tests/integration_tests/modules/test_cli.py
@@ -13,11 +13,18 @@ runcmd:
   - echo 'hi' > /var/tmp/test
 """
 
-INVALID_USER_DATA = """\
+INVALID_USER_DATA_HEADER = """\
 runcmd:
   - echo 'hi' > /var/tmp/test
 """
 
+INVALID_USER_DATA_SCHEMA = """\
+#cloud-config
+updates:
+ notnetwork: -1
+apt_pipelining: bogus
+"""
+
 
 @pytest.mark.sru_2020_11
 @pytest.mark.user_data(VALID_USER_DATA)
@@ -29,10 +36,15 @@ def test_valid_userdata(client: IntegrationInstance):
     result = client.execute("cloud-init devel schema --system")
     assert result.ok
     assert "Valid cloud-config: system userdata" == result.stdout.strip()
+    result = client.execute("cloud-init status --long")
+    if not result.ok:
+        raise AssertionError(
+            f"Unexpected error from cloud-init status: {result}"
+        )
 
 
 @pytest.mark.sru_2020_11
-@pytest.mark.user_data(INVALID_USER_DATA)
+@pytest.mark.user_data(INVALID_USER_DATA_HEADER)
 def test_invalid_userdata(client: IntegrationInstance):
     """Test `cloud-init devel schema` with invalid userdata.
 
@@ -42,3 +54,30 @@ def test_invalid_userdata(client: IntegrationInstance):
     assert not result.ok
     assert "Cloud config schema errors" in result.stderr
     assert 'needs to begin with "#cloud-config"' in result.stderr
+    result = client.execute("cloud-init status --long")
+    if not result.ok:
+        raise AssertionError(
+            f"Unexpected error from cloud-init status: {result}"
+        )
+
+
+@pytest.mark.user_data(INVALID_USER_DATA_SCHEMA)
+def test_invalid_userdata_schema(client: IntegrationInstance):
+    """Test invalid schema represented as Warnings, not fatal
+
+    PR #1175
+    """
+    result = client.execute("cloud-init status --long")
+    assert result.ok
+    log = client.read_from_file("/var/log/cloud-init.log")
+    warning = (
+        "[WARNING]: Invalid cloud-config provided:\napt_pipelining: 'bogus'"
+        " is not valid under any of the given schemas\nupdates: Additional"
+        " properties are not allowed ('notnetwork' was unexpected)"
+    )
+    assert warning in log
+    result = client.execute("cloud-init status --long")
+    if not result.ok:
+        raise AssertionError(
+            f"Unexpected error from cloud-init status: {result}"
+        )