Single JSON schema validation in early boot (#1175)

Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.

Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.

This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.

The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.

Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:

- get_schema: source base schema file from cloud-init-schema.json
  and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
  no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
  on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param


Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
  if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI

1 files changed, 9 insertions, 6 deletions

diff --git a/tests/unittests/config/test_cc_write_files.py b/tests/unittests/config/test_cc_write_files.py
index 7eea99d3..faea5885 100644
--- a/tests/unittests/config/test_cc_write_files.py
+++ b/tests/unittests/config/test_cc_write_files.py
@@ -79,9 +79,11 @@ class TestWriteFilesSchema(CiTestCase):
         cc = self.tmp_cloud("ubuntu")
         valid_config = {"write_files": [{"path": "/some/path"}]}
         handle("cc_write_file", valid_config, cc, LOG, [])
-        self.assertNotIn("Invalid config:", self.logs.getvalue())
+        self.assertNotIn(
+            "Invalid cloud-config provided:", self.logs.getvalue()
+        )
         handle("cc_write_file", INVALID_SCHEMA, cc, LOG, [])
-        self.assertIn("Invalid config:", self.logs.getvalue())
+        self.assertIn("Invalid cloud-config provided:", self.logs.getvalue())
         self.assertIn("'path' is a required property", self.logs.getvalue())
 
     def test_schema_validation_warns_non_string_type_for_files(
@@ -105,7 +107,7 @@ class TestWriteFilesSchema(CiTestCase):
                 "write_files.0.%s: 1 is not of type '%s'" % (key, key_type),
                 self.logs.getvalue(),
             )
-        self.assertIn("Invalid config:", self.logs.getvalue())
+        self.assertIn("Invalid cloud-config provided:", self.logs.getvalue())
 
     def test_schema_validation_warns_on_additional_undefined_propertes(
         self, m_write_files
@@ -116,8 +118,8 @@ class TestWriteFilesSchema(CiTestCase):
         invalid_config["write_files"][0]["bogus"] = "value"
         handle("cc_write_file", invalid_config, cc, LOG, [])
         self.assertIn(
-            "Invalid config:\nwrite_files.0: Additional properties"
-            " are not allowed ('bogus' was unexpected)",
+            "Invalid cloud-config provided:\nwrite_files.0: Additional"
+            " properties are not allowed ('bogus' was unexpected)",
             self.logs.getvalue(),
         )
 
@@ -139,7 +141,8 @@ class TestWriteFiles(FilesystemMockingTestCase):
         with self.assertRaises(TypeError):
             handle("cc_write_file", invalid_config, cc, LOG, [])
         self.assertIn(
-            "Invalid config:\nwrite_files: 1 is not of type 'array'",
+            "Invalid cloud-config provided:\nwrite_files: 1 is not of type"
+            " 'array'",
             self.logs.getvalue(),
         )