Single JSON schema validation in early boot (#1175)

Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.

Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.

This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.

The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.

Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:

- get_schema: source base schema file from cloud-init-schema.json
  and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
  no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
  on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param


Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
  if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI

1 files changed, 12 insertions, 9 deletions

diff --git a/tests/unittests/config/test_cc_ntp.py b/tests/unittests/config/test_cc_ntp.py
index 7da82cee..fba141aa 100644
--- a/tests/unittests/config/test_cc_ntp.py
+++ b/tests/unittests/config/test_cc_ntp.py
@@ -385,7 +385,9 @@ class TestNtp(FilesystemMockingTestCase):
                         "servers []\npools {0}\n".format(pools),
                         util.load_file(confpath),
                     )
-            self.assertNotIn("Invalid config:", self.logs.getvalue())
+            self.assertNotIn(
+                "Invalid cloud-config provided:", self.logs.getvalue()
+            )
 
     @skipUnlessJsonSchema()
     @mock.patch("cloudinit.config.cc_ntp.select_ntp_client")
@@ -404,8 +406,8 @@ class TestNtp(FilesystemMockingTestCase):
             m_sel.return_value = ntpconfig
             cc_ntp.handle("cc_ntp", invalid_config, mycloud, None, [])
             self.assertIn(
-                "Invalid config:\nntp.pools.0: 123 is not of type 'string'\n"
-                "ntp.servers.1: None is not of type 'string'",
+                "Invalid cloud-config provided:\nntp.pools.0: 123 is not of"
+                " type 'string'\nntp.servers.1: None is not of type 'string'",
                 self.logs.getvalue(),
             )
             self.assertEqual(
@@ -431,8 +433,8 @@ class TestNtp(FilesystemMockingTestCase):
             m_select.return_value = ntpconfig
             cc_ntp.handle("cc_ntp", invalid_config, mycloud, None, [])
             self.assertIn(
-                "Invalid config:\nntp.pools: 123 is not of type 'array'\n"
-                "ntp.servers: 'non-array' is not of type 'array'",
+                "Invalid cloud-config provided:\nntp.pools: 123 is not of type"
+                " 'array'\nntp.servers: 'non-array' is not of type 'array'",
                 self.logs.getvalue(),
             )
             self.assertEqual(
@@ -459,8 +461,9 @@ class TestNtp(FilesystemMockingTestCase):
                 m_select.return_value = ntpconfig
                 cc_ntp.handle("cc_ntp", invalid_config, mycloud, None, [])
                 self.assertIn(
-                    "Invalid config:\nntp: Additional properties are not "
-                    "allowed ('invalidkey' was unexpected)",
+                    "Invalid cloud-config provided:\nntp: Additional"
+                    " properties are not allowed ('invalidkey' was"
+                    " unexpected)",
                     self.logs.getvalue(),
                 )
                 self.assertEqual(
@@ -488,8 +491,8 @@ class TestNtp(FilesystemMockingTestCase):
             m_select.return_value = ntpconfig
             cc_ntp.handle("cc_ntp", invalid_config, mycloud, None, [])
             self.assertIn(
-                "Invalid config:\nntp.pools: ['0.mypool.org', '0.mypool.org']"
-                " has non-unique elements\nntp.servers: "
+                "Invalid cloud-config provided:\nntp.pools: ['0.mypool.org',"
+                " '0.mypool.org'] has non-unique elements\nntp.servers: "
                 "['10.0.0.1', '10.0.0.1'] has non-unique elements",
                 self.logs.getvalue(),
             )