Single JSON schema validation in early boot (#1175)

Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.

Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.

This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.

The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.

Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:

- get_schema: source base schema file from cloud-init-schema.json
  and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
  no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
  on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param


Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
  if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI

1 files changed, 34 insertions, 29 deletions

diff --git a/cloudinit/config/cc_apt_pipelining.py b/cloudinit/config/cc_apt_pipelining.py
index 569849d1..34b6ac0e 100644
--- a/cloudinit/config/cc_apt_pipelining.py
+++ b/cloudinit/config/cc_apt_pipelining.py
@@ -4,54 +4,59 @@
 #
 # This file is part of cloud-init. See LICENSE file for license information.
 
-"""
-Apt Pipelining
---------------
-**Summary:** configure apt pipelining
+"""Apt Pipelining: configure apt pipelining."""
 
-This module configures apt's ``Acquite::http::Pipeline-Depth`` option, which
-controls how apt handles HTTP pipelining. It may be useful for pipelining to be
-disabled, because some web servers, such as S3 do not pipeline properly (LP:
-#948461). The ``apt_pipelining`` config key may be set to ``false`` to disable
-pipelining altogether. This is the default behavior. If it is set to ``none``,
-``unchanged``, or ``os``, no change will be made to apt configuration and the
-default setting for the distro will be used. The pipeline depth can also be
-manually specified by setting ``apt_pipelining`` to a number. However, this is
-not recommended.
-
-**Internal name:** ``cc_apt_pipelining``
-
-**Module frequency:** per instance
-
-**Supported distros:** ubuntu, debian
-
-**Config keys**::
-    apt_pipelining: <false/none/unchanged/os/number>
-"""
+from textwrap import dedent
 
 from cloudinit import util
+from cloudinit.config.schema import get_meta_doc
 from cloudinit.settings import PER_INSTANCE
 
 frequency = PER_INSTANCE
-
 distros = ["ubuntu", "debian"]
-
 DEFAULT_FILE = "/etc/apt/apt.conf.d/90cloud-init-pipelining"
-
 APT_PIPE_TPL = (
     "//Written by cloud-init per 'apt_pipelining'\n"
     'Acquire::http::Pipeline-Depth "%s";\n'
 )
-
 # Acquire::http::Pipeline-Depth can be a value
 # from 0 to 5 indicating how many outstanding requests APT should send.
 # A value of zero MUST be specified if the remote host does not properly linger
 # on TCP connections - otherwise data corruption will occur.
 
+meta = {
+    "id": "cc_apt_pipelining",
+    "name": "Apt Pipelining",
+    "title": "Configure apt pipelining",
+    "description": dedent(
+        """\
+        This module configures apt's ``Acquite::http::Pipeline-Depth`` option,
+        which controls how apt handles HTTP pipelining. It may be useful for
+        pipelining to be disabled, because some web servers, such as S3 do not
+        pipeline properly (LP: #948461).
+
+        Value configuration options for this module are:
+
+        * ``false`` (Default): disable pipelining altogether
+        * ``none``, ``unchanged``, or ``os``: use distro default
+        * ``<number>``: Manually specify pipeline depth. This is not recommended."""  # noqa: E501
+    ),
+    "distros": distros,
+    "frequency": frequency,
+    "examples": [
+        "apt_pipelining: false",
+        "apt_pipelining: none",
+        "apt_pipelining: unchanged",
+        "apt_pipelining: os",
+        "apt_pipelining: 3",
+    ],
+}
+
+__doc__ = get_meta_doc(meta)
 
-def handle(_name, cfg, _cloud, log, _args):
 
-    apt_pipe_value = util.get_cfg_option_str(cfg, "apt_pipelining", "os")
+def handle(_name, cfg, _cloud, log, _args):
+    apt_pipe_value = cfg.get("apt_pipelining", "os")
     apt_pipe_value_s = str(apt_pipe_value).lower().strip()
 
     if apt_pipe_value_s == "false":