Improve module documentation and doc cleanup.

This adds lots of config module documentation in a standard format.
It will greatly improve the content at readthedocs.
Additionally:
 * Add a 'doc' env to tox.ini
 * Changed default highlight language for sphinx conf from python to yaml
   most examples in documentation are yaml configs
 * Updated datasource examples to highlight sh code properly

1 files changed, 207 insertions, 0 deletions

diff --git a/cloudinit/config/cc_apt_configure.py b/cloudinit/config/cc_apt_configure.py
index fa9505a7..6145fcd2 100644
--- a/cloudinit/config/cc_apt_configure.py
+++ b/cloudinit/config/cc_apt_configure.py
@@ -18,6 +18,213 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+"""
+Apt Configure
+-------------
+**Summary:** configure apt
+
+This module handles both configuration of apt options and adding source lists.
+There are configuration options such as ``apt_get_wrapper`` and
+``apt_get_command`` that control how cloud-init invokes apt-get.
+These configuration options are handled on a per-distro basis, so consult
+documentation for cloud-init's distro support for instructions on using
+these config options.
+
+.. note::
+    To ensure that apt configuration is valid yaml, any strings containing
+    special characters, especially ``:`` should be quoted.
+
+.. note::
+    For more information about apt configuration, see the
+    ``Additional apt configuration`` example.
+
+**Preserve sources.list:**
+
+By default, cloud-init will generate a new sources list in
+``/etc/apt/sources.list.d`` based on any changes specified in cloud config.
+To disable this behavior and preserve the sources list from the pristine image,
+set ``preserve_sources_list`` to ``true``.
+
+.. note::
+    The ``preserve_sources_list`` option overrides all other config keys that
+    would alter ``sources.list`` or ``sources.list.d``, **except** for
+    additional sources to be added to ``sources.list.d``.
+
+**Disable source suites:**
+
+Entries in the sources list can be disabled using ``disable_suites``, which
+takes a list of suites to be disabled. If the string ``$RELEASE`` is present in
+a suite in the ``disable_suites`` list, it will be replaced with the release
+name. If a suite specified in ``disable_suites`` is not present in
+``sources.list`` it will be ignored. For convenience, several aliases are
+provided for ``disable_suites``:
+
+    - ``updates`` => ``$RELEASE-updates``
+    - ``backports`` => ``$RELEASE-backports``
+    - ``security`` => ``$RELEASE-security``
+    - ``proposed`` => ``$RELEASE-proposed``
+    - ``release`` => ``$RELEASE``
+
+.. note::
+    When a suite is disabled using ``disable_suites``, its entry in
+    ``sources.list`` is not deleted; it is just commented out.
+
+**Configure primary and security mirrors:**
+
+The primary and security archive mirrors can be specified using the ``primary``
+and ``security`` keys, respectively. Both the ``primary`` and ``security`` keys
+take a list of configs, allowing mirrors to be specified on a per-architecture
+basis. Each config is a dictionary which must have an entry for ``arches``,
+specifying which architectures that config entry is for. The keyword
+``default`` applies to any architecture not explicitly listed. The mirror url
+can be specified with the ``url`` key, or a list of mirrors to check can be
+provided in order, with the first mirror that can be resolved being selected.
+This allows the same configuration to be used in different environment, with
+different hosts used for a local apt mirror. If no mirror is provided by uri or
+search, ``search_dns`` may be used to search for dns names in the format
+``<distro>-mirror`` in each of the following:
+
+    - fqdn of this host per cloud metadata
+    - localdomain
+    - domains listed in ``/etc/resolv.conf``
+
+If there is a dns entry for ``<distro>-mirror``, then it is assumed that there
+is a distro mirror at ``http://<distro>-mirror.<domain>/<distro>``. If the
+``primary`` key is defined, but not the ``security`` key, then then
+configuration for ``primary`` is also used for ``security``. If ``search_dns``
+is used for the ``security`` key, the search pattern will be.
+``<distro>-security-mirror``.
+
+If no mirrors are specified, or all lookups fail, then default mirrors defined
+in the datasource are used. If none are present in the datasource either the
+following defaults are used:
+
+    - primary: ``http://archive.ubuntu.com/ubuntu``
+    - security: ``http://security.ubuntu.com/ubuntu``
+
+**Specify sources.list template:**
+
+A custom template for rendering ``sources.list`` can be specefied with
+``sources_list``. If no ``sources_list`` template is given, cloud-init will
+use sane default. Within this template, the following strings will be replaced
+with the appropriate values:
+
+    - ``$MIRROR``
+    - ``$RELEASE``
+    - ``$PRIMARY``
+    - ``$SECURITY``
+
+**Pass configuration to apt:**
+
+Apt configuration can be specified using ``conf``. Configuration is specified
+as a string. For multiline apt configuration, make sure to follow yaml syntax.
+
+**Configure apt proxy:**
+
+Proxy configuration for apt can be specified using ``conf``, but proxy config
+keys also exist for convenience. The proxy config keys, ``http_proxy``,
+``ftp_proxy``, and ``https_proxy`` may be used to specify a proxy for http, ftp
+and https protocols respectively. The ``proxy`` key also exists as an alias for
+``http_proxy``. Proxy url is specified in the format
+``<protocol>://[[user][:pass]@]host[:port]/``.
+
+**Add apt repos by regex:**
+
+All source entries in ``apt-sources`` that match regex in
+``add_apt_repo_match`` will be added to the system using
+``add-apt-repository``. If ``add_apt_repo_match`` is not specified, it defaults
+to ``^[\w-]+:\w``
+
+**Add source list entries:**
+
+Source list entries can be specified as a dictionary under the ``sources``
+config key, with key in the dict representing a different source file. The key
+The key of each source entry will be used as an id that can be referenced in
+other config entries, as well as the filename for the source's configuration
+under ``/etc/apt/sources.list.d``. If the name does not end with ``.list``,
+it will be appended. If there is no configuration for a key in ``sources``, no
+file will be written, but the key may still be referred to as an id in other
+``sources`` entries.
+
+Each entry under ``sources`` is a dictionary which may contain any of the
+following optional keys:
+
+    - ``source``: a sources.list entry (some variable replacements apply)
+    - ``keyid``: a key to import via shortid or fingerprint
+    - ``key``: a raw PGP key
+    - ``keyserver``: alternate keyserver to pull ``keyid`` key from
+
+The ``source`` key supports variable replacements for the following strings:
+
+    - ``$MIRROR``
+    - ``$PRIMARY``
+    - ``$SECURITY``
+    - ``$RELEASE``
+
+**Internal name:** ``cc_apt_configure``
+
+**Module frequency:** per instance
+
+**Supported distros:** ubuntu, debian
+
+**Config keys**::
+
+    apt:
+        preserve_sources_list: <true/false>
+        disable_suites:
+            - $RELEASE-updates
+            - backports
+            - $RELEASE
+            - mysuite
+        primary:
+            - arches:
+                - amd64
+                - i386
+                - default
+              uri: "http://us.archive.ubuntu.com/ubuntu"
+              search:
+                - "http://cool.but-sometimes-unreachable.com/ubuntu"
+                - "http://us.archive.ubuntu.com/ubuntu"
+              search_dns: <true/false>
+            - arches:
+                - s390x
+                - arm64
+              uri: "http://archive-to-use-for-arm64.example.com/ubuntu"
+        security:
+            - arches:
+                - default
+              search_dns: true
+        sources_list: |
+            deb $MIRROR $RELEASE main restricted
+            deb-src $MIRROR $RELEASE main restricted
+            deb $PRIMARY $RELEASE universe restricted
+            deb $SECURITY $RELEASE-security multiverse
+        conf: |
+            APT {
+                Get {
+                    Assume-Yes "true";
+                    Fix-Broken "true";
+                }
+            }
+        proxy: "http://[[user][:pass]@]host[:port]/"
+        http_proxy: "http://[[user][:pass]@]host[:port]/"
+        ftp_proxy: "ftp://[[user][:pass]@]host[:port]/"
+        https_proxy: "https://[[user][:pass]@]host[:port]/"
+        sources:
+            source1:
+                keyid: "keyid"
+                keyserver: "keyserverurl"
+                source: "deb http://<url>/ xenial main"
+            source2:
+                source "ppa:<ppa-name>"
+            source3:
+                source "deb $MIRROR $RELEASE multiverse"
+                key: |
+                    ------BEGIN PGP PUBLIC KEY BLOCK-------
+                    <key data>
+                    ------END PGP PUBLIC KEY BLOCK-------
+"""
+
 import glob
 import os
 import re