user-data: jinja template to render instance-data.json in cloud-config

Allow users to provide '## template: jinja' as the first line or their
#cloud-config or custom script user-data parts. When this header exists,
the cloud-config or script will be rendered as a jinja template.

All instance metadata keys and values present in
/run/cloud-init/instance-data.json will be available as jinja variables
for the template. This means any cloud-config module or script can
reference any standardized instance data in templates and scripts.

Additionally, any standardized instance-data.json keys scoped below a
'<v#>' key will be promoted as a top-level key for ease of reference in
templates. This means that '{{ local_hostname }}' is the same as using the
latest '{{ v#.local_hostname }}'.

Since instance-data is written to /run/cloud-init/instance-data.json, make
sure it is persisted across reboots when the cached datasource opject is
reloaded.

LP: #1791781

3 files changed, 76 insertions, 7 deletions

diff --git a/doc/rtd/topics/capabilities.rst b/doc/rtd/topics/capabilities.rst
index 3e2c9e31..2d8e2538 100644
--- a/doc/rtd/topics/capabilities.rst
+++ b/doc/rtd/topics/capabilities.rst
@@ -16,13 +16,15 @@ User configurability
 
 `Cloud-init`_ 's behavior can be configured via user-data.
 
-    User-data can be given by the user at instance launch time.
+    User-data can be given by the user at instance launch time. See
+    :ref:`user_data_formats` for acceptable user-data content.
+    
 
 This is done via the ``--user-data`` or ``--user-data-file`` argument to
 ec2-run-instances for example.
 
-* Check your local clients documentation for how to provide a `user-data`
-  string or `user-data` file for usage by cloud-init on instance creation.
+* Check your local client's documentation for how to provide a `user-data`
+  string or `user-data` file to cloud-init on instance creation.
 
 
 Feature detection
@@ -166,6 +168,13 @@ likely be promoted to top-level subcommands when stable.
    validation is work in progress and supports a subset of cloud-config
    modules.
 
+ * ``cloud-init devel render``: Use cloud-init's jinja template render to
+   process  **#cloud-config** or **custom-scripts**, injecting any variables
+   from ``/run/cloud-init/instance-data.json``. It accepts a user-data file
+   containing  the jinja template header ``## template: jinja`` and renders
+   that content with any instance-data.json variables present.
+
+
 .. _cli_clean:
 
 cloud-init clean
diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst
index 83034589..14432e65 100644
--- a/doc/rtd/topics/datasources.rst
+++ b/doc/rtd/topics/datasources.rst
@@ -18,6 +18,8 @@ single way to access the different cloud systems methods to provide this data
 through the typical usage of subclasses.
 
 
+.. _instance_metadata:
+
 instance-data
 -------------
 For reference, cloud-init stores all the metadata, vendordata and userdata
@@ -110,6 +112,51 @@ Below is an instance-data.json example from an OpenStack instance:
    }
   }
 
+ 
+As of cloud-init v. 18.4, any values present in
+``/run/cloud-init/instance-data.json`` can be used in cloud-init user data
+scripts or cloud config data. This allows consumers to use cloud-init's
+vendor-neutral, standardized metadata keys as well as datasource-specific
+content for any scripts or cloud-config modules they are using.
+
+To use instance-data.json values in scripts and **#config-config** files the
+user-data will need to contain the following header as the first line **## template: jinja**. Cloud-init will source all variables defined in
+``/run/cloud-init/instance-data.json`` and allow scripts or cloud-config files 
+to reference those paths. Below are two examples::
+
+ * Cloud config calling home with the ec2 public hostname and avaliability-zone
+    ```
+    ## template: jinja
+    #cloud-config
+    runcmd:
+        - echo 'EC2 public hostname allocated to instance: {{ ds.meta_data.public_hostname }}' > /tmp/instance_metadata
+        - echo 'EC2 avaiability zone: {{ v1.availability_zone }}' >> /tmp/instance_metadata 
+        - curl -X POST -d '{"hostname": "{{ds.meta_data.public_hostname }}", "availability-zone": "{{ v1.availability_zone }}"}'  https://example.com.com
+    ```
+
+ * Custom user script performing different operations based on region
+    ```
+    ## template: jinja
+    #!/bin/bash
+    {% if v1.region == 'us-east-2' -%}
+    echo 'Installing custom proxies for {{ v1.region }}
+    sudo apt-get install my-xtra-fast-stack
+    {%- endif %}
+    ...
+
+    ```
+
+.. note::
+  Trying to reference jinja variables that don't exist in
+  instance-data.json will result in warnings in ``/var/log/cloud-init.log``
+  and the following string in your rendered user-data:
+  ``CI_MISSING_JINJA_VAR/<your_varname>``.
+  
+.. note::
+  To save time designing your user-data for a specific cloud's
+  instance-data.json, use the 'render' cloud-init command on an
+  instance booted on your favorite cloud. See :ref:`cli_devel` for more
+  information.
 
 
 Datasource API
diff --git a/doc/rtd/topics/format.rst b/doc/rtd/topics/format.rst
index 1b0ff366..15234d21 100644
--- a/doc/rtd/topics/format.rst
+++ b/doc/rtd/topics/format.rst
@@ -1,6 +1,8 @@
-*******
-Formats
-*******
+.. _user_data_formats:
+
+*****************
+User-Data Formats
+*****************
 
 User data that will be acted upon by cloud-init must be in one of the following types.
 
@@ -65,6 +67,11 @@ Typically used by those who just want to execute a shell script.
 
 Begins with: ``#!`` or ``Content-Type: text/x-shellscript`` when using a MIME archive.
 
+.. note::
+   New in cloud-init v. 18.4: User-data scripts can also render cloud instance
+   metadata variables using jinja templating. See
+   :ref:`instance_metadata` for more information.
+
 Example
 -------
 
@@ -103,12 +110,18 @@ These things include:
 - certain ssh keys should be imported
 - *and many more...*
 
-**Note:** The file must be valid yaml syntax.
+.. note::
+   This file must be valid yaml syntax.
 
 See the :ref:`yaml_examples` section for a commented set of examples of supported cloud config formats.
 
 Begins with: ``#cloud-config`` or ``Content-Type: text/cloud-config`` when using a MIME archive.
 
+.. note::
+   New in cloud-init v. 18.4: Cloud config dta can also render cloud instance
+   metadata variables using jinja templating. See
+   :ref:`instance_metadata` for more information.
+
 Upstart Job
 ===========