renderer: support unicode in render_from_file.

If a file passed to render_from_file had non-ascii text then
jinja in python2 would decode as ascii, which would cause
UnicodeDecodeError.  This issue can be re-created in python2
with just:
 'can\xe2\x80\x99t'.decode()

The solution here is to explicitly pass in unicode supporting
type (py3 str, py2 unicode).  Those are six.text_type.
Then jinja does not try to decode.

The reason we hit this is that load_file calls decode_binary.
decode_binary believes it has no work to do if it got a six.string_types.
  isinstance('can\xe2\x80\x99t', six.string_types) == True
So it returns the original string which will blow up for jinja.

Our fix here then is to load the file in binary mode and explicitly
decode it to utf-8.  Then in python2 we'll have a unicode type
and in python3 we'll have a string type.

1 files changed, 9 insertions, 1 deletions

diff --git a/cloudinit/templater.py b/cloudinit/templater.py
index b3ea64e4..9a087e1c 100644
--- a/cloudinit/templater.py
+++ b/cloudinit/templater.py
@@ -121,7 +121,11 @@ def detect_template(text):
 def render_from_file(fn, params):
     if not params:
         params = {}
-    template_type, renderer, content = detect_template(util.load_file(fn))
+    # jinja in python2 uses unicode internally.  All py2 str will be decoded.
+    # If it is given a str that has non-ascii then it will raise a
+    # UnicodeDecodeError.  So we explicitly convert to unicode type here.
+    template_type, renderer, content = detect_template(
+        util.load_file(fn, decode=False).decode('utf-8'))
     LOG.debug("Rendering content of '%s' using renderer %s", fn, template_type)
     return renderer(content, params)
 
@@ -132,11 +136,15 @@ def render_to_file(fn, outfn, params, mode=0o644):
 
 
 def render_string_to_file(content, outfn, params, mode=0o644):
+    """Render string (or py2 unicode) to file.
+    Warning: py2 str with non-ascii chars will cause UnicodeDecodeError."""
     contents = render_string(content, params)
     util.write_file(outfn, contents, mode=mode)
 
 
 def render_string(content, params):
+    """Render string (or py2 unicode).
+    Warning: py2 str with non-ascii chars will cause UnicodeDecodeError."""
     if not params:
         params = {}
     template_type, renderer, content = detect_template(content)