Merge branch 'current' of github.com:vyos/vyos-1x into equuleus

* 'current' of github.com:vyos/vyos-1x:
  T1762: adjust the set_level() calls to use the new list representation.
  [vyos.config] T1764: support both string and list arguments in config functions.
  T1759: bug fixes, missing interface IP
  [vyos.config] T1758: use vyos.configtree for reading values, instead of calling cli-shell-api.
  [HTTP API] Add endpoints for config file and image management.
  ddclient: T1030: add cloudflare zone config entry
  [service https] T1443: organize internal data by server block
  [vyos.config] T1758: check that config setup has completed before calling showConfig, else, default to config.boot
  [HTTP API] Use a decorator for functions that require authentication.
  ddclient: T1030: adjust to latest syntax
  ddclient: T1030: auto create runtime directories
  ddclient: T1030: use new default configuration file path
  T1759: Migrating interfaces
  T1755: fixes issue with 'show vpn ipsec sa' command where lack of keysize (encr-keysize) will result in KeyError - such as for CHACHA20_POLY1305
  T1755: fixes issue with 'show vpn ipsec sa' command where lack of hash (integ-alg) will result in KeyError - such as with GCM based options

1 files changed, 125 insertions, 114 deletions

diff --git a/python/vyos/config.py b/python/vyos/config.py
index c9c73b971..3a340b2da 100644
--- a/python/vyos/config.py
+++ b/python/vyos/config.py
@@ -1,4 +1,4 @@
-# Copyright 2017 VyOS maintainers and contributors <maintainers@vyos.io>
+# Copyright 2017, 2019 VyOS maintainers and contributors <maintainers@vyos.io>
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -17,7 +17,7 @@
 A library for reading VyOS running config data.
 
 This library is used internally by all config scripts of VyOS,
-but its API should be considered stable and it is safe to use
+but its API should be considered stable and safe to use
 in user scripts.
 
 Note that this module will not work outside VyOS.
@@ -43,8 +43,8 @@ For example, under "system", the names of all valid child nodes are predefined
 
 To the contrary, children of the "system task-scheduler task" node can have arbitrary names.
 Such nodes are called *tag nodes*. This terminology is confusing but we keep using it for lack
-of a better word. The knowledge of whether in "task Foo" the "tag" is "task" or "Foo" is lost
-in time, luckily, the distinction is irrelevant in practice.
+of a better word. No one remembers if the "tag" in "task Foo" is "task" or "Foo",
+but the distinction is irrelevant in practice.
 
 Configuration modes
 ###################
@@ -53,21 +53,21 @@ VyOS has two distinct modes: operational mode and configuration mode. When a use
 the CLI is in the operational mode. In this mode, only the running (effective) config is accessible for reading.
 
 When a user enters the "configure" command, a configuration session is setup. Every config session
-has its *proposed* config built on top of the current running config. When changes are commited, if commit succeeds,
+has its *proposed* (or *session*) config built on top of the current running config. When changes are commited, if commit succeeds,
 the proposed config is merged into the running config.
 
-For this reason, this library has two sets of functions. The base versions, such as ``exists`` or ``return_value``
-are only usable in configuration mode. They take all nodes into account, in both proposed and running configs.
-Configuration scripts require access to uncommited changes for obvious reasons. Configuration mode completion helpers
-should also use these functions because not having nodes you've just created in completion is annoying.
+In configuration mode, "base" functions like `exists`, `return_value` return values from the session config,
+while functions prefixed "effective" return values from the running config.
+
+In operational mode, all functions return values from the running config.
 
-However, in operational mode, only the running config is available. Currently, you need to use special functions
-for reading it from operational mode scripts, they can be distinguished by the word "effective" in their names.
-In the future base versions may be made to detect if they are called from a config session or not.
 """
 
-import subprocess
+import os
 import re
+import subprocess
+
+import vyos.configtree
 
 
 class VyOSError(Exception):
@@ -89,17 +89,51 @@ class Config(object):
     """
     def __init__(self, session_env=None):
         self._cli_shell_api = "/bin/cli-shell-api"
-        self._level = ""
+        self._level = []
         if session_env:
             self.__session_env = session_env
         else:
             self.__session_env = None
 
+        # Running config can be obtained either from op or conf mode, it always succeeds
+        # (if config system is initialized at all).
+        if os.path.isfile('/tmp/vyos-config-status'):
+            running_config_text = self._run([self._cli_shell_api, '--show-active-only', '--show-show-defaults', 'showConfig'])
+        else:
+            with open('/opt/vyatta/etc/config/config.boot') as f:
+                running_config_text = f.read()
+
+        # Session config ("active") only exists in conf mode.
+        # Trying to obtain it from op mode will cause a fatal cli-shell-api error.
+        # If that happens, we assume that a script is running from op mode and use the running config
+        # for the "session config" variable as well.
+        try:
+            session_config_text = self._run([self._cli_shell_api, '--show-working-only', '--show-show-defaults', 'showConfig'])
+        except VyOSError:
+            session_config_text = running_config_text
+
+        self._session_config = vyos.configtree.ConfigTree(session_config_text)
+        self._running_config = vyos.configtree.ConfigTree(running_config_text)
+
     def _make_command(self, op, path):
         args = path.split()
         cmd = [self._cli_shell_api, op] + args
         return cmd
 
+    def _make_path(self, path):
+        # Backwards-compatibility stuff: original implementation used string paths
+        # libvyosconfig paths are lists, but since node names cannot contain whitespace,
+        # splitting at whitespace is reasonably safe.
+        # It may cause problems with exists() when it's used for checking values,
+        # since values may contain whitespace.
+        if isinstance(path, str):
+            path = re.split(r'\s*', path)
+        elif isinstance(path, list):
+            pass
+        else:
+            raise TypeError("Path must be a whitespace-separated string or a list")
+        return (self._level + path)
+
     def _run(self, cmd):
         if self.__session_env:
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=self.__session_env)
@@ -126,7 +160,12 @@ class Config(object):
         # Make sure there's always a space between default path (level)
         # and path supplied as method argument
         # XXX: for small strings in-place concatenation is not a problem
-        self._level = path + " "
+        if isinstance(path, str):
+            self._level = re.split(r'\s*', path)
+        elif isinstance(path, list):
+            self._level = path
+        else:
+            raise TypeError("Level path must be either a whitespace-separated string or a list")
 
     def get_level(self):
         """
@@ -135,7 +174,7 @@ class Config(object):
         Returns:
             str: current edit level
         """
-        return(self._level.strip())
+        return(self._level)
 
     def exists(self, path):
         """
@@ -148,11 +187,20 @@ class Config(object):
             This function cannot be used outside a configuration sessions.
             In operational mode scripts, use ``exists_effective``.
         """
-        try:
-            self._run(self._make_command('exists', self._level + path))
+        if self._session_config.exists(self._make_path(path)):
             return True
-        except VyOSError:
-            return False
+        else:
+            # libvyosconfig exists() works only for _nodes_, not _values_
+            # libvyattacfg one also worked for values, so we emulate that case here
+            path = re.split(r'\s*', path)
+            path_without_value = path[:-1]
+            path_str = " ".join(path_without_value)
+            try:
+                value = self._session_config.return_value(self._make_path(path_str))
+                return (value == path[-1])
+            except vyos.configtree.ConfigTreeError:
+                # node doesn't exist at all
+                return False
 
     def session_changed(self):
         """
@@ -203,7 +251,8 @@ class Config(object):
             It also returns False if node doesn't exist.
         """
         try:
-            self._run(self._make_command('isMulti', self._level + path))
+            path = " ".join(self._level) + " " + path
+            self._run(self._make_command('isMulti', path))
             return True
         except VyOSError:
             return False
@@ -220,7 +269,8 @@ class Config(object):
             It also returns False if node doesn't exist.
         """
         try:
-            self._run(self._make_command('isTag', self._level + path))
+            path = " ".join(self._level) + " " + path
+            self._run(self._make_command('isTag', path))
             return True
         except VyOSError:
             return False
@@ -237,7 +287,8 @@ class Config(object):
             It also returns False if node doesn't exist.
         """
         try:
-            self._run(self._make_command('isLeaf', self._level + path))
+            path = " ".join(self._level) + " " + path
+            self._run(self._make_command('isLeaf', path))
             return True
         except VyOSError:
             return False
@@ -254,9 +305,6 @@ class Config(object):
             str: Node value, if it has any
             None: if node is valueless *or* if it doesn't exist
 
-        Raises:
-            VyOSError: if node is not a single-value leaf node
-
         Note:
             Due to the issue with treatment of valueless nodes by this function,
             valueless nodes should be checked with ``exists`` instead.
@@ -264,17 +312,15 @@ class Config(object):
             This function cannot be used outside a configuration session.
             In operational mode scripts, use ``return_effective_value``.
         """
-        full_path = self._level + path
-        if self.is_multi(path):
-            raise VyOSError("Cannot use return_value on multi node: {0}".format(full_path))
-        elif not self.is_leaf(path):
-            raise VyOSError("Cannot use return_value on non-leaf node: {0}".format(full_path))
+        try:
+            value = self._session_config.return_value(self._make_path(path))
+        except vyos.configtree.ConfigTreeError:
+            value = None
+
+        if not value:
+            return(default)
         else:
-            try:
-                out = self._run(self._make_command('returnValue', full_path))
-                return out
-            except VyOSError:
-                return(default)
+            return(value)
 
     def return_values(self, path, default=[]):
         """
@@ -285,27 +331,21 @@ class Config(object):
 
         Returns:
             str list: Node values, if it has any
-            None: if node does not exist
-
-        Raises:
-            VyOSError: if node is not a multi-value leaf node
+            []: if node does not exist
 
         Note:
             This function cannot be used outside a configuration session.
             In operational mode scripts, use ``return_effective_values``.
         """
-        full_path = self._level + path
-        if not self.is_multi(path):
-            raise VyOSError("Cannot use return_values on non-multi node: {0}".format(full_path))
-        elif not self.is_leaf(path):
-            raise VyOSError("Cannot use return_values on non-leaf node: {0}".format(full_path))
+        try:
+            values = self._session_config.return_values(self._make_path(path))
+        except vyos.configtree.ConfigTreeError:
+            values = []
+
+        if not values:
+            return(default)
         else:
-            try:
-                out = self._run(self._make_command('returnValues', full_path))
-                values = re.findall(r"\'(.*?)\'", out)
-                return values
-            except VyOSError:
-                return(default)
+            return(values)
 
     def list_nodes(self, path, default=[]):
         """
@@ -317,26 +357,16 @@ class Config(object):
         Returns:
             string list: child node names
 
-        Raises:
-            VyOSError: if the node is not a tag node
-
-        Note:
-            There is no way to list all children of a non-tag node in
-            the current config backend.
-
-            This function cannot be used outside a configuration session.
-            In operational mode scripts, use ``list_effective_nodes``.
         """
-        full_path = self._level + path
-        if self.is_tag(path):
-            try:
-                out = self._run(self._make_command('listNodes', full_path))
-                values = re.findall(r"\'(.*?)\'", out)
-                return values
-            except VyOSError:
-                return(default)
+        try:
+            nodes = self._session_config.list_nodes(self._make_path(path))
+        except vyos.configtree.ConfigTreeError:
+            nodes = []
+
+        if not nodes:
+            return(default)
         else:
-            raise VyOSError("Cannot use list_nodes on a non-tag node: {0}".format(full_path))
+            return(nodes)
 
     def exists_effective(self, path):
         """
@@ -352,11 +382,7 @@ class Config(object):
             This function is safe to use in operational mode. In configuration mode,
             it ignores uncommited changes.
         """
-        try:
-            self._run(self._make_command('existsEffective', self._level + path))
-            return True
-        except VyOSError:
-            return False
+        return(self._running_config.exists(self._make_path(path)))
 
     def return_effective_value(self, path, default=None):
         """
@@ -368,21 +394,17 @@ class Config(object):
 
         Returns:
             str: Node value
-
-        Raises:
-            VyOSError: if node is not a multi-value leaf node
         """
-        full_path = self._level + path
-        if self.is_multi(path):
-            raise VyOSError("Cannot use return_effective_value on multi node: {0}".format(full_path))
-        elif not self.is_leaf(path):
-            raise VyOSError("Cannot use return_effective_value on non-leaf node: {0}".format(full_path))
+        try:
+            value = self._running_config.return_value(self._make_path(path))
+        except vyos.configtree.ConfigTreeError:
+            value = None
+
+        if not value:
+            return(default)
         else:
-            try:
-                out = self._run(self._make_command('returnEffectiveValue', full_path))
-                return out
-            except VyOSError:
-                return(default)
+            return(value)
+
 
     def return_effective_values(self, path, default=[]):
         """
@@ -393,22 +415,16 @@ class Config(object):
 
         Returns:
             str list: A list of values
-
-        Raises:
-            VyOSError: if node is not a multi-value leaf node
         """
-        full_path = self._level + path
-        if not self.is_multi(path):
-            raise VyOSError("Cannot use return_effective_values on non-multi node: {0}".format(full_path))
-        elif not self.is_leaf(path):
-            raise VyOSError("Cannot use return_effective_values on non-leaf node: {0}".format(full_path))
+        try:
+            values = self._running_config.return_values(self._make_path(path))
+        except vyos.configtree.ConfigTreeError:
+            values = []
+
+        if not values:
+            return(default)
         else:
-            try:
-                out = self._run(self._make_command('returnEffectiveValues', full_path))
-                values = re.findall(r"\'(.*?)\'", out)
-                return values
-            except VyOSError:
-                return(default)
+            return(values)
 
     def list_effective_nodes(self, path, default=[]):
         """
@@ -422,18 +438,13 @@ class Config(object):
 
         Raises:
             VyOSError: if the node is not a tag node
-
-        Note:
-            There is no way to list all children of a non-tag node in
-            the current config backend.
         """
-        full_path = self._level + path
-        if self.is_tag(path):
-            try:
-                out = self._run(self._make_command('listEffectiveNodes', full_path))
-                values = out.split()
-                return list(map(lambda x: re.sub(r'^\'(.*)\'$', r'\1',x), values))
-            except VyOSError:
-                return(default)
+        try:
+            nodes = self._running_config.list_nodes(self._make_path(path))
+        except vyos.configtree.ConfigTreeError:
+            nodes = []
+
+        if not nodes:
+            return(default)
         else:
-            raise VyOSError("Cannot use list_effective_nodes on a non-tag node: {0}".format(full_path))
+            return(nodes)