HTTP-API: rewrite and add config multiple commands

3 files changed, 364 insertions, 140 deletions

diff --git a/docs/automation/index.rst b/docs/automation/index.rst
index e07dfecc..c19d819b 100644
--- a/docs/automation/index.rst
+++ b/docs/automation/index.rst
@@ -5,11 +5,11 @@ VyOS Automation
 
   * Ansible
   * Saltstack
-  * HTTP-API
   * startup scripts
 
 
 .. toctree::
    :maxdepth: 1
-
+   
+   vyos-api
    command-scripting
\ No newline at end of file
diff --git a/docs/automation/vyos-api.rst b/docs/automation/vyos-api.rst
new file mode 100644
index 00000000..c34f4f52
--- /dev/null
+++ b/docs/automation/vyos-api.rst
@@ -0,0 +1,315 @@
+.. _vyosapi:
+
+########
+VyOS API
+########
+
+for configuration and enabling the API see :ref:`httpapi`
+
+**************
+Authentication
+**************
+
+All Endpoint only listen on HTTP POST requests and the API KEY must set as
+``key`` in the formdata.
+
+Below see one example or curl and one for python.
+In the following, the documentation is reduced to curl.
+
+.. code-block:: none
+
+   curl --location --request POST 'https://vyos/retrieve' \
+   --form data='{"op": "showConfig", "path": []}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+.. code-block:: python
+
+   import requests
+   url = "https://vyos/retrieve"
+   payload={'data': '{"op": "showConfig", "path": []}',
+            'key': 'MY-HTTPS-API-PLAINTEXT-KEY'
+           }
+   headers = {}
+   response = requests.request("POST", url, headers=headers, data=payload)
+   print(response.text)
+
+
+*************
+API Endpoints
+*************
+
+/retrieve
+=========
+
+With the ``retrieve`` endpoint you get parts or the whole configuration.
+
+To get the whole configuration, pass an empty list to the ``path`` field 
+
+.. code-block:: none
+
+   curl --location --request POST 'https://vyos/retrieve' \
+   --form data='{"op": "showConfig", "path": []}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+
+   response (shorted)
+   {
+      "success": true,
+      "data": {
+         "interfaces": {
+               "ethernet": {
+                  "eth0": {
+                     "address": "dhcp",
+                     "duplex": "auto",
+                     "hw-id": "50:00:00:01:00:00",
+                     "speed": "auto"
+                  },
+                  "eth1": {
+                     "duplex": "auto",
+                     "hw-id": "50:00:00:01:00:01",
+                     "speed": "auto"
+      ...
+      },
+      "error": null
+   }
+
+
+only get a part of the configuration, 
+for example ``system syslog``.
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/retrieve' \
+   --form data='{"op": "showConfig", "path": ["system", "syslog"]}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+
+   response:
+   {
+      "success": true,
+      "data": {
+         "global": {
+               "facility": {
+                  "all": {
+                     "level": "info"
+                  },
+                  "protocols": {
+                     "level": "debug"
+                  }
+               }
+         }
+      },
+      "error": null
+   }
+
+if you just want the Value of a multi-valued node, use the ``returnValues``
+operation.
+
+for example get the addresses of a ``dum0`` interface
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/retrieve' \
+   --form data='{"op": "returnValues", "path": ["interfaces","dummy","dum0","address"]}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   respone:
+   {
+      "success": true,
+      "data": [
+         "10.10.10.10/24",
+         "10.10.10.11/24",
+         "10.10.10.12/24"
+      ],
+      "error": null
+   }
+
+/image
+======
+
+To add or delete an image, use the ``/image`` endpoint.
+
+add an image
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/image' \
+   --form data='{"op": "add", "url": "https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso"}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   respone (shorted):
+   {
+      "success": true,
+      "data": "Trying to fetch ISO file from https://downloads.vyos.io/rolling-latest.iso\n
+               ...
+               Setting up grub configuration...\nDone.\n",
+      "error": null
+   }
+
+delete an image, for example ``1.3-rolling-202006070117``
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/image' \
+   --form data='{"op": "delete", "name": "1.3-rolling-202006070117"}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": "Deleting the \"1.3-rolling-202006070117\" image...\nDone\n",
+      "error": null
+   }
+
+
+/show
+=====
+
+The ``/show`` endpoint is to show everthing in operational mode
+
+for example which images are installed
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/show' \
+   --form data='{"op": "show", "path": ["system", "image"]}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": "The system currently has the following image(s) installed:\n\n
+                1: 1.4-rolling-202102280559 (default boot)\n
+                2: 1.4-rolling-202102230218\n
+                3: 1.3-beta-202102210443\n\n",
+      "error": null
+   }
+
+
+/generate
+=========
+
+to run a ``generate`` command use the
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/generate' \
+   --form data='{"op": "generate", "path": ["wireguard", "default-keypair"]}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": "",
+      "error": null
+   }
+
+
+/configure
+==========
+
+You can pass a ``set``, ``delete`` or ``comment`` command to the
+``/configure`` endpoint.
+
+``set`` a single command
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/configure' \
+   --form data='{"op": "set", "path": ["interfaces", "dummy", "dum1", "address", "10.11.0.1/32"]}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": null,
+      "error": null
+   }
+
+
+``delete`` a single command
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/configure' \
+   --form data='{"op": "delete", "path": ["interfaces", "dummy", "dum1", "address", "10.11.0.1/32"]}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": null,
+      "error": null
+   }
+
+The API will push the command to a session and commit after each HTTP reguest.
+The Endpoint will process multiple commands when you pass them as a list to
+the ``data`` field.
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/configure' \
+   --form data='[{"op": "delete", "path": ["interfaces", "dummy", "dum1", "address"]},{"op": "set", "path": ["interfaces", "dummy", "dum1", "address", "10.11.0.1/32"]}]' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": null,
+      "error": null
+   }
+
+
+/config-file
+============
+
+The endpoint ``/config-file`` is to save or load a configuration.
+
+Save a running configuration to the startup configuration.
+When you don't specify the file when saving, it saves to
+``/config/config.boot``.
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/config-file' \
+   --form data='{"op": "save"}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": "Saving configuration to '/config/config.boot'...\nDone\n",
+      "error": null
+   }
+
+
+Save a running configuration to a file.
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/config-file' \
+   --form data='{"op": "save", "file": "/config/test.config"}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": "Saving configuration to '/config/test.config'...\nDone\n",
+      "error": null
+   }
+
+
+To Load a configuration file.
+
+.. code-block:: none
+
+   curl -k --location --request POST 'https://vyos/config-file' \
+   --form data='{"op": "load", "file": "/config/test.config"}' \
+   --form key='MY-HTTPS-API-PLAINTEXT-KEY'
+
+   response:
+   {
+      "success": true,
+      "data": null,
+      "error": null
+   }
\ No newline at end of file
diff --git a/docs/configuration/service/https.rst b/docs/configuration/service/https.rst
index b9c691da..3cd4d5ab 100644
--- a/docs/configuration/service/https.rst
+++ b/docs/configuration/service/https.rst
@@ -4,178 +4,87 @@
 HTTP-API
 ########
 
-Enabling HTTP-API
------------------
+VyOS provide a HTTP API. You can use it to execute op-mode commands,
+update VyOS, set or delete config.
 
-VyOS HTTP API can be enabled through the ``set service https api`` command.
-
-.. code-block:: none
-
-  set service https api debug
-  set service https api keys id MY-HTTP-API-ID key MY-HTTP-API-PLAINTEXT-KEY
-
-The local API process listens on localhost:8080, and nginx exposes it on all
-virtual servers, by default. For the purpose of illustration below, we will
-assume nginx is running at https://192.168.122.127.
-
-One can limit proxying to specific listen addresses/ports/server-names by
-defining a ``service https virtual-host <id>``, and setting ``service https
-api-restrict virtual-host <id>``.
-
-.. code-block:: none
-
-  set service https virtual-host example listen-address 192.168.122.127
-  set service https virtual-host example listen-port 44302
-  set service https virtual-host example server-name example.net
-
-  set service https api-restrict virtual-host example
-
-In this example, nginx will proxy only those requests to
-192.168.122.127:44302 or example.net:44302 (assuming the DNS record is
-viable). Omitting any of listen-address, listen-port, or server-name, will
-leave appropriate defaults in the nginx directive. Multiple instances of
-``service https api-restrict virtual-host`` may be set.
-
-Configuration mode requests
----------------------------
-
-In our example, we are creating a dummy interface and assigning an address to
-it:
-
-.. code-block:: none
-
-  curl -k -X POST -F data='{"op": "set", "path": ["interfaces", "dummy", "dum1", "address"], "value": "203.0.113.76/32"}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/configure
-
-The ``/configure`` endpoint takes a request serialized in JSON. The only HTTP
-method it uses is POST. Request data is passed in the ``data=`` field and the
-API key is passed in the ``key=`` field. Key identifiers from the config are
-purely informational and the application doesn't need to know them, they only
-appear in the server logs to avoid exposing keys in log files, you only need
-the key itself.
-
-Since internally there is no distinction between a path and a value, you can
-omit the value field and include the value in the path like it's done in the
-shell commands:
-
-.. code-block:: none
-
-  curl -k -X POST -F data='{"op": "set", "path": ["interfaces", "dummy", "dum10", "address", "203.0.113.99/32"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/configure
-
-Separate value field make the semantics more clear though, and also makes it
-easier to create a command template once and update it with different values
-as needed.
-
-You can pass the ``set``, ``delete`` or ``comment`` command to it.
-The API will push the command to the session and commit.
-
-To retrieve a value:
-
-.. code-block:: none
-
-  curl -k -X POST -F data='{"op": "returnValue", "path": ["interfaces", "dummy", "dum1", "address"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/retrieve
-
-Use ``returnValues`` for multi-valued nodes.
+Please take a look at the :ref:`vyosapi` page for an detailed how-to.
 
+*************
+Configuration
+*************
 
-Show config
-"""""""""""
+.. cfgcmd:: set service https api keys id <name> key <apikey>
 
-To retrieve the full config under a path:
+   Set an named api key, every key have the same, full permissions
+   on the system.
 
-.. code-block:: none
-
-  # curl -k -X POST -F data='{"op": "showConfig", "path": ["interfaces", "dummy"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/retrieve
+.. cfgcmd:: set service https api debug
 
-It will return:
-
-.. code-block:: none
+   To enable debug messages. Available via :opcmd:`show log` or 
+   :opcmd:`monitor log`
 
-  {"success": true, "data": {"dummy": {"dum1": {"address": "203.0.113.76/32"}}}, "error": null}
-
-Passing an empty path will return the full config:
-
-.. code-block:: none
+.. cfgcmd:: set service https api port
 
-  # curl -k -X POST -F data='{"op": "showConfig", "path": []}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/retrieve
+   Set the listen port of the local API, this have non effect of the
+   webserver. The default is port 8080
 
+.. cfgcmd:: set service https api strict
 
-Configuration management requests
----------------------------------
+   Enforce strict path checking
 
-When saving or loading a configuration, the endpoint is ``/config-file`` and
-you can pass the ``save`` or ``load`` command.
+.. cfgcmd:: set service https virtual-host <vhost> listen-address
 
-If you don't specify the file when saving, it saves to ``/config/config.boot``.
-Here's an example:
+   Address to listen for HTTPS requests
 
-.. code-block:: none
+.. cfgcmd:: set service https virtual-host <vhost> listen-port <1-65535>
 
-  # curl -k -X POST -F key=MY-HTTP-API-PLAINTEXT-KEY -Fdata='{"op": "save", "file": "/config/config.boot"}' https://192.168.122.127/config-file
+   Port to listen for HTTPS requests; default 443
 
-Image management requests
--------------------------
+.. cfgcmd:: set service https virtual-host <vhost> server-name <text>
 
-One may ``add`` or ``delete`` a system image using the endpoint ``/image``.
-Here are the respective examples:
+   Server names for virtual hosts it ca be exact, wildcard or regex.
 
-``add`` from ``url``. Here we use the URL of the latest rolling release:
+.. cfgcmd:: set service https api-restrict virtual-host <vhost>
 
-.. code-block:: none
+   Nginx exposes the local API on all virtual servers, by default
+   Use this to restrict nginx to one or more virtual hosts.
 
-  # curl -k -X POST -F data='{"op": "add", "url": "https://downloads.vyos.io/rolling/current/amd64/vyos-rolling-latest.iso"}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/image
+.. cfgcmd:: set service https certificates certbot domain-name <text>
 
-``delete`` by image ``name``. For example:
+   Domain name(s) for which to obtain certificate
 
-.. code-block:: none
+.. cfgcmd:: set service https certificates certbot email
 
-  # curl -k -X POST -F data='{"op": "delete", "name": "1.3-rolling-202006070117"}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/image
+   Email address to associate with certificate
 
-To list the available system images by name, one may use the operational mode
-request ``show`` discussed in the next section; in this setting it would be:
+.. cfgcmd:: set service https certificates system-generated-certificate
 
-.. code-block:: none
+   Use an automatically generated self-signed certificate
 
-  # curl -k -X POST -F data='{"op": "show", "path": ["system", "image"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/show
+.. cfgcmd:: set service https certificates system-generated-certificate lifetime <days>
 
-Operational mode requests
--------------------------
+   Lifetime in days; default is 365
 
-It is possible to run ``show`` and ``generate`` commands:
 
+*********************
+Example Configuration
+*********************
 
-Request:
+Set an API-KEY is the minimal configuration to get a working API Endpoint.
 
 .. code-block:: none
 
-  curl -k -X POST -F data='{"op": "generate", "path": ["wireguard", "default-keypair"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/generate
-
-Response:
-
-.. code-block:: none
+   set service https api keys id MY-HTTPS-API-ID key MY-HTTPS-API-PLAINTEXT-KEY
 
-  {"success": true, "data": "", "error": null}
 
-Request:
+To use this full configuration we asume a publice accessable hostname.
 
 .. code-block:: none
 
-  curl -k -X POST -F data='{"op": "show", "path": ["wireguard", "keypairs", "pubkey", "default"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/show
-
-Response:
-
-.. code-block:: none
-
-  {"success": true, "data": "<some pubkey>=\n", "error": null}
-
-Request:
-
-.. code-block:: none
-
-  curl -k -X POST -F data='{"op": "show", "path": ["ip", "route"]}' -F key=MY-HTTP-API-PLAINTEXT-KEY https://192.168.122.127/show
-
-Response:
-
-.. code-block:: none
-
-  {"success": true, "data": "Codes: K - kernel route, C - connected, S - static, R - RIP,\n       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,\n       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,\n       F - PBR, f - OpenFabric,\n       > - selected route, * - FIB route, q - queued route, r - rejected route\n\nS>* 0.0.0.0/0 [210/0] via 192.168.100.1, eth0, 01:41:05\nC>* 192.168.0.0/24 is directly connected, eth1, 01:41:09\nC>* 192.168.100.0/24 is directly connected, eth0, 01:41:05\nC>* 203.0.113.76/32 is directly connected, dum1, 01:38:40\n", "error": null}
-
+   set service https api keys id MY-HTTPS-API-ID key MY-HTTPS-API-PLAINTEXT-KEY
+   set service https certificates certbot domain-name rtr01.example.com
+   set service https certificates certbot email mail@example.com
+   set service https virtual-host rtr01 listen-address 198.51.100.2
+   set service https virtual-host rtr01 listen-port 11443
+   set service https virtual-host rtr01 server-name rtr01.example.com
+   set service https api-restrict virtual-host rtr01.example.com
\ No newline at end of file