pki: T5886: add support for ACME protocol (LetsEncrypt)

The "idea" of this PR is to add new CLI nodes under the pki subsystem to
activate ACME for any given certificate.

vyos@vyos# set pki certificate NAME acme
Possible completions:
+  domain-name          Domain Name
   email                Email address to associate with certificate
   listen-address       Local IPv4 addresses to listen on
   rsa-key-size         Size of the RSA key (default: 2048)
   url                  Remote URL (default:
                        https://acme-v02.api.letsencrypt.org/directory)

Users choose if the CLI based custom certificates are used
  set pki certificate EXAMPLE acme certificate <base64>
or if it should be generated via ACME.

The ACME server URL defaults to LetsEncrypt but can be changed to their staging
API for testing to not get blacklisted.
  set pki certificate EXAMPLE acme url https://acme-staging-v02.api.letsencrypt.org/directory

Certificate retrieval has a certbot --dry-run stage in verify() to see if it
can be generated.

After successful generation, the certificate is stored in under
/config/auth/letsencrypt. Once a certificate is referenced in the CLI (e.g. set
interfaces ethernet eth0 eapol certificate EXAMPLE) we call
vyos.config.get_config_dict() which will (if with_pki=True is set) blend in the
base64 encoded certificate into the JSON data structure normally used when
using a certificate set by the CLI.

Using this "design" does not need any change to any other code referencing the
PKI system, as the base64 encoded certificate is already there.

certbot renewal will call the PKI python script to trigger dependency updates.

(cherry picked from commit b8db1a9d7baf91b70c1b735e58710f1e2bc9fc7a)

# Conflicts:
#	debian/control

8 files changed, 287 insertions, 53 deletions

diff --git a/debian/control b/debian/control
index 3f1ba1a63..bbe9df1f1 100644
--- a/debian/control
+++ b/debian/control
@@ -39,6 +39,7 @@ Depends:
   beep,
   bmon,
   bsdmainutils,
+  certbot,
   charon-systemd,
   conntrack,
   conntrackd,
diff --git a/interface-definitions/include/constraint/email.xml.i b/interface-definitions/include/constraint/email.xml.i
new file mode 100644
index 000000000..b19a88d64
--- /dev/null
+++ b/interface-definitions/include/constraint/email.xml.i
@@ -0,0 +1,3 @@
+<!-- include start from constraint/email.xml.i -->
+<regex>[^\s@]+@([^\s@.,]+\.)+[^\s@.,]{2,}</regex>
+<!-- include end -->
diff --git a/interface-definitions/pki.xml.in b/interface-definitions/pki.xml.in
index 097c541ac..0ed199539 100644
--- a/interface-definitions/pki.xml.in
+++ b/interface-definitions/pki.xml.in
@@ -81,6 +81,60 @@
               <constraintErrorMessage>Certificate is not base64-encoded</constraintErrorMessage>
             </properties>
           </leafNode>
+          <node name="acme">
+            <properties>
+              <help>Automatic Certificate Management Environment (ACME) request</help>
+            </properties>
+            <children>
+              #include <include/url-http-https.xml.i>
+              <leafNode name="url">
+                <defaultValue>https://acme-v02.api.letsencrypt.org/directory</defaultValue>
+              </leafNode>
+              <leafNode name="domain-name">
+                <properties>
+                  <help>Domain Name</help>
+                  <constraint>
+                    <validator name="fqdn"/>
+                  </constraint>
+                  <constraintErrorMessage>Invalid domain name (RFC 1123 section 2).\nMay only contain letters, numbers and .-_</constraintErrorMessage>
+                  <multi/>
+                </properties>
+              </leafNode>
+              <leafNode name="email">
+                <properties>
+                  <help>Email address to associate with certificate</help>
+                  <constraint>
+                    #include <include/constraint/email.xml.i>
+                  </constraint>
+                </properties>
+              </leafNode>
+              #include <include/listen-address-ipv4-single.xml.i>
+              <leafNode name="rsa-key-size">
+                <properties>
+                  <help>Size of the RSA key</help>
+                  <completionHelp>
+                    <list>2048 3072 4096</list>
+                  </completionHelp>
+                  <valueHelp>
+                    <format>2048</format>
+                    <description>RSA key length 2048 bit</description>
+                  </valueHelp>
+                  <valueHelp>
+                    <format>3072</format>
+                    <description>RSA key length 3072 bit</description>
+                  </valueHelp>
+                  <valueHelp>
+                    <format>4096</format>
+                    <description>RSA key length 4096 bit</description>
+                  </valueHelp>
+                  <constraint>
+                    <regex>(2048|3072|4096)</regex>
+                  </constraint>
+                </properties>
+                <defaultValue>2048</defaultValue>
+              </leafNode>
+            </children>
+          </node>
           #include <include/generic-description.xml.i>
           <node name="private">
             <properties>
diff --git a/python/vyos/config.py b/python/vyos/config.py
index ca7b035e5..bee85315d 100644
--- a/python/vyos/config.py
+++ b/python/vyos/config.py
@@ -92,6 +92,38 @@ def config_dict_merge(src: dict, dest: Union[dict, ConfigDict]) -> ConfigDict:
         dest = ConfigDict(dest)
     return ext_dict_merge(src, dest)
 
+def config_dict_mangle_acme(name, cli_dict):
+    """
+    Load CLI PKI dictionary and if an ACME certificate is used, load it's content
+    and place it into the CLI dictionary as it would be a "regular" CLI PKI based
+    certificate with private key
+    """
+    from vyos.base import ConfigError
+    from vyos.defaults import directories
+    from vyos.utils.file import read_file
+    from vyos.pki import encode_certificate
+    from vyos.pki import encode_private_key
+    from vyos.pki import load_certificate
+    from vyos.pki import load_private_key
+
+    try:
+        vyos_certbot_dir = directories['certbot']
+
+        if 'acme' in cli_dict:
+            tmp = read_file(f'{vyos_certbot_dir}/live/{name}/cert.pem')
+            tmp = load_certificate(tmp, wrap_tags=False)
+            cert_base64 = "".join(encode_certificate(tmp).strip().split("\n")[1:-1])
+
+            tmp = read_file(f'{vyos_certbot_dir}/live/{name}/privkey.pem')
+            tmp = load_private_key(tmp, wrap_tags=False)
+            key_base64 = "".join(encode_private_key(tmp).strip().split("\n")[1:-1])
+            # install ACME based PEM keys into "regular" CLI config keys
+            cli_dict.update({'certificate' : cert_base64, 'private' : {'key' : key_base64}})
+    except:
+        raise ConfigError(f'Unable to load ACME certificates for "{name}"!')
+
+    return cli_dict
+
 class Config(object):
     """
     The class of config access objects.
@@ -306,6 +338,11 @@ class Config(object):
                                             no_tag_node_value_mangle=True,
                                             get_first_key=True)
             if pki_dict:
+                if 'certificate' in pki_dict:
+                    for certificate in pki_dict['certificate']:
+                        pki_dict['certificate'][certificate] = config_dict_mangle_acme(
+                            certificate, pki_dict['certificate'][certificate])
+
                 conf_dict['pki'] = pki_dict
 
         # save optional args for a call to get_config_defaults
diff --git a/src/conf_mode/pki.py b/src/conf_mode/pki.py
index f7e14aa16..310519abd 100755
--- a/src/conf_mode/pki.py
+++ b/src/conf_mode/pki.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 #
-# Copyright (C) 2021 VyOS maintainers and contributors
+# Copyright (C) 2021-2024 VyOS maintainers and contributors
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 or later as
@@ -14,59 +14,66 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import os
+
+from sys import argv
 from sys import exit
 
 from vyos.config import Config
-from vyos.configdep import set_dependents, call_dependents
+from vyos.config import config_dict_merge
+from vyos.configdep import set_dependents
+from vyos.configdep import call_dependents
 from vyos.configdict import node_changed
+from vyos.configdiff import Diff
+from vyos.defaults import directories
 from vyos.pki import is_ca_certificate
 from vyos.pki import load_certificate
 from vyos.pki import load_public_key
 from vyos.pki import load_private_key
 from vyos.pki import load_crl
 from vyos.pki import load_dh_parameters
+from vyos.utils.boot import boot_configuration_complete
+from vyos.utils.dict import dict_search
 from vyos.utils.dict import dict_search_args
 from vyos.utils.dict import dict_search_recursive
+from vyos.utils.process import call
+from vyos.utils.process import cmd
+from vyos.utils.process import is_systemd_service_active
 from vyos import ConfigError
 from vyos import airbag
 airbag.enable()
 
-# keys to recursively search for under specified path, script to call if update required
+vyos_certbot_dir = directories['certbot']
+
+# keys to recursively search for under specified path
 sync_search = [
     {
         'keys': ['certificate'],
         'path': ['service', 'https'],
-        'script': '/usr/libexec/vyos/conf_mode/service_https.py'
     },
     {
         'keys': ['certificate', 'ca_certificate'],
         'path': ['interfaces', 'ethernet'],
-        'script': '/usr/libexec/vyos/conf_mode/interfaces_ethernet.py'
     },
     {
         'keys': ['certificate', 'ca_certificate', 'dh_params', 'shared_secret_key', 'auth_key', 'crypt_key'],
         'path': ['interfaces', 'openvpn'],
-        'script': '/usr/libexec/vyos/conf_mode/interfaces_openvpn.py'
     },
     {
         'keys': ['ca_certificate'],
         'path': ['interfaces', 'sstpc'],
-        'script': '/usr/libexec/vyos/conf_mode/interfaces_sstpc.py'
     },
     {
         'keys': ['certificate', 'ca_certificate', 'local_key', 'remote_key'],
         'path': ['vpn', 'ipsec'],
-        'script': '/usr/libexec/vyos/conf_mode/vpn_ipsec.py'
     },
     {
         'keys': ['certificate', 'ca_certificate'],
         'path': ['vpn', 'openconnect'],
-        'script': '/usr/libexec/vyos/conf_mode/vpn_openconnect.py'
     },
     {
         'keys': ['certificate', 'ca_certificate'],
         'path': ['vpn', 'sstp'],
-        'script': '/usr/libexec/vyos/conf_mode/vpn_sstp.py'
     }
 ]
 
@@ -82,6 +89,33 @@ sync_translate = {
     'crypt_key': 'openvpn'
 }
 
+def certbot_delete(certificate):
+    if not boot_configuration_complete():
+        return
+    if os.path.exists(f'{vyos_certbot_dir}/renewal/{certificate}.conf'):
+        cmd(f'certbot delete --non-interactive --config-dir {vyos_certbot_dir} --cert-name {certificate}')
+
+def certbot_request(name: str, config: dict, dry_run: bool=True):
+    # We do not call certbot when booting the system - there is no need to do so and
+    # request new certificates during boot/image upgrade as the certbot configuration
+    # is stored persistent under /config - thus we do not open the door to transient
+    # errors
+    if not boot_configuration_complete():
+        return
+
+    domains = '--domains ' + ' --domains '.join(config['domain_name'])
+    tmp = f'certbot certonly --config-dir {vyos_certbot_dir} --cert-name {name} '\
+            f'--non-interactive --standalone --agree-tos --no-eff-email --expand '\
+            f'--server {config["url"]} --email {config["email"]} '\
+            f'--key-type rsa --rsa-key-size {config["rsa_key_size"]} {domains}'
+    if 'listen_address' in config:
+        tmp += f' --http-01-address {config["listen_address"]}'
+    # verify() does not need to actually request a cert but only test for plausability
+    if dry_run:
+        tmp += ' --dry-run'
+
+    cmd(tmp, raising=ConfigError, message=f'ACME certbot request failed for "{name}"!')
+
 def get_config(config=None):
     if config:
         conf = config
@@ -93,25 +127,62 @@ def get_config(config=None):
                                      get_first_key=True,
                                      no_tag_node_value_mangle=True)
 
-    pki['changed'] = {}
+    if len(argv) > 1 and argv[1] == 'certbot_renew':
+        pki['certbot_renew'] = {}
+
     tmp = node_changed(conf, base + ['ca'], key_mangling=('-', '_'), recursive=True)
-    if tmp: pki['changed'].update({'ca' : tmp})
+    if tmp:
+        if 'changed' not in pki: pki.update({'changed':{}})
+        pki['changed'].update({'ca' : tmp})
 
-    tmp = node_changed(conf, base + ['certificate'], key_mangling=('-', '_'), recursive=True)
-    if tmp: pki['changed'].update({'certificate' : tmp})
+    tmp = node_changed(conf, base + ['certificate'], key_mangling=('-', '_'),
+                       recursive=True, expand_nodes=Diff.ADD|Diff.DELETE)
+    if tmp:
+        if 'changed' not in pki: pki.update({'changed':{}})
+        pki['changed'].update({'certificate' : tmp})
 
     tmp = node_changed(conf, base + ['dh'], key_mangling=('-', '_'), recursive=True)
-    if tmp: pki['changed'].update({'dh' : tmp})
+    if tmp:
+        if 'changed' not in pki: pki.update({'changed':{}})
+        pki['changed'].update({'dh' : tmp})
 
     tmp = node_changed(conf, base + ['key-pair'], key_mangling=('-', '_'), recursive=True)
-    if tmp: pki['changed'].update({'key_pair' : tmp})
+    if tmp:
+        if 'changed' not in pki: pki.update({'changed':{}})
+        pki['changed'].update({'key_pair' : tmp})
 
-    tmp = node_changed(conf, base + ['openvpn', 'shared-secret'], key_mangling=('-', '_'), recursive=True)
-    if tmp: pki['changed'].update({'openvpn' : tmp})
+    tmp = node_changed(conf, base + ['openvpn', 'shared-secret'], key_mangling=('-', '_'),
+                       recursive=True)
+    if tmp:
+        if 'changed' not in pki: pki.update({'changed':{}})
+        pki['changed'].update({'openvpn' : tmp})
 
     # We only merge on the defaults of there is a configuration at all
     if conf.exists(base):
-        pki = conf.merge_defaults(pki, recursive=True)
+        # We have gathered the dict representation of the CLI, but there are default
+        # options which we need to update into the dictionary retrived.
+        default_values = conf.get_config_defaults(**pki.kwargs, recursive=True)
+        # remove ACME default configuration if unused by CLI
+        if 'certificate' in pki:
+            for name, cert_config in pki['certificate'].items():
+                if 'acme' not in cert_config:
+                    # Remove ACME default values
+                    del default_values['certificate'][name]['acme']
+
+        # merge CLI and default dictionary
+        pki = config_dict_merge(default_values, pki)
+
+    # Certbot triggered an external renew of the certificates.
+    # Mark all ACME based certificates as "changed" to trigger
+    # update of dependent services
+    if 'certificate' in pki and 'certbot_renew' in pki:
+        renew = []
+        for name, cert_config in pki['certificate'].items():
+            if 'acme' in cert_config:
+                renew.append(name)
+        # If triggered externally by certbot, certificate key is not present in changed
+        if 'changed' not in pki: pki.update({'changed':{}})
+        pki['changed'].update({'certificate' : renew})
 
     # We need to get the entire system configuration to verify that we are not
     # deleting a certificate that is still referenced somewhere!
@@ -119,38 +190,34 @@ def get_config(config=None):
                                          get_first_key=True,
                                          no_tag_node_value_mangle=True)
 
-    if 'changed' in pki:
-        for search in sync_search:
-            for key in search['keys']:
-                changed_key = sync_translate[key]
-
-                if changed_key not in pki['changed']:
-                    continue
-
-                for item_name in pki['changed'][changed_key]:
-                    node_present = False
-                    if changed_key == 'openvpn':
-                        node_present = dict_search_args(pki, 'openvpn', 'shared_secret', item_name)
-                    else:
-                        node_present = dict_search_args(pki, changed_key, item_name)
-
-                    if node_present:
-                        search_dict = dict_search_args(pki['system'], *search['path'])
-
-                        if not search_dict:
-                            continue
-
-                        for found_name, found_path in dict_search_recursive(search_dict, key):
-                            if found_name == item_name:
-                                path = search['path']
-                                path_str = ' '.join(path + found_path)
-                                print(f'pki: Updating config: {path_str} {found_name}')
-
-                                if path[0] == 'interfaces':
-                                    ifname = found_path[0]
-                                    set_dependents(path[1], conf, ifname)
-                                else:
-                                    set_dependents(path[1], conf)
+    for search in sync_search:
+        for key in search['keys']:
+            changed_key = sync_translate[key]
+            if 'changed' not in pki or changed_key not in pki['changed']:
+                continue
+
+            for item_name in pki['changed'][changed_key]:
+                node_present = False
+                if changed_key == 'openvpn':
+                    node_present = dict_search_args(pki, 'openvpn', 'shared_secret', item_name)
+                else:
+                    node_present = dict_search_args(pki, changed_key, item_name)
+
+                if node_present:
+                    search_dict = dict_search_args(pki['system'], *search['path'])
+                    if not search_dict:
+                        continue
+                    for found_name, found_path in dict_search_recursive(search_dict, key):
+                        if found_name == item_name:
+                            path = search['path']
+                            path_str = ' '.join(path + found_path)
+                            print(f'pki: Updating config: {path_str} {found_name}')
+
+                            if path[0] == 'interfaces':
+                                ifname = found_path[0]
+                                set_dependents(path[1], conf, ifname)
+                            else:
+                                set_dependents(path[1], conf)
 
     return pki
 
@@ -223,6 +290,22 @@ def verify(pki):
                 if not is_valid_private_key(private['key'], protected):
                     raise ConfigError(f'Invalid private key on certificate "{name}"')
 
+            if 'acme' in cert_conf:
+                if 'domain_name' not in cert_conf['acme']:
+                    raise ConfigError(f'At least one domain-name is required to request '\
+                                    f'certificate for "{name}" via ACME!')
+
+                if 'email' not in cert_conf['acme']:
+                    raise ConfigError(f'An email address is required to request '\
+                                    f'certificate for "{name}" via ACME!')
+
+                if 'certbot_renew' not in pki:
+                    # Only run the ACME command if something on this entity changed,
+                    # as this is time intensive
+                    tmp = dict_search('changed.certificate', pki)
+                    if tmp != None and name in tmp:
+                        certbot_request(name, cert_conf['acme'])
+
     if 'dh' in pki:
         for name, dh_conf in pki['dh'].items():
             if 'parameters' in dh_conf:
@@ -283,12 +366,50 @@ def generate(pki):
     if not pki:
         return None
 
+    # Certbot renewal only needs to re-trigger the services to load up the
+    # new PEM file
+    if 'certbot_renew' in pki:
+        return None
+
+    # list of certificates issued via certbot
+    certbot_list = []
+    if 'certificate' in pki:
+        for name, cert_conf in pki['certificate'].items():
+            if 'acme' in cert_conf:
+                certbot_list.append(name)
+                # when something for the certificate changed, we should delete it
+                if name in dict_search('changed.certificate', pki):
+                    certbot_delete(name)
+                    certbot_request(name, cert_conf['acme'], dry_run=False)
+
+    # Cleanup certbot configuration and certificates if no longer in use by CLI
+    # Get foldernames under vyos_certbot_dir which each represent a certbot cert
+    if os.path.exists(f'{vyos_certbot_dir}/live'):
+        for cert in [f.path.split('/')[-1] for f in os.scandir(f'{vyos_certbot_dir}/live') if f.is_dir()]:
+            if cert not in certbot_list:
+                # certificate is no longer active on the CLI - remove it
+                certbot_delete(cert)
+
     return None
 
 def apply(pki):
+    systemd_certbot_name = 'certbot.timer'
     if not pki:
+        call(f'systemctl stop {systemd_certbot_name}')
         return None
 
+    has_certbot = False
+    if 'certificate' in pki:
+        for name, cert_conf in pki['certificate'].items():
+            if 'acme' in cert_conf:
+                has_certbot = True
+                break
+
+    if not has_certbot:
+        call(f'systemctl stop {systemd_certbot_name}')
+    elif has_certbot and not is_systemd_service_active(systemd_certbot_name):
+        call(f'systemctl restart {systemd_certbot_name}')
+
     if 'changed' in pki:
         call_dependents()
 
diff --git a/src/etc/systemd/system/certbot.service.d/10-override.conf b/src/etc/systemd/system/certbot.service.d/10-override.conf
new file mode 100644
index 000000000..542f77eb2
--- /dev/null
+++ b/src/etc/systemd/system/certbot.service.d/10-override.conf
@@ -0,0 +1,7 @@
+[Unit]
+After=
+After=vyos-router.service
+
+[Service]
+ExecStart=
+ExecStart=/usr/bin/certbot renew --config-dir /config/auth/letsencrypt --no-random-sleep-on-renew --post-hook "/usr/libexec/vyos/vyos-certbot-renew-pki.sh"
diff --git a/src/helpers/vyos-certbot-renew-pki.sh b/src/helpers/vyos-certbot-renew-pki.sh
new file mode 100755
index 000000000..d0b663f7b
--- /dev/null
+++ b/src/helpers/vyos-certbot-renew-pki.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+source /opt/vyatta/etc/functions/script-template
+/usr/libexec/vyos/conf_mode/pki.py certbot_renew
diff --git a/src/op_mode/pki.py b/src/op_mode/pki.py
index 6c854afb5..ad2c1ada0 100755
--- a/src/op_mode/pki.py
+++ b/src/op_mode/pki.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 #
-# Copyright (C) 2021-2023 VyOS maintainers and contributors
+# Copyright (C) 2021-2024 VyOS maintainers and contributors
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 or later as
@@ -25,6 +25,7 @@ from cryptography import x509
 from cryptography.x509.oid import ExtendedKeyUsageOID
 
 from vyos.config import Config
+from vyos.config import config_dict_mangle_acme
 from vyos.pki import encode_certificate, encode_public_key, encode_private_key, encode_dh_parameters
 from vyos.pki import get_certificate_fingerprint
 from vyos.pki import create_certificate, create_certificate_request, create_certificate_revocation_list
@@ -79,9 +80,14 @@ def get_config_certificate(name=None):
         if not conf.exists(base + ['private', 'key']) or not conf.exists(base + ['certificate']):
             return False
 
-    return conf.get_config_dict(base, key_mangling=('-', '_'),
+    pki = conf.get_config_dict(base, key_mangling=('-', '_'),
                                 get_first_key=True,
                                 no_tag_node_value_mangle=True)
+    if pki:
+        for certificate in pki:
+            pki[certificate] = config_dict_mangle_acme(certificate, pki[certificate])
+
+    return pki
 
 def get_certificate_ca(cert, ca_certs):
     # Find CA certificate for given certificate
@@ -1073,7 +1079,9 @@ if __name__ == '__main__':
                 show_crl(None if args.crl == 'all' else args.crl, args.pem)
             else:
                 show_certificate_authority()
+                print('\n')
                 show_certificate()
+                print('\n')
                 show_crl()
     except KeyboardInterrupt:
         print("Aborted")