freebsd: introduce the freebsd renderer (#61)

* freebsd: introduce the freebsd renderer

Refactoring of the FreeBSD code base to provide a real network renderer
for FreeBSD.
Use the generic update_sysconfig_file() from rhel_util to handle the
access to /etc/rc.conf.
Interfaces are not automatically renamed by FreeBSD using
the following configuration in /etc/rc.conf:

```
ifconfig_fxp0_name="eth0"
```

* freesd: use regex named groups

Reduce the complexity of `get_interfaces_by_mac_on_freebsd()` with
named groups.

* freebsd: breaks up _write_network() in tree small functions

- `_write_ifconfig_entries()`
- `_write_route_entries()`
- `_write_resolve_conf()`

* extend find_fallback_nic() to support FreeBSD

this uses `route -n show default` to find the default interface

* freebsd: use dns keys from NetworkState class

The NetworkState class (settings instance) exposes the DNS configuration
in two keys:

- `dns_nameservers`
- `dns_searchdomains`

On OpenStack, these keys are set when a global DNS server is set. The
alternative is the `dns_nameservers` and `dns_search` keys from each
subdomain. We continue to read those.

* freebsd: properly target the /etc/resolv.conf file

* freebsd: ignore 'service routing restart' ret code

On FreeBSD 10, the restart of routing and dhclient is likely to fail because
- routing: it cannot remove the loopback route, but it will still set up
  the default route as expected.
- dhclient: it cannot stop the dhclient started by the netif service.
In both case, the situation is ok, and we can proceed.

* freebsd: handle case when metadata MAC local locally

Handle the case where the metadata configuration comes with a MAC that
does not exist locally.

See:
- https://github.com/canonical/cloud-init/pull/61/files/635ce14b3153934ba1041be48b7245062f21e960#r359600604
- https://github.com/canonical/cloud-init/pull/61/files/635ce14b3153934ba1041be48b7245062f21e960#r359600966

* freebsd: show up a warning if several subnet found

The FreeBSD provider currently only allow one subnet per interface.

* freebsd: honor the target parameter in _write_network

* freebsd: log when a bad route is found

* freebsd: pass _postcmds to start_services()

* freebsd: updatercconf() is depercated

Replace `updatercconf()` by `rhel_util.update_sysconfig_file()`.

* freebsd: ensure gateway is ipv4 before using it

With the legacy ENI format, an IPv6 gateway may be pushed. This instead
of the expected IPv4.

* freebsd: find_fallback_nic, support FB10

On FreeBSD <= 10, `ifconfig -l` ignores the down interfaces.

* freebsd: use util.target_path() to load resolv.conf

Ensure we access `/etc/resolv.conf`, not `etc/resolv.conf`.

* freebsd: skip subnet without netmask

Those are likely to be either invalid of in IPv6 format. IPv6 support
will be addressed later in a new patchset.

* freebsd: get_devicelist returns netif list

Ensure `get_devicelist()` returns the list of known netif on FreeBSD.

* replace rhel_util.update_sysconfig_file wrapper call, with a wrapper function

* reverse if condition to remove an indent

Co-authored-by: Igor Galić <me+github@igalic.co>

2 files changed, 24 insertions, 420 deletions

diff --git a/cloudinit/distros/__init__.py b/cloudinit/distros/__init__.py
index 6d69e6ca..cdce26f2 100755
--- a/cloudinit/distros/__init__.py
+++ b/cloudinit/distros/__init__.py
@@ -145,7 +145,7 @@ class Distro(object):
         # Write it out
 
         # pylint: disable=assignment-from-no-return
-        # We have implementations in arch, freebsd and gentoo still
+        # We have implementations in arch and gentoo still
         dev_names = self._write_network(settings)
         # pylint: enable=assignment-from-no-return
         # Now try to bring them up
diff --git a/cloudinit/distros/freebsd.py b/cloudinit/distros/freebsd.py
index 8e5ae96c..95cabc5c 100644
--- a/cloudinit/distros/freebsd.py
+++ b/cloudinit/distros/freebsd.py
@@ -13,12 +13,10 @@ import re
 from cloudinit import distros
 from cloudinit import helpers
 from cloudinit import log as logging
+from cloudinit import net
 from cloudinit import ssh_util
 from cloudinit import util
-
-from cloudinit.distros import net_util
-from cloudinit.distros.parsers.resolv_conf import ResolvConf
-
+from cloudinit.distros import rhel_util
 from cloudinit.settings import PER_INSTANCE
 
 LOG = logging.getLogger(__name__)
@@ -29,9 +27,8 @@ class Distro(distros.Distro):
     rc_conf_fn = "/etc/rc.conf"
     login_conf_fn = '/etc/login.conf'
     login_conf_fn_bak = '/etc/login.conf.orig'
-    resolv_conf_fn = '/etc/resolv.conf'
     ci_sudoers_fn = '/usr/local/etc/sudoers.d/90-cloud-init-users'
-    default_primary_nic = 'hn0'
+    hostname_conf_fn = '/etc/rc.conf'
 
     def __init__(self, name, cfg, paths):
         distros.Distro.__init__(self, name, cfg, paths)
@@ -40,99 +37,8 @@ class Distro(distros.Distro):
         # should only happen say once per instance...)
         self._runner = helpers.Runners(paths)
         self.osfamily = 'freebsd'
-        self.ipv4_pat = re.compile(r"\s+inet\s+\d+[.]\d+[.]\d+[.]\d+")
         cfg['ssh_svcname'] = 'sshd'
 
-    # Updates a key in /etc/rc.conf.
-    def updatercconf(self, key, value):
-        LOG.debug("Checking %s for: %s = %s", self.rc_conf_fn, key, value)
-        conf = self.loadrcconf()
-        config_changed = False
-        if key not in conf:
-            LOG.debug("Adding key in %s: %s = %s", self.rc_conf_fn, key,
-                      value)
-            conf[key] = value
-            config_changed = True
-        else:
-            for item in conf.keys():
-                if item == key and conf[item] != value:
-                    conf[item] = value
-                    LOG.debug("Changing key in %s: %s = %s", self.rc_conf_fn,
-                              key, value)
-                    config_changed = True
-
-        if config_changed:
-            LOG.info("Writing %s", self.rc_conf_fn)
-            buf = StringIO()
-            for keyval in conf.items():
-                buf.write('%s="%s"\n' % keyval)
-            util.write_file(self.rc_conf_fn, buf.getvalue())
-
-    # Load the contents of /etc/rc.conf and store all keys in a dict. Make sure
-    # quotes are ignored:
-    #  hostname="bla"
-    def loadrcconf(self):
-        RE_MATCH = re.compile(r'^(\w+)\s*=\s*(.*)\s*')
-        conf = {}
-        lines = util.load_file(self.rc_conf_fn).splitlines()
-        for line in lines:
-            m = RE_MATCH.match(line)
-            if not m:
-                LOG.debug("Skipping line from /etc/rc.conf: %s", line)
-                continue
-            key = m.group(1).rstrip()
-            val = m.group(2).rstrip()
-            # Kill them quotes (not completely correct, aka won't handle
-            # quoted values, but should be ok ...)
-            if val[0] in ('"', "'"):
-                val = val[1:]
-            if val[-1] in ('"', "'"):
-                val = val[0:-1]
-            if len(val) == 0:
-                LOG.debug("Skipping empty value from /etc/rc.conf: %s", line)
-                continue
-            conf[key] = val
-        return conf
-
-    def readrcconf(self, key):
-        conf = self.loadrcconf()
-        try:
-            val = conf[key]
-        except KeyError:
-            val = None
-        return val
-
-    # NOVA will inject something like eth0, rewrite that to use the FreeBSD
-    # adapter. Since this adapter is based on the used driver, we need to
-    # figure out which interfaces are available. On KVM platforms this is
-    # vtnet0, where Xen would use xn0.
-    def getnetifname(self, dev):
-        LOG.debug("Translating network interface %s", dev)
-        if dev.startswith('lo'):
-            return dev
-
-        n = re.search(r'\d+$', dev)
-        index = n.group(0)
-
-        (out, _err) = util.subp(['ifconfig', '-a'])
-        ifconfigoutput = [x for x in (out.strip()).splitlines()
-                          if len(x.split()) > 0]
-        bsddev = 'NOT_FOUND'
-        for line in ifconfigoutput:
-            m = re.match(r'^\w+', line)
-            if m:
-                if m.group(0).startswith('lo'):
-                    continue
-                # Just settle with the first non-lo adapter we find, since it's
-                # rather unlikely there will be multiple nicdrivers involved.
-                bsddev = m.group(0)
-                break
-
-        # Replace the index with the one we're after.
-        bsddev = re.sub(r'\d+$', index, bsddev)
-        LOG.debug("Using network interface %s", bsddev)
-        return bsddev
-
     def _select_hostname(self, hostname, fqdn):
         # Should be FQDN if available. See rc.conf(5) in FreeBSD
         if fqdn:
@@ -140,21 +46,18 @@ class Distro(distros.Distro):
         return hostname
 
     def _read_system_hostname(self):
-        sys_hostname = self._read_hostname(filename=None)
-        return ('rc.conf', sys_hostname)
+        sys_hostname = self._read_hostname(self.hostname_conf_fn)
+        return (self.hostname_conf_fn, sys_hostname)
 
     def _read_hostname(self, filename, default=None):
-        hostname = None
-        try:
-            hostname = self.readrcconf('hostname')
-        except IOError:
-            pass
-        if not hostname:
+        (_exists, contents) = rhel_util.read_sysconfig_file(filename)
+        if contents.get('hostname'):
+            return contents['hostname']
+        else:
             return default
-        return hostname
 
     def _write_hostname(self, hostname, filename):
-        self.updatercconf('hostname', hostname)
+        rhel_util.update_sysconfig_file(filename, {'hostname': hostname})
 
     def create_group(self, name, members):
         group_add_cmd = ['pw', '-n', name]
@@ -282,309 +185,16 @@ class Distro(distros.Distro):
             keys = set(kwargs['ssh_authorized_keys']) or []
             ssh_util.setup_user_keys(keys, name, options=None)
 
-    @staticmethod
-    def get_ifconfig_list():
-        cmd = ['ifconfig', '-l']
-        (nics, err) = util.subp(cmd, rcs=[0, 1])
-        if len(err):
-            LOG.warning("Error running %s: %s", cmd, err)
-            return None
-        return nics
-
-    @staticmethod
-    def get_ifconfig_ifname_out(ifname):
-        cmd = ['ifconfig', ifname]
-        (if_result, err) = util.subp(cmd, rcs=[0, 1])
-        if len(err):
-            LOG.warning("Error running %s: %s", cmd, err)
-            return None
-        return if_result
-
-    @staticmethod
-    def get_ifconfig_ether():
-        cmd = ['ifconfig', '-l', 'ether']
-        (nics, err) = util.subp(cmd, rcs=[0, 1])
-        if len(err):
-            LOG.warning("Error running %s: %s", cmd, err)
-            return None
-        return nics
-
-    @staticmethod
-    def get_interface_mac(ifname):
-        if_result = Distro.get_ifconfig_ifname_out(ifname)
-        for item in if_result.splitlines():
-            if item.find('ether ') != -1:
-                mac = str(item.split()[1])
-                if mac:
-                    return mac
-
-    @staticmethod
-    def get_devicelist():
-        nics = Distro.get_ifconfig_list()
-        return nics.split()
-
-    @staticmethod
-    def get_ipv6():
-        ipv6 = []
-        nics = Distro.get_devicelist()
-        for nic in nics:
-            if_result = Distro.get_ifconfig_ifname_out(nic)
-            for item in if_result.splitlines():
-                if item.find("inet6 ") != -1 and item.find("scopeid") == -1:
-                    ipv6.append(nic)
-        return ipv6
-
-    def get_ipv4(self):
-        ipv4 = []
-        nics = Distro.get_devicelist()
-        for nic in nics:
-            if_result = Distro.get_ifconfig_ifname_out(nic)
-            for item in if_result.splitlines():
-                print(item)
-                if self.ipv4_pat.match(item):
-                    ipv4.append(nic)
-        return ipv4
-
-    def is_up(self, ifname):
-        if_result = Distro.get_ifconfig_ifname_out(ifname)
-        pat = "^" + ifname
-        for item in if_result.splitlines():
-            if re.match(pat, item):
-                flags = item.split('<')[1].split('>')[0]
-                if flags.find("UP") != -1:
-                    return True
-
-    def _get_current_rename_info(self, check_downable=True):
-        """Collect information necessary for rename_interfaces."""
-        names = Distro.get_devicelist()
-        bymac = {}
-        for n in names:
-            bymac[Distro.get_interface_mac(n)] = {
-                'name': n, 'up': self.is_up(n), 'downable': None}
-
-        nics_with_addresses = set()
-        if check_downable:
-            nics_with_addresses = set(self.get_ipv4() + self.get_ipv6())
-
-        for d in bymac.values():
-            d['downable'] = (d['up'] is False or
-                             d['name'] not in nics_with_addresses)
-
-        return bymac
-
-    def _rename_interfaces(self, renames):
-        if not len(renames):
-            LOG.debug("no interfaces to rename")
-            return
-
-        current_info = self._get_current_rename_info()
-
-        cur_bymac = {}
-        for mac, data in current_info.items():
-            cur = data.copy()
-            cur['mac'] = mac
-            cur_bymac[mac] = cur
-
-        def update_byname(bymac):
-            return dict((data['name'], data)
-                        for data in bymac.values())
-
-        def rename(cur, new):
-            util.subp(["ifconfig", cur, "name", new], capture=True)
-
-        def down(name):
-            util.subp(["ifconfig", name, "down"], capture=True)
-
-        def up(name):
-            util.subp(["ifconfig", name, "up"], capture=True)
-
-        ops = []
-        errors = []
-        ups = []
-        cur_byname = update_byname(cur_bymac)
-        tmpname_fmt = "cirename%d"
-        tmpi = -1
-
-        for mac, new_name in renames:
-            cur = cur_bymac.get(mac, {})
-            cur_name = cur.get('name')
-            cur_ops = []
-            if cur_name == new_name:
-                # nothing to do
-                continue
-
-            if not cur_name:
-                errors.append("[nic not present] Cannot rename mac=%s to %s"
-                              ", not available." % (mac, new_name))
-                continue
-
-            if cur['up']:
-                msg = "[busy] Error renaming mac=%s from %s to %s"
-                if not cur['downable']:
-                    errors.append(msg % (mac, cur_name, new_name))
-                    continue
-                cur['up'] = False
-                cur_ops.append(("down", mac, new_name, (cur_name,)))
-                ups.append(("up", mac, new_name, (new_name,)))
-
-            if new_name in cur_byname:
-                target = cur_byname[new_name]
-                if target['up']:
-                    msg = "[busy-target] Error renaming mac=%s from %s to %s."
-                    if not target['downable']:
-                        errors.append(msg % (mac, cur_name, new_name))
-                        continue
-                    else:
-                        cur_ops.append(("down", mac, new_name, (new_name,)))
-
-                tmp_name = None
-                while tmp_name is None or tmp_name in cur_byname:
-                    tmpi += 1
-                    tmp_name = tmpname_fmt % tmpi
-
-                cur_ops.append(("rename", mac, new_name, (new_name, tmp_name)))
-                target['name'] = tmp_name
-                cur_byname = update_byname(cur_bymac)
-                if target['up']:
-                    ups.append(("up", mac, new_name, (tmp_name,)))
-
-            cur_ops.append(("rename", mac, new_name, (cur['name'], new_name)))
-            cur['name'] = new_name
-            cur_byname = update_byname(cur_bymac)
-            ops += cur_ops
-
-        opmap = {'rename': rename, 'down': down, 'up': up}
-        if len(ops) + len(ups) == 0:
-            if len(errors):
-                LOG.debug("unable to do any work for renaming of %s", renames)
-            else:
-                LOG.debug("no work necessary for renaming of %s", renames)
-        else:
-            LOG.debug("achieving renaming of %s with ops %s",
-                      renames, ops + ups)
-
-            for op, mac, new_name, params in ops + ups:
-                try:
-                    opmap.get(op)(*params)
-                except Exception as e:
-                    errors.append(
-                        "[unknown] Error performing %s%s for %s, %s: %s" %
-                        (op, params, mac, new_name, e))
-        if len(errors):
-            raise Exception('\n'.join(errors))
-
-    def apply_network_config_names(self, netcfg):
-        renames = []
-        for ent in netcfg.get('config', {}):
-            if ent.get('type') != 'physical':
-                continue
-            mac = ent.get('mac_address')
-            name = ent.get('name')
-            if not mac:
-                continue
-            renames.append([mac, name])
-        return self._rename_interfaces(renames)
-
-    @classmethod
     def generate_fallback_config(self):
-        nics = Distro.get_ifconfig_ether()
-        if nics is None:
-            LOG.debug("Fail to get network interfaces")
-            return None
-        potential_interfaces = nics.split()
-        connected = []
-        for nic in potential_interfaces:
-            pat = "^" + nic
-            if_result = Distro.get_ifconfig_ifname_out(nic)
-            for item in if_result.split("\n"):
-                if re.match(pat, item):
-                    flags = item.split('<')[1].split('>')[0]
-                    if flags.find("RUNNING") != -1:
-                        connected.append(nic)
-        if connected:
-            potential_interfaces = connected
-        names = list(sorted(potential_interfaces))
-        default_pri_nic = Distro.default_primary_nic
-        if default_pri_nic in names:
-            names.remove(default_pri_nic)
-            names.insert(0, default_pri_nic)
-        target_name = None
-        target_mac = None
-        for name in names:
-            mac = Distro.get_interface_mac(name)
-            if mac:
-                target_name = name
-                target_mac = mac
-                break
-        if target_mac and target_name:
-            nconf = {'config': [], 'version': 1}
+        nconf = {'config': [], 'version': 1}
+        for mac, name in net.get_interfaces_by_mac().items():
             nconf['config'].append(
-                {'type': 'physical', 'name': target_name,
-                 'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]})
-            return nconf
-        else:
-            return None
-
-    def _write_network(self, settings):
-        entries = net_util.translate_network(settings)
-        nameservers = []
-        searchdomains = []
-        dev_names = entries.keys()
-        for (device, info) in entries.items():
-            # Skip the loopback interface.
-            if device.startswith('lo'):
-                continue
-
-            dev = self.getnetifname(device)
-
-            LOG.info('Configuring interface %s', dev)
-
-            if info.get('bootproto') == 'static':
-                LOG.debug('Configuring dev %s with %s / %s', dev,
-                          info.get('address'), info.get('netmask'))
-                # Configure an ipv4 address.
-                ifconfig = (info.get('address') + ' netmask ' +
-                            info.get('netmask'))
-
-                # Configure the gateway.
-                self.updatercconf('defaultrouter', info.get('gateway'))
-
-                if 'dns-nameservers' in info:
-                    nameservers.extend(info['dns-nameservers'])
-                if 'dns-search' in info:
-                    searchdomains.extend(info['dns-search'])
-            else:
-                ifconfig = 'DHCP'
-
-            self.updatercconf('ifconfig_' + dev, ifconfig)
-
-        # Try to read the /etc/resolv.conf or just start from scratch if that
-        # fails.
-        try:
-            resolvconf = ResolvConf(util.load_file(self.resolv_conf_fn))
-            resolvconf.parse()
-        except IOError:
-            util.logexc(LOG, "Failed to parse %s, use new empty file",
-                        self.resolv_conf_fn)
-            resolvconf = ResolvConf('')
-            resolvconf.parse()
-
-        # Add some nameservers
-        for server in nameservers:
-            try:
-                resolvconf.add_nameserver(server)
-            except ValueError:
-                util.logexc(LOG, "Failed to add nameserver %s", server)
-
-        # And add any searchdomains.
-        for domain in searchdomains:
-            try:
-                resolvconf.add_search_domain(domain)
-            except ValueError:
-                util.logexc(LOG, "Failed to add search domain %s", domain)
-        util.write_file(self.resolv_conf_fn, str(resolvconf), 0o644)
+                {'type': 'physical', 'name': name,
+                 'mac_address': mac, 'subnets': [{'type': 'dhcp'}]})
+        return nconf
 
-        return dev_names
+    def _write_network_config(self, netconfig):
+        return self._supported_write_network_config(netconfig)
 
     def apply_locale(self, locale, out_fn=None):
         # Adjust the locals value to the new value
@@ -612,18 +222,12 @@ class Distro(distros.Distro):
                 util.logexc(LOG, "Failed to restore %s backup",
                             self.login_conf_fn)
 
-    def _bring_up_interface(self, device_name):
-        if device_name.startswith('lo'):
-            return
-        dev = self.getnetifname(device_name)
-        cmd = ['/etc/rc.d/netif', 'start', dev]
-        LOG.debug("Attempting to bring up interface %s using command %s",
-                  dev, cmd)
-        # This could return 1 when the interface has already been put UP by the
-        # OS. This is just fine.
-        (_out, err) = util.subp(cmd, rcs=[0, 1])
-        if len(err):
-            LOG.warning("Error running %s: %s", cmd, err)
+    def apply_network_config_names(self, netconfig):
+        # This is handled by the freebsd network renderer. It writes in
+        # /etc/rc.conf a line with the following format:
+        #    ifconfig_OLDNAME_name=NEWNAME
+        # FreeBSD network script will rename the interface automatically.
+        return
 
     def install_packages(self, pkglist):
         self.update_package_sources()