freebsd: introduce the freebsd renderer (#61)

* freebsd: introduce the freebsd renderer

Refactoring of the FreeBSD code base to provide a real network renderer
for FreeBSD.
Use the generic update_sysconfig_file() from rhel_util to handle the
access to /etc/rc.conf.
Interfaces are not automatically renamed by FreeBSD using
the following configuration in /etc/rc.conf:

```
ifconfig_fxp0_name="eth0"
```

* freesd: use regex named groups

Reduce the complexity of `get_interfaces_by_mac_on_freebsd()` with
named groups.

* freebsd: breaks up _write_network() in tree small functions

- `_write_ifconfig_entries()`
- `_write_route_entries()`
- `_write_resolve_conf()`

* extend find_fallback_nic() to support FreeBSD

this uses `route -n show default` to find the default interface

* freebsd: use dns keys from NetworkState class

The NetworkState class (settings instance) exposes the DNS configuration
in two keys:

- `dns_nameservers`
- `dns_searchdomains`

On OpenStack, these keys are set when a global DNS server is set. The
alternative is the `dns_nameservers` and `dns_search` keys from each
subdomain. We continue to read those.

* freebsd: properly target the /etc/resolv.conf file

* freebsd: ignore 'service routing restart' ret code

On FreeBSD 10, the restart of routing and dhclient is likely to fail because
- routing: it cannot remove the loopback route, but it will still set up
  the default route as expected.
- dhclient: it cannot stop the dhclient started by the netif service.
In both case, the situation is ok, and we can proceed.

* freebsd: handle case when metadata MAC local locally

Handle the case where the metadata configuration comes with a MAC that
does not exist locally.

See:
- https://github.com/canonical/cloud-init/pull/61/files/635ce14b3153934ba1041be48b7245062f21e960#r359600604
- https://github.com/canonical/cloud-init/pull/61/files/635ce14b3153934ba1041be48b7245062f21e960#r359600966

* freebsd: show up a warning if several subnet found

The FreeBSD provider currently only allow one subnet per interface.

* freebsd: honor the target parameter in _write_network

* freebsd: log when a bad route is found

* freebsd: pass _postcmds to start_services()

* freebsd: updatercconf() is depercated

Replace `updatercconf()` by `rhel_util.update_sysconfig_file()`.

* freebsd: ensure gateway is ipv4 before using it

With the legacy ENI format, an IPv6 gateway may be pushed. This instead
of the expected IPv4.

* freebsd: find_fallback_nic, support FB10

On FreeBSD <= 10, `ifconfig -l` ignores the down interfaces.

* freebsd: use util.target_path() to load resolv.conf

Ensure we access `/etc/resolv.conf`, not `etc/resolv.conf`.

* freebsd: skip subnet without netmask

Those are likely to be either invalid of in IPv6 format. IPv6 support
will be addressed later in a new patchset.

* freebsd: get_devicelist returns netif list

Ensure `get_devicelist()` returns the list of known netif on FreeBSD.

* replace rhel_util.update_sysconfig_file wrapper call, with a wrapper function

* reverse if condition to remove an indent

Co-authored-by: Igor Galić <me+github@igalic.co>

3 files changed, 244 insertions, 1 deletions

diff --git a/cloudinit/net/__init__.py b/cloudinit/net/__init__.py
index bd806378..1d5eb535 100644
--- a/cloudinit/net/__init__.py
+++ b/cloudinit/net/__init__.py
@@ -307,6 +307,9 @@ def device_devid(devname):
 
 
 def get_devicelist():
+    if util.is_FreeBSD():
+        return list(get_interfaces_by_mac().values())
+
     try:
         devs = os.listdir(get_sys_class_path())
     except OSError as e:
@@ -329,6 +332,35 @@ def is_disabled_cfg(cfg):
 
 def find_fallback_nic(blacklist_drivers=None):
     """Return the name of the 'fallback' network device."""
+    if util.is_FreeBSD():
+        return find_fallback_nic_on_freebsd(blacklist_drivers)
+    else:
+        return find_fallback_nic_on_linux(blacklist_drivers)
+
+
+def find_fallback_nic_on_freebsd(blacklist_drivers=None):
+    """Return the name of the 'fallback' network device on FreeBSD.
+
+    @param blacklist_drivers: currently ignored
+    @return default interface, or None
+
+
+    we'll use the first interface from ``ifconfig -l -u ether``
+    """
+    stdout, _stderr = util.subp(['ifconfig', '-l', '-u', 'ether'])
+    values = stdout.split()
+    if values:
+        return values[0]
+    # On FreeBSD <= 10, 'ifconfig -l' ignores the interfaces with DOWN
+    # status
+    values = list(get_interfaces_by_mac().values())
+    values.sort()
+    if values:
+        return values[0]
+
+
+def find_fallback_nic_on_linux(blacklist_drivers=None):
+    """Return the name of the 'fallback' network device on Linux."""
     if not blacklist_drivers:
         blacklist_drivers = []
 
@@ -765,6 +797,40 @@ def get_ib_interface_hwaddr(ifname, ethernet_format):
 
 
 def get_interfaces_by_mac():
+    if util.is_FreeBSD():
+        return get_interfaces_by_mac_on_freebsd()
+    else:
+        return get_interfaces_by_mac_on_linux()
+
+
+def get_interfaces_by_mac_on_freebsd():
+    (out, _) = util.subp(['ifconfig', '-a', 'ether'])
+
+    # flatten each interface block in a single line
+    def flatten(out):
+        curr_block = ''
+        for l in out.split('\n'):
+            if l.startswith('\t'):
+                curr_block += l
+            else:
+                if curr_block:
+                    yield curr_block
+                curr_block = l
+        yield curr_block
+
+    # looks for interface and mac in a list of flatten block
+    def find_mac(flat_list):
+        for block in flat_list:
+            m = re.search(
+                r"^(?P<ifname>\S*): .*ether\s(?P<mac>[\da-f:]{17}).*",
+                block)
+            if m:
+                yield (m.group('mac'), m.group('ifname'))
+    results = {mac: ifname for mac, ifname in find_mac(flatten(out))}
+    return results
+
+
+def get_interfaces_by_mac_on_linux():
     """Build a dictionary of tuples {mac: name}.
 
     Bridges and any devices that have a 'stolen' mac are excluded."""
diff --git a/cloudinit/net/freebsd.py b/cloudinit/net/freebsd.py
new file mode 100644
index 00000000..d6f61da3
--- /dev/null
+++ b/cloudinit/net/freebsd.py
@@ -0,0 +1,175 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+import re
+
+from cloudinit import log as logging
+from cloudinit import net
+from cloudinit import util
+from cloudinit.distros import rhel_util
+from cloudinit.distros.parsers.resolv_conf import ResolvConf
+
+from . import renderer
+
+LOG = logging.getLogger(__name__)
+
+
+class Renderer(renderer.Renderer):
+    resolv_conf_fn = 'etc/resolv.conf'
+    rc_conf_fn = 'etc/rc.conf'
+
+    def __init__(self, config=None):
+        if not config:
+            config = {}
+        self.dhcp_interfaces = []
+        self._postcmds = config.get('postcmds', True)
+
+    def _update_rc_conf(self, settings, target=None):
+        fn = util.target_path(target, self.rc_conf_fn)
+        rhel_util.update_sysconfig_file(fn, settings)
+
+    def _write_ifconfig_entries(self, settings, target=None):
+        ifname_by_mac = net.get_interfaces_by_mac()
+        for interface in settings.iter_interfaces():
+            device_name = interface.get("name")
+            device_mac = interface.get("mac_address")
+            if device_name and re.match(r'^lo\d+$', device_name):
+                continue
+            if device_mac not in ifname_by_mac:
+                LOG.info('Cannot find any device with MAC %s', device_mac)
+            elif device_mac and device_name:
+                cur_name = ifname_by_mac[device_mac]
+                if cur_name != device_name:
+                    LOG.info('netif service will rename interface %s to %s',
+                             cur_name, device_name)
+                    self._update_rc_conf(
+                        {'ifconfig_%s_name' % cur_name: device_name},
+                        target=target)
+            else:
+                device_name = ifname_by_mac[device_mac]
+
+            LOG.info('Configuring interface %s', device_name)
+            ifconfig = 'DHCP'  # default
+
+            for subnet in interface.get("subnets", []):
+                if ifconfig != 'DHCP':
+                    LOG.info('The FreeBSD provider only set the first subnet.')
+                    break
+                if subnet.get('type') == 'static':
+                    if not subnet.get('netmask'):
+                        LOG.debug(
+                                'Skipping IP %s, because there is no netmask',
+                                subnet.get('address'))
+                        continue
+                    LOG.debug('Configuring dev %s with %s / %s', device_name,
+                              subnet.get('address'), subnet.get('netmask'))
+                # Configure an ipv4 address.
+                    ifconfig = (
+                            subnet.get('address') + ' netmask ' +
+                            subnet.get('netmask'))
+
+            if ifconfig == 'DHCP':
+                self.dhcp_interfaces.append(device_name)
+            self._update_rc_conf(
+                {'ifconfig_' + device_name: ifconfig},
+                target=target)
+
+    def _write_route_entries(self, settings, target=None):
+        routes = list(settings.iter_routes())
+        for interface in settings.iter_interfaces():
+            subnets = interface.get("subnets", [])
+            for subnet in subnets:
+                if subnet.get('type') != 'static':
+                    continue
+                gateway = subnet.get('gateway')
+                if gateway and len(gateway.split('.')) == 4:
+                    routes.append({
+                        'network': '0.0.0.0',
+                        'netmask': '0.0.0.0',
+                        'gateway': gateway})
+                routes += subnet.get('routes', [])
+        route_cpt = 0
+        for route in routes:
+            network = route.get('network')
+            if not network:
+                LOG.debug('Skipping a bad route entry')
+                continue
+            netmask = route.get('netmask')
+            gateway = route.get('gateway')
+            route_cmd = "-route %s/%s %s" % (network, netmask, gateway)
+            if network == '0.0.0.0':
+                self._update_rc_conf(
+                    {'defaultrouter': gateway}, target=target)
+            else:
+                self._update_rc_conf(
+                    {'route_net%d' % route_cpt: route_cmd}, target=target)
+                route_cpt += 1
+
+    def _write_resolve_conf(self, settings, target=None):
+        nameservers = settings.dns_nameservers
+        searchdomains = settings.dns_searchdomains
+        for interface in settings.iter_interfaces():
+            for subnet in interface.get("subnets", []):
+                if 'dns_nameservers' in subnet:
+                    nameservers.extend(subnet['dns_nameservers'])
+                if 'dns_search' in subnet:
+                    searchdomains.extend(subnet['dns_search'])
+        # Try to read the /etc/resolv.conf or just start from scratch if that
+        # fails.
+        try:
+            resolvconf = ResolvConf(util.load_file(util.target_path(
+                target, self.resolv_conf_fn)))
+            resolvconf.parse()
+        except IOError:
+            util.logexc(LOG, "Failed to parse %s, use new empty file",
+                        util.target_path(target, self.resolv_conf_fn))
+            resolvconf = ResolvConf('')
+            resolvconf.parse()
+
+        # Add some nameservers
+        for server in nameservers:
+            try:
+                resolvconf.add_nameserver(server)
+            except ValueError:
+                util.logexc(LOG, "Failed to add nameserver %s", server)
+
+        # And add any searchdomains.
+        for domain in searchdomains:
+            try:
+                resolvconf.add_search_domain(domain)
+            except ValueError:
+                util.logexc(LOG, "Failed to add search domain %s", domain)
+        util.write_file(
+            util.target_path(target, self.resolv_conf_fn),
+            str(resolvconf), 0o644)
+
+    def _write_network(self, settings, target=None):
+        self._write_ifconfig_entries(settings, target=target)
+        self._write_route_entries(settings, target=target)
+        self._write_resolve_conf(settings, target=target)
+
+        self.start_services(run=self._postcmds)
+
+    def render_network_state(self, network_state, templates=None, target=None):
+        self._write_network(network_state, target=target)
+
+    def start_services(self, run=False):
+        if not run:
+            LOG.debug("freebsd generate postcmd disabled")
+            return
+
+        util.subp(['service', 'netif', 'restart'], capture=True)
+        # On FreeBSD 10, the restart of routing and dhclient is likely to fail
+        # because
+        # - routing: it cannot remove the loopback route, but it will still set
+        #   up the default route as expected.
+        # - dhclient: it cannot stop the dhclient started by the netif service.
+        # In both case, the situation is ok, and we can proceed.
+        util.subp(['service', 'routing', 'restart'], capture=True, rcs=[0, 1])
+        for dhcp_interface in self.dhcp_interfaces:
+            util.subp(['service', 'dhclient', 'restart', dhcp_interface],
+                      rcs=[0, 1],
+                      capture=True)
+
+
+def available(target=None):
+    return util.is_FreeBSD()
diff --git a/cloudinit/net/renderers.py b/cloudinit/net/renderers.py
index 5117b4a5..b98dbbe3 100644
--- a/cloudinit/net/renderers.py
+++ b/cloudinit/net/renderers.py
@@ -1,17 +1,19 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
 from . import eni
+from . import freebsd
 from . import netplan
 from . import RendererNotFoundError
 from . import sysconfig
 
 NAME_TO_RENDERER = {
     "eni": eni,
+    "freebsd": freebsd,
     "netplan": netplan,
     "sysconfig": sysconfig,
 }
 
-DEFAULT_PRIORITY = ["eni", "sysconfig", "netplan"]
+DEFAULT_PRIORITY = ["eni", "sysconfig", "netplan", "freebsd"]
 
 
 def search(priority=None, target=None, first=False):