freebsd: introduce the freebsd renderer (#61)

* freebsd: introduce the freebsd renderer

Refactoring of the FreeBSD code base to provide a real network renderer
for FreeBSD.
Use the generic update_sysconfig_file() from rhel_util to handle the
access to /etc/rc.conf.
Interfaces are not automatically renamed by FreeBSD using
the following configuration in /etc/rc.conf:

```
ifconfig_fxp0_name="eth0"
```

* freesd: use regex named groups

Reduce the complexity of `get_interfaces_by_mac_on_freebsd()` with
named groups.

* freebsd: breaks up _write_network() in tree small functions

- `_write_ifconfig_entries()`
- `_write_route_entries()`
- `_write_resolve_conf()`

* extend find_fallback_nic() to support FreeBSD

this uses `route -n show default` to find the default interface

* freebsd: use dns keys from NetworkState class

The NetworkState class (settings instance) exposes the DNS configuration
in two keys:

- `dns_nameservers`
- `dns_searchdomains`

On OpenStack, these keys are set when a global DNS server is set. The
alternative is the `dns_nameservers` and `dns_search` keys from each
subdomain. We continue to read those.

* freebsd: properly target the /etc/resolv.conf file

* freebsd: ignore 'service routing restart' ret code

On FreeBSD 10, the restart of routing and dhclient is likely to fail because
- routing: it cannot remove the loopback route, but it will still set up
  the default route as expected.
- dhclient: it cannot stop the dhclient started by the netif service.
In both case, the situation is ok, and we can proceed.

* freebsd: handle case when metadata MAC local locally

Handle the case where the metadata configuration comes with a MAC that
does not exist locally.

See:
- https://github.com/canonical/cloud-init/pull/61/files/635ce14b3153934ba1041be48b7245062f21e960#r359600604
- https://github.com/canonical/cloud-init/pull/61/files/635ce14b3153934ba1041be48b7245062f21e960#r359600966

* freebsd: show up a warning if several subnet found

The FreeBSD provider currently only allow one subnet per interface.

* freebsd: honor the target parameter in _write_network

* freebsd: log when a bad route is found

* freebsd: pass _postcmds to start_services()

* freebsd: updatercconf() is depercated

Replace `updatercconf()` by `rhel_util.update_sysconfig_file()`.

* freebsd: ensure gateway is ipv4 before using it

With the legacy ENI format, an IPv6 gateway may be pushed. This instead
of the expected IPv4.

* freebsd: find_fallback_nic, support FB10

On FreeBSD <= 10, `ifconfig -l` ignores the down interfaces.

* freebsd: use util.target_path() to load resolv.conf

Ensure we access `/etc/resolv.conf`, not `etc/resolv.conf`.

* freebsd: skip subnet without netmask

Those are likely to be either invalid of in IPv6 format. IPv6 support
will be addressed later in a new patchset.

* freebsd: get_devicelist returns netif list

Ensure `get_devicelist()` returns the list of known netif on FreeBSD.

* replace rhel_util.update_sysconfig_file wrapper call, with a wrapper function

* reverse if condition to remove an indent

Co-authored-by: Igor Galić <me+github@igalic.co>

2 files changed, 105 insertions, 117 deletions

diff --git a/tests/unittests/test_distros/test_netconfig.py b/tests/unittests/test_distros/test_netconfig.py
index 67209955..a1611a07 100644
--- a/tests/unittests/test_distros/test_netconfig.py
+++ b/tests/unittests/test_distros/test_netconfig.py
@@ -1,5 +1,6 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
+import copy
 import os
 from six import StringIO
 from textwrap import dedent
@@ -14,7 +15,7 @@ from cloudinit.distros.parsers.sys_conf import SysConf
 from cloudinit import helpers
 from cloudinit import settings
 from cloudinit.tests.helpers import (
-    FilesystemMockingTestCase, dir2dict, populate_dir)
+    FilesystemMockingTestCase, dir2dict)
 from cloudinit import util
 
 
@@ -213,128 +214,95 @@ class TestNetCfgDistroBase(FilesystemMockingTestCase):
             self.assertEqual(v, b2[k])
 
 
-class TestNetCfgDistroFreebsd(TestNetCfgDistroBase):
+class TestNetCfgDistroFreeBSD(TestNetCfgDistroBase):
 
-    frbsd_ifout = """\
-hn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
-        options=51b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,LRO>
-        ether 00:15:5d:4c:73:00
-        inet6 fe80::215:5dff:fe4c:7300%hn0 prefixlen 64 scopeid 0x2
-        inet 10.156.76.127 netmask 0xfffffc00 broadcast 10.156.79.255
-        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
-        media: Ethernet autoselect (10Gbase-T <full-duplex>)
-        status: active
+    def setUp(self):
+        super(TestNetCfgDistroFreeBSD, self).setUp()
+        self.distro = self._get_distro('freebsd', renderers=['freebsd'])
+
+    def _apply_and_verify_freebsd(self, apply_fn, config, expected_cfgs=None,
+                                  bringup=False):
+        if not expected_cfgs:
+            raise ValueError('expected_cfg must not be None')
+
+        tmpd = None
+        with mock.patch('cloudinit.net.freebsd.available') as m_avail:
+            m_avail.return_value = True
+            with self.reRooted(tmpd) as tmpd:
+                util.ensure_dir('/etc')
+                util.ensure_file('/etc/rc.conf')
+                util.ensure_file('/etc/resolv.conf')
+                apply_fn(config, bringup)
+
+        results = dir2dict(tmpd)
+        for cfgpath, expected in expected_cfgs.items():
+            print("----------")
+            print(expected)
+            print("^^^^ expected | rendered VVVVVVV")
+            print(results[cfgpath])
+            print("----------")
+            self.assertEqual(
+                set(expected.split('\n')),
+                set(results[cfgpath].split('\n')))
+            self.assertEqual(0o644, get_mode(cfgpath, tmpd))
+
+    @mock.patch('cloudinit.net.get_interfaces_by_mac')
+    def test_apply_network_config_freebsd_standard(self, ifaces_mac):
+        ifaces_mac.return_value = {
+            '00:15:5d:4c:73:00': 'eth0',
+        }
+        rc_conf_expected = """\
+defaultrouter=192.168.1.254
+ifconfig_eth0='192.168.1.5 netmask 255.255.255.0'
+ifconfig_eth1=DHCP
 """
 
-    @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_list')
-    @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_ifname_out')
-    def test_get_ip_nic_freebsd(self, ifname_out, iflist):
-        frbsd_distro = self._get_distro('freebsd')
-        iflist.return_value = "lo0 hn0"
-        ifname_out.return_value = self.frbsd_ifout
-        res = frbsd_distro.get_ipv4()
-        self.assertEqual(res, ['lo0', 'hn0'])
-        res = frbsd_distro.get_ipv6()
-        self.assertEqual(res, [])
-
-    @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_ether')
-    @mock.patch('cloudinit.distros.freebsd.Distro.get_ifconfig_ifname_out')
-    @mock.patch('cloudinit.distros.freebsd.Distro.get_interface_mac')
-    def test_generate_fallback_config_freebsd(self, mac, ifname_out, if_ether):
-        frbsd_distro = self._get_distro('freebsd')
-
-        if_ether.return_value = 'hn0'
-        ifname_out.return_value = self.frbsd_ifout
-        mac.return_value = '00:15:5d:4c:73:00'
-        res = frbsd_distro.generate_fallback_config()
-        self.assertIsNotNone(res)
-
-    def test_simple_write_freebsd(self):
-        fbsd_distro = self._get_distro('freebsd')
-
-        rc_conf = '/etc/rc.conf'
-        read_bufs = {
-            rc_conf: 'initial-rc-conf-not-validated',
-            '/etc/resolv.conf': 'initial-resolv-conf-not-validated',
+        expected_cfgs = {
+            '/etc/rc.conf': rc_conf_expected,
+            '/etc/resolv.conf': ''
         }
+        self._apply_and_verify_freebsd(self.distro.apply_network_config,
+                                       V1_NET_CFG,
+                                       expected_cfgs=expected_cfgs.copy())
 
-        tmpd = self.tmp_dir()
-        populate_dir(tmpd, read_bufs)
-        with self.reRooted(tmpd):
-            with mock.patch("cloudinit.distros.freebsd.util.subp",
-                            return_value=('vtnet0', '')):
-                fbsd_distro.apply_network(BASE_NET_CFG, False)
-                results = dir2dict(tmpd)
-
-        self.assertIn(rc_conf, results)
-        self.assertCfgEquals(
-            dedent('''\
-                ifconfig_vtnet0="192.168.1.5 netmask 255.255.255.0"
-                ifconfig_vtnet1="DHCP"
-                defaultrouter="192.168.1.254"
-                '''), results[rc_conf])
-        self.assertEqual(0o644, get_mode(rc_conf, tmpd))
-
-    def test_simple_write_freebsd_from_v2eni(self):
-        fbsd_distro = self._get_distro('freebsd')
-
-        rc_conf = '/etc/rc.conf'
-        read_bufs = {
-            rc_conf: 'initial-rc-conf-not-validated',
-            '/etc/resolv.conf': 'initial-resolv-conf-not-validated',
+    @mock.patch('cloudinit.net.get_interfaces_by_mac')
+    def test_apply_network_config_freebsd_ifrename(self, ifaces_mac):
+        ifaces_mac.return_value = {
+            '00:15:5d:4c:73:00': 'vtnet0',
         }
+        rc_conf_expected = """\
+ifconfig_vtnet0_name=eth0
+defaultrouter=192.168.1.254
+ifconfig_eth0='192.168.1.5 netmask 255.255.255.0'
+ifconfig_eth1=DHCP
+"""
 
-        tmpd = self.tmp_dir()
-        populate_dir(tmpd, read_bufs)
-        with self.reRooted(tmpd):
-            with mock.patch("cloudinit.distros.freebsd.util.subp",
-                            return_value=('vtnet0', '')):
-                fbsd_distro.apply_network(BASE_NET_CFG_FROM_V2, False)
-                results = dir2dict(tmpd)
-
-        self.assertIn(rc_conf, results)
-        self.assertCfgEquals(
-            dedent('''\
-                ifconfig_vtnet0="192.168.1.5 netmask 255.255.255.0"
-                ifconfig_vtnet1="DHCP"
-                defaultrouter="192.168.1.254"
-                '''), results[rc_conf])
-        self.assertEqual(0o644, get_mode(rc_conf, tmpd))
-
-    def test_apply_network_config_fallback_freebsd(self):
-        fbsd_distro = self._get_distro('freebsd')
-
-        # a weak attempt to verify that we don't have an implementation
-        # of _write_network_config or apply_network_config in fbsd now,
-        # which would make this test not actually test the fallback.
-        self.assertRaises(
-            NotImplementedError, fbsd_distro._write_network_config,
-            BASE_NET_CFG)
-
-        # now run
-        mynetcfg = {
-            'config': [{"type": "physical", "name": "eth0",
-                        "mac_address": "c0:d6:9f:2c:e8:80",
-                        "subnets": [{"type": "dhcp"}]}],
-            'version': 1}
-
-        rc_conf = '/etc/rc.conf'
-        read_bufs = {
-            rc_conf: 'initial-rc-conf-not-validated',
-            '/etc/resolv.conf': 'initial-resolv-conf-not-validated',
+        V1_NET_CFG_RENAME = copy.deepcopy(V1_NET_CFG)
+        V1_NET_CFG_RENAME['config'][0]['mac_address'] = '00:15:5d:4c:73:00'
+
+        expected_cfgs = {
+            '/etc/rc.conf': rc_conf_expected,
+            '/etc/resolv.conf': ''
         }
+        self._apply_and_verify_freebsd(self.distro.apply_network_config,
+                                       V1_NET_CFG_RENAME,
+                                       expected_cfgs=expected_cfgs.copy())
 
-        tmpd = self.tmp_dir()
-        populate_dir(tmpd, read_bufs)
-        with self.reRooted(tmpd):
-            with mock.patch("cloudinit.distros.freebsd.util.subp",
-                            return_value=('vtnet0', '')):
-                fbsd_distro.apply_network_config(mynetcfg, bring_up=False)
-                results = dir2dict(tmpd)
+    @mock.patch('cloudinit.net.get_interfaces_by_mac')
+    def test_apply_network_config_freebsd_nameserver(self, ifaces_mac):
+        ifaces_mac.return_value = {
+            '00:15:5d:4c:73:00': 'eth0',
+        }
 
-        self.assertIn(rc_conf, results)
-        self.assertCfgEquals('ifconfig_vtnet0="DHCP"', results[rc_conf])
-        self.assertEqual(0o644, get_mode(rc_conf, tmpd))
+        V1_NET_CFG_DNS = copy.deepcopy(V1_NET_CFG)
+        ns = ['1.2.3.4']
+        V1_NET_CFG_DNS['config'][0]['subnets'][0]['dns_nameservers'] = ns
+        expected_cfgs = {
+            '/etc/resolv.conf': 'nameserver 1.2.3.4\n'
+        }
+        self._apply_and_verify_freebsd(self.distro.apply_network_config,
+                                       V1_NET_CFG_DNS,
+                                       expected_cfgs=expected_cfgs.copy())
 
 
 class TestNetCfgDistroUbuntuEni(TestNetCfgDistroBase):
@@ -694,10 +662,11 @@ class TestNetCfgDistroArch(TestNetCfgDistroBase):
                 """),
         }
 
-        self._apply_and_verify(self.distro.apply_network_config,
-                               V1_NET_CFG,
-                               expected_cfgs=expected_cfgs.copy(),
-                               with_netplan=True)
+        with mock.patch('cloudinit.util.is_FreeBSD', return_value=False):
+            self._apply_and_verify(self.distro.apply_network_config,
+                                   V1_NET_CFG,
+                                   expected_cfgs=expected_cfgs.copy(),
+                                   with_netplan=True)
 
 
 def get_mode(path, target=None):
diff --git a/tests/unittests/test_net_freebsd.py b/tests/unittests/test_net_freebsd.py
new file mode 100644
index 00000000..48296c30
--- /dev/null
+++ b/tests/unittests/test_net_freebsd.py
@@ -0,0 +1,19 @@
+from cloudinit import net
+
+from cloudinit.tests.helpers import (CiTestCase, mock, readResource)
+
+SAMPLE_FREEBSD_IFCONFIG_OUT = readResource("netinfo/freebsd-ifconfig-output")
+
+
+class TestInterfacesByMac(CiTestCase):
+
+    @mock.patch('cloudinit.util.subp')
+    @mock.patch('cloudinit.util.is_FreeBSD')
+    def test_get_interfaces_by_mac(self, mock_is_FreeBSD, mock_subp):
+        mock_is_FreeBSD.return_value = True
+        mock_subp.return_value = (SAMPLE_FREEBSD_IFCONFIG_OUT, 0)
+        a = net.get_interfaces_by_mac()
+        assert a == {'52:54:00:50:b7:0d': 'vtnet0',
+                     '80:00:73:63:5c:48': 're0.33',
+                     '02:14:39:0e:25:00': 'bridge0',
+                     '02:ff:60:8c:f3:72': 'vnet0:11'}