ec2: Allow Ec2 to run in init-local using dhclient in a sandbox.

This branch is a prerequisite for IPv6 support in AWS by allowing Ec2
datasource to query the metadata source version 2016-09-02 about whether
or not it needs to configure IPv6 on interfaces. If version 2016-09-02
is not present, fallback to the min_metadata_version of 2009-04-04. The
DataSourceEc2Local not run on FreeBSD because dhclient in doesn't
support the -sf flag allowing us to run dhclient without filesystem
side-effects.

To query AWS' metadata address @ 169.254.169.254, the instance must have
a dhcp-allocated address configured. Configuring IPv4 link-local
addresses result in timeouts from the metadata service. We introduced a
DataSourceEc2Local subclass which will perform a sandboxed dhclient
discovery which obtains an authorized IP address on eth0 and crawl
metadata about full instance network configuration.

Since ec2 IPv6 metadata is not sufficient in itself to tell us all the
ipv6 knownledge we need, it only be used as a boolean to tell us which
nics need IPv6. Cloud-init will then configure desired interfaces to
DHCPv6 versus DHCPv4.

Performance side note: Shifting the dhcp work into init-local for Ec2
actually gets us 1 second faster deployments by skipping init-network
phase of alternate datasource checks because Ec2Local is configured in
an ealier boot stage. In 3 test runs prior to this change: cloud-init
runs were 5.5 seconds, with the change we now average 4.6 seconds.

This efficiency could be even further improved if we avoiding dhcp
discovery in order to talk to the metadata service from an AWS
authorized dhcp address if there were some way to advertize the dhcp
configuration via DMI/SMBIOS or system environment variables.

Inspecting time costs of the dhclient setup/teardown in 3 live runs the
time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
         dhcp discovery + metadata: 0.347 seconds
         metadata alone: 0.271 seconds
test 2: 88 milliseconds
         dhcp discovery + metadata: 0.388 seconds
         metadata alone: 0.300 seconds
test 3: 75 milliseconds
         dhcp discovery + metadata: 0.366 seconds
         metadata alone: 0.291 seconds

LP: #1709772

1 files changed, 119 insertions, 0 deletions

diff --git a/cloudinit/net/dhcp.py b/cloudinit/net/dhcp.py
new file mode 100644
index 00000000..c7febc57
--- /dev/null
+++ b/cloudinit/net/dhcp.py
@@ -0,0 +1,119 @@
+# Copyright (C) 2017 Canonical Ltd.
+#
+# Author: Chad Smith <chad.smith@canonical.com>
+#
+# This file is part of cloud-init. See LICENSE file for license information.
+
+import logging
+import os
+import re
+
+from cloudinit.net import find_fallback_nic, get_devicelist
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+
+class InvalidDHCPLeaseFileError(Exception):
+    """Raised when parsing an empty or invalid dhcp.leases file.
+
+    Current uses are DataSourceAzure and DataSourceEc2 during ephemeral
+    boot to scrape metadata.
+    """
+    pass
+
+
+def maybe_perform_dhcp_discovery(nic=None):
+    """Perform dhcp discovery if nic valid and dhclient command exists.
+
+    If the nic is invalid or undiscoverable or dhclient command is not found,
+    skip dhcp_discovery and return an empty dict.
+
+    @param nic: Name of the network interface we want to run dhclient on.
+    @return: A dict of dhcp options from the dhclient discovery if run,
+        otherwise an empty dict is returned.
+    """
+    if nic is None:
+        nic = find_fallback_nic()
+        if nic is None:
+            LOG.debug(
+                'Skip dhcp_discovery: Unable to find fallback nic.')
+            return {}
+    elif nic not in get_devicelist():
+        LOG.debug(
+            'Skip dhcp_discovery: nic %s not found in get_devicelist.', nic)
+        return {}
+    dhclient_path = util.which('dhclient')
+    if not dhclient_path:
+        LOG.debug('Skip dhclient configuration: No dhclient command found.')
+        return {}
+    with util.tempdir(prefix='cloud-init-dhcp-') as tmpdir:
+        return dhcp_discovery(dhclient_path, nic, tmpdir)
+
+
+def parse_dhcp_lease_file(lease_file):
+    """Parse the given dhcp lease file for the most recent lease.
+
+    Return a dict of dhcp options as key value pairs for the most recent lease
+    block.
+
+    @raises: InvalidDHCPLeaseFileError on empty of unparseable leasefile
+        content.
+    """
+    lease_regex = re.compile(r"lease {(?P<lease>[^}]*)}\n")
+    dhcp_leases = []
+    lease_content = util.load_file(lease_file)
+    if len(lease_content) == 0:
+        raise InvalidDHCPLeaseFileError(
+            'Cannot parse empty dhcp lease file {0}'.format(lease_file))
+    for lease in lease_regex.findall(lease_content):
+        lease_options = []
+        for line in lease.split(';'):
+            # Strip newlines, double-quotes and option prefix
+            line = line.strip().replace('"', '').replace('option ', '')
+            if not line:
+                continue
+            lease_options.append(line.split(' ', 1))
+        dhcp_leases.append(dict(lease_options))
+    if not dhcp_leases:
+        raise InvalidDHCPLeaseFileError(
+            'Cannot parse dhcp lease file {0}. No leases found'.format(
+                lease_file))
+    return dhcp_leases
+
+
+def dhcp_discovery(dhclient_cmd_path, interface, cleandir):
+    """Run dhclient on the interface without scripts or filesystem artifacts.
+
+    @param dhclient_cmd_path: Full path to the dhclient used.
+    @param interface: Name of the network inteface on which to dhclient.
+    @param cleandir: The directory from which to run dhclient as well as store
+        dhcp leases.
+
+    @return: A dict of dhcp options parsed from the dhcp.leases file or empty
+        dict.
+    """
+    LOG.debug('Performing a dhcp discovery on %s', interface)
+
+    # XXX We copy dhclient out of /sbin/dhclient to avoid dealing with strict
+    # app armor profiles which disallow running dhclient -sf <our-script-file>.
+    # We want to avoid running /sbin/dhclient-script because of side-effects in
+    # /etc/resolv.conf any any other vendor specific scripts in
+    # /etc/dhcp/dhclient*hooks.d.
+    sandbox_dhclient_cmd = os.path.join(cleandir, 'dhclient')
+    util.copy(dhclient_cmd_path, sandbox_dhclient_cmd)
+    pid_file = os.path.join(cleandir, 'dhclient.pid')
+    lease_file = os.path.join(cleandir, 'dhcp.leases')
+
+    # ISC dhclient needs the interface up to send initial discovery packets.
+    # Generally dhclient relies on dhclient-script PREINIT action to bring the
+    # link up before attempting discovery. Since we are using -sf /bin/true,
+    # we need to do that "link up" ourselves first.
+    util.subp(['ip', 'link', 'set', 'dev', interface, 'up'], capture=True)
+    cmd = [sandbox_dhclient_cmd, '-1', '-v', '-lf', lease_file,
+           '-pf', pid_file, interface, '-sf', '/bin/true']
+    util.subp(cmd, capture=True)
+    return parse_dhcp_lease_file(lease_file)
+
+
+# vi: ts=4 expandtab