commit to push for fear of loss.

== background ==
DataSource Mode (dsmode) is present in many datasources in cloud-init.
dsmode was originally added to cloud-init to specify when this datasource
should be 'realized'.

cloud-init has 4 stages of boot.
 a.) cloud-init --local . network is guaranteed not present.
 b.) cloud-init (--network). network is guaranteed present.
 c.) cloud-config
 d.) cloud-init final

'init_modules' [1] are run "as early as possible".  And as such, are executed
in either 'a' or 'b' based on the datasource.  However, executing them means
that user-data has been fully consumed.  User-data and vendor-data may have
'#include http://...' which then rely on the network being present.  boothooks
are an example of the things run in init_modules.

The 'dsmode' was a way for a user to indicate that init_modules
should run at 'a' (dsmode=local) or 'b' (dsmode=net) directly.

Things were further confused when a datasource could provide networking
configuration.  Then, we needed to apply the networking config at 'a'
but if the user had provided boothooks that expected networking, then the
init_modules would need to be executed at 'b'.  The config drive datasource
hacked its way through this and applies networking if *it* detects it is
a new instance.

== Suggested Change ==
The plan is to 
 1. incorporate 'dsmode' into DataSource superclass
 2. make all existing datasources default to network
 3. apply any networking configuration from a datasource on first boot only
    apply_networking will always rename network devices when it runs.
    for bug 1579130.
 4. run init_modules at cloud-init (network) time frame unless datasource
    is 'local'.
 5. Datasources can provide a 'first_boot' method that will be called when
    a new instance_id is found.  This will allow the config drive's write_files
    to be applied once.

Over all, this will very much simplify things.  We'll no longer have
2 sources like DataSourceNoCloud and DataSourceNoCloudNet, but would just
have one source with a dsmode.

== Concerns ==
Some things have odd reliance on dsmode.  For example, OpenNebula's get_hostname
uses it to determine if it should do a lookup of an ip address.

== Bugs to fix here ==
http://pad.lv/1577982 ConfigDrive: cloud-init fails to configure network from network_data.json
http://pad.lv/1579130 need to support systemd.link renaming of devices in container 
http://pad.lv/1577844 Drop unnecessary blocking of all net udev rules 

1 files changed, 23 insertions, 67 deletions

diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py
index 52a9f543..20df5fcd 100644
--- a/cloudinit/sources/DataSourceConfigDrive.py
+++ b/cloudinit/sources/DataSourceConfigDrive.py
@@ -22,6 +22,7 @@ import copy
 import os
 
 from cloudinit import log as logging
+from cloudinit import net
 from cloudinit import sources
 from cloudinit import util
 
@@ -35,7 +36,6 @@ DEFAULT_MODE = 'pass'
 DEFAULT_METADATA = {
     "instance-id": DEFAULT_IID,
 }
-VALID_DSMODES = ("local", "net", "pass", "disabled")
 FS_TYPES = ('vfat', 'iso9660')
 LABEL_TYPES = ('config-2',)
 POSSIBLE_MOUNTS = ('sr', 'cd')
@@ -47,12 +47,12 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
     def __init__(self, sys_cfg, distro, paths):
         super(DataSourceConfigDrive, self).__init__(sys_cfg, distro, paths)
         self.source = None
-        self.dsmode = 'local'
         self.seed_dir = os.path.join(paths.seed_dir, 'config_drive')
         self.version = None
         self.ec2_metadata = None
         self._network_config = None
         self.network_json = None
+        self.network_eni = None
         self.files = {}
 
     def __str__(self):
@@ -98,38 +98,22 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
 
         md = results.get('metadata', {})
         md = util.mergemanydict([md, DEFAULT_METADATA])
-        user_dsmode = results.get('dsmode', None)
-        if user_dsmode not in VALID_DSMODES + (None,):
-            LOG.warn("User specified invalid mode: %s", user_dsmode)
-            user_dsmode = None
 
-        dsmode = get_ds_mode(cfgdrv_ver=results['version'],
-                             ds_cfg=self.ds_cfg.get('dsmode'),
-                             user=user_dsmode)
+        self.dsmode = self._determine_dsmode(
+            [results.get('dsmode'), self.ds_cfg.get('dsmode'),
+             sources.DSMODE_PASS if results['version'] == 1 else None])
 
-        if dsmode == "disabled":
-            # most likely user specified
+        if self.dsmode == sources.DSMODE_DISABLED:
             return False
 
-        # TODO(smoser): fix this, its dirty.
-        # we want to do some things (writing files and network config)
-        # only on first boot, and even then, we want to do so in the
-        # local datasource (so they happen earlier) even if the configured
-        # dsmode is 'net' or 'pass'. To do this, we check the previous
-        # instance-id
+        # This is legacy and sneaky.  If dsmode is 'pass' then write
+        # 'injected files' and apply legacy ENI network format.
         prev_iid = get_previous_iid(self.paths)
         cur_iid = md['instance-id']
-        if prev_iid != cur_iid and self.dsmode == "local":
+        if prev_iid != cur_iid and self.dsmode == sources.DSMODE_PASS:
             on_first_boot(results, distro=self.distro)
-
-        # dsmode != self.dsmode here if:
-        #  * dsmode = "pass",  pass means it should only copy files and then
-        #    pass to another datasource
-        #  * dsmode = "net" and self.dsmode = "local"
-        #    so that user boothooks would be applied with network, the
-        #    local datasource just gets out of the way, and lets the net claim
-        if dsmode != self.dsmode:
-            LOG.debug("%s: not claiming datasource, dsmode=%s", self, dsmode)
+            LOG.debug("%s: not claiming datasource, dsmode=%s", self,
+                      self.dsmode)
             return False
 
         self.source = found
@@ -147,12 +131,11 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
             LOG.warn("Invalid content in vendor-data: %s", e)
             self.vendordata_raw = None
 
-        try:
-            self.network_json = results.get('networkdata')
-        except ValueError as e:
-            LOG.warn("Invalid content in network-data: %s", e)
-            self.network_json = None
-
+        # network_config is an /etc/network/interfaces formated file and is
+        # obsolete compared to networkdata (from network_data.json) but both
+        # might be present.
+        self.network_eni = results.get("network_config")
+        self.network_json = results.get('networkdata')
         return True
 
     def check_instance_id(self, sys_cfg):
@@ -164,40 +147,11 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
         if self._network_config is None:
             if self.network_json is not None:
                 self._network_config = convert_network_data(self.network_json)
+            elif self.network_eni is not None:
+                self._network_config = net.convert_eni_data(self.network_eni)
         return self._network_config
 
 
-class DataSourceConfigDriveNet(DataSourceConfigDrive):
-    def __init__(self, sys_cfg, distro, paths):
-        DataSourceConfigDrive.__init__(self, sys_cfg, distro, paths)
-        self.dsmode = 'net'
-
-
-def get_ds_mode(cfgdrv_ver, ds_cfg=None, user=None):
-    """Determine what mode should be used.
-    valid values are 'pass', 'disabled', 'local', 'net'
-    """
-    # user passed data trumps everything
-    if user is not None:
-        return user
-
-    if ds_cfg is not None:
-        return ds_cfg
-
-    # at config-drive version 1, the default behavior was pass.  That
-    # meant to not use use it as primary data source, but expect a ec2 metadata
-    # source. for version 2, we default to 'net', which means
-    # the DataSourceConfigDriveNet, would be used.
-    #
-    # this could change in the future.  If there was definitive metadata
-    # that indicated presense of an openstack metadata service, then
-    # we could change to 'pass' by default also. The motivation for that
-    # would be 'cloud-init query' as the web service could be more dynamic
-    if cfgdrv_ver == 1:
-        return "pass"
-    return "net"
-
-
 def read_config_drive(source_dir):
     reader = openstack.ConfigDriveReader(source_dir)
     finders = [
@@ -231,9 +185,12 @@ def on_first_boot(data, distro=None):
                         % (type(data)))
     net_conf = data.get("network_config", '')
     if net_conf and distro:
-        LOG.debug("Updating network interfaces from config drive")
+        LOG.warn("Updating network interfaces from config drive")
         distro.apply_network(net_conf)
-    files = data.get('files', {})
+    write_injected_files(data.get('files'))
+
+
+def write_injected_files(files):
     if files:
         LOG.debug("Writing %s injected files", len(files))
         for (filename, content) in files.items():
@@ -296,7 +253,6 @@ def find_candidate_devs(probe_optical=True):
 # Used to match classes to dependencies
 datasources = [
     (DataSourceConfigDrive, (sources.DEP_FILESYSTEM, )),
-    (DataSourceConfigDriveNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
 ]