commit to push for fear of loss.

== background ==
DataSource Mode (dsmode) is present in many datasources in cloud-init.
dsmode was originally added to cloud-init to specify when this datasource
should be 'realized'.

cloud-init has 4 stages of boot.
 a.) cloud-init --local . network is guaranteed not present.
 b.) cloud-init (--network). network is guaranteed present.
 c.) cloud-config
 d.) cloud-init final

'init_modules' [1] are run "as early as possible".  And as such, are executed
in either 'a' or 'b' based on the datasource.  However, executing them means
that user-data has been fully consumed.  User-data and vendor-data may have
'#include http://...' which then rely on the network being present.  boothooks
are an example of the things run in init_modules.

The 'dsmode' was a way for a user to indicate that init_modules
should run at 'a' (dsmode=local) or 'b' (dsmode=net) directly.

Things were further confused when a datasource could provide networking
configuration.  Then, we needed to apply the networking config at 'a'
but if the user had provided boothooks that expected networking, then the
init_modules would need to be executed at 'b'.  The config drive datasource
hacked its way through this and applies networking if *it* detects it is
a new instance.

== Suggested Change ==
The plan is to 
 1. incorporate 'dsmode' into DataSource superclass
 2. make all existing datasources default to network
 3. apply any networking configuration from a datasource on first boot only
    apply_networking will always rename network devices when it runs.
    for bug 1579130.
 4. run init_modules at cloud-init (network) time frame unless datasource
    is 'local'.
 5. Datasources can provide a 'first_boot' method that will be called when
    a new instance_id is found.  This will allow the config drive's write_files
    to be applied once.

Over all, this will very much simplify things.  We'll no longer have
2 sources like DataSourceNoCloud and DataSourceNoCloudNet, but would just
have one source with a dsmode.

== Concerns ==
Some things have odd reliance on dsmode.  For example, OpenNebula's get_hostname
uses it to determine if it should do a lookup of an ip address.

== Bugs to fix here ==
http://pad.lv/1577982 ConfigDrive: cloud-init fails to configure network from network_data.json
http://pad.lv/1579130 need to support systemd.link renaming of devices in container 
http://pad.lv/1577844 Drop unnecessary blocking of all net udev rules 

1 files changed, 34 insertions, 44 deletions

diff --git a/cloudinit/sources/DataSourceNoCloud.py b/cloudinit/sources/DataSourceNoCloud.py
index 48c61a90..7e30118c 100644
--- a/cloudinit/sources/DataSourceNoCloud.py
+++ b/cloudinit/sources/DataSourceNoCloud.py
@@ -24,6 +24,7 @@ import errno
 import os
 
 from cloudinit import log as logging
+from cloudinit import net
 from cloudinit import sources
 from cloudinit import util
 
@@ -35,7 +36,6 @@ class DataSourceNoCloud(sources.DataSource):
         sources.DataSource.__init__(self, sys_cfg, distro, paths)
         self.dsmode = 'local'
         self.seed = None
-        self.cmdline_id = "ds=nocloud"
         self.seed_dirs = [os.path.join(paths.seed_dir, 'nocloud'),
                           os.path.join(paths.seed_dir, 'nocloud-net')]
         self.seed_dir = None
@@ -58,7 +58,7 @@ class DataSourceNoCloud(sources.DataSource):
         try:
             # Parse the kernel command line, getting data passed in
             md = {}
-            if parse_cmdline_data(self.cmdline_id, md):
+            if load_cmdline_data(md):
                 found.append("cmdline")
                 mydata = _merge_new_seed(mydata, {'meta-data': md})
         except Exception:
@@ -123,12 +123,6 @@ class DataSourceNoCloud(sources.DataSource):
 
                     mydata = _merge_new_seed(mydata, seeded)
 
-                    # For seed from a device, the default mode is 'net'.
-                    # that is more likely to be what is desired.  If they want
-                    # dsmode of local, then they must specify that.
-                    if 'dsmode' not in mydata['meta-data']:
-                        mydata['meta-data']['dsmode'] = "net"
-
                     LOG.debug("Using data from %s", dev)
                     found.append(dev)
                     break
@@ -144,7 +138,6 @@ class DataSourceNoCloud(sources.DataSource):
         if len(found) == 0:
             return False
 
-        seeded_network = None
         # The special argument "seedfrom" indicates we should
         # attempt to seed the userdata / metadata from its value
         # its primarily value is in allowing the user to type less
@@ -160,10 +153,6 @@ class DataSourceNoCloud(sources.DataSource):
                 LOG.debug("Seed from %s not supported by %s", seedfrom, self)
                 return False
 
-            if (mydata['meta-data'].get('network-interfaces') or
-                    mydata.get('network-config')):
-                seeded_network = self.dsmode
-
             # This could throw errors, but the user told us to do it
             # so if errors are raised, let them raise
             (md_seed, ud) = util.read_seeded(seedfrom, timeout=None)
@@ -179,35 +168,21 @@ class DataSourceNoCloud(sources.DataSource):
         mydata['meta-data'] = util.mergemanydict([mydata['meta-data'],
                                                   defaults])
 
-        netdata = {'format': None, 'data': None}
-        if mydata['meta-data'].get('network-interfaces'):
-            netdata['format'] = 'interfaces'
-            netdata['data'] = mydata['meta-data']['network-interfaces']
-        elif mydata.get('network-config'):
-            netdata['format'] = 'network-config'
-            netdata['data'] = mydata['network-config']
-
-        # if this is the local datasource or 'seedfrom' was used
-        # and the source of the seed was self.dsmode.
-        # Then see if there is network config to apply.
-        # note this is obsolete network-interfaces style seeding.
-        if self.dsmode in ("local", seeded_network):
-            if mydata['meta-data'].get('network-interfaces'):
-                LOG.debug("Updating network interfaces from %s", self)
-                self.distro.apply_network(
-                    mydata['meta-data']['network-interfaces'])
-
-        if mydata['meta-data']['dsmode'] == self.dsmode:
-            self.seed = ",".join(found)
-            self.metadata = mydata['meta-data']
-            self.userdata_raw = mydata['user-data']
-            self.vendordata_raw = mydata['vendor-data']
-            self._network_config = mydata['network-config']
-            return True
+        self.dsmode = self._determine_dsmode(
+            [mydata['meta-data'].get('dsmode')])
 
-        LOG.debug("%s: not claiming datasource, dsmode=%s", self,
-                  mydata['meta-data']['dsmode'])
-        return False
+        if self.dsmode == sources.DSMODE_DISABLED:
+            LOG.debug("%s: not claiming datasource, dsmode=%s", self,
+                      self.dsmode)
+            return False
+
+        self.seed = ",".join(found)
+        self.metadata = mydata['meta-data']
+        self.userdata_raw = mydata['user-data']
+        self.vendordata_raw = mydata['vendor-data']
+        self._network_config = mydata['network-config']
+        self._network_eni = mydata['meta-data'].get('network-interfaces')
+        return True
 
     def check_instance_id(self, sys_cfg):
         # quickly (local check only) if self.instance_id is still valid
@@ -227,6 +202,9 @@ class DataSourceNoCloud(sources.DataSource):
 
     @property
     def network_config(self):
+        if self._network_config is None:
+            if self.network_eni is not None:
+                self._network_config = net.convert_eni_data(self.network_eni)
         return self._network_config
 
 
@@ -254,8 +232,22 @@ def _quick_read_instance_id(cmdline_id, dirs=None):
     return None
 
 
+def load_cmdline_data(fill, cmdline=None):
+    pairs = [("ds=nocloud", sources.DSMODE_LOCAL),
+             ("ds=nocloud-net", sources.DSMODE_NETWORK)]
+    for idstr, dsmode in pairs:
+        if parse_cmdline_data(idstr, fill, cmdline):
+            # if dsmode was explicitly in the commanad line, then
+            # prefer it to the dsmode based on the command line id
+            if 'dsmode' not in fill:
+                fill['dsmode'] = dsmode
+            return True
+    return False
+
+
 # Returns true or false indicating if cmdline indicated
-# that this module should be used
+# that this module should be used.  Updates dictionary 'fill'
+# with data that was found.
 # Example cmdline:
 #  root=LABEL=uec-rootfs ro ds=nocloud
 def parse_cmdline_data(ds_id, fill, cmdline=None):
@@ -319,9 +311,7 @@ def _merge_new_seed(cur, seeded):
 class DataSourceNoCloudNet(DataSourceNoCloud):
     def __init__(self, sys_cfg, distro, paths):
         DataSourceNoCloud.__init__(self, sys_cfg, distro, paths)
-        self.cmdline_id = "ds=nocloud-net"
         self.supported_seed_starts = ("http://", "https://", "ftp://")
-        self.dsmode = "net"
 
 
 # Used to match classes to dependencies