diff options
119 files changed, 1659 insertions, 764 deletions
@@ -6,6 +6,37 @@ - SmartOS test: do not require existance of /dev/ttyS1. [LP: #1316597] - doc: fix user-groups doc to reference plural ssh-authorized-keys (LP: #1327065) [Joern Heissler] + - fix 'make test' in python 2.6 + - support jinja2 as a templating engine. Drop the hard requirement on + cheetah. This helps in python3 effort. (LP: #1219223) + - change install path for systemd files to /lib/systemd/system + [Dimitri John Ledkov] + - change trunk debian packaging to use pybuild and drop cdbs. + [Dimitri John Ledkov] + - SeLinuxGuard: remove invalid check that looked for stat.st_mode in os.lstat. + - do not write comments in /etc/timezone (LP: #1341710) + - ubuntu: provide 'ubuntu-init-switch' module to aid in systemd testing. + - status/result json: remove 'end' entry which was always null + - systemd: make cloud-init block ssh service startup to guarantee keys + are generated. [Jordan Evans] (LP: #1333920) + - default settings: fix typo resulting in OpenStack and GCE not working + unless config explicitly provided (LP: #1329583) [Garrett Holmstrom]) + - fix rendering resolv.conf if no 'options' are provided (LP: #1328953) + - docs: fix disk-setup to reference 'table_type' [Rail Aliiev] (LP: #1313114) + - ssh_authkey_fingerprints: fix bug that prevented disabling the module. + (LP: #1340903) [Patrick Lucas] + - no longer use pylint as a checker, fix pep8 [Jay Faulkner]. + - Openstack: do not load some urls twice. + - FreeBsd: fix initscripts and add working config file [Harm Weites] + - Datasource: fix broken logic to provide hostname if datasource does not + provide one + - Improved and less verbose logging. + - resizefs: first check that device is writable. + - configdrive: fix reading of vendor data to be like metadata service reader. + [Jay Faulkner] + - resizefs: fix broken background resizing [Jay Faulkner] (LP: #1338614) + - cc_grub_dpkg: fix EC2 hvm instances to avoid prompt on grub update. + (LP: #1336855) 0.7.5: - open 0.7.5 - Add a debug log message around import failures diff --git a/HACKING.rst b/HACKING.rst index 66bf7c90..6bfe4b4d 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -19,9 +19,9 @@ To get changes into cloud-init, the process to follow is: - ``bzr commit`` -* Check pylint and pep8 and test, and address any issues: +* Check pep8 and test, and address any issues: - - ``make test pylint pep8`` + - ``make test pep8`` * Push to launchpad to a personal branch: @@ -19,9 +19,6 @@ all: test check_version pep8: @$(CWD)/tools/run-pep8 $(PY_FILES) -pylint: - @$(CWD)/tools/run-pylint $(PY_FILES) - pyflakes: pyflakes $(PY_FILES) @@ -61,5 +58,5 @@ rpm: deb: ./packages/bddeb -.PHONY: test pylint pyflakes 2to3 clean pep8 rpm deb yaml check_version +.PHONY: test pyflakes 2to3 clean pep8 rpm deb yaml check_version .PHONY: pip-test-requirements pip-requirements clean_pyc @@ -1,46 +0,0 @@ -- Consider a 'failsafe' DataSource - If all others fail, setting a default that - - sets the user password, writing it to console - - logs to console that this happened -- Consider a 'previous' DataSource - If no other data source is found, fall back to the 'previous' one - keep a indication of what instance id that is in /var/lib/cloud -- Rewrite "cloud-init-query" (currently not implemented) - Possibly have DataSource and cloudinit expose explicit fields - - instance-id - - hostname - - mirror - - release - - ssh public keys -- Remove the conversion of the ubuntu network interface format conversion - to a RH/fedora format and replace it with a top level format that uses - the netcf libraries format instead (which itself knows how to translate - into the specific formats) -- Replace the 'apt*' modules with variants that now use the distro classes - to perform distro independent packaging commands (where possible) -- Canonicalize the semaphore/lock name for modules and user data handlers - a. It is most likely a bug that currently exists that if a module in config - alters its name and it has already ran, then it will get ran again since - the lock name hasn't be canonicalized -- Replace some the LOG.debug calls with a LOG.info where appropriate instead - of how right now there is really only 2 levels (WARN and DEBUG) -- Remove the 'cc_' for config modules, either have them fully specified (ie - 'cloudinit.config.resizefs') or by default only look in the 'cloudinit.config' - for these modules (or have a combination of the above), this avoids having - to understand where your modules are coming from (which can be altered by - the current python inclusion path) -- Depending on if people think the wrapper around 'os.path.join' provided - by the 'paths' object is useful (allowing us to modify based off a 'read' - and 'write' configuration based 'root') or is just to confusing, it might be - something to remove later, and just recommend using 'chroot' instead (or the X - different other options which are similar to 'chroot'), which is might be more - natural and less confusing... -- Instead of just warning when a module is being ran on a 'unknown' distribution - perhaps we should not run that module in that case? Or we might want to start - reworking those modules so they will run on all distributions? Or if that is - not the case, then maybe we want to allow fully specified python paths for - modules and start encouraging packages of 'ubuntu' modules, packages of 'rhel' - specific modules that people can add instead of having them all under the - cloud-init 'root' tree? This might encourage more development of other modules - instead of having to go edit the cloud-init code to accomplish this. - diff --git a/TODO.rst b/TODO.rst new file mode 100644 index 00000000..7d126864 --- /dev/null +++ b/TODO.rst @@ -0,0 +1,43 @@ +============================================== +Things that cloud-init may do (better) someday +============================================== + +- Consider making ``failsafe`` ``DataSource`` + - sets the user password, writing it to console + +- Consider a ``previous`` ``DataSource``, if no other data source is + found, fall back to the ``previous`` one that worked. +- Rewrite ``cloud-init-query`` (currently not implemented) +- Possibly have a ``DataSource`` expose explicit fields: + + - instance-id + - hostname + - mirror + - release + - ssh public keys + +- Remove the conversion of the ubuntu network interface format conversion + to a RH/fedora format and replace it with a top level format that uses + the netcf libraries format instead (which itself knows how to translate + into the specific formats). See for example `netcf`_ which seems to be + an active project that has this capability. +- Replace the ``apt*`` modules with variants that now use the distro classes + to perform distro independent packaging commands (wherever possible). +- Replace some the LOG.debug calls with a LOG.info where appropriate instead + of how right now there is really only 2 levels (``WARN`` and ``DEBUG``) +- Remove the ``cc_`` prefix for config modules, either have them fully + specified (ie ``cloudinit.config.resizefs``) or by default only look in + the ``cloudinit.config`` namespace for these modules (or have a combination + of the above), this avoids having to understand where your modules are + coming from (which can be altered by the current python inclusion path) +- Instead of just warning when a module is being ran on a ``unknown`` + distribution perhaps we should not run that module in that case? Or we might + want to start reworking those modules so they will run on all + distributions? Or if that is not the case, then maybe we want to allow + fully specified python paths for modules and start encouraging + packages of ``ubuntu`` modules, packages of ``rhel`` specific modules that + people can add instead of having them all under the cloud-init ``root`` + tree? This might encourage more development of other modules instead of + having to go edit the cloud-init code to accomplish this. + +.. _netcf: https://fedorahosted.org/netcf/ diff --git a/bin/cloud-init b/bin/cloud-init index 6ede60af..866f8ca4 100755 --- a/bin/cloud-init +++ b/bin/cloud-init @@ -224,6 +224,9 @@ def main_init(name, args): LOG.debug("Exiting early due to the existence of %s files", existing_files) return (None, []) + else: + LOG.debug("Execution continuing, no previous run detected that" + " would allow us to stop early.") else: # The cache is not instance specific, so it has to be purged # but we want 'start' to benefit from a cache if @@ -475,7 +478,7 @@ def status_wrapper(name, args, data_d=None, link_d=None): nullstatus = { 'errors': [], 'start': None, - 'end': None, + 'finished': None, } status = {'v1': {}} for m in modes: diff --git a/cloudinit/config/cc_apt_configure.py b/cloudinit/config/cc_apt_configure.py index 29c13a3d..f10b76a3 100644 --- a/cloudinit/config/cc_apt_configure.py +++ b/cloudinit/config/cc_apt_configure.py @@ -235,7 +235,7 @@ def find_apt_mirror_info(cloud, cfg): mirror = util.search_for_mirror(search) if (not mirror and - util.get_cfg_option_bool(cfg, "apt_mirror_search_dns", False)): + util.get_cfg_option_bool(cfg, "apt_mirror_search_dns", False)): mydom = "" doms = [] diff --git a/cloudinit/config/cc_byobu.py b/cloudinit/config/cc_byobu.py index 92d428b7..ef0ce7ab 100644 --- a/cloudinit/config/cc_byobu.py +++ b/cloudinit/config/cc_byobu.py @@ -43,7 +43,7 @@ def handle(name, cfg, cloud, log, args): valid = ("enable-user", "enable-system", "enable", "disable-user", "disable-system", "disable") - if not value in valid: + if value not in valid: log.warn("Unknown value %s for byobu_by_default", value) mod_user = value.endswith("-user") diff --git a/cloudinit/config/cc_chef.py b/cloudinit/config/cc_chef.py index 727769cd..806deed9 100644 --- a/cloudinit/config/cc_chef.py +++ b/cloudinit/config/cc_chef.py @@ -87,7 +87,8 @@ def handle(name, cfg, cloud, log, _args): # If chef is not installed, we install chef based on 'install_type' if (not os.path.isfile('/usr/bin/chef-client') or - util.get_cfg_option_bool(chef_cfg, 'force_install', default=False)): + util.get_cfg_option_bool(chef_cfg, + 'force_install', default=False)): install_type = util.get_cfg_option_str(chef_cfg, 'install_type', 'packages') diff --git a/cloudinit/config/cc_disk_setup.py b/cloudinit/config/cc_disk_setup.py index 0b970e4e..1660832b 100644 --- a/cloudinit/config/cc_disk_setup.py +++ b/cloudinit/config/cc_disk_setup.py @@ -271,7 +271,7 @@ def find_device_node(device, fs_type=None, label=None, valid_targets=None, return ('/dev/%s' % d['name'], False) if (d['fstype'] == fs_type and - ((label_match and d['label'] == label) or not label_match)): + ((label_match and d['label'] == label) or not label_match)): # If we find a matching device, we return that return ('/dev/%s' % d['name'], True) @@ -447,7 +447,7 @@ def get_partition_mbr_layout(size, layout): return "0," if ((len(layout) == 0 and isinstance(layout, list)) or - not isinstance(layout, list)): + not isinstance(layout, list)): raise Exception("Partition layout is invalid") last_part_num = len(layout) @@ -484,7 +484,7 @@ def get_partition_mbr_layout(size, layout): def purge_disk_ptable(device): # wipe the first and last megabyte of a disk (or file) # gpt stores partition table both at front and at end. - null = '\0' # pylint: disable=W1401 + null = '\0' start_len = 1024 * 1024 end_len = 1024 * 1024 with open(device, "rb+") as fp: diff --git a/cloudinit/config/cc_grub_dpkg.py b/cloudinit/config/cc_grub_dpkg.py index d96e85cf..e3219e81 100644 --- a/cloudinit/config/cc_grub_dpkg.py +++ b/cloudinit/config/cc_grub_dpkg.py @@ -36,7 +36,8 @@ def handle(_name, cfg, _cloud, log, _args): "grub-pc/install_devices_empty", None) if ((os.path.exists("/dev/sda1") and not os.path.exists("/dev/sda")) or - (os.path.exists("/dev/xvda1") and not os.path.exists("/dev/xvda"))): + (os.path.exists("/dev/xvda1") + and not os.path.exists("/dev/xvda"))): if idevs is None: idevs = "" if idevs_empty is None: diff --git a/cloudinit/config/cc_mounts.py b/cloudinit/config/cc_mounts.py index 80590118..ba1303d1 100644 --- a/cloudinit/config/cc_mounts.py +++ b/cloudinit/config/cc_mounts.py @@ -18,7 +18,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -from string import whitespace # pylint: disable=W0402 +from string import whitespace import logging import os.path diff --git a/cloudinit/config/cc_phone_home.py b/cloudinit/config/cc_phone_home.py index 2e058ccd..5bc68b83 100644 --- a/cloudinit/config/cc_phone_home.py +++ b/cloudinit/config/cc_phone_home.py @@ -47,7 +47,7 @@ def handle(name, cfg, cloud, log, args): if len(args) != 0: ph_cfg = util.read_conf(args[0]) else: - if not 'phone_home' in cfg: + if 'phone_home' not in cfg: log.debug(("Skipping module named %s, " "no 'phone_home' configuration found"), name) return diff --git a/cloudinit/config/cc_power_state_change.py b/cloudinit/config/cc_power_state_change.py index 8f99e887..09d37371 100644 --- a/cloudinit/config/cc_power_state_change.py +++ b/cloudinit/config/cc_power_state_change.py @@ -89,8 +89,9 @@ def load_power_state(cfg): mode = pstate.get("mode") if mode not in opt_map: - raise TypeError("power_state[mode] required, must be one of: %s." % - ','.join(opt_map.keys())) + raise TypeError( + "power_state[mode] required, must be one of: %s. found: '%s'." % + (','.join(opt_map.keys()), mode)) delay = pstate.get("delay", "now") # convert integer 30 or string '30' to '+30' @@ -100,7 +101,9 @@ def load_power_state(cfg): pass if delay != "now" and not re.match(r"\+[0-9]+", delay): - raise TypeError("power_state[delay] must be 'now' or '+m' (minutes).") + raise TypeError( + "power_state[delay] must be 'now' or '+m' (minutes)." + " found '%s'." % delay) args = ["shutdown", opt_map[mode], delay] if pstate.get("message"): @@ -116,7 +119,7 @@ def load_power_state(cfg): def doexit(sysexit): - os._exit(sysexit) # pylint: disable=W0212 + os._exit(sysexit) def execmd(exe_args, output=None, data_in=None): @@ -124,7 +127,7 @@ def execmd(exe_args, output=None, data_in=None): proc = subprocess.Popen(exe_args, stdin=subprocess.PIPE, stdout=output, stderr=subprocess.STDOUT) proc.communicate(data_in) - ret = proc.returncode # pylint: disable=E1101 + ret = proc.returncode except Exception: doexit(EXIT_FAIL) doexit(ret) diff --git a/cloudinit/config/cc_resizefs.py b/cloudinit/config/cc_resizefs.py index be406034..cbc07853 100644 --- a/cloudinit/config/cc_resizefs.py +++ b/cloudinit/config/cc_resizefs.py @@ -28,19 +28,19 @@ from cloudinit import util frequency = PER_ALWAYS -def _resize_btrfs(mount_point, devpth): # pylint: disable=W0613 +def _resize_btrfs(mount_point, devpth): return ('btrfs', 'filesystem', 'resize', 'max', mount_point) -def _resize_ext(mount_point, devpth): # pylint: disable=W0613 +def _resize_ext(mount_point, devpth): return ('resize2fs', devpth) -def _resize_xfs(mount_point, devpth): # pylint: disable=W0613 +def _resize_xfs(mount_point, devpth): return ('xfs_growfs', devpth) -def _resize_ufs(mount_point, devpth): # pylint: disable=W0613 +def _resize_ufs(mount_point, devpth): return ('growfs', devpth) # Do not use a dictionary as these commands should be able to be used @@ -98,14 +98,14 @@ def handle(name, cfg, _cloud, log, args): (devpth, fs_type, mount_point) = result - # Ensure the path is a block device. info = "dev=%s mnt_point=%s path=%s" % (devpth, mount_point, resize_what) log.debug("resize_info: %s" % info) container = util.is_container() + # Ensure the path is a block device. if (devpth == "/dev/root" and not os.path.exists(devpth) and - not container): + not container): devpth = rootdev_from_cmdline(util.get_cmdline()) if devpth is None: log.warn("Unable to find device '/dev/root'") @@ -117,14 +117,22 @@ def handle(name, cfg, _cloud, log, args): except OSError as exc: if container and exc.errno == errno.ENOENT: log.debug("Device '%s' did not exist in container. " - "cannot resize: %s" % (devpth, info)) + "cannot resize: %s", devpth, info) elif exc.errno == errno.ENOENT: - log.warn("Device '%s' did not exist. cannot resize: %s" % - (devpth, info)) + log.warn("Device '%s' did not exist. cannot resize: %s", + devpth, info) else: raise exc return + if not os.access(devpth, os.W_OK): + if container: + log.debug("'%s' not writable in container. cannot resize: %s", + devpth, info) + else: + log.warn("'%s' not writable. cannot resize: %s", devpth, info) + return + if not stat.S_ISBLK(statret.st_mode) and not stat.S_ISCHR(statret.st_mode): if container: log.debug("device '%s' not a block device in container." @@ -154,8 +162,8 @@ def handle(name, cfg, _cloud, log, args): # Fork to a child that will run # the resize command util.fork_cb( - util.log_time(logfunc=log.debug, msg="backgrounded Resizing", - func=do_resize, args=(resize_cmd, log))) + util.log_time, logfunc=log.debug, msg="backgrounded Resizing", + func=do_resize, args=(resize_cmd, log)) else: util.log_time(logfunc=log.debug, msg="Resizing", func=do_resize, args=(resize_cmd, log)) diff --git a/cloudinit/config/cc_resolv_conf.py b/cloudinit/config/cc_resolv_conf.py index 879b62b1..bbaa6c63 100644 --- a/cloudinit/config/cc_resolv_conf.py +++ b/cloudinit/config/cc_resolv_conf.py @@ -49,23 +49,22 @@ # +from cloudinit import log as logging from cloudinit.settings import PER_INSTANCE from cloudinit import templater from cloudinit import util +LOG = logging.getLogger(__name__) + frequency = PER_INSTANCE distros = ['fedora', 'rhel', 'sles'] -def generate_resolv_conf(cloud, log, params): - template_fn = cloud.get_template_filename('resolv.conf') - if not template_fn: - log.warn("No template found, not rendering /etc/resolv.conf") - return - +def generate_resolv_conf(template_fn, params, target_fname="/etc/resolv.conf"): flags = [] false_flags = [] + if 'options' in params: for key, val in params['options'].iteritems(): if type(val) == bool: @@ -77,12 +76,15 @@ def generate_resolv_conf(cloud, log, params): for flag in flags + false_flags: del params['options'][flag] + if not params.get('options'): + params['options'] = {} + params['flags'] = flags - log.debug("Writing resolv.conf from template %s" % template_fn) - templater.render_to_file(template_fn, '/etc/resolv.conf', params) + LOG.debug("Writing resolv.conf from template %s" % template_fn) + templater.render_to_file(template_fn, target_fname, params) -def handle(name, cfg, _cloud, log, _args): +def handle(name, cfg, cloud, log, _args): """ Handler for resolv.conf @@ -102,8 +104,13 @@ def handle(name, cfg, _cloud, log, _args): " 'manage_resolv_conf' present but set to False"), name) return - if not "resolv_conf" in cfg: + if "resolv_conf" not in cfg: log.warn("manage_resolv_conf True but no parameters provided!") - generate_resolv_conf(_cloud, log, cfg["resolv_conf"]) + template_fn = cloud.get_template_filename('resolv.conf') + if not template_fn: + log.warn("No template found, not rendering /etc/resolv.conf") + return + + generate_resolv_conf(template_fn=template_fn, params=cfg["resolv_conf"]) return diff --git a/cloudinit/config/cc_rightscale_userdata.py b/cloudinit/config/cc_rightscale_userdata.py index c771728d..7d2ec10a 100644 --- a/cloudinit/config/cc_rightscale_userdata.py +++ b/cloudinit/config/cc_rightscale_userdata.py @@ -18,22 +18,22 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -## -## The purpose of this script is to allow cloud-init to consume -## rightscale style userdata. rightscale user data is key-value pairs -## in a url-query-string like format. -## -## for cloud-init support, there will be a key named -## 'CLOUD_INIT_REMOTE_HOOK'. -## -## This cloud-config module will -## - read the blob of data from raw user data, and parse it as key/value -## - for each key that is found, download the content to -## the local instance/scripts directory and set them executable. -## - the files in that directory will be run by the user-scripts module -## Therefore, this must run before that. -## -## +# +# The purpose of this script is to allow cloud-init to consume +# rightscale style userdata. rightscale user data is key-value pairs +# in a url-query-string like format. +# +# for cloud-init support, there will be a key named +# 'CLOUD_INIT_REMOTE_HOOK'. +# +# This cloud-config module will +# - read the blob of data from raw user data, and parse it as key/value +# - for each key that is found, download the content to +# the local instance/scripts directory and set them executable. +# - the files in that directory will be run by the user-scripts module +# Therefore, this must run before that. +# +# import os @@ -58,7 +58,7 @@ def handle(name, _cfg, cloud, log, _args): try: mdict = parse_qs(ud) - if not mdict or not MY_HOOKNAME in mdict: + if mdict or MY_HOOKNAME not in mdict: log.debug(("Skipping module %s, " "did not find %s in parsed" " raw userdata"), name, MY_HOOKNAME) diff --git a/cloudinit/config/cc_rsyslog.py b/cloudinit/config/cc_rsyslog.py index 0c2c6880..57486edc 100644 --- a/cloudinit/config/cc_rsyslog.py +++ b/cloudinit/config/cc_rsyslog.py @@ -35,7 +35,7 @@ def handle(name, cfg, cloud, log, _args): # *.* @@syslogd.example.com # process 'rsyslog' - if not 'rsyslog' in cfg: + if 'rsyslog' not in cfg: log.debug(("Skipping module named %s," " no 'rsyslog' key in configuration"), name) return @@ -46,7 +46,7 @@ def handle(name, cfg, cloud, log, _args): files = [] for i, ent in enumerate(cfg['rsyslog']): if isinstance(ent, dict): - if not "content" in ent: + if "content" not in ent: log.warn("No 'content' entry in config entry %s", i + 1) continue content = ent['content'] diff --git a/cloudinit/config/cc_set_passwords.py b/cloudinit/config/cc_set_passwords.py index 4a3b21af..4ca85e21 100644 --- a/cloudinit/config/cc_set_passwords.py +++ b/cloudinit/config/cc_set_passwords.py @@ -28,7 +28,7 @@ from cloudinit import distros as ds from cloudinit import ssh_util from cloudinit import util -from string import letters, digits # pylint: disable=W0402 +from string import letters, digits # We are removing certain 'painful' letters/numbers PW_SET = (letters.translate(None, 'loLOI') + @@ -132,7 +132,7 @@ def handle(_name, cfg, cloud, log, args): 'PasswordAuthentication', pw_auth)) - lines = [str(e) for e in new_lines] + lines = [str(l) for l in new_lines] util.write_file(ssh_util.DEF_SSHD_CFG, "\n".join(lines)) try: diff --git a/cloudinit/config/cc_ssh.py b/cloudinit/config/cc_ssh.py index 64a5e3cb..4c76581c 100644 --- a/cloudinit/config/cc_ssh.py +++ b/cloudinit/config/cc_ssh.py @@ -75,7 +75,7 @@ def handle(_name, cfg, cloud, log, _args): util.write_file(tgt_fn, val, tgt_perms) for (priv, pub) in PRIV_2_PUB.iteritems(): - if pub in cfg['ssh_keys'] or not priv in cfg['ssh_keys']: + if pub in cfg['ssh_keys'] or priv not in cfg['ssh_keys']: continue pair = (KEY_2_FILE[priv][0], KEY_2_FILE[pub][0]) cmd = ['sh', '-xc', KEY_GEN_TPL % pair] diff --git a/cloudinit/config/cc_ssh_authkey_fingerprints.py b/cloudinit/config/cc_ssh_authkey_fingerprints.py index be8083db..51580633 100644 --- a/cloudinit/config/cc_ssh_authkey_fingerprints.py +++ b/cloudinit/config/cc_ssh_authkey_fingerprints.py @@ -55,7 +55,7 @@ def _gen_fingerprint(b64_text, hash_meth='md5'): def _is_printable_key(entry): if any([entry.keytype, entry.base64, entry.comment, entry.options]): if (entry.keytype and - entry.keytype.lower().strip() in ['ssh-dss', 'ssh-rsa']): + entry.keytype.lower().strip() in ['ssh-dss', 'ssh-rsa']): return True return False @@ -92,9 +92,10 @@ def _pprint_key_entries(user, key_fn, key_entries, hash_meth='md5', def handle(name, cfg, cloud, log, _args): - if 'no_ssh_fingerprints' in cfg: + if util.is_true(cfg.get('no_ssh_fingerprints', False)): log.debug(("Skipping module named %s, " "logging of ssh fingerprints disabled"), name) + return hash_meth = util.get_cfg_option_str(cfg, "authkey_hash", "md5") (users, _groups) = ds.normalize_users_groups(cfg, cloud.distro) diff --git a/cloudinit/config/cc_ssh_import_id.py b/cloudinit/config/cc_ssh_import_id.py index 76c1663d..2d480d7e 100644 --- a/cloudinit/config/cc_ssh_import_id.py +++ b/cloudinit/config/cc_ssh_import_id.py @@ -85,7 +85,7 @@ def import_ssh_ids(ids, user, log): return try: - _check = pwd.getpwnam(user) + pwd.getpwnam(user) except KeyError as exc: raise exc diff --git a/cloudinit/config/cc_ubuntu_init_switch.py b/cloudinit/config/cc_ubuntu_init_switch.py new file mode 100644 index 00000000..6f994bff --- /dev/null +++ b/cloudinit/config/cc_ubuntu_init_switch.py @@ -0,0 +1,164 @@ +# vi: ts=4 expandtab +# +# Copyright (C) 2014 Canonical Ltd. +# +# Author: Scott Moser <scott.moser@canonical.com> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 3, as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +""" +ubuntu_init_switch: reboot system into another init + +This provides a way for the user to boot with systemd even if the +image is set to boot with upstart. It should be run as one of the first +cloud_init_modules, and will switch the init system and then issue a reboot. +The next boot will come up in the target init system and no action will +be taken. + +This should be inert on non-ubuntu systems, and also exit quickly. + +config is comes under the top level 'init_switch' dictionary. + +#cloud-config +init_switch: + target: systemd + reboot: true + +'target' can be 'systemd' or 'upstart'. Best effort is made, but its possible +this system will break, and probably won't interact well with any other +mechanism you've used to switch the init system. + +'reboot': [default=true]. + true: reboot if a change was made. + false: do not reboot. +""" + +from cloudinit.settings import PER_INSTANCE +from cloudinit import log as logging +from cloudinit import util +from cloudinit.distros import ubuntu + +import os +import time + +frequency = PER_INSTANCE +REBOOT_CMD = ["/sbin/reboot", "--force"] + +DEFAULT_CONFIG = { + 'init_switch': {'target': None, 'reboot': True} +} + +SWITCH_INIT = """ +#!/bin/sh +# switch_init: [upstart | systemd] + +is_systemd() { + [ "$(dpkg-divert --listpackage /sbin/init)" = "systemd-sysv" ] +} +debug() { echo "$@" 1>&2; } +fail() { echo "$@" 1>&2; exit 1; } + +if [ "$1" = "systemd" ]; then + if is_systemd; then + debug "already systemd, nothing to do" + else + [ -f /lib/systemd/systemd ] || fail "no systemd available"; + dpkg-divert --package systemd-sysv --divert /sbin/init.diverted \\ + --rename /sbin/init + fi + [ -f /sbin/init ] || ln /lib/systemd/systemd /sbin/init +elif [ "$1" = "upstart" ]; then + if is_systemd; then + rm -f /sbin/init + dpkg-divert --package systemd-sysv --rename --remove /sbin/init + else + debug "already upstart, nothing to do." + fi +else + fail "Error. expect 'upstart' or 'systemd'" +fi +""" + + +def handle(name, cfg, cloud, log, args): + + if not isinstance(cloud.distro, ubuntu.Distro): + log.debug("%s: distro is '%s', not ubuntu. returning", + name, cloud.distro.__class__) + return + + cfg = util.mergemanydict([cfg, DEFAULT_CONFIG]) + target = cfg['init_switch']['target'] + reboot = cfg['init_switch']['reboot'] + + if len(args) != 0: + target = args[0] + if len(args) > 1: + reboot = util.is_true(args[1]) + + if not target: + log.debug("%s: target=%s. nothing to do", name, target) + return + + if not util.which('dpkg'): + log.warn("%s: 'dpkg' not available. Assuming not ubuntu", name) + return + + supported = ('upstart', 'systemd') + if target not in supported: + log.warn("%s: target set to %s, expected one of: %s", + name, target, str(supported)) + + if os.path.exists("/run/systemd/system"): + current = "systemd" + else: + current = "upstart" + + if current == target: + log.debug("%s: current = target = %s. nothing to do", name, target) + return + + try: + util.subp(['sh', '-s', target], data=SWITCH_INIT) + except util.ProcessExecutionError as e: + log.warn("%s: Failed to switch to init '%s'. %s", name, target, e) + return + + if util.is_false(reboot): + log.info("%s: switched '%s' to '%s'. reboot=false, not rebooting.", + name, current, target) + return + + try: + log.warn("%s: switched '%s' to '%s'. rebooting.", + name, current, target) + logging.flushLoggers(log) + _fire_reboot(log, wait_attempts=4, initial_sleep=4) + except Exception as e: + util.logexc(log, "Requested reboot did not happen!") + raise + + +def _fire_reboot(log, wait_attempts=6, initial_sleep=1, backoff=2): + util.subp(REBOOT_CMD) + start = time.time() + wait_time = initial_sleep + for _i in range(0, wait_attempts): + time.sleep(wait_time) + wait_time *= backoff + elapsed = time.time() - start + log.debug("Rebooted, but still running after %s seconds", int(elapsed)) + # If we got here, not good + elapsed = time.time() - start + raise RuntimeError(("Reboot did not happen" + " after %s seconds!") % (int(elapsed))) diff --git a/cloudinit/config/cc_yum_add_repo.py b/cloudinit/config/cc_yum_add_repo.py index 5c273825..0d836f28 100644 --- a/cloudinit/config/cc_yum_add_repo.py +++ b/cloudinit/config/cc_yum_add_repo.py @@ -42,7 +42,7 @@ def _format_repo_value(val): return val -## TODO(harlowja): move to distro? +# TODO(harlowja): move to distro? # See man yum.conf def _format_repository_config(repo_id, repo_config): to_be = configobj.ConfigObj() @@ -89,7 +89,7 @@ def handle(name, cfg, _cloud, log, _args): repo_config = n_repo_config missing_required = 0 for req_field in ['baseurl']: - if not req_field in repo_config: + if req_field not in repo_config: log.warn(("Repository %s does not contain a %s" " configuration 'required' entry"), repo_id, req_field) diff --git a/cloudinit/distros/__init__.py b/cloudinit/distros/__init__.py index 55d6bcbc..2599d9f2 100644 --- a/cloudinit/distros/__init__.py +++ b/cloudinit/distros/__init__.py @@ -167,7 +167,7 @@ class Distro(object): def expand_osfamily(family_list): distros = [] for family in family_list: - if not family in OSFAMILIES: + if family not in OSFAMILIES: raise ValueError("No distibutions found for osfamily %s" % (family)) distros.extend(OSFAMILIES[family]) @@ -218,7 +218,7 @@ class Distro(object): fn) if (sys_hostname and prev_hostname and - sys_hostname != prev_hostname): + sys_hostname != prev_hostname): LOG.debug("%s differs from %s, assuming user maintained hostname.", prev_hostname_fn, sys_fn) @@ -847,12 +847,19 @@ def extract_default(users, default_name=None, default_config=None): def fetch(name): - locs = importer.find_module(name, - ['', __name__], - ['Distro']) + locs, looked_locs = importer.find_module(name, ['', __name__], ['Distro']) if not locs: - raise ImportError("No distribution found for distro %s" - % (name)) + raise ImportError("No distribution found for distro %s (searched %s)" + % (name, looked_locs)) mod = importer.import_module(locs[0]) cls = getattr(mod, 'Distro') return cls + + +def set_etc_timezone(tz, tz_file=None, tz_conf="/etc/timezone", + tz_local="/etc/localtime"): + util.write_file(tz_conf, str(tz).rstrip() + "\n") + # This ensures that the correct tz will be used for the system + if tz_local and tz_file: + util.copy(tz_file, tz_local) + return diff --git a/cloudinit/distros/arch.py b/cloudinit/distros/arch.py index 310c3dff..005a0dd4 100644 --- a/cloudinit/distros/arch.py +++ b/cloudinit/distros/arch.py @@ -32,8 +32,6 @@ LOG = logging.getLogger(__name__) class Distro(distros.Distro): locale_conf_fn = "/etc/locale.gen" network_conf_dir = "/etc/netctl" - tz_conf_fn = "/etc/timezone" - tz_local_fn = "/etc/localtime" resolve_conf_fn = "/etc/resolv.conf" init_cmd = ['systemctl'] # init scripts @@ -161,16 +159,7 @@ class Distro(distros.Distro): return hostname def set_timezone(self, tz): - tz_file = self._find_tz_file(tz) - # Note: "" provides trailing newline during join - tz_lines = [ - util.make_header(), - str(tz), - "", - ] - util.write_file(self.tz_conf_fn, "\n".join(tz_lines)) - # This ensures that the correct tz will be used for the system - util.copy(tz_file, self.tz_local_fn) + distros.set_etc_timezone(tz=tz, tz_file=self._find_tz_file(tz)) def package_command(self, command, args=None, pkgs=None): if pkgs is None: diff --git a/cloudinit/distros/debian.py b/cloudinit/distros/debian.py index 1ae232fd..010be67d 100644 --- a/cloudinit/distros/debian.py +++ b/cloudinit/distros/debian.py @@ -46,8 +46,6 @@ class Distro(distros.Distro): hostname_conf_fn = "/etc/hostname" locale_conf_fn = "/etc/default/locale" network_conf_fn = "/etc/network/interfaces" - tz_conf_fn = "/etc/timezone" - tz_local_fn = "/etc/localtime" def __init__(self, name, cfg, paths): distros.Distro.__init__(self, name, cfg, paths) @@ -133,16 +131,7 @@ class Distro(distros.Distro): return "127.0.1.1" def set_timezone(self, tz): - tz_file = self._find_tz_file(tz) - # Note: "" provides trailing newline during join - tz_lines = [ - util.make_header(), - str(tz), - "", - ] - util.write_file(self.tz_conf_fn, "\n".join(tz_lines)) - # This ensures that the correct tz will be used for the system - util.copy(tz_file, self.tz_local_fn) + distros.set_etc_timezone(tz=tz, tz_file=self._find_tz_file(tz)) def package_command(self, command, args=None, pkgs=None): if pkgs is None: diff --git a/cloudinit/distros/freebsd.py b/cloudinit/distros/freebsd.py index d98f9578..cff10387 100644 --- a/cloudinit/distros/freebsd.py +++ b/cloudinit/distros/freebsd.py @@ -26,6 +26,9 @@ from cloudinit import log as logging from cloudinit import ssh_util from cloudinit import util +from cloudinit.distros import net_util +from cloudinit.distros.parsers.resolv_conf import ResolvConf + LOG = logging.getLogger(__name__) @@ -33,6 +36,8 @@ class Distro(distros.Distro): rc_conf_fn = "/etc/rc.conf" login_conf_fn = '/etc/login.conf' login_conf_fn_bak = '/etc/login.conf.orig' + resolv_conf_fn = '/etc/resolv.conf' + ci_sudoers_fn = '/usr/local/etc/sudoers.d/90-cloud-init-users' def __init__(self, name, cfg, paths): distros.Distro.__init__(self, name, cfg, paths) @@ -44,30 +49,53 @@ class Distro(distros.Distro): # Updates a key in /etc/rc.conf. def updatercconf(self, key, value): - LOG.debug("updatercconf: %s => %s", key, value) + LOG.debug("Checking %s for: %s = %s", self.rc_conf_fn, key, value) conf = self.loadrcconf() config_changed = False - for item in conf: - if item == key and conf[item] != value: - conf[item] = value - LOG.debug("[rc.conf]: Value %s for key %s needs to be changed", - value, key) - config_changed = True + if key not in conf: + LOG.debug("Adding key in %s: %s = %s", self.rc_conf_fn, key, + value) + conf[key] = value + config_changed = True + else: + for item in conf.keys(): + if item == key and conf[item] != value: + conf[item] = value + LOG.debug("Changing key in %s: %s = %s", self.rc_conf_fn, + key, value) + config_changed = True if config_changed: - LOG.debug("Writing new %s file", self.rc_conf_fn) + LOG.info("Writing %s", self.rc_conf_fn) buf = StringIO() for keyval in conf.items(): - buf.write("%s=%s\n" % keyval) + buf.write('%s="%s"\n' % keyval) util.write_file(self.rc_conf_fn, buf.getvalue()) - # Load the contents of /etc/rc.conf and store all keys in a dict. + # Load the contents of /etc/rc.conf and store all keys in a dict. Make sure + # quotes are ignored: + # hostname="bla" def loadrcconf(self): + RE_MATCH = re.compile(r'^(\w+)\s*=\s*(.*)\s*') conf = {} lines = util.load_file(self.rc_conf_fn).splitlines() for line in lines: - tok = line.split('=') - conf[tok[0]] = tok[1].rstrip() + m = RE_MATCH.match(line) + if not m: + LOG.debug("Skipping line from /etc/rc.conf: %s", line) + continue + key = m.group(1).rstrip() + val = m.group(2).rstrip() + # Kill them quotes (not completely correct, aka won't handle + # quoted values, but should be ok ...) + if val[0] in ('"', "'"): + val = val[1:] + if val[-1] in ('"', "'"): + val = val[0:-1] + if len(val) == 0: + LOG.debug("Skipping empty value from /etc/rc.conf: %s", line) + continue + conf[key] = val return conf def readrcconf(self, key): @@ -192,10 +220,6 @@ class Distro(distros.Distro): util.logexc(LOG, "Failed to lock user %s", name) raise e - # TODO: - def write_sudo_rules(self, name, rules, sudo_file=None): - LOG.debug("[write_sudo_rules] Name: %s", name) - def create_user(self, name, **kwargs): self.add_user(name, **kwargs) @@ -218,7 +242,60 @@ class Distro(distros.Distro): ssh_util.setup_user_keys(keys, name, options=None) def _write_network(self, settings): - return + entries = net_util.translate_network(settings) + nameservers = [] + searchdomains = [] + dev_names = entries.keys() + for (dev, info) in entries.iteritems(): + # Skip the loopback interface. + if dev.startswith('lo'): + continue + + LOG.info('Configuring interface %s', dev) + + if info.get('bootproto') == 'static': + LOG.debug('Configuring dev %s with %s / %s', dev, info.get('address'), info.get('netmask')) + # Configure an ipv4 address. + ifconfig = info.get('address') + ' netmask ' + info.get('netmask') + + # Configure the gateway. + self.updatercconf('defaultrouter', info.get('gateway')) + + if 'dns-nameservers' in info: + nameservers.extend(info['dns-nameservers']) + if 'dns-search' in info: + searchdomains.extend(info['dns-search']) + else: + ifconfig = 'DHCP' + + self.updatercconf('ifconfig_' + dev, ifconfig) + + # Try to read the /etc/resolv.conf or just start from scratch if that + # fails. + try: + resolvconf = ResolvConf(util.load_file(self.resolv_conf_fn)) + resolvconf.parse() + except IOError: + util.logexc(LOG, "Failed to parse %s, use new empty file", self.resolv_conf_fn) + resolvconf = ResolvConf('') + resolvconf.parse() + + # Add some nameservers + for server in nameservers: + try: + resolvconf.add_nameserver(server) + except ValueError: + util.logexc(LOG, "Failed to add nameserver %s", server) + + # And add any searchdomains. + for domain in searchdomains: + try: + resolvconf.add_search_domain(domain) + except ValueError: + util.logexc(LOG, "Failed to add search domain %s", domain) + util.write_file(self.resolv_conf_fn, str(resolvconf), 0644) + + return dev_names def apply_locale(self, locale, out_fn=None): # Adjust the locals value to the new value diff --git a/cloudinit/distros/gentoo.py b/cloudinit/distros/gentoo.py index 09f8d8ea..45c2e658 100644 --- a/cloudinit/distros/gentoo.py +++ b/cloudinit/distros/gentoo.py @@ -31,8 +31,6 @@ LOG = logging.getLogger(__name__) class Distro(distros.Distro): locale_conf_fn = "/etc/locale.gen" network_conf_fn = "/etc/conf.d/net" - tz_conf_fn = "/etc/timezone" - tz_local_fn = "/etc/localtime" init_cmd = [''] # init scripts def __init__(self, name, cfg, paths): @@ -140,16 +138,7 @@ class Distro(distros.Distro): return hostname def set_timezone(self, tz): - tz_file = self._find_tz_file(tz) - # Note: "" provides trailing newline during join - tz_lines = [ - util.make_header(), - str(tz), - "", - ] - util.write_file(self.tz_conf_fn, "\n".join(tz_lines)) - # This ensures that the correct tz will be used for the system - util.copy(tz_file, self.tz_local_fn) + distros.set_etc_timezone(tz=tz, tz_file=self._find_tz_file(tz)) def package_command(self, command, args=None, pkgs=None): if pkgs is None: diff --git a/cloudinit/distros/parsers/resolv_conf.py b/cloudinit/distros/parsers/resolv_conf.py index 1be9d46b..5733c25a 100644 --- a/cloudinit/distros/parsers/resolv_conf.py +++ b/cloudinit/distros/parsers/resolv_conf.py @@ -137,8 +137,8 @@ class ResolvConf(object): self._contents.append(('option', ['search', s_list, ''])) return flat_sds - @local_domain.setter # pl51222 pylint: disable=E1101 - def local_domain(self, domain): # pl51222 pylint: disable=E0102 + @local_domain.setter + def local_domain(self, domain): self.parse() self._remove_option('domain') self._contents.append(('option', ['domain', str(domain), ''])) diff --git a/cloudinit/ec2_utils.py b/cloudinit/ec2_utils.py index a7c9c9ab..0c751140 100644 --- a/cloudinit/ec2_utils.py +++ b/cloudinit/ec2_utils.py @@ -166,7 +166,9 @@ def get_instance_metadata(api_version='latest', metadata_address='http://169.254.169.254', ssl_details=None, timeout=5, retries=5): md_url = url_helper.combine_url(metadata_address, api_version) - md_url = url_helper.combine_url(md_url, 'meta-data') + # Note, 'meta-data' explicitly has trailing /. + # this is required for CloudStack (LP: #1356855) + md_url = url_helper.combine_url(md_url, 'meta-data/') caller = functools.partial(util.read_file_or_url, ssl_details=ssl_details, timeout=timeout, retries=retries) diff --git a/cloudinit/handlers/boot_hook.py b/cloudinit/handlers/boot_hook.py index 1848ce2c..3a50cf87 100644 --- a/cloudinit/handlers/boot_hook.py +++ b/cloudinit/handlers/boot_hook.py @@ -53,8 +53,7 @@ class BootHookPartHandler(handlers.Handler): util.write_file(filepath, contents.lstrip(), 0700) return filepath - def handle_part(self, _data, ctype, filename, # pylint: disable=W0221 - payload, frequency): # pylint: disable=W0613 + def handle_part(self, data, ctype, filename, payload, frequency): if ctype in handlers.CONTENT_SIGNALS: return diff --git a/cloudinit/handlers/cloud_config.py b/cloudinit/handlers/cloud_config.py index 4232700f..bf994e33 100644 --- a/cloudinit/handlers/cloud_config.py +++ b/cloudinit/handlers/cloud_config.py @@ -138,8 +138,7 @@ class CloudConfigPartHandler(handlers.Handler): self.file_names = [] self.cloud_buf = None - def handle_part(self, _data, ctype, filename, # pylint: disable=W0221 - payload, _frequency, headers): # pylint: disable=W0613 + def handle_part(self, data, ctype, filename, payload, frequency, headers): if ctype == handlers.CONTENT_START: self._reset() return diff --git a/cloudinit/handlers/shell_script.py b/cloudinit/handlers/shell_script.py index 30c1ed89..9755ab05 100644 --- a/cloudinit/handlers/shell_script.py +++ b/cloudinit/handlers/shell_script.py @@ -44,8 +44,7 @@ class ShellScriptPartHandler(handlers.Handler): handlers.type_from_starts_with(SHELL_PREFIX), ] - def handle_part(self, _data, ctype, filename, # pylint: disable=W0221 - payload, frequency): # pylint: disable=W0613 + def handle_part(self, data, ctype, filename, payload, frequency): if ctype in handlers.CONTENT_SIGNALS: # TODO(harlowja): maybe delete existing things here return diff --git a/cloudinit/handlers/upstart_job.py b/cloudinit/handlers/upstart_job.py index bac4cad2..50d193c4 100644 --- a/cloudinit/handlers/upstart_job.py +++ b/cloudinit/handlers/upstart_job.py @@ -44,8 +44,7 @@ class UpstartJobPartHandler(handlers.Handler): handlers.type_from_starts_with(UPSTART_PREFIX), ] - def handle_part(self, _data, ctype, filename, # pylint: disable=W0221 - payload, frequency): + def handle_part(self, data, ctype, filename, payload, frequency): if ctype in handlers.CONTENT_SIGNALS: return diff --git a/cloudinit/importer.py b/cloudinit/importer.py index a094141a..fb57253c 100644 --- a/cloudinit/importer.py +++ b/cloudinit/importer.py @@ -22,10 +22,6 @@ import sys -from cloudinit import log as logging - -LOG = logging.getLogger(__name__) - def import_module(module_name): __import__(module_name) @@ -33,27 +29,24 @@ def import_module(module_name): def find_module(base_name, search_paths, required_attrs=None): - found_places = [] if not required_attrs: required_attrs = [] # NOTE(harlowja): translate the search paths to include the base name. - real_paths = [] + lookup_paths = [] for path in search_paths: real_path = [] if path: real_path.extend(path.split(".")) real_path.append(base_name) full_path = '.'.join(real_path) - real_paths.append(full_path) - LOG.debug("Looking for modules %s that have attributes %s", - real_paths, required_attrs) - for full_path in real_paths: + lookup_paths.append(full_path) + found_paths = [] + for full_path in lookup_paths: mod = None try: mod = import_module(full_path) - except ImportError as e: - LOG.debug("Failed at attempted import of '%s' due to: %s", - full_path, e) + except ImportError: + pass if not mod: continue found_attrs = 0 @@ -61,7 +54,5 @@ def find_module(base_name, search_paths, required_attrs=None): if hasattr(mod, attr): found_attrs += 1 if found_attrs == len(required_attrs): - found_places.append(full_path) - LOG.debug("Found %s with attributes %s in %s", base_name, - required_attrs, found_places) - return found_places + found_paths.append(full_path) + return (found_paths, lookup_paths) diff --git a/cloudinit/mergers/__init__.py b/cloudinit/mergers/__init__.py index 0978b2c6..03aa1ee1 100644 --- a/cloudinit/mergers/__init__.py +++ b/cloudinit/mergers/__init__.py @@ -55,9 +55,6 @@ class UnknownMerger(object): if not meth: meth = self._handle_unknown args.insert(0, method_name) - LOG.debug("Merging '%s' into '%s' using method '%s' of '%s'", - type_name, type_utils.obj_name(merge_with), - meth.__name__, self) return meth(*args) @@ -84,8 +81,6 @@ class LookupMerger(UnknownMerger): # First one that has that method/attr gets to be # the one that will be called meth = getattr(merger, meth_wanted) - LOG.debug(("Merging using located merger '%s'" - " since it had method '%s'"), merger, meth_wanted) break if not meth: return UnknownMerger._handle_unknown(self, meth_wanted, @@ -148,12 +143,14 @@ def construct(parsed_mergers): for (m_name, m_ops) in parsed_mergers: if not m_name.startswith(MERGER_PREFIX): m_name = MERGER_PREFIX + str(m_name) - merger_locs = importer.find_module(m_name, - [__name__], - [MERGER_ATTR]) + merger_locs, looked_locs = importer.find_module(m_name, + [__name__], + [MERGER_ATTR]) if not merger_locs: msg = ("Could not find merger module named '%s' " - "with attribute '%s'") % (m_name, MERGER_ATTR) + "with attribute '%s' (searched %s)") % (m_name, + MERGER_ATTR, + looked_locs) raise ImportError(msg) else: mod = importer.import_module(merger_locs[0]) diff --git a/cloudinit/mergers/m_list.py b/cloudinit/mergers/m_list.py index 62999b4e..3b87b0fc 100644 --- a/cloudinit/mergers/m_list.py +++ b/cloudinit/mergers/m_list.py @@ -53,7 +53,7 @@ class Merger(object): def _on_list(self, value, merge_with): if (self._method == 'replace' and - not isinstance(merge_with, (tuple, list))): + not isinstance(merge_with, (tuple, list))): return merge_with # Ok we now know that what we are merging with is a list or tuple. diff --git a/cloudinit/netinfo.py b/cloudinit/netinfo.py index 30b6f3b3..8d4df342 100644 --- a/cloudinit/netinfo.py +++ b/cloudinit/netinfo.py @@ -21,10 +21,13 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. import cloudinit.util as util +from cloudinit.log import logging import re from prettytable import PrettyTable +LOG = logging.getLogger() + def netdev_info(empty=""): fields = ("hwaddr", "addr", "bcast", "mask") @@ -56,8 +59,8 @@ def netdev_info(empty=""): # newer (freebsd and fedora) show 'inet xx.yy' # just skip this 'inet' entry. (LP: #1285185) try: - if (toks[i] in ("inet", "inet6") and - toks[i + 1].startswith("addr:")): + if ((toks[i] in ("inet", "inet6") and + toks[i + 1].startswith("addr:"))): continue except IndexError: pass @@ -168,8 +171,9 @@ def route_pformat(): lines = [] try: routes = route_info() - except Exception: + except Exception as e: lines.append(util.center('Route info failed', '!', 80)) + util.logexc(LOG, "Route info failed: %s" % e) routes = None if routes is not None: fields = ['Route', 'Destination', 'Gateway', diff --git a/cloudinit/patcher.py b/cloudinit/patcher.py index 0f3c034e..f6609d6f 100644 --- a/cloudinit/patcher.py +++ b/cloudinit/patcher.py @@ -41,7 +41,7 @@ def _patch_logging(): fallback_handler = QuietStreamHandler(sys.stderr) fallback_handler.setFormatter(logging.Formatter(FALL_FORMAT)) - def handleError(self, record): # pylint: disable=W0613 + def handleError(self, record): try: fallback_handler.handle(record) fallback_handler.flush() diff --git a/cloudinit/settings.py b/cloudinit/settings.py index 37d4958b..5efcb0b0 100644 --- a/cloudinit/settings.py +++ b/cloudinit/settings.py @@ -37,7 +37,7 @@ CFG_BUILTIN = { 'OVF', 'MAAS', 'GCE', - 'OpenStack' + 'OpenStack', 'Ec2', 'CloudSigma', 'CloudStack', diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py index bd75e6d8..09bc196d 100644 --- a/cloudinit/sources/DataSourceAzure.py +++ b/cloudinit/sources/DataSourceAzure.py @@ -452,7 +452,7 @@ def load_azure_ovf_pubkeys(sshnode): continue if (len(child.childNodes) != 1 or - child.childNodes[0].nodeType != text_node): + child.childNodes[0].nodeType != text_node): continue cur[name] = child.childNodes[0].wholeText.strip() @@ -521,7 +521,7 @@ def read_azure_ovf(contents): simple = False value = "" if (len(child.childNodes) == 1 and - child.childNodes[0].nodeType == dom.TEXT_NODE): + child.childNodes[0].nodeType == dom.TEXT_NODE): simple = True value = child.childNodes[0].wholeText diff --git a/cloudinit/sources/DataSourceCloudStack.py b/cloudinit/sources/DataSourceCloudStack.py index 08f661e4..1bbeca59 100644 --- a/cloudinit/sources/DataSourceCloudStack.py +++ b/cloudinit/sources/DataSourceCloudStack.py @@ -78,7 +78,8 @@ class DataSourceCloudStack(sources.DataSource): (max_wait, timeout) = self._get_url_settings() - urls = [self.metadata_address + "/latest/meta-data/instance-id"] + urls = [uhelp.combine_url(self.metadata_address, + 'latest/meta-data/instance-id')] start_time = time.time() url = uhelp.wait_for_url(urls=urls, max_wait=max_wait, timeout=timeout, status_cb=LOG.warn) diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py index 0c35f83a..4e5d90de 100644 --- a/cloudinit/sources/DataSourceConfigDrive.py +++ b/cloudinit/sources/DataSourceConfigDrive.py @@ -125,7 +125,15 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource): self.userdata_raw = results.get('userdata') self.version = results['version'] self.files.update(results.get('files', {})) - self.vendordata_raw = results.get('vendordata') + + vd = results.get('vendordata') + self.vendordata_pure = vd + try: + self.vendordata_raw = openstack.convert_vendordata_json(vd) + except ValueError as e: + LOG.warn("Invalid content in vendor-data: %s", e) + self.vendordata_raw = None + return True @@ -160,10 +168,10 @@ def get_ds_mode(cfgdrv_ver, ds_cfg=None, user=None): return "net" -def read_config_drive(source_dir, version="2012-08-10"): +def read_config_drive(source_dir): reader = openstack.ConfigDriveReader(source_dir) finders = [ - (reader.read_v2, [], {'version': version}), + (reader.read_v2, [], {}), (reader.read_v1, [], {}), ] excps = [] diff --git a/cloudinit/sources/DataSourceNoCloud.py b/cloudinit/sources/DataSourceNoCloud.py index a315aae0..c26a645c 100644 --- a/cloudinit/sources/DataSourceNoCloud.py +++ b/cloudinit/sources/DataSourceNoCloud.py @@ -181,7 +181,7 @@ class DataSourceNoCloud(sources.DataSource): # and the source of the seed was self.dsmode # ('local' for NoCloud, 'net' for NoCloudNet') if ('network-interfaces' in mydata['meta-data'] and - (self.dsmode in ("local", seeded_interfaces))): + (self.dsmode in ("local", seeded_interfaces))): LOG.debug("Updating network interfaces from %s", self) self.distro.apply_network( mydata['meta-data']['network-interfaces']) diff --git a/cloudinit/sources/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py index 77b43e17..2f53c1ba 100644 --- a/cloudinit/sources/DataSourceOVF.py +++ b/cloudinit/sources/DataSourceOVF.py @@ -107,7 +107,7 @@ class DataSourceOVF(sources.DataSource): return True def get_public_ssh_keys(self): - if not 'public-keys' in self.metadata: + if 'public-keys' not in self.metadata: return [] pks = self.metadata['public-keys'] if isinstance(pks, (list)): @@ -205,7 +205,7 @@ def transport_iso9660(require_iso=True): fullp = os.path.join("/dev/", dev) if (fullp in mounts or - not cdmatch.match(dev) or os.path.isdir(fullp)): + not cdmatch.match(dev) or os.path.isdir(fullp)): continue try: diff --git a/cloudinit/sources/DataSourceOpenNebula.py b/cloudinit/sources/DataSourceOpenNebula.py index 34557f8b..e2469f6e 100644 --- a/cloudinit/sources/DataSourceOpenNebula.py +++ b/cloudinit/sources/DataSourceOpenNebula.py @@ -28,7 +28,7 @@ import base64 import os import pwd import re -import string # pylint: disable=W0402 +import string from cloudinit import log as logging from cloudinit import sources diff --git a/cloudinit/sources/DataSourceOpenStack.py b/cloudinit/sources/DataSourceOpenStack.py index 0970d07b..469c2e2a 100644 --- a/cloudinit/sources/DataSourceOpenStack.py +++ b/cloudinit/sources/DataSourceOpenStack.py @@ -88,11 +88,9 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource): md_urls = [] url2base = {} for url in urls: - for version in openstack.OS_VERSIONS + (openstack.OS_LATEST,): - md_url = url_helper.combine_url(url, 'openstack', - version, 'meta_data.json') - md_urls.append(md_url) - url2base[md_url] = url + md_url = url_helper.combine_url(url, 'openstack') + md_urls.append(md_url) + url2base[md_url] = url (max_wait, timeout) = self._get_url_settings() start_time = time.time() @@ -119,8 +117,7 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource): 'Crawl of openstack metadata service', read_metadata_service, args=[self.metadata_address], - kwargs={'ssl_details': self.ssl_details, - 'version': openstack.OS_HAVANA}) + kwargs={'ssl_details': self.ssl_details}) except openstack.NonReadable: return False except (openstack.BrokenMetadata, IOError): @@ -143,20 +140,20 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource): self.version = results['version'] self.files.update(results.get('files', {})) - # if vendordata includes 'cloud-init', then read that explicitly - # for cloud-init (for namespacing). vd = results.get('vendordata') - if isinstance(vd, dict) and 'cloud-init' in vd: - self.vendordata_raw = vd['cloud-init'] - else: - self.vendordata_raw = vd + self.vendordata_pure = vd + try: + self.vendordata_raw = openstack.convert_vendordata_json(vd) + except ValueError as e: + LOG.warn("Invalid content in vendor-data: %s", e) + self.vendordata_raw = None return True -def read_metadata_service(base_url, version=None, ssl_details=None): +def read_metadata_service(base_url, ssl_details=None): reader = openstack.MetadataReader(base_url, ssl_details=ssl_details) - return reader.read_v2(version=version) + return reader.read_v2() # Used to match classes to dependencies diff --git a/cloudinit/sources/DataSourceSmartOS.py b/cloudinit/sources/DataSourceSmartOS.py index 65ec0339..2733a2f6 100644 --- a/cloudinit/sources/DataSourceSmartOS.py +++ b/cloudinit/sources/DataSourceSmartOS.py @@ -41,7 +41,7 @@ import serial LOG = logging.getLogger(__name__) SMARTOS_ATTRIB_MAP = { - #Cloud-init Key : (SmartOS Key, Strip line endings) + # Cloud-init Key : (SmartOS Key, Strip line endings) 'local-hostname': ('hostname', True), 'public-keys': ('root_authorized_keys', True), 'user-script': ('user-script', False), @@ -96,21 +96,21 @@ BUILTIN_CLOUD_CONFIG = { 'device': 'ephemeral0'}], } -## builtin vendor-data is a boothook that writes a script into -## /var/lib/cloud/scripts/per-boot. *That* script then handles -## executing the 'operator-script' and 'user-script' files -## that cloud-init writes into /var/lib/cloud/instance/data/ -## if they exist. -## -## This is all very indirect, but its done like this so that at -## some point in the future, perhaps cloud-init wouldn't do it at -## all, but rather the vendor actually provide vendor-data that accomplished -## their desires. (That is the point of vendor-data). -## -## cloud-init does cheat a bit, and write the operator-script and user-script -## itself. It could have the vendor-script do that, but it seems better -## to not require the image to contain a tool (mdata-get) to read those -## keys when we have a perfectly good one inside cloud-init. +# builtin vendor-data is a boothook that writes a script into +# /var/lib/cloud/scripts/per-boot. *That* script then handles +# executing the 'operator-script' and 'user-script' files +# that cloud-init writes into /var/lib/cloud/instance/data/ +# if they exist. +# +# This is all very indirect, but its done like this so that at +# some point in the future, perhaps cloud-init wouldn't do it at +# all, but rather the vendor actually provide vendor-data that accomplished +# their desires. (That is the point of vendor-data). +# +# cloud-init does cheat a bit, and write the operator-script and user-script +# itself. It could have the vendor-script do that, but it seems better +# to not require the image to contain a tool (mdata-get) to read those +# keys when we have a perfectly good one inside cloud-init. BUILTIN_VENDOR_DATA = """\ #cloud-boothook #!/bin/sh diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py index fef4d460..7c7ef9ab 100644 --- a/cloudinit/sources/__init__.py +++ b/cloudinit/sources/__init__.py @@ -66,7 +66,7 @@ class DataSource(object): name = name[0:-3] self.ds_cfg = util.get_cfg_by_path(self.sys_cfg, - ("datasource", name), {}) + ("datasource", name), {}) if not ud_proc: self.ud_proc = ud.UserDataProcessor(self.paths) else: @@ -166,7 +166,7 @@ class DataSource(object): defhost = "localhost" domain = defdomain - if not self.metadata or not 'local-hostname' in self.metadata: + if not self.metadata or 'local-hostname' not in self.metadata: # this is somewhat questionable really. # the cloud datasource was asked for a hostname # and didn't have one. raising error might be more appropriate @@ -272,9 +272,9 @@ def list_sources(cfg_list, depends, pkg_list): for ds_name in cfg_list: if not ds_name.startswith(DS_PREFIX): ds_name = '%s%s' % (DS_PREFIX, ds_name) - m_locs = importer.find_module(ds_name, - pkg_list, - ['get_datasource_list']) + m_locs, _looked_locs = importer.find_module(ds_name, + pkg_list, + ['get_datasource_list']) for m_loc in m_locs: mod = importer.import_module(m_loc) lister = getattr(mod, "get_datasource_list") diff --git a/cloudinit/sources/helpers/openstack.py b/cloudinit/sources/helpers/openstack.py index 0fac0335..b7e19314 100644 --- a/cloudinit/sources/helpers/openstack.py +++ b/cloudinit/sources/helpers/openstack.py @@ -21,6 +21,7 @@ import abc import base64 import copy +import functools import os from cloudinit import ec2_utils @@ -48,6 +49,7 @@ OS_LATEST = 'latest' OS_FOLSOM = '2012-08-10' OS_GRIZZLY = '2013-04-04' OS_HAVANA = '2013-10-17' +# keep this in chronological order. new supported versions go at the end. OS_VERSIONS = ( OS_FOLSOM, OS_GRIZZLY, @@ -150,17 +152,40 @@ class BaseReader(object): pass @abc.abstractmethod - def _path_exists(self, path): + def _path_read(self, path): pass @abc.abstractmethod - def _path_read(self, path): + def _fetch_available_versions(self): pass @abc.abstractmethod def _read_ec2_metadata(self): pass + def _find_working_version(self): + try: + versions_available = self._fetch_available_versions() + except Exception as e: + LOG.debug("Unable to read openstack versions from %s due to: %s", + self.base_path, e) + versions_available = [] + + # openstack.OS_VERSIONS is stored in chronological order, so + # reverse it to check newest first. + supported = [v for v in reversed(list(OS_VERSIONS))] + selected_version = OS_LATEST + + for potential_version in supported: + if potential_version not in versions_available: + continue + selected_version = potential_version + break + + LOG.debug("Selected version '%s' from %s", selected_version, + versions_available) + return selected_version + def _read_content_path(self, item): path = item.get('content_path', '').lstrip("/") path_pieces = path.split("/") @@ -170,24 +195,7 @@ class BaseReader(object): path = self._path_join(self.base_path, "openstack", *path_pieces) return self._path_read(path) - def _find_working_version(self, version): - search_versions = [version] + list(OS_VERSIONS) - for potential_version in search_versions: - if not potential_version: - continue - path = self._path_join(self.base_path, "openstack", - potential_version) - if self._path_exists(path): - if potential_version != version: - LOG.debug("Version '%s' not available, attempting to use" - " version '%s' instead", version, - potential_version) - return potential_version - LOG.debug("Version '%s' not available, attempting to use '%s'" - " instead", version, OS_LATEST) - return OS_LATEST - - def read_v2(self, version=None): + def read_v2(self): """Reads a version 2 formatted location. Return a dict with metadata, userdata, ec2-metadata, dsmode, @@ -196,6 +204,9 @@ class BaseReader(object): If not a valid location, raise a NonReadable exception. """ + load_json_anytype = functools.partial( + util.load_json, root_types=(dict, basestring, list)) + def datafiles(version): files = {} files['metadata'] = ( @@ -214,29 +225,32 @@ class BaseReader(object): files['vendordata'] = ( self._path_join("openstack", version, 'vendor_data.json'), False, - util.load_json, + load_json_anytype, ) return files - version = self._find_working_version(version) results = { 'userdata': '', 'version': 2, } - data = datafiles(version) + data = datafiles(self._find_working_version()) for (name, (path, required, translator)) in data.iteritems(): path = self._path_join(self.base_path, path) data = None found = False - if self._path_exists(path): - try: - data = self._path_read(path) - except IOError: - raise NonReadable("Failed to read: %s" % path) - found = True + try: + data = self._path_read(path) + except IOError as e: + if not required: + LOG.debug("Failed reading optional path %s due" + " to: %s", path, e) + else: + LOG.debug("Failed reading mandatory path %s due" + " to: %s", path, e) else: - if required: - raise NonReadable("Missing mandatory path: %s" % path) + found = True + if required and not found: + raise NonReadable("Missing mandatory path: %s" % path) if found and translator: try: data = translator(data) @@ -304,21 +318,27 @@ class BaseReader(object): class ConfigDriveReader(BaseReader): def __init__(self, base_path): super(ConfigDriveReader, self).__init__(base_path) + self._versions = None def _path_join(self, base, *add_ons): components = [base] + list(add_ons) return os.path.join(*components) - def _path_exists(self, path): - return os.path.exists(path) - def _path_read(self, path): return util.load_file(path) + def _fetch_available_versions(self): + if self._versions is None: + path = self._path_join(self.base_path, 'openstack') + found = [d for d in os.listdir(path) + if os.path.isdir(os.path.join(path))] + self._versions = found + return self._versions + def _read_ec2_metadata(self): path = self._path_join(self.base_path, 'ec2', 'latest', 'meta-data.json') - if not self._path_exists(path): + if not os.path.exists(path): return {} else: try: @@ -338,7 +358,7 @@ class ConfigDriveReader(BaseReader): found = {} for name in FILES_V1.keys(): path = self._path_join(self.base_path, name) - if self._path_exists(path): + if os.path.exists(path): found[name] = path if len(found) == 0: raise NonReadable("%s: no files found" % (self.base_path)) @@ -400,17 +420,26 @@ class MetadataReader(BaseReader): self.ssl_details = ssl_details self.timeout = float(timeout) self.retries = int(retries) + self._versions = None + + def _fetch_available_versions(self): + # <baseurl>/openstack/ returns a newline separated list of versions + if self._versions is not None: + return self._versions + found = [] + version_path = self._path_join(self.base_path, "openstack") + content = self._path_read(version_path) + for line in content.splitlines(): + line = line.strip() + if not line: + continue + found.append(line) + self._versions = found + return self._versions def _path_read(self, path): - response = url_helper.readurl(path, - retries=self.retries, - ssl_details=self.ssl_details, - timeout=self.timeout) - return response.contents - def _path_exists(self, path): - - def should_retry_cb(request, cause): + def should_retry_cb(_request_args, cause): try: code = int(cause.code) if code >= 400: @@ -420,15 +449,12 @@ class MetadataReader(BaseReader): pass return True - try: - response = url_helper.readurl(path, - retries=self.retries, - ssl_details=self.ssl_details, - timeout=self.timeout, - exception_cb=should_retry_cb) - return response.ok() - except IOError: - return False + response = url_helper.readurl(path, + retries=self.retries, + ssl_details=self.ssl_details, + timeout=self.timeout, + exception_cb=should_retry_cb) + return response.contents def _path_join(self, base, *add_ons): return url_helper.combine_url(base, *add_ons) @@ -437,3 +463,28 @@ class MetadataReader(BaseReader): return ec2_utils.get_instance_metadata(ssl_details=self.ssl_details, timeout=self.timeout, retries=self.retries) + + +def convert_vendordata_json(data, recurse=True): + """ data: a loaded json *object* (strings, arrays, dicts). + return something suitable for cloudinit vendordata_raw. + + if data is: + None: return None + string: return string + list: return data + the list is then processed in UserDataProcessor + dict: return convert_vendordata_json(data.get('cloud-init')) + """ + if not data: + return None + if isinstance(data, (str, unicode, basestring)): + return data + if isinstance(data, list): + return copy.deepcopy(data) + if isinstance(data, dict): + if recurse is True: + return convert_vendordata_json(data.get('cloud-init'), + recurse=False) + raise ValueError("vendordata['cloud-init'] cannot be dict") + raise ValueError("Unknown data type for vendordata: %s" % type(data)) diff --git a/cloudinit/stages.py b/cloudinit/stages.py index 58349ffc..67f467f7 100644 --- a/cloudinit/stages.py +++ b/cloudinit/stages.py @@ -386,19 +386,19 @@ class Init(object): potential_handlers = util.find_modules(path) for (fname, mod_name) in potential_handlers.iteritems(): try: - mod_locs = importer.find_module(mod_name, [''], - ['list_types', - 'handle_part']) + mod_locs, looked_locs = importer.find_module( + mod_name, [''], ['list_types', 'handle_part']) if not mod_locs: - LOG.warn(("Could not find a valid user-data handler" - " named %s in file %s"), mod_name, fname) + LOG.warn("Could not find a valid user-data handler" + " named %s in file %s (searched %s)", + mod_name, fname, looked_locs) continue mod = importer.import_module(mod_locs[0]) mod = handlers.fixup_handler(mod) types = c_handlers.register(mod) if types: - LOG.debug("Added custom handler for %s from %s", - types, fname) + LOG.debug("Added custom handler for %s [%s] from %s", + types, mod, fname) except Exception: util.logexc(LOG, "Failed to register handler from %s", fname) @@ -621,11 +621,11 @@ class Modules(object): " has an unknown frequency %s"), raw_name, freq) # Reset it so when ran it will get set to a known value freq = None - mod_locs = importer.find_module(mod_name, - ['', type_utils.obj_name(config)], - ['handle']) + mod_locs, looked_locs = importer.find_module( + mod_name, ['', type_utils.obj_name(config)], ['handle']) if not mod_locs: - LOG.warn("Could not find module named %s", mod_name) + LOG.warn("Could not find module named %s (searched %s)", + mod_name, looked_locs) continue mod = config.fixup_module(importer.import_module(mod_locs[0])) mostly_mods.append([mod, raw_name, freq, run_args]) @@ -642,8 +642,10 @@ class Modules(object): # Try the modules frequency, otherwise fallback to a known one if not freq: freq = mod.frequency - if not freq in FREQUENCIES: + if freq not in FREQUENCIES: freq = PER_INSTANCE + LOG.debug("Running module %s (%s) with frequency %s", + name, mod, freq) # Use the configs logger and not our own # TODO(harlowja): possibly check the module @@ -657,7 +659,7 @@ class Modules(object): run_name = "config-%s" % (name) cc.run(run_name, mod.handle, func_args, freq=freq) except Exception as e: - util.logexc(LOG, "Running %s (%s) failed", name, mod) + util.logexc(LOG, "Running module %s (%s) failed", name, mod) failures.append((name, e)) return (which_ran, failures) diff --git a/cloudinit/templater.py b/cloudinit/templater.py index 77af1270..02f6261d 100644 --- a/cloudinit/templater.py +++ b/cloudinit/templater.py @@ -20,13 +20,119 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -from Cheetah.Template import Template +import collections +import re +try: + from Cheetah.Template import Template as CTemplate + CHEETAH_AVAILABLE = True +except (ImportError, AttributeError): + CHEETAH_AVAILABLE = False + +try: + import jinja2 + from jinja2 import Template as JTemplate + JINJA_AVAILABLE = True +except (ImportError, AttributeError): + JINJA_AVAILABLE = False + +from cloudinit import log as logging +from cloudinit import type_utils as tu from cloudinit import util +LOG = logging.getLogger(__name__) +TYPE_MATCHER = re.compile(r"##\s*template:(.*)", re.I) +BASIC_MATCHER = re.compile(r'\$\{([A-Za-z0-9_.]+)\}|\$([A-Za-z0-9_.]+)') + + +def basic_render(content, params): + """This does simple replacement of bash variable like templates. + + It identifies patterns like ${a} or $a and can also identify patterns like + ${a.b} or $a.b which will look for a key 'b' in the dictionary rooted + by key 'a'. + """ + + def replacer(match): + # Only 1 of the 2 groups will actually have a valid entry. + name = match.group(1) + if name is None: + name = match.group(2) + if name is None: + raise RuntimeError("Match encountered but no valid group present") + path = collections.deque(name.split(".")) + selected_params = params + while len(path) > 1: + key = path.popleft() + if not isinstance(selected_params, dict): + raise TypeError("Can not traverse into" + " non-dictionary '%s' of type %s while" + " looking for subkey '%s'" + % (selected_params, + tu.obj_name(selected_params), + key)) + selected_params = selected_params[key] + key = path.popleft() + if not isinstance(selected_params, dict): + raise TypeError("Can not extract key '%s' from non-dictionary" + " '%s' of type %s" + % (key, selected_params, + tu.obj_name(selected_params))) + return str(selected_params[key]) + + return BASIC_MATCHER.sub(replacer, content) + + +def detect_template(text): + + def cheetah_render(content, params): + return CTemplate(content, searchList=[params]).respond() + + def jinja_render(content, params): + return JTemplate(content, + undefined=jinja2.StrictUndefined, + trim_blocks=True).render(**params) + + if text.find("\n") != -1: + ident, rest = text.split("\n", 1) + else: + ident = text + rest = '' + type_match = TYPE_MATCHER.match(ident) + if not type_match: + if not CHEETAH_AVAILABLE: + LOG.warn("Cheetah not available as the default renderer for" + " unknown template, reverting to the basic renderer.") + return ('basic', basic_render, text) + else: + return ('cheetah', cheetah_render, text) + else: + template_type = type_match.group(1).lower().strip() + if template_type not in ('jinja', 'cheetah', 'basic'): + raise ValueError("Unknown template rendering type '%s' requested" + % template_type) + if template_type == 'jinja' and not JINJA_AVAILABLE: + LOG.warn("Jinja not available as the selected renderer for" + " desired template, reverting to the basic renderer.") + return ('basic', basic_render, rest) + elif template_type == 'jinja' and JINJA_AVAILABLE: + return ('jinja', jinja_render, rest) + if template_type == 'cheetah' and not CHEETAH_AVAILABLE: + LOG.warn("Cheetah not available as the selected renderer for" + " desired template, reverting to the basic renderer.") + return ('basic', basic_render, rest) + elif template_type == 'cheetah' and CHEETAH_AVAILABLE: + return ('cheetah', cheetah_render, rest) + # Only thing left over is the basic renderer (it is always available). + return ('basic', basic_render, rest) + def render_from_file(fn, params): - return render_string(util.load_file(fn), params) + if not params: + params = {} + template_type, renderer, content = detect_template(util.load_file(fn)) + LOG.debug("Rendering content of '%s' using renderer %s", fn, template_type) + return renderer(content, params) def render_to_file(fn, outfn, params, mode=0644): @@ -37,4 +143,5 @@ def render_to_file(fn, outfn, params, mode=0644): def render_string(content, params): if not params: params = {} - return Template(content, searchList=[params]).respond() + template_type, renderer, content = detect_template(content) + return renderer(content, params) diff --git a/cloudinit/type_utils.py b/cloudinit/type_utils.py index 2decbfc5..cc3d9495 100644 --- a/cloudinit/type_utils.py +++ b/cloudinit/type_utils.py @@ -19,8 +19,6 @@ # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# pylint: disable=C0302 import types diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py index 4a83169a..3074dd08 100644 --- a/cloudinit/url_helper.py +++ b/cloudinit/url_helper.py @@ -44,7 +44,7 @@ try: from distutils.version import LooseVersion import pkg_resources _REQ = pkg_resources.get_distribution('requests') - _REQ_VER = LooseVersion(_REQ.version) # pylint: disable=E1103 + _REQ_VER = LooseVersion(_REQ.version) if _REQ_VER >= LooseVersion('0.8.8'): SSL_ENABLED = True if _REQ_VER >= LooseVersion('0.7.0') and _REQ_VER < LooseVersion('1.0.0'): @@ -54,7 +54,7 @@ except: def _cleanurl(url): - parsed_url = list(urlparse(url, scheme='http')) # pylint: disable=E1123 + parsed_url = list(urlparse(url, scheme='http')) if not parsed_url[1] and parsed_url[2]: # Swap these since this seems to be a common # occurrence when given urls like 'www.google.com' @@ -90,7 +90,7 @@ class StringResponse(object): self.contents = contents self.url = None - def ok(self, *args, **kwargs): # pylint: disable=W0613 + def ok(self, *args, **kwargs): if self.code != 200: return False return True @@ -150,7 +150,7 @@ class UrlError(IOError): def _get_ssl_args(url, ssl_details): ssl_args = {} - scheme = urlparse(url).scheme # pylint: disable=E1101 + scheme = urlparse(url).scheme if scheme == 'https' and ssl_details: if not SSL_ENABLED: LOG.warn("SSL is not supported in requests v%s, " @@ -227,18 +227,17 @@ def readurl(url, data=None, timeout=None, retries=0, sec_between=1, r = requests.request(**req_args) if check_status: - r.raise_for_status() # pylint: disable=E1103 + r.raise_for_status() LOG.debug("Read from %s (%s, %sb) after %s attempts", url, - r.status_code, len(r.content), # pylint: disable=E1103 - (i + 1)) + r.status_code, len(r.content), (i + 1)) # Doesn't seem like we can make it use a different # subclass for responses, so add our own backward-compat # attrs return UrlResponse(r) except exceptions.RequestException as e: if (isinstance(e, (exceptions.HTTPError)) - and hasattr(e, 'response') # This appeared in v 0.10.8 - and hasattr(e.response, 'status_code')): + and hasattr(e, 'response') # This appeared in v 0.10.8 + and hasattr(e.response, 'status_code')): excps.append(UrlError(e, code=e.response.status_code, headers=e.response.headers)) else: diff --git a/cloudinit/util.py b/cloudinit/util.py index 06039ee2..946059e9 100644 --- a/cloudinit/util.py +++ b/cloudinit/util.py @@ -19,8 +19,6 @@ # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -# -# pylint: disable=C0302 from StringIO import StringIO @@ -42,7 +40,7 @@ import re import shutil import socket import stat -import string # pylint: disable=W0402 +import string import subprocess import sys import tempfile @@ -146,23 +144,23 @@ class SeLinuxGuard(object): return False def __exit__(self, excp_type, excp_value, excp_traceback): - if self.selinux and self.selinux.is_selinux_enabled(): - path = os.path.realpath(os.path.expanduser(self.path)) - # path should be a string, not unicode - path = str(path) - do_restore = False - try: - # See if even worth restoring?? - stats = os.lstat(path) - if stat.ST_MODE in stats: - self.selinux.matchpathcon(path, stats[stat.ST_MODE]) - do_restore = True - except OSError: - pass - if do_restore: - LOG.debug("Restoring selinux mode for %s (recursive=%s)", - path, self.recursive) - self.selinux.restorecon(path, recursive=self.recursive) + if not self.selinux or not self.selinux.is_selinux_enabled(): + return + if not os.path.lexists(self.path): + return + + path = os.path.realpath(self.path) + # path should be a string, not unicode + path = str(path) + try: + stats = os.lstat(path) + self.selinux.matchpathcon(path, stats[stat.ST_MODE]) + except OSError: + return + + LOG.debug("Restoring selinux mode for %s (recursive=%s)", + path, self.recursive) + self.selinux.restorecon(path, recursive=self.recursive) class MountFailedError(Exception): @@ -193,16 +191,16 @@ def ExtendedTemporaryFile(**kwargs): return fh -def fork_cb(child_cb, *args): +def fork_cb(child_cb, *args, **kwargs): fid = os.fork() if fid == 0: try: - child_cb(*args) - os._exit(0) # pylint: disable=W0212 + child_cb(*args, **kwargs) + os._exit(0) except: logexc(LOG, "Failed forking and calling callback %s", type_utils.obj_name(child_cb)) - os._exit(1) # pylint: disable=W0212 + os._exit(1) else: LOG.debug("Forked child %s who will run callback %s", fid, type_utils.obj_name(child_cb)) @@ -423,7 +421,7 @@ def get_cfg_option_list(yobj, key, default=None): @return: The configuration option as a list of strings or default if key is not found. """ - if not key in yobj: + if key not in yobj: return default if yobj[key] is None: return [] @@ -487,7 +485,7 @@ def redirect_output(outfmt, errfmt, o_out=None, o_err=None): new_fp = open(arg, owith) elif mode == "|": proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE) - new_fp = proc.stdin # pylint: disable=E1101 + new_fp = proc.stdin else: raise TypeError("Invalid type for output format: %s" % outfmt) @@ -509,7 +507,7 @@ def redirect_output(outfmt, errfmt, o_out=None, o_err=None): new_fp = open(arg, owith) elif mode == "|": proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE) - new_fp = proc.stdin # pylint: disable=E1101 + new_fp = proc.stdin else: raise TypeError("Invalid type for error format: %s" % errfmt) @@ -937,7 +935,7 @@ def is_resolvable(name): should also not exist. The random entry will be resolved inside the search list. """ - global _DNS_REDIRECT_IP # pylint: disable=W0603 + global _DNS_REDIRECT_IP if _DNS_REDIRECT_IP is None: badips = set() badnames = ("does-not-exist.example.com.", "example.invalid.", @@ -1148,7 +1146,7 @@ def chownbyname(fname, user=None, group=None): # this returns the specific 'mode' entry, cleanly formatted, with value def get_output_cfg(cfg, mode): ret = [None, None] - if not cfg or not 'output' in cfg: + if cfg or 'output' not in cfg: return ret outcfg = cfg['output'] @@ -1532,7 +1530,7 @@ def subp(args, data=None, rcs=None, env=None, capture=True, shell=False, (out, err) = sp.communicate(data) except OSError as e: raise ProcessExecutionError(cmd=args, reason=e) - rc = sp.returncode # pylint: disable=E1101 + rc = sp.returncode if rc not in rcs: raise ProcessExecutionError(stdout=out, stderr=err, exit_code=rc, @@ -1745,7 +1743,7 @@ def parse_mount_info(path, mountinfo_lines, log=LOG): # Ignore mount points higher than an already seen mount # point. if (match_mount_point_elements is not None and - len(match_mount_point_elements) > len(mount_point_elements)): + len(match_mount_point_elements) > len(mount_point_elements)): continue # Find the '-' which terminates a list of optional columns to diff --git a/config/cloud.cfg b/config/cloud.cfg index b746e3db..200050d3 100644 --- a/config/cloud.cfg +++ b/config/cloud.cfg @@ -24,6 +24,7 @@ preserve_hostname: false # The modules that run in the 'init' stage cloud_init_modules: - migrator + - ubuntu-init-switch - seed_random - bootcmd - write-files diff --git a/config/cloud.cfg-freebsd b/config/cloud.cfg-freebsd new file mode 100644 index 00000000..bb3a4a51 --- /dev/null +++ b/config/cloud.cfg-freebsd @@ -0,0 +1,88 @@ +# The top level settings are used as module +# and system configuration. + +syslog_fix_perms: root:wheel + +# This should not be required, but leave it in place until the real cause of +# not beeing able to find -any- datasources is resolved. +datasource_list: ['OpenStack'] + +# A set of users which may be applied and/or used by various modules +# when a 'default' entry is found it will reference the 'default_user' +# from the distro configuration specified below +users: + - default + +# If this is set, 'root' will not be able to ssh in and they +# will get a message to login instead as the above $user (ubuntu) +disable_root: false + +# This will cause the set+update hostname module to not operate (if true) +preserve_hostname: false + +# Example datasource config +# datasource: +# Ec2: +# metadata_urls: [ 'blah.com' ] +# timeout: 5 # (defaults to 50 seconds) +# max_wait: 10 # (defaults to 120 seconds) + +# The modules that run in the 'init' stage +cloud_init_modules: +# - migrator + - seed_random + - bootcmd +# - write-files + - growpart + - resizefs + - set_hostname + - update_hostname +# - update_etc_hosts +# - ca-certs +# - rsyslog + - users-groups + - ssh + +# The modules that run in the 'config' stage +cloud_config_modules: +# - disk_setup +# - mounts + - ssh-import-id + - locale +# - set-passwords +# - package-update-upgrade-install +# - landscape +# - timezone +# - puppet +# - chef +# - salt-minion +# - mcollective + - disable-ec2-metadata + - runcmd +# - byobu + +# The modules that run in the 'final' stage +cloud_final_modules: + - rightscale_userdata + - scripts-vendor + - scripts-per-once + - scripts-per-boot + - scripts-per-instance + - scripts-user + - ssh-authkey-fingerprints + - keys-to-console + - phone-home + - final-message + - power-state-change + +# System and/or distro specific settings +# (not accessible to handlers/transforms) +system_info: + distro: freebsd + default_user: + name: beastie + lock_passwd: True + gecos: FreeBSD + groups: [wheel] + sudo: ["ALL=(ALL) NOPASSWD:ALL"] + shell: /bin/sh diff --git a/doc/examples/cloud-config-disk-setup.txt b/doc/examples/cloud-config-disk-setup.txt index 0dfef8e8..3e46a22e 100644 --- a/doc/examples/cloud-config-disk-setup.txt +++ b/doc/examples/cloud-config-disk-setup.txt @@ -7,7 +7,7 @@ disk_setup: ephmeral0: - type: 'mbr' + table_type: 'mbr' layout: True overwrite: False @@ -23,7 +23,7 @@ fs_setup: device_aliases: {'ephemeral0': '/dev/sdb'} disk_setup: ephemeral0: - type: mbr + table_type: mbr layout: True overwrite: False @@ -40,7 +40,7 @@ fs_setup: device_aliases: {'ephemeral0': '/dev/sdb'} disk_setup: ephemeral0: - type: mbr + table_type: mbr layout: False overwrite: False @@ -63,10 +63,10 @@ fs_setup: disk_setup: ephmeral0: - type: 'mbr' + table_type: 'mbr' layout: 'auto' /dev/xvdh: - type: 'mbr' + table_type: 'mbr' layout: - 33 - [33, 82] @@ -79,7 +79,7 @@ disk_setup: # The general format is: # disk_setup: # <DEVICE>: -# type: 'mbr' +# table_type: 'mbr' # layout: <LAYOUT|BOOL> # overwrite: <BOOL> # @@ -96,7 +96,7 @@ disk_setup: # Note: At this time, there is no handling or setup of # device mapper targets. # -# type=<TYPE>: Currently the following are supported: +# table_type=<TYPE>: Currently the following are supported: # 'mbr': default and setups a MS-DOS partition table # # Note: At this time only 'mbr' partition tables are allowed. @@ -116,7 +116,7 @@ disk_setup: # partition having a swap label, taking 1/3 of the disk space # and the remainder being used as the second partition. # /dev/xvdh': -# type: 'mbr' +# table_type: 'mbr' # layout: # - [33,82] # - 66 diff --git a/packages/bddeb b/packages/bddeb index 9552aa40..9d264f92 100755 --- a/packages/bddeb +++ b/packages/bddeb @@ -31,6 +31,7 @@ PKG_MP = { 'argparse': 'python-argparse', 'cheetah': 'python-cheetah', 'configobj': 'python-configobj', + 'jinja2': 'python-jinja2', 'jsonpatch': 'python-jsonpatch | python-json-patch', 'oauth': 'python-oauth', 'prettytable': 'python-prettytable', @@ -38,7 +39,7 @@ PKG_MP = { 'pyyaml': 'python-yaml', 'requests': 'python-requests', } -DEBUILD_ARGS = ["-us", "-S", "-uc", "-d"] +DEBUILD_ARGS = ["-S", "-d"] def write_debian_folder(root, version, revno, append_requires=[]): @@ -75,7 +76,7 @@ def write_debian_folder(root, version, revno, append_requires=[]): params={'requires': requires}) # Just copy the following directly - for base_fn in ['dirs', 'copyright', 'compat', 'pycompat', 'rules']: + for base_fn in ['dirs', 'copyright', 'compat', 'rules']: shutil.copy(util.abs_join(find_root(), 'packages', 'debian', base_fn), util.abs_join(deb_dir, base_fn)) @@ -98,15 +99,23 @@ def main(): parser.add_argument("--init-system", dest="init_system", help=("build deb with INIT_SYSTEM=xxx" " (default: %(default)s"), - default=os.environ.get("INIT_SYSTEM", "upstart")) + default=os.environ.get("INIT_SYSTEM", + "upstart,systemd")) for ent in DEBUILD_ARGS: parser.add_argument(ent, dest="debuild_args", action='append_const', - const=ent, help=("pass through '%s' to debuild" % ent)) + const=ent, help=("pass through '%s' to debuild" % ent), + default=[]) + + parser.add_argument("--sign", default=False, action='store_true', + help="sign result. do not pass -us -uc to debuild") args = parser.parse_args() + if not args.sign: + args.debuild_args.extend(['-us', '-uc']) + os.environ['INIT_SYSTEM'] = args.init_system capture = True @@ -166,7 +175,8 @@ def main(): print("Copied that archive to %r for local usage (if desired)." % (util.abs_join(os.getcwd(), tar_fn))) - print("Running 'debuild' in %r" % (xdir)) + print("Running 'debuild %s' in %r" % (' '.join(args.debuild_args), + xdir)) with util.chdir(xdir): cmd = ['debuild', '--preserve-envvar', 'INIT_SYSTEM'] if args.debuild_args: diff --git a/packages/brpm b/packages/brpm index f8ba1db1..b8bbff9d 100755 --- a/packages/brpm +++ b/packages/brpm @@ -37,6 +37,7 @@ PKG_MP = { 'redhat': { 'argparse': 'python-argparse', 'cheetah': 'python-cheetah', + 'jinja2': 'python-jinja2', 'configobj': 'python-configobj', 'jsonpatch': 'python-jsonpatch', 'oauth': 'python-oauth', diff --git a/packages/debian/compat b/packages/debian/compat index 7ed6ff82..ec635144 100644 --- a/packages/debian/compat +++ b/packages/debian/compat @@ -1 +1 @@ -5 +9 diff --git a/packages/debian/control.in b/packages/debian/control.in index 7e42b94b..9207e5f4 100644 --- a/packages/debian/control.in +++ b/packages/debian/control.in @@ -1,18 +1,22 @@ ## This is a cheetah template Source: cloud-init Section: admin -Priority: extra +Priority: optional Maintainer: Scott Moser <smoser@ubuntu.com> -Build-Depends: cdbs, - debhelper (>= 5.0.38), +Build-Depends: debhelper (>= 9), + dh-python, + dh-systemd, python (>= 2.6.6-3~), python-nose, pyflakes, - pylint, python-setuptools, + python-selinux, python-cheetah, python-mocker, - python-setuptools + python-httpretty, +#for $r in $requires + ${r}, +#end for XS-Python-Version: all Standards-Version: 3.9.3 diff --git a/packages/debian/pycompat b/packages/debian/pycompat deleted file mode 100644 index 0cfbf088..00000000 --- a/packages/debian/pycompat +++ /dev/null @@ -1 +0,0 @@ -2 diff --git a/packages/debian/rules b/packages/debian/rules index 7623ac9d..9e0c5ddb 100755 --- a/packages/debian/rules +++ b/packages/debian/rules @@ -1,18 +1,17 @@ #!/usr/bin/make -f -DEB_PYTHON2_MODULE_PACKAGES = cloud-init -INIT_SYSTEM ?= upstart +INIT_SYSTEM ?= upstart,systemd +export PYBUILD_INSTALL_ARGS=--init-system=$(INIT_SYSTEM) -binary-install/cloud-init::cloud-init-fixups +%: + dh $@ --with python2,systemd --buildsystem pybuild -include /usr/share/cdbs/1/rules/debhelper.mk -include /usr/share/cdbs/1/class/python-distutils.mk +override_dh_install: + dh_install + install -d debian/cloud-init/etc/rsyslog.d + cp tools/21-cloudinit.conf debian/cloud-init/etc/rsyslog.d/21-cloudinit.conf -DEB_PYTHON_INSTALL_ARGS_ALL += --init-system=$(INIT_SYSTEM) - -DEB_DH_INSTALL_SOURCEDIR := debian/tmp - -cloud-init-fixups: - install -d $(DEB_DESTDIR)/etc/rsyslog.d - cp tools/21-cloudinit.conf $(DEB_DESTDIR)/etc/rsyslog.d/21-cloudinit.conf - +override_dh_auto_test: + # Becuase setup tools didn't copy data... + cp -r tests/data .pybuild/pythonX.Y_2.7/build/tests + http_proxy= dh_auto_test -- --test-nose diff --git a/pylintrc b/pylintrc deleted file mode 100644 index ee886510..00000000 --- a/pylintrc +++ /dev/null @@ -1,19 +0,0 @@ -[General] -init-hook='import sys; sys.path.append("tests/")' - -[MESSAGES CONTROL] -# See: http://pylint-messages.wikidot.com/all-codes -# W0142: *args and **kwargs are fine. -# W0511: TODOs in code comments are fine. -# W0702: No exception type(s) specified -# W0703: Catch "Exception" -# C0103: Invalid name -# C0111: Missing docstring -disable=W0142,W0511,W0702,W0703,C0103,C0111 - -[REPORTS] -reports=no -include-ids=yes - -[FORMAT] -max-line-length=79 diff --git a/requirements.txt b/requirements.txt index fdcbd143..943dbef7 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,6 +2,7 @@ # Used for untemplating any files or strings with parameters. cheetah +jinja2 # This is used for any pretty printing of tabular data. PrettyTable @@ -36,21 +36,6 @@ def is_f(p): return os.path.isfile(p) -INITSYS_FILES = { - 'sysvinit': [f for f in glob('sysvinit/redhat/*') if is_f(f)], - 'sysvinit_deb': [f for f in glob('sysvinit/debian/*') if is_f(f)], - 'systemd': [f for f in glob('systemd/*') if is_f(f)], - 'upstart': [f for f in glob('upstart/*') if is_f(f)], -} -INITSYS_ROOTS = { - 'sysvinit': '/etc/rc.d/init.d', - 'sysvinit_deb': '/etc/init.d', - 'systemd': '/etc/systemd/system/', - 'upstart': '/etc/init/', -} -INITSYS_TYPES = sorted(list(INITSYS_ROOTS.keys())) - - def tiny_p(cmd, capture=True): # Darn python 2.6 doesn't have check_output (argggg) stdout = subprocess.PIPE @@ -61,13 +46,46 @@ def tiny_p(cmd, capture=True): sp = subprocess.Popen(cmd, stdout=stdout, stderr=stderr, stdin=None) (out, err) = sp.communicate() - ret = sp.returncode # pylint: disable=E1101 + ret = sp.returncode if ret not in [0]: raise RuntimeError("Failed running %s [rc=%s] (%s, %s)" % (cmd, ret, out, err)) return (out, err) +def systemd_unitdir(): + cmd = ['pkg-config', '--variable=systemdsystemunitdir', 'systemd'] + try: + (path, err) = tiny_p(cmd) + except: + return '/lib/systemd/system' + return str(path).strip() + +INITSYS_FILES = { + 'sysvinit': [f for f in glob('sysvinit/redhat/*') if is_f(f)], + 'sysvinit_freebsd': [f for f in glob('sysvinit/freebsd/*') if is_f(f)], + 'sysvinit_deb': [f for f in glob('sysvinit/debian/*') if is_f(f)], + 'systemd': [f for f in glob('systemd/*') if is_f(f)], + 'upstart': [f for f in glob('upstart/*') if is_f(f)], +} +INITSYS_ROOTS = { + 'sysvinit': '/etc/rc.d/init.d', + 'sysvinit_freebsd': '/usr/local/etc/rc.d', + 'sysvinit_deb': '/etc/init.d', + 'systemd': systemd_unitdir(), + 'upstart': '/etc/init/', +} +INITSYS_TYPES = sorted(list(INITSYS_ROOTS.keys())) + +# Install everything in the right location and take care of Linux (default) and +# FreeBSD systems. +USR = "/usr" +ETC = "/etc" +if os.uname()[0] == 'FreeBSD': + USR = "/usr/local" + ETC = "/usr/local/etc" + + def get_version(): cmd = ['tools/read-version'] (ver, _e) = tiny_p(cmd) @@ -86,26 +104,35 @@ class InitsysInstallData(install): user_options = install.user_options + [ # This will magically show up in member variable 'init_sys' ('init-system=', None, - ('init system to configure (%s) [default: None]') % + ('init system(s) to configure (%s) [default: None]') % (", ".join(INITSYS_TYPES)) ), ] def initialize_options(self): install.initialize_options(self) - self.init_system = None + self.init_system = "" def finalize_options(self): install.finalize_options(self) - if self.init_system and self.init_system not in INITSYS_TYPES: + + if self.init_system and isinstance(self.init_system, str): + self.init_system = self.init_system.split(",") + + if len(self.init_system) == 0: raise DistutilsArgError(("You must specify one of (%s) when" - " specifying a init system!") % (", ".join(INITSYS_TYPES))) - elif self.init_system: + " specifying init system(s)!") % (", ".join(INITSYS_TYPES))) + + bad = [f for f in self.init_system if f not in INITSYS_TYPES] + if len(bad) != 0: + raise DistutilsArgError( + "Invalid --init-system: %s" % (','.join(bad))) + + for sys in self.init_system: self.distribution.data_files.append( - (INITSYS_ROOTS[self.init_system], - INITSYS_FILES[self.init_system])) - # Force that command to reinitalize (with new file list) - self.distribution.reinitialize_command('install_data', True) + (INITSYS_ROOTS[sys], INITSYS_FILES[sys])) + # Force that command to reinitalize (with new file list) + self.distribution.reinitialize_command('install_data', True) setuptools.setup(name='cloud-init', @@ -119,18 +146,17 @@ setuptools.setup(name='cloud-init', 'tools/cloud-init-per', ], license='GPLv3', - data_files=[('/etc/cloud', glob('config/*.cfg')), - ('/etc/cloud/cloud.cfg.d', glob('config/cloud.cfg.d/*')), - ('/etc/cloud/templates', glob('templates/*')), - ('/usr/share/cloud-init', []), - ('/usr/lib/cloud-init', + data_files=[(ETC + '/cloud', glob('config/*.cfg')), + (ETC + '/cloud/cloud.cfg.d', glob('config/cloud.cfg.d/*')), + (ETC + '/cloud/templates', glob('templates/*')), + (USR + '/lib/cloud-init', ['tools/uncloud-init', 'tools/write-ssh-key-fingerprints']), - ('/usr/share/doc/cloud-init', + (USR + '/share/doc/cloud-init', [f for f in glob('doc/*') if is_f(f)]), - ('/usr/share/doc/cloud-init/examples', + (USR + '/share/doc/cloud-init/examples', [f for f in glob('doc/examples/*') if is_f(f)]), - ('/usr/share/doc/cloud-init/examples/seed', + (USR + '/share/doc/cloud-init/examples/seed', [f for f in glob('doc/examples/seed/*') if is_f(f)]), ], install_requires=read_requires(), diff --git a/systemd/cloud-init.service b/systemd/cloud-init.service index 018a1fa8..6b0c7229 100644 --- a/systemd/cloud-init.service +++ b/systemd/cloud-init.service @@ -1,8 +1,9 @@ [Unit] Description=Initial cloud-init job (metadata service crawler) After=local-fs.target network.target cloud-init-local.service +Before=sshd.service sshd-keygen.service Requires=network.target -Wants=local-fs.target cloud-init-local.service +Wants=local-fs.target cloud-init-local.service sshd.service sshd-keygen.service [Service] Type=oneshot diff --git a/sysvinit/freebsd/cloudconfig b/sysvinit/freebsd/cloudconfig index 15d7ab95..44c216b3 100755 --- a/sysvinit/freebsd/cloudconfig +++ b/sysvinit/freebsd/cloudconfig @@ -6,28 +6,28 @@ . /etc/rc.subr +export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg + name="cloudconfig" -command="/usr/bin/cloud-init" +command="/usr/local/bin/cloud-init" start_cmd="cloudconfig_start" stop_cmd=":" rcvar="cloudinit_enable" start_precmd="cloudinit_override" start_cmd="cloudconfig_start" -: ${cloudinit_config:="/etc/cloud/cloud.cfg"} - cloudinit_override() { - # If there exist sysconfig/default variable override files use it... - if [ -f /etc/default/cloud-init ]; then - . /etc/default/cloud-init + # If there exist sysconfig/defaults variable override files use it... + if [ -f /etc/defaults/cloud-init ]; then + . /etc/defaults/cloud-init fi } cloudconfig_start() { echo "${command} starting" - ${command} ${cloudinit_config} modules --mode config + ${command} modules --mode config } load_rc_config $name diff --git a/sysvinit/freebsd/cloudfinal b/sysvinit/freebsd/cloudfinal index 49945ecd..f668e036 100755 --- a/sysvinit/freebsd/cloudfinal +++ b/sysvinit/freebsd/cloudfinal @@ -6,28 +6,28 @@ . /etc/rc.subr +export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg + name="cloudfinal" -command="/usr/bin/cloud_init" +command="/usr/local/bin/cloud-init" start_cmd="cloudfinal_start" stop_cmd=":" rcvar="cloudinit_enable" start_precmd="cloudinit_override" start_cmd="cloudfinal_start" -: ${cloudinit_config:="/etc/cloud/cloud.cfg"} - cloudinit_override() { - # If there exist sysconfig/default variable override files use it... - if [ -f /etc/default/cloud-init ]; then - . /etc/default/cloud-init + # If there exist sysconfig/defaults variable override files use it... + if [ -f /etc/defaults/cloud-init ]; then + . /etc/defaults/cloud-init fi } cloudfinal_start() { echo -n "${command} starting" - ${command} ${cloudinit_config} modules --mode final + ${command} modules --mode final } load_rc_config $name diff --git a/sysvinit/freebsd/cloudinit b/sysvinit/freebsd/cloudinit index 8d5ff10e..c5478678 100755 --- a/sysvinit/freebsd/cloudinit +++ b/sysvinit/freebsd/cloudinit @@ -6,28 +6,28 @@ . /etc/rc.subr +export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg + name="cloudinit" -command="/usr/bin/cloud_init" +command="/usr/local/bin/cloud-init" start_cmd="cloudinit_start" stop_cmd=":" rcvar="cloudinit_enable" start_precmd="cloudinit_override" start_cmd="cloudinit_start" -: ${cloudinit_config:="/etc/cloud/cloud.cfg"} - cloudinit_override() { - # If there exist sysconfig/default variable override files use it... - if [ -f /etc/default/cloud-init ]; then - . /etc/default/cloud-init + # If there exist sysconfig/defaults variable override files use it... + if [ -f /etc/defaults/cloud-init ]; then + . /etc/defaults/cloud-init fi } cloudinit_start() { echo -n "${command} starting" - ${command} ${cloudinit_config} init + ${command} init } load_rc_config $name diff --git a/sysvinit/freebsd/cloudinitlocal b/sysvinit/freebsd/cloudinitlocal index b55705c0..c340d5d0 100755 --- a/sysvinit/freebsd/cloudinitlocal +++ b/sysvinit/freebsd/cloudinitlocal @@ -6,28 +6,28 @@ . /etc/rc.subr +export CLOUD_CFG=/usr/local/etc/cloud/cloud.cfg + name="cloudinitlocal" -command="/usr/bin/cloud-init" +command="/usr/local/bin/cloud-init" start_cmd="cloudlocal_start" stop_cmd=":" rcvar="cloudinit_enable" start_precmd="cloudinit_override" start_cmd="cloudlocal_start" -: ${cloudinit_config:="/etc/cloud/cloud.cfg"} - cloudinit_override() { - # If there exist sysconfig/default variable override files use it... - if [ -f /etc/default/cloud-init ]; then - . /etc/default/cloud-init + # If there exist sysconfig/defaults variable override files use it... + if [ -f /etc/defaults/cloud-init ]; then + . /etc/defaults/cloud-init fi } cloudlocal_start() { echo -n "${command} starting" - ${command} ${cloudinit_config} init --local + ${command} init --local } load_rc_config $name diff --git a/templates/chef_client.rb.tmpl b/templates/chef_client.rb.tmpl index 7981cba7..538850ca 100644 --- a/templates/chef_client.rb.tmpl +++ b/templates/chef_client.rb.tmpl @@ -1,25 +1,25 @@ -#* - This file is only utilized if the module 'cc_chef' is enabled in - cloud-config. Specifically, in order to enable it - you need to add the following to config: - chef: - validation_key: XYZ - validation_cert: XYZ - validation_name: XYZ - server_url: XYZ -*# +## template:jinja +{# +This file is only utilized if the module 'cc_chef' is enabled in +cloud-config. Specifically, in order to enable it +you need to add the following to config: + chef: + validation_key: XYZ + validation_cert: XYZ + validation_name: XYZ + server_url: XYZ +-#} log_level :info log_location "/var/log/chef/client.log" ssl_verify_mode :verify_none -validation_client_name "$validation_name" +validation_client_name "{{validation_name}}" validation_key "/etc/chef/validation.pem" client_key "/etc/chef/client.pem" -chef_server_url "$server_url" -environment "$environment" -node_name "$node_name" +chef_server_url "{{server_url}}" +environment "{{environment}}" +node_name "{{node_name}}" json_attribs "/etc/chef/firstboot.json" file_cache_path "/var/cache/chef" file_backup_path "/var/backups/chef" pid_file "/var/run/chef/client.pid" Chef::Log::Formatter.show_time = true - diff --git a/templates/hosts.debian.tmpl b/templates/hosts.debian.tmpl index ae120b02..a1d97212 100644 --- a/templates/hosts.debian.tmpl +++ b/templates/hosts.debian.tmpl @@ -1,19 +1,19 @@ -## This file (/etc/cloud/templates/hosts.tmpl) is only utilized -## if enabled in cloud-config. Specifically, in order to enable it -## you need to add the following to config: -## manage_etc_hosts: True -## -## Note, double-hash commented lines will not appear in /etc/hosts -# +## template:jinja +{# +This file (/etc/cloud/templates/hosts.tmpl) is only utilized +if enabled in cloud-config. Specifically, in order to enable it +you need to add the following to config: + manage_etc_hosts: True +-#} # Your system has configured 'manage_etc_hosts' as True. # As a result, if you wish for changes to this file to persist # then you will need to either # a.) make changes to the master file in /etc/cloud/templates/hosts.tmpl # b.) change or remove the value of 'manage_etc_hosts' in # /etc/cloud/cloud.cfg or cloud-config from user-data -# -## The value '$hostname' will be replaced with the local-hostname -127.0.1.1 $fqdn $hostname +# +{# The value '{{hostname}}' will be replaced with the local-hostname -#} +127.0.1.1 {{fqdn}} {{hostname}} 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts @@ -23,3 +23,4 @@ ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts + diff --git a/templates/hosts.redhat.tmpl b/templates/hosts.redhat.tmpl index 80459d95..bc5da32c 100644 --- a/templates/hosts.redhat.tmpl +++ b/templates/hosts.redhat.tmpl @@ -1,9 +1,10 @@ -#* - This file /etc/cloud/templates/hosts.redhat.tmpl is only utilized - if enabled in cloud-config. Specifically, in order to enable it - you need to add the following to config: - manage_etc_hosts: True -*# +## template:jinja +{# +This file /etc/cloud/templates/hosts.redhat.tmpl is only utilized +if enabled in cloud-config. Specifically, in order to enable it +you need to add the following to config: + manage_etc_hosts: True +-#} # Your system has configured 'manage_etc_hosts' as True. # As a result, if you wish for changes to this file to persist # then you will need to either @@ -12,12 +13,12 @@ # /etc/cloud/cloud.cfg or cloud-config from user-data # # The following lines are desirable for IPv4 capable hosts -127.0.0.1 ${fqdn} ${hostname} +127.0.0.1 {{fqdn}} {{hostname}} 127.0.0.1 localhost.localdomain localhost 127.0.0.1 localhost4.localdomain4 localhost4 # The following lines are desirable for IPv6 capable hosts -::1 ${fqdn} ${hostname} +::1 {{fqdn}} {{hostname}} ::1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 diff --git a/templates/hosts.suse.tmpl b/templates/hosts.suse.tmpl index 5d3d57e4..b6082692 100644 --- a/templates/hosts.suse.tmpl +++ b/templates/hosts.suse.tmpl @@ -1,9 +1,10 @@ -#* - This file /etc/cloud/templates/hosts.suse.tmpl is only utilized - if enabled in cloud-config. Specifically, in order to enable it - you need to add the following to config: - manage_etc_hosts: True -*# +## template:jinja +{# +This file /etc/cloud/templates/hosts.suse.tmpl is only utilized +if enabled in cloud-config. Specifically, in order to enable it +you need to add the following to config: + manage_etc_hosts: True +-#} # Your system has configured 'manage_etc_hosts' as True. # As a result, if you wish for changes to this file to persist # then you will need to either @@ -22,3 +23,4 @@ ff00::0 ipv6-mcastprefix ff02::1 ipv6-allnodes ff02::2 ipv6-allrouters ff02::3 ipv6-allhosts + diff --git a/templates/resolv.conf.tmpl b/templates/resolv.conf.tmpl index b7e97b13..1300156c 100644 --- a/templates/resolv.conf.tmpl +++ b/templates/resolv.conf.tmpl @@ -1,39 +1,30 @@ -# +## template:jinja # Your system has been configured with 'manage-resolv-conf' set to true. # As a result, cloud-init has written this file with configuration data # that it has been provided. Cloud-init, by default, will write this file # a single time (PER_ONCE). # +{% if nameservers is defined %} +{% for server in nameservers %} +nameserver {{server}} +{% endfor %} -#if $varExists('nameservers') -#for $server in $nameservers -nameserver $server -#end for -#end if -#if $varExists('searchdomains') -search #slurp -#for $search in $searchdomains -$search #slurp -#end for +{% endif -%} +{% if searchdomains is defined %} +search {% for search in searchdomains %}{{search}} {% endfor %} -#end if -#if $varExists('domain') -domain $domain -#end if -#if $varExists('sortlist') -sortlist #slurp -#for $sort in $sortlist -$sort #slurp -#end for +{% endif %} +{% if domain is defined %} +domain {{domain}} +{% endif %} +{% if sortlist is defined %} -#end if -#if $varExists('options') or $varExists('flags') -options #slurp -#for $flag in $flags -$flag #slurp -#end for -#for $key, $value in $options.items() -$key:$value #slurp -#end for +sortlist {% for sort in sortlist %}{{sort}} {% endfor %} +{% endif %} +{% if options or flags %} -#end if +options {% for flag in flags %}{{flag}} {% endfor %} +{% for key, value in options.iteritems() -%} + {{key}}:{{value}} +{% endfor %} +{% endif %} diff --git a/templates/sources.list.debian.tmpl b/templates/sources.list.debian.tmpl index 609bc6bd..c8043f76 100644 --- a/templates/sources.list.debian.tmpl +++ b/templates/sources.list.debian.tmpl @@ -1,28 +1,32 @@ -\## Note, this file is written by cloud-init on first boot of an instance -\## modifications made here will not survive a re-bundle. -\## if you wish to make changes you can: -\## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg -\## or do the same in user-data -\## b.) add sources in /etc/apt/sources.list.d -\## c.) make changes to template file /etc/cloud/templates/sources.list.debian.tmpl -\### +## template:jinja +## Note, this file is written by cloud-init on first boot of an instance +## modifications made here will not survive a re-bundle. +## if you wish to make changes you can: +## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg +## or do the same in user-data +## b.) add sources in /etc/apt/sources.list.d +## c.) make changes to template file /etc/cloud/templates/sources.list.debian.tmpl +### # See http://www.debian.org/releases/stable/i386/release-notes/ch-upgrading.html # for how to upgrade to newer versions of the distribution. -deb $mirror $codename main contrib non-free -deb-src $mirror $codename main contrib non-free +deb {{mirror}} {{codename}} main contrib non-free +deb-src {{mirror}} {{codename}} main contrib non-free -\## Major bug fix updates produced after the final release of the -\## distribution. -deb $security $codename/updates main contrib non-free -deb-src $security $codename/updates main contrib non-free -deb $mirror $codename-updates main contrib non-free -deb-src $mirror $codename-updates main contrib non-free +## Major bug fix updates produced after the final release of the +## distribution. +deb {{security}} {{codename}}/updates main contrib non-free +deb-src {{security}} {{codename}}/updates main contrib non-free +deb {{mirror}} {{codename}}-updates main contrib non-free +deb-src {{mirror}} {{codename}}-updates main contrib non-free -\## Uncomment the following two lines to add software from the 'backports' -\## repository. -\## N.B. software from this repository may not have been tested as -\## extensively as that contained in the main release, although it includes -\## newer versions of some applications which may provide useful features. -# deb http://backports.debian.org/debian-backports $codename-backports main contrib non-free -# deb-src http://backports.debian.org/debian-backports $codename-backports main contrib non-free +## Uncomment the following two lines to add software from the 'backports' +## repository. +## +## N.B. software from this repository may not have been tested as +## extensively as that contained in the main release, although it includes +## newer versions of some applications which may provide useful features. +{# +deb http://backports.debian.org/debian-backports {{codename}}-backports main contrib non-free +deb-src http://backports.debian.org/debian-backports {{codename}}-backports main contrib non-free +-#} diff --git a/templates/sources.list.ubuntu.tmpl b/templates/sources.list.ubuntu.tmpl index ce395b3d..4b1b019a 100644 --- a/templates/sources.list.ubuntu.tmpl +++ b/templates/sources.list.ubuntu.tmpl @@ -1,60 +1,60 @@ -\## Note, this file is written by cloud-init on first boot of an instance -\## modifications made here will not survive a re-bundle. -\## if you wish to make changes you can: -\## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg -\## or do the same in user-data -\## b.) add sources in /etc/apt/sources.list.d -\## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl -\### +## template:jinja +## Note, this file is written by cloud-init on first boot of an instance +## modifications made here will not survive a re-bundle. +## if you wish to make changes you can: +## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg +## or do the same in user-data +## b.) add sources in /etc/apt/sources.list.d +## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to # newer versions of the distribution. -deb $mirror $codename main -deb-src $mirror $codename main +deb {{mirror}} {{codename}} main +deb-src {{mirror}} {{codename}} main -\## Major bug fix updates produced after the final release of the -\## distribution. -deb $mirror $codename-updates main -deb-src $mirror $codename-updates main +## Major bug fix updates produced after the final release of the +## distribution. +deb {{mirror}} {{codename}}-updates main +deb-src {{mirror}} {{codename}}-updates main -\## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu -\## team. Also, please note that software in universe WILL NOT receive any -\## review or updates from the Ubuntu security team. -deb $mirror $codename universe -deb-src $mirror $codename universe -deb $mirror $codename-updates universe -deb-src $mirror $codename-updates universe +## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu +## team. Also, please note that software in universe WILL NOT receive any +## review or updates from the Ubuntu security team. +deb {{mirror}} {{codename}} universe +deb-src {{mirror}} {{codename}} universe +deb {{mirror}} {{codename}}-updates universe +deb-src {{mirror}} {{codename}}-updates universe -\## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu -\## team, and may not be under a free licence. Please satisfy yourself as to -\## your rights to use the software. Also, please note that software in -\## multiverse WILL NOT receive any review or updates from the Ubuntu -\## security team. -# deb $mirror $codename multiverse -# deb-src $mirror $codename multiverse -# deb $mirror $codename-updates multiverse -# deb-src $mirror $codename-updates multiverse +## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu +## team, and may not be under a free licence. Please satisfy yourself as to +## your rights to use the software. Also, please note that software in +## multiverse WILL NOT receive any review or updates from the Ubuntu +## security team. +# deb {{mirror}} {{codename}} multiverse +# deb-src {{mirror}} {{codename}} multiverse +# deb {{mirror}} {{codename}}-updates multiverse +# deb-src {{mirror}} {{codename}}-updates multiverse -\## Uncomment the following two lines to add software from the 'backports' -\## repository. -\## N.B. software from this repository may not have been tested as -\## extensively as that contained in the main release, although it includes -\## newer versions of some applications which may provide useful features. -\## Also, please note that software in backports WILL NOT receive any review -\## or updates from the Ubuntu security team. -# deb $mirror $codename-backports main restricted universe multiverse -# deb-src $mirror $codename-backports main restricted universe multiverse +## Uncomment the following two lines to add software from the 'backports' +## repository. +## N.B. software from this repository may not have been tested as +## extensively as that contained in the main release, although it includes +## newer versions of some applications which may provide useful features. +## Also, please note that software in backports WILL NOT receive any review +## or updates from the Ubuntu security team. +# deb {{mirror}} {{codename}}-backports main restricted universe multiverse +# deb-src {{mirror}} {{codename}}-backports main restricted universe multiverse -\## Uncomment the following two lines to add software from Canonical's -\## 'partner' repository. -\## This software is not part of Ubuntu, but is offered by Canonical and the -\## respective vendors as a service to Ubuntu users. -# deb http://archive.canonical.com/ubuntu $codename partner -# deb-src http://archive.canonical.com/ubuntu $codename partner +## Uncomment the following two lines to add software from Canonical's +## 'partner' repository. +## This software is not part of Ubuntu, but is offered by Canonical and the +## respective vendors as a service to Ubuntu users. +# deb http://archive.canonical.com/ubuntu {{codename}} partner +# deb-src http://archive.canonical.com/ubuntu {{codename}} partner -deb $security $codename-security main -deb-src $security $codename-security main -deb $security $codename-security universe -deb-src $security $codename-security universe -# deb $security $codename-security multiverse -# deb-src $security $codename-security multiverse +deb {{security}} {{codename}}-security main +deb-src {{security}} {{codename}}-security main +deb {{security}} {{codename}}-security universe +deb-src {{security}} {{codename}}-security universe +# deb {{security}} {{codename}}-security multiverse +# deb-src {{security}} {{codename}}-security multiverse diff --git a/test-requirements.txt b/test-requirements.txt index 4be0211d..2edb8066 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -1,6 +1,5 @@ httpretty>=0.7.1 mocker nose -pep8 +pep8==1.5.7 pyflakes -pylint diff --git a/tests/unittests/helpers.py b/tests/unittests/helpers.py index 5bed13cc..9700a4ca 100644 --- a/tests/unittests/helpers.py +++ b/tests/unittests/helpers.py @@ -52,6 +52,30 @@ if PY26: standardMsg = standardMsg % (value) self.fail(self._formatMessage(msg, standardMsg)) + def assertDictContainsSubset(self, expected, actual, msg=None): + missing = [] + mismatched = [] + for k, v in expected.iteritems(): + if k not in actual: + missing.append(k) + elif actual[k] != v: + mismatched.append('%r, expected: %r, actual: %r' + % (k, v, actual[k])) + + if len(missing) == 0 and len(mismatched) == 0: + return + + standardMsg = '' + if missing: + standardMsg = 'Missing: %r' % ','.join(m for m in missing) + if mismatched: + if standardMsg: + standardMsg += '; ' + standardMsg += 'Mismatched values: %s' % ','.join(mismatched) + + self.fail(self._formatMessage(msg, standardMsg)) + + else: class TestCase(unittest.TestCase): pass @@ -209,6 +233,21 @@ class FilesystemMockingTestCase(ResourceUsingTestCase): self.patched_funcs.append((mod, f, func)) +class HttprettyTestCase(TestCase): + # necessary as http_proxy gets in the way of httpretty + # https://github.com/gabrielfalcao/HTTPretty/issues/122 + def setUp(self): + self.restore_proxy = os.environ.get('http_proxy') + if self.restore_proxy is not None: + del os.environ['http_proxy'] + super(HttprettyTestCase, self).setUp() + + def tearDown(self): + if self.restore_proxy: + os.environ['http_proxy'] = self.restore_proxy + super(HttprettyTestCase, self).tearDown() + + def populate_dir(path, files): if not os.path.exists(path): os.makedirs(path) diff --git a/tests/unittests/test__init__.py b/tests/unittests/test__init__.py index 03065c8b..17965488 100644 --- a/tests/unittests/test__init__.py +++ b/tests/unittests/test__init__.py @@ -18,8 +18,7 @@ class FakeModule(handlers.Handler): def list_types(self): return self.types - def handle_part(self, _data, ctype, filename, # pylint: disable=W0221 - payload, frequency): + def handle_part(self, data, ctype, filename, payload, frequency): pass diff --git a/tests/unittests/test_builtin_handlers.py b/tests/unittests/test_builtin_handlers.py index b387f13b..af7f442e 100644 --- a/tests/unittests/test_builtin_handlers.py +++ b/tests/unittests/test_builtin_handlers.py @@ -2,7 +2,7 @@ import os -from tests.unittests import helpers as test_helpers +from . import helpers as test_helpers from cloudinit import handlers from cloudinit import helpers diff --git a/tests/unittests/test_data.py b/tests/unittests/test_data.py index 68729c57..fd6bd8a1 100644 --- a/tests/unittests/test_data.py +++ b/tests/unittests/test_data.py @@ -20,7 +20,7 @@ from cloudinit import util INSTANCE_ID = "i-testing" -from tests.unittests import helpers +from . import helpers class FakeDataSource(sources.DataSource): @@ -106,7 +106,7 @@ class TestConsumeUserData(helpers.FilesystemMockingTestCase): initer.read_cfg() initer.initialize() initer.fetch() - _iid = initer.instancify() + initer.instancify() initer.update() initer.cloudify().run('consume_data', initer.consume_data, @@ -145,7 +145,7 @@ class TestConsumeUserData(helpers.FilesystemMockingTestCase): initer.read_cfg() initer.initialize() initer.fetch() - _iid = initer.instancify() + initer.instancify() initer.update() initer.cloudify().run('consume_data', initer.consume_data, @@ -221,7 +221,7 @@ run: initer.read_cfg() initer.initialize() initer.fetch() - _iid = initer.instancify() + initer.instancify() initer.update() initer.cloudify().run('consume_data', initer.consume_data, @@ -256,7 +256,7 @@ vendor_data: initer.read_cfg() initer.initialize() initer.fetch() - _iid = initer.instancify() + initer.instancify() initer.update() initer.cloudify().run('consume_data', initer.consume_data, @@ -264,7 +264,6 @@ vendor_data: freq=PER_INSTANCE) mods = stages.Modules(initer) (_which_ran, _failures) = mods.run_section('cloud_init_modules') - _cfg = mods.cfg vendor_script = initer.paths.get_ipath_cur('vendor_scripts') vendor_script_fns = "%s%s/part-001" % (new_root, vendor_script) self.assertTrue(os.path.exists(vendor_script_fns)) diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py index ccfd672a..e992a006 100644 --- a/tests/unittests/test_datasource/test_azure.py +++ b/tests/unittests/test_datasource/test_azure.py @@ -1,7 +1,7 @@ from cloudinit import helpers from cloudinit.util import load_file from cloudinit.sources import DataSourceAzure -from tests.unittests.helpers import populate_dir +from ..helpers import populate_dir import base64 import crypt @@ -235,7 +235,7 @@ class TestAzureDataSource(MockerTestCase): self.assertEqual(dsrc.userdata_raw, mydata) def test_no_datasource_expected(self): - #no source should be found if no seed_dir and no devs + # no source should be found if no seed_dir and no devs data = {} dsrc = self._get_ds({}) ret = dsrc.get_data() diff --git a/tests/unittests/test_datasource/test_cloudsigma.py b/tests/unittests/test_datasource/test_cloudsigma.py index f92e07b7..306ac7d8 100644 --- a/tests/unittests/test_datasource/test_cloudsigma.py +++ b/tests/unittests/test_datasource/test_cloudsigma.py @@ -1,10 +1,11 @@ # coding: utf-8 import copy -from unittest import TestCase from cloudinit.cs_utils import Cepko from cloudinit.sources import DataSourceCloudSigma +from .. import helpers as test_helpers + SERVER_CONTEXT = { "cpu": 1000, @@ -36,7 +37,7 @@ class CepkoMock(Cepko): return self -class DataSourceCloudSigmaTest(TestCase): +class DataSourceCloudSigmaTest(test_helpers.TestCase): def setUp(self): self.datasource = DataSourceCloudSigma.DataSourceCloudSigma("", "", "") self.datasource.is_running_in_cloudsigma = lambda: True diff --git a/tests/unittests/test_datasource/test_configdrive.py b/tests/unittests/test_datasource/test_configdrive.py index 4404668e..d88066e5 100644 --- a/tests/unittests/test_datasource/test_configdrive.py +++ b/tests/unittests/test_datasource/test_configdrive.py @@ -12,7 +12,7 @@ from cloudinit.sources import DataSourceConfigDrive as ds from cloudinit.sources.helpers import openstack from cloudinit import util -from tests.unittests import helpers as unit_helpers +from .. import helpers as unit_helpers PUBKEY = u'ssh-rsa AAAAB3NzaC1....sIkJhq8wdX+4I3A4cYbYP ubuntu@server-460\n' EC2_META = { diff --git a/tests/unittests/test_datasource/test_gce.py b/tests/unittests/test_datasource/test_gce.py index d91bd531..60a0ce48 100644 --- a/tests/unittests/test_datasource/test_gce.py +++ b/tests/unittests/test_datasource/test_gce.py @@ -15,7 +15,6 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -import unittest import httpretty import re @@ -25,6 +24,8 @@ from cloudinit import settings from cloudinit import helpers from cloudinit.sources import DataSourceGCE +from .. import helpers as test_helpers + GCE_META = { 'instance/id': '123', 'instance/zone': 'foo/bar', @@ -54,12 +55,13 @@ def _request_callback(method, uri, headers): return (404, headers, '') -class TestDataSourceGCE(unittest.TestCase): +class TestDataSourceGCE(test_helpers.HttprettyTestCase): def setUp(self): self.ds = DataSourceGCE.DataSourceGCE( settings.CFG_BUILTIN, None, helpers.Paths({})) + super(TestDataSourceGCE, self).setUp() @httpretty.activate def test_connection(self): diff --git a/tests/unittests/test_datasource/test_maas.py b/tests/unittests/test_datasource/test_maas.py index 73cfadcb..c157beb8 100644 --- a/tests/unittests/test_datasource/test_maas.py +++ b/tests/unittests/test_datasource/test_maas.py @@ -3,7 +3,7 @@ import os from cloudinit.sources import DataSourceMAAS from cloudinit import url_helper -from tests.unittests.helpers import populate_dir +from ..helpers import populate_dir import mocker diff --git a/tests/unittests/test_datasource/test_nocloud.py b/tests/unittests/test_datasource/test_nocloud.py index a65833eb..e9235951 100644 --- a/tests/unittests/test_datasource/test_nocloud.py +++ b/tests/unittests/test_datasource/test_nocloud.py @@ -1,7 +1,7 @@ from cloudinit import helpers from cloudinit.sources import DataSourceNoCloud from cloudinit import util -from tests.unittests.helpers import populate_dir +from ..helpers import populate_dir from mocker import MockerTestCase import os @@ -50,14 +50,13 @@ class TestNoCloudDataSource(MockerTestCase): self.assertTrue(ret) def test_fs_label(self): - #find_devs_with should not be called ff fs_label is None + # find_devs_with should not be called ff fs_label is None ds = DataSourceNoCloud.DataSourceNoCloud class PsuedoException(Exception): pass def my_find_devs_with(*args, **kwargs): - _f = (args, kwargs) raise PsuedoException self.apply_patches([(util, 'find_devs_with', my_find_devs_with)]) @@ -74,7 +73,7 @@ class TestNoCloudDataSource(MockerTestCase): self.assertFalse(ret) def test_no_datasource_expected(self): - #no source should be found if no cmdline, config, and fs_label=None + # no source should be found if no cmdline, config, and fs_label=None sys_cfg = {'datasource': {'NoCloud': {'fs_label': None}}} ds = DataSourceNoCloud.DataSourceNoCloud diff --git a/tests/unittests/test_datasource/test_opennebula.py b/tests/unittests/test_datasource/test_opennebula.py index ec6b752b..b4fd1f4d 100644 --- a/tests/unittests/test_datasource/test_opennebula.py +++ b/tests/unittests/test_datasource/test_opennebula.py @@ -2,7 +2,7 @@ from cloudinit import helpers from cloudinit.sources import DataSourceOpenNebula as ds from cloudinit import util from mocker import MockerTestCase -from tests.unittests.helpers import populate_dir +from ..helpers import populate_dir from base64 import b64encode import os diff --git a/tests/unittests/test_datasource/test_openstack.py b/tests/unittests/test_datasource/test_openstack.py index 3a64430a..7b4e651a 100644 --- a/tests/unittests/test_datasource/test_openstack.py +++ b/tests/unittests/test_datasource/test_openstack.py @@ -19,12 +19,13 @@ import copy import json import re +import unittest from StringIO import StringIO from urlparse import urlparse -from tests.unittests import helpers as test_helpers +from .. import helpers as test_helpers from cloudinit import helpers from cloudinit import settings @@ -67,8 +68,8 @@ OSTACK_META = { CONTENT_0 = 'This is contents of /etc/foo.cfg\n' CONTENT_1 = '# this is /etc/bar/bar.cfg\n' OS_FILES = { - 'openstack/2012-08-10/meta_data.json': json.dumps(OSTACK_META), - 'openstack/2012-08-10/user_data': USER_DATA, + 'openstack/latest/meta_data.json': json.dumps(OSTACK_META), + 'openstack/latest/user_data': USER_DATA, 'openstack/content/0000': CONTENT_0, 'openstack/content/0001': CONTENT_1, 'openstack/latest/meta_data.json': json.dumps(OSTACK_META), @@ -78,6 +79,9 @@ OS_FILES = { EC2_FILES = { 'latest/user-data': USER_DATA, } +EC2_VERSIONS = [ + 'latest', +] def _register_uris(version, ec2_files, ec2_meta, os_files): @@ -85,10 +89,13 @@ def _register_uris(version, ec2_files, ec2_meta, os_files): same data returned by the openstack metadata service (and ec2 service).""" def match_ec2_url(uri, headers): + path = uri.path.strip("/") + if len(path) == 0: + return (200, headers, "\n".join(EC2_VERSIONS)) path = uri.path.lstrip("/") if path in ec2_files: return (200, headers, ec2_files.get(path)) - if path == 'latest/meta-data': + if path == 'latest/meta-data/': buf = StringIO() for (k, v) in ec2_meta.items(): if isinstance(v, (list, tuple)): @@ -97,7 +104,7 @@ def _register_uris(version, ec2_files, ec2_meta, os_files): buf.write("%s" % (k)) buf.write("\n") return (200, headers, buf.getvalue()) - if path.startswith('latest/meta-data'): + if path.startswith('latest/meta-data/'): value = None pieces = path.split("/") if path.endswith("/"): @@ -110,24 +117,33 @@ def _register_uris(version, ec2_files, ec2_meta, os_files): return (200, headers, str(value)) return (404, headers, '') - def get_request_callback(method, uri, headers): - uri = urlparse(uri) + def match_os_uri(uri, headers): + path = uri.path.strip("/") + if path == 'openstack': + return (200, headers, "\n".join([openstack.OS_LATEST])) path = uri.path.lstrip("/") if path in os_files: return (200, headers, os_files.get(path)) + return (404, headers, '') + + def get_request_callback(method, uri, headers): + uri = urlparse(uri) + path = uri.path.lstrip("/").split("/") + if path[0] == 'openstack': + return match_os_uri(uri, headers) return match_ec2_url(uri, headers) hp.register_uri(hp.GET, re.compile(r'http://169.254.169.254/.*'), body=get_request_callback) -class TestOpenStackDataSource(test_helpers.TestCase): +class TestOpenStackDataSource(test_helpers.HttprettyTestCase): VERSION = 'latest' @hp.activate def test_successful(self): _register_uris(self.VERSION, EC2_FILES, EC2_META, OS_FILES) - f = ds.read_metadata_service(BASE_URL, version=self.VERSION) + f = ds.read_metadata_service(BASE_URL) self.assertEquals(VENDOR_DATA, f.get('vendordata')) self.assertEquals(CONTENT_0, f['files']['/etc/foo.cfg']) self.assertEquals(CONTENT_1, f['files']['/etc/bar/bar.cfg']) @@ -149,7 +165,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): @hp.activate def test_no_ec2(self): _register_uris(self.VERSION, {}, {}, OS_FILES) - f = ds.read_metadata_service(BASE_URL, version=self.VERSION) + f = ds.read_metadata_service(BASE_URL) self.assertEquals(VENDOR_DATA, f.get('vendordata')) self.assertEquals(CONTENT_0, f['files']['/etc/foo.cfg']) self.assertEquals(CONTENT_1, f['files']['/etc/bar/bar.cfg']) @@ -165,7 +181,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): os_files.pop(k, None) _register_uris(self.VERSION, {}, {}, os_files) self.assertRaises(openstack.NonReadable, ds.read_metadata_service, - BASE_URL, version=self.VERSION) + BASE_URL) @hp.activate def test_bad_uuid(self): @@ -177,7 +193,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): os_files[k] = json.dumps(os_meta) _register_uris(self.VERSION, {}, {}, os_files) self.assertRaises(openstack.BrokenMetadata, ds.read_metadata_service, - BASE_URL, version=self.VERSION) + BASE_URL) @hp.activate def test_userdata_empty(self): @@ -186,7 +202,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): if k.endswith('user_data'): os_files.pop(k, None) _register_uris(self.VERSION, {}, {}, os_files) - f = ds.read_metadata_service(BASE_URL, version=self.VERSION) + f = ds.read_metadata_service(BASE_URL) self.assertEquals(VENDOR_DATA, f.get('vendordata')) self.assertEquals(CONTENT_0, f['files']['/etc/foo.cfg']) self.assertEquals(CONTENT_1, f['files']['/etc/bar/bar.cfg']) @@ -199,7 +215,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): if k.endswith('vendor_data.json'): os_files.pop(k, None) _register_uris(self.VERSION, {}, {}, os_files) - f = ds.read_metadata_service(BASE_URL, version=self.VERSION) + f = ds.read_metadata_service(BASE_URL) self.assertEquals(CONTENT_0, f['files']['/etc/foo.cfg']) self.assertEquals(CONTENT_1, f['files']['/etc/bar/bar.cfg']) self.assertFalse(f.get('vendordata')) @@ -212,7 +228,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): os_files[k] = '{' # some invalid json _register_uris(self.VERSION, {}, {}, os_files) self.assertRaises(openstack.BrokenMetadata, ds.read_metadata_service, - BASE_URL, version=self.VERSION) + BASE_URL) @hp.activate def test_metadata_invalid(self): @@ -222,7 +238,7 @@ class TestOpenStackDataSource(test_helpers.TestCase): os_files[k] = '{' # some invalid json _register_uris(self.VERSION, {}, {}, os_files) self.assertRaises(openstack.BrokenMetadata, ds.read_metadata_service, - BASE_URL, version=self.VERSION) + BASE_URL) @hp.activate def test_datasource(self): @@ -241,7 +257,8 @@ class TestOpenStackDataSource(test_helpers.TestCase): self.assertEquals(EC2_META, ds_os.ec2_metadata) self.assertEquals(USER_DATA, ds_os.userdata_raw) self.assertEquals(2, len(ds_os.files)) - self.assertEquals(VENDOR_DATA, ds_os.vendordata_raw) + self.assertEquals(VENDOR_DATA, ds_os.vendordata_pure) + self.assertEquals(ds_os.vendordata_raw, None) @hp.activate def test_bad_datasource_meta(self): @@ -299,3 +316,34 @@ class TestOpenStackDataSource(test_helpers.TestCase): found = ds_os.get_data() self.assertFalse(found) self.assertIsNone(ds_os.version) + + +class TestVendorDataLoading(unittest.TestCase): + def cvj(self, data): + return openstack.convert_vendordata_json(data) + + def test_vd_load_none(self): + # non-existant vendor-data should return none + self.assertIsNone(self.cvj(None)) + + def test_vd_load_string(self): + self.assertEqual(self.cvj("foobar"), "foobar") + + def test_vd_load_list(self): + data = [{'foo': 'bar'}, 'mystring', list(['another', 'list'])] + self.assertEqual(self.cvj(data), data) + + def test_vd_load_dict_no_ci(self): + self.assertEqual(self.cvj({'foo': 'bar'}), None) + + def test_vd_load_dict_ci_dict(self): + self.assertRaises(ValueError, self.cvj, + {'foo': 'bar', 'cloud-init': {'x': 1}}) + + def test_vd_load_dict_ci_string(self): + data = {'foo': 'bar', 'cloud-init': 'VENDOR_DATA'} + self.assertEqual(self.cvj(data), data['cloud-init']) + + def test_vd_load_dict_ci_list(self): + data = {'foo': 'bar', 'cloud-init': ['VD_1', 'VD_2']} + self.assertEqual(self.cvj(data), data['cloud-init']) diff --git a/tests/unittests/test_datasource/test_smartos.py b/tests/unittests/test_datasource/test_smartos.py index f64aea07..b197b600 100644 --- a/tests/unittests/test_datasource/test_smartos.py +++ b/tests/unittests/test_datasource/test_smartos.py @@ -25,7 +25,7 @@ import base64 from cloudinit import helpers as c_helpers from cloudinit.sources import DataSourceSmartOS -from tests.unittests import helpers +from .. import helpers import os import os.path import re diff --git a/tests/unittests/test_distros/test_generic.py b/tests/unittests/test_distros/test_generic.py index 7befb8c8..db6aa0e8 100644 --- a/tests/unittests/test_distros/test_generic.py +++ b/tests/unittests/test_distros/test_generic.py @@ -1,7 +1,7 @@ from cloudinit import distros from cloudinit import util -from tests.unittests import helpers +from .. import helpers import os @@ -26,8 +26,8 @@ package_mirrors = [ unknown_arch_info ] -gpmi = distros._get_package_mirror_info # pylint: disable=W0212 -gapmi = distros._get_arch_package_mirror_info # pylint: disable=W0212 +gpmi = distros._get_package_mirror_info +gapmi = distros._get_arch_package_mirror_info class TestGenericDistro(helpers.FilesystemMockingTestCase): @@ -193,7 +193,7 @@ class TestGenericDistro(helpers.FilesystemMockingTestCase): 'security': 'http://security-mirror2-intel'}) -#def _get_package_mirror_info(mirror_info, availability_zone=None, +# def _get_package_mirror_info(mirror_info, availability_zone=None, # mirror_filter=util.search_for_mirror): diff --git a/tests/unittests/test_distros/test_netconfig.py b/tests/unittests/test_distros/test_netconfig.py index 9763b14b..ed997a1d 100644 --- a/tests/unittests/test_distros/test_netconfig.py +++ b/tests/unittests/test_distros/test_netconfig.py @@ -173,3 +173,60 @@ NETWORKING=yes ''' self.assertCfgEquals(expected_buf, str(write_buf)) self.assertEquals(write_buf.mode, 0644) + + def test_simple_write_freebsd(self): + fbsd_distro = self._get_distro('freebsd') + util_mock = self.mocker.replace(util.write_file, + spec=False, passthrough=False) + exists_mock = self.mocker.replace(os.path.isfile, + spec=False, passthrough=False) + load_mock = self.mocker.replace(util.load_file, + spec=False, passthrough=False) + + exists_mock(mocker.ARGS) + self.mocker.count(0, None) + self.mocker.result(False) + + write_bufs = {} + read_bufs = { + '/etc/rc.conf': '', + } + + def replace_write(filename, content, mode=0644, omode="wb"): + buf = WriteBuffer() + buf.mode = mode + buf.omode = omode + buf.write(content) + write_bufs[filename] = buf + + def replace_read(fname, read_cb=None, quiet=False): + if fname not in read_bufs: + if fname in write_bufs: + return str(write_bufs[fname]) + raise IOError("%s not found" % fname) + else: + if fname in write_bufs: + return str(write_bufs[fname]) + return read_bufs[fname] + + util_mock(mocker.ARGS) + self.mocker.call(replace_write) + self.mocker.count(0, None) + + load_mock(mocker.ARGS) + self.mocker.call(replace_read) + self.mocker.count(0, None) + + self.mocker.replay() + fbsd_distro.apply_network(BASE_NET_CFG, False) + + self.assertIn('/etc/rc.conf', write_bufs) + write_buf = write_bufs['/etc/rc.conf'] + expected_buf = ''' +ifconfig_eth0="192.168.1.5 netmask 255.255.255.0" +ifconfig_eth1="DHCP" +defaultrouter="192.168.1.254" +''' + self.assertCfgEquals(expected_buf, str(write_buf)) + self.assertEquals(write_buf.mode, 0644) + diff --git a/tests/unittests/test_ec2_util.py b/tests/unittests/test_ec2_util.py index dd87665d..84aa002e 100644 --- a/tests/unittests/test_ec2_util.py +++ b/tests/unittests/test_ec2_util.py @@ -1,4 +1,4 @@ -from tests.unittests import helpers +from . import helpers from cloudinit import ec2_utils as eu from cloudinit import url_helper as uh @@ -6,7 +6,7 @@ from cloudinit import url_helper as uh import httpretty as hp -class TestEc2Util(helpers.TestCase): +class TestEc2Util(helpers.HttprettyTestCase): VERSION = 'latest' @hp.activate @@ -44,7 +44,7 @@ class TestEc2Util(helpers.TestCase): @hp.activate def test_metadata_fetch_no_keys(self): - base_url = 'http://169.254.169.254/%s/meta-data' % (self.VERSION) + base_url = 'http://169.254.169.254/%s/meta-data/' % (self.VERSION) hp.register_uri(hp.GET, base_url, status=200, body="\n".join(['hostname', 'instance-id', @@ -62,7 +62,7 @@ class TestEc2Util(helpers.TestCase): @hp.activate def test_metadata_fetch_key(self): - base_url = 'http://169.254.169.254/%s/meta-data' % (self.VERSION) + base_url = 'http://169.254.169.254/%s/meta-data/' % (self.VERSION) hp.register_uri(hp.GET, base_url, status=200, body="\n".join(['hostname', 'instance-id', @@ -83,7 +83,7 @@ class TestEc2Util(helpers.TestCase): @hp.activate def test_metadata_fetch_with_2_keys(self): - base_url = 'http://169.254.169.254/%s/meta-data' % (self.VERSION) + base_url = 'http://169.254.169.254/%s/meta-data/' % (self.VERSION) hp.register_uri(hp.GET, base_url, status=200, body="\n".join(['hostname', 'instance-id', @@ -108,7 +108,7 @@ class TestEc2Util(helpers.TestCase): @hp.activate def test_metadata_fetch_bdm(self): - base_url = 'http://169.254.169.254/%s/meta-data' % (self.VERSION) + base_url = 'http://169.254.169.254/%s/meta-data/' % (self.VERSION) hp.register_uri(hp.GET, base_url, status=200, body="\n".join(['hostname', 'instance-id', diff --git a/tests/unittests/test_filters/test_launch_index.py b/tests/unittests/test_filters/test_launch_index.py index 773bb312..2f4c2fda 100644 --- a/tests/unittests/test_filters/test_launch_index.py +++ b/tests/unittests/test_filters/test_launch_index.py @@ -1,6 +1,6 @@ import copy -from tests.unittests import helpers +from .. import helpers import itertools diff --git a/tests/unittests/test_handler/test_handler_growpart.py b/tests/unittests/test_handler/test_handler_growpart.py index f2ed4597..5d0636d1 100644 --- a/tests/unittests/test_handler/test_handler_growpart.py +++ b/tests/unittests/test_handler/test_handler_growpart.py @@ -53,7 +53,7 @@ class TestDisabled(MockerTestCase): self.handle = cc_growpart.handle def test_mode_off(self): - #Test that nothing is done if mode is off. + # Test that nothing is done if mode is off. # this really only verifies that resizer_factory isn't called config = {'growpart': {'mode': 'off'}} @@ -109,7 +109,7 @@ class TestConfig(MockerTestCase): self.assertTrue(isinstance(ret, cc_growpart.ResizeGrowPart)) def test_handle_with_no_growpart_entry(self): - #if no 'growpart' entry in config, then mode=auto should be used + # if no 'growpart' entry in config, then mode=auto should be used myresizer = object() @@ -141,7 +141,7 @@ class TestResize(MockerTestCase): self.mocker.order() def test_simple_devices(self): - #test simple device list + # test simple device list # this patches out devent2dev, os.stat, and device_part_info # so in the end, doesn't test a lot devs = ["/dev/XXda1", "/dev/YYda2"] @@ -187,9 +187,9 @@ class TestResize(MockerTestCase): find("/dev/YYda2", resized)[1]) self.assertEqual(cc_growpart.RESIZE.SKIPPED, find(enoent[0], resized)[1]) - #self.assertEqual(resize_calls, - #[("/dev/XXda", "1", "/dev/XXda1"), - #("/dev/YYda", "2", "/dev/YYda2")]) + # self.assertEqual(resize_calls, + # [("/dev/XXda", "1", "/dev/XXda1"), + # ("/dev/YYda", "2", "/dev/YYda2")]) finally: cc_growpart.device_part_info = opinfo os.stat = real_stat @@ -203,8 +203,6 @@ def simple_device_part_info(devpath): class Bunch(object): - st_mode = None # fix pylint complaint - def __init__(self, **kwds): self.__dict__.update(kwds) diff --git a/tests/unittests/test_handler/test_handler_locale.py b/tests/unittests/test_handler/test_handler_locale.py index 72ad00fd..eb251636 100644 --- a/tests/unittests/test_handler/test_handler_locale.py +++ b/tests/unittests/test_handler/test_handler_locale.py @@ -25,7 +25,7 @@ from cloudinit import util from cloudinit.sources import DataSourceNoCloud -from tests.unittests import helpers as t_help +from .. import helpers as t_help from configobj import ConfigObj diff --git a/tests/unittests/test_handler/test_handler_power_state.py b/tests/unittests/test_handler/test_handler_power_state.py index f6e37fa5..2f86b8f8 100644 --- a/tests/unittests/test_handler/test_handler_power_state.py +++ b/tests/unittests/test_handler/test_handler_power_state.py @@ -1,6 +1,6 @@ from cloudinit.config import cc_power_state_change as psc -from tests.unittests import helpers as t_help +from .. import helpers as t_help class TestLoadPowerState(t_help.TestCase): @@ -67,7 +67,7 @@ def check_lps_ret(psc_return, mode=None): cmd = psc_return[0] timeout = psc_return[1] - if not 'shutdown' in psc_return[0][0]: + if 'shutdown' not in psc_return[0][0]: errs.append("string 'shutdown' not in cmd") if mode is not None: diff --git a/tests/unittests/test_handler/test_handler_seed_random.py b/tests/unittests/test_handler/test_handler_seed_random.py index be2fa4a4..40481f16 100644 --- a/tests/unittests/test_handler/test_handler_seed_random.py +++ b/tests/unittests/test_handler/test_handler_seed_random.py @@ -1,4 +1,4 @@ - # Copyright (C) 2013 Hewlett-Packard Development Company, L.P. +# Copyright (C) 2013 Hewlett-Packard Development Company, L.P. # # Author: Juerg Haefliger <juerg.haefliger@hp.com> # @@ -31,7 +31,7 @@ from cloudinit import util from cloudinit.sources import DataSourceNone -from tests.unittests import helpers as t_help +from .. import helpers as t_help import logging diff --git a/tests/unittests/test_handler/test_handler_set_hostname.py b/tests/unittests/test_handler/test_handler_set_hostname.py index 6344ec0c..03004ab9 100644 --- a/tests/unittests/test_handler/test_handler_set_hostname.py +++ b/tests/unittests/test_handler/test_handler_set_hostname.py @@ -5,7 +5,7 @@ from cloudinit import distros from cloudinit import helpers from cloudinit import util -from tests.unittests import helpers as t_help +from .. import helpers as t_help import logging diff --git a/tests/unittests/test_handler/test_handler_timezone.py b/tests/unittests/test_handler/test_handler_timezone.py index 40b69773..874db340 100644 --- a/tests/unittests/test_handler/test_handler_timezone.py +++ b/tests/unittests/test_handler/test_handler_timezone.py @@ -25,7 +25,7 @@ from cloudinit import util from cloudinit.sources import DataSourceNoCloud -from tests.unittests import helpers as t_help +from .. import helpers as t_help from configobj import ConfigObj diff --git a/tests/unittests/test_handler/test_handler_yum_add_repo.py b/tests/unittests/test_handler/test_handler_yum_add_repo.py index 7c6f7c40..21b89c34 100644 --- a/tests/unittests/test_handler/test_handler_yum_add_repo.py +++ b/tests/unittests/test_handler/test_handler_yum_add_repo.py @@ -2,7 +2,7 @@ from cloudinit import util from cloudinit.config import cc_yum_add_repo -from tests.unittests import helpers +from .. import helpers import logging @@ -24,7 +24,7 @@ class TestConfig(helpers.FilesystemMockingTestCase): 'epel-testing': { 'name': 'Extra Packages for Enterprise Linux 5 - Testing', # Missing this should cause the repo not to be written - #'baseurl': 'http://blah.org/pub/epel/testing/5/$basearch', + # 'baseurl': 'http://blah.org/pub/epel/testing/5/$basearch', 'enabled': False, 'gpgcheck': True, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL', diff --git a/tests/unittests/test_merging.py b/tests/unittests/test_merging.py index 486b9158..07b610f7 100644 --- a/tests/unittests/test_merging.py +++ b/tests/unittests/test_merging.py @@ -1,4 +1,4 @@ -from tests.unittests import helpers +from . import helpers from cloudinit.handlers import cloud_config from cloudinit.handlers import (CONTENT_START, CONTENT_END) @@ -11,7 +11,7 @@ import glob import os import random import re -import string # pylint: disable=W0402 +import string SOURCE_PAT = "source*.*yaml" EXPECTED_PAT = "expected%s.yaml" diff --git a/tests/unittests/test_pathprefix2dict.py b/tests/unittests/test_pathprefix2dict.py index c68c263c..590c4b82 100644 --- a/tests/unittests/test_pathprefix2dict.py +++ b/tests/unittests/test_pathprefix2dict.py @@ -1,7 +1,7 @@ from cloudinit import util from mocker import MockerTestCase -from tests.unittests.helpers import populate_dir +from .helpers import populate_dir class TestPathPrefix2Dict(MockerTestCase): diff --git a/tests/unittests/test_runs/test_merge_run.py b/tests/unittests/test_runs/test_merge_run.py index 5ffe95a2..977adb34 100644 --- a/tests/unittests/test_runs/test_merge_run.py +++ b/tests/unittests/test_runs/test_merge_run.py @@ -1,6 +1,6 @@ import os -from tests.unittests import helpers +from .. import helpers from cloudinit.settings import (PER_INSTANCE) from cloudinit import stages @@ -33,7 +33,7 @@ class TestMergeRun(helpers.FilesystemMockingTestCase): initer.initialize() initer.fetch() initer.datasource.userdata_raw = ud - _iid = initer.instancify() + initer.instancify() initer.update() initer.cloudify().run('consume_data', initer.consume_data, diff --git a/tests/unittests/test_runs/test_simple_run.py b/tests/unittests/test_runs/test_simple_run.py index 9a7178d1..c9ba949e 100644 --- a/tests/unittests/test_runs/test_simple_run.py +++ b/tests/unittests/test_runs/test_simple_run.py @@ -1,6 +1,6 @@ import os -from tests.unittests import helpers +from .. import helpers from cloudinit.settings import (PER_INSTANCE) from cloudinit import stages diff --git a/tests/unittests/test_templating.py b/tests/unittests/test_templating.py new file mode 100644 index 00000000..87681f0f --- /dev/null +++ b/tests/unittests/test_templating.py @@ -0,0 +1,107 @@ +# vi: ts=4 expandtab +# +# Copyright (C) 2014 Yahoo! Inc. +# +# Author: Joshua Harlow <harlowja@yahoo-inc.com> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 3, as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +from . import helpers as test_helpers +import textwrap + +from cloudinit import templater + + +class TestTemplates(test_helpers.TestCase): + def test_render_basic(self): + in_data = textwrap.dedent(""" + ${b} + + c = d + """) + in_data = in_data.strip() + expected_data = textwrap.dedent(""" + 2 + + c = d + """) + out_data = templater.basic_render(in_data, {'b': 2}) + self.assertEqual(expected_data.strip(), out_data) + + def test_detection(self): + blob = "## template:cheetah" + + (template_type, renderer, contents) = templater.detect_template(blob) + self.assertIn("cheetah", template_type) + self.assertEqual("", contents.strip()) + + blob = "blahblah $blah" + (template_type, renderer, contents) = templater.detect_template(blob) + self.assertIn("cheetah", template_type) + self.assertEquals(blob, contents) + + blob = '##template:something-new' + self.assertRaises(ValueError, templater.detect_template, blob) + + def test_render_cheetah(self): + blob = '''## template:cheetah +$a,$b''' + c = templater.render_string(blob, {"a": 1, "b": 2}) + self.assertEquals("1,2", c) + + def test_render_jinja(self): + blob = '''## template:jinja +{{a}},{{b}}''' + c = templater.render_string(blob, {"a": 1, "b": 2}) + self.assertEquals("1,2", c) + + def test_render_default(self): + blob = '''$a,$b''' + c = templater.render_string(blob, {"a": 1, "b": 2}) + self.assertEquals("1,2", c) + + def test_render_basic_deeper(self): + hn = 'myfoohost.yahoo.com' + expected_data = "h=%s\nc=d\n" % hn + in_data = "h=$hostname.canonical_name\nc=d\n" + params = { + "hostname": { + "canonical_name": hn, + }, + } + out_data = templater.render_string(in_data, params) + self.assertEqual(expected_data, out_data) + + def test_render_basic_no_parens(self): + hn = "myfoohost" + in_data = "h=$hostname\nc=d\n" + expected_data = "h=%s\nc=d\n" % hn + out_data = templater.basic_render(in_data, {'hostname': hn}) + self.assertEqual(expected_data, out_data) + + def test_render_basic_parens(self): + hn = "myfoohost" + in_data = "h = ${hostname}\nc=d\n" + expected_data = "h = %s\nc=d\n" % hn + out_data = templater.basic_render(in_data, {'hostname': hn}) + self.assertEqual(expected_data, out_data) + + def test_render_basic2(self): + mirror = "mymirror" + codename = "zany" + in_data = "deb $mirror $codename-updates main contrib non-free" + ex_data = "deb %s %s-updates main contrib non-free" % (mirror, codename) + + out_data = templater.basic_render(in_data, + {'mirror': mirror, 'codename': codename}) + self.assertEqual(ex_data, out_data) diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py index 38ab0c96..35e92445 100644 --- a/tests/unittests/test_util.py +++ b/tests/unittests/test_util.py @@ -1,12 +1,10 @@ -# pylint: disable=C0301 -# the mountinfo data lines are too long import os import stat import yaml from mocker import MockerTestCase -from tests.unittests import helpers -from unittest import TestCase +from . import helpers +import unittest from cloudinit import importer from cloudinit import util @@ -18,7 +16,7 @@ class FakeSelinux(object): self.match_what = match_what self.restored = [] - def matchpathcon(self, path, mode): # pylint: disable=W0613 + def matchpathcon(self, path, mode): if path == self.match_what: return else: @@ -27,11 +25,11 @@ class FakeSelinux(object): def is_selinux_enabled(self): return True - def restorecon(self, path, recursive): # pylint: disable=W0613 + def restorecon(self, path, recursive): self.restored.append(path) -class TestGetCfgOptionListOrStr(TestCase): +class TestGetCfgOptionListOrStr(unittest.TestCase): def test_not_found_no_default(self): """None is returned if key is not found and no default given.""" config = {} @@ -124,16 +122,21 @@ class TestWriteFile(MockerTestCase): def test_restorecon_if_possible_is_called(self): """Make sure the selinux guard is called correctly.""" + my_file = os.path.join(self.tmp, "my_file") + with open(my_file, "w") as fp: + fp.write("My Content") + import_mock = self.mocker.replace(importer.import_module, passthrough=False) import_mock('selinux') - fake_se = FakeSelinux('/etc/hosts') + + fake_se = FakeSelinux(my_file) self.mocker.result(fake_se) self.mocker.replay() - with util.SeLinuxGuard("/etc/hosts") as is_on: + with util.SeLinuxGuard(my_file) as is_on: self.assertTrue(is_on) self.assertEqual(1, len(fake_se.restored)) - self.assertEqual('/etc/hosts', fake_se.restored[0]) + self.assertEqual(my_file, fake_se.restored[0]) class TestDeleteDirContents(MockerTestCase): @@ -201,20 +204,20 @@ class TestDeleteDirContents(MockerTestCase): self.assertDirEmpty(self.tmp) -class TestKeyValStrings(TestCase): +class TestKeyValStrings(unittest.TestCase): def test_keyval_str_to_dict(self): expected = {'1': 'one', '2': 'one+one', 'ro': True} cmdline = "1=one ro 2=one+one" self.assertEqual(expected, util.keyval_str_to_dict(cmdline)) -class TestGetCmdline(TestCase): +class TestGetCmdline(unittest.TestCase): def test_cmdline_reads_debug_env(self): os.environ['DEBUG_PROC_CMDLINE'] = 'abcd 123' self.assertEqual(os.environ['DEBUG_PROC_CMDLINE'], util.get_cmdline()) -class TestLoadYaml(TestCase): +class TestLoadYaml(unittest.TestCase): mydefault = "7b03a8ebace993d806255121073fed52" def test_simple(self): diff --git a/tools/build-on-freebsd b/tools/build-on-freebsd new file mode 100755 index 00000000..65d783f7 --- /dev/null +++ b/tools/build-on-freebsd @@ -0,0 +1,58 @@ +#!/bin/sh +# Since there is no official FreeBSD port yet, we need some way of building and +# installing cloud-init. This script takes care of building and installing. It +# will optionally make a first run at the end. + +fail() { echo "FAILED:" "$@" 1>&2; exit 1; } + +# Check dependencies: +depschecked=/tmp/c-i.dependencieschecked +pkgs=" + dmidecode + py27-argparse + py27-boto gpart sudo + py27-configobj py27-yaml + py27-Jinja2 + py27-oauth py27-serial + py27-prettytable + py27-requests py27-six + python py27-cheetah + py27-jsonpointer py27-jsonpatch +" +[ -f "$depschecked" ] || pkg install ${pkgs} || fail "install packages" +touch $depschecked + +# Required but unavailable port/pkg: py27-jsonpatch py27-jsonpointer +# Luckily, the install step will take care of this by installing it from pypi... + +# Build the code and install in /usr/local/: +python setup.py build +python setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd + +# Install the correct config file: +cp config/cloud.cfg-freebsd /usr/local/etc/cloud/cloud.cfg + +# Enable cloud-init in /etc/rc.conf: +sed -i.bak -e "/cloudinit_enable=.*/d" /etc/rc.conf +echo 'cloudinit_enable="YES"' >> /etc/rc.conf + +echo "Installation completed." + +if [ "$1" = "run" ]; then + echo "Ok, now let's see if it works." + + # Backup SSH keys + mv /etc/ssh/ssh_host_* /tmp/ + + # Remove old metadata + rm -rf /var/lib/cloud + + # Just log everything, quick&dirty + rm /usr/local/etc/cloud/cloud.cfg.d/05_logging.cfg + + # Start: + /usr/local/etc/rc.d/cloudinit start + + # Restore SSH keys + mv /tmp/ssh_host_* /etc/ssh/ +fi diff --git a/tools/hacking.py b/tools/hacking.py index 26a07c53..e7797564 100755 --- a/tools/hacking.py +++ b/tools/hacking.py @@ -71,7 +71,7 @@ def cloud_import_alphabetical(physical_line, line_number, lines): # with or without "as y" length = [2, 4] if (len(split_line) in length and len(split_previous) in length and - split_line[0] == "import" and split_previous[0] == "import"): + split_line[0] == "import" and split_previous[0] == "import"): if split_line[1] < split_previous[1]: return (0, "N306: imports not in alphabetical order (%s, %s)" % (split_previous[1], split_line[1])) @@ -154,7 +154,7 @@ def add_cloud(): if not inspect.isfunction(function): continue if name.startswith("cloud_"): - exec("pep8.%s = %s" % (name, name)) # pylint: disable=W0122 + exec("pep8.%s = %s" % (name, name)) if __name__ == "__main__": # NOVA based 'hacking.py' error codes start with an N @@ -163,7 +163,7 @@ if __name__ == "__main__": pep8.current_file = current_file pep8.readlines = readlines try: - pep8._main() # pylint: disable=W0212 + pep8._main() finally: if len(_missingImport) > 0: print >> sys.stderr, ("%i imports missing in this test environment" diff --git a/tools/mock-meta.py b/tools/mock-meta.py index c79f0598..dfbc2a71 100755 --- a/tools/mock-meta.py +++ b/tools/mock-meta.py @@ -23,7 +23,7 @@ import json import logging import os import random -import string # pylint: disable=W0402 +import string import sys import yaml @@ -306,7 +306,7 @@ class UserDataHandler(object): blob = "\n".join(lines) return blob.strip() - def get_data(self, params, who, **kwargs): # pylint: disable=W0613 + def get_data(self, params, who, **kwargs): if not params: return self._get_user_blob(who=who) return NOT_IMPL_RESPONSE @@ -427,8 +427,8 @@ def extract_opts(): def setup_fetchers(opts): - global meta_fetcher # pylint: disable=W0603 - global user_fetcher # pylint: disable=W0603 + global meta_fetcher + global user_fetcher meta_fetcher = MetaDataHandler(opts) user_fetcher = UserDataHandler(opts) diff --git a/tools/run-pep8 b/tools/run-pep8 index cfce5edd..d0a131f6 100755 --- a/tools/run-pep8 +++ b/tools/run-pep8 @@ -13,7 +13,7 @@ else base=`pwd`/tools/ fi -IGNORE="E501" # Line too long (these are caught by pylint) +IGNORE="" # King Arthur: Be quiet! ... Be Quiet! I Order You to Be Quiet. IGNORE="$IGNORE,E121" # Continuation line indentation is not a multiple of four diff --git a/tools/run-pylint b/tools/run-pylint deleted file mode 100755 index 0fe0c64a..00000000 --- a/tools/run-pylint +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash - -if [ $# -eq 0 ]; then - files=( bin/cloud-init $(find * -name "*.py" -type f) ) -else - files=( "$@" ); -fi - -RC_FILE="pylintrc" -if [ ! -f $RC_FILE ]; then - RC_FILE="../pylintrc" -fi - -cmd=( - pylint - --rcfile=$RC_FILE - --disable=R - --disable=I - --dummy-variables-rgx="_" - "${files[@]}" -) - -echo -e "\nRunning pylint:" -echo "${cmd[@]}" -"${cmd[@]}" - diff --git a/upstart/cloud-init-blocknet.conf b/upstart/cloud-init-blocknet.conf new file mode 100644 index 00000000..be09e7d8 --- /dev/null +++ b/upstart/cloud-init-blocknet.conf @@ -0,0 +1,83 @@ +# cloud-init-blocknet +# the purpose of this job is +# * to block networking from coming up until cloud-init-nonet has run +# * timeout if they all do not come up in a reasonable amount of time +description "block networking until cloud-init-local" +start on (starting network-interface + or starting network-manager + or starting networking) +stop on stopped cloud-init-local + +instance $JOB${INTERFACE:+/}${INTERFACE:-} +export INTERFACE +task + +script + set +e # you cannot trap TERM reliably with 'set -e' + SLEEP_CHILD="" + + static_network_up() { + local emitted="/run/network/static-network-up-emitted" + # /run/network/static-network-up-emitted is written by + # upstart (via /etc/network/if-up.d/upstart). its presense would + # indicate that static-network-up has already fired. + [ -e "$emitted" -o -e "/var/$emitted" ] + } + msg() { + local uptime="" idle="" msg="" + if [ -r /proc/uptime ]; then + read uptime idle < /proc/uptime + fi + msg="${UPSTART_INSTANCE}${uptime:+[${uptime}]}: $*" + echo "$msg" + } + + handle_sigterm() { + # if we received sigterm and static networking is up then it probably + # came from upstart as a result of 'stop on static-network-up' + msg "got sigterm" + if [ -n "$SLEEP_CHILD" ]; then + if ! kill $SLEEP_CHILD 2>/dev/null; then + [ ! -d "/proc/$SLEEP_CHILD" ] || + msg "hm.. failed to kill sleep pid $SLEEP_CHILD" + fi + fi + msg "stopped" + exit 0 + } + + dowait() { + msg "blocking $1 seconds" + # all this 'exec -a' does is get me a nicely named process in 'ps' + # ie, 'sleep-block-network-interface.eth1' + if [ -x /bin/bash ]; then + bash -c 'exec -a sleep-block-$1 sleep $2' -- "$UPSTART_INSTANCE" "$1" & + else + sleep "$1" & + fi + SLEEP_CHILD=$! + msg "sleepchild=$SLEEP_CHILD" + wait $SLEEP_CHILD + SLEEP_CHILD="" + } + + trap handle_sigterm TERM + + if [ -n "$INTERFACE" -a "${INTERFACE#lo}" != "${INTERFACE}" ]; then + msg "ignoring interface ${INTERFACE}"; + exit 0; + fi + + # static_network_up already occurred + static_network_up && { msg "static_network_up already"; exit 0; } + + # local-finished cloud-init-local success or failure + lfin="/run/cloud-init/local-finished" + disable="/etc/cloud/no-blocknet" + [ -f "$lfin" ] && { msg "$lfin found"; exit 0; } + [ -f "$disable" ] && { msg "$disable found"; exit 0; } + + dowait 120 + msg "gave up waiting for $lfin" + exit 1 +end script diff --git a/upstart/cloud-init-local.conf b/upstart/cloud-init-local.conf index 061fe406..5def043d 100644 --- a/upstart/cloud-init-local.conf +++ b/upstart/cloud-init-local.conf @@ -1,9 +1,16 @@ # cloud-init - the initial cloud-init job # crawls metadata service, emits cloud-config -start on mounted MOUNTPOINT=/ +start on mounted MOUNTPOINT=/ and mounted MOUNTPOINT=/run task console output -exec /usr/bin/cloud-init init --local +script + lfin=/run/cloud-init/local-finished + ret=0 + cloud-init init --local || ret=$? + [ -r /proc/uptime ] && read up idle < /proc/uptime || up="N/A" + echo "$ret up $up" > "$lfin" + exit $ret +end script diff --git a/upstart/cloud-init-nonet.conf b/upstart/cloud-init-nonet.conf index e8ebee96..6abf6573 100644 --- a/upstart/cloud-init-nonet.conf +++ b/upstart/cloud-init-nonet.conf @@ -46,7 +46,7 @@ script } dowait() { - msg "waiting $1 seconds for network device" + [ $# -eq 2 ] || msg "waiting $1 seconds for network device" sleep "$1" & SLEEP_CHILD=$! wait $SLEEP_CHILD @@ -58,12 +58,9 @@ script # static_network_up already occurred static_network_up && exit 0 - # obj.pkl comes from cloud-init-local (or previous boot and - # manual_cache_clean) - [ -f /var/lib/cloud/instance/obj.pkl ] && exit 0 - + dowait 5 silent dowait 10 - dowait 120 + dowait 115 msg "gave up waiting for a network device." : > /var/lib/cloud/data/no-net end script |