From 34595e9b4abacc10ac599aad97c95861af34ea54 Mon Sep 17 00:00:00 2001 From: Joshua Powers Date: Thu, 7 Dec 2017 12:54:46 -0800 Subject: tests: Enable AWS EC2 Integration Testing This enables integration tests to utilize AWS EC2 as a testing platform by utilizing the boto3 Python library. Usage will create and delete a custom VPC for every run. All resources will be tagged with the ec2 tag, 'cii', and the date (e.g. cii-20171220-102452). The VPC is setup with both IPv4 and IPv6 capabilities, but will only hand out IPv4 addresses by default. Instances will have complete Internet access and have full ingress and egress access (i.e. no firewall). SSH keys are generated with each run of the integration tests with the key getting uploaded to AWS at the start of tests and deleted on exit. To enable creation when the platform is setup the SSH generation code is moved to be completed by the platform setup and not during image setup. The nocloud-kvm platform was updated with this change. Creating a custom image will utilize the same clean script, boot_clean_script, that the LXD platform uses as well. The custom AMI is generated, used, and de-registered after a test run. The default instance type is set to t2.micro. This is one of the smallest instance types and is free tier eligible. The default timeout for ec2 was increased to 300 from 120 as many tests hit up against the 2 minute timeout and depending on region load can go over. Documentation for the AWS platform was added with the expected configuration files for the platform to be used. There are some additional whitespace changes included as well. pylint exception was added for paramiko and simplestreams. In the past these were not already flagged due to no __init__.py in the subdirectories of files that used these. boto3 was added to the list of dependencies in the tox ci-test runner. In order to grab console logs on EC2 the harness will now shut down an instance before terminating and before collecting the console log. This is to address a behavior of EC2 where the console log is refreshed very infrequently, but one point when it is refreshed is after shutdown. --- doc/rtd/topics/tests.rst | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst index d668e3f4..bf04bb3c 100644 --- a/doc/rtd/topics/tests.rst +++ b/doc/rtd/topics/tests.rst @@ -118,19 +118,19 @@ TreeRun and TreeCollect If working on a cloud-init feature or resolving a bug, it may be useful to run the current copy of cloud-init in the integration testing environment. -The integration testing suite can automatically build a deb based on the +The integration testing suite can automatically build a deb based on the current working tree of cloud-init and run the test suite using this deb. The ``tree_run`` and ``tree_collect`` commands take the same arguments as -the ``run`` and ``collect`` commands. These commands will build a deb and -write it into a temporary file, then start the test suite and pass that deb +the ``run`` and ``collect`` commands. These commands will build a deb and +write it into a temporary file, then start the test suite and pass that deb in. To build a deb only, and not run the test suite, the ``bddeb`` command can be used. Note that code in the cloud-init working tree that has not been committed when the cloud-init deb is built will still be included. To build a cloud-init deb from or use the ``tree_run`` command using a copy of -cloud-init located in a different directory, use the option ``--cloud-init +cloud-init located in a different directory, use the option ``--cloud-init /path/to/cloud-init``. .. code-block:: bash @@ -383,7 +383,7 @@ Development Checklist * Valid unit tests validating output collected * Passes pylint & pep8 checks * Placed in the appropriate sub-folder in the test cases directory -* Tested by running the test: +* Tested by running the test: .. code-block:: bash @@ -392,6 +392,32 @@ Development Checklist --test modules/your_test.yaml \ [--deb ] + +Platforms +========= + +EC2 +--- +To run on the EC2 platform it is required that the user has an AWS credentials +configuration file specifying his or her access keys and a default region. +These configuration files are the standard that the AWS cli and other AWS +tools utilize for interacting directly with AWS itself and are normally +generated when running ``aws configure``: + +.. code-block:: bash + + $ cat $HOME/.aws/credentials + [default] + aws_access_key_id = + aws_secret_access_key = + +.. code-block:: bash + + $ cat $HOME/.aws/config + [default] + region = us-west-2 + + Architecture ============ @@ -455,7 +481,7 @@ replace the default. If the data is a dictionary then the value will be the result of merging that dictionary from the default config and that dictionary from the overrides. -Merging is done using the function +Merging is done using the function ``tests.cloud_tests.config.merge_config``, which can be examined for more detail on config merging behavior. -- cgit v1.2.3 From df182de41fd75bc0cf2b1de2b511c37527d45475 Mon Sep 17 00:00:00 2001 From: aRkadeFR Date: Tue, 23 Jan 2018 15:48:56 -0500 Subject: docs: Fix typos in docs and one debug message. Fix obvious typos. Replace 'for for' with a 'for'. --- cloudinit/config/cc_rsyslog.py | 10 +++++----- cloudinit/sources/__init__.py | 2 +- doc/rtd/topics/network-config-format-v1.rst | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/cloudinit/config/cc_rsyslog.py b/cloudinit/config/cc_rsyslog.py index 50ff9e35..af08788c 100644 --- a/cloudinit/config/cc_rsyslog.py +++ b/cloudinit/config/cc_rsyslog.py @@ -20,15 +20,15 @@ which defaults to ``20-cloud-config.conf``. The rsyslog config directory to write config files to may be specified in ``config_dir``, which defaults to ``/etc/rsyslog.d``. -A list of configurations for for rsyslog can be specified under the ``configs`` -key in the ``rsyslog`` config. Each entry in ``configs`` is either a string or -a dictionary. Each config entry contains a configuration string and a file to +A list of configurations for rsyslog can be specified under the ``configs`` key +in the ``rsyslog`` config. Each entry in ``configs`` is either a string or a +dictionary. Each config entry contains a configuration string and a file to write it to. For config entries that are a dictionary, ``filename`` sets the target filename and ``content`` specifies the config string to write. For config entries that are only a string, the string is used as the config string to write. If the filename to write the config to is not specified, the value of -the ``config_filename`` key is used. A file with the selected filename will -be written inside the directory specified by ``config_dir``. +the ``config_filename`` key is used. A file with the selected filename will be +written inside the directory specified by ``config_dir``. The command to use to reload the rsyslog service after the config has been updated can be specified in ``service_reload_command``. If this is set to diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py index 4b819ce6..a05ca2f6 100644 --- a/cloudinit/sources/__init__.py +++ b/cloudinit/sources/__init__.py @@ -448,7 +448,7 @@ def find_source(sys_cfg, distro, paths, ds_deps, cfg_list, pkg_list, reporter): # Return an ordered list of classes that match (if any) def list_sources(cfg_list, depends, pkg_list): src_list = [] - LOG.debug(("Looking for for data source in: %s," + LOG.debug(("Looking for data source in: %s," " via packages %s that matches dependencies %s"), cfg_list, pkg_list, depends) for ds_name in cfg_list: diff --git a/doc/rtd/topics/network-config-format-v1.rst b/doc/rtd/topics/network-config-format-v1.rst index ce3a1bde..2f8ab54c 100644 --- a/doc/rtd/topics/network-config-format-v1.rst +++ b/doc/rtd/topics/network-config-format-v1.rst @@ -349,7 +349,7 @@ For any network device (one of the Config Types) users can define a list of entries will create interface alias allowing a single interface to use different ip configurations. -Valid keys for for ``subnets`` include the following: +Valid keys for ``subnets`` include the following: - ``type``: Specify the subnet type. - ``control``: Specify manual, auto or hotplug. Indicates how the interface -- cgit v1.2.3 From f7deaf15acf382d62554e2b1d70daa9a9109d542 Mon Sep 17 00:00:00 2001 From: Chad Smith Date: Mon, 29 Jan 2018 13:04:37 -0700 Subject: docs: Update RTD content for cloud-init subcommands. Give a bit more detailed information which others can quickly reference to discover new CLI subcommand functionality. This section was a bit stale as we've introduced cloud-init status, clean and analyze content that was a bit under represented. Since we've had a few request of externals who try to run cloud-init subcommands on the commandline instead of rebooting, it's probably worth a bit of content here to describe how those init and module subcommands work. --- doc/rtd/topics/boot.rst | 13 +++- doc/rtd/topics/capabilities.rst | 160 ++++++++++++++++++++++++++++++++++++++-- doc/rtd/topics/debugging.rst | 1 + doc/rtd/topics/modules.rst | 2 + 4 files changed, 166 insertions(+), 10 deletions(-) (limited to 'doc') diff --git a/doc/rtd/topics/boot.rst b/doc/rtd/topics/boot.rst index 859409a6..f2976fdf 100644 --- a/doc/rtd/topics/boot.rst +++ b/doc/rtd/topics/boot.rst @@ -1,3 +1,5 @@ +.. _boot_stages: + *********** Boot Stages *********** @@ -74,7 +76,7 @@ Network * **systemd service**: ``cloud-init.service`` * **runs**: After local stage and configured networking is up. * **blocks**: As much of remaining boot as possible. - * **modules**: ``init_modules`` + * **modules**: ``cloud_init_modules`` in **/etc/cloud/cloud.cfg** This stage requires all configured networking to be online, as it will fully process any user-data that is found. Here, processing means: @@ -104,7 +106,7 @@ Config * **systemd service**: ``cloud-config.service`` * **runs**: After network stage. * **blocks**: None. - * **modules**: ``config_modules`` + * **modules**: ``cloud_config_modules`` in **/etc/cloud/cloud.cfg** This stage runs config modules only. Modules that do not really have an effect on other stages of boot are run here. @@ -115,7 +117,7 @@ Final * **systemd service**: ``cloud-final.service`` * **runs**: As final part of boot (traditional "rc.local") * **blocks**: None. - * **modules**: ``final_modules`` + * **modules**: ``cloud_final_modules`` in **/etc/cloud/cloud.cfg** This stage runs as late in boot as possible. Any scripts that a user is accustomed to running after logging into a system should run correctly here. @@ -125,4 +127,9 @@ Things that run here include * configuration management plugins (puppet, chef, salt-minion) * user-scripts (including ``runcmd``). +For scripts external to cloud-init looking to wait until cloud-init +finished, the ``cloud-init status`` subcommand can help block external +scripts until cloud-init is done without having to write your own systemd +units dependency chains. See :ref:`cli_status` for more info. + .. vi: textwidth=78 diff --git a/doc/rtd/topics/capabilities.rst b/doc/rtd/topics/capabilities.rst index 31eaba53..ae3a0c74 100644 --- a/doc/rtd/topics/capabilities.rst +++ b/doc/rtd/topics/capabilities.rst @@ -1,3 +1,5 @@ +.. _capabilities: + ************ Capabilities ************ @@ -39,17 +41,19 @@ Currently defined feature names include: see :ref:`network_config_v2` documentation for examples. -CLI Interface : +CLI Interface +============= -``cloud-init features`` will print out each feature supported. If cloud-init -does not have the features subcommand, it also does not support any features -described in this document. + The command line documentation is accessible on any cloud-init +installed system: .. code-block:: bash % cloud-init --help - usage: cloud-init [-h] [--version] [--file FILES] [--debug] [--force] - {init,modules,query,single,dhclient-hook,features} ... + usage: cloud-init [-h] [--version] [--file FILES] + [--debug] [--force] + {init,modules,single,dhclient-hook,features,analyze,devel,collect-logs,clean,status} + ... optional arguments: -h, --help show this help message and exit @@ -61,7 +65,7 @@ described in this document. your own risk) Subcommands: - {init,modules,single,dhclient-hook,features,analyze,devel} + {init,modules,single,dhclient-hook,features,analyze,devel,collect-logs,clean,status} init initializes cloud-init and performs initial modules modules activates modules using a given configuration key single run a single module @@ -69,11 +73,153 @@ described in this document. features list defined features analyze Devel tool: Analyze cloud-init logs and data devel Run development tools + collect-logs Collect and tar all cloud-init debug info + clean Remove logs and artifacts so cloud-init can re-run. + status Report cloud-init status or wait on completion. + +CLI Subcommand details +====================== + +.. _cli_features: + +cloud-init features +------------------- +Print out each feature supported. If cloud-init does not have the +features subcommand, it also does not support any features described in +this document. + +.. code-block:: bash % cloud-init features NETWORK_CONFIG_V1 NETWORK_CONFIG_V2 +.. _cli_status: + +cloud-init status +----------------- +Report whether cloud-init is running, done, disabled or errored. Exits +non-zero if an error is detected in cloud-init. + * **--long**: Detailed status information. + * **--wait**: Block until cloud-init completes. + +.. code-block:: bash + + % cloud-init status --long + status: done + time: Wed, 17 Jan 2018 20:41:59 +0000 + detail: + DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net] + + # Cloud-init running still short versus long options + % cloud-init status + status: running + % cloud-init status --long + status: running + time: Fri, 26 Jan 2018 21:39:43 +0000 + detail: + Running in stage: init-local + +.. _cli_collect_logs: + +cloud-init collect-logs +----------------------- +Collect and tar cloud-init generated logs, data files and system +information for triage. This subcommand is integrated with apport. + +**Note**: Ubuntu users can file bugs with `ubuntu-bug cloud-init` to +automaticaly attach these logs to a bug report. + +Logs collected are: + + * /var/log/cloud-init*log + * /run/cloud-init + * cloud-init package version + * dmesg output + * journalctl output + * /var/lib/cloud/instance/user-data.txt + +.. _cli_analyze: + +cloud-init analyze +------------------ +Get detailed reports of where cloud-init spends most of its time. See +:ref:`boot_time_analysis` for more info. + + * **blame** Report ordered by most costly operations. + * **dump** Machine-readable JSON dump of all cloud-init tracked events. + * **show** show time-ordered report of the cost of operations during each + boot stage. + +.. _cli_devel: + +cloud-init devel +---------------- +Collection of development tools under active development. These tools will +likely be promoted to top-level subcommands when stable. + + * ``cloud-init devel schema``: A **#cloud-config** format and schema + validator. It accepts a cloud-config yaml file and annotates potential + schema errors locally without the need for deployment. Schema + validation is work in progress and supports a subset of cloud-config + modules. + +.. _cli_clean: + +cloud-init clean +---------------- +Remove cloud-init artifacts from /var/lib/cloud and optionally reboot the +machine to so cloud-init re-runs all stages as it did on first boot. + + * **--logs**: Optionally remove /var/log/cloud-init*log files. + * **--reboot**: Reboot the system after removing artifacts. + +.. _cli_init: + +cloud-init init +--------------- +Generally run by OS init systems to execute cloud-init's stages +*init* and *init-local*. See :ref:`boot_stages` for more info. +Can be run on the commandline, but is generally gated to run only once +due to semaphores in **/var/lib/cloud/instance/sem/** and +**/var/lib/cloud/sem**. + + * **--local**: Run *init-local* stage instead of *init*. + +.. _cli_modules: + +cloud-init modules +------------------ +Generally run by OS init systems to execute *modules:config* and +*modules:final* boot stages. This executes cloud config :ref:`modules` +configured to run in the init, config and final stages. The modules are +declared to run in various boot stages in the file +**/etc/cloud/cloud.cfg** under keys **cloud_init_modules**, +**cloud_init_modules** and **cloud_init_modules**. Can be run on the +commandline, but each module is gated to run only once due to semaphores +in ``/var/lib/cloud/``. + + * **--mode (init|config|final)**: Run *modules:init*, *modules:config* or + *modules:final* cloud-init stages. See :ref:`boot_stages` for more info. + +.. _cli_single: + +cloud-init single +----------------- +Attempt to run a single named cloud config module. The following example +re-runs the cc_set_hostname module ignoring the module default frequency +of once-per-instance: + + * **--name**: The cloud-config module name to run + * **--frequency**: Optionally override the declared module frequency + with one of (always|once-per-instance|once) + +.. code-block:: bash + + % cloud-init single --name set_hostname --frequency always + +**Note**: Mileage may vary trying to re-run each cloud-config module, as +some are not idempotent. .. _Cloud-init: https://launchpad.net/cloud-init .. vi: textwidth=78 diff --git a/doc/rtd/topics/debugging.rst b/doc/rtd/topics/debugging.rst index 4e43dd57..c2b47edc 100644 --- a/doc/rtd/topics/debugging.rst +++ b/doc/rtd/topics/debugging.rst @@ -7,6 +7,7 @@ Overview This topic will discuss general approaches for test and debug of cloud-init on deployed instances. +.. _boot_time_analysis: Boot Time Analysis - cloud-init analyze ====================================== diff --git a/doc/rtd/topics/modules.rst b/doc/rtd/topics/modules.rst index cdb0f419..7b146751 100644 --- a/doc/rtd/topics/modules.rst +++ b/doc/rtd/topics/modules.rst @@ -1,3 +1,5 @@ +.. _modules: + ******* Modules ******* -- cgit v1.2.3