| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
Cloud-init caches any cloud metadata crawled during boot in the file
/run/cloud-init/instance-data.json. Cloud-init also standardizes some of
that metadata across all clouds. The command 'cloud-init query' surfaces a
simple CLI to query or format any cached instance metadata so that scripts
or end-users do not have to write tools to crawl metadata themselves.
Since 'cloud-init query' is runnable by non-root users, redact any
sensitive data from instance-data.json and provide a root-readable
unredacted instance-data-sensitive.json. Datasources can now define a
sensitive_metadata_keys tuple which will redact any matching keys
which could contain passwords or credentials from instance-data.json.
Also add the following standardized 'v1' instance-data.json keys:
- user_data: The base64encoded user-data provided at instance launch
- vendor_data: Any vendor_data provided to the instance at launch
- underscore_delimited versions of existing hyphenated keys:
instance_id, local_hostname, availability_zone, cloud_name
|
|
Allow users to provide '## template: jinja' as the first line or their
#cloud-config or custom script user-data parts. When this header exists,
the cloud-config or script will be rendered as a jinja template.
All instance metadata keys and values present in
/run/cloud-init/instance-data.json will be available as jinja variables
for the template. This means any cloud-config module or script can
reference any standardized instance data in templates and scripts.
Additionally, any standardized instance-data.json keys scoped below a
'<v#>' key will be promoted as a top-level key for ease of reference in
templates. This means that '{{ local_hostname }}' is the same as using the
latest '{{ v#.local_hostname }}'.
Since instance-data is written to /run/cloud-init/instance-data.json, make
sure it is persisted across reboots when the cached datasource opject is
reloaded.
LP: #1791781
|
|
Both landscape and puppet modules had issues with the way they wrote
/etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or
python2. This branch adds initial unit tests for both modules which will
get better exercise under both python2 and python3.
The unit tests shed light on a few issues:
- In the cc_landscape module py3 can't provide six.StringIO content to
ConfigParser.write, so we need to use six.BytesIO instead
- In the cc_puppet module, python <= 2.7 doesn't support using
six.StringIO as a context manager, so we drop the context manager
fanciness and directly set outputstream = StringIO().
- The docstring in cc_puppet is fixed to document the 'conf'
sub-key requiring valid puppet section names for each
key-value list.
LP: #1699282
LP: #1710932
|
|
This will change all instances of LOG.warn to LOG.warning as warn
is now a deprecated method. It will also make sure any logging
uses lazy logging by passing string format arguments as function
parameters.
|
|
This moves the warning code that was added specifically for
EC2 into a generic path at cloudinit/warnings.py.
It also adds support for writing warning files into the
warnings directory to be shown by Z99-cloudinit-warnings.sh.
|
|
When manual_cache_clean is enabled, write a file to
/var/lib/cloud/instance/manual-clean. That file can then be read by
ds-identify or another tool to indicate that manual cleaning is in place.
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
i dont want to expose this as i'd rather have some json there
or write to /run/cloud-init/status.json . would also like to indicate
'first_boot' somewhere.
|
|
previously, if you did: paths.get_ipath("bogus")
it would silenetly hand you back just the directory. now it
will fail, which seems much more sane.
|
|
== background ==
DataSource Mode (dsmode) is present in many datasources in cloud-init.
dsmode was originally added to cloud-init to specify when this datasource
should be 'realized'.
cloud-init has 4 stages of boot.
a.) cloud-init --local . network is guaranteed not present.
b.) cloud-init (--network). network is guaranteed present.
c.) cloud-config
d.) cloud-init final
'init_modules' [1] are run "as early as possible". And as such, are executed
in either 'a' or 'b' based on the datasource. However, executing them means
that user-data has been fully consumed. User-data and vendor-data may have
'#include http://...' which then rely on the network being present. boothooks
are an example of the things run in init_modules.
The 'dsmode' was a way for a user to indicate that init_modules
should run at 'a' (dsmode=local) or 'b' (dsmode=net) directly.
Things were further confused when a datasource could provide networking
configuration. Then, we needed to apply the networking config at 'a'
but if the user had provided boothooks that expected networking, then the
init_modules would need to be executed at 'b'. The config drive datasource
hacked its way through this and applies networking if *it* detects it is
a new instance.
== Suggested Change ==
The plan is to
1. incorporate 'dsmode' into DataSource superclass
2. make all existing datasources default to network
3. apply any networking configuration from a datasource on first boot only
apply_networking will always rename network devices when it runs.
for bug 1579130.
4. run init_modules at cloud-init (network) time frame unless datasource
is 'local'.
5. Datasources can provide a 'first_boot' method that will be called when
a new instance_id is found. This will allow the config drive's write_files
to be applied once.
Over all, this will very much simplify things. We'll no longer have
2 sources like DataSourceNoCloud and DataSourceNoCloudNet, but would just
have one source with a dsmode.
== Concerns ==
Some things have odd reliance on dsmode. For example, OpenNebula's get_hostname
uses it to determine if it should do a lookup of an ip address.
== Bugs to fix here ==
http://pad.lv/1577982 ConfigDrive: cloud-init fails to configure network from network_data.json
http://pad.lv/1579130 need to support systemd.link renaming of devices in container
http://pad.lv/1577844 Drop unnecessary blocking of all net udev rules
|
|
|
|
If the datasource's instance id contained a '/' then the instance_id path
would not be as expected under /var/lib/cloud/instances/instance_id.
LP: #1575938
|
|
r1213 (Ensure instance path is a child of cloud_dir) stripped the
leading path separator. This patch goes further by replacing all
path seperators with '_' which will avoid a deep directory structure
under /var/lib/cloud/instances.
LP: #1575938
|
|
A cloud has an instance-id metadata value in the form:
/Compute-$TENANT/$CLOUDUSERNAME/$UUID
The leading '/' causes /var/lib/cloud/instance to link to
/Compute-$TENANT/$CLOUDUSERNAME/$UUID rather than
/var/lib/cloud/instances/Compute-$TENANT/$CLOUDUSERNAME/$UUID
This patch strips the leading path separator from the instance-id.
LP: #1575938
|
|
Update make check target to run pep8 and run pyflakes or pyflakes3
depending on the value of 'PYVER'. This way the python3 build
environment does not need python2 and vice versa.
Also have make check run the 'yaml' test.
tox: have tox run pep8 in the pyflakes
|
|
|
|
|
|
to be behind trunk.
`tox -e py27` passes full test suite. Now to work on replacing mocker.
|
|
|
|
vendordata.
Vendordata is a datasource provided userdata-like blob that is parsed
similiarly to userdata, execept at the user's pleasure.
cloudinit/config/cc_scripts_vendor.py: added vendor script cloud config
cloudinit/config/cc_vendor_scripts_per_boot.py: added vendor per boot
cloud config
cloudinit/config/cc_vendor_scripts_per_instance.py: added vendor per
instance vendor cloud config
cloudinit/config/cc_vendor_scripts_per_once.py: added per once vendor
cloud config script
doc/examples/cloud-config-vendor-data.txt: documentation of vendor-data
examples
doc/vendordata.txt: documentation of vendordata for vendors
(RENAMED) tests/unittests/test_userdata.py => tests/unittests/test_userdata.py
TO: tests/unittests/test_userdata.py => tests/unittests/test_data.py:
userdata test cases are not expanded to confirm superiority over vendor
data.
bin/cloud-init: change instances of 'consume_userdata' to 'consume_data'
cloudinit/handlers/cloud_config.py: Added vendor script handling to default
cloud-config modules
cloudinit/handlers/shell_script.py: Added ability to change the path key to
support vendor provided 'vendor-scripts'. Defaults to 'script'.
cloudinit/helpers.py:
- Changed ConfigMerger to include handling of vendordata.
- Changed helpers to include paths for vendordata.
cloudinit/sources/__init__.py: Added functions for helping vendordata
- get_vendordata_raw(): returns vendordata unprocessed
- get_vendordata(): returns vendordata through userdata processor
- has_vendordata(): indicator if vendordata is present
- consume_vendordata(): datasource directive for indicating explict
user approval of vendordata consumption. Defaults to 'false'
cloudinit/stages.py: Re-jiggered for handling of vendordata
- _initial_subdirs(): added vendor script definition
- update(): added self._store_vendordata()
- [ADDED] _store_vendordata(): store vendordata
- _get_default_handlers(): modified to allow for filtering
which handlers will run against vendordata
- [ADDED] _do_handlers(): moved logic from consume_userdata
to _do_handlers(). This allows _consume_vendordata() and
_consume_userdata() to use the same code path.
- [RENAMED] consume_userdata() to _consume_userdata()
- [ADDED] _consume_vendordata() for handling vendordata
- run after userdata to get user cloud-config
- uses ConfigMerger to get the configuration from the
instance perspective about whether or not to use
vendordata
- [ADDED] consume_data() to call _consume_{user,vendor}data
cloudinit/util.py:
- [ADDED] get_nested_option_as_list() used by cc_vendor* for
getting a nested value from a dict and returned as a list
- runparts(): added 'exe_prefix' for running exe with a prefix,
used by cc_vendor*
config/cloud.cfg: Added vendor script execution as default
tests/unittests/test_runs/test_merge_run.py: changed consume_userdata() to
consume_data()
tests/unittests/test_runs/test_simple_run.py: changed consume_userdata() to
consume_data()
|
|
|
|
Instead of having a register default handler and a register
custom handler, just use the same function to do both but
provide a parameter to affect how overwritting of previously
existing content-types (which default handlers use to not
overwrite custom ones).
|
|
|
|
LP: #1203368
|
|
|
|
|
|
This check is a waste of a stat any time after the migrator module
had run. As it would take care of moving markers.
However, if the user runs:
sudo cloud-init modules --mode final
after an upgrade, they'd otherwise run any module that had a '-' in its
name again. To avoid that, we just return true in that case, and
inform the user how to run the migrator themselves.
|
|
|
|
that can be used to aid in the moving
of older versions of cloud-inits data
to newer versions of cloud-inits data.
1. Move the semaphores for the current instance
to there canonicalized names and use the
canonicalized in the file 'locking' code
|
|
1. Remove the usage of the path.join function
now that all code should be going through
the util file methods (and they can be
mocked out as needed).
2. Adjust all occurences of the above join
function to either not use it or replace
it with the standard os.path.join (which
can also be mocked out as needed)
3. Fix pylint from complaining about the
tests folder 'helpers.py' not being found
4. Add a pylintrc file that is used instead
of the options hidden in the 'run_pylint'
tool.
|
|
locked).
|
|
|
|
that will be the central point of config fetching
for both the 'init' stage and the 'module' stages
a. This helps those who want to understand exactly
what configs are fetched and how the merging occurs.
|
|
2. Fix pylint warning on set method of config parser.
|
|
2. Adjust comment on sources list from depends
3. For the /etc/timezone 'writing', add a header that says created by cloud-init
|
|
2. Use this object when returning semaphores
3. Adjust the debug statement to show this object
|
|
|
|
confusing when there is also a directory named 'config'
|
|
|
|
|
|
with non-real read/write paths is easier.
|
|
that use the path object.
1. This basically allows configuration to specify 'read_root' which will be used for read operations, right now just templates
and a 'write_root' via config that will be used for non read operations (ie 'var/lib/cloud' operations where most of the writing now happens)
|
|
stated if it ran.
|
|
ipath.
|
|
|
|
2. Ensure if iid is found that it is converted to a string before path joining.
|
|
|
|
2. Throw lock specific exception when semaphores can't be obtained
3. Make the paths constructor config variable named better to its actual usage.
|
