| Age | Commit message (Collapse) | Author |
|---|
|
This was added in d00126c167fc06d913d99cfc184bf3402cb8cf53, but not
removed in ef041fd822a2cf3a4022525e942ce988b1f95180 which removed the
one usage of it from the original commit.
|
|
Commit d00126c167fc06d913d99cfc184bf3402cb8cf53 regressed cloud-init
handling in multipart MIME user-data. Specifically, cloud-init would
examine the payload of the MIME part to determine what the content
type and subsequently which handler to use. This meant that user-data
which had shellscript payloads (starts with #!) were always handled
as shellscripts, rather than their declared MIME type and affected
when the payload was handled.
One failing scenario was a MIME part with text/cloud-boothook type
declared and a shellscript payload. This was run at shellscript
processing time rather than boothook time resulting in an change in
behavior from previous cloud-init releases.
To continue to support known scenarios where clouds have specifed
a MIME type of text/x-shellscript but provided a payload of something
other than shellscripts, we're changing the lookup logic to check for
the TYPES_NEEDED (text/plain, text/x-not-multipart) and only
text/x-shellscript.
It is safe to check text/x-shellscript parts as all shellscripts must
include the #! marker and will be detected as text/x-shellscript types.
If the content is missing the #! marker, it will not be excuted. If
the content is detected as something cloud-init supports, such as
#cloud-config the appropriate cloud-init handler will be used.
This change will fix hanldling for parts which were shellscripts but
ran with the wrong handler due to ignoring of the provided mime-type.
LP: #1888822
|
|
Build time feature flags are now defined in cloudinit/features.py.
Feature flags can be added to toggle configuration options or
deprecated features. Feature flag overrides can be placed in
cloudinit/feature_overrides.py. Further documentation can be found in
HACKING.rst.
Additionally, updated default behavior to exit with an exception
if #include can't retrieve resources as expected. This behavior
can be toggled with a feature flag.
LP: #1734939
|
|
On some platforms (old heat on OpenTelekomCloud), the user-data mime part
is mislabeled x-shellscript. cloud-init would not accept this unexpected
mime-type in multipart user-data. Cloud-init will now run find_ctype()
on the content of the mime-part to check if it matches known include types.
This patch is from Ryan Harper (inspired by my bug report and a
suggestion from Scott Moser) from the discussion on PR #234.
Signed-off-by: Kurt Garloff <kurt@garloff.de>
|
|
* url_helper: drop six
* url_helper: sort imports
* log: drop six
* log: sort imports
* handlers/__init__: drop six
* handlers/__init__: sort imports
* user_data: drop six
* user_data: sort imports
* sources/__init__: drop six
* sources/__init__: sort imports
* DataSourceOVF: drop six
* DataSourceOVF: sort imports
* sources/helpers/openstack: drop six
* sources/helpers/openstack: sort imports
* mergers/m_str: drop six
This also allowed simplification of the logic, as we will never
encounter a non-string text type.
* type_utils: drop six
* mergers/m_dict: drop six
* mergers/m_list: drop six
* cmd/query: drop six
* mergers/__init__: drop six
* net/cmdline: drop six
* reporting/handlers: drop six
* reporting/handlers: sort imports
|
|
Headers param was accidentally omitted and no longer passed through to
readurl due to a previous commit.
To avoid this omission of params in the future, drop positional param
definitions from read_file_or_url and pass all kwargs through to readurl
when we are not operating on a file.
In util:read_seeded, correct the case where invalid positional param
file_retries was being passed into read_file_or_url.
Also drop duplicated file:// prefix addition from read_seeded because
read_file_or_url does that work anyway.
LP: #1854084
|
|
When creating the multipart mime message that is written as
user-data.txt.i, cloud-init losing data on conversion to some things
as a string.
LP: #1768600
Author: Scott Moser <smoser@ubuntu.com>
Co-Authored-By: Chad Smith <chad.smith@canonical.com>
|
|
The result of a read_file_or_url on a file and on a url would differ
in behavior.
str(UrlResponse) would return UrlResponse.contents.decode('utf-8')
while
str(FileResponse) would return str(FileResponse.contents)
The difference being "b'foo'" versus "foo".
As part of the general goal of cleaning util, move read_file_or_url
into url_helper.
|
|
On python 2.7 and earlier (CentOS 6 & 7), UrlErrors raised by requests do
not report the url which failed. In such cases, append the url if not
present in the error message.
This fixes nightly CI failures at
https://jenkins.ubuntu.com/server/view/cloud-init/.
|
|
Without this the entire stage can fail, which will leave an instance
unaccessible.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
Reviewed-by: Ben Cressey <bcressey@amazon.com>
|
|
This will change all instances of LOG.warn to LOG.warning as warn
is now a deprecated method. It will also make sure any logging
uses lazy logging by passing string format arguments as function
parameters.
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
when user-data was not decodable, cloud-init would raise exception.
LP: #1532072
|
|
|
|
handling of cloud-config-archive input would fail in fully_decoded_payload.
part.get_charset() would return a Charset object, but
get_charset.input_codec is a string suitable for passing to decode.
This handles that correctly, and is more careful about binary data inside
input.
The test added verifies that cloud-config inside a cloud-config-archive
is handled correctly and also that binary data there is ignored without
exceptions raised.
LP: #1445143
|
|
This is to work-around the fact that email.message_from_string uses
cStringIO in Python 2.6, which can't handle Unicode.
|
|
|
|
an include can include a gzip type also or binary data
so avoid conversion here.
|
|
just seems to make more sense to decode here.
Add a test showing the previous failure (testBytesInPayload)
And one that should pass (testStringInPayload)
Also, add a test for unencoded content in the ovf xml (test_userdata_plain)
And explicitly set encoding on another test (test_userdata_found).
|
|
base64 encoded data.
|
|
|
|
- Refactor "fully" decoding the payload of a text/* part. In Python 3,
decode=True only means to decode according to Content-Transfer-Encoding, not
according to any charset in the Content-Type header. So do that.
|
|
- Remove str() wrappers to second argument to write_files() where it is no
longer necessary.
Also: Fixed a couple of other octal literals which clearly weren't being
tested.
|
|
|
|
|
|
to be behind trunk.
`tox -e py27` passes full test suite. Now to work on replacing mocker.
|
|
This fixes up many long lines to be < 80 chars and some other
pylint issues. pylint 1.1 (in trusty) is now complaining about
the lazy logging, so I'll clean that up when I touch things.
|
|
vendordata.
Vendordata is a datasource provided userdata-like blob that is parsed
similiarly to userdata, execept at the user's pleasure.
cloudinit/config/cc_scripts_vendor.py: added vendor script cloud config
cloudinit/config/cc_vendor_scripts_per_boot.py: added vendor per boot
cloud config
cloudinit/config/cc_vendor_scripts_per_instance.py: added vendor per
instance vendor cloud config
cloudinit/config/cc_vendor_scripts_per_once.py: added per once vendor
cloud config script
doc/examples/cloud-config-vendor-data.txt: documentation of vendor-data
examples
doc/vendordata.txt: documentation of vendordata for vendors
(RENAMED) tests/unittests/test_userdata.py => tests/unittests/test_userdata.py
TO: tests/unittests/test_userdata.py => tests/unittests/test_data.py:
userdata test cases are not expanded to confirm superiority over vendor
data.
bin/cloud-init: change instances of 'consume_userdata' to 'consume_data'
cloudinit/handlers/cloud_config.py: Added vendor script handling to default
cloud-config modules
cloudinit/handlers/shell_script.py: Added ability to change the path key to
support vendor provided 'vendor-scripts'. Defaults to 'script'.
cloudinit/helpers.py:
- Changed ConfigMerger to include handling of vendordata.
- Changed helpers to include paths for vendordata.
cloudinit/sources/__init__.py: Added functions for helping vendordata
- get_vendordata_raw(): returns vendordata unprocessed
- get_vendordata(): returns vendordata through userdata processor
- has_vendordata(): indicator if vendordata is present
- consume_vendordata(): datasource directive for indicating explict
user approval of vendordata consumption. Defaults to 'false'
cloudinit/stages.py: Re-jiggered for handling of vendordata
- _initial_subdirs(): added vendor script definition
- update(): added self._store_vendordata()
- [ADDED] _store_vendordata(): store vendordata
- _get_default_handlers(): modified to allow for filtering
which handlers will run against vendordata
- [ADDED] _do_handlers(): moved logic from consume_userdata
to _do_handlers(). This allows _consume_vendordata() and
_consume_userdata() to use the same code path.
- [RENAMED] consume_userdata() to _consume_userdata()
- [ADDED] _consume_vendordata() for handling vendordata
- run after userdata to get user cloud-config
- uses ConfigMerger to get the configuration from the
instance perspective about whether or not to use
vendordata
- [ADDED] consume_data() to call _consume_{user,vendor}data
cloudinit/util.py:
- [ADDED] get_nested_option_as_list() used by cc_vendor* for
getting a nested value from a dict and returned as a list
- runparts(): added 'exe_prefix' for running exe with a prefix,
used by cc_vendor*
config/cloud.cfg: Added vendor script execution as default
tests/unittests/test_runs/test_merge_run.py: changed consume_userdata() to
consume_data()
tests/unittests/test_runs/test_simple_run.py: changed consume_userdata() to
consume_data()
|
|
|
|
|
|
|
|
|
|
Instead of being restricted to only gzip
compressing the overall mime segment or
individual included segments, allow for
each mime segment to be gzip compressed.
LP: #1203203
|
|
|
|
|
|
|
|
LP: #1065116
|
|
its original content type said it is, make sure we set
the new value, also unsure if the old top level message
should have the same header (which will flip-flop).
|
|
indexes (since they will be handled beforehand) and fix
the types being checked on the root of the archive format
to be a tuple instead of a list (which oddly causes complaints).
|
|
variable has a little more meaning and by default look in
metadata for 'launch-index' and have ec2 instead look for
a different variable (thus allowing more datasources to just work).
|
|
'launch-index' key that we copy that key over to the right
header (which will then be used later when assigning the
'real' header when the message is attached)
|
|
before we look into the payload as well as make the skip
test a function that the datasource module can also use.
|
|
userdata based on a launch-index (or leave userdata
alone if none is provided by the datasource). This
works by doing the following.
1. Adjusting the userdata processor to attempt to
inject a "Launch-Index" header into the messages
headers (by either taking a header that already exists
or by looking into the payload to see if it exists
there).
2. Adjust the get_userdata ds function to apply a filter
on the returned userdata (defaulting to false) that
will now use the datasources get_launch_index value
to restrict the 'final' message used in consuming
user data (the same behavior if not existent).
3. Further down the line processes that use the 'resultant'
userdata now will only see the ones for there own launch
index (ie cloud-config will be restricted automatically
and so on) and are unaffected (although they can now
ask the cloud object or the datasource for its launch index
via the above new ds method.
|
|
make pep8 now is silent on precise's pep8 ( 0.6.1-2ubuntu2).
|
|
other code to have user/group parsing in util instead
of in stages.py, renames decomp_str to decomp_gzip since
it is more meaningful when named that (as thats all it can
decompress).
|
|
2. Fix pylint warning in userdata about unused variable
|
|
easily
|
|
|
|
turn it back off, if one of those is encountered
|
|
a. This allows for more properties to be added as needed in the future, instead of being very restrictive.
2. Fix up all uses of the url reading to now use this new response object.
3. Also fixup user data including, such that if no response actual occurs the url content is not further processed.
|
