| Age | Commit message (Collapse) | Author |
|---|
|
Applied Black and isort, fixed any linting issues, updated tox.ini
and CI.
|
|
Instead of logging the token values used log the headers and replace the actual
values with the string 'REDACTED'. This allows users to examine cloud-init.log
and see that the IMDSv2 token header is being used but avoids leaving the value
used in the log file itself.
LP: #1863943
|
|
* ec2: Add support for AWS IMDS v2 (session-oriented)
AWS now supports a new version of fetching Instance Metadata[1].
Update cloud-init's ec2 utility functions and update ec2 derived
datasources accordingly. For DataSourceEc2 (versus ec2-look-alikes)
cloud-init will issue the PUT request to obtain an API token for
the maximum lifetime and then all subsequent interactions with the
IMDS will include the token in the header.
If the API token endpoint is unreachable on Ec2 platform, log a
warning and fallback to using IMDS v1 and which does not use
session tokens when communicating with the Instance metadata
service.
We handle read errors, typically seen if the IMDS is beyond one
etwork hop (IMDSv2 responses have a ttl=1), by setting the api token
to a disabled value and then using IMDSv1 paths.
To support token-based headers, ec2_utils functions were updated
to support custom headers_cb and exception_cb callback functions
so Ec2 could store, or refresh API tokens in the event of token
becoming stale.
[1] https://docs.aws.amazon.com/AWSEC2/latest/ \
UserGuide/ec2-instance-metadata.html \
#instance-metadata-v2-how-it-works
|
|
The result of a read_file_or_url on a file and on a url would differ
in behavior.
str(UrlResponse) would return UrlResponse.contents.decode('utf-8')
while
str(FileResponse) would return str(FileResponse.contents)
The difference being "b'foo'" versus "foo".
As part of the general goal of cleaning util, move read_file_or_url
into url_helper.
|
|
The recent change to exception_cb missed this caller.
The result was a slow test.
|
|
The instance identity document is a better source for region information,
partly because region isn't actually in meta-data at all, only
availability-zone, which happens to be named similarly.
Reviewed-by: Ethan Faust <efaust@amazon.com>
Reviewed-by: Cyle Riggs <cyler@amazon.com>
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
[ajorgens@amazon.com: rebase onto 0.7.9]
[ajorgens@amazon.com: changes per merge proposal discussions]
|
|
This will change all instances of LOG.warn to LOG.warning as warn
is now a deprecated method. It will also make sure any logging
uses lazy logging by passing string format arguments as function
parameters.
|
|
the MetadataLeafDecoder would return a bytes value b'' instead of
an empty string if the value of a key was empty. In all other cases
the value would be a string.
This was discovered when trying to json.dumps(get_instance_metadata())
on a recent OpenStack, where the value of 'public-ipv4' was empty.
The attempt to dump that with json would raise
TypeError: b'' is not JSON serializable
|
|
On EC2, instance metadata can include credentials that remain valid for as
much as 6 hours. Reading these and allowing them to be pickled represents
a potential vulnerability if a snapshot of the disk is taken and shared as
part of an AMI.
This skips security-credentials when walking the meta-data tree.
LP: #1638312
Reviewed-by: Ian Weller <iweller@amazon.com>
Reviewed-by: Ben Cressey <bcressey@amazon.com>
Reported-by: Kyle Barnes <barnesky@amazon.com>
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
The bit.ly link seems to have expired. Replace it with link to
'latest' version of EC2 docs.
|
|
|
|
The change to get_instance_userdata is to fix an issue that
was causing retry in the test when it was not desired.
if user_data returned 404 it means "there was no user-data", so
dont bother retrying. However, _skip_retry_on_codes was returning
False indicating that readurl should retry.
test_merging was creating 2500 random tests, shrink that down to 100.
test_seed_runs is still on my system the slowest test, but
taking < .5 seconds where it was taking > 3.
|
|
|
|
UrlResponse: biggest change... make readurl return bytes, making user
know what to do with it.
util: add load_tfile_or_url for loading text file or url
as read_file_or_url now returns bytes
ec2_utils: all meta-data is text, remove non-obvious string translations
DigitalOcean: adjust for ec2_utils
DataSourceGCE, DataSourceMAAS: user-data is binary other fields are text.
openstack.py: read paths without decoding to text. This is ok as paths
other than user-data are json, and load_json will handle
load_file still returns text, and that is what most things use.
|
|
to be behind trunk.
`tox -e py27` passes full test suite. Now to work on replacing mocker.
|
|
|
|
This change works around a bug in CloudStack's EC2 api implementation.
That is filed upstream at [1].
The fix is safe for openstack and EC2 use cases as well.
EC2 and OpenStacks' EC2 metadata service both return a list with
access to either of:
<url_base>/latest/meta-data
<url_base>/latest/meta-data/
Additionally, the responses explicitly contain a trailing '/' for
items that have a child. The ec2_utils code then just re-uses the trailng
/ there. Thus, only the top level request for 'meta-data/' needs
the explicit fix.
This also changes test cases. Those test cases failed without fixing
them. If ever this regressed, those would fail again.
--
[1] https://issues.apache.org/jira/browse/CLOUDSTACK-7405
LP: #1356855
|
|
|
|
|
|
Openstack has a unique derivative datasource
that is gaining usage. Previously the config
drive datasource provided part of this functionality
as well as the ec2 datasource, but since new
functionality is being added to openstack is
seems benefical to combine the used parts into
one datasource just made for handling openstack
deployments.
This patch factors out the common logic shared
between the config drive and the openstack
metadata datasource and places that in a shared
helper file and then creates a new openstack
datasource that readers from the openstack metadata
service and refactors the config drive datasource
to use this common logic.
|
|
|
|
When a 404 http code comes back from the fetching of ec2
data, instead of retrying immediatly stop the fetching process
and in the userdata fetching function handle this case as a
special case of no userdata being fetched (an empty string
in this case).
|
|
the Requires would get that string rendered into the package's
Depends/Requires (rather than BuildDepends/BuildRequires).
We should have BuildDepends/BuildRequires too, but since
trunk's package builds do not run 'make test', this isn't a big deal.
This also adds 'test-requires' for httpretty.
|
|
|
|
|
|
|
|
|
|
This replacement uses our own userdata/metadata ec2 webservice
parser that we can easily modify, it also automatically allows
for reading the ec2 userdata/metdata from files and also brings
in the usage of requests instead of boto's usage of urllib which
did not support ssl properly.
|
|
|
|
cloudinit/sources/DataSourceCloudStack.py and
cloudinit/sources/DataSourceEc2.py are reverted entirely back to trunk versions
now, rather than the non-behavior change that was left in place.
Also, remove inadvertantly added trailing newline from cloudinit/ec2_utils.py
Overall, this just makes the diff when merged to trunk carry more focused
changes.
|
|
|
|
|
|
|
|
this shouldn't change anything, only the signatures of the methods.
|
|
* cloudinit/distros/parsers/resolv_conf.py
added some pylint overrides with 'plXXXXX' syntax.
example: # pl51222 pylint: disable=E0102
The pl51222 there means: http://www.logilab.org/ticket/51222
This specific issue is present in 12.04 pylint, but not 13.04.
* pylint doesn't like the requests special handling we have.
which makes sense as it is only checking versus one specific version.
* general pep8 and pylint cleanups.
|
|
|
|
|
|
|
|
It seems like its possible that boto 2.5.2 and below have
the lazy loading metadata dictionary so as a precaution
we will always take the hit of unlazying the metadata dictionary
by traversing it which in the non-lazy dictionary case has
no effect (its marginal). This also removes the need to check
the boto version and the dependency on setup tools just for
this case.
|
|
|
|
|
|
|
|
LP: #1068801
|
|
are used that lazily load the metadata from the
ec2 metadata service.
1. Add a ec2_utils module that checks which version
of boto is being used and under the right versions
the metadata dictionary will be expanded.
2. Use this new ec2_utils module in the cloudstack and ec2
datasources as there entrypoints into boto.
|
|
userdata and metadata. Add in this crawling functionality
to the ec2_utils module that will fully crawl (not lazily)
the ec2 metadata and parse it in the same manner as boto.
1. Make the ec2 datasource + cloudstack now call into these.
2. Fix phone_home due to urllib3 change (TBD)
|
|
use ssl correctly (they can not do validation due to
the underlying usage of urllib/httplib). Adjust to
use urllib3 which can in cloud-init url fetching. For
now also take the metadata/userdata extraction code
from boto (small amount) and fix it in a new local file
called 'ec2_utils'.
|
