| Age | Commit message (Collapse) | Author |
|---|
|
The result of a read_file_or_url on a file and on a url would differ
in behavior.
str(UrlResponse) would return UrlResponse.contents.decode('utf-8')
while
str(FileResponse) would return str(FileResponse.contents)
The difference being "b'foo'" versus "foo".
As part of the general goal of cleaning util, move read_file_or_url
into url_helper.
|
|
This enables warnings produced by pylint for unused variables (W0612),
and fixes the existing errors.
|
|
Reducing timeout to 1 second as IMDS responds within a handful
of milliseconds. Also get rid of max_retries to prevent exiting
out of polling loop early due to IMDS outage / upgrade.
Reduce Azure PreProvisioning HTTP timeouts during polling to
avoid waiting an extra minute.
LP: #1752977
|
|
In commit e9e8616, there was an inversion of the logic of the
exception_cb return value meaning, breaking the (network) OpenStack
DataSource, which implemented exception_cb as should_retry_cb, returning
True when a retry should be done and False when the retry loop should
be broken and the exception reraised again immediately.
The OpenStack DS was the only user of this callback at the time and not
touched by the commit (nor did the commit message mention an intended
change), so this almost certainly happened by mistake.
These days, we have a second user of the callback in DataSourceScaleway.
It uses the new logic, so it needs change if we fix the meaning of the
return value.
This patch reverts the meaning of url_helper.read_url() execption_cb
to the old semantics. It updates the comment and adjusts the Scaleway
datasource.
The patch has been tested on Open Telekom Cloud (which uses the
OpenStack network Datasource) where previously a missing user_data
and network_data.json would be retried 6 times each despite them
not being present (they are optional!) and the server repsonding
with a correct 404. After the patch, boot times are 10s faster,
as we no longer pointlessly retry these files.
LP: #1702160
LP: #1298921
|
|
Just replace a couple things like:
if b > a and b < c:
with:
if a < b < c:
|
|
This change will enable azure vms to report provisioning has completed
twice, first to tell the fabric it has completed then a second time to
enable customer settings. The datasource for the second provisioning is
the Instance Metadata Service (IMDS),and the VM will poll indefinitely for
the new ovf-env.xml from IMDS.
This branch introduces EphemeralDHCPv4 which encapsulates common logic
used by both DataSourceEc2 an DataSourceAzure for temporary DHCP
interactions without side-effects.
LP: #1734991
|
|
oauth_headers is the only function which requires oauthlib, move the
import and ImportError handling inside this function to only attempt
loading at runtime if called. This will allow us to build on platforms
that don't have python-oauthlib installed by default. Add simple unittests
around the missing oauthlib dependencies to make sure the function
performs as intended and raises and NotImplementedError if oauthlib can't
be imported.
|
|
We are unable to ship python-oauthlib in RHEL. This commit allows
imports of url_helper to succeed even when oauthlib is unavailable
and OauthUrlHelper.oauth_headers to raise a NotImplementedException
when called.
LP: #1713760
|
|
Here we add and enable by default a datasource for Scaleway cloud.
The datasource quickly exits unless one of three things:
a.) 'Scaleway' found as the system vendor
b.) 'scaleway' found on the kernel command line.
c.) the directory /var/run/scaleway exists (this is currently created
by the scaleway initramfs module).
One interesting bit of this particular datasource is that it requires
the source port of the http request to be < 1024.
|
|
This will change all instances of LOG.warn to LOG.warning as warn
is now a deprecated method. It will also make sure any logging
uses lazy logging by passing string format arguments as function
parameters.
|
|
Now tox will run pylint. The .pylintrc file sets pylint to only produce
errors, and will ignore certain classes that are known problematic (six).
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
|
|
|
|
This would cause problems in the event that we actually had a bad
clock. We add a retry in the main (for test) also, to ensure that
the oauth timestamp fix gets in place.
LP: #1499869
|
|
we want cloud-init user agent to be present even if the user
provided some headers. In the event that they provided User-Agent,
this will respect their wishes.
|
|
the previous version was broken. The vital fixes here are:
* adding parsedate and oauth1 imports to url_helper
* fix skew_data usage intending to use self.skew_data
Additionally:
* reorder imports in url_helper
* fixes to python3 -m cloudinit.sources.DataSourceMaas
LP: #1488507
|
|
|
|
|
|
|
|
|
|
python2 scoping is different and running wait_for_url in python3
results in a use of undeclared variable 'e'.
$ python3 -c 'from cloudinit import url_helper; \
url_helper.wait_for_url("o", max_wait=3,timeout=1, exception_cb=print)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "cloudinit/url_helper.py", line 358, in wait_for_url
exception_cb(msg=status_msg, exception=e)
|
|
UrlResponse: biggest change... make readurl return bytes, making user
know what to do with it.
util: add load_tfile_or_url for loading text file or url
as read_file_or_url now returns bytes
ec2_utils: all meta-data is text, remove non-obvious string translations
DigitalOcean: adjust for ec2_utils
DataSourceGCE, DataSourceMAAS: user-data is binary other fields are text.
openstack.py: read paths without decoding to text. This is ok as paths
other than user-data are json, and load_json will handle
load_file still returns text, and that is what most things use.
|
|
to be behind trunk.
`tox -e py27` passes full test suite. Now to work on replacing mocker.
|
|
This just removes comments '# pylint:' things and other code
remnents of pylint.
|
|
Fixed all complaints from running "make pep8". Also version locked
pep8 in test-requirements.txt to ensure that pep8 requirements don't
change without an explicit commit.
|
|
Waiting around for a metadata service in a given datasource means that if its
not there all the subsequent datasources have to wait, and boot is slowed down.
As it is right now, EC2 is the only one that has the right to wait. In the
past, we had to wait around for the EC2 metadata service. I really do not want
to extend that courtesy to other cloud platforms. A network based metadata
service should be up as soon as networking is up.
|
|
|
|
|
|
|
|
|
|
Openstack has a unique derivative datasource
that is gaining usage. Previously the config
drive datasource provided part of this functionality
as well as the ec2 datasource, but since new
functionality is being added to openstack is
seems benefical to combine the used parts into
one datasource just made for handling openstack
deployments.
This patch factors out the common logic shared
between the config drive and the openstack
metadata datasource and places that in a shared
helper file and then creates a new openstack
datasource that readers from the openstack metadata
service and refactors the config drive datasource
to use this common logic.
|
|
This makes it easier to call read_file_or_url and handle file or url
errors. Now read_file_or_url will raise a UrlError in either case
on errors.
|
|
|
|
When a 404 http code comes back from the fetching of ec2
data, instead of retrying immediatly stop the fetching process
and in the userdata fetching function handle this case as a
special case of no userdata being fetched (an empty string
in this case).
|
|
|
|
Couple of things here:
* do not re-try on user-data (404 means 'not here')
* re-generate headers on retry requests
LP: #1172742
|
|
the previous implementation of url_helper.readurl() would default
to allow_redirects being true.
So, for backwards compat, we should keep that behavior.
|
|
if the error is an ssl error, its extremely unlikely that it would be fixed by
waiting a few seconds and trying again.
|
|
* cloudinit/distros/parsers/resolv_conf.py
added some pylint overrides with 'plXXXXX' syntax.
example: # pl51222 pylint: disable=E0102
The pl51222 there means: http://www.logilab.org/ticket/51222
This specific issue is present in 12.04 pylint, but not 13.04.
* pylint doesn't like the requests special handling we have.
which makes sense as it is only checking versus one specific version.
* general pep8 and pylint cleanups.
|
|
in earlier versions of requests (pre 0.10.8).
|
|
|
|
|
|
|
|
|
|
1. Handle our own retries (so that we can sleep in between)
2. Cleanup the url so that partially invalid (no scheme) urls
will continue working.
3. Allow redirects option now a param.
|
|
|
|
since it provides the same support (and it uses
that library). Urllib3 doesn't seem to exist as
its own package (requests hides it) so solve that
by using requests and handling its oddness ourself.
|
|
userdata and metadata. Add in this crawling functionality
to the ec2_utils module that will fully crawl (not lazily)
the ec2 metadata and parse it in the same manner as boto.
1. Make the ec2 datasource + cloudstack now call into these.
2. Fix phone_home due to urllib3 change (TBD)
|
|
use ssl correctly (they can not do validation due to
the underlying usage of urllib/httplib). Adjust to
use urllib3 which can in cloud-init url fetching. For
now also take the metadata/userdata extraction code
from boto (small amount) and fix it in a new local file
called 'ec2_utils'.
|
