| Age | Commit message (Collapse) | Author |
|---|
|
If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.
This branch does two things:
- Add a fallback_interface property to DatasourceEC2 to support reading the
old .fallback_nic attribute if it was set. New versions will
call net.find_fallback_nic() if there has not been one found.
- Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
metadata
LP: #1732917
|
|
VPC instances have the option to specific local only IPv4 addresses. Allow
Ec2Datasource to enable dhcp4 on instances even if local-ipv4s is
configured on an instance.
Also limit network_configuration to only the primary (fallback) nic.
LP: #1728152
|
|
Systems that used systemd-networkd's dhcp client would not be able to get
information on the Azure endpoint (placed in Option 245) or the CloudStack
server (in 'server_address').
The change here supports reading these files in /run/systemd/netif/leases.
The files declare that "This is private data. Do not parse.", but at this
point we do not have another option.
LP: #1718029
|
|
Previously we had hard coded paths in /sbin for the udevadm and modprobe
programs invoked by AltCloud. Its more flexible to expect the PATH to
be set correctly.
Debian: #852564
|
|
DataSourceOVF attempts to find iso files via walking os.listdir('/dev/')
which is far too wide. This approach is too invasive and can sometimes
race with systemd attempting to fsck and mount devices.
Instead, utilize cloudinit.util.find_devs_with to filter devices by
criteria (which uses blkid under the covers). This results in fewer
attempts to mount block devices which do not contain iso filesystems.
Unittest changes include:
- cloudinit.tests.helpers; introduce add_patch() helper
- Add unittest coverage for DataSourceOVF use of transport_iso9660
LP: #1718287
|
|
This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data that was read in
self.userdata_raw, rather than self.userdata. That is consistent
with other datasources and ulitimately how it was before the refactor.
The main is updated to address the fact that user-data is binary data
and may not be able to be printed.
LP: #1717598
|
|
A regression in 'get_latest_lease' made it ignore files starting with
'dhclient-' rather than just 'dhclient.'. The fix here is to allow those
files to be considered.
There is a lot more we could do here to better ensure that we pick the
most recent lease, but this change fixes the regression.
LP: #1717147
|
|
This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:
a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)
b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.
The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.
Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.
LP: #1715128
|
|
During boot, the usage of /tmp is not safe. In systemd systems,
systemd-tmpfiles-clean may run at any point and clear out a temp file
while cloud-init is using it. The solution here is to use
/run/cloud-init/tmp.
LP: #1707222
|
|
This missed mock in test_openstack resulted in a costly unit test timeout.
LP: #1714376
|
|
This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.
|
|
DataSourceEc2 behavior changed to first check a minimum acceptable
metadata version uri http://169.154.169.254/<min_version>/instance-id,
retrying on 404, until the metadata service is available. After the
metadata service is up, the datasource inspects preferred
extended_metadata_versions for availability. Unit tests only mocked the
preferred extended_metadata_version so all Ec2 tests were retrying
attempts against
http://169.254.169.254/meta-data/<min-version>/instance-id adding a lot of
time cost to the unit test runs.
This branch uses httpretty to properly mock the following:
- 404s from metadata on undesired extended_metadata_version test routes
- https://169.254.169.254/meta-data/2016-09-02/instance-id
- full metadata dictionary represented on min_metadata_version
- https://169.254.169.254/meta-data/2016-09-02/*
The branch also tightens httpretty to raise a MockError for any URL which
isn't mocked via httpretty.HTTPretty.allow_net_connect=False.
LP: #1714117
|
|
DataSourceEc2 now parses the metadata for each nic to determine if
configured for ipv6 and/or ipv4 addresses. In AWS for metadata version
2016-09-02, nics configured for ipv4 or ipv6 addresses will have non-zero
values stored in metadata at network/interfaces/macs/<MAC>/public-ipv4 or
ipv6s respectively. Those metadata files are only non-zero when an ipv4 or
ipv6 ip is associated to the specific nic. A new
DataSourceEc2.network_config property is added which parses the metadata
and renders a network version 1 dictionary representing both dhcp4 and
dhcp6 configuration for associated nics.
The network configuration returned from the datasource will also 'pin' the
nic name to the name presented on the instance for each nic.
LP: #1639030
|
|
This branch is a prerequisite for IPv6 support in AWS by allowing Ec2
datasource to query the metadata source version 2016-09-02 about whether
or not it needs to configure IPv6 on interfaces. If version 2016-09-02
is not present, fallback to the min_metadata_version of 2009-04-04. The
DataSourceEc2Local not run on FreeBSD because dhclient in doesn't
support the -sf flag allowing us to run dhclient without filesystem
side-effects.
To query AWS' metadata address @ 169.254.169.254, the instance must have
a dhcp-allocated address configured. Configuring IPv4 link-local
addresses result in timeouts from the metadata service. We introduced a
DataSourceEc2Local subclass which will perform a sandboxed dhclient
discovery which obtains an authorized IP address on eth0 and crawl
metadata about full instance network configuration.
Since ec2 IPv6 metadata is not sufficient in itself to tell us all the
ipv6 knownledge we need, it only be used as a boolean to tell us which
nics need IPv6. Cloud-init will then configure desired interfaces to
DHCPv6 versus DHCPv4.
Performance side note: Shifting the dhcp work into init-local for Ec2
actually gets us 1 second faster deployments by skipping init-network
phase of alternate datasource checks because Ec2Local is configured in
an ealier boot stage. In 3 test runs prior to this change: cloud-init
runs were 5.5 seconds, with the change we now average 4.6 seconds.
This efficiency could be even further improved if we avoiding dhcp
discovery in order to talk to the metadata service from an AWS
authorized dhcp address if there were some way to advertize the dhcp
configuration via DMI/SMBIOS or system environment variables.
Inspecting time costs of the dhclient setup/teardown in 3 live runs the
time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
dhcp discovery + metadata: 0.347 seconds
metadata alone: 0.271 seconds
test 2: 88 milliseconds
dhcp discovery + metadata: 0.388 seconds
metadata alone: 0.300 seconds
test 3: 75 milliseconds
dhcp discovery + metadata: 0.366 seconds
metadata alone: 0.291 seconds
LP: #1709772
|
|
EC2 was the original, but this adds some initial tests for that datasource.
Also updates a docstring for an internal method.
|
|
Here we add and enable by default a datasource for Scaleway cloud.
The datasource quickly exits unless one of three things:
a.) 'Scaleway' found as the system vendor
b.) 'scaleway' found on the kernel command line.
c.) the directory /var/run/scaleway exists (this is currently created
by the scaleway initramfs module).
One interesting bit of this particular datasource is that it requires
the source port of the http request to be < 1024.
|
|
The usage of mock in this test was simply invalid and only worked by
happenstance.
|
|
The mock of platform_reports_gce is created with a True return value in
tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.setUp().
But, the final test_get_data_returns_false_if_not_on_gce incorrectly
attempts to override the mocked return_value of True to False by setting
self.m_platform_gce.return_value = False. But, since the mock is already
initialized, the updated False is not honored. Instead we should use the
patch decorator on the specific unit test to override the return_value of
DataSourceGCE.platform_reports_gce to False.
A False from platform_reports_gce allows DataSourceGCE.get_data to
immediately return False instead of trying to contact
metadata.google.internal as the related bug references.
|
|
On systems with network devices with duplicate mac addresses, cloud-init
will fail to rename the devices according to the specified network
configuration. Refactor net layer to search by device driver and device
id if available. Azure systems may have duplicate mac addresses by
design.
Update Azure datasource to run at init-local time and let Azure datasource
generate a fallback networking config to handle advanced networking
configurations.
Lastly, add a 'setup' method to the datasources that is called before
userdata/vendordata is processed but after networking is up. That is
used here on Azure to interact with the 'fabric'.
|
|
The previous commit caused test failure.
This separates out _check_freebsd_cdrom and mocks it in a test
rather than patching open.
|
|
The current method is to attempt to mount the cdrom (/dev/cd0), if it is
successful, /dev/cd0 is configured, otherwise, it is not configured. The
problem is it forgets to check whether the mounting destination folder is
created or not. As a result, mounting attempt failed even if cdrom is
ready.
LP: #1696295
|
|
Unix file modes are usually represented as octal, but they were being
interpreted as decimal, for example 0o644 would be printed as '420'.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
|
|
Azure sets a known chassis asset tag to 7783-7084-3265-9085-8269-3286-77.
We can inspect this in both ds-identify and DataSource.get_data to
determine whether we are on Azure.
Added unit tests to cover these changes
and some minor tweaks to Exception error message content to give more
context on malformed or missing ovf-env.xml files.
LP: #1693939
|
|
AliYun cloud platform is now identifying themselves by setting the dmi
product id to the well known value "Alibaba Cloud ECS". The changes here
identify that properly in tools/ds-identify and in the DataSourceAliYun.
Since the 'get_data' for AliYun now identifies itself correctly, we can
enable AliYun by default.
LP: #1638931
|
|
Per the documentation at
https://cloud.google.com/compute/docs/storing-retrieving-metadata
The instance-level SSH key was named 'sshKeys' and now is 'ssh-keys'.
The project-level SSH key attribute has not changed so is intentionally
not changed here.
LP: #1693582
|
|
python2.6 unittest.TestCase does not have the assertIsNone or
assertIsNotNone. We just have to explicitly use the unittest2
version, which we get from helpers.
The desire to use assertIsNone comes from flake8 (through hacking,
I believe).
Also, fix "{}.format('foo')" which is not valid in python2.6.
|
|
This just moves flake8 and related tools up to newer versions and fixes
the complaints associated with that.
We added to the list of flake8 ignores:
H102: do not put vim info in source files
H304: no relative imports
Also updates and pins the following in the flake8 environment:
pep8: 1.7.0 => drop (although hacking still pulls it in).
pyflakes 1.1.0 => 1.5.0
hacking 0.10.2 => 0.13.0
flake8 2.5.4 => 3.3.0
pycodestyle none => 2.3.1
|
|
Previously, the datasource for DigitalOcean allowed for a gateway on each
NIC. As a result, on Ubuntu 16.04, networking.service was broken. For
17.04 and later, Ubuntu _replaces_ the default gateway with the second
gateway on 'ifup' after reboot.
DigitalOcean is looking at changing the meta-data, however, this will
result in another version of the meta-data JSON.
LP: #1681531.
|
|
on centos/fedora/rhel, /var/lib/NetworkManager has mode 700, causing
the cloudstack unit tests to fail when run as a non-root user. This
mocks out get_latest_lease so that we no longer try to read dhcp lease
information during the unit tests.
|
|
Large instance types have a different disk format on the newly
partitioned ephemeral drive. So we have to adjust the logic in the
Azure datasource to recognize that a disk with 2 partitions and
an empty ntfs filesystem on the second one is acceptable.
This also adjusts the datasources's builtin fs_setup config to remove
the 'replace_fs' entry. This entry was previously ignored, and confusing.
I've clarified the doc on that also.
LP: #1686514
|
|
This patch targets to make FreeBSD 10.3 or 11 work on Azure. The
modifications abide by the rule of:
* making as less modification as possible
* delegate to the distro or datasource where possible.
The main modifications are:
1. network configuration improvements, and movement into distro path.
2. Fix setting of password.
Password setting through "pw" can only work through pipe.
3. Add 'root:wheel' to syslog_fix_perms field.
4. Support resizing default file system (ufs)
5. copy cloud.cfg for freebsd to /etc/cloud/cloud.cfg rather than
/usr/local/etc/cloud/cloud.cfg.
6. Azure specific changes:
a. When reading the azure endpoint, search in a different path
and read a different option name (option-245 vs. unknown-245).
so, the lease file path should be generated according to platform.
b. adjust the handling of ephemeral mounts for ufs filesystem and
for finding the ephemeral device.
c. fix mounting of cdrom
LP: #1636345
|
|
This change makes the DigitalOcean datasource consistent with OpenStack and
Joyent by binding the resolver addresses to the loopback interface. This _is_
a work-around to bug 1675571.
Part of bug 1676908.
|
|
This just mocks out the AltCloud tests to not invoke blkid.
Our tests should not rely on system command returning any specific
value.
Also, shorten long lines with change in the import name of
DataSourceAltCloud.
LP: #1636531
|
|
Mock the use use of get_interfaces_by_mac in Digital Ocean and OpenNebula.
Its best to mock this for the tests as the results aren't expecting
it to fail.
Note, as it stands, OpenNebula relies on devices named 'eth0'.
The metadata (context) does not provide mac addresses.
|
|
While documentation indicates that the smbios product name should
contain 'Google Compute Engine', experimentation and bug reports
indicate that is not always the case. The change here is to change
the check for GCE to also consider a serial number that starts with
'GoogleCompute-'.
Also, ds-identify was not currently searching for GCE if no config of
datasource_list was found. Most images have a datasource_list defined.
So update the list to include GCE.
LP: #1674861
|
|
render_network_state should default to rendering on /.
The changes here just make it so render_network_state does not
require a target, but defaults to None, and uses target_path
to handle that.
|
|
No shipping ubuntu has a python 3.4 that is less than 3.4.2.
Remove this workaround to avoid unnecessary complexity.
This reverts 04a60cf949.
|
|
this ensures that we call httpretty.reset() before calling
httppretty.register_uri(...), which ensures that we get a fresh
callback with the expected version of the metadata.
LP: #1658200
|
|
This modifies get_data in DataSourceOpenStack.py to get the timeout
and retries values from the data source configuration, rather than
from keyword arguments. This permits get_data to use the same timeout
as other methods, and allows an operator to increase the timeout in
environments where the metadata service takes longer than five seconds
to respond.
LP: #1657130
Resolves: rhbz#1408589
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
CloudSigma would not get any datasources loaded during cloud-init local.
Thus, when the network datasource was removed, *no* CloudSigma
datasources would be loaded.
LP: #1648380
|
|
Cloud-init has for some time relied on walinuxagent to do some bits
of work necessary for instance initialization. That reliance has
not been needed for a while, but we have still defaulted to it.
This change uses the "builtin" path that Daniel Watkins added
some time ago by default. Also, Adjust tests that assumed the
non-__builtin__ Azure agent_command.
LP: #1538522
|
|
This adds a call to 'activate_datasource'. That will be called
during init stage (or init-local in the event of a 'local' dsmode).
It is present so that the datasource can do platform specific operations
that may be necessary. It is passed the fully rendered cloud-config
and whether or not the instance is a new instance.
The Azure datasource uses this to address formatting of the ephemeral
devices. It does so by
a.) waiting for the device to come online
b.) removing the marker files for the disk_setup and mounts modules
if it finds that the ephemeral device has been reset.
LP: #1611074
|
|
An obvious fix for an issue raised by pyflakes 1.3.
|
|
Support AliYun(Ali-Cloud ECS). This datasource inherits from EC2,
the main difference is the meta-server address is changed to
100.100.100.200.
The datasource behaves similarly to EC2 and relies on network polling.
As such, it is not enabled by default.
|
|
Replace the parsing of 'ip' to get a link and mac address list
in OpenNebula's datasource with usage of cloudinit.net.
This makes test cases there not depend on 'ip' availability
and also uses common code.
|
|
Some of the new DigitalOcean unittests were written to use 'assert_called',
which is only available in mock versions 2.0. Because of this, the failure
would only occur in releases less than yakkety and not in 'tox'.
Add a 'xenial' entry to tox.ini with versions from xenial.
|
|
On DigitalOcean, Network information is provided via Meta-data.
It changes the datasource to be a local datasource, meaning it
will run before fallback networking is configured.
The advantage of that is that before networking is configured it
can bring up a network device with ipv4 link-local and hit the
metadata service that lives at 169.254.169.254 to find its networking
configuration. It then takes down the link local address and lets
cloud-init configure networking.
The configuring of a network device to go looking for a metadata
service is gated by a check of data in the smbios. This guarantees
that the code will not run on another system.
|
|
Treat null type as yet another physical type, seen in real-world
openstack cloud.
Also, support the case where network_data.json provides mac addresses
in upper case. Rackspace public cloud currently does that.
LP: #1621968
|
|
When user-data was provided in the ovf environment python3 would call
base64.decodestring() with a string rather than bytes and an exception
would occur.
This fixes the broken path and adds unit test. Also changes to
return None rather than empty string when there is no user-data and
when there is user-data return that as bytes instead of string.
LP: #1619394
|
