| Age | Commit message (Collapse) | Author |
|---|
|
Fix obvious typos. Replace 'for for' with a 'for'.
|
|
Network configuration in OpenNebula would only work if the host correctly
guessed the names of the devices in the guest. OpenNebula provided data
in its context.sh like 'ETH0_NETWORK', but if the guest named devices
differently then results were not predictable. This would occur with
Predictable Network Interface Names. To address this,
newer versions (of OpenNebula provide the mac address ETH0_MAC.
This function is present in 4.14 and documented officially in 5.0 docs.
This provides support for reading the mac addresses from the context.sh.
It also fixes cases where context.sh provided a field (ETH0_NETWORK
or ETH0_MASK) with a empty string. Previously the empty string would
be used rather than falling back to the default.
LP: #1719157, #1716397, #1736750
|
|
The behavior changes and improvements include:
- Only import keys into the default user that contain the name of the
default user ('ubuntu', or 'centos') or that contain 'cloudinit'.
- Use instance or project level keys based on GCE convention.
- Respect expiration time when keys are set.
Do not import expired keys.
- Support ssh-keys in project level metadata (the GCE default).
As part of this change, we also update the request header when talking
to the metadata server based on the documentation:
https://cloud.google.com/compute/docs/storing-retrieving-metadata#querying
LP: #1670456, #1707033, #1707037, #1707039
|
|
New mkfs.vfat and fatlabel tools included in the dosfsutils package no
longer support creating vfat disks with lowercase labels. They silently
default to an all uppercase label eg CONFIG-2 instead of config-2. This
change makes cloud-init handle either upper or lower case.
LP: #1598783
|
|
This stores a hash of the OAuth tokens as an 'id' for the maas
datasource. Since new instances get new tokens created and those tokens
are written by curtin into datasource system config this will provide
a way to identify a new "instance" (install).
LP: #1712680
|
|
This fixes a traceback when attempting to bounce the network after
hostname resets.
In artful and bionic ifupdown package is no longer installed in default
cloud images. As such, Azure can't use those tools to bounce the network
informing DDNS about hostname changes. This doesn't affect DDNS updates
though because systemd-networkd is now watching hostname deltas and with
default behavior to SendHostname=True over dhcp for all hostname updates
which publishes DDNS for us.
LP: #1722668
|
|
This branch resolves lints seen by pylint revision 1.8.1 and updates our
pinned tox pylint dependency used by our tox pylint target.
|
|
The instance identity document is a better source for region information,
partly because region isn't actually in meta-data at all, only
availability-zone, which happens to be named similarly.
Reviewed-by: Ethan Faust <efaust@amazon.com>
Reviewed-by: Cyle Riggs <cyler@amazon.com>
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
[ajorgens@amazon.com: rebase onto 0.7.9]
[ajorgens@amazon.com: changes per merge proposal discussions]
|
|
In the VMware customization workflow, we have some options for the user
to upload scripts for additional customization. Based on user request,
those custom scripts can be either run before regular customization or
after. For post customization scripts, we decide whether to run the scripts
just after customization or post system reboot.
|
|
Each DataSource subclass must define its own get_data method. This branch
formalizes our DataSource class to require that subclasses define an
explicit dsname for sourcing cloud-config datasource configuration.
Subclasses must also override the _get_data method or a
NotImplementedError is raised.
The branch also writes /run/cloud-init/instance-data.json. This file
contains all meta-data, user-data and vendor-data and a standardized set
of metadata keys in a json blob which other utilities with root-access
could make use of. Because some meta-data or user-data is potentially
sensitive the file is only readable by root.
Generally most metadata content types should be json serializable. If
specific keys or values are not serializable, those specific values will
be base64encoded and the key path will be listed under the top-level key
'base64-encoded-keys' in instance-data.json. If json writing fails due to
other TypeErrors or UnicodeDecodeErrors, a warning log will be emitted to
/var/log/cloud-init.log and no instance-data.json will be created.
|
|
There is a race condition where our sandboxed dhclient properly writes a
lease file but has not yet written a pid file. If the sandbox temporary
directory is torn down before the dhclient subprocess writes a pidfile
DataSourceEc2Local gets a traceback and the instance will fallback to
DataSourceEc2 in the init-network stage. This wastes boot cycles we'd
rather not spend.
Fix handling of sandboxed dhclient to wait for both pidfile and leasefile
before proceding. If either file doesn't show in 5 seconds, log a warning
and return empty lease results {}.
LP: #1735331
|
|
If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.
This branch does two things:
- Add a fallback_interface property to DatasourceEC2 to support reading the
old .fallback_nic attribute if it was set. New versions will
call net.find_fallback_nic() if there has not been one found.
- Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
metadata
LP: #1732917
|
|
Azure kernel now configures the SRIOV devices itself so cloud-init
does not need to provide any SRIOV device configuration or udev
naming rules.
LP: #1721579
|
|
VPC instances have the option to specific local only IPv4 addresses. Allow
Ec2Datasource to enable dhcp4 on instances even if local-ipv4s is
configured on an instance.
Also limit network_configuration to only the primary (fallback) nic.
LP: #1728152
|
|
Systems that used systemd-networkd's dhcp client would not be able to get
information on the Azure endpoint (placed in Option 245) or the CloudStack
server (in 'server_address').
The change here supports reading these files in /run/systemd/netif/leases.
The files declare that "This is private data. Do not parse.", but at this
point we do not have another option.
LP: #1718029
|
|
Previously we had hard coded paths in /sbin for the udevadm and modprobe
programs invoked by AltCloud. Its more flexible to expect the PATH to
be set correctly.
Debian: #852564
|
|
DataSourceOVF attempts to find iso files via walking os.listdir('/dev/')
which is far too wide. This approach is too invasive and can sometimes
race with systemd attempting to fsck and mount devices.
Instead, utilize cloudinit.util.find_devs_with to filter devices by
criteria (which uses blkid under the covers). This results in fewer
attempts to mount block devices which do not contain iso filesystems.
Unittest changes include:
- cloudinit.tests.helpers; introduce add_patch() helper
- Add unittest coverage for DataSourceOVF use of transport_iso9660
LP: #1718287
|
|
Currently the Azure data source waits up to 60 seconds. This has proven
not to be sufficient to provide resiliency to unrelated transient failures
in other parts of the infrastructure. Azure already has logic outside of
the VM to abort hung provisioning. This changes lengthens the time out to
15 minutes.
LP: #1717611
|
|
This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data that was read in
self.userdata_raw, rather than self.userdata. That is consistent
with other datasources and ulitimately how it was before the refactor.
The main is updated to address the fact that user-data is binary data
and may not be able to be printed.
LP: #1717598
|
|
A regression in 'get_latest_lease' made it ignore files starting with
'dhclient-' rather than just 'dhclient.'. The fix here is to allow those
files to be considered.
There is a lot more we could do here to better ensure that we pick the
most recent lease, but this change fixes the regression.
LP: #1717147
|
|
The network devices should be enabled before sending the 'SUCCESS' event
to the underlying hypervisor.
|
|
For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.
This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.
Modify the code to read the customization configuration and return the
converted data.
Added few tests.
LP: #1675063
|
|
This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:
a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)
b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.
The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.
Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.
LP: #1715128
|
|
During boot, the usage of /tmp is not safe. In systemd systems,
systemd-tmpfiles-clean may run at any point and clear out a temp file
while cloud-init is using it. The solution here is to use
/run/cloud-init/tmp.
LP: #1707222
|
|
Currently the cloud-init default locale (en_US.UTF-8) is set by
the base datasource class. This patch allows a distro to overide
the fallback value with one that's available in the distro but continues
to respect an image which has preconfigured a locale.
- Distro object now has a get_locale method which will return a
preconfigure locale setting by checking the distros locale system
configuration file. If not set or not present, return the default
locale of en_US.UTF-8 which retains behavior of all previous cloud-init
releases.
- Apply locale now handles regenerating locales or system configuration
files as needed.
- Adjust apply_locale logic to skip locale-regen if the specified LANG
value is C.UTF-8,C, or POSIX; they do not require regeneration.
- Further add unittests to exercise the default paths for Ubuntu and
non-ubuntu paths to validate they get the LANG expected.
|
|
This just adds a main to the GCE datasource so that it is easily
callable: python3 -m cloudinit.sources.DataSourceGCE
It also adds a log of the time it took to crawl.
|
|
DataSourceEc2 now parses the metadata for each nic to determine if
configured for ipv6 and/or ipv4 addresses. In AWS for metadata version
2016-09-02, nics configured for ipv4 or ipv6 addresses will have non-zero
values stored in metadata at network/interfaces/macs/<MAC>/public-ipv4 or
ipv6s respectively. Those metadata files are only non-zero when an ipv4 or
ipv6 ip is associated to the specific nic. A new
DataSourceEc2.network_config property is added which parses the metadata
and renders a network version 1 dictionary representing both dhcp4 and
dhcp6 configuration for associated nics.
The network configuration returned from the datasource will also 'pin' the
nic name to the name presented on the instance for each nic.
LP: #1639030
|
|
This feature enables the following VMware VCloud Director functionality:
1. Setting admin password
2. Expire password.
3. Set admin password and expire.
Password configuration is triggered only as part of a full
recustomization, that happens either on first power on or when
"poweron and full recustomization" is selected. Full customization
flow is determined by marker files. Unique marker ids are
generated when full recustomization is requested. And marker file based
on these marker ids help to determine if we need to execute the above
configuration.
|
|
This branch is a prerequisite for IPv6 support in AWS by allowing Ec2
datasource to query the metadata source version 2016-09-02 about whether
or not it needs to configure IPv6 on interfaces. If version 2016-09-02
is not present, fallback to the min_metadata_version of 2009-04-04. The
DataSourceEc2Local not run on FreeBSD because dhclient in doesn't
support the -sf flag allowing us to run dhclient without filesystem
side-effects.
To query AWS' metadata address @ 169.254.169.254, the instance must have
a dhcp-allocated address configured. Configuring IPv4 link-local
addresses result in timeouts from the metadata service. We introduced a
DataSourceEc2Local subclass which will perform a sandboxed dhclient
discovery which obtains an authorized IP address on eth0 and crawl
metadata about full instance network configuration.
Since ec2 IPv6 metadata is not sufficient in itself to tell us all the
ipv6 knownledge we need, it only be used as a boolean to tell us which
nics need IPv6. Cloud-init will then configure desired interfaces to
DHCPv6 versus DHCPv4.
Performance side note: Shifting the dhcp work into init-local for Ec2
actually gets us 1 second faster deployments by skipping init-network
phase of alternate datasource checks because Ec2Local is configured in
an ealier boot stage. In 3 test runs prior to this change: cloud-init
runs were 5.5 seconds, with the change we now average 4.6 seconds.
This efficiency could be even further improved if we avoiding dhcp
discovery in order to talk to the metadata service from an AWS
authorized dhcp address if there were some way to advertize the dhcp
configuration via DMI/SMBIOS or system environment variables.
Inspecting time costs of the dhclient setup/teardown in 3 live runs the
time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
dhcp discovery + metadata: 0.347 seconds
metadata alone: 0.271 seconds
test 2: 88 milliseconds
dhcp discovery + metadata: 0.388 seconds
metadata alone: 0.300 seconds
test 3: 75 milliseconds
dhcp discovery + metadata: 0.366 seconds
metadata alone: 0.291 seconds
LP: #1709772
|
|
EC2 was the original, but this adds some initial tests for that datasource.
Also updates a docstring for an internal method.
|
|
Here we add and enable by default a datasource for Scaleway cloud.
The datasource quickly exits unless one of three things:
a.) 'Scaleway' found as the system vendor
b.) 'scaleway' found on the kernel command line.
c.) the directory /var/run/scaleway exists (this is currently created
by the scaleway initramfs module).
One interesting bit of this particular datasource is that it requires
the source port of the http request to be < 1024.
|
|
On systems with network devices with duplicate mac addresses, cloud-init
will fail to rename the devices according to the specified network
configuration. Refactor net layer to search by device driver and device
id if available. Azure systems may have duplicate mac addresses by
design.
Update Azure datasource to run at init-local time and let Azure datasource
generate a fallback networking config to handle advanced networking
configurations.
Lastly, add a 'setup' method to the datasources that is called before
userdata/vendordata is processed but after networking is up. That is
used here on Azure to interact with the 'fabric'.
|
|
- Simplify the logic of 'variant' in util.system_info
much of the data from
https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
- fix get_resource_disk_on_freebsd when running on a system without
an Azure resource disk.
- fix tools/build-on-freebsd to replace oauth with oauthlib and add
bash which is a dependency for tests.
- update a fiew places that were checking for freebsd but not using
the util.is_FreeBSD()
|
|
The previous commit caused test failure.
This separates out _check_freebsd_cdrom and mocks it in a test
rather than patching open.
|
|
Fix the issue caused by different commands on Linux and FreeBSD. On Linux,
we used ifdown and ifup to enable and disable a NIC, but on FreeBSD, the
counterpart is "ifconfig down" and "ifconfig up".
LP: #1697815
|
|
The current method is to attempt to mount the cdrom (/dev/cd0), if it is
successful, /dev/cd0 is configured, otherwise, it is not configured. The
problem is it forgets to check whether the mounting destination folder is
created or not. As a result, mounting attempt failed even if cdrom is
ready.
LP: #1696295
|
|
This allows the user to seed NoCloud in a trivial way from qemu/libvirt,
by using a stock image and passing a single command line flag. No custom
command line, no filesystem modification, no bootstrap disk image.
This is particularly handy now that Ec2 backend is discouraged from use
under bug 1660385.
LP: #1691772
|
|
In previous commit I inadvertantly left two calls to
asset_tag = util.read_dmi_data('chassis-asset-tag')
The second did not do anything useful. Thus, remove it.
|
|
Azure sets a known chassis asset tag to 7783-7084-3265-9085-8269-3286-77.
We can inspect this in both ds-identify and DataSource.get_data to
determine whether we are on Azure.
Added unit tests to cover these changes
and some minor tweaks to Exception error message content to give more
context on malformed or missing ovf-env.xml files.
LP: #1693939
|
|
AliYun cloud platform is now identifying themselves by setting the dmi
product id to the well known value "Alibaba Cloud ECS". The changes here
identify that properly in tools/ds-identify and in the DataSourceAliYun.
Since the 'get_data' for AliYun now identifies itself correctly, we can
enable AliYun by default.
LP: #1638931
|
|
Per the documentation at
https://cloud.google.com/compute/docs/storing-retrieving-metadata
The instance-level SSH key was named 'sshKeys' and now is 'ssh-keys'.
The project-level SSH key attribute has not changed so is intentionally
not changed here.
LP: #1693582
|
|
This just moves flake8 and related tools up to newer versions and fixes
the complaints associated with that.
We added to the list of flake8 ignores:
H102: do not put vim info in source files
H304: no relative imports
Also updates and pins the following in the flake8 environment:
pep8: 1.7.0 => drop (although hacking still pulls it in).
pyflakes 1.1.0 => 1.5.0
hacking 0.10.2 => 0.13.0
flake8 2.5.4 => 3.3.0
pycodestyle none => 2.3.1
|
|
Previously, the datasource for DigitalOcean allowed for a gateway on each
NIC. As a result, on Ubuntu 16.04, networking.service was broken. For
17.04 and later, Ubuntu _replaces_ the default gateway with the second
gateway on 'ifup' after reboot.
DigitalOcean is looking at changing the meta-data, however, this will
result in another version of the meta-data JSON.
LP: #1681531.
|
|
Large instance types have a different disk format on the newly
partitioned ephemeral drive. So we have to adjust the logic in the
Azure datasource to recognize that a disk with 2 partitions and
an empty ntfs filesystem on the second one is acceptable.
This also adjusts the datasources's builtin fs_setup config to remove
the 'replace_fs' entry. This entry was previously ignored, and confusing.
I've clarified the doc on that also.
LP: #1686514
|
|
There was a copy/paste error in _get_url_settings such that the error
message would complain about max wait when in fact it was talking
about retries.
|
|
This patch targets to make FreeBSD 10.3 or 11 work on Azure. The
modifications abide by the rule of:
* making as less modification as possible
* delegate to the distro or datasource where possible.
The main modifications are:
1. network configuration improvements, and movement into distro path.
2. Fix setting of password.
Password setting through "pw" can only work through pipe.
3. Add 'root:wheel' to syslog_fix_perms field.
4. Support resizing default file system (ufs)
5. copy cloud.cfg for freebsd to /etc/cloud/cloud.cfg rather than
/usr/local/etc/cloud/cloud.cfg.
6. Azure specific changes:
a. When reading the azure endpoint, search in a different path
and read a different option name (option-245 vs. unknown-245).
so, the lease file path should be generated according to platform.
b. adjust the handling of ephemeral mounts for ufs filesystem and
for finding the ephemeral device.
c. fix mounting of cdrom
LP: #1636345
|
|
This will change all instances of LOG.warn to LOG.warning as warn
is now a deprecated method. It will also make sure any logging
uses lazy logging by passing string format arguments as function
parameters.
|
|
To query the metadata, the Cloudstack source currently scans a predefined
DHCP lease directories to find the IP of the DHCP server. This list does
not include "/var/lib/NetworkManager/" which is the default directory in
CentOS7. Add that directory to the list.
|
|
Previously the IPv4LL address for metadata discovery was assigned to the
first interfaces from an alphabetic sort. On DigitalOcean, the metadata
is only accessible from the first interface. This fixes a problem where the
IPv4LL address is bound to the wrong interface with snapshots.
This is part of general improvements to the DigitalOcean Datasource in
bug 1676908.
|
|
Instead of only configuring 'public' and 'private' interfaces, we want
to configure any that has been defined in the meta-data. For legacy reasons,
the 'public' and 'private' interfaces are maintained as 'eth0' and 'eth1'
respectively.
This is part of bug 1676908 for general DigitalOcean datasource fixups.
|
