| Age | Commit message (Collapse) | Author |
|---|
|
LP: #1959118
|
|
Add DataSourceLXD which knows how to talk to the dev-lxd socket to
obtain all instance metadata API:
https://linuxcontainers.org/lxd/docs/master/dev-lxd.
This first branch is to deliver feature parity with the existing
NoCloud datasource which is currently used to intialize LXC instances
on first boot.
Introduce a SocketConnectionPool and LXDSocketAdapter to support
performing HTTP GETs on the following routes which are surfaced by the
LXD host to all containers:
http://unix.socket/1.0/meta-data
http://unix.socket/1.0/config/user.user-data
http://unix.socket/1.0/config/user.network-config
http://unix.socket/1.0/config/user.vendor-data
These 4 routes minimally replace the static content provided in the
following nocloud-net seed files:
/var/lib/cloud/nocloud-net/{meta-data,vendor-data,user-data,network-config}
The intent of this commit is to set a foundation for LXD socket
communication that will allow us to build network hot-plug features
by eventually consuming LXD's websocket upgrade route 1.0/events to
react to network, meta-data and user-data config changes over time.
In the event that no custom network-config is provided, default to the
same network-config definition provided by LXD to the NoCloud
network-config seed file.
Supplemental features above NoCloud datasource:
surface all custom instance data config keys via cloud-init query ds
which aids in discoverability of features/tags/labels as well as
conditional #cloud-config jinja templates operations based on custom
config options.
TBD: better cloud-init query support for dot-delimited keys
|
|
Due to multiarch, the libdeployPkgPlugin.so is deployed into dir
/usr/lib/<multiarch name>/open-vm-tools, we need to add this path
into search_paths.
LP: #1944946
|
|
This patch updates the ds-identify file to pass the
popular shell linter, shellcheck. Updates include:
* Global ignore for error code 3043, the warning about
the "local" keyword not being POSIX compliant. This
keyword is used everywhere and cannot be easily removed.
* Point-of-use ignore for error code 3013, the warning
about the "-nt" comparator not being POSIX compliant.
There's no POSIX compliant way to compare timestamps
on two files without forking a new process, so this
comparator remains.
* Several point-of-use ignores for error code 2254, the
warning about quoting parameters. These warnings are
ignored for the *_matches functions where wildcard
tokens are expected to be used. Quoting the parameters
would prevent the desired globbing and cause calls
to these functions with wildcards to fail.
* A point-of-use ignore for error code 2086 for the
same reason as the above ignore -- allowing globbing
to take place for matching.
* Quoted parameters in the remainder of the locations
where it was safe to do so and did not break the
tests.
|
|
This patch reverts an unnecessary lcase optimization in the
ds-identify script. SystemD documents the values produced by
the systemd-detect-virt command are lower case, and the mapping
table used by the FreeBSD check is also lower-case.
The optimization added two new forked processes, needlessly
causing overhead.
|
|
This patch updates the dscheck_VMware function's use of "vmware-rpctool".
When checking to see if a "guestinfo" property is set.
Because a successful exit code can occur even if there is an empty
string returned, it is possible that the VMware datasource will be
loaded as a false-positive. This patch ensures that in addition to
validating the exit code, the emitted output is also examined to ensure
a non-empty value is returned by rpctool before returning "${DS_FOUND}"
from "dscheck_VMware()".
|
|
This patch finally introduces the Cloud-Init Datasource for VMware
GuestInfo as a part of cloud-init proper. This datasource has existed
since 2018, and rapidly became the de facto datasource for developers
working with Packer, Terraform, for projects like kube-image-builder,
and the de jure datasource for Photon OS.
The major change to the datasource from its previous incarnation is
the name. Now named DatasourceVMware, this new version of the
datasource will allow multiple transport types in addition to
GuestInfo keys.
This datasource includes several unique features developed to address
real-world situations:
* Support for reading any key (metadata, userdata, vendordata) both
from the guestinfo table when running on a VM in vSphere as well as
from an environment variable when running inside of a container,
useful for rapid dev/test.
* Allows booting with DHCP while still providing full participation
in Cloud-Init instance data and Jinja queries. The netifaces library
provides the ability to inspect the network after it is online,
and the runtime network configuration is then merged into the
existing metadata and persisted to disk.
* Advertises the local_ipv4 and local_ipv6 addresses via guestinfo
as well. This is useful as Guest Tools is not always able to
identify what would be considered the local address.
The primary author and current steward of this datasource spoke at
Cloud-Init Con 2020 where there was interest in contributing this datasource
to the Cloud-Init codebase.
The datasource currently lives in its own GitHub repository at
https://github.com/vmware/cloud-init-vmware-guestinfo. Once the datasource
is merged into Cloud-Init, the old repository will be deprecated.
|
|
This PR adds in support so that cloud-init can run on instances
deployed on Vultr cloud. This was originally brought up in #628.
Co-authored-by: Eric Benner <ebenner@vultr.com>
|
|
New datasource utilizing UpCloud metadata API, including relevant unit
tests and documentation.
|
|
FreeBSD lets us read out kernel parameters with kenv(1), a user-space
utility that's shipped in "base" We can use it in place of dmidecode(8),
thus removing the dependency on sysutils/dmidecode, and the restrictions
to i386 and x86_64 architectures that this utility imposes on FreeBSD.
Co-authored-by: Scott Moser <smoser@brickies.net>
|
|
FreeBSD doesn't have blkid, so we want to use geom to list devices and
their fstypes and labels.
This PR also adds `jail` to the list of is_container()
And we now also properly cache geom and blkid output!
A test is added to verify the new behaviour by correctly identifying
NoCloud on FreeBSD.
Co-authored-by: Scott Moser <smoser@brickies.net>
|
|
Update DataSourceNoCloud and ds-identify to recognize LABEL_FATBOOT labels from blkid.
Also updated associated tests.
LP: #1841466
|
|
add SAP Converged Cloud as cloud provider
|
|
On non-Linux systems, `/sys` won't be available. In these cases, we can query `dmidecode(8)` directly. This PR implements a dmi_decode function to query the same fields ds-identify
would otherwise read from /sys. This path is taken when /sys isn't present. In addition to
adding dmidecode support, non-Linux systems also need to map in virtualization detection
as systemd-detect-virt is not present; on FreeBSD, use sysctl kern.vm_guest and provide a
mapping[1] between BSD values and those that match with systemd-detect-virt[2].
1. https://github.com/freebsd/freebsd/blob/master/sys/kern/subr_param.c#L149-L157
2. https://www.freedesktop.org/software/systemd/man/systemd-detect-virt.html
LP: #1852442
|
|
e24cloud provides an EC2 compatible datasource.
This just identifies their platform based on dmi 'system-vendor'
having 'e24cloud'. https://www.e24cloud.com/en/ .
Updated chassis typo in zstack unit test docstring.
LP: #1696476
|
|
|
|
Zstack platform provides a AWS Ec2 metadata service, and
identifies their platform to the guest by setting the 'chassis asset tag'
to a string that ends with '.zstack.io'.
LP: #1841181
|
|
The detection for brightbox in both ds-identify and in
identify_brightbox would incorrectly match the domain 'bobrightbox',
which is not a brightbox platform. The fix here is to restrict
matching to '*.brightbox.com' rather than '*brightbox.com'
Also, while here remove a url to bug 1661693 which added the
knowledge of brightbox.
|
|
- dsidentify switches to the new Exoscale datasource on matching DMI name
- New Exoscale datasource added
Signed-off-by: Mathieu Corbin <mathieu.corbin@exoscale.ch>
|
|
With a large size ISO file attached to iso dev, ds-identify might
grep it entirely if iso dev is ISO9660, it takes very long time to
start OS.
Resolve this by:
- Adding a checker to read the ISO size (from sysfs). If
the size of the ISO filesystem is > 10MiB then the ISO will be
ignored (logged as oversized).
- Move the ovf vmware guest customization checker to be
ahead of cdrom ovf checker, so no need check the ISO size if vmware
guest customization is enabled.
LP: #1806701
|
|
When OpenStack is deployed on some hypervisors (such as VMware
vSphere), cloud-init doesn't detect that it needs to probe the
metadata service because the DMI product name field can't be set
to a field that is recognized by cloud-init. However, the asset
tag field can be set via flavor extra specs or image metadata.
A similar approach is already used to identify Open Telekom Cloud.
This patch allows cloud init to recognize "OpenStack Nova" or
"OpenStack Compute" in the asset tag field as an indication that
the instance being configured is running on an OpenStack platform.
LP: #1669875
|
|
NoCloud data source now accepts both 'cidata' and 'CIDATA'
as filesystem labels. This is similar to DataSourceConfigDrive's
support for 'config-2' and 'CONFIG-2'.
|
|
ovf_vmware_transport_guestinfo is not currently tested.
It used '$1' instead of '$out' when checking for xml content in
the output of vmware-rpctool.
|
|
This adds support for reading OVF information over the
'com.vmware.guestInfo' tranport. The current implementation requires
vmware-rpctool be installed in the system.
LP: #1807466
|
|
When deploying an OVA, at least some versions of vmware
attach a cdrom with an ISO9660 filesystem label of 'OVF ENV'.
This was seen on Vmware vCenter Server, 6.0.0, 2776510.
In order to accomplish this we had to change the content of
the DI_ISO9660_DEVS variable to be comma delimited rather
than space delimited.
|
|
There was a typo in the doc string at the top of ds-identify
(disable -> disabled). That is fixed here as well as adding some
better examples on content in /etc/cloud/ds-identify.cfg.
|
|
This adds a Oracle specific datasource that functions with OCI.
It is a simplified version of the OpenStack metadata server
with support for vendor-data.
It does not support the OCI-C (classic) platform.
Also here is a move of BrokenMetadata to common 'sources'
as this was the third occurrence of that class.
|
|
SuSE builds were not getting a PATH set in generator's environment.
This may seem like mis-configuration on the system, but caused ds-identify
to fail to find blkid (or any other program).
The change here just ensures that we get /sbin /usr/sbin /bin /usr/bin
into the PATH when main is run.
LP: #1771382
|
|
In playing with a SmartOS container I found that ds-identify did
not identify the container there as a container. Systemd-detect-virt
identifies it as 'container-other'.
Also here are tests for ds-identify for the SmartOS platform
identification, and some indentation fixes in ds-identify.
|
|
We had two calls to is_ds_enabled, and the debug message looked
something like this:
is_ds_enabled returned 1: ConfigDrive NoCloud
Now instead we have just one call, and the debug message like:
is_ds_enabled(IBMCloud) = true
|
|
This fixes warnings reported by shellcheck at 0.4.6.
The complaints that we are ignoring globally (top of the file) are:
2015: Note that A && B || C is not if-then-else. C may run if A is true.
2039: In POSIX sh, 'local' is undefined.
2162: read without -r will mangle backslashes.
2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.
Most of the complaints were just noise, but a few unused variables
were reported and fixed.
Related shellcheck issues opened:
- https://github.com/koalaman/shellcheck/issues/1191
- https://github.com/koalaman/shellcheck/issues/1192
- https://github.com/koalaman/shellcheck/issues/1193
- https://github.com/koalaman/shellcheck/issues/1194
|
|
Ubuntu images on IBMCloud for 16.04 have some seed data in
/var/lib/cloud/data/seed/nocloud-net. In order to have systems with
IBMCloud enabled, we modified ds-identify detection to skip that seed
if the system was on IBMCloud. That change did not consider the
fact that IBMCloud might not be in the datasource list.
There was similar logic in the ConfigDrive datasource in ds-identify
and the datasource itself.
Config drive is now updated to only check and avoid IBMCloud if IBMCloud
is enabled. The check in ds-identify for nocloud was dropped. If a
user provides a nocloud seed on IBMCloud, then that can be used.
This means that systems running Xenial will continue to get their
old datasources.
LP: #1766401
|
|
When images are deployed from template in a production environment
the artifacts of the provisioning stage (provisioningConfiguration.cfg)
that cloud-init referenced are cleaned up. However, when provisioned
in "debug" mode (internal to IBM) the artifacts are left.
This changes the 'is_ibm_provisioning' implementations in both
ds-identify and in the IBM datasource to identify the provisioning
stage more correctly. The change is to consider provisioning only
if the provisioing file existed and there was no log file or
the log file was older than this boot.
LP: #1767166
|
|
This adds a specific IBM Cloud datasource.
IBM Cloud is identified by:
a.) running on xen
b.) one of a LABEL=METADATA disk or a LABEL=config-2 disk with
UUID=9796-932E
The datasource contains its own config-drive reader that reads
only the currently supported portion of config-drive needed for
ibm cloud.
During the provisioning boot, cloud-init is disabled.
See the docstring in DataSourceIBMCloud.py for more more information.
|
|
Open Telekom Cloud gen1 (Xen) hosts do not provide nova product
names in DMI but Xen HVM domU. They can however be safely identified
by the OpenTelekomCloud Chassis asset tag. OpenTelekomCloud does
use the network OpenStack DataSource, so we better detect it.
LP: #1756471
|
|
The Hetzner Cloud metadata service is an AWS-style service available
over HTTP via the link local address 169.254.169.254.
https://hetzner.com/cloud
https://docs.hetzner.cloud/
|
|
On few 64-bit platforms, the open-vm-tools package is installed at
/usr/lib64/. The DataSourceOVF is changed to search look there for the
'customization plugin'
|
|
This fixes a bug in parsing of 'blkid -o export' output. The result
of the bug meant that DI_ISO9660_DEVS did not get set correctly and
is_cdrom_ovf would not identify devices in most cases.
The tests are improved to demonstrate both multiple iso devices
and also a cdrom that doesn't sort "last" in blkid output.
The code change is to use DEVNAME as the record separator when
parsing blkid -o export rather than relying on being able to read
the empty line.
LP: #1749980
|
|
Ubuntu core seeds information to nocloud via a bind-mount of
/writable/system-data/var/lib/cloud over /var/lib/cloud.
When ds-identify runs as a systemd generator that mount is not
guaranteed to have been done. It is guaranteed at
cloud-init-local.service time, but not generator time.
Images built with 'ubuntu-image --cloud-init=user-data-file'
would have cloud-init disabled.
The fix here is just to consider the seed dir under /writable/system-data.
LP: #1747070
|
|
Fujitsu Cloud Service attaches a ovf iso transport with a label
'OVFENV'. This seems to be a reasonable value as a label.
While the for bug 1731868 would likely fix cloud-init on fujitsu
cloud, this change will find it faster.
LP: #1698669
|
|
New mkfs.vfat and fatlabel tools included in the dosfsutils package no
longer support creating vfat disks with lowercase labels. They silently
default to an all uppercase label eg CONFIG-2 instead of config-2. This
change makes cloud-init handle either upper or lower case.
LP: #1598783
|
|
The previous OVF datasource change added a debug message that referenced
an un-used variable. The failure path would be triggered if an image was
booted with a iso9660 filesystem attached to a device that was not a
cdrom.
A unit test is added for the specific failure found.
Additional safety to avoid 'cidata' labels is also added to the OVF
checker.
LP: #1737704
|
|
Previously the OVF transport would not be identified except for when
config files set 'ovf_vmware_guest_customization'. It would also
return DS_MAYBE almost always.
The change here is to add support to ds-identify for storing the
iso9660 filesystems that it finds (ISO9660_DEVS). Then the OVF check
will check that the iso9660 filesystem has ovf-env.xml on it. The least
wonderful part of this is that the check is done by 'grep' for case
insensitive ovf-env.xml.
Future improvement would be to identify VMware's OVF by label or UUID
so we could avoid the grep.
LP: #1731868
|
|
OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.
An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.
LP: #1715241
|
|
Here we add and enable by default a datasource for Scaleway cloud.
The datasource quickly exits unless one of three things:
a.) 'Scaleway' found as the system vendor
b.) 'scaleway' found on the kernel command line.
c.) the directory /var/run/scaleway exists (this is currently created
by the scaleway initramfs module).
One interesting bit of this particular datasource is that it requires
the source port of the http request to be < 1024.
|
|
This allows the user to seed NoCloud in a trivial way from qemu/libvirt,
by using a stock image and passing a single command line flag. No custom
command line, no filesystem modification, no bootstrap disk image.
This is particularly handy now that Ec2 backend is discouraged from use
under bug 1660385.
LP: #1691772
|
|
Azure sets a known chassis asset tag to 7783-7084-3265-9085-8269-3286-77.
We can inspect this in both ds-identify and DataSource.get_data to
determine whether we are on Azure.
Added unit tests to cover these changes
and some minor tweaks to Exception error message content to give more
context on malformed or missing ovf-env.xml files.
LP: #1693939
|
|
AliYun cloud platform is now identifying themselves by setting the dmi
product id to the well known value "Alibaba Cloud ECS". The changes here
identify that properly in tools/ds-identify and in the DataSourceAliYun.
Since the 'get_data' for AliYun now identifies itself correctly, we can
enable AliYun by default.
LP: #1638931
|
|
If the user configured:
datasource_list: ["Ec2", "None"]
then ds-identify would write
datasource_list: ["Ec2", "None", "None"]
which would break the logic to avoid warning.
|
|
This adds several unit tests for ds-identify, and fixes a bug
in Ec2 detection that I found while writing these tests.
The method of testing is to use the ds-identify code as a shell
library. The TestDsIdentify:call basically does:
* populate a (temp) directory with files that represent what
ds-identify would see in /sys or other locations it reads.
* create a file '_shwrap' that replaces the 3 programs that are executed
in ds-identify code path. It supports setting their stdout, stderr,
and exit code.
* set the default policies explicitly (DI_DEFAULT_POLICY) so we can
support testing different builtins. This is necessary because the
Ubuntu branches patch the builtin value. If we did not explicilty set
it, then testing there would fail.
* execute sh to source the script and call its main.
|
