| Age | Commit message (Collapse) | Author |
|---|
|
Systems that used systemd-networkd's dhcp client would not be able to get
information on the Azure endpoint (placed in Option 245) or the CloudStack
server (in 'server_address').
The change here supports reading these files in /run/systemd/netif/leases.
The files declare that "This is private data. Do not parse.", but at this
point we do not have another option.
LP: #1718029
|
|
This removes shlex and converts the subprocess commands to use a
list over a string.
|
|
Previously we had hard coded paths in /sbin for the udevadm and modprobe
programs invoked by AltCloud. Its more flexible to expect the PATH to
be set correctly.
Debian: #852564
|
|
DataSourceOVF attempts to find iso files via walking os.listdir('/dev/')
which is far too wide. This approach is too invasive and can sometimes
race with systemd attempting to fsck and mount devices.
Instead, utilize cloudinit.util.find_devs_with to filter devices by
criteria (which uses blkid under the covers). This results in fewer
attempts to mount block devices which do not contain iso filesystems.
Unittest changes include:
- cloudinit.tests.helpers; introduce add_patch() helper
- Add unittest coverage for DataSourceOVF use of transport_iso9660
LP: #1718287
|
|
The bootcmd test was leaving files in the tmpdir named
ci-FakeExtendedTempFile.XXXXXX.
This cleans those up.
|
|
Create a copy of each modules schema attribute when generating sphinx docs
to avoid altering the actual module dict in memory. This avoids illegible
rendering of module examples and distros where each character of a list
was represented on a separate line by itself.
Fixes ntp, resizefs, runcmd and bootcmd docs.
|
|
The variable DI_EC2_STRICT_ID_DEFAULT was not being set in unit tests
so when 16.04 built, which changed that setting in patches the tests
would unexpectedly fail.
|
|
This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data that was read in
self.userdata_raw, rather than self.userdata. That is consistent
with other datasources and ulitimately how it was before the refactor.
The main is updated to address the fact that user-data is binary data
and may not be able to be printed.
LP: #1717598
|
|
Add a new collect-logs sub command to the cloud-init CLI. This script
will collect all logs pertinent to a cloud-init run and store them in a
compressed tar-gzipped file. This tarfile can be attached to any
cloud-init bug filed in order to aid in bug triage and resolution.
A cloudinit.apport module is also added that allows apport interaction.
Here is an example bug filed via ubuntu-bug cloud-init: LP: #1716975.
Once the apport launcher is packaged in cloud-init, bugs can be filed
against cloud-init with the following command:
ubuntu-bug cloud-init
LP: #1607345
|
|
A regression in 'get_latest_lease' made it ignore files starting with
'dhclient-' rather than just 'dhclient.'. The fix here is to allow those
files to be considered.
There is a lot more we could do here to better ensure that we pick the
most recent lease, but this change fixes the regression.
LP: #1717147
|
|
As root user, os.access(<path>, os.W_OK) will always return True so that
path will never get executed. Also avoid a warning if the root is
overlayroot, which is the common case on a MAAS booted 'ephemeral' system.
|
|
The NoCloud KVM platform includes:
* Downloads daily Ubuntu images using streams and store in
/srv/images
* Image customization, if required, is done using
mount-image-callback otherwise image is untouched
* Launches KVM via the xkvm script, a wrapper around
qemu-system, and sets custom port for SSH
* Generation and inject an SSH (RSA 4096) key pair to use for
communication with the guest to collect test artifacts
* Add method to produce safe shell strings by base64 encoding
the command
Additional Changes:
* Set default backend to use LXD
* Verify not running script as root in order to prevent images
from becoming owned by root
* Removed extra quotes around that were added when collecting
the cloud-init version from the image
* Added info about each release as previously the lxd backend
was able to query that information from pylxd image info,
however, other backends will not be able to obtain the same
information as easily
|
|
Modules can optionally define a list of supported distros on which they can run
by declaring a distros attribute in the cc_*py module. This branch fixes
handling of cloudinit.stages.Modules.run_section. The behavior of run_section
is now the following:
- always run a module if the module doesn't declare a distros attribute
- always run a module if the module declares distros = [ALL_DISTROS]
- skip a module if the distribution on which we run isn't in module.distros
- force a run of a skipped module if unverified_modules configuration contains
the module name
LP: #1715738
LP: #1715690
|
|
Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.
This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.
LP: #1462693
|
|
If a string is passed to execute, then invoke 'bash', '-c',
'string'. That allows the less verbose execution of simple
commands:
image.execute("ls /run")
compared to the more explicit but longer winded:
image.execute(["ls", "/run"])
If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.
|
|
Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.
This branch also does the following:
- drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
option only created the specified directory and didn't make use of
that directory for any resize operations.
- Drop yaml.dumps calls from schema documentation generation to avoid
yaml import costs on module load
- Add __doc__ = get_schema_doc(schema) definitions it each module to
supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
cc_resizefs
- Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
contain more than one example
|
|
For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.
This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.
Modify the code to read the customization configuration and return the
converted data.
Added few tests.
LP: #1675063
|
|
This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:
a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)
b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.
The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.
Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.
LP: #1715128
|
|
During boot, the usage of /tmp is not safe. In systemd systems,
systemd-tmpfiles-clean may run at any point and clear out a temp file
while cloud-init is using it. The solution here is to use
/run/cloud-init/tmp.
LP: #1707222
|
|
OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.
An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.
LP: #1715241
|
|
This missed mock in test_openstack resulted in a costly unit test timeout.
LP: #1714376
|
|
This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.
|
|
The ubuntu-init-switch module allowed the use to launch an instance that
was booted with upstart and have it switch its init system to systemd and
then reboot itself. It was only useful for the time period when Ubuntu was
transitioning to systemd but only produced images using upstart.
Also, do not run setup with --init-system=upstart. This means that by
default, debian packages built with packages/bddeb will not have upstart
unit files included. No other removal is done here.
|
|
DataSourceEc2 behavior changed to first check a minimum acceptable
metadata version uri http://169.154.169.254/<min_version>/instance-id,
retrying on 404, until the metadata service is available. After the
metadata service is up, the datasource inspects preferred
extended_metadata_versions for availability. Unit tests only mocked the
preferred extended_metadata_version so all Ec2 tests were retrying
attempts against
http://169.254.169.254/meta-data/<min-version>/instance-id adding a lot of
time cost to the unit test runs.
This branch uses httpretty to properly mock the following:
- 404s from metadata on undesired extended_metadata_version test routes
- https://169.254.169.254/meta-data/2016-09-02/instance-id
- full metadata dictionary represented on min_metadata_version
- https://169.254.169.254/meta-data/2016-09-02/*
The branch also tightens httpretty to raise a MockError for any URL which
isn't mocked via httpretty.HTTPretty.allow_net_connect=False.
LP: #1714117
|
|
Currently the cloud-init default locale (en_US.UTF-8) is set by
the base datasource class. This patch allows a distro to overide
the fallback value with one that's available in the distro but continues
to respect an image which has preconfigured a locale.
- Distro object now has a get_locale method which will return a
preconfigure locale setting by checking the distros locale system
configuration file. If not set or not present, return the default
locale of en_US.UTF-8 which retains behavior of all previous cloud-init
releases.
- Apply locale now handles regenerating locales or system configuration
files as needed.
- Adjust apply_locale logic to skip locale-regen if the specified LANG
value is C.UTF-8,C, or POSIX; they do not require regeneration.
- Further add unittests to exercise the default paths for Ubuntu and
non-ubuntu paths to validate they get the LANG expected.
|
|
test_set_locale_sles and test_set_locale_sles_default were incorrectly
testing for truth of <distro_object>.uses_systemd rather than calling
that function and checking its result.
The error was only seen if the system running the tests was not using
systemd.
|
|
This gets initial opensuse and SLES support back to a working state.
Still missing is more complete network file writing and unit tests.
|
|
DataSourceEc2 now parses the metadata for each nic to determine if
configured for ipv6 and/or ipv4 addresses. In AWS for metadata version
2016-09-02, nics configured for ipv4 or ipv6 addresses will have non-zero
values stored in metadata at network/interfaces/macs/<MAC>/public-ipv4 or
ipv6s respectively. Those metadata files are only non-zero when an ipv4 or
ipv6 ip is associated to the specific nic. A new
DataSourceEc2.network_config property is added which parses the metadata
and renders a network version 1 dictionary representing both dhcp4 and
dhcp6 configuration for associated nics.
The network configuration returned from the datasource will also 'pin' the
nic name to the name presented on the instance for each nic.
LP: #1639030
|
|
Currently the python logging module will default to a local time which may
contain an TZ offset in the values it produces, but the logged time format
does not contain the offset. Switching to UTC time for logging produces
consistent values in the cloud-init.log file and avoids issues when the
timezone is changed during boot.
LP: #1713158
|
|
A patch to allow scripts missing a #! to run by using shell=True was
proposed but rejected. Instead we emit a log message to help the user
understand what went wrong.
|
|
In an effort to save file load cost during system boot, certain
subcommands, analyze and devel, do not get loaded unless the subcommand is
specified on the commandline. Because setup.py entrypoint for cloud-init
script doesn't specify sysv_args parameter when calling the CLI's main()
we need main to read sys.argv into sysv_args so our subparser loading
continues to work.
LP: #1712676
|
|
Both landscape and puppet modules had issues with the way they wrote
/etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or
python2. This branch adds initial unit tests for both modules which will
get better exercise under both python2 and python3.
The unit tests shed light on a few issues:
- In the cc_landscape module py3 can't provide six.StringIO content to
ConfigParser.write, so we need to use six.BytesIO instead
- In the cc_puppet module, python <= 2.7 doesn't support using
six.StringIO as a context manager, so we drop the context manager
fanciness and directly set outputstream = StringIO().
- The docstring in cc_puppet is fixed to document the 'conf'
sub-key requiring valid puppet section names for each
key-value list.
LP: #1699282
LP: #1710932
|
|
This branch does a few things:
- Add 'schema' subcommand to cloud-init CLI for validating
cloud-config files against strict module jsonschema definitions
- Add --annotate parameter to 'cloud-init schema' to annotate
existing cloud-config file content with validation errors
- Add jsonschema definition to cc_runcmd
- Add unit test coverage for cc_runcmd
- Update CLI capabilities documentation
This branch only imports development (and analyze) subparsers when the
specific subcommand is provided on the CLI to avoid adding costly unused
file imports during cloud-init system boot.
The schema command allows a person to quickly validate a cloud-config text
file against cloud-init's known module schemas to avoid costly roundtrips
deploying instances in their cloud of choice. As of this branch, only
cc_ntp and cc_runcmd cloud-config modules define schemas. Schema
validation will ignore all undefined config keys until all modules define
a strict schema.
To perform validation of runcmd and ntp sections of a cloud-config file:
$ cat > cloud.cfg <<EOF
runcmd: bogus
EOF
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg \
--annotate
Once jsonschema is defined for all ~55 cc modules, we will move this
schema subcommand up as a proper subcommand of the cloud-init CLI.
|
|
This branch adds cloudinit-analyze into cloud-init proper. It adds an
"analyze" subcommand to the cloud-init command line utility for quick
performance assessment of cloud-init stages and events.
On a cloud-init configured instance, running "cloud-init analyze blame"
will now report which cloud-init events cost the most wall time. This
allows for quick assessment of the most costly stages of cloud-init.
This functionality is pulled from Ryan Harper's analyze work.
The cloudinit-analyze main script itself has been refactored a bit for
inclusion as a subcommand of cloud-init CLI. There will be a followup
branch at some point which will optionally instrument detailed strace
profiling, but that approach needs a bit more discussion first.
This branch also adds:
* additional debugging topic to the sphinx-generated docs describing
cloud-init analyze, dump and show as well as cloud-init single usage.
* Updates the Makefile unittests target to include cloudinit directory
because we now have unittests within that package.
LP: #1709761
|
|
If the network-config sent to cloud-init is in version: 2 format then
when rendering netplan, we can pass the content through and avoid
consuming network_state elements. This removes the need for trying to
map many v2 features onto network state where other renderers won't be
able to use anyhow (for example match parameters for multi-interface
configuration and wifi configuration support).
Additionally ensure we retain bond/bridge v2 configuration in network
state so when rendering to eni or sysconfig we don't lose the configuration
- Drop the NotImplemented wifi exception, log a warning that it works for
netplan only
- Adjust unittests to new code path and output
- Fix issue with v2 macaddress values getting dropped
- Add unittests for consuming/validating v2 configurations
LP: #1709180
|
|
This feature enables the following VMware VCloud Director functionality:
1. Setting admin password
2. Expire password.
3. Set admin password and expire.
Password configuration is triggered only as part of a full
recustomization, that happens either on first power on or when
"poweron and full recustomization" is selected. Full customization
flow is determined by marker files. Unique marker ids are
generated when full recustomization is requested. And marker file based
on these marker ids help to determine if we need to execute the above
configuration.
|
|
This branch is a prerequisite for IPv6 support in AWS by allowing Ec2
datasource to query the metadata source version 2016-09-02 about whether
or not it needs to configure IPv6 on interfaces. If version 2016-09-02
is not present, fallback to the min_metadata_version of 2009-04-04. The
DataSourceEc2Local not run on FreeBSD because dhclient in doesn't
support the -sf flag allowing us to run dhclient without filesystem
side-effects.
To query AWS' metadata address @ 169.254.169.254, the instance must have
a dhcp-allocated address configured. Configuring IPv4 link-local
addresses result in timeouts from the metadata service. We introduced a
DataSourceEc2Local subclass which will perform a sandboxed dhclient
discovery which obtains an authorized IP address on eth0 and crawl
metadata about full instance network configuration.
Since ec2 IPv6 metadata is not sufficient in itself to tell us all the
ipv6 knownledge we need, it only be used as a boolean to tell us which
nics need IPv6. Cloud-init will then configure desired interfaces to
DHCPv6 versus DHCPv4.
Performance side note: Shifting the dhcp work into init-local for Ec2
actually gets us 1 second faster deployments by skipping init-network
phase of alternate datasource checks because Ec2Local is configured in
an ealier boot stage. In 3 test runs prior to this change: cloud-init
runs were 5.5 seconds, with the change we now average 4.6 seconds.
This efficiency could be even further improved if we avoiding dhcp
discovery in order to talk to the metadata service from an AWS
authorized dhcp address if there were some way to advertize the dhcp
configuration via DMI/SMBIOS or system environment variables.
Inspecting time costs of the dhclient setup/teardown in 3 live runs the
time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
dhcp discovery + metadata: 0.347 seconds
metadata alone: 0.271 seconds
test 2: 88 milliseconds
dhcp discovery + metadata: 0.388 seconds
metadata alone: 0.300 seconds
test 3: 75 milliseconds
dhcp discovery + metadata: 0.366 seconds
metadata alone: 0.291 seconds
LP: #1709772
|
|
Some systems like Ubuntu-Core do not provide an ntp package for
installation but do include systemd-timesyncd (an ntp client).
On such systems cloud-init will generate a timesyncd configuration
using the 'servers' and 'pools' values as ntp hosts for timesyncd to use.
LP: #1686485
|
|
The build deb command was no longer working becasue it had
assumed that you were in the root of the cloud-init directory.
This changes where the deb is built and changes how the
dependencies are determined as well as uses the built-in tools
for determining build dependencies.
|
|
The sysconfig renderer duplicates the cloud-init header string
when rendering resolv.conf file. This leads to resolv.conf file
growing with every reboot of a system. Fix this by checking for
the header when loading content from existing file.
Update one of the sysconfig unittests with multiple render calls
to simulate the reboot to check that we don't repeat the header.
LP: #1701420
|
|
If no dns nameservers were provided a stack trace would occur.
The changes here add some unit tests for the arch distro.
Also avoids rendering an 'lo' interface.
LP: #1663045
LP: #1706593
|
|
The locale integration test started failing with commit 0ef61b28.
This was just because the test's expectations on the content/formatting
of /etc/default/locale were too strict. The change here is to read
the file as a set of shell variables and assert that the values are
set correctly.
|
|
This is not strictly necessary, but since yakkety is no longer supported
we will remove it from the releases.yaml file.
|
|
EC2 was the original, but this adds some initial tests for that datasource.
Also updates a docstring for an internal method.
|
|
If the system configure default in /etc/default/locale is set to the same
value that is provided for cloud-init's "locale" setting, then do not
re-run locale-gen. This allows images built with a locale already
generated to not re-run locale-gen (which can be very heavy).
Also here is a fix to invoke update-locale correctly and remove the
internal writing of /etc/default/locale. We were calling
update-locale <locale>
This ends up having no affect. The more correct invocation is:
update-locale LANG=<locale>
Also added some support here should we ever want to change setting
LANG to setting LC_ALL (or any other key).
Lastly, a test change to allow us to use assert_not_called from mock.
Versions of mock in CentOS 6 do not have assert_not_called.
|
|
The subnet type 'manual' was used as a way to declare a device
and set an MTU for it but not assign network addresses.
This updates the manual example config to handle that case and
provides expected rendered output for sysconfig, eni, and netplan.
|
|
Previously, sysconfig rendered HWADDR for all interface types, but
that value is only used to identify physical devices. Instead use
MACADDR to configure the MAC on virtual devices, like bonds and
bridges.
- Sort bond slave list to ensure consistent ordering in sysconfig
rendered files.
- Add unittests for sysconfig rendering of bonds/bridges with
mac_address
LP: #1701417
|
|
In some network configurations a network value of '::' and a
netmask value of '::' were used to indicate a default IPV6 gateway.
Commit d00da2d5 removed ipv6 'netmask' attributes and calculate
a prefix length value instead. The eni route rendering failed to update
the check to use prefix value of 0 to indicate the presence of an IPV6
default route.
A broken ipv6 default route rendered like:
post-up route add -net :: netmask :: gw 2001:4800:78ff:1b::1 || true
And with this patch, it now renders like:
post-up route add -A inet6 default gw 2001:4800:78ff:1b::1 || true
LP: #1701097
|
|
Render MTU values if present in subnet and route configurations
for v4 and v6.
LP: #1702513
|
|
Implement manual control for sysconfig by using ONBOOT=N. This
allows an interface to be configured but not brought up.
Note that ONBOOT is per-interface not per address.
LP: #1687725
|
