| Age | Commit message (Collapse) | Author |
|---|
|
This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.
|
|
The ubuntu-init-switch module allowed the use to launch an instance that
was booted with upstart and have it switch its init system to systemd and
then reboot itself. It was only useful for the time period when Ubuntu was
transitioning to systemd but only produced images using upstart.
Also, do not run setup with --init-system=upstart. This means that by
default, debian packages built with packages/bddeb will not have upstart
unit files included. No other removal is done here.
|
|
DataSourceEc2 behavior changed to first check a minimum acceptable
metadata version uri http://169.154.169.254/<min_version>/instance-id,
retrying on 404, until the metadata service is available. After the
metadata service is up, the datasource inspects preferred
extended_metadata_versions for availability. Unit tests only mocked the
preferred extended_metadata_version so all Ec2 tests were retrying
attempts against
http://169.254.169.254/meta-data/<min-version>/instance-id adding a lot of
time cost to the unit test runs.
This branch uses httpretty to properly mock the following:
- 404s from metadata on undesired extended_metadata_version test routes
- https://169.254.169.254/meta-data/2016-09-02/instance-id
- full metadata dictionary represented on min_metadata_version
- https://169.254.169.254/meta-data/2016-09-02/*
The branch also tightens httpretty to raise a MockError for any URL which
isn't mocked via httpretty.HTTPretty.allow_net_connect=False.
LP: #1714117
|
|
Currently the cloud-init default locale (en_US.UTF-8) is set by
the base datasource class. This patch allows a distro to overide
the fallback value with one that's available in the distro but continues
to respect an image which has preconfigured a locale.
- Distro object now has a get_locale method which will return a
preconfigure locale setting by checking the distros locale system
configuration file. If not set or not present, return the default
locale of en_US.UTF-8 which retains behavior of all previous cloud-init
releases.
- Apply locale now handles regenerating locales or system configuration
files as needed.
- Adjust apply_locale logic to skip locale-regen if the specified LANG
value is C.UTF-8,C, or POSIX; they do not require regeneration.
- Further add unittests to exercise the default paths for Ubuntu and
non-ubuntu paths to validate they get the LANG expected.
|
|
test_set_locale_sles and test_set_locale_sles_default were incorrectly
testing for truth of <distro_object>.uses_systemd rather than calling
that function and checking its result.
The error was only seen if the system running the tests was not using
systemd.
|
|
This gets initial opensuse and SLES support back to a working state.
Still missing is more complete network file writing and unit tests.
|
|
DataSourceEc2 now parses the metadata for each nic to determine if
configured for ipv6 and/or ipv4 addresses. In AWS for metadata version
2016-09-02, nics configured for ipv4 or ipv6 addresses will have non-zero
values stored in metadata at network/interfaces/macs/<MAC>/public-ipv4 or
ipv6s respectively. Those metadata files are only non-zero when an ipv4 or
ipv6 ip is associated to the specific nic. A new
DataSourceEc2.network_config property is added which parses the metadata
and renders a network version 1 dictionary representing both dhcp4 and
dhcp6 configuration for associated nics.
The network configuration returned from the datasource will also 'pin' the
nic name to the name presented on the instance for each nic.
LP: #1639030
|
|
Currently the python logging module will default to a local time which may
contain an TZ offset in the values it produces, but the logged time format
does not contain the offset. Switching to UTC time for logging produces
consistent values in the cloud-init.log file and avoids issues when the
timezone is changed during boot.
LP: #1713158
|
|
A patch to allow scripts missing a #! to run by using shell=True was
proposed but rejected. Instead we emit a log message to help the user
understand what went wrong.
|
|
In an effort to save file load cost during system boot, certain
subcommands, analyze and devel, do not get loaded unless the subcommand is
specified on the commandline. Because setup.py entrypoint for cloud-init
script doesn't specify sysv_args parameter when calling the CLI's main()
we need main to read sys.argv into sysv_args so our subparser loading
continues to work.
LP: #1712676
|
|
Both landscape and puppet modules had issues with the way they wrote
/etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or
python2. This branch adds initial unit tests for both modules which will
get better exercise under both python2 and python3.
The unit tests shed light on a few issues:
- In the cc_landscape module py3 can't provide six.StringIO content to
ConfigParser.write, so we need to use six.BytesIO instead
- In the cc_puppet module, python <= 2.7 doesn't support using
six.StringIO as a context manager, so we drop the context manager
fanciness and directly set outputstream = StringIO().
- The docstring in cc_puppet is fixed to document the 'conf'
sub-key requiring valid puppet section names for each
key-value list.
LP: #1699282
LP: #1710932
|
|
This branch does a few things:
- Add 'schema' subcommand to cloud-init CLI for validating
cloud-config files against strict module jsonschema definitions
- Add --annotate parameter to 'cloud-init schema' to annotate
existing cloud-config file content with validation errors
- Add jsonschema definition to cc_runcmd
- Add unit test coverage for cc_runcmd
- Update CLI capabilities documentation
This branch only imports development (and analyze) subparsers when the
specific subcommand is provided on the CLI to avoid adding costly unused
file imports during cloud-init system boot.
The schema command allows a person to quickly validate a cloud-config text
file against cloud-init's known module schemas to avoid costly roundtrips
deploying instances in their cloud of choice. As of this branch, only
cc_ntp and cc_runcmd cloud-config modules define schemas. Schema
validation will ignore all undefined config keys until all modules define
a strict schema.
To perform validation of runcmd and ntp sections of a cloud-config file:
$ cat > cloud.cfg <<EOF
runcmd: bogus
EOF
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg \
--annotate
Once jsonschema is defined for all ~55 cc modules, we will move this
schema subcommand up as a proper subcommand of the cloud-init CLI.
|
|
This branch adds cloudinit-analyze into cloud-init proper. It adds an
"analyze" subcommand to the cloud-init command line utility for quick
performance assessment of cloud-init stages and events.
On a cloud-init configured instance, running "cloud-init analyze blame"
will now report which cloud-init events cost the most wall time. This
allows for quick assessment of the most costly stages of cloud-init.
This functionality is pulled from Ryan Harper's analyze work.
The cloudinit-analyze main script itself has been refactored a bit for
inclusion as a subcommand of cloud-init CLI. There will be a followup
branch at some point which will optionally instrument detailed strace
profiling, but that approach needs a bit more discussion first.
This branch also adds:
* additional debugging topic to the sphinx-generated docs describing
cloud-init analyze, dump and show as well as cloud-init single usage.
* Updates the Makefile unittests target to include cloudinit directory
because we now have unittests within that package.
LP: #1709761
|
|
If the network-config sent to cloud-init is in version: 2 format then
when rendering netplan, we can pass the content through and avoid
consuming network_state elements. This removes the need for trying to
map many v2 features onto network state where other renderers won't be
able to use anyhow (for example match parameters for multi-interface
configuration and wifi configuration support).
Additionally ensure we retain bond/bridge v2 configuration in network
state so when rendering to eni or sysconfig we don't lose the configuration
- Drop the NotImplemented wifi exception, log a warning that it works for
netplan only
- Adjust unittests to new code path and output
- Fix issue with v2 macaddress values getting dropped
- Add unittests for consuming/validating v2 configurations
LP: #1709180
|
|
This feature enables the following VMware VCloud Director functionality:
1. Setting admin password
2. Expire password.
3. Set admin password and expire.
Password configuration is triggered only as part of a full
recustomization, that happens either on first power on or when
"poweron and full recustomization" is selected. Full customization
flow is determined by marker files. Unique marker ids are
generated when full recustomization is requested. And marker file based
on these marker ids help to determine if we need to execute the above
configuration.
|
|
This branch is a prerequisite for IPv6 support in AWS by allowing Ec2
datasource to query the metadata source version 2016-09-02 about whether
or not it needs to configure IPv6 on interfaces. If version 2016-09-02
is not present, fallback to the min_metadata_version of 2009-04-04. The
DataSourceEc2Local not run on FreeBSD because dhclient in doesn't
support the -sf flag allowing us to run dhclient without filesystem
side-effects.
To query AWS' metadata address @ 169.254.169.254, the instance must have
a dhcp-allocated address configured. Configuring IPv4 link-local
addresses result in timeouts from the metadata service. We introduced a
DataSourceEc2Local subclass which will perform a sandboxed dhclient
discovery which obtains an authorized IP address on eth0 and crawl
metadata about full instance network configuration.
Since ec2 IPv6 metadata is not sufficient in itself to tell us all the
ipv6 knownledge we need, it only be used as a boolean to tell us which
nics need IPv6. Cloud-init will then configure desired interfaces to
DHCPv6 versus DHCPv4.
Performance side note: Shifting the dhcp work into init-local for Ec2
actually gets us 1 second faster deployments by skipping init-network
phase of alternate datasource checks because Ec2Local is configured in
an ealier boot stage. In 3 test runs prior to this change: cloud-init
runs were 5.5 seconds, with the change we now average 4.6 seconds.
This efficiency could be even further improved if we avoiding dhcp
discovery in order to talk to the metadata service from an AWS
authorized dhcp address if there were some way to advertize the dhcp
configuration via DMI/SMBIOS or system environment variables.
Inspecting time costs of the dhclient setup/teardown in 3 live runs the
time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
dhcp discovery + metadata: 0.347 seconds
metadata alone: 0.271 seconds
test 2: 88 milliseconds
dhcp discovery + metadata: 0.388 seconds
metadata alone: 0.300 seconds
test 3: 75 milliseconds
dhcp discovery + metadata: 0.366 seconds
metadata alone: 0.291 seconds
LP: #1709772
|
|
Some systems like Ubuntu-Core do not provide an ntp package for
installation but do include systemd-timesyncd (an ntp client).
On such systems cloud-init will generate a timesyncd configuration
using the 'servers' and 'pools' values as ntp hosts for timesyncd to use.
LP: #1686485
|
|
The build deb command was no longer working becasue it had
assumed that you were in the root of the cloud-init directory.
This changes where the deb is built and changes how the
dependencies are determined as well as uses the built-in tools
for determining build dependencies.
|
|
The sysconfig renderer duplicates the cloud-init header string
when rendering resolv.conf file. This leads to resolv.conf file
growing with every reboot of a system. Fix this by checking for
the header when loading content from existing file.
Update one of the sysconfig unittests with multiple render calls
to simulate the reboot to check that we don't repeat the header.
LP: #1701420
|
|
If no dns nameservers were provided a stack trace would occur.
The changes here add some unit tests for the arch distro.
Also avoids rendering an 'lo' interface.
LP: #1663045
LP: #1706593
|
|
The locale integration test started failing with commit 0ef61b28.
This was just because the test's expectations on the content/formatting
of /etc/default/locale were too strict. The change here is to read
the file as a set of shell variables and assert that the values are
set correctly.
|
|
This is not strictly necessary, but since yakkety is no longer supported
we will remove it from the releases.yaml file.
|
|
EC2 was the original, but this adds some initial tests for that datasource.
Also updates a docstring for an internal method.
|
|
If the system configure default in /etc/default/locale is set to the same
value that is provided for cloud-init's "locale" setting, then do not
re-run locale-gen. This allows images built with a locale already
generated to not re-run locale-gen (which can be very heavy).
Also here is a fix to invoke update-locale correctly and remove the
internal writing of /etc/default/locale. We were calling
update-locale <locale>
This ends up having no affect. The more correct invocation is:
update-locale LANG=<locale>
Also added some support here should we ever want to change setting
LANG to setting LC_ALL (or any other key).
Lastly, a test change to allow us to use assert_not_called from mock.
Versions of mock in CentOS 6 do not have assert_not_called.
|
|
The subnet type 'manual' was used as a way to declare a device
and set an MTU for it but not assign network addresses.
This updates the manual example config to handle that case and
provides expected rendered output for sysconfig, eni, and netplan.
|
|
Previously, sysconfig rendered HWADDR for all interface types, but
that value is only used to identify physical devices. Instead use
MACADDR to configure the MAC on virtual devices, like bonds and
bridges.
- Sort bond slave list to ensure consistent ordering in sysconfig
rendered files.
- Add unittests for sysconfig rendering of bonds/bridges with
mac_address
LP: #1701417
|
|
In some network configurations a network value of '::' and a
netmask value of '::' were used to indicate a default IPV6 gateway.
Commit d00da2d5 removed ipv6 'netmask' attributes and calculate
a prefix length value instead. The eni route rendering failed to update
the check to use prefix value of 0 to indicate the presence of an IPV6
default route.
A broken ipv6 default route rendered like:
post-up route add -net :: netmask :: gw 2001:4800:78ff:1b::1 || true
And with this patch, it now renders like:
post-up route add -A inet6 default gw 2001:4800:78ff:1b::1 || true
LP: #1701097
|
|
Render MTU values if present in subnet and route configurations
for v4 and v6.
LP: #1702513
|
|
Implement manual control for sysconfig by using ONBOOT=N. This
allows an interface to be configured but not brought up.
Note that ONBOOT is per-interface not per address.
LP: #1687725
|
|
Currently only the subnet is checked for 'ipv6' setting, however, the
routes array may include a mix of v4 or v6 configurations, in particular,
the gateway in a route may be ipv6, and if so, should export the value via
IPV6_DEFAULTGW in the ifcfg-XXXX file.
Additionally, if the route is v6, it should rendering a routes6-XXXX file;
this is present but missing the 'dev <interface>' scoping.
LP: #1694801
|
|
Previously, virtual types (bond, bridge, vlan) were almost completely
broken. They would not get any network configuration (ip addresses or
dhcp config) and or routes rendered. This fixes those issues.
For bonds we now correctly render BONDING_SLAVE entries.
Also add tests for simple bond, bridge and vlan.
LP: #1695092
|
|
With this change, entries in IPV6ADDR and IPV6ADDR_SECONDARIES will now
always be in format addr/prefix. When a subnet has a gateway will be
written. If the gateway is ipv6, use the key IPV6_DEFAULTGW rather than
GATEWAY.
LP: #1704872
|
|
The network device renaming code previously required the case of
the mac address input to match that of the data read from the system.
For example, if user provided network config with mac address
in upper case, then cloud-init would not rename the device correctly
as /sys/class/net/address stores lower case values.
The fix here is to always compare lower case mac addresses.
LP: #1705147
|
|
This includes a few fixes found when testing with python 3.6.
- fix eni renderer when target is None
This just uses the util.target_path() in the event that target is None.
- change test cases to not rely on the cached result of
util.get_cmdline() and other cached globals. Update the base TestCase
to unset that cache.
- mock calls to system_is_snappy from the create_users test cases.
- drop unused _pp_root in test_simple_run.py
LP: #1703697
|
|
Render the GATEWAY= value in interface files which have a gateway in the
subnet configuration.
LP: #1686856
|
|
Here we add and enable by default a datasource for Scaleway cloud.
The datasource quickly exits unless one of three things:
a.) 'Scaleway' found as the system vendor
b.) 'scaleway' found on the kernel command line.
c.) the directory /var/run/scaleway exists (this is currently created
by the scaleway initramfs module).
One interesting bit of this particular datasource is that it requires
the source port of the http request to be < 1024.
|
|
load_shell_content previously would not allow shell comment characters
in the content being parsed. If comments=True is not passed then an
exception would previously be raised as the line would not be guaranteed to
have an '=' in it.
|
|
The usage of mock in this test was simply invalid and only worked by
happenstance.
|
|
The mock of platform_reports_gce is created with a True return value in
tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.setUp().
But, the final test_get_data_returns_false_if_not_on_gce incorrectly
attempts to override the mocked return_value of True to False by setting
self.m_platform_gce.return_value = False. But, since the mock is already
initialized, the updated False is not honored. Instead we should use the
patch decorator on the specific unit test to override the return_value of
DataSourceGCE.platform_reports_gce to False.
A False from platform_reports_gce allows DataSourceGCE.get_data to
immediately return False instead of trying to contact
metadata.google.internal as the related bug references.
|
|
The test is currently importing the incorrect keyid. It specifies
the curtin developers ppa, rather than the cloud-init ppa. On
Artful this causes failures as a check is made to verify the
correct key is imported for the ppa, whereas on previous releases
only a warning was issued.
Also, change to use a full key fingerprint.
LP: #1702717
|
|
Instead of passing around a 'log' reference to functions, just import
logging and use that. This is the pattern that is now more common in
cloud-init.
|
|
This fixes stacktrace and warning message that would be printed
to the log if running inside a container and read_dmi_data tried
to access a key that was not present.
In a container, the /sys/class/dmi/id data is not relevant to the
but to the host. Additionally an unpriviledged container might see
strange behavior:
# cd /sys/class/dmi/id/
# id -u
0
# ls -l chassis_serial
-r-------- 1 nobody nogroup 4096 Jun 29 16:49 chassis_serial
# cat chassis_serial
cat: /sys/class/dmi/id/chassis_serial: Permission denied
The solution here is to just always return None when running in a
container.
LP: #1701325
|
|
On systems with network devices with duplicate mac addresses, cloud-init
will fail to rename the devices according to the specified network
configuration. Refactor net layer to search by device driver and device
id if available. Azure systems may have duplicate mac addresses by
design.
Update Azure datasource to run at init-local time and let Azure datasource
generate a fallback networking config to handle advanced networking
configurations.
Lastly, add a 'setup' method to the datasources that is called before
userdata/vendordata is processed but after networking is up. That is
used here on Azure to interact with the 'fabric'.
|
|
I want to be able to add additional SSH keys to my account, therefore I
should not be limiting these tests to look for one specific key. Instead
we confirm that the comment in authorized_users has the specified users.
|
|
Recent change to ntp in artful has added the sntp package whenever
ntp is installed. The tests, rather poorly, did a dpkg -l instead
of checking with `which`. This fixes the ntp tests to all use
`which` over expecting a certain number of lines using dpkg and
as a result make the tests OS independent.
|
|
- Simplify the logic of 'variant' in util.system_info
much of the data from
https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
- fix get_resource_disk_on_freebsd when running on a system without
an Azure resource disk.
- fix tools/build-on-freebsd to replace oauth with oauthlib and add
bash which is a dependency for tests.
- update a fiew places that were checking for freebsd but not using
the util.is_FreeBSD()
|
|
The previous commit caused test failure.
This separates out _check_freebsd_cdrom and mocks it in a test
rather than patching open.
|
|
The current method is to attempt to mount the cdrom (/dev/cd0), if it is
successful, /dev/cd0 is configured, otherwise, it is not configured. The
problem is it forgets to check whether the mounting destination folder is
created or not. As a result, mounting attempt failed even if cdrom is
ready.
LP: #1696295
|
|
Unix file modes are usually represented as octal, but they were being
interpreted as decimal, for example 0o644 would be printed as '420'.
Reviewed-by: Tom Kirchner <tjk@amazon.com>
|
|
These changes are all in an effort to get tools/run-centos using
read-dependencies rather than the 'setup-centos' script with a separate
set of dependencies listed.
- tools/read-dependencies: support taking multiple --requirements
options. This allows run-centos to get both test and build
dependencies. Ultimately, I think it might be nicer for
read-dependencies to take a list of "goals" (build, test, run or
test-tox) rather than having the caller need to know to provide
multiple --requirements.
- packages/pkg-deps.json: drop the version on the sudo package.
centos 6 has newer (1.8.6p3) version than listed, so its not a problem.
- test_handler_disk_setup.py: a test case here was using assertLogs
which is not present in the version of unittest2 that is available in
centos 6 epel. We just adjust it to use with_logs = True.
- tools/run-cents:
- improve usage with example
- add 'inside_as_cd' to provide the dir you want to cd first to.
- avoid the intermediate tarball on disk in the container.
- add 'prep' subcommand and use it to install pre-dependencies.
- use read-dependencies.
|
