| Age | Commit message (Collapse) | Author |
|---|
|
LP: #1754495
|
|
This makes 2 changes to shellify's behavior:
a.) raise a TypeError rather than a RuntimeError.
b.) raise a TypeError if input is not a list or tuple.
|
|
Previously the module was not working under FreeBSD due to a different
package name and some different paths. The module now has OS specific
default values which can even be customized via corresponding cloud config
variables.
LP: #1721503
|
|
Last set of changes to GCE datasource broke reading of user-data
unless the user had base64 encoded their user-data and also set
user-data-encoding to 'base64'.
This fixes the issue.
LP: #1752711
|
|
If python starts up without a locale set, then its default encoding
ends up set as ascii. That is not easily changed with the likes of
setlocale. In order to avoid UnicodeDecodeErrors cloud-init will
encode to bytes a python3 string or python2 basestring so that the
values passed to Popen are already bytes.
LP: #1751051
|
|
While salt grains can be configured in the minion config file, it is
usually better to configure it in the /etc/salt/grains file.
This allows that to be done.
|
|
On few 64-bit platforms, the open-vm-tools package is installed at
/usr/lib64/. The DataSourceOVF is changed to search look there for the
'customization plugin'
|
|
This fixes a bug in parsing of 'blkid -o export' output. The result
of the bug meant that DI_ISO9660_DEVS did not get set correctly and
is_cdrom_ovf would not identify devices in most cases.
The tests are improved to demonstrate both multiple iso devices
and also a cdrom that doesn't sort "last" in blkid output.
The code change is to use DEVNAME as the record separator when
parsing blkid -o export rather than relying on being able to read
the empty line.
LP: #1749980
|
|
Ubuntu core seeds information to nocloud via a bind-mount of
/writable/system-data/var/lib/cloud over /var/lib/cloud.
When ds-identify runs as a systemd generator that mount is not
guaranteed to have been done. It is guaranteed at
cloud-init-local.service time, but not generator time.
Images built with 'ubuntu-image --cloud-init=user-data-file'
would have cloud-init disabled.
The fix here is just to consider the seed dir under /writable/system-data.
LP: #1747070
|
|
When we moved some tests to live under cloudinit/ we inadvertantly
failed to change all things that would run nose to include that
directory.
This changes all the 'nose' invocations to consistently run with
tests/unittests and cloudinit/.
Also, it works around, more correctly this time, a python2.6-ism with
the following code:
with assertRaises(SystemExit) as cm:
sys.exit(2)
|
|
This puts in place detection for if 'show-log' will work with lxc
client, and uses that if present. The 'lxc console --show-log' is
not expected to work until lxd/liblxc3.0. That should come in a
few months. The hope is that when that function arrives, this
code will move over to using it.
For other scenarios (all current lxd installs) this will now
support getting logs from a snap installed lxd or a package installed
lxd via the old 'lxc.console.logfile'.
If installed from snap, a platform error will be raised until
the user does:
sudo mkdir --mode=1777 -p /var/snap/lxd/common/consoles
LP: #1745663
|
|
Fix an issue in EC2 where the datasource.identity had not been
initialized before being used when restoring datasource from pickle.
This is exposed in upgrade and reboot path.
LP: #1748354
|
|
This fixes a bug where invalid keys would sneak into authorized_keys.
|
|
Fujitsu Cloud Service attaches a ovf iso transport with a label
'OVFENV'. This seems to be a reasonable value as a label.
While the for bug 1731868 would likely fix cloud-init on fujitsu
cloud, this change will find it faster.
LP: #1698669
|
|
This adds collection a gzip compressed systemd journal on systemd systems.
The file can later be reviewed with:
zcat system.journal.gz > system.journal
journalctl --file=system.journal [-o short-monotonic ..]
To support this:
* modify test harness infrastructure to not assume content is utf-8.
* fix lxd platform to support make '_execute' return bytes rather
than a string. https://github.com/lxc/pylxd/issues/268
Also switched the base collectors to use /bin/sh as others already did.
|
|
This change will enable azure vms to report provisioning has completed
twice, first to tell the fabric it has completed then a second time to
enable customer settings. The datasource for the second provisioning is
the Instance Metadata Service (IMDS),and the VM will poll indefinitely for
the new ovf-env.xml from IMDS.
This branch introduces EphemeralDHCPv4 which encapsulates common logic
used by both DataSourceEc2 an DataSourceAzure for temporary DHCP
interactions without side-effects.
LP: #1734991
|
|
Resize of btrfs fails if the mount point for the file system we are trying
to resize, i.e. the root of the filesystem is read only. With this change
we use a known (currently snapper specific) rw location to work around a
flaw that blocks resizing of the ro filesystem.
LP: #1734787
|
|
Network configuration in OpenNebula would only work if the host correctly
guessed the names of the devices in the guest. OpenNebula provided data
in its context.sh like 'ETH0_NETWORK', but if the guest named devices
differently then results were not predictable. This would occur with
Predictable Network Interface Names. To address this,
newer versions (of OpenNebula provide the mac address ETH0_MAC.
This function is present in 4.14 and documented officially in 5.0 docs.
This provides support for reading the mac addresses from the context.sh.
It also fixes cases where context.sh provided a field (ETH0_NETWORK
or ETH0_MASK) with a empty string. Previously the empty string would
be used rather than falling back to the default.
LP: #1719157, #1716397, #1736750
|
|
The EC2 test platform uses boto, and boto decodes console output
with decode('utf-8', 'replace). It is known that Ubuntu consoles
contain non-utf8 characters, making this call lossy.
The change here is to patch the boto session to include a OutputBytes
entry in the console_output response, and then to utilize that in
console_log.
More information on problem and solution at:
https://github.com/boto/botocore/issues/1351
|
|
The previous commit added a test that would attempt to create and use
/run/cloud-init/. This just modifies it to use a temp dir instead.
|
|
The behavior changes and improvements include:
- Only import keys into the default user that contain the name of the
default user ('ubuntu', or 'centos') or that contain 'cloudinit'.
- Use instance or project level keys based on GCE convention.
- Respect expiration time when keys are set.
Do not import expired keys.
- Support ssh-keys in project level metadata (the GCE default).
As part of this change, we also update the request header when talking
to the metadata server based on the documentation:
https://cloud.google.com/compute/docs/storing-retrieving-metadata#querying
LP: #1670456, #1707033, #1707037, #1707039
|
|
When subp raised a ProcessExecutionError, that exception's stderr and
stdout might end up being the string '-' rather than bytes.
This mean that:
try:
subp(mycommand, decode=False)
except ProcessExecutionError as e:
pass
Would have 'e.stdout' set to '-' while the caller would expect bytes.
Also reduce the try/except block in subp to a specifically the two lines
that may raise an OSError.
|
|
Tests run on EC2 would successfully resolve the ipv4 dns address
and that caused false positives on failure reports.
Basically, dns lookup of 172.16.15.14 would return
ip-172-16-15-14.us-east-2.compute.internal.
which then shows up in the ntpq output unless you provide -n.
|
|
New mkfs.vfat and fatlabel tools included in the dosfsutils package no
longer support creating vfat disks with lowercase labels. They silently
default to an all uppercase label eg CONFIG-2 instead of config-2. This
change makes cloud-init handle either upper or lower case.
LP: #1598783
|
|
Zesty goes EOL as of January 13, 2017. This removes it as a valid
OS for testing.
|
|
This issue was first identified when manual_cache_clean was set, as
ds-identify would write /run/cloud-init/cloud.cfg with
# manual_cache_clean
that would generate a warning as cloud-init expected to load a dict.
Any other "empty" config would also log such a warning.
Also fix reading of di_report to allow it to be None, as ds-identify
would write:
di_report:
# manual_cache_clean
which reads as 'di_report: None' rather than di_report: {}.
LP: #1742479
|
|
This stores a hash of the OAuth tokens as an 'id' for the maas
datasource. Since new instances get new tokens created and those tokens
are written by curtin into datasource system config this will provide
a way to identify a new "instance" (install).
LP: #1712680
|
|
Due to additional platforms getting added this test was not taking into
account platform specific mirrors nor was it checking that no additional
entries were added.
|
|
This fixes the incorrectly named 'family' value for images as 'os'.
Families are already defined in util.py:OS_FAMILY_MAPPING and a family
is a collection of OSes.
This makes the properties function part of the super class of image as
it is only overrided by the lxd backend.
|
|
While the generated ssh keys are throw away keys, generating SSH keys
with the default name (e.g. id_rsa) can trigger security scanners or
draw unnecessary attention.
The change here simply renames 'id_rsa' to 'cloud_init_rsa' to avoid
a false positive reported by a scanning tool.
|
|
This enables integration tests to utilize AWS EC2 as a testing platform by
utilizing the boto3 Python library.
Usage will create and delete a custom VPC for every run. All resources
will be tagged with the ec2 tag, 'cii', and the date (e.g.
cii-20171220-102452). The VPC is setup with both IPv4 and IPv6
capabilities, but will only hand out IPv4 addresses by default. Instances
will have complete Internet access and have full ingress and egress access
(i.e. no firewall).
SSH keys are generated with each run of the integration tests with the key
getting uploaded to AWS at the start of tests and deleted on exit. To
enable creation when the platform is setup the SSH generation code is
moved to be completed by the platform setup and not during image setup.
The nocloud-kvm platform was updated with this change.
Creating a custom image will utilize the same clean script,
boot_clean_script, that the LXD platform uses as well. The custom AMI is
generated, used, and de-registered after a test run.
The default instance type is set to t2.micro. This is one of the smallest
instance types and is free tier eligible.
The default timeout for ec2 was increased to 300 from 120 as many tests
hit up against the 2 minute timeout and depending on region load can
go over.
Documentation for the AWS platform was added with the expected
configuration files for the platform to be used. There are some
additional whitespace changes included as well.
pylint exception was added for paramiko and simplestreams. In the past
these were not already flagged due to no __init__.py in the subdirectories
of files that used these. boto3 was added to the list of dependencies in
the tox ci-test runner.
In order to grab console logs on EC2 the harness will now shut down an
instance before terminating and before collecting the console log. This
is to address a behavior of EC2 where the console log is refreshed very
infrequently, but one point when it is refreshed is after shutdown.
|
|
This simply increases test coverage in rendering of network config
for SUSE and SLES.
|
|
This fixes a traceback when attempting to bounce the network after
hostname resets.
In artful and bionic ifupdown package is no longer installed in default
cloud images. As such, Azure can't use those tools to bounce the network
informing DDNS about hostname changes. This doesn't affect DDNS updates
though because systemd-networkd is now watching hostname deltas and with
default behavior to SendHostname=True over dhcp for all hostname updates
which publishes DDNS for us.
LP: #1722668
|
|
The cli help docs and argument parser allow the 'init' mode value
which caused a traceback.
Fix the cli to support 'init', 'config' and 'final' modes for the
cloud-init modules subcommand.
Add a check in the cli to raise a ValueError if a new
subcommand ends up allowing an unsupported/unimplemented modes.
Drive by unit test additions for a bit better coverage of error
handling.
LP: #1736600
|
|
The previous OVF datasource change added a debug message that referenced
an un-used variable. The failure path would be triggered if an image was
booted with a iso9660 filesystem attached to a device that was not a
cdrom.
A unit test is added for the specific failure found.
Additional safety to avoid 'cidata' labels is also added to the OVF
checker.
LP: #1737704
|
|
The exception was incorrectly creating a string and not a bytes object.
|
|
The instance identity document is a better source for region information,
partly because region isn't actually in meta-data at all, only
availability-zone, which happens to be named similarly.
Reviewed-by: Ethan Faust <efaust@amazon.com>
Reviewed-by: Cyle Riggs <cyler@amazon.com>
Reviewed-by: Tom Kirchner <tjk@amazon.com>
Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com>
[ajorgens@amazon.com: rebase onto 0.7.9]
[ajorgens@amazon.com: changes per merge proposal discussions]
|
|
Make sure that some temporary files used by the config drive tests get
cleaned up properly.
|
|
Previously the OVF transport would not be identified except for when
config files set 'ovf_vmware_guest_customization'. It would also
return DS_MAYBE almost always.
The change here is to add support to ds-identify for storing the
iso9660 filesystems that it finds (ISO9660_DEVS). Then the OVF check
will check that the iso9660 filesystem has ovf-env.xml on it. The least
wonderful part of this is that the check is done by 'grep' for case
insensitive ovf-env.xml.
Future improvement would be to identify VMware's OVF by label or UUID
so we could avoid the grep.
LP: #1731868
|
|
In the VMware customization workflow, we have some options for the user
to upload scripts for additional customization. Based on user request,
those custom scripts can be either run before regular customization or
after. For post customization scripts, we decide whether to run the scripts
just after customization or post system reboot.
|
|
The NoCloudKVM platform was inserting ssh keys via user-data
rather than through meta-data like it is done on other platforms.
This way we are not forced to change the user-data provided.
Also, provide meta-data including a uuid as the instance-id.
|
|
The motivation for this is that
a.) 1.7.1 runs with python 3.6 (bionic)
b.) we want to run pylint on tests/ and tools for the same reasons
that we want to run it on cloudinit/
The changes are described below.
- Update tox.ini to invoke pylint v1.7.1.
- Modify .pylintrc generated-members ignore mocked object members (m_.*)
- Replace "dangerous" params defaulting to {}
- Fix up cloud_tests use of platforms
- Cast some instance objects to with dict()
- Handle python2.7 vs 3+ ConfigParser use of readfp (deprecated)
- Update use of assertEqual(<boolean>, value) to assert<Boolean>(value)
- replace depricated assertRegexp -> assertRegex
- Remove useless test-class calls to super class
- Assign class property accessors a result and use it
- Fix missing class member in CepkoResultTests
- Fix Cheetah test import
|
|
Each DataSource subclass must define its own get_data method. This branch
formalizes our DataSource class to require that subclasses define an
explicit dsname for sourcing cloud-config datasource configuration.
Subclasses must also override the _get_data method or a
NotImplementedError is raised.
The branch also writes /run/cloud-init/instance-data.json. This file
contains all meta-data, user-data and vendor-data and a standardized set
of metadata keys in a json blob which other utilities with root-access
could make use of. Because some meta-data or user-data is potentially
sensitive the file is only readable by root.
Generally most metadata content types should be json serializable. If
specific keys or values are not serializable, those specific values will
be base64encoded and the key path will be listed under the top-level key
'base64-encoded-keys' in instance-data.json. If json writing fails due to
other TypeErrors or UnicodeDecodeErrors, a warning log will be emitted to
/var/log/cloud-init.log and no instance-data.json will be created.
|
|
The 'cloud-init clean' command allows a user or script to clear cloud-init
artifacts from the system so that cloud-init sees the system as
unconfigured upon reboot. Optional parameters can be provided to remove
cloud-init logs and reboot after clean.
The 'cloud-init status' command allows the user or script to check whether
cloud-init has finished all configuration stages and whether errors
occurred. An optional --wait argument will poll on a 0.25 second interval
until cloud-init configuration is complete. The benefit here is scripts
can block on cloud-init completion before performing post-config tasks.
|
|
This groups up each test platform into its own directory rather
than having files spread between four different directories for
one platform. Platforms tend to be worked on one at a time and
so having the platforms together makes more sense than apart.
|
|
There is a race condition where our sandboxed dhclient properly writes a
lease file but has not yet written a pid file. If the sandbox temporary
directory is torn down before the dhclient subprocess writes a pidfile
DataSourceEc2Local gets a traceback and the instance will fallback to
DataSourceEc2 in the init-network stage. This wastes boot cycles we'd
rather not spend.
Fix handling of sandboxed dhclient to wait for both pidfile and leasefile
before proceding. If either file doesn't show in 5 seconds, log a warning
and return empty lease results {}.
LP: #1735331
|
|
The NoCloudKVMImage.execute() would modify the image in /srv/citest
that meant that after the first time you ran a test, the image was
dirty.
The change here is to make the image operate on a qcow backed image.
Also modify Snapshot to then copy the qcow rather
than creating another chained qcow. The reason being that the image
might go away or change after the snapshot has been returned.
Also
* drop use of 'override_templates' which was only relevant to LXD.
* NoCloudKVM.create_image() returned an instance before
now it has create_instance which creates an instance.
* NoCloudKVMInstance has a 'disk' attribute separate from 'name'
|
|
|
|
Instead of using 'dpkg -i' to install a package and then running
apt-get -f install, to hope that it would install needed dependencies
we can just use 'apt-get' directly to do the install.
The 'dpkg/apt-get -f' path was a problem if the installed deb was
older than the available deb. In that case it would get replaced.
|
|
Currently when dns and dns search info is provided, it is not rendered
when outputting to sysconfig format.
This patch causes the DNS and DOMAIN lines to be written out rendering
sysconfig.
LP: #1705804
|
