| Age | Commit message (Collapse) | Author |
|---|
|
This was painful, but it finishes a TODO from cloudinit/subp.py.
It moves the following from util to subp:
ProcessExecutionError
subp
which
target_path
I moved subp_blob_in_tempfile into cc_chef, which is its only caller.
That saved us from having to deal with it using write_file
and temp_utils from subp (which does not import any cloudinit things now).
It is arguable that 'target_path' could be moved to a 'path_utils' or
something, but in order to use it from subp and also from utils,
we had to get it out of utils.
|
|
Build time feature flags are now defined in cloudinit/features.py.
Feature flags can be added to toggle configuration options or
deprecated features. Feature flag overrides can be placed in
cloudinit/feature_overrides.py. Further documentation can be found in
HACKING.rst.
Additionally, updated default behavior to exit with an exception
if #include can't retrieve resources as expected. This behavior
can be toggled with a feature flag.
LP: #1734939
|
|
Improving the debugability of this code path by logging the thrown exception details for the non 404 exceptions.
Retry IMDS on HTTP Error 404 and 410, re-run DHCP on other exceptions.
|
|
This fixes issues with closing brackets not matching the opening
bracket's line and continuation line under-idented for hanging indent.
|
|
Remove extra spaces after a ','
|
|
This removes the use of variables named ‘l’, ‘O’, or ‘I’. Generally
these are used in list comprehension to read the line of lines.
|
|
|
|
Co-authored-by: Daniel Watkins <oddbloke@ubuntu.com>
|
|
We live in the future now.
|
|
This ensures that Travis will not kill our tests if fetching images is
taking a long time.
In implementation terms, this introduces a context manager which will
spin up a multiprocessing.Process in the background and print a dot to
stdout every 10 seconds. The process is terminated when the context
manager exits.
This also drop the use of travis_wait, which was being used to work
around this issue.
|
|
Create a schema object for the `apt_configure` module and
validate this schema in the `handle` function of the module.
There are some considerations regarding this PR:
* The `primary` and `security` keys have the exact same properties. I
tried to eliminate this redundancy by moving their properties to a
common place and then just referencing it for both security and
primary. Similar to what is documented here:
https://json-schema.org/understanding-json-schema/structuring.html
under the `Reuse` paragraph. However, this approach does not work,
because the `#` pointer goes to the beginning of the file, which is
a python module instead of a json file, not allowing the pointer to
find the correct definition. What I did was to create a separate dict
for the mirror config and reuse it for primary and security, but
maybe there are better approaches to do that.
* There was no documentation for the config `debconf_selections`. I
tried to infer what it supposed to do by looking at the code and the
`debconf-set-selections` manpage, but my description may not be
accurate or complete.
* Add a _parse_description function to schema.py to render multi-line
preformatted content instead of squashing all whitespace
LP: #1858884
|
|
Also update all examples to include the cloud-config header if they don't have it
LP: #1876414
|
|
Move from 127.0.0.1 to 127.0.1.1 for localhost IP addr for opensuse and sles
|
|
Add a unit test to validate if the examples provided in the config
modules are conforming to the concatenated schema of all config
modules. The rationale behind that is not only to verify if the
examples are correctly written but to assert that no config schema
is interfering with each other.
Failures in validate_cloudconfig_schema raise the
SchemaValidationError by using strict=True, so I have
only called the function passing the right schema examples to
validate.
This branch also fixes an invalid schema example in cc_snap.
LP: #1876412
|
|
Implement the upgrade support:
- FreeBSD: using `pkg upgrade`
- NetBSD: with `pkgin`
|
|
We want to set route-metrics such that NICs are configured with the priority that they are given in the network metadata that we receive from the IMDS. (This switches away from using MAC ordering.)
This also required the following test changes:
* reverse the sort order of the MACs in test data (so that they would trigger the bug being fixed)
* fix up the key names in `NIC2_MD` (which were under_scored instead of dash-separated)
* use a full interface dict (rather than a minimal one) for `TestConvertEc2MetadataNetworkConfig`
LP: #1876312
|
|
|
|
Add schema definition to cc_write_files.py
Cloud-config containing write_files config directives will now
emit warnings for invalid config keys or values for the write_files
module.
Add an extension to JSON schema's draft4validator to permit either
binary or text values for 'string' objects.
This allows for JSON schema validating the YAML declaration of binary
valiues in cloud-config using YAML's '!!binary' syntax.
Add the ability to pass a specific module name to
`cloud-init devel schema --docs <module_name>|all` to optionally
limit doc output during development to a single schema doc.
|
|
Refactoring of the `find_devs_with_*bsd()` methods:
- centralize everything in `util.py`
- add test coverage
|
|
These libraries provide backports of Python 3's stdlib components to Python 2. As we only support Python 3, we can simply use the stdlib now. This pull request does the following:
* removes some unneeded compatibility code for the old spelling of `assertRaisesRegex`
* replaces invocations of the Python 2-only `assertItemsEqual` with its new name, `assertCountEqual`
* replaces all usage of `unittest2` with `unittest`
* replaces all usage of `contextlib2` with `contextlib`
* drops `unittest2` and `contextlib2` from requirements files and tox.ini
It also rewrites some `test_azure` helpers to use bare asserts. We were seeing a strange error in xenial builds of this branch which appear to be stemming from the AssertionError that pytest produces being _different_ from the standard AssertionError. This means that the modified helpers weren't behaving correctly, because they weren't catching AssertionErrors as one would expect. (I believe this is related, in some way, to https://github.com/pytest-dev/pytest/issues/645, but the only version of pytest where we're affected is so far in the past that it's not worth pursuing it any further as we have a workaround.)
|
|
We currently have a test system where get_interfaces_by_mac raises an
exception, which is causing these tests to fail as they aren't mocking
get_interfaces_by_mac out.
LP: #1873910
|
|
The cc_mounts module does not support NFS mounts in the form of hostname:/ or hostname:/path.
This PR adds support for NFS-style paths in the fs_spec field.
LP: #1870370
|
|
This shim was required to support Python 2.6, so we no longer need it.
|
|
|
|
On some platforms (old heat on OpenTelekomCloud), the user-data mime part
is mislabeled x-shellscript. cloud-init would not accept this unexpected
mime-type in multipart user-data. Cloud-init will now run find_ctype()
on the content of the mime-part to check if it matches known include types.
This patch is from Ryan Harper (inspired by my bug report and a
suggestion from Scott Moser) from the discussion on PR #234.
Signed-off-by: Kurt Garloff <kurt@garloff.de>
|
|
On Focal and later, Ubuntu will prioritize netplan renderer over eni,
even if ifupdown and netplan are both installed.
ENI on Focal and later is considered an unsupported configuration so
cloud-init should generally prefer netplan. On many cloud images,
the /etc/network/interfaces config file does not include the dir
/etc/network/interfaces.d thereby ignoring cloud-init's
/etc/network/interfaces.d/50-cloud-init.cfg file.
LP: #1867029
|
|
(And sort some imports where I was changing them.)
|
|
add SAP Converged Cloud as cloud provider
|
|
- tested on OpenBSD 6.6
- tested on OpenStack without config drive, and NoCloud with ISO config
drive
|
|
|
|
These two implementations had drifted away from one another very
slightly. Reconcile them and then remove the one in cc_apt_configure.
|
|
These classes don't use `self.logs` anywhere in their body, so we can
remove the `with_logs = True` setting from them.
These instances were found using astpath[0], with the following
invocation:
astpath "//Name[@id='with_logs' and not(ancestor::ClassDef//Attribute[@attr='logs'])]"
[0] https://github.com/hchasestevens/astpath
|
|
Add support for rendering secondary static IPv4/IPv6 addresses on
any NIC attached to the machine. In order to see secondary IP
addresses in Ec2 IMDS network config, cloud-init now reads metadata
version 2018-09-24. Metadata services which do not support the Ec2
API version will not get secondary IP addresses configured.
In order to discover secondary IP address config, cloud-init now
relies on metadata API Parse local-ipv4s, ipv6s,
subnet-ipv4-cidr-block and subnet-ipv6-cidr-block metadata keys to
determine additional IPs and appropriate subnet prefix to set for a
nic.
Also add the datasource config option apply_full_imds_netork_config
which defaults to true to allow cloud-init to automatically configure
secondary IP addresses. Setting this option to false will tell
cloud-init to avoid setting up secondary IP addresses.
Also in this branch:
- Shift Ec2 datasource to emit network config v2 instead of v1.
LP: #1866930
|
|
The 'merged_cfg' key introduced in 71af48d was incorrectly referenced
to as 'ci_cfg' in the json validation test for ec2.
|
|
Quote the Ubuntu version numbers in releases.yaml to make sure they're
treated as strings and not as floats.
|
|
Add support for the NetBSD Operating System.
Features in this branch:
* Add BSD distro parent class from which NetBSD and FreeBSD can
specialize
* Add *bsd util functions to cloudinit.net and cloudinit.net.bsd_utils
* subclass cloudinit.distro.freebsd.Distro from bsd.Distro
* Add new cloudinit.distro.netbsd and cloudinit.net.renderer for
netbsd
* Add lru_cached util.is_NetBSD functions
* Add NetBSD detection for ConfigDrive and NoCloud datasources
This branch has been tested with:
- NoCloud and OpenStack (with and without config-drive)
- NetBSD 8.1. and 9.0
- FreeBSD 11.2 and 12.1
- Python 3.7 only, because of the dependency oncrypt.METHOD_BLOWFISH.
This version is available in NetBSD 7, 8 and 9 anyway
|
|
Cloud-config userdata provided as jinja templates are now distro,
platform and merged cloud config aware. The cloud-init query command
will also surface this config data.
Now users can selectively render portions of cloud-config based on:
* distro name, version, release
* python version
* merged cloud config values
* machine platform
* kernel
To support template handling of this config, add new top-level
keys to /run/cloud-init/instance-data.json.
The new 'merged_cfg' key represents merged cloud config from
/etc/cloud/cloud.cfg and /etc/cloud/cloud.cfg.d/*.
The new 'sys_info' key which captures distro and platform
info from cloudinit.util.system_info.
Cloud config userdata templates can render conditional content
based on these additional environmental checks such as the following
simple example:
```
## template: jinja
#cloud-config
runcmd:
{% if distro == 'opensuse' %}
- sh /custom-setup-sles
{% elif distro == 'centos' %}
- sh /custom-setup-centos
{% elif distro == 'debian' %}
- sh /custom-setup-debian
{% endif %}
```
To see all values: sudo cloud-init query --all
Any keys added to the standardized v1 keys are guaranteed to not
change or drop on future released of cloud-init. 'v1' keys will be retained
for backward-compatibility even if a new standardized 'v2' set of keys
are introduced
The following standardized v1 keys are added:
* distro, distro_release, distro_version, kernel_version, machine,
python_version, system_platform, variant
LP: #1865969
|
|
The EC2 Data Source needs to handle 3 states of the Instance
Metadata Service configured for a given instance:
1. HttpTokens : optional & HttpEndpoint : enabled
Either IMDSv2 or IMDSv1 can be used.
2. HttpTokens : required & HttpEndpoint : enabled
Calls to IMDS without a valid token (IMDSv1 or IMDSv2 with expired token)
will return a 401 error.
3. HttpEndpoint : disabled
The IMDS http endpoint will return a 403 error.
Previous work to support IMDSv2 in cloud-init handled case 1 and case 2.
This commit handles case 3 by bypassing the retry block when IMDS returns HTTP
status code >= 400 on official AWS cloud platform.
It shaves 2 minutes when rebooting an instance that has its IMDS http token endpoint
disabled but creates some inconsistencies. An instance that doesn't set
"manual_cache_clean" to "True" will have its /var/lib/cloud/instance symlink
removed altogether after it has failed to find a datasource.
|
|
Allow disabling cloud-init's network configuration via a plain-text kernel cmdline
Cloud-init docs indicate that users can disable cloud-init networking via kernel
command line parameter 'network-config=<YAML>'. This does not work unless
the <YAML> payload base64 encoded. Document the base64 encoding
requirement and add a plain-text value for disabling cloud-init network config:
network-config=disabled
Also:
- Log an error and ignore any plain-text network-config payloads that are
not specifically 'network-config=disabled'.
- Log a warning if network-config kernel param is invalid yaml but do not
raise an exception, allowing boot to continue and use fallback networking.
LP: #1862702
|
|
|
|
Instead of logging the token values used log the headers and replace the actual
values with the string 'REDACTED'. This allows users to examine cloud-init.log
and see that the IMDSv2 token header is being used but avoids leaving the value
used in the log file itself.
LP: #1863943
|
|
The azurecloud platform did not always start instances
during collect runs. This was a result of two issues. First
the image class _instance method did not invoke the start()
method which then allowed collect stage to attempt to run
scripts without an endpoint. Second, azurecloud used the
image_id as both an instance handle (which is typically
vmName in azure api) as well as an image handle (for image
capture). Resolve this by adding a .vm_name property to
the AzureCloudInstance and reference this property in
AzureCloudImage.
Also in this branch
- Fix error encoding user-data when value is None
- Add additional logging in AzureCloud platform
- Update logging format to print pathname,funcName and line number
This greatly eases debugging.
LP: #1861921
|
|
|
|
- Introduce the "flavor" configuration option for the sysconfig renderer
this is necessary to account for differences in the handling of the
BOOTPROTO setting between distributions (lp#1858808)
+ Thanks to Petr Pavlu for the idea
- Network config clean up for sysconfig renderer
+ The introduction of the "flavor" renderer configuration allows us
to only write values that are pertinent for the given distro
- Set the DHCPv6 client mode on SUSE (lp#1800854)
Co-authored-by: Chad Smith <chad.smith@canonical.com>
LP: #1800854
|
|
|
|
It is proto 'none', not 'static' as was mistakenly implemented in
initramfs-tools/cloud-init in the past, yet was never the case in the
klibc ipconfig state file output.
LP: #1861412
|
|
* cloudinit: replace "import mock" with "from unittest import mock"
* test-requirements.txt: drop mock
Co-authored-by: Chad Smith <chad.smith@canonical.com>
|
|
Make sure network_config is created when self._network_config is unset.
Co-authored-by: Scott Moser <smoser@brickies.net>
|
|
We only run on Python 3 now, so we can unambiguously expect
unittest.mock to exist.
|
|
When creating a swap file on an xfs filesystem, fallocate cannot be used.
Doing so results in failure of swapon and a message like:
swapon: swapfile has holes
The solution here is to maintain a list (currently containing only XFS)
of filesystems where fallocate cannot be used. The, on those fileystems
use the slower but functional 'dd' method.
Signed-off-by: Eduardo Otubo <otubo@redhat.com>
Co-authored-by: Adam Dobrawy <naczelnik@jawnosc.tk>
Co-authored-by: Scott Moser <smoser@brickies.net>
Co-authored-by: Daniel Watkins <daniel@daniel-watkins.co.uk>
LP: #1781781
|
