| Age | Commit message (Collapse) | Author |
|---|
|
Handlers for per-boot/per-instance/per-once multipart MIME
Add handlers for adding scripts to userdata that can be run at various
frequencies. Scripts of type x-shellscript-per-boot,
x-shellscript-per-instance, or x-shellscript-per-once can be added
to a multipart MIME userdata message as part of instance userdata.
These scripts will then be added to the appropriate per-boot,
per-instance, or per-once directory in /var/lib/cloud/scripts/
during processing of userdata.
|
|
snapd currrently looks for 'not run' from cloud-init status[1].
Avoid changing this behavior and revert "not-run" value to
"not run".
This avoids having to get snapd to change implementation and
release updates as far back as Bionic to handle a hyphenated
not-run string.
[1]: https://github.com/snapcore/snapd/blob/master/sysconfig/\
cloudinit.go#L802
|
|
Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.
Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.
This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.
The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.
Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:
- get_schema: source base schema file from cloud-init-schema.json
and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param
Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI
|
|
This fix has two elements:
- cloud-init status will not correctly report 'not-run' prior to systemd
generator running. Only report "disabled" when generator has run
and /run/cloud-init/disabled exists.
- Expose not-run and disabled state in cloud-id responses
- Add unique error codes from cloud-id for error, disabled and not-run.
The new cloud-id exit codes:
0: success
1: error
2: cloud-init is in disabled state
3: cloud-init generator has not run yet
|
|
Applied Black and isort, fixed any linting issues, updated tox.ini
and CI.
|
|
Improve schema validation.
This adds strict validation of config module definitions at testing
time, with plumbing included for future runtime validation. This
eliminates a class of bugs resulting from schemas that have definitions
that are incorrect, but get interpreted by jsonschema as
"additionalProperties" that are therefore ignored.
- Add strict meta-schema for jsonschema unit test validation
- Separate schema from module metadata structure
- Improve type annotations for various functions and data types
Cleanup:
- Remove unused jsonschema "required" elements
- Eliminate manual memoization in schema.py:get_schema(),
reference module.__doc__ directly
|
|
This attempts to standardize unit test file location under test/unittests/
such that any source file located at cloudinit/path/to/file.py may have a
corresponding unit test file at test/unittests/path/to/test_file.py.
Noteworthy Comments:
====================
Four different duplicate test files existed:
test_{gpg,util,cc_mounts,cc_resolv_conf}.py
Each of these duplicate file pairs has been merged together. This is a
break in git history for these files.
The test suite appears to have a dependency on test order. Changing test
order causes some tests to fail. This should be rectified, but for now
some tests have been modified in
tests/unittests/config/test_set_passwords.py.
A helper class name starts with "Test" which causes pytest to try
executing it as a test case, which then throws warnings "due to Class
having __init__()". Silence by changing the name of the class.
# helpers.py is imported in many test files, import paths change
cloudinit/tests/helpers.py -> tests/unittests/helpers.py
# Move directories:
cloudinit/distros/tests -> tests/unittests/distros
cloudinit/cmd/devel/tests -> tests/unittests/cmd/devel
cloudinit/cmd/tests -> tests/unittests/cmd/
cloudinit/sources/helpers/tests -> tests/unittests/sources/helpers
cloudinit/sources/tests -> tests/unittests/sources
cloudinit/net/tests -> tests/unittests/net
cloudinit/config/tests -> tests/unittests/config
cloudinit/analyze/tests/ -> tests/unittests/analyze/
# Standardize tests already in tests/unittests/
test_datasource -> sources
test_distros -> distros
test_vmware -> sources/vmware
test_handler -> config # this contains cloudconfig module tests
test_runs -> runs
|
|
(#1123)
Allow #cloud-config and cloud-init query to use underscore-delimited
"jinja-safe" key aliases for any instance-data.json keys
containing jinja operator characters.
This provides a means to use Jinja's dot-notation instead of square brackets
and quoting to reference "unsafe" obtain attribute names.
Support for these aliased keys is available to both #cloud-config user-data and
`cloud-init query`.
For example #cloud-config alias access can look like:
{{ ds.config.user_network_config }}
- instead of -
{{ ds.config["user.network-config"] }}
|
|
This commit removes automatically installing udev rules for hotplug
and adds a module to install them instead.
Automatically including the udev rules and checking if hotplug was
enabled consumed too many resources in certain circumstances. Moving the
rules to a module ensures we don't spend extra extra cycles on hotplug
if hotplug functionality isn't desired.
LP: #1946003
|
|
In #919 (81299de), we refactored some of the code used to bring up
networks across distros. Previously, the call to bring up network
interfaces during 'init' stage unintentionally resulted in a no-op
such that network interfaces were NEVER brought up by cloud-init, even
if new network interfaces were found after crawling the metadata.
The code was altered to bring up these discovered network interfaces.
On ubuntu, this results in a 'netplan apply' call during 'init' stage
for any ubuntu-based distro on a datasource that has a NETWORK
dependency. On GCE, this additional 'netplan apply' conflicts with the
google-guest-agent service, resulting in an instance that can no
be connected to.
This commit adds a 'disable_network_activation' option that can be
enabled in /etc/cloud.cfg to disable the activation of network
interfaces in 'init' stage.
LP: #1938299
|
|
Using flake8 inplace of pyflakes
Renamed run-pyflakes -> run-flake8
Changed target name to flake8 in Makefile
With pyflakes we can't suppress warnings/errors in few required places.
flake8 is flexible in that regard. Hence using flake8 seems to be a
better choice here.
flake8 does the job of pep8 anyway.
So, removed pep8 target from Makefile along with tools/run-pep8 script.
Included setup.py in flake8 checks
|
|
Update "cloud-init collect-logs" to ignore
/run/cloud-init/hook-hotplug-cmd as this will raise the error
"/run/cloud-init/hook-hotplug-cmd` is a named pipe" if included.
Also updated logs.py to continue writing the tarball if it fails
collecting a file rather than let the exception bubble up.
LP: #1940235
|
|
Alters hotplug hook to have a query mechanism checking if the
functionality is enabled. This allows us to avoid using the hotplug
socket and service when hotplug is disabled.
|
|
In CI run against pylint 2.9.3 and fix occurrences of:
- W0237 (arguments-renamed)
- W0402 (deprecated-module)
The W0402 deprecated-module was about module `imp`:
cloudinit/patcher.py:9: [W0402(deprecated-module), ]
Uses of a deprecated module 'imp'
The imp module is deprecated and replaced by importlib, which according
to the documentation has no replacement for acquire_lock() and
release_lock(), which are the only reason why `imp` is imported.
Nothing about the code using this lock that actually requires it.
Let's remove the locking code and the import altogether.
Dropping the locking makes patcher.patch() an empty wrapper around
_patch_logging(). Rename _patch_logging() to patch_logging() and
call it directly instead. Drop patch().
|
|
Adds a udev script which will invoke a hotplug hook script on all net
add events. The script will write some udev arguments to a systemd FIFO
socket (to ensure we have only instance of cloud-init running at a
time), which is then read by a new service that calls a new 'cloud-init
devel hotplug-hook' command to handle the new event.
This hotplug-hook command will:
- Fetch the pickled datsource
- Verify that the hotplug event is supported/enabled
- Update the metadata for the datasource
- Ensure the hotplugged device exists within the datasource
- Apply the config change on the datasource metadata
- Bring up the new interface (or apply global network configuration)
- Save the updated metadata back to the pickle cache
Also scattered in some unrelated typing where helpful
|
|
Currently _bring_up_interfaces() is a no-op for any distro using
renderers. We need to be able to support bringing up a single
interfaces, a list of interfaces, and all interfaces. This should be
independent of the renderers, as the network config is often
generated independent of the mechanism used to apply it.
Additionally, I included a refactor to remove
"_supported_write_network_config". We had a confusing call chain of
apply_network_config->_write_network_config->_supported_write_network_config.
The last two have been combined.
|
|
summary: Clear cache when a Python version change is detected
When a distribution gets updated it is possible that the Python version
changes. Python makes no guarantee that pickle is consistent across
versions as such we need to purge the cache and start over.
Co-authored-by: James Falcon <therealfalcon@gmail.com>
|
|
Also added a new (currently experimental) systemd-networkd renderer,
and includes a small refactor to cc_resolv_conf.py to support the
resolved.conf used by systemd-resolved.
|
|
Add support for openstack's dynamic vendor data, which appears under openstack/latest/vendor_data2.json
This adds vendor_data2 to all pathways; it should be a no-op for non-OpenStack providers.
LP: #1841104
|
|
|
|
Fedora build system's rpmlint is complaining that there is a file with a
shebang but no executable flag set. No need to have shebang on this
file, so drop it.
Signed-off-by: Eduardo Otubo <otubo@redhat.com>
Co-authored-by: Scott Moser <smoser@brickies.net>
|
|
Changes:
tox: bump the pylint version to 2.6.0 in the default run
Fix pylint 2.6.0 W0707 warnings (raise-missing-from)
|
|
|
|
cloud-init query tries to directly load and decode
raw user-data from /var/lib/cloud/instance/user-data.txt.
This results in UnicodeDecodeErrors on some platforms which
provide compressed content.
Avoid UnicodeDecoderErrors when parsing compressed user-data at
/var/lib/cloud/instance/user-data.txt.
LP: #1889938
|
|
* cli: add devel make-mime subcommand
Cloud-init documents an in-source-tree tool, make-mime.py used to
help users create multi-part mime user-data. This tool is not shipped
in the cloud-init install and unavailable at runtime. This patch
takes tools/make-mime.py and makes the functionality available via
the devel subcommand.
The primary interface of --attach file:content-type is still present.
The cli now adds:
-l, --list-types Print out a list of supported content-types
-f, --force Ignore errors for unsupported content-types
The tool will now raise a RunTime error if the supplied content-type
is not supported (or more likely a typo:
x-shell-script vs. x-shellscript)
* make-mime: write to stderr and exit 1 instead of raising RuntimeError
* Update example to match docs
* Update docs for make-mime subcommand
* Remove tools/make-mime.py; replaced by cloud-init devel make-mime
Co-authored-by: Rick Harding <rharding@mitechie.com>
|
|
|
|
This was painful, but it finishes a TODO from cloudinit/subp.py.
It moves the following from util to subp:
ProcessExecutionError
subp
which
target_path
I moved subp_blob_in_tempfile into cc_chef, which is its only caller.
That saved us from having to deal with it using write_file
and temp_utils from subp (which does not import any cloudinit things now).
It is arguable that 'target_path' could be moved to a 'path_utils' or
something, but in order to use it from subp and also from utils,
we had to get it out of utils.
|
|
Remove extra spaces after a ','
|
|
This shim was required to support Python 2.6, so we no longer need it.
|
|
These classes don't use `self.logs` anywhere in their body, so we can
remove the `with_logs = True` setting from them.
These instances were found using astpath[0], with the following
invocation:
astpath "//Name[@id='with_logs' and not(ancestor::ClassDef//Attribute[@attr='logs'])]"
[0] https://github.com/hchasestevens/astpath
|
|
|
|
|
|
* url_helper: drop six
* url_helper: sort imports
* log: drop six
* log: sort imports
* handlers/__init__: drop six
* handlers/__init__: sort imports
* user_data: drop six
* user_data: sort imports
* sources/__init__: drop six
* sources/__init__: sort imports
* DataSourceOVF: drop six
* DataSourceOVF: sort imports
* sources/helpers/openstack: drop six
* sources/helpers/openstack: sort imports
* mergers/m_str: drop six
This also allowed simplification of the logic, as we will never
encounter a non-string text type.
* type_utils: drop six
* mergers/m_dict: drop six
* mergers/m_list: drop six
* cmd/query: drop six
* mergers/__init__: drop six
* net/cmdline: drop six
* reporting/handlers: drop six
* reporting/handlers: sort imports
|
|
The query command checks the user's uid when running and takes two
different code paths. As a normal user is returns fake data, that these
tests were expecting. As a root user, the actual user and vendor data
files are ready.
LP: #1856096
|
|
netplan introduced an 'info' subcommand which emits yaml describing
implemented features that indicate new or changed fields and values
in the yaml that it accepts. Previously, cloud-init emitted the key
'mtu6' for ipv6 MTU values. This is not correct and netplan will
fail to parse these values. Netplan as of 0.98 supports both the
info subcommand and the ipv6-mtu key.
This branch modifies the netplan renderer to collect the netplan
info output into a 'features' property which is a list of available
feature flags which the renderer can use to modify its output. If
the command is not available, no feature flags are set and
cloud-init will render IPv6 MTU values just as MTU for the subnet.
|
|
Here we replace uses of the pyyaml module directly with functions
provided by cloudinit.safeyaml. Also, change/move
cloudinit.util.yaml_dumps
to
cloudinit.safeyaml.dumps
LP: #1849640
|
|
Cloud-init's main.py will fail when presented with a new
stage name 'modules-init' if upgrading an older cloud-init.
Fix this by initializing unknown stage names before accessing.
LP: #1815109
|
|
Previously, init.paths.cloud_dir has a trailing slash, which meant that
"/var/lib/cloud//seed" was being compared to "/var/lib/cloud/seed" and
(of course), never matching.
In this commit, switch to using os.path.join to avoid this case (and
update the tests to catch it in future).
LP: #1818571
|
|
Avoid traceback when cloud-init clean is run from within
/var/lib/cloud/ deleted dirs.
LP: #1795508
|
|
I noticed a bug in dhclient_hook on the 'down' event, using 'is'
operator rather than '==' (if self.net_action is 'down').
This refactors/simplifies the code a bit for easier testing and adds
tests. The reason for the rename of 'action' to 'event' is to just
be internally consistent. The word and Namespace 'action' is used
by cloud-init main, so it was not really usable here.
Also adds a main which can easily be debugged with:
CI_DHCP_HOOK_DATA_D=./my.d python -m cloudinit.dhclient_hook up eth0
|
|
Move routes under the nic's subnet rather than use top-level
("global") route config ensuring all net renderers will provide the
configured route.
Also updated cloudinit/cmd/devel/net_convert.py:
- Add input type 'vmware-imc' for OVF customization config files
- Fix bug when output-type was netplan which invoked netplan
generate/apply and attempted to write to
/etc/netplan/50-cloud-init.yaml instead of joining with the
output directory.
LP: #1806103
|
|
Since /run/cloud-init/instance-data-sensitive.json is root read-only,
ignore this file if non-root user runs collect-logs.
If --include-userdata is provided on the command line, exit in error
if non-root user attempts this operation.
Lastly, update the __main__ to exit based on return value of main.
LP: #1805201
|
|
Emit a permissions error instead of "Missing instance-data.json" when
non-root user doesn't have read-permission on
/run/cloud-init/instance-data.json
|
|
On cloud-init upgrade path from 18.3 to 18.4 cloud-init changed how
instance-data is written. Cloud-init changes instance-data.json from root
read-only to redacted world-readable content, and provided a separate
unredacted instance-data-sensitive.json which is read-only root.
Since instance-data is only rewritten from cache on
reboot, the query and render tools needed fallback to use the 'old'
instance-data.json if the new sensitive file isn't yet present.
This avoids error messages from tools about an absebt
/run/instance-data-sensitive.json file.
LP: #1798189
|
|
Add a quick cloud lookup utility in order to more easily determine
the cloud on which an instance is running.
The utility parses standardized attributes from
/run/cloud-init/instance-data.json to print the canonical cloud-id
for the instance. It uses known region maps if necessary to determine
on which specific cloud the instance is running.
Examples:
aws, aws-gov, aws-china, rackspace, azure-china, lxd, openstack, unknown
|
|
Cloud-init caches any cloud metadata crawled during boot in the file
/run/cloud-init/instance-data.json. Cloud-init also standardizes some of
that metadata across all clouds. The command 'cloud-init query' surfaces a
simple CLI to query or format any cached instance metadata so that scripts
or end-users do not have to write tools to crawl metadata themselves.
Since 'cloud-init query' is runnable by non-root users, redact any
sensitive data from instance-data.json and provide a root-readable
unredacted instance-data-sensitive.json. Datasources can now define a
sensitive_metadata_keys tuple which will redact any matching keys
which could contain passwords or credentials from instance-data.json.
Also add the following standardized 'v1' instance-data.json keys:
- user_data: The base64encoded user-data provided at instance launch
- vendor_data: Any vendor_data provided to the instance at launch
- underscore_delimited versions of existing hyphenated keys:
instance_id, local_hostname, availability_zone, cloud_name
|
|
If the user has removed the default configuration file or does
not set the syslog_fix_perms config option the user still ends
up with a warning on SUSE distributions. Add root:root to the
default builtin config.
|
|
Allow users to provide '## template: jinja' as the first line or their
#cloud-config or custom script user-data parts. When this header exists,
the cloud-config or script will be rendered as a jinja template.
All instance metadata keys and values present in
/run/cloud-init/instance-data.json will be available as jinja variables
for the template. This means any cloud-config module or script can
reference any standardized instance data in templates and scripts.
Additionally, any standardized instance-data.json keys scoped below a
'<v#>' key will be promoted as a top-level key for ease of reference in
templates. This means that '{{ local_hostname }}' is the same as using the
latest '{{ v#.local_hostname }}'.
Since instance-data is written to /run/cloud-init/instance-data.json, make
sure it is persisted across reboots when the cached datasource opject is
reloaded.
LP: #1791781
|
|
In many cases, cloud-init uses 'util.subp' to run a subprocess.
This is not really desirable in our unit tests as it makes the tests
dependent upon existance of those utilities.
The change here is to modify the base test case class (CiTestCase) to
raise exception any time subp is called. Then, fix all callers.
For cases where subp is necessary or actually desired, we can use it
via
a.) context hander CiTestCase.allow_subp(value)
b.) class level self.allowed_subp = value
Both cases the value is a list of acceptable executable names that
will be called (essentially argv[0]).
Some cleanups in AltCloud were done as the code was being updated.
|
|
Multiple distros use sysconfig format but have different content
and paths to certain files. Update distros to specify these
template paths in their renderer_configs dictionary.
|
