| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
We were seeing issues where if anything showed up before the
expected first adapter, booting could fail. This switches to seeking
for a working interface to handle edge cases.
Also fixes region code handling.
|
|
- Update EphemeralDHCPv4WithReporting to subclass EphemeralDHCPv4 for
consistency (non-functional change).
- Replace all usage of EphemeralDHCPv4 with EphemeralDHCPv4WithReporting.
- Converging to one DHCP class exposed an issue with ExitStack patches
being mixed with decorators. Specifically, it appeared that tests
that did not enable azure.EphemeralDHCPv4WithReporting mocks had it
applied anyways from previous tests.
Presumably ExitStack was overwriting the actual value with the
mock provided by the decorator? For now, remove some mock patches
that trigger failures, but future work should move towards a
consistent approach to prevent undetected effects.
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
Applied Black and isort, fixed any linting issues, updated tox.ini
and CI.
|
|
This attempts to standardize unit test file location under test/unittests/
such that any source file located at cloudinit/path/to/file.py may have a
corresponding unit test file at test/unittests/path/to/test_file.py.
Noteworthy Comments:
====================
Four different duplicate test files existed:
test_{gpg,util,cc_mounts,cc_resolv_conf}.py
Each of these duplicate file pairs has been merged together. This is a
break in git history for these files.
The test suite appears to have a dependency on test order. Changing test
order causes some tests to fail. This should be rectified, but for now
some tests have been modified in
tests/unittests/config/test_set_passwords.py.
A helper class name starts with "Test" which causes pytest to try
executing it as a test case, which then throws warnings "due to Class
having __init__()". Silence by changing the name of the class.
# helpers.py is imported in many test files, import paths change
cloudinit/tests/helpers.py -> tests/unittests/helpers.py
# Move directories:
cloudinit/distros/tests -> tests/unittests/distros
cloudinit/cmd/devel/tests -> tests/unittests/cmd/devel
cloudinit/cmd/tests -> tests/unittests/cmd/
cloudinit/sources/helpers/tests -> tests/unittests/sources/helpers
cloudinit/sources/tests -> tests/unittests/sources
cloudinit/net/tests -> tests/unittests/net
cloudinit/config/tests -> tests/unittests/config
cloudinit/analyze/tests/ -> tests/unittests/analyze/
# Standardize tests already in tests/unittests/
test_datasource -> sources
test_distros -> distros
test_vmware -> sources/vmware
test_handler -> config # this contains cloudconfig module tests
test_runs -> runs
|
|
Vultr uses 169.254.169.254 for the metadata server. Some distros are
having trouble with this on IPv6 only servers because the route is
not being assigned to the link-local interface by default as it is in
other distros. This change sets that route before attempting to fetch
the metadata avoiding the current issue.
|
|
For Debian, the network configure file was named
/etc/network/interfaces.d/50-cloud-init, not the 50-cloud-init.cfg,
related to
https://github.com/canonical/cloud-init/blob/62721ae71057530e41779ff02ce578b7b802a60f/cloudinit/distros/debian.py#L56
the static IP customization on Debian will fail owing to
"source /etc/network/interfaces.d/*.cfg".
This change will fix this issue.
LP: #1950136
|
|
Add MTU, accept-ra, routes, options and a direct way to provide intact
cloud configs for networking opposed to relying on configurations that
may need changed often.
|
|
Offload Vultr's vendordata assembly to the backend, correct vendordata
storage and parsing, allow passing critical data via the useragent,
better networking configuration for additional interfaces.
|
|
Add connectivity_url to Oracle's EphemeralDHCPv4
On bionic, when trying to bring up the EphemeralDHCPv4, it's possible
that we already have a route defined, which will result in an error when
trying to add the DHCP route. Use the connectivity_url to check if we
can reach the metadata service, and if so, skip the EphemeralDHCPv4.
The has_url_connectivity function has also been modified to take
a dict of kwargs to send to readurl.
LP: #1939603
|
|
Azure Linux Agent (WaLinuxAgent) waits for the ovf-env.xml file
to be written by cloud-init when cloud-init provisions the VM. This
file is written whenever cloud-init reads its contents from the
provisioning ISO.
With this change, when there is no provisioning ISO,
DataSourceAzure will generate the ovf-env.xml file based on the
metadata obtained from Azure IMDS.
|
|
Due to hyper-v implementations, iso ejection is more efficient if performed
from within the guest. The code will attempt to perform a best-effort ejection.
Failure during ejection will not prevent reporting ready from happening. If iso
ejection is successful, later iso ejection from the platform will be a no-op.
In the event the iso ejection from the guest fails, iso ejection will still happen at
the platform level.
|
|
This PR adds in support so that cloud-init can run on instances
deployed on Vultr cloud. This was originally brought up in #628.
Co-authored-by: Eric Benner <ebenner@vultr.com>
|
|
Ensure that the Azure helper's http handler sleeps a fixed duration
between retry failure attempts. The http handler will sleep a fixed
duration between failed attempts regardless of whether the attempt
failed due to (1) request timing out or (2) instant failure (no
timeout).
Due to certain platform issues, the http request to the Azure endpoint
may instantly fail without reaching the http timeout duration. Without
sleeping a fixed duration in between retry attempts, the http handler
will loop through the max retry attempts quickly. This causes the
communication between cloud-init and the Azure platform to be less
resilient due to the short total duration if there is no sleep in
between retries.
|
|
`get_interfaces` is used to in two ways, broadly: firstly, to determine
the available interfaces when converting cloud network configuration
formats to cloud-init's network configuration formats; and, secondly, to
ensure that any interfaces which are specified in network configuration
are (a) available, and (b) named correctly. The first of these is
unaffected by this commit, as no clouds support Open vSwitch
configuration in their network configuration formats.
For the second, we check that MAC addresses of physical devices are
unique. In some OVS configurations, there are OVS-created devices which
have duplicate MAC addresses, either with each other or with physical
devices. As these interfaces are created by OVS, we can be confident
that (a) they will be available when appropriate, and (b) that OVS will
name them correctly. As such, this commit excludes any OVS-internal
interfaces from the set of interfaces returned by `get_interfaces`.
LP: #1912844
|
|
New datasource utilizing UpCloud metadata API, including relevant unit
tests and documentation.
|
|
Add support for openstack's dynamic vendor data, which appears under openstack/latest/vendor_data2.json
This adds vendor_data2 to all pathways; it should be a no-op for non-OpenStack providers.
LP: #1841104
|
|
This feature will modify VMware datasource to read from meta data and user data which are specified by VMware vSphere user. If meta data/user data are found in cloud-init configuration directory, datasource will parse the meta data/network and user data from the configuration file, otherwise it will continue to parse them from traditional customization configuration file as before. The supported meta data file is in json or yaml format.
|
|
Adds the ability to run the Azure preprovisioned VMs as NIC-less and
then hot-attach them when assigned for reprovision.
The NIC on the preprovisioned VM is hot-detached as soon as it reports
ready and goes into wait for one or more interfaces to be hot-attached.
Once they are attached, cloud-init gets the expected number of NICs (in
case there are more than one) that will be attached from IMDS and waits
until all of them are attached. After all the NICs are attached,
reprovision proceeds as usual.
|
|
Pushing dmesg log to KVP to help troubleshoot VM boot issues
|
|
Increase Azure Endpoint HTTP retries to handle
occasional platform network blips.
Introduce a common method http_with_retries
in the azure.py helper, which will serve as
the common HTTP request handler for
all HTTP requests with the Azure endpoint.
This method has builtin retries and
reporting diagnostics logic.
|
|
On systems where the Azure datasource
is a viable platform for crawling metadata,
cloud-init occasionally encounters fatal
irrecoverable errors during the crawling
of the Azure datasource.
When this happens, cloud-init crashes,
and Azure VM provisioning would fail.
However, instead of failing immediately,
the user will continue seeing provisioning
for a long time until it times out with
"OS Provisioning Timed Out" message.
In these situations, cloud-init should
report failure to the Azure datasource
endpoint indicating provisioning failure.
The user will immediately see provisioning
terminate, giving them a much better
failure experience instead of pointlessly
waiting for OS provisioning timeout.
|
|
This allows the cloud-init log to be pushed multiple times during boot,
with the latest lines being pushed each time.
|
|
* Include both Ubuntu-style cfg file, and Debian-style directory in ENI
* Add WebSpider as contributor
|
|
This just separates the reading of dmi values into its own file.
Some things of note:
* left import of util in dmi.py only for 'is_container'
It'd be good if is_container was not in util.
* just the use of 'util.is_x86' to dmi.py
* open() is used directly rather than load_file.
|
|
The static and static6 subnet types for network_data.json were
being ignored by the Openstack handler, this would cause the code to
break and not function properly.
As of today, if a static6 configuration is chosen, the interface will
still eventually be available to receive router advertisements or be set
from NetworkManager to wait for them and cycle the interface in negative
case.
It is safe to assume that if the interface is manually configured to use
static ipv6 address, there's no need to wait for router advertisements.
This patch will set automatically IPV6_AUTOCONF and IPV6_FORCE_ACCEPT_RA
both to "no" in this case.
This patch fixes the specific behavior only for RHEL flavor and
sysconfig renderer. It also introduces new unit tests for the specific
case as well as adjusts some existent tests to be compatible with the
new options. This patch also addresses this problem by assigning the
appropriate subnet type for each case on the openstack handler.
rhbz: #1889635
rhbz: #1889635
Signed-off-by: Eduardo Otubo otubo@redhat.com
|
|
This moves logging into `report_diagnostic_event`, to clean up its callsites.
|
|
Co-authored-by: Rick Harding <rharding@mitechie.com>
|
|
* pull ssh keys from imds first and fall back to ovf if unavailable
* refactor log and diagnostic messages
* refactor the OpenSSLManager instantiation and certificate usage
* fix unit test where exception was being silenced for generate cert
* fix tests now that certificate is not always generated
* add documentation for ssh key retrieval
* add ability to check if http client has security enabled
* refactor certificate logic to GoalState
|
|
This reverts commit 8d25d5e6fac39ab3319ec5d37d23196429fb0c95.
|
|
Changes:
tox: bump the pylint version to 2.6.0 in the default run
Fix pylint 2.6.0 W0707 warnings (raise-missing-from)
|
|
Push the cloud-init.log file (Up to 500KB at once) to the KVP before reporting ready to the Azure platform.
Based on the analysis done on a large sample of cloud-init.log files, Here's the statistics collected on the log file size:
P50 P90 P95 P99 P99.9 P99.99
137K 423K 537K 3.5MB 6MB 16MB
This change limits the size of cloud-init.log file data that gets dumped to KVP to 500KB. So for ~95% of the cases, the whole log file will be dumped and for the remaining ~5%, we will get the last 500KB of the cloud-init.log file.
To asses the performance of the 500KB limit, 250 VM were deployed with a 500KB cloud-init.log file and the time taken to compress, encode and dump the entries to KVP was measured. Here's the time in milliseconds percentiles:
P50 P99 P999
75.705 232.701 1169.636
Another 250 VMs were deployed with this logic dumping their normal cloud-init.log file to KVP, the same timing was measured as above. Here's the time in milliseconds percentiles:
P50 P99 P999
1.88 5.277 6.992
Added excluded_handlers to the report_event function to be able to opt-out from reporting the events of the compressed cloud-init.log file to the cloud-init.log file.
The KVP break_down logic had a bug, where it will reuse the same key for all the split chunks of KVP which results in overwriting the split KVPs by the last one when consumed by Hyper-V. I added the split chunk index as a differentiator to the KVP key.
The Hyper-V consumes the KVPs from the KVP file as chunks whose key is 512KB and value is 2048KB but the Azure platform expects the value to be 1024KB, thus I introduced the Azure value limit.
|
|
This PR refactors Azure report ready code to include more robust tests and telemetry.
|
|
Add support for VMware's vCD configuration setting DEFAULT-RUN-POST-CUST-SCRIPT.
When set True, it will default vms to run post customization scripts if the VM has not been configured in VMTools with "enable-custom-scripts" set False.
Add datasource documentation with a bit more context about this interaction on VMware products.
With this fix, the behavior will be:
* If VM administrator doesn't want others to execute a script on this VM, VMtools can set "enable-custom-scripts" to false from the utility "vmware-toolbox-cmd".
* If VM administrator doesn't set value to "enable-custom-scripts", then by default this script is disabled for security purpose.
* For VMware's vCD product , the preference is to enable the script if "enable-custom-scripts" is not set. vCD will generate a configuration file with "DEFAULT-RUN-POST-CUST-SCRIPT" set to true. This flag works for both VMware customization engine and cloud-init.
|
|
* cloudinit: remove global disable of pylint W0107 and fix errors
This includes removing a test class which contained no tests but wasn't
detected as empty because of an errant pass statement.
* .pylintrc: update disable comment to match arguments
|
|
Specifically:
* disable E1102 in cloudinit/sources/helpers/openstack.py for reasons
described in a comment, and
* refactor `abs_join` to require at least one positional argument; this
matches os.path.join's signature, and that mismatch is what was
causing pylint to emit a warning
* bump to pylint 2.4.2
|
|
As the first refactor PR, this also includes the initial structure for tests.
LP: #1884619
|
|
Hetzner cloud only supports user-data as a string (presumably utf-8).
In order to allow users on Hetzner to provide binary data to cloud-init,
we will attempt to base64decode the userdata.
The change here adds a 'maybe_b64decode' function that will decode data
if and only if is base64 encoded.
The reason for not using util.b64d is that we do not want the return value
decoded to a string, and util.b64d will do that if it can. Additionally
we call decode with validate=True which oddly is not the default.
LP: #1884071
|
|
This introduces a way to log the dhclient error stream, and uses it for the Azure datasource (where we have a specific requirement for this data to be logged).
|
|
|
|
deployPkg enable-custom-scripts", the return code will be EX_UNAVAILABLE(69), on this condition, it should not take it as error. (#413)
|
|
This was painful, but it finishes a TODO from cloudinit/subp.py.
It moves the following from util to subp:
ProcessExecutionError
subp
which
target_path
I moved subp_blob_in_tempfile into cc_chef, which is its only caller.
That saved us from having to deal with it using write_file
and temp_utils from subp (which does not import any cloudinit things now).
It is arguable that 'target_path' could be moved to a 'path_utils' or
something, but in order to use it from subp and also from utils,
we had to get it out of utils.
|
|
This fixes issues with closing brackets not matching the opening
bracket's line and continuation line under-idented for hanging indent.
|
|
This removes the use of variables named ‘l’, ‘O’, or ‘I’. Generally
these are used in list comprehension to read the line of lines.
|
|
We recently discovered that pylint is failing to report some errors when
invoked across our entire codebase (see
https://github.com/PyCQA/pylint/issues/3611). I've run pylint across
every Python file under cloudinit/[0], and this commit fixes the issues
so-discovered.
[0] find cloudinit/ -name "*.py" | xargs -n 1 -t .tox/pylint/bin/python -m pylint
|
|
|
|
* Add physical network type: cascading to openstack helpers
* add new helpers test for checking all openstack KNOWN_PHYSICAL_TYPES get type 'physical'.
|
|
* url_helper: drop six
* url_helper: sort imports
* log: drop six
* log: sort imports
* handlers/__init__: drop six
* handlers/__init__: sort imports
* user_data: drop six
* user_data: sort imports
* sources/__init__: drop six
* sources/__init__: sort imports
* DataSourceOVF: drop six
* DataSourceOVF: sort imports
* sources/helpers/openstack: drop six
* sources/helpers/openstack: sort imports
* mergers/m_str: drop six
This also allowed simplification of the logic, as we will never
encounter a non-string text type.
* type_utils: drop six
* mergers/m_dict: drop six
* mergers/m_list: drop six
* cmd/query: drop six
* mergers/__init__: drop six
* net/cmdline: drop six
* reporting/handlers: drop six
* reporting/handlers: sort imports
|
|
Azure stores the instance ID with an incorrect byte ordering for the
first three hyphen delimited parts. This results in invalid
is_new_instance checks forcing Azure datasource to recrawl the metadata
service.
When persisting instance-id from the metadata service, swap the
instance-id string byte order such that it is consistent with
that returned by dmi information. Check whether the instance-id
string is a byte-swapped match when determining correctly whether
the Azure platform instance-id has actually changed.
|
