| Age | Commit message (Collapse) | Author |
|---|
|
Use PEP 589 syntax for TypeDict annotation.
Also fixes previously broken typing MetaSchema typing implementation.
|
|
Applied Black and isort, fixed any linting issues, updated tox.ini
and CI.
|
|
Improve schema validation.
This adds strict validation of config module definitions at testing
time, with plumbing included for future runtime validation. This
eliminates a class of bugs resulting from schemas that have definitions
that are incorrect, but get interpreted by jsonschema as
"additionalProperties" that are therefore ignored.
- Add strict meta-schema for jsonschema unit test validation
- Separate schema from module metadata structure
- Improve type annotations for various functions and data types
Cleanup:
- Remove unused jsonschema "required" elements
- Eliminate manual memoization in schema.py:get_schema(),
reference module.__doc__ directly
|
|
- Mostly based on FreeBSD, the main exception is that
`find_devs_with_on_freebsd` does not work.
- Since we cannot get the CDROM or the partition labels,
`find_devs_with_on_dragonflybsd()` has a more naive approach and
returns all the block devices.
|
|
On FreeBSD, if a UFS has trim: (-t) or MAC multilabel: (-l) flag, resize
FS fail, because the _can_skip_ufs_resize check gets tripped up by the
missing options.
This was reported at FreeBSD Bugzilla:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250496 and as
LP: #1901958
Rather than fixing the parser as in the patches proposed there (and
attempted in #636) this pull-request rips out all of it, and simplifies
the code. We now use `growfs -N` and check if that returns an error. If
it returns the correct kind of error, we can skip the resize, because we
either are at the correct size, or the filesystem in question is broken
or not UFS. If it returns the wrong kind of error, we just re-raise it.
LP: #1901958
|
|
This includes a fix to a test that had a string concatenation issue, and
so was only testing a prefix of what was intended.
|
|
This was painful, but it finishes a TODO from cloudinit/subp.py.
It moves the following from util to subp:
ProcessExecutionError
subp
which
target_path
I moved subp_blob_in_tempfile into cc_chef, which is its only caller.
That saved us from having to deal with it using write_file
and temp_utils from subp (which does not import any cloudinit things now).
It is arguable that 'target_path' could be moved to a 'path_utils' or
something, but in order to use it from subp and also from utils,
we had to get it out of utils.
|
|
Commit 6797e822959b84c98cf73e02b2a6e3d6ab3fd4fe replaced
the LOG.warn calls that linters were warning about; this
also replaces calls that linters would not have recognised
(as `log` is generally a parameter in these scenarios).
LP: #1508442
|
|
- UFS file system support
- GPT partition table support
- add support for newfs's -L parameter (label)
- move freebsd specific test from Azure to freebsd
|
|
In some environments, like FreeBSD, gpart can return the device basename
instead of the full path. If this discovered devpath does not exist and
is missing the '/dev/' prefix, add that prefix in an attempt to find the
device.
|
|
By default, FreeBSD's growfs runs interactively asking a question
which can be mitigated using the '-y' command line option. The fix
here is simply to pass -y to growfs to avoid the prompt.
LP: #1404745
|
|
This enables warnings produced by pylint for unused variables (W0612),
and fixes the existing errors.
|
|
The zfs/zpool commands will hang for 10 seconds if /dev/zfs is not
present (bug 1760173). This is a common occurence for containers
using zfs as rootfs. Additionally handle missing zpool command or
other errors that may occur while executing the zpool command.
|
|
Previously there was no support at all for zfs file system. With this
change it is now possible to use the resizefs module to grow a zpool to
its maximum partition size on FreeBSD.
LP: #1721243
|
|
Resize of btrfs fails if the mount point for the file system we are trying
to resize, i.e. the root of the filesystem is read only. With this change
we use a known (currently snapper specific) rw location to work around a
flaw that blocks resizing of the ro filesystem.
LP: #1734787
|
|
A recent cleanup of the resizefs module broke resizing when a system was
booted with root=PARTUUID=<uuid> and the device /dev/root does not exist.
This path is exposed with the Ubuntu 16.04 but not with Ubuntu 17.10. A
recreate exists under bug 1684869.
LP: #1725067
|
|
As root user, os.access(<path>, os.W_OK) will always return True so that
path will never get executed. Also avoid a warning if the root is
overlayroot, which is the common case on a MAAS booted 'ephemeral' system.
|
|
Supposedly it was never a feature to be able to pass a path to a block
device to xfs_growfs and have it grow the filesystem. The behavior changed
upstream recently. It is only supported to pass the mount point of a mounted
XFS filesystem. This causes breakages in cloud-init.
Upstream xfs change was commit b97815a0321072a7154ecab63e297af84066fc78.
https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/commit/?id=b97815a0321
rhbz: rhbz: https://bugzilla.redhat.com/show_bug.cgi?id=1490505
Signed-off-by: Dusty Mabe <dusty@dustymabe.com>
|
|
Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.
This branch also does the following:
- drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
option only created the specified directory and didn't make use of
that directory for any resize operations.
- Drop yaml.dumps calls from schema documentation generation to avoid
yaml import costs on module load
- Add __doc__ = get_schema_doc(schema) definitions it each module to
supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
cc_resizefs
- Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
contain more than one example
|
|
This patch targets to make FreeBSD 10.3 or 11 work on Azure. The
modifications abide by the rule of:
* making as less modification as possible
* delegate to the distro or datasource where possible.
The main modifications are:
1. network configuration improvements, and movement into distro path.
2. Fix setting of password.
Password setting through "pw" can only work through pipe.
3. Add 'root:wheel' to syslog_fix_perms field.
4. Support resizing default file system (ufs)
5. copy cloud.cfg for freebsd to /etc/cloud/cloud.cfg rather than
/usr/local/etc/cloud/cloud.cfg.
6. Azure specific changes:
a. When reading the azure endpoint, search in a different path
and read a different option name (option-245 vs. unknown-245).
so, the lease file path should be generated according to platform.
b. adjust the handling of ephemeral mounts for ufs filesystem and
for finding the ephemeral device.
c. fix mounting of cdrom
LP: #1636345
|
|
When booted without an initramfs, the root device will be /dev/root, not a
named device. There is partial support for this when resizing filesystems,
but not for growing partitions, without which it doesn't do much good. Move
the /dev/root resolution code to util.py and use it from cc_growpart.py.
Also, booting without an initramfs only works with a root= argument that's
either a kernel device name (which is unstable) or a partition UUID. Handle
the case of root=PARTUUID=value, not just LABEL and UUID.
LP: #1677376
|
|
This has been a recurring ask and we had initially just made the change to
the cloud-init 2.0 codebase. As the current thinking is we'll just
continue to enhance the current codebase, its desirable to relicense to
match what we'd intended as part of the 2.0 plan here.
- put a brief description of license in LICENSE file
- put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0
- simplify the per-file header to reference LICENSE
- tox: ignore H102 (Apache License Header check)
Add license header to files that ship.
Reformat headers, make sure everything has vi: at end of file.
Non-shipping files do not need the copyright header,
but at the moment tests/ have it.
|
|
pycodestyle 2.1.0 is in Ubuntu zesty, and complained about the
changes made here. Simple style changes. This makes 'make pep8'
pass again when built in a zesty build system with proposed enabled.
|
|
This adds lots of config module documentation in a standard format.
It will greatly improve the content at readthedocs.
Additionally:
* Add a 'doc' env to tox.ini
* Changed default highlight language for sphinx conf from python to yaml
most examples in documentation are yaml configs
* Updated datasource examples to highlight sh code properly
|
|
|
|
|
|
util.log_time()'s return value was what was being sent to fork_cb. This means
the resize ran in parallel and the call to fork_cb threw a traceback (trying
to call Nonetype).
By permitting fork_cb to take kwargs, and using the correct method syntax,
this now forks and resizes in the background as appropriate.
|
|
In a container the device nodes may exist but not be writable.
I'm seeing this on trusty host with trusty containers, the root
device ends up looking like it is to /dev/loop0.
LP: #1366891
|
|
This just removes comments '# pylint:' things and other code
remnents of pylint.
|
|
Fixed all complaints from running "make pep8". Also version locked
pep8 in test-requirements.txt to ensure that pep8 requirements don't
change without an explicit commit.
|
|
|
|
If mount_info says that the root filesystem is on /dev/root and
/dev/root does not exist, then we'll try to glean that information
from the linux kernel cmdline.
This situation occurs at least when you boot without an initramfs
for the current ppc64el cloud images:
qemu-system-ppc64 ... -kernel my.kernel -append 'root=/dev/sda'
When doing that, /proc/1/mountinfo will say '/dev/root' for '/'.
|
|
current form its still missing some modules though.
Supported:
-SSH-keys
-growpart
-growfs
-adduser
-powerstate
|
|
The reason for this is that more and more things I was wanting to be able to
see how long they took. This puts that time logic into a single place. It
also supports (by default) reading from /proc/uptime as the timing mechanism.
While that is almost certainly slower than time.time(), it does give
millisecond granularity and is not affected by 'ntpdate' having
run in between the two events.
|
|
Now, errors will not be so annoying if the device doesn't exist.
Specifically, if there is no device in a container, only debug messages
will be logged.
LP: #1160462
|
|
LP: #1136936
|
|
The existing code has two issues with btrfs:
1) The command to resize a btrfs filesystem uses a path to the mount
point, not the underlying device:
$ btrfs filesystem resize max /dev/vda1
ERROR: unable to resize '/dev/vda1' - Inappropriate ioctl for device
Resize '/dev/vda1' of 'max'
$ btrfs filesystem resize max /
Resize '/' of 'max'
2) The code that is given a path and finds the ID of the device where
the path is mounted doesn't work for btrfs:
Use /proc/$$/mountinfo to find the device where path is mounted.
This is done because with a btrfs filesystem using os.stat(path)
does not return the ID of the device.
Here, / has a device of 18 (decimal).
$ stat /
File: '/'
Size: 234 Blocks: 0 IO Block: 4096 directory
Device: 12h/18d Inode: 256 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-01-13 07:31:04.358011255 +0000
Modify: 2013-01-13 18:48:25.930011255 +0000
Change: 2013-01-13 18:48:25.930011255 +0000
Birth: -
Find where / is mounted:
$ mount | grep ' / '
/dev/vda1 on / type btrfs (rw,subvol=@,compress=lzo)
And the device ID for /dev/vda1 is not 18:
$ ls -l /dev/vda1
brw-rw---- 1 root disk 253, 1 Jan 13 08:29 /dev/vda1
So use /proc/$$/mountinfo to find the device underlying the input
path.
|
|
The existing code has two issues with btrfs:
1) The command to resize a btrfs filesystem uses a path to the mount
point, not the underlying device:
$ btrfs filesystem resize max /dev/vda1
ERROR: unable to resize '/dev/vda1' - Inappropriate ioctl for device
Resize '/dev/vda1' of 'max'
$ btrfs filesystem resize max /
Resize '/' of 'max'
2) The code that is given a path and finds the ID of the device where
the path is mounted doesn't work for btrfs:
Use /proc/$$/mountinfo to find the device where path is mounted.
This is done because with a btrfs filesystem using os.stat(path)
does not return the ID of the device.
Here, / has a device of 18 (decimal).
$ stat /
File: '/'
Size: 234 Blocks: 0 IO Block: 4096 directory
Device: 12h/18d Inode: 256 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-01-13 07:31:04.358011255 +0000
Modify: 2013-01-13 18:48:25.930011255 +0000
Change: 2013-01-13 18:48:25.930011255 +0000
Birth: -
Find where / is mounted:
$ mount | grep ' / '
/dev/vda1 on / type btrfs (rw,subvol=@,compress=lzo)
And the device ID for /dev/vda1 is not 18:
$ ls -l /dev/vda1
brw-rw---- 1 root disk 253, 1 Jan 13 08:29 /dev/vda1
So use /proc/$$/mountinfo to find the device underlying the input
path.
|
|
resize_root: noblock
has been broken in the 0.7.0 series. Using it would disable resizing.
LP: #1080985
|
|
1. Remove the usage of the path.join function
now that all code should be going through
the util file methods (and they can be
mocked out as needed).
2. Adjust all occurences of the above join
function to either not use it or replace
it with the standard os.path.join (which
can also be mocked out as needed)
3. Fix pylint from complaining about the
tests folder 'helpers.py' not being found
4. Add a pylintrc file that is used instead
of the options hidden in the 'run_pylint'
tool.
|
|
make pep8 now is silent on precise's pep8 ( 0.6.1-2ubuntu2).
|
|
Show 3 decimal places for how long a resize took. 1 second granularity was
too large.
|
|
a. Instead of executing a bash string, write out a temporary file
and then just execute '/bin/sh' on that file with the right arguments instead.
2. Rename util.SilentTemporaryFile to util.ExtendedTemporaryFile and update the usages
of the previous name accordingly, this better reflects what this temp file is.
3. More teenie pep8 line length fixings
|
|
2. Fixing up more cloud.path.joins found to use the right ro/rw filename
|
|
|
|
some protection against module name collisions when importing.
|
