CephFS EINVAL specified for ceph.dir.subvolume

Written by
Ronald Reitsema & Alex Boonstra
Published on 2023-02-06

What could be the case when creating a subvolume in CephFS throws an EINVAL VolumeException?

A client of ours recently got errors when creating a 24GiB subvolume:

ceph fs subvolume create cephfs-filesystem \
  example1 --group_name=thegroup \
  --size=25769803776

But, instead of silence and a new subvolume, they got this in their face: a big Python backtrace.

Error EINVAL: Traceback (most recent call last):
  File "ceph/mgr/volumes/fs/operations/versions/subvolume_base.py",
      line 271, in discover
    self.fs.stat(self.base_path)
  File "cephfs.pyx", line 1338, in cephfs.LibCephFS.stat
cephfs.ObjectNotFound: error in stat: /volumes/thegroup/example1:
  No such file or directory [Errno 2]

During handling of the above exception, another exception occurred:
...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py",
      line 101, in mark_subvolume
    self.fs.setxattr(self.base_path, 'ceph.dir.subvolume', b'1', 0)
  File "cephfs.pyx", line 1264, in cephfs.LibCephFS.setxattr
cephfs.InvalidValue: error in setxattr: Invalid argument [Errno 22]

During handling of the above exception, another exception occurred:
...
  File "ceph/mgr/volumes/fs/operations/trash.py", line 101, in dump
    raise VolumeException(-e.args[0], e.args[1])
TypeError: bad operand type for unary -: 'str'

(Actually the backtrace was a lot bigger. It’s trimmed here for clarity.)

We see:

A self.fs.stat() call fails: there is no example1 (yet);
marking the path as subvolume using setxattr() fails;
then some more failures during exception handling;
and lastly, a string being treated as an integer.

In our case it was actually beneficial for debugging that there was an unfixed bug in this old CephFS 15 Octopus: Bad exception thrown (fix).

Without it, we would have only gotten a Error EINVAL: invalid value specified for ceph.dir.subvolume error string. But now, it pointed us straight to the call that failed:

self.fs.setxattr(
    self.base_path,
    'ceph.dir.subvolume',
    b'1',
    0)

It turns out, Ceph configures various properties using the setxattr() filesystem interface. The extended attributes (xattr(7)) can for instance store quota, or, in this case, flag a path as being a subvolume.

When you use ceph fs subvolume create, a directory (example1) gets created within the subvolumegroup (thegroup). The ceph.dir.subvolume of this directory gets set to 1 (turning the directory into a subvolume).

If you were to create a Ceph subvolume in a subvolumegroup while omitting a name, the ceph.dir.subvolume attribute of the subvolumegroup directory (the parent) gets set to 1 rather than raising an error or warning. But since you cannot create subvolumes inside subvolumes, this makes it impossible to use this group as a subvolumegroup any longer.

One can reproduce with the following example. (We use a subvolumegroup argument for clarity. When omitted, the default _nogroup subvolumegroup would be used.)

ceph fs subvolumegroup create ceph-filesystem thegroup
ceph fs subvolume create ceph-filesystem example1 --group_name thegroup
ceph fs subvolume create ceph-filesystem "" --group_name thegroup
ceph fs subvolume create ceph-filesystem example2 --group_name thegroup

That third command broke things, causing the fourth and last command to print the following error:

Error EINVAL: invalid value specified for ceph.dir.subvolume

The error is correct, but not really helpful:

The “invalid value specified” seems to be meant for the subvolumegroup, not for the subvolume we try to create. This is confusing and misleading.
It is unclear which value is invalid. In this case a 1 needs to be a 0, but it requires thorough investigation to come to this conclusion.

And to add to the confusion; this extended attribute can be set with setfattr(2), but it cannot be read with getfattr(2):

# mount -t ceph \
    10.1.2.100:6789,10.1.2.101:6789,10.1.2.102:6789:/ \
    /mnt/cephfs/ -o name=admin,secret=admin_secret

# getfattr -n ceph.dir.subvolume \
    /mnt/cephfs/volumes/thegroup
ceph.dir.subvolume: No such attribute

Fortunately, we can set attributes and unbreak the situation. To resolve the issue, you need to change the ceph.dir.subvolume on the filesystem to 0:

# setfattr -n ceph.dir.subvolume -v 0 \
    /mnt/cephfs/volumes/thegroup

Note that the broken example2 subvolume was still created – even though it showed an error. It is unusable, but can be deleted with a normal rm call.

# ceph fs subvolume rm ceph-filesystem \
    example2 --group_name thegroup

This error had us pulling our hair out for a while. While we’re not entirely sure that an empty-name volume was the cause for our troubles, setting the ceph.dir.subvolume xattr back to 0 was the solution.

And because of the extra details in the backtrace we got to the setxattr(2) call sooner.

Naturally we filed a bug report in the CephFS issue tracker. But not before we tested that this as still an issue in newer CephFS:

CephFS 17 Quincy shows only this error:

Error EINVAL: invalid value specified for ceph.dir.subvolume

CephFS 16 Pacific and CephFs 15 Octopus both show the big backtrace. You can expand the details to view full example:

(expand ceph client backtrace)

Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 271, in discover
    self.fs.stat(self.base_path)
  File "cephfs.pyx", line 1338, in cephfs.LibCephFS.stat
cephfs.ObjectNotFound: error in stat: /volumes/thegroup/example2: No such file or directory [Errno 2]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 164, in create_subvolume
    with open_subvol(self.mgr, fs_handle, self.volspec, group, subvolname, SubvolumeOpType.CREATE) as subvolume:
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 72, in open_subvol
    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py", line 95, in get_subvolume_object
    subvolume.discover()
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 283, in discover
    raise VolumeException(-errno.ENOENT, "subvolume '{0}' does not exist".format(self.subvolname))
volumes.fs.exception.VolumeException: -2 (subvolume 'example2' does not exist)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 101, in mark_subvolume
    self.fs.setxattr(self.base_path, 'ceph.dir.subvolume', b'1', 0)
  File "cephfs.pyx", line 1264, in cephfs.LibCephFS.setxattr
cephfs.InvalidValue: error in setxattr: Invalid argument [Errno 22]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 171, in create
    self.mark_subvolume()
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 103, in mark_subvolume
    raise VolumeException(-errno.EINVAL, "invalid value specified for ceph.dir.subvolume")
volumes.fs.exception.VolumeException: -22 (invalid value specified for ceph.dir.subvolume)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 99, in dump
    self.fs.rename(path, self.unique_trash_path)
  File "cephfs.pyx", line 1545, in cephfs.LibCephFS.rename
cephfs.Error: error in rename /volumes/thegroup/example2 to /volumes/_deleting/4dd2069d-7cf9-44fe-b1c9-1c3c8e80ae28: Invalid cross-device link [Errno 18]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1212, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 466, in handle_command
    return handler(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 34, in wrap
    return f(self, inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 520, in _cmd_fs_subvolume_create
    namespace_isolated=cmd.get('namespace_isolated', False))
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 176, in create_subvolume
    self._create_subvolume(fs_handle, volname, group, subvolname, **kwargs)
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 142, in _create_subvolume
    self.mgr, fs_handle, self.volspec, group, subvolname, size, isolate_nspace, pool, oct_mode, uid, gid)
  File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 26, in create_subvol
    subvolume.create(size, isolate_nspace, pool, mode, uid, gid)
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 193, in create
    self._remove_on_failure(subvol_path, retained)
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 151, in _remove_on_failure
    self.remove()
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 346, in remove
    self.trash_base_dir()
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 301, in trash_base_dir
    self._trash_dir(self.base_path)
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 289, in _trash_dir
    trashcan.dump(path)
  File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 101, in dump
    raise VolumeException(-e.args[0], e.args[1])
TypeError: bad operand type for unary -: 'str'

Yea. So the newer versions were all affected. Glad we got it sorted. (And glad it wasn’t our own fault for running old software.)

CephFS EINVAL specified for ceph.dir.subvolume

CephFS EINVAL specified for ceph.dir.subvolume

Location

Links

OSSO B.V.