CephFS authorize fails with unknown cap type

[Billy Olsen]

[Impact]

Attempting to provide access to a user within Ceph to a specific mount path fails with unknown cap type. This appears to be due to the monitor not knowing how to validate the caps that are provided with the mount path per upstream bug https://tracker.ceph.com/issues/39395 and subsequent pull requests.

This is fixed in Mimic (13.1.0+) and included in the current Luminous devel release (upcoming 12.2.13).

[Test Case]

Steps to recreate:

1. Install ceph w/ ceph-fs.

2. Mount ceph filesystem and create subdirectory for restricting access
$ ceph-fuse -k /etc/ceph/ceph.client.foo.keyring --id foo -m 10.5.0.5:6789 /mnt/ceph-fs
$ mkdir /mnt/ceph-fs/bar

3. Authorize access for ceph user to rw a directory
$ ceph fs authorize ceph-fs client.foo /bar rw

Expected Results:

The authorize command to succeed

Actual Results:

Error EINVAL: unknown cap type '/bar'

[Regression Potential]

Regression potential is low as this has already been fixed upstream and has seen additional testing without additional problem reports from the change. The change does affect the validation of capabilities, so if a problem were to arise it would likely be in the verification of capabilities when the code is parsing.

[Other Info]

Upstream pull-request: https://github.com/ceph/ceph/pull/28666

[Billy Olsen (billy-olsen)]

This appears to have been broken by commit d63fccb52241a216a08a92e615bcff008d365392 which added a validation for all of the capabilities in this code path. The fs authorize command works by adding a new capability keyed by the path. This capability with the path key was not understood by the standard capability check method being invoked so it returned False to indicate the capabilities are not valid, which results in an EINVAL value.

The patch included in the pull request fixes this by changing to only check the mds and osd capabilities instead of the full set.

[Billy Olsen (billy-olsen)]

I've added the regression-update tag due to this being introduced in the 12.2.12 update via the SRU process.

I've also built a package in a PPA with the latest version for testing purposes. You can find it at https://launchpad.net/~billy-olsen/+archive/ubuntu/lp1847822 and can verify that commit 9c18bd4e from the aforementioned PR fixes this.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "bionic patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Timo Aaltonen (tjaalton)]

Hello Billy, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.12-0ubuntu0.18.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

Hello Billy, or anyone else affected,

Accepted ceph into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Billy, or anyone else affected,

Accepted ceph into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[gerald.yang (gerald-yang-tw)]

The ceph packages in queens-proposed fixes this issue

Some details on different nodes

On ceph-mon node:

ubuntu@juju-590627-246395-0:~$ apt-cache policy ceph
ceph:
  Installed: 12.2.12-0ubuntu0.18.04.4
  Candidate: 12.2.12-0ubuntu0.18.04.4
  Version table:
 *** 12.2.12-0ubuntu0.18.04.4 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     12.2.12-0ubuntu0.18.04.3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     12.2.12-0ubuntu0.18.04.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     12.2.4-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
ubuntu@juju-590627-246395-0:~$ ceph fs authorize ceph-fs client.foo / rw
[client.foo]
 key = AQCiAeZd+PTmHhAAh2VDxBu1JJR7qr5lfLavaw==
ubuntu@juju-590627-246395-0:~$ ceph fs authorize ceph-fs client.bar /bar rw
[client.bar]
 key = AQC2COZdxTytDxAAtY3oVcjK8XqzlFf0oRlzWQ==

On ceph-fs node:

ubuntu@juju-590627-246395-7:~$ apt-cache policy ceph
ceph:
  Installed: 12.2.12-0ubuntu0.18.04.4
  Candidate: 12.2.12-0ubuntu0.18.04.4
  Version table:
 *** 12.2.12-0ubuntu0.18.04.4 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     12.2.12-0ubuntu0.18.04.3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     12.2.12-0ubuntu0.18.04.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     12.2.4-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
ubuntu@juju-590627-246395-7:~$ sudo ceph-fuse -c ceph.conf --id foo -k client.foo.keyring mnt
ceph-fuse[4198]: starting ceph client
2019-12-03 07:01:22.855304 7fb6a87f6500 -1 init, newargv = 0x559f708262e0 newargc=9
ceph-fuse[4198]: starting fuse
ubuntu@juju-590627-246395-7:~$ ls mnt/
ubuntu@juju-590627-246395-7:~$ sudo mkdir mnt/bar
ubuntu@juju-590627-246395-7:~$ ls mnt
bar

[gerald.yang (gerald-yang-tw)]

The ceph packages in bionic-proposed fixes this issue

Some details on different nodes

On ceph-mon node:

ubuntu@juju-aad762-246395-1-0:~$ apt-cache policy ceph
ceph:
  Installed: 12.2.12-0ubuntu0.18.04.4
  Candidate: 12.2.12-0ubuntu0.18.04.4
  Version table:
 *** 12.2.12-0ubuntu0.18.04.4 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     12.2.12-0ubuntu0.18.04.3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     12.2.12-0ubuntu0.18.04.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     12.2.4-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
ubuntu@juju-aad762-246395-1-0:~$ ceph fs authorize ceph-fs client.foo / rw
[client.foo]
 key = AQC8EuZdrBm7ERAA7ZaAruEDz3glRMDnPvmWRQ==
ubuntu@juju-aad762-246395-1-0:~$ ceph fs authorize ceph-fs client.bar /bar rw
[client.bar]
 key = AQA+E+ZdoyyENxAAqNXma0Pdq9CaG0rlLL0otQ==

On ceph-fs node:

ubuntu@juju-aad762-246395-1-7:~$ apt-cache policy ceph
ceph:
  Installed: 12.2.12-0ubuntu0.18.04.4
  Candidate: 12.2.12-0ubuntu0.18.04.4
  Version table:
 *** 12.2.12-0ubuntu0.18.04.4 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     12.2.12-0ubuntu0.18.04.3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     12.2.12-0ubuntu0.18.04.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     12.2.4-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
ubuntu@juju-aad762-246395-1-7:~$ sudo ceph-fuse -c ceph.conf --id foo -k client.foo.keyring mnt
ceph-fuse[2553]: starting ceph client
2019-12-03 07:47:40.128237 7f423c183500 -1 init, newargv = 0x55a9692d02e0 newargc=9
ceph-fuse[2553]: starting fuse
ubuntu@juju-aad762-246395-1-7:~$ ls mnt
ubuntu@juju-aad762-246395-1-7:~$ sudo mkdir mnt/bar
ubuntu@juju-aad762-246395-1-7:~$ ls mnt/
bar

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 12.2.12-0ubuntu0.18.04.4

---------------
ceph (12.2.12-0ubuntu0.18.04.4) bionic; urgency=medium

  [ Billy Olsen ]
  * Do not validate fs caps on authorize (LP: #1847822):
    - d/p/dont-validate-fs-caps-on-authorize.patch: Do not validate
      the filesystem caps with a new client connection to the monitor
      when authorizing a client connection.

  [ Dan Hill ]
  * d/p/issue38454.patch: Cherry pick of fixes for misc RGW bugs
    and cleanup of garbage collection code (LP: #1843085).

  [ Dariusz Gadomski ]
  * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries
    in ceph-volume, resolving performance issues in systems under
    heavy load or with large numbers of disks (LP: #1850754).

 -- James Page <email address hidden> Thu, 28 Nov 2019 10:27:34 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 12.2.12-0ubuntu0.18.04.4~cloud0
---------------

 ceph (12.2.12-0ubuntu0.18.04.4~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (12.2.12-0ubuntu0.18.04.4) bionic; urgency=medium
 .
   [ Billy Olsen ]
   * Do not validate fs caps on authorize (LP: #1847822):
     - d/p/dont-validate-fs-caps-on-authorize.patch: Do not validate
       the filesystem caps with a new client connection to the monitor
       when authorizing a client connection.
 .
   [ Dan Hill ]
   * d/p/issue38454.patch: Cherry pick of fixes for misc RGW bugs
     and cleanup of garbage collection code (LP: #1843085).
 .
   [ Dariusz Gadomski ]
   * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries
     in ceph-volume, resolving performance issues in systems under
     heavy load or with large numbers of disks (LP: #1850754).

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 12.2.12-0ubuntu0.18.04.4~cloud0
---------------

 ceph (12.2.12-0ubuntu0.18.04.4~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (12.2.12-0ubuntu0.18.04.4) bionic; urgency=medium
 .
   [ Billy Olsen ]
   * Do not validate fs caps on authorize (LP: #1847822):
     - d/p/dont-validate-fs-caps-on-authorize.patch: Do not validate
       the filesystem caps with a new client connection to the monitor
       when authorizing a client connection.
 .
   [ Dan Hill ]
   * d/p/issue38454.patch: Cherry pick of fixes for misc RGW bugs
     and cleanup of garbage collection code (LP: #1843085).
 .
   [ Dariusz Gadomski ]
   * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries
     in ceph-volume, resolving performance issues in systems under
     heavy load or with large numbers of disks (LP: #1850754).