manila-share not working with ceph mimic (13.2) nor ceph nautilus (14.2)

[Shi Yan]

We are using the rocky release manila, and after we upgrade the ceph version from 12.2.11 (luminous) to 13.2.6 (mimic) the breakage happens:

Rolling back to luminous is working again.

 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager [req-961ba3d1-273e-48fb-bed7-8e872ba62683 - - - - -] Caught exception trying ensure share 'e587deff-e01d-4d a4-b44c-7a9533967bb3'.: OperationNotSupported: [Errno 95] error in setxattr
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager Traceback (most recent call last):
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager File "/usr/lib/python2.7/dist-packages/manila/share/manager.py", line 478, in _ensure_share
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager share_server=share_instance['share_server'])
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager File "/usr/lib/python2.7/dist-packages/manila/share/drivers/cephfs/driver.py", line 277, in ensure_share
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager return self.create_share(context, share, share_server)
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager File "/usr/lib/python2.7/dist-packages/manila/share/drivers/cephfs/driver.py", line 256, in create_share
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager mode=self._cephfs_volume_mode)
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager File "/usr/lib/python2.7/dist-packages/ceph_volume_client.py", line 631, in create_volume
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager self.fs.setxattr(path, 'ceph.quota.max_bytes', to_bytes(size), 0)
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager File "cephfs.pyx", line 990, in cephfs.LibCephFS.setxattr
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager OperationNotSupported: [Errno 95] error in setxattr
 2019-08-05 15:52:54.527 27245 ERROR manila.share.manager

Confirmed to still break with Nautilus. (see comments)

[Tom Barron (tpb)]

It seems odd that size 0 is being passed to setxattr for the valude of ceph.quota.max_bytes. Ceph quotas are used to implement manila share sizes, which are never zero.

[Tom Barron (tpb)]

Well as gouthamr pointed out to me, that value is always set to zero [1]. I guess it's not supposed to be the value of the CephFS quota as I understood it (implementation of manila share size).

[1] https://github.com/ceph/ceph/blob/2c8b4eaaa78c7040c5b41fbe71caac7d7adaa7ca/src/pybind/ceph_volume_client.py#L660

[Goutham Pacha Ravi (gouthamr)]

Related bug: https://bugs.launchpad.net/manila/+bug/1855885

[Radosław Piliszek (yoctozepto)]

I merged both bugs because they are essentially the same (the same line in manila's ceph driver triggered the same error message in the same daemon).

Extended mnasiadka's summary below:

Situation in Kolla (both servers and clients):

Ubuntu/Rocky: Luminous (12.2) <- presumably works fine
CentOS/Rocky: Luminous (12.2) <- presumably works fine

Ubuntu/Stein: Luminous (12.2) <- presumably works fine
CentOS/Stein: Nautilus (14.2, Source: RDO) <- breaks

Ubuntu/Train: Nautilus (14.2) <- no idea
CentOS/Train: Nautilus (14.2) <- no idea

[Radosław Piliszek (yoctozepto)]

Kolla-Ansible Rocky is affected in case of an external Ceph.

All are affected in case of internal (too new).

Kolla waits for Manila and Ceph to get compatible again, will need to upgrade the code.

[Goutham Pacha Ravi (gouthamr)]

Revisiting this bug to note a couple of things:

On my test environment, I've the following:

Ceph container OS: Red Hat Enterprise Linux release 8.1 (Ootpa)
ceph version 14.2.4-114.el8cp (8d1742d1e853d6bfb69a73026f07dc7fa27f7459) nautilus (stable)

I do not see any errors when creating/exporting, extending, shrinking shares with CephFS-NFS

So this doesn't appear to be a manila bug to me; unless someone points out that there is something that the driver can do to workaround a potential incompatibility within Ceph or the OS that it runs on.

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-05-28-15.00.log.html

[Victoria Martinez de la Cruz (vkmc)]

Shi Yan/Radosław we are running the CephFS Native and CephFS NFS jobs on Nautilus now, not hitting any issues. Not so sure about Mimic. Could you confirm you are still seeing this?

[Shi Yan (yanshi-403)]

Sorry, our environment has been upgraded and we do not have available ones at present to roll back and test.

[Radosław Piliszek (yoctozepto)]

We no longer care about Mimic either. I guess it was really the same issue so if Nautilus works, then Mimic likely too. Looks closable to me.

[Goutham Pacha Ravi (gouthamr)]

Rounding back here in case someone else runs into this. This bug is characteristic of the ceph environment. It can manifest when using the ceph kernel client, and an older version of the kernel which doesn't support setting xattrs [1]; or when using a newer Ceph client (ex: Nautilus) against an older ceph cluster (ex: Luminous). See relevant bugzilla here: https://bugzilla.redhat.com/show_bug.cgi?id=1820346 and the CephFS documentation regarding quotas: https://docs.ceph.com/docs/mimic/cephfs/quota/

This isn't a problem that can be worked around in the manila driver.