Add support for CEPH ceph.file.layout and ceph.dir.layout xattr

[Fabio Augusto Miranda Martins]

Ceph supports using extended attributes (xattrs) to configure a file or directory layout, so as to tell Ceph which pool to place the objects related to that file or directory. This is mostly relevant for when you are using CephFS and have multiple pools in the Ceph volume, as documented here [1] and [2].

However, if you have NFS-Ganesha serving the CephFS filesystem to NFS clients, it will fail if you try to setfattr

ubuntu@jammy-183048:~$ sudo setfattr -n ceph.dir.layout.pool -v ceph-fs_ecdata /mnt/nfs1/ecpool
setfattr: /mnt/nfs1/ecpool: Operation not supported

So, it seems there's a limitation on what kind of Ceph xattr ganesha would support.

In this bug, there are some discussions around some CEPH FSALs that were enabled.

https://github.com/nfs-ganesha/nfs-ganesha/issues/526

And we can see them at:

https://github.com/nfs-ganesha/nfs-ganesha/blob/f495f2fd8a3529065cd5c7eef5276910e11abb1a/src/FSAL/FSAL_CEPH/internal.h#L201-L204

However none of them are related to the xattr required by Ceph for disk layouts to work (i.e.: ceph.dir.layout.pool)

Hence, although ceph-fs supports file-layouts, it doesn't seem that NFS Ganesha contains the necessary features to support it in a Ceph environment.

It would be good to have implemented in NFS-Ganesha, so we can relay the CephFS requirements to configure file-layouts.

[1] https://docs.ceph.com/en/latest/cephfs/file-layouts/#adding-a-data-pool-to-the-file-system
[2] https://docs.ceph.com/en/latest/cephfs/file-layouts/#adding-a-data-pool-to-the-file-system

[Peter Sabaini (peter-sabaini)]

Fabio, can I ask you to provide some detail on the setup.

- Can you confirm which NFS client you are using?
- Would you be able to share logs. Ideally provide an sosreport, but if that is not feasible /v/l/kern.log and /v/l/syslog would be great

Thank you.

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Peter,

The NFS client is a Jammy VM running nfs-common 1:2.6.1-1ubuntu1.2 and kernel 5.15.0-102-generic.

I'm running a sosreport and will attach it here soon.

Regards,
Fabio

[Fabio Augusto Miranda Martins (fabio.martins)]

This is also being discussed upstream: https://github.com/nfs-ganesha/nfs-ganesha/issues/1123

[Fabio Augusto Miranda Martins (fabio.martins)]

Based on discussions at https://github.com/nfs-ganesha/nfs-ganesha/issues/1123, it seems this is something released with nfs-ganesha V4, so IIUC, this is something that will only be available to an Openstack + Manila + Ceph + NFS Ganesha in Noble Numbat, since it offers nfs-ganesha 4.3-8ubuntu1