Currently the client to mount volumes in the ceph-fs storage class in a deployment of CK with ceph-fs is not configurable. My understanding of the code is that it will end up as ceph-fuse if available and not explicitly set otherwise [1][2].
I have done some testing with fio and found that using a volume mounted with ceph-fuse is much slower than using the kernel client:
In a pod using ceph-fuse:
READ: bw=781KiB/s (800kB/s), 781KiB/s-781KiB/s (800kB/s-800kB/s), io=768MiB (805MB), run=1005823-1005823msec
WRITE: bw=261KiB/s (267kB/s), 261KiB/s-261KiB/s (267kB/s-267kB/s), io=256MiB (269MB), run=1005823-1005823msec
Directly mounted to the worker:
READ: bw=3998KiB/s (4094kB/s), 3998KiB/s-3998KiB/s (4094kB/s-4094kB/s), io=768MiB (805MB), run=196586-196586msec
WRITE: bw=1336KiB/s (1368kB/s), 1336KiB/s-1336KiB/s (1368kB/s-1368kB/s), io=256MiB (269MB), run=196586-196586msec
I then manually created a new storage class using `mounter: kernel` and tested inside of a new pod:
READ: bw=4327KiB/s (4431kB/s), 4327KiB/s-4327KiB/s (4431kB/s-4431kB/s), io=768MiB (805MB), run=181628-181628msec
WRITE: bw=1446KiB/s (1480kB/s), 1446KiB/s-1446KiB/s (1480kB/s-1480kB/s), io=256MiB (269MB), run=181628-181628msec
It seems like we should atleast investigate making this configurable, if not default.
[1] https://github.com/charmed-kubernetes/cdk-addons/blob/master/get-addon-templates#L407
[2] https://github.com/ceph/ceph-csi/blob/master/examples/cephfs/storageclass.yaml#L39
Additional references:
https://github.com/rook/rook/issues/3619
https://github.com/rook/rook/issues/3315
https://docs.ceph.com/docs/master/cephfs/kernel-features/#quotas
Relevantly I have two PR's linked to a bug filed against the kubernetes-master charm for this [0] to enable the necessary config.
[0] https:/