Investigate enabling ceph-fs to be mounted using the kernel client mounter

Bug #1895352 reported by Chris Johnston on 2020-09-11
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
CDK Addons
High
Unassigned
Kubernetes Master Charm
High
Unassigned

Bug Description

Currently the client to mount volumes in the ceph-fs storage class in a deployment of CK with ceph-fs is not configurable. My understanding of the code is that it will end up as ceph-fuse if available and not explicitly set otherwise [1][2].

I have done some testing with fio and found that using a volume mounted with ceph-fuse is much slower than using the kernel client:

In a pod using ceph-fuse:
   READ: bw=781KiB/s (800kB/s), 781KiB/s-781KiB/s (800kB/s-800kB/s), io=768MiB (805MB), run=1005823-1005823msec
  WRITE: bw=261KiB/s (267kB/s), 261KiB/s-261KiB/s (267kB/s-267kB/s), io=256MiB (269MB), run=1005823-1005823msec

Directly mounted to the worker:
   READ: bw=3998KiB/s (4094kB/s), 3998KiB/s-3998KiB/s (4094kB/s-4094kB/s), io=768MiB (805MB), run=196586-196586msec
  WRITE: bw=1336KiB/s (1368kB/s), 1336KiB/s-1336KiB/s (1368kB/s-1368kB/s), io=256MiB (269MB), run=196586-196586msec

I then manually created a new storage class using `mounter: kernel` and tested inside of a new pod:
   READ: bw=4327KiB/s (4431kB/s), 4327KiB/s-4327KiB/s (4431kB/s-4431kB/s), io=768MiB (805MB), run=181628-181628msec
  WRITE: bw=1446KiB/s (1480kB/s), 1446KiB/s-1446KiB/s (1480kB/s-1480kB/s), io=256MiB (269MB), run=181628-181628msec

It seems like we should atleast investigate making this configurable, if not default.

[1] https://github.com/charmed-kubernetes/cdk-addons/blob/master/get-addon-templates#L407
[2] https://github.com/ceph/ceph-csi/blob/master/examples/cephfs/storageclass.yaml#L39

Additional references:
https://github.com/rook/rook/issues/3619
https://github.com/rook/rook/issues/3315
https://docs.ceph.com/docs/master/cephfs/kernel-features/#quotas

Tags: sts Edit Tag help
Brett Milford (brettmilford) wrote :

Relevantly I have two PR's linked to a bug filed against the kubernetes-master charm for this [0] to enable the necessary config.

[0] https://bugs.launchpad.net/charm-kubernetes-master/+bug/1895428

tags: added: sts
George Kraft (cynerva) wrote :
Changed in cdk-addons:
importance: Undecided → High
Changed in charm-kubernetes-master:
importance: Undecided → High
milestone: none → 1.19+ck1
Changed in cdk-addons:
milestone: none → 1.19+ck1
George Kraft (cynerva) on 2020-09-14
Changed in cdk-addons:
status: New → Triaged
Changed in charm-kubernetes-master:
status: New → Triaged
George Kraft (cynerva) on 2020-09-15
Changed in cdk-addons:
status: Triaged → Fix Committed
Changed in charm-kubernetes-master:
status: Triaged → Fix Committed
George Kraft (cynerva) wrote :
Changed in charm-kubernetes-master:
status: Fix Committed → In Progress
tags: added: review-needed
George Kraft (cynerva) on 2020-09-18
Changed in charm-kubernetes-master:
status: In Progress → Fix Committed
tags: removed: review-needed
tags: added: backport-needed
tags: removed: backport-needed
George Kraft (cynerva) wrote :

Follow-up PR: https://github.com/charmed-kubernetes/cdk-addons/pull/199

The cdk-addons commits have been reverted from release-1.19, release-1.18, release-1.17. We'll need to backport to those again.

tags: added: backport-needed review-needed
Changed in cdk-addons:
status: Fix Committed → In Progress
George Kraft (cynerva) on 2020-11-13
Changed in cdk-addons:
status: In Progress → Fix Committed
tags: removed: review-needed
Changed in cdk-addons:
status: Fix Committed → Fix Released
Changed in charm-kubernetes-master:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers