FS not created when using erasure coding

[Liam Young]

When the ceph-mon charm tries to create a new fs it fails if an erasure-coded data pool is being used. This is because the client tool requires a '--force' option as using an ec default data pool is not recommended *1 *2. The suggested approach is to use a replicated pool for metatdata and a replicated pool as the default data pool. The add an additional EC data pool. Access to the datapools is the specified by the client using xattrs.

*1 https://tracker.ceph.com/issues/42450
*2 https://docs.ceph.com/docs/master/cephfs/createfs/#creating-pools

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/751017

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751017
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-fs/commit/?id=051ba2a55fe821611fb1ae250770ceebea4de2d4
Submitter: Zuul
Branch: master

commit 051ba2a55fe821611fb1ae250770ceebea4de2d4
Author: Liam Young <email address hidden>
Date: Thu Sep 10 14:49:10 2020 +0000

    Align EC usage with best practice

    When using erasure coding three pools are needed. One metadata
    pool which is replicated, one default data pool which is replicated
    and one erasure coded pool.

    Add an additional ec-pool-weight option to allow the erasure coded
    pool usage to be modelled within the deployment.

    The consumer of the share will need to use xattrs to ensure that
    the hierarchy of directories and files are stored in EC pool while
    leaving the default replicated data pool for file backtraces.

    Depends-On: Ib19e545fc676fa9e986f3fddcfb92cf99e778a3f
    Depends-On: I48348ab2b6c8952c5e22008a074a60c1f35be952
    Change-Id: I1af386d5e1d031f2f0387c7381af683f948172d9
    Closes-Bug: #1895154