Charm errs if any rbd pool is erasure encoded

[Vern Hart]

As near as I can tell, erasure-coded pools are not supported as the source with rbd-mirror. When the ceph-rbd-mirror charm tries to "mirror pool enable" on an ec pool we get the following error:

  librbd::api::Mirror: mode_set: failed to allocate mirroring uuid: (95) Operation not supported

To work around this, I modified eligible pools in lib/charm/openstack/ceph_rbd_mirror.py to exclude pools that have an erasure_code_profile parameter:

--- lib/charm/openstack/ceph_rbd_mirror.py.orig 2020-10-05 04:05:01.126851853 +0000
+++ lib/charm/openstack/ceph_rbd_mirror.py 2020-10-05 04:05:12.739024844 +0000
@@ -71,7 +71,8 @@
         :rtype: dict
         """
         return {pool: attrs for pool, attrs in pools.items()
- if 'rbd' in attrs['applications']}
+ if ('rbd' in attrs['applications']
+ and 'erasure_code_profile' not in attrs['parameters'])}

     def custom_assess_status_check(self):
         """Provide mirrored pool statistics through juju status."""

The full trace of the above error:

2020-10-05 04:15:55.440 7f14a0fce0c0 -1 librbd::api::Mirror: mode_set: failed to allocate mirroring uuid: (95) Operation not supported
Traceback (most recent call last):
  File "hooks/ceph-local-relation-changed", line 22, in <module>
    main()
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/charm/reactive/ceph_rbd_mirror_handlers.py", line 104, in configure_pools
    charm_instance.mirror_pool_enable(pool)
  File "lib/charm/openstack/ceph_rbd_mirror.py", line 155, in mirror_pool_enable
    subprocess.check_call(base_cmd + ['enable', pool, 'pool'])
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rbd', '--id', 'rbd-mirror.juju-cf9afe-8-lxd-0', 'mirror', 'pool', 'enable', 'cinder-ceph-az1-arbor', 'pool']' returned non-zero exit status 95.

[Vern Hart (vern)]

Subscribed field-medium to see if we can get this change (or something like it) into the charm.

[Aurelien Lourot (aurelien-lourot)]

Hi Vern, thanks for reporting! Could you please provide a juju deployment bundle (so we can see which config options are being used) and are there any additional steps to reproduce other than just deploying? Thanks a lot!

[Vern Hart (vern)]

I have attached the sanitized bundle for this deployment. We have two ceph clusters, which is necessary for ceph-rbd-mirror, obviously, but they are in a single model. I don't believe this is germane to this bug so two models with cross model relations should also be sufficient.

We have multiple cinder-ceph charms deployed here but only one with pool-type=erasure-coded should be necessary.

Once the cinder-ceph charm creates its pool, rbd-mirror will try to mirror it and the above error will show up in the /var/log/ceph/ceph-client.rbd-mirror.*.log file.

Alternately, one could manually create a ceph pool with erasure coding and not need cinder-ceph at all.

[James Page (james-page)]

The lack of support for erasure coded pools is in the rbd-mirror application itself and is noted in the charm deployment guide section for use of erasure coded pools:

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-erasure-coding.html

[James Page (james-page)]

(your proposed fix to exclude any pools using erasure coding makes sense)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/784630
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.