ceph-rbd-mirror function tests fail on focal

[Liam Young]

Functional tests fail on focal. The initial deployment gets stuck with the ceph-rbd-mirror* units in a blocked state.

The first step to fixes this seems to be to add an extra permission:

# rbd --id rbd-mirror.juju-250193-zaza-2f6ad28ff600-20 mirror pool status --format json --verbose cinder-ceph
rbd: failed to query services: (13) Permission denied
{"summary":{"health":"UNKNOWN","daemon_health":"UNKNOWN","image_health":"OK","states":{}},"daemons":[],"images":[]}

Adding mon allow r gives:

# rbd --id rbd-mirror.juju-250193-zaza-2f6ad28ff600-20 mirror pool status --format json --verbose cinder-ceph
{"summary":{"health":"OK","daemon_health":"OK","image_health":"OK","states":{}},"daemons":[{"service_id":"6822","instance_id":"6858","client_id":"juju-250193-zaza-2f6ad28ff600-20","hostname":"juju-250193-zaza-2f6ad28ff600-20","ceph_version":"15.2.1","leader":true,"health":"OK"}],"images":[]}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/729170
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=59160b08151a4a8e5a743c141f45c529547254af
Submitter: Zuul
Branch: master

commit 59160b08151a4a8e5a743c141f45c529547254af
Author: Liam Young <email address hidden>
Date: Tue May 19 08:18:36 2020 +0000

    Sync charms.ceph

    Sync charms.ceph to pick up fix for LP #1851869

    Change-Id: Ie3027d030c71faec0108fcde7fdd624b29623d49
    Closes-Bug: #1851869
    Partial-Bug: #1879464

[Ryan Beisner (1chb1n)]

FYI, failing on the latest run @ https://review.opendev.org/#/c/717068/ with artifacts @ https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-rbd-mirror/717068/5/5855/index.html

ceph-rbd-mirror-b/0* waiting executing 20 172.17.110.19 Waiting for pools to be created
ceph-rbd-mirror/0* blocked idle 13 172.17.110.10 Unit is ready (Pools ERROR (1) OK (1) )

...

2020-05-20 23:11:45 INFO juju-log Pools: "{'cinder-ceph': {'applications': {'rbd': {}}, 'parameters': {'pg_num': '32', 'size': '3'}, 'quota': {}}, 'device_health_metrics': {'applications': {'mgr_devicehealth': {}}, 'parameters': {'pg_num': '1', 'size': '3'}, 'quota': {}}, 'glance': {'applications': {'rbd': {}}, 'parameters': {'pg_num': '128', 'size': '3'}, 'quota': {}}}"
2020-05-20 23:11:45 DEBUG update-status creating /var/lib/ceph/rbd-mirror/ceph.client.rbd-mirror.juju-ba0618-zaza-89869a5cef84-13.keyring
2020-05-20 23:11:45 DEBUG update-status added entity client.rbd-mirror.juju-ba0618-zaza-89869a5cef84-13 auth(key=AQDbscVeezgELxAAl5eTogKSgfJ3dykmZ1McTA==)
2020-05-20 23:11:45 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2020-05-20 23:11:46 DEBUG juju-log Changing permissions on existing content: 33184 -> 416
2020-05-20 23:11:46 INFO juju-log Invoking reactive handler: reactive/ceph_rbd_mirror_handlers.py:93:configure_pools
2020-05-20 23:11:46 INFO juju-log Invoking reactive handler: hooks/relations/ceph-rbd-mirror/requires.py:51:joined:ceph-local
2020-05-20 23:11:46 INFO juju-log Invoking reactive handler: hooks/relations/ceph-rbd-mirror/requires.py:51:joined:ceph-remote
2020-05-20 23:11:46 INFO juju-log Invoking reactive handler: hooks/relations/ceph-rbd-mirror/requires.py:55:changed:ceph-local
2020-05-20 23:11:46 INFO juju-log Invoking reactive handler: hooks/relations/ceph-rbd-mirror/requires.py:55:changed:ceph-remote
2020-05-20 23:11:46 INFO juju-log Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
2020-05-20 23:11:46 DEBUG juju-log Running _assess_status()
2020-05-20 23:11:47 DEBUG juju-log mirror_pools_summary = "{'pool_health': defaultdict(<class 'int'>, {'ERROR': 1, 'OK': 1}), 'image_states': defaultdict(<class 'int'>, {})}"

...

2020-05-20T22:40:53.600+0000 7fabe6ae8500 0 ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable), process rbd-mirror, pid 18606
2020-05-20T22:40:53.624+0000 7fabe6ae8500 1 mgrc service_daemon_register rbd-mirror.4917 metadata {arch=x86_64,ceph_release=octopus,ceph_version=ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable),ceph_version_short=15.2.1,cpu=Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz,distro=ubuntu,distro_description=Ubuntu 20.04 LTS,distro_version=20.04,hostname=juju-ba0618-zaza-89869a5cef84-13,id=juju-ba0618-zaza-89869a5cef84-13,instance_id=4917,kernel_description=#33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020,kernel_version=5.4.0-29-generic,mem_swap_kb=0,mem_total_kb=2035420,os=Linux}
2020-05-20T22:41:53.724+0000 7fabe6ae8500 -1 rbd::mirror::PoolReplayer: 0x55f357b49200 init: error accessing remote poo...