Unable to create/delete volumes after series upgrade

[Giuseppe Petralia]

This is Openstack Mitaka that is currently going through series upgrade from trusty to xenial.

After xenial upgrade we have the following packages on cinder:

# dpkg -l | grep -E "cinder|ceph"
ii ceph-common 10.2.11-0ubuntu0.16.04.2 amd64 common utilities to mount and interact with a ceph storage cluster
ii cinder-api 2:8.1.1-0ubuntu3 all Cinder storage service - API server
ii cinder-backup 2:8.1.1-0ubuntu3 all Cinder storage service - Scheduler server
ii cinder-common 2:8.1.1-0ubuntu3 all Cinder storage service - common files
ii cinder-scheduler 2:8.1.1-0ubuntu3 all Cinder storage service - Scheduler server
ii cinder-volume 2:8.1.1-0ubuntu3 all Cinder storage service - Volume server
ii libcephfs1 10.2.11-0ubuntu0.16.04.2 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.11-0ubuntu0.16.04.2 amd64 Python libraries for the Ceph libcephfs library
ii python-cinder 2:8.1.1-0ubuntu3 all Cinder Python libraries

And cinder charm is at revision 303
We are using ceph behind cinder and ceph-mon is at revision 48

After upgrading series from trusty to xenial,
we were not able to create or delete Volumes and all remained in pending state. However no errors were logged in cinder logs.

Checking ceph auth caps for cinder client in ceph we found that caps were the following:

# ceph auth get client.cinder
exported keyring for client.cinder
[client.cinder]
        key = <REDACTED>
        caps mon = "allow r; allow command "osd blacklist""
        caps osd = "allow rwx"

While checking the same caps on a vanilla trusty/mistaka (w/o upgrades) caps are the following:

# ceph auth get client.cinder
exported keyring for client.cinder
[client.cinder]
 key = <REDACTED>
 caps mon = "allow rw"
 caps osd = "allow rwx"

We updated the caps to be the same as the vanilla trusty/mitaka with:

# ceph auth caps client.cinder osd "allow rwx" mon "allow rw"

and we are able again to create/delete volumes.

[Corey Bryant (corey.bryant)]

I've added charms.ceph to this bug. It looks like this code lives in charms.ceph around https://github.com/openstack/charms.ceph/blob/master/charms_ceph/utils.py#L1150