Hello, 

Thanks Jahson.

I was able to reproduce this bug, and break my openstack infrastructure.
Specifically, manila's access control allows three kinds of damage to existing
users:


1) "manila access-allow" *can* reset the "caps" (capabilities) of pre-existing
   ceph users, thereby breaking any ceph user's non-manila workloads 
2) "manila access-deny" can delete ceph users, even users that were not created
   by "manila access-allow"
3) "manila access-list" can leak privileged/infrastructure users' access-keys:
   in cases where pre-existing ceph capabilities are maintained, leaking ceph
   keys through manila would mean users can wreak havoc on the ceph cluster.

lets break it down a bit more:

a) The CephFS driver in manila interacts with ceph via the python-cephfs 
   package, which is derived from a python module in the ceph repository [1].
   When you allow access to a ceph client user, this volume client code 
   checks whether the client user exists, or it will go ahead and create one.
   If the ceph client user exists, it checks whether there are any 
   pre-existing "mds" caps to carefully craft an update [2]. If there are no
   mds caps,
   the pre-existing caps are ignored and overwritten. This seems to be a 
   problem, because ceph client user may not have any mds caps, but may have
   osd/mgr/mon caps. So, if we fix the check in [2], you will not have manila
   breaking other users' pre-existing capabilities.
b) On removing access from a share (cephfs subvolume), if a ceph client user 
   has access to no other cephfs subvolume, the ceph-side code deletes the user. 
   This is again a huge problem, ceph client users may have non-manila 
   workloads and deleting the user will break those workloads. 
c) There's a question of whether manila (or specifically the cephfs driver) 
   should be modifying users that it doesn't create at all. 
   It's possible that that's desired - you may have pre-created cephfs client 
   users, and would like them to use manila shares.
   However, currently, there's no way to prevent anyone but the manila service 
   user [3] to be used for authentication. 
   There's two ways to achieve this Should we have a "denylist" of users that 
   are not allowed to be used for auth with manila?
   We have two places where this could go:
   - We can make a mutable configuration option in manila to specify a
     list/regex of client users that the driver can prevent using for auth. This
     gives an OpenStack administrator the power to protect their infra users.
   - We can make ceph handle a denylist - I dunno how we can achieve this piece,
     we can consult with ceph developers - this may be more dynamic, since we
     can add or remove users from such a deny list without having to mess with
     configuration.

Introducing a configurable denylist *somewhere* may prevent all three problems
highlighted. Does anyone think of any drawbacks to this approach?

[1] https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py 
[2] https://github.com/ceph/ceph/blob/a0e1a8f17372b361db68cc4994120e729d8e484a/src/pybind/ceph_volume_client.py#L1115-L1116
[3] https://opendev.org/openstack/manila/src/commit/22d6fe98a3f437709901fac4e4ec65fec414f7d0/manila/share/drivers/cephfs/driver.py#L413-L417