I was able to reproduce this bug, and break my openstack infrastructure.
Specifically, manila's access control allows three kinds of damage to existing
users:
1) "manila access-allow" *can* reset the "caps" (capabilities) of pre-existing
ceph users, thereby breaking any ceph user's non-manila workloads
2) "manila access-deny" can delete ceph users, even users that were not created
by "manila access-allow"
3) "manila access-list" can leak privileged/infrastructure users' access-keys:
in cases where pre-existing ceph capabilities are maintained, leaking ceph
keys through manila would mean users can wreak havoc on the ceph cluster.
lets break it down a bit more:
a) The CephFS driver in manila interacts with ceph via the python-cephfs
package, which is derived from a python module in the ceph repository [1].
When you allow access to a ceph client user, this volume client code
checks whether the client user exists, or it will go ahead and create one.
If the ceph client user exists, it checks whether there are any
pre-existing "mds" caps to carefully craft an update [2]. If there are no
mds caps,
the pre-existing caps are ignored and overwritten. This seems to be a
problem, because ceph client user may not have any mds caps, but may have
osd/mgr/mon caps. So, if we fix the check in [2], you will not have manila
breaking other users' pre-existing capabilities.
b) On removing access from a share (cephfs subvolume), if a ceph client user
has access to no other cephfs subvolume, the ceph-side code deletes the user.
This is again a huge problem, ceph client users may have non-manila
workloads and deleting the user will break those workloads.
c) There's a question of whether manila (or specifically the cephfs driver)
should be modifying users that it doesn't create at all.
It's possible that that's desired - you may have pre-created cephfs client
users, and would like them to use manila shares.
However, currently, there's no way to prevent anyone but the manila service
user [3] to be used for authentication.
There's two ways to achieve this Should we have a "denylist" of users that
are not allowed to be used for auth with manila?
We have two places where this could go:
- We can make a mutable configuration option in manila to specify a
list/regex of client users that the driver can prevent using for auth. This
gives an OpenStack administrator the power to protect their infra users.
- We can make ceph handle a denylist - I dunno how we can achieve this piece,
we can consult with ceph developers - this may be more dynamic, since we
can add or remove users from such a deny list without having to mess with
configuration.
Introducing a configurable denylist *somewhere* may prevent all three problems
highlighted. Does anyone think of any drawbacks to this approach?
Hello,
Thanks Jahson.
I was able to reproduce this bug, and break my openstack infrastructure.
Specifically, manila's access control allows three kinds of damage to existing
users:
1) "manila access-allow" *can* reset the "caps" (capabilities) of pre-existing infrastructure users' access-keys:
ceph users, thereby breaking any ceph user's non-manila workloads
2) "manila access-deny" can delete ceph users, even users that were not created
by "manila access-allow"
3) "manila access-list" can leak privileged/
in cases where pre-existing ceph capabilities are maintained, leaking ceph
keys through manila would mean users can wreak havoc on the ceph cluster.
lets break it down a bit more:
a) The CephFS driver in manila interacts with ceph via the python-cephfs
package, which is derived from a python module in the ceph repository [1].
When you allow access to a ceph client user, this volume client code
checks whether the client user exists, or it will go ahead and create one.
If the ceph client user exists, it checks whether there are any
pre-existing "mds" caps to carefully craft an update [2]. If there are no
mds caps,
the pre-existing caps are ignored and overwritten. This seems to be a
problem, because ceph client user may not have any mds caps, but may have
osd/mgr/mon caps. So, if we fix the check in [2], you will not have manila
breaking other users' pre-existing capabilities.
b) On removing access from a share (cephfs subvolume), if a ceph client user
has access to no other cephfs subvolume, the ceph-side code deletes the user.
This is again a huge problem, ceph client users may have non-manila
workloads and deleting the user will break those workloads.
c) There's a question of whether manila (or specifically the cephfs driver)
should be modifying users that it doesn't create at all.
It's possible that that's desired - you may have pre-created cephfs client
users, and would like them to use manila shares.
However, currently, there's no way to prevent anyone but the manila service
user [3] to be used for authentication.
There's two ways to achieve this Should we have a "denylist" of users that
are not allowed to be used for auth with manila?
We have two places where this could go:
- We can make a mutable configuration option in manila to specify a
list/regex of client users that the driver can prevent using for auth. This
gives an OpenStack administrator the power to protect their infra users.
- We can make ceph handle a denylist - I dunno how we can achieve this piece,
we can consult with ceph developers - this may be more dynamic, since we
can add or remove users from such a deny list without having to mess with
configuration.
Introducing a configurable denylist *somewhere* may prevent all three problems
highlighted. Does anyone think of any drawbacks to this approach?
[1] https:/ /github. com/ceph/ ceph/blob/ master/ src/pybind/ ceph_volume_ client. py /github. com/ceph/ ceph/blob/ a0e1a8f17372b36 1db68cc4994120e 729d8e484a/ src/pybind/ ceph_volume_ client. py#L1115- L1116 /opendev. org/openstack/ manila/ src/commit/ 22d6fe98a3f4377 09901fac4e4ec65 fec414f7d0/ manila/ share/drivers/ cephfs/ driver. py#L413- L417
[2] https:/
[3] https:/