Guestagent config leaks rabbit password

[Mark Kirkwood]

A running guest vm has the guestagent service running. Included in this is the trave-guestagent.conf file. This contains (at least) the rabbit password.

It is pretty easy to extract this as an unprivileged user - given that the guest image is publicly available, it can be downloaded,
and (if needed) converted to raw and mounted. From this either:

- config can be immediately read if guestagent is pre-installed (or)
- rsync command and ip + location of config files can be gleaned from
the init script

In the second case it is then pretty easy to boot a vm on the
appropriate network and rsync the config files using the above gleaned
command(s) as required (e.g add keys to the previously downloaded trove
guest image, upload it to glance then run it directly from nova and ssh
in...).

I'm thinking that we need to setup the guestagent so it does *not* need to know this level of detail about the inner workings of Openstack.

[Mark Kirkwood (mark-kirkwood)]

I note a number of us have mentioned this issue (or parts of it) on the mailing list... in hindsight probably should have filed one of these 1st - but I suspect most of us figured that we were simply mistaken.

[Jeremy Stanley (fungi)]

Why is the guest image publicly available? It seems to me that's a big part of the issue right there. Is it necessary to make that image accessible by tenants who make use of Trove services? Or is there some (perhaps poorly or even undocumented) expectation that the image will be secured against unprivileged access?

As for the bug status, I agree that the mailing list thread makes it pointless to leave this bug embargoed. Also there was a previous Trove bug report very similar to this one about sensitive information on the instance filesystem, but we did not issue an advisory for it because tenants are not supposed to have any access to the instance filesystem at all (modulo exploiting information leaks in mysqld which might make it possible to read files I guess?).

[Mark Kirkwood (mark-kirkwood)]

I'm examining this using a devstack setup, however I think the issues you raise apply equally well to any trove installation.

So - image access: the way this appears to work is that the guest image *must* be available to the tenant/user creating the trove instance (I experimented with making it available to - say - admin only and instances cannot be created at all then). This automatically means the tenant can (with some playing about) read the rabbit password.

Even *if* we could protect the image, then I think the tenant/user can simply look at the running trove instance via the *cinder* api and snapshot the volume and mount it somewhere else to examine at leasure (I have not tried this yet, but see no reason why it would not work)...

 I notice Nikhil has just suggested using a separate rabbit server for trove in the email thread, so I guess this might mean a separate rabbit server (or perhaps rabbit users) per tenant? This seems to lead to an endless proliferation of... rabbits (ironic eh), and clearly (to me) seems simply wrong.

[Tristan Cacqueray (tristan-cacqueray)]

Beside having such information in the guest image, there is also the issue about instances having access to management network. The documentation (http://docs.openstack.org/admin-guide-cloud/content/section_create-datastore.html) is not very clear about having to deploy a special controller for Trove operations.

[Tristan Cacqueray (tristan-cacqueray)]

Subscribing ossg-coresec for further debunking. I think this issue better deserve an OSSN rather than an OSSA.
I vote for a B2 type of bug ( http://security.openstack.org/vmt-process.html#incident-report-taxonomy )

[Nikhil Manchanda (slicknik)]

So there are a couple of points I wanted to add here for more contextl:

1. The way Trove is deployed in devstack is to deploy Trove into the user's tenant directly and is inherently insecure. However it is the simplest way to deploy Trove and suffices for dev testing. For a production deploy, I'd suggest deploying Trove into its own tenant or even in its own private cloud. If Trove is deployed in its own Tenant, the Trove guest images are uploaded only for this tenant, and should NOT be made public. Also users cannot access cinder volumes in the Trove tenant. The Trove deployment docs are lacking in this regard, and this is identified as something that we need to work on updating in Liberty.

2. Also Trove should not use the same RabbitMQ server as the rest of the services (undercloud). The reason for this is that this RabbitMQ server is usually on bare metal, and the Trove services are all usually deployed as a workload in the cloud. Having a users' instances reach out from inside the cloud to something on bare-metal is not a good idea. Given Trove should use a separate RabbitMQ server for its use (one user is sufficient, not one user per tenant), it does not need access to the infrastructure RabbitMQ.

[Mark Kirkwood (mark-kirkwood)]

Right, and it looks to me that 2) above only really works if you have 1) i.e: a separate rabbit mq with one user is fine *if* you have made trove run its instances in its own tenant.

And 1) needs code changes in (or in addition to) trove/common.remote.py so that nova_client uses said tenant.

[Jeremy Stanley (fungi)]

It looks like this is a fairly significant implication of the way Trove is currently designed to operate. If the Trove developers commit to documenting a recommended secure deployment model using dedicated tenant and/or message queues, then we could probably address this with a security note. However, I don't foresee this getting fixed under embargo via a patch we can backport and for which we could issue an advisory.

[Tristan Cacqueray (tristan-cacqueray)]

This is a B2 type of bug ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ), I propose we close the OSSA task and open the bug next Monday, unless someone complain.

[Amrith Kumar (amrith)]

flwang asked questions about this on IRC today. I'll update the bug with the known avoidance and explanations on how to securely deploy trove.