Ceph credentials included in logs using older libvirt/qemu

[Matthew Booth]

Older versions of libvirt included network storage authentication information on the qemu command line. If libvirt raises an exception which logs the qemu command line it used, for example an error starting a domain, this authentication information will end up in the logs. There is an existing CVE for this issue here:

  https://access.redhat.com/security/cve/CVE-2015-5160

Specifically, if a deployment is using ceph, a libvirt error starting a domain would log the cephx secret key and the monitor addresses on the qemu command line.

The issue has been resolved upstream. Users running qemu version 2.6 or later, and libvirt version 2.2 or later, are not vulnerable. No change is required in Nova to resolve this issue.

Red Hat users running RHEL 7.3 or later are not vulnerable.

It's not 100% clear to me that an OpenStack CVE is required here as it's not a bug in an OpenStack component, and it's already fixed upstream. However, it did come to my attention after a user publicly posted their ceph credentials on IRC, so evidently some OpenStack users are running vulnerable systems, and this is a very common configuration.

In Nova, we currently have:

MIN_LIBVIRT_VERSION = (1, 2, 9)
MIN_QEMU_VERSION = (2, 1, 0)

so anybody running the minimum supported versions will be vulnerable.

[Matthew Booth (mbooth-9)]

Incidentally, Dan Berrangé points out that the libvirt and qemu bugs were never embargoed as the issues had been public knowledge for a long time.

[Daniel Berrange (berrange)]

Public way back to day 1 https://www.redhat.com/archives/libvir-list/2011-November/msg00853.html

and I'm fairly sure they've been discussed on openstack bugs / mailing lists / irc several times before, though I don't recall specific links.

[Daniel Berrange (berrange)]

That all said, the scope of the libvirt / qemu bug was a local user information leak.

If Nova turns that compute-node local info leak, into a remote user information leak by publicizing the error message somewhere, that could be considered a genuinely new security bug in nova.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Given the public nature of the issue, I'm inclined to switch this report to public. It doesn't sound like there's much point to an embargo here unless it's possible for unprivileged users to intentionally trigger an exception (in which case the value may be leaked in Nova's service logs?).

Since a dependency upgrade mitigates the issue and it sounds like there won't be any associated Nova patches, I'm inclined to treat this as C2 in our taxonomy (a vulnerability, but not in OpenStack supported code, e.g., in a dependency, https://security.openstack.org/vmt-process.html#incident-report-taxonomy ) and recommend an OSSN letting consumers of Nova know they should upgrade to at least qemu 2.6 and libvirt 2.2 (or backport the relevant fixes to their qemu/libvirt versions). I'm subscribing the ossg-coresec team for input on this as well.

[Daniel Berrange (berrange)]

One further thing - the same issue affects the in-qemu iscsi client, and has *not* yet been fixed because the neccessary QEMU fix was only released last week in qemu 2.9. While nova has code for the in-qemu iSCSI client, I don't think that is actually possible to be enabled any more - it was never the default either since it lacked multi-path support.

[Matthew Booth (mbooth-9)]

I don't see any immediate reason to keep this private. Ideally it wouldn't be possible for a user to trigger an error like this, although in practise there will be bugs. However, even when triggered I don't think we ever intentionally expose stack traces to non-admin users. In that case, I don't think this is a new issue and we should just warn users as you say.

I'm going to request a second opinion on whether users should ever be able to see stack traces.

[Matt Riedemann (mriedem)]

We don't intentionally expose stacktraces to the user, and don't consider the logs for the user. Nova records exception traces via instance faults for some operations, but we only show those details for an admin context:

https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L301

[Tristan Cacqueray (tristan-cacqueray)]

I agree with above comments, this sounds like a C2 class. Unless there is an objection let's open this bug.

[Matthew Booth (mbooth-9)]

Ubuntu Xenial has libvirt 1.3.1 and qemu 2.5, so unless these packages have backported fixes (I haven't checked), Ubuntu users will be vulnerable.

[Sean Dague (sdague)]

Making this public, all the issues in here have been publicly discussed in the libvirt/qemu projects, and being public at least makes people more aware of the potential leak.

[Jeremy Stanley (fungi)]

Triaged as class C2 (a vulnerability, but not in OpenStack supported code, e.g., in a dependency; won't fix OSSA; potential OSSN) based on prior discussion in this bug report.

[Sean Dague (sdague)]

I don't think there is really a Nova fix here, just an FYI