Nova qemu hypervisor host smbios serial number is leaked to guest

[Tristan Cacqueray]

Erwan Velu from eNovance reported a vulnerability in OpenStack Nova.

The hypervisor is passing host system uuid (-smbios version) to guests, and this happen to be a critical info leak.
The defect have been pinpointed to:
 https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3054

From a simple virtual machine, this may allow numerous info leak like:
    Allow compute hardware enumeration from guests
    Deduce service tag and get all hardware configuration
    Ability to know if two instances are on the same compute

Dell hardware is particulary impacted as :
    - the uuid encodes the service tag
    - the service tag can be used on support site to determine:
    - detailled hardware configuration
    - date & country where the hw was shipped
    - date & type of support contract
    - amount of servers bought during this shipment

If there is no use case for this, we should scrambled that piece of information.

[Tristan Cacqueray (tristan-cacqueray)]

Here is a fix for master, thanks to Vladik

[Erwan Velu (erwan-t)]

Any input or remarks on this bug ? Sounds like an important leak isn't it ?

[Thierry Carrez (ttx)]

Looks OSSA-worthy, although could be considered a Dell-specific issue. If the fix is backportable we'll do one.

[Thierry Carrez (ttx)]

Could we get a bit of nova-coresec review on that proposed patch ?

[Andrew Laski (alaski)]

The fix looks sane to me and would address the particular issue raised here. And it should be backportable though some methods may have changed since Icehouse. I would like to get a second opinion though.

[Daniel Berrange (berrange)]

Exposing the host UUID to guests is an intentional step whose goal is to allow OS/software inventory programs and/or license validation programs to track the deployment of instances on hosts. Replacing this will a randomly generated UUID is certainly not a viable approach. If they wish to, admins can replace the host UUID with a different static uuid by setting the 'host_uuid' parameter in /etc/libvirt/libvirtd.conf

That Dell allows you to obtain so much sensitive information from an identifier is never considered or intended to be a secret is a flaw in itself.

[Erwan Velu (erwan-t)]

So you consider we shall keep the same uuid exposed to the VMs of a single hypervisor but maybe faking the UUID at the host level only for the Dell Servers ?

I did proove that Dell is putting some interesting data in their UUID that can be seen from a single VM but there is a great chance some other vendors does it too.

Decoding some Lenovo UUID shows they always use the same clock and keep the payload change across servers. But I don't know yet what's inside.

So can we consider generating a once-for-all random UUID for the host that is exposed to the VMs ? I mean it should be done the same for all servers brands and in an automatic fashion to avoid any possible leak of data. We cannot expect people knowing they are exposing data by running simple VMs.

[Tristan Cacqueray (tristan-cacqueray)]

@Daniel Berrange Having a same host uuid for every guest does not look like a good idea to me...

Though, this is more likely a Dell issue and OpenStack already offers an override mechanism so we may step down on this one after all. I'm adding ossg-coresec group for additional inputs if this still makes sense...

[Daniel Berrange (berrange)]

@Truistan: the host uuid is *not* used to populate the guest UUID. Every guest still has their own unique UUID populated in the SMBIOS system UUID field. The host UUID is provided separately for the purposes I desribe above.

[Robert Clark (robert-clark)]

Very interesting, it seems everyone is agreed that this is a significant issue but there's a little discussion regarding the patch, about which I have some thoughts too!

Rather than generating completely random UUIDs, is there any value in taking the system generated UUID, hashing it and providing that upwards as required? This way the same hosts will always provide the same information but any sensitive information should be masked in a way that makes in computationally unfeasible to reverse (subject to an appropriate hashing algorithm being chosen)?

[Tristan Cacqueray (tristan-cacqueray)]

@Robert Thanks you for joining in!

Hashing the UUID would be a good middle-ground solution but there may be concern about migration for OS/Software inventory programs...

[Daniel Berrange (berrange)]

I don't think hashing is too great. The point of using a standard UUID from the host is that it gives integration with 3rd party software which is already tracking hardware inventory based on UUID. If we hash the host UUID by default then any software needs to now be openstack-aware.

As mentioned earlier, libvirt already provides a mechanism for setting a different host UUID that would at least mean all applications using libvirt would be consistent within themselves. Common provisioning tools like PackStart, Triple-O should/could set this when they deploy libvirt and see that it is running on Dell hardware.

[Nathan Kinder (nkinder)]

We need the UUID to be consistent for all guests running on a particular host for the reasons that Daniel mentions, which the proposed patch would break. I also have concerns about using some format other than an actual UUID, which would be a problem if we just used a hash of the real UUID.

This could be addressed as an OSSN recommending that one generate their own UUID and configure it in /etc/libvirt/libvirtd.conf. I do agree that deployment tools should be responsible for setting these UUIDs instead of putting the burden on the deployer themselves.

[Erwan Velu (erwan-t)]

@nkinder We now know that this could be a serious leak not to hide the UUID to the VM.

Could we agree that we shall generate automatically a new host_uuid if a Dell host is detected (and only once of course to keep a single uuid exposed to VMs) ?
Letting this task to some external tools means that many will let that unconfigured generating the current leak situation.

Even if 3rd party configuration can do that, it sounds safer for me to make this automatic, insuring a better protection to our users.

BTW, don't you consider that being in position to estimate if several VMs are running on the same host could be an interesting information to some attacks ? For any public cloud provider, that could be an issue isn't it ? We can count the number of hypervisors etc..

My 2 cents,

[Robert Clark (robert-clark)]

I think that if you go down this route you'll end up with little bits of bespoke code for every vendor, I doubt that it's only Dell that suffers from such issues.

If we are generating a new UUID, how is that better than hashing the _real_ one? I suppose it would make it fit the correct format etc...?

[Daniel Berrange (berrange)]

On Linux hosts using systemd, there is a new standard OS level UUID defined in /etc/machine-id. Since basically every major Linux distro has switched or is about to switch to systemd, I think it is valid to consider use of the machine-id as an alternative source of UUIDs. Much as I hate to suggest adding nova config options, I think we should add a 'host_uuid_source' parameter to nova.conf taking values:

 - 'hardware' - use the UUID of the host hardware, eg from SMBIOS on x86 or equivalent. ie what we do today
 - 'os' - use the operating system UUID, eg /etc/machine-id on Linux systems.
 - Not specified - try to use 'os', fallback to 'hardware'

That would ensure we're safe on all future Linux distros. For people who have deployed on traditional Linux hosts pre-dating systemd, they can still generate a new UUID for libvirtd.conf.

[Nathan Kinder (nkinder)]

@robert-clark
I absolutely agree that we shouldn't do something Dell specific for the reasons you point out. The behavior should be the same regardless of hardware vendor.

@erwan-t
The information leak is definitely a concern. The ability to tell that multiple hypervisors are running on the same host is an interesting case too, which may be something that a deployer does not want to expose.

I think the ideal solution would be to expose as little real information as possible by default, while still allowing for the real hardware UUID to be exposed to instances if desired. Chaging the default behavior for existing releases is difficult though, as there is a possibility that someone is relying on the current behavior (passing the UUID through).

If we don't want to rely on the libvirt configuration for setting a UUID, we will likely need to have some new configuration in nova itself as suggested by Daniel. It may make sense to have a 'random' source setting that generates a new UUID when nova is started in addition to the 'hardware' and 'os' sources that were recommended. Adding new configuration isn't really acceptable for existing stable releases though.

As a short term solution for stable releases, I see the following options:

- Consider this as a vulnerability and modify nova's behavior to generate a random UUID (or hash the system UUID). This may break behavior that people are relying on, and they would have no way around it without a new configuration option. The configuration option could be added as an exception to the stable update policy if we think it important enough.

- Consider this as a potentially dangerous information exposure and recommend a workaround of using the libvirt config in an OSSN, while fixing this with new configuration options for future releases. The downside is that people may miss/overlook the OSSN and be vulnerable to unwanted information exposure.

What are others opinions between these two approaches?

[Erwan Velu (erwan-t)]

I do like the @berrange option to solve this issue. That offer the flexibility required by some while providing a "default" protection for the others.

I would have added another option to host_uuid_source. The "random" one which cover the behaviour that was proposed initially in this thread. Some operators will not like letting customers being able to count the number of HV and locate the VM on them. Adding this option would generate a random UUID for each started VM.

This way we cover all the cases reported in this thread.

To summarize the possible values of host_uuid_source:
 - 'hardware' - use the UUID of the host hardware, eg from SMBIOS on x86 or equivalent. ie what we do today
 - 'os' - use the operating system UUID, eg /etc/machine-id on Linux systems.
- 'random' - generate a random UUID for every single VM
 - Not specified - try to use 'os', fallback to 'hardware'

About the transition for existing platforms we surely have to come through the OSSN. The matter is that it will take long before platforms gets into systemd. If we do have the patch for getting the /etc/machine-id used in host_uuid_source, wouldn't it be acceptable to generate one if the file isn't available ?
- On the systems that runs systemd, we do have the proper behavior
- On legacy systems, we generate the expected file and gain the expected behavior from the host_uuid_source paramater

I'm not a great expert of this code but the patch sounds pretty light isn't it ?

[Daniel Berrange (berrange)]

Every VM already gets it own random UUID. There is no point in generating yet another random UUID to represent the host since it defeats the point of having this information there in the first place - a random host UUIUD is useless for its intended purpose.

[Erwan Velu (erwan-t)]

I do agree for the UUID but as the serial of thoses VMs is made of the host uuid, that still allow simple users being able to count the hypervisors and discover the colocation of VMs.

If you want to DDOS the platform or attack an HV, that's a very interesting information.

That's where the fully random mode would solve this issue by providing a unique serial number per VM.

[Daniel Berrange (berrange)]

This patch introduces a new configuration parameter to control the data exposed in the 'serial' field of guest SMBIOS. The impact of applying this patch

 - Distros providing /etc/machine-id -> fixed with no additional work needed by admin beyond restarting nova
 - Other distros - admin must either set 'host_uuid' in /etc/libvirt/libvirtd.conf & restart libvirtd or run 'uuidgen > /etc/machine-id' and restart Nova

Those clouds wishing to be paranoid against the so called 'hypervisor counting' problem can set the config parameter to 'none' to disable data entirely. I don't believe "hypervisor counting" is sufficiently serious to disable this by default.

It explicitly does *not* introduce any 'random' UUID value since this has no value over the existing random UUID reported in another SMBIOS field. Again the 'none' value should simply be used to disable reporting instead.

Those wishing to maintain back compat with existing deployed behaviour can force the setting to be 'hardware'

[Thierry Carrez (ttx)]

Note that the "new config option" approach (which I think is the right choice to fix this issue) is not really backportable to stable/*, so it's not appropriate for an OSSA patch.

But then I think any fix for this would introduce a behavior change that is not welcome in stable/* -- so we should probably just fix this in master, make the patch available for backport for those who wish to carry it in their deployments, and (maybe) issue an OSSN about it to raise awareness around the issue.

Thoughts?

[Robert Clark (robert-clark)]

We could certainly consider it for an OSSN

[Tristan Cacqueray (tristan-cacqueray)]

@ttx, I agree with your comment #22, sounds like the better way to take care of this.

[Thierry Carrez (ttx)]

OK, I propose we open this publicly so that iteration on the juno patch can go public on Gerrit. Unless someone complains, will open on Thursday.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113311

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113311
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4431eec1c94c4a353b45e5d873854b3fb1eaa11b
Submitter: Jenkins
Branch: master

commit 4431eec1c94c4a353b45e5d873854b3fb1eaa11b
Author: Daniel P. Berrange <email address hidden>
Date: Mon Jul 28 15:15:44 2014 +0100

    libvirt: make sysinfo serial number configurable

    The 'serial' field in guest SMBIOS tables gets populated
    based on the libvirt reported UUID of the host hardware.
    The rationale is to allow correlation of guests running
    on the same host.

    Unfortunately some hardware vendors use a subset of the
    host UUID as a key for retrieving hardware support
    contract information without requiring any authentication.
    So exposing the host UUID to the guest is an information
    leak for those vendors. It is possible to override the
    use of SMBIOS data by libvirt in /etc/libvirt/libvirtd.conf
    by setting the 'host_uuid' parameter.

    As a way to reduce the configuration burden though, it is
    preferrable to use the /etc/machine-id UUID, instead of
    the host hardware UUID. The former is a recent standard
    for Linux distros introduced by systemd to provide a UUID
    that is unique per operating system install. This means
    that even containers will see a separate /etc/machine-id
    value. This /etc/machine-id can be expected to be widely
    available in current and future distros. If missing, it
    is still possible to fallback to the libvirt reported
    host UUID.

    The host UUID exposed could theoretically be leveraged
    by a cloud user to get an approximate count of the number
    of unique hosts available to them in the cloud by launching
    many short lived VMs. Administrators concerned about this
    may wish to disable reporting of any sysinfo serial field
    at all.

    Introduce a 'sysinfo_serial' config parameter to the libvirt
    driver to control behaviour, accepting values:

     - 'auto' - try /etc/machine-id, fallback to
       libvirt reported host UUID (new default)
     - 'hardware' - always use libvirt host UUID (old default)
     - 'os' - always use /etc/machine-id, error if missing
     - 'none' - do not report any value to the guest

    DocImpact: new libvirt.sysinfo_serial config parameter
    SecurityImpact
    Closes-bug: #1337349
    Change-Id: I7ba7dbd65e913a66efe35a1d6490a85bec8413da

[Nathan Kinder (nkinder)]

This issue has been published as OSSN-0028:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0028