Trusted vm can be powered on untrusted host

[Wei Wang]

This is related to the trusted compute.

I recently setup trusted compute pool in my company and have observed that although new trusted vm is not able to be launched from an untrusted host, but if an trusted vm that have launched earlier on a trusted host which is compromised later on, that VM can still be powered on.

1. Exact version of Nova/Openstack:
[root@grunt2 ~]# rpm -qa | grep nova
python-nova-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-network-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-compute-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-conductor-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-cells-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-api-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-console-2014.1.2-100+45c2cbc.fc20.noarch
python-novaclient-2.17.0-2.fc21.noarch
openstack-nova-cert-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-scheduler-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-objectstore-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-common-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-novncproxy-2014.1.2-100+45c2cbc.fc20.noarch
openstack-nova-doc-2014.1.2-100+45c2cbc.fc20.noarch

2. Relevant log files:
this is not a error, don't think logs will help..

3. Reproduce steps:

* create trusted compute pool with only one compute node
* create an trusted VM on that compute node
* compromise the trusted compute node by changing the boot order
* power on the trusted Vm created earlier.

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Can a Nova core confirm that report ?

[Wei Wang (wei-w-wang)]

Thanks Tristan. Is it something I need to do?

[Malini Bhandaru (malini-k-bhandaru)]

Boot order change compromise. Jimmy, Tan Lin, any comment?
If there is a re-boot, my understanding is the hashes change and if the whitelist did not change, the VM cannot launch.
Is the issue that existing VMs just come right back up on the host without re-check for VMs that are supposed to launch on
Trusted hosts?

[Nathan Reller (rellerreller)]

Can you confirm that the hashes have changed? If you make the boot order change you mentioned above and try to launch a trusted VM does that result in a failure to launch? I'm trying to confirm that this actual results in different hash and that PCR is checked against.

Does Trusted Pools do a re-check on machine reboot or is it only invoked with the scheduler?

[Malini Bhandaru (malini-k-bhandaru)]

Something to consider: how legitimate is this test scenario in a production cloud.
1) A host upgrade, boot order change -- this would be in the realm of admin privileges.
2) Should this be part of scheduled maintenance, more likely than not, the admin would have evacuated all virtual machines from said host prior to upgrade, in which case Nova scheduler would be involved and the VM would be migrated to another trusted host.

And something to consider with respect to security hardening:
1) On host re-boot, as part of initialization step, it would be good to determine if it still is trusted if earlier on it was a trusted machine.

[Wei Wang (wei-w-wang)]

The boot order change is the simplest way to make the Attestation server
think this specific compute node has been "compromised", and it worked.

Please be noted this vm was previously launched from the same host, which
were in trusted status. But, we expect the same VM not be able to powered
back on when the attestation has marked the host as untrusted per the
design.

On Mon, Jun 8, 2015 at 3:07 PM, Malini Bhandaru <<email address hidden>
> wrote:

> Something to consider: how legitimate is this test scenario in a
> production cloud.
> 1) A host upgrade, boot order change -- this would be in the realm of
> admin privileges.
> 2) Should this be part of scheduled maintenance, more likely than not, the
> admin would have evacuated all virtual machines from said host prior to
> upgrade, in which case Nova scheduler would be involved and the VM would be
> migrated to another trusted host.
>
> And something to consider with respect to security hardening:
> 1) On host re-boot, as part of initialization step, it would be good to
> determine if it still is trusted if earlier on it was a trusted machine.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1456228
>
> Title:
> Trusted vm can be powered on untrusted host
>
> Status in OpenStack Compute (Nova):
> New
> Status in OpenStack Security Advisories:
> Incomplete
>
> Bug description:
> This is related to the trusted compute.
>
> I recently setup trusted compute pool in my company and have observed
> that although new trusted vm is not able to be launched from an
> untrusted host, but if an trusted vm that have launched earlier on a
> trusted host which is compromised later on, that VM can still be
> powered on.
>
> 1. Exact version of Nova/Openstack:
> [root@grunt2 ~]# rpm -qa | grep nova
> python-nova-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-network-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-compute-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-conductor-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-cells-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-api-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-console-2014.1.2-100+45c2cbc.fc20.noarch
> python-novaclient-2.17.0-2.fc21.noarch
> openstack-nova-cert-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-scheduler-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-objectstore-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-common-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-novncproxy-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-doc-2014.1.2-100+45c2cbc.fc20.noarch
>
> 2. Relevant log files:
> this is not a error, don't think logs will help..
>
> 3. Reproduce steps:
>
> * create trusted compute pool with only one compute node
> * create an trusted VM on that compute node
> * compromise the trusted compute node by changing the boot order
> * power on the trusted Vm created earlier.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1456228/+subscriptions
>

[Nathan Reller (rellerreller)]

Does the design specify that the VM should not be powered back on when the system reboots? I ask because I think that is a different problem. With the scheduling problem there is an outside entity from Nova that can ask for attestation and enforce not giving the VM to an untrusted host.

In this use case Nova would be responsible (I'm asking here and not stating, so please clarify if wrong) for asking for attestation and enforcing not launching a VM. If it is then this is risky because Nova would report measurements and enforce the decision as to whether or not to launch the VM. If Nova has been compromised then it could ignore whatever response is from attestation service or ignore it completely.

I'm not sure what is in scope for requirements and design of trusted pools.

[Sylvain Bauza (sylvain-bauza)]

The TrustedFilter filter actually doesn't check by itself but rather calls the Attestation API to know if the destination host is correct or not. That way, it's just when the instance is scheduled that it goes to the scheduler, then finds an acceptable destination (so calls the Attestation API for each host to see if it's compromised or not) and then calls the corresponding compute node to spawn that VM.

Once the VM is spawned, the scheduler is no longer involved unless a migration, a resize or an evacuation is asked for that VM.
That means that having a valid host, running a VM, stopping it, compromising the host, then restarting the VM is something that Nova doesn't check, because it's not its responsibility.

To be clear, Nova doesn't want to support that feature.

[Wei Wang (wei-w-wang)]

Wow, seriously? Sounds like joke...
 On Jun 23, 2015 3:56 AM, "Sylvain Bauza" <email address hidden> wrote:

> The TrustedFilter filter actually doesn't check by itself but rather
> calls the Attestation API to know if the destination host is correct or
> not. That way, it's just when the instance is scheduled that it goes to
> the scheduler, then finds an acceptable destination (so calls the
> Attestation API for each host to see if it's compromised or not) and
> then calls the corresponding compute node to spawn that VM.
>
> Once the VM is spawned, the scheduler is no longer involved unless a
> migration, a resize or an evacuation is asked for that VM.
> That means that having a valid host, running a VM, stopping it,
> compromising the host, then restarting the VM is something that Nova
> doesn't check, because it's not its responsibility.
>
> To be clear, Nova doesn't want to support that feature.
>
> ** Changed in: nova
> Status: New => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1456228
>
> Title:
> Trusted vm can be powered on untrusted host
>
> Status in OpenStack Compute (Nova):
> Invalid
> Status in OpenStack Security Advisories:
> Incomplete
>
> Bug description:
> This is related to the trusted compute.
>
> I recently setup trusted compute pool in my company and have observed
> that although new trusted vm is not able to be launched from an
> untrusted host, but if an trusted vm that have launched earlier on a
> trusted host which is compromised later on, that VM can still be
> powered on.
>
> 1. Exact version of Nova/Openstack:
> [root@grunt2 ~]# rpm -qa | grep nova
> python-nova-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-network-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-compute-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-conductor-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-cells-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-api-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-console-2014.1.2-100+45c2cbc.fc20.noarch
> python-novaclient-2.17.0-2.fc21.noarch
> openstack-nova-cert-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-scheduler-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-objectstore-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-common-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-novncproxy-2014.1.2-100+45c2cbc.fc20.noarch
> openstack-nova-doc-2014.1.2-100+45c2cbc.fc20.noarch
>
> 2. Relevant log files:
> this is not a error, don't think logs will help..
>
> 3. Reproduce steps:
>
> * create trusted compute pool with only one compute node
> * create an trusted VM on that compute node
> * compromise the trusted compute node by changing the boot order
> * power on the trusted Vm created earlier.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1456228/+subscriptions
>

[Tristan Cacqueray (tristan-cacqueray)]

Sylvain, if the trusted computing feature of Nova doesn't prevent an instance to spawn on a compromised node, then maybe the feature should be removed, or at least have a clear mention about that behavior.

According to vulnerability taxonomy, this issue falls in the B2 class ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ), I've set the OSSA tasks to won't fix and added a new OSSN tasks.

[Sylvain Bauza (sylvain-bauza)]

Tristan, to be clear, instances are spawned correctly. The problem is not at the creation stage or if a migration/evacuation is done, but rather what happens if one host gets compromised for the running instances. That's where the flaw is.

[Sylvain Bauza (sylvain-bauza)]

Also, I raised the problem here and why I consider this filter should be removed from in-tree :
http://lists.openstack.org/pipermail/openstack-dev/2015-June/067766.html

Comments welcome.

[Doug Chivers (doug-chivers)]

It is unclear if this is OSSN worthy, as it may be expected (if sub-optimal) behaviour. To recap, the steps listed to reproduce described above:

* create trusted compute pool with only one compute node
* create an trusted VM on that compute node
* compromise the trusted compute node by changing the boot order
* power on the trusted Vm created earlier.

I am under the impression that the trust status is only checked at boot time, so unless the node is rebooted after changing the boot order, the attestation information available to trusted_filter will still say that the node is trusted, and therefore will allow the instance to start. However, if the node is rebooted, the attestation information should prevent that node from restarting - this should be verified, although Slynvain states on 2015-06-24 that instances are spawned correctly, which implies that nova is functioning as it should.

Clearly this is sub-optimal to boot instances on a compromised nova node, but I believe this is expected behaviour for a trusted boot system (it is one of the known defects with trusted computing) . If I am correct, we should document this via the compute section of the security guide, rather than issuing an OSSN. Any update to the security guide should include the expected situations that the attestation service should prevent instance boot, and the cases where instances can still boot.

[Will Auld (will-auld)]

I concur with Doug. This is not a bug but expected behavior. Another example to illustrate: Boot a node into the trusted pool. This node is later compromised while it is running several VMs and afterward continues to receive newly launched VMs. The attestation is attesting to the node booting up with known software not that it will remain uncompromised.

[Wei Wang (wei-w-wang)]

"I am under the impression that the trust status is only checked at boot time, so unless the node is rebooted after changing the boot order, the attestation information available to trusted_filter will still say that the node is trusted, and therefore will allow the instance to start. However, if the node is rebooted, the attestation information should prevent that node from restarting"
This is not true. Currently, tboot will still let the server boot up even if server has been compromised(say by changing the grub and get into single user mode then change root password and boot to normal mode). Attestation server will say this node is no longer trusted and openstack then can't launch a new VM on this cumpute node.

However, for the VMs that were already on this compromised node, openstack still let them start(power on).

To me, openstack does check with attestation server before it launch a new VM on a compute node, however if it does not do so when it power on a VM that are supposed to be on a trusted node, I don't really see the point of this design.

Maybe I am underestimated it, but isn't there a obvious fix, whatever checks performed before lunching a new trusted VM, we need to do the same for power on a trusted VM..

[Michael Xin (michael-xin)]

"Maybe I am underestimated it, but isn't there a obvious fix, whatever checks performed before lunching a new trusted VM, we need to do the same for power on a trusted VM.."
I agree your point. How to implement this? Will Nova be able to address this?

[Dave McCowan (dave-mccowan)]

The trusted check runs as a filter in Nova's scheduler. It only runs when a VM is launched and automatically scheduled. In addition to powering on, you can also migrate a trusted VM to an untrusted host after it is scheduled. The trust check involves sending a request to an external trust attestation service. The Nova team will likely not accept any patches that add a check like that to any other operation.

[Sylvain Bauza (sylvain-bauza)]

So, since the scheduler is only checked when booting or migrating a VM (for the latter, that's only partially true) and since by design, the Nova scheduler is not verifying the status for the existing VMs, IHMO I think that's this bug is Invalid.

If someone would want to write a feature getting this checked, it should be out of Nova, like in Congress or whatever else, but I don't think it's an OSSN, just a misunderstanding of what means a Nova Filter.

[Dave McCowan (dave-mccowan)]

+1 Nova Filter clarification should go in Nova's docs, not in an OSSN.

[Wei Wang (wei-w-wang)]

I understand that it has been conclude that nova is not the place to address this security issue from above discussions.. However documenting will not make this issue goes away, while it actually probable give a clue to people who really want to compromise a trusted host..

Can anyone shed some lights on, which component should perform such check so a VM that are supposed on trusted host only can't not be powered on on a compromised host, especially in this case attestation server has already figured out the hose is no longer trusted?

[Malini Bhandaru (malini-k-bhandaru)]

As I originally mentioned, the scenario described in this bug is very unlikely in a production environment, a host being compromised in a way detectable by secure boot or trusted boot requires a reboot, at which time those VMs would be down, most likely something that Heat along with some monitoring system would have detected and acted upon, such as restarting the VMs in a new location meeting their original trust requirements.

Yet another alternative would be a VM aware of its trust needs, able to test that it is running on a trusted host else emit an alarm message and shutdown. Congress or some other OpenStack project could detect the alarm and migrate the VM. Or Heat and a monitoring application as described in the previous paragraph.

Another approach is to combine secure boot with trusted boot. In this scenario, if the boot routines are not signed and certified the host will fail to launch. Trusted boot would then capture the launch measurements.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/194592
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e134536d0a7a555088f1af5d75d573ed50352f1a
Submitter: Jenkins
Branch: master

commit e134536d0a7a555088f1af5d75d573ed50352f1a
Author: Sylvain Bauza <email address hidden>
Date: Fri Sep 18 12:38:12 2015 +0200

    Set TrustedFilter as experimental

    The TrustedFilter is the only in-tree scheduler filter that calls an external
    3rd-party service (OpenAttestation) for decision-making. Thus, the OAT service
    is not listed as an official Nova dependency and consequently not even gated,
    even by a 3rd-party CI.

    Besides, some discussions have been captured in a ML thread that show that
    running this filter is not the best way for enforcing trusted compute nodes [1]
    but I leave that out of the review (just a FYI) because the main reason for
    making experimental the filter is to send a signal to operators that they will
    either have to find another solution or accept the current gaps.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2015-June/067766.html

    Related-Bug: #1456228

    Change-Id: I6ab013faf22a0e88424207830ec399724f827622

[Nathan Kinder (nkinder)]

This has been published as OSSN-0059:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0059