[OSSA-2017-005] nova rebuild ignores all image properties and scheduler filters (CVE-2017-16239)

[George Shuklin]

Big picture: If some image has some restriction on aggregates or hosts it can be run on, tenant may use nova rebuild command to circumvent those restrictions. Main issue is with ImagePropertiesFilter, but it may cause issues with combination of flavor/image (for example allows to run license restricted OS (Windows) on host which has no such license, or rebuild instance with cheap flavor with image which is restricted only for high-priced flavors).

I don't know if this is a security bug or not, if you would find it non-security issue, please remove the security flag.

Steps to reproduce:

1. Set up nova with ImagePropertiesFilter or IsolatedHostsFilter active. They should allows to run 'image1' only on 'host1', but never on 'host2'.
2. Boot instance with some other (non-restricted) image on 'host2'.
3. Use nova rebuild INSTANCE image1

Expected result:

nova rejects rebuild because given image ('image1') may not run on 'host2'.

Actual result:

nova happily rebuild instance with image1 on host2, violating restrictions.

Checked affected version: mitaka.

I believe, due to the way 'rebuild' command is working, newton and master are affected too.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Matt Riedemann (mriedem)]

I believe the issue is that we bypass the scheduler filters because in the compute API for rebuild we pass the existing host for the instance to conductor here:

https://github.com/openstack/nova/blob/93bf6ba5186a3663606aa843a2f247709173f073/nova/compute/api.py#L3015

And then in the conductor manager rebuild_instance method, this "if not host" line is False because the compute API provided the host (the current host the instance is running on):

https://github.com/openstack/nova/blob/93bf6ba5186a3663606aa843a2f247709173f073/nova/conductor/manager.py#L698

And then we happily cast to nova-compute to rebuild the instance with the new image, bypassing the filter checks:

https://github.com/openstack/nova/blob/93bf6ba5186a3663606aa843a2f247709173f073/nova/conductor/manager.py#L757

If the compute API didn't provide the host (like in the case of evacuate), then we'd go through the scheduler to pick a host based on the filters:

https://github.com/openstack/nova/blob/93bf6ba5186a3663606aa843a2f247709173f073/nova/conductor/manager.py#L722

In the case of evacuate, this is where we make sure to pass host=None to conductor's rebuild_instance method so that we go through the scheduler:

https://github.com/openstack/nova/blob/93bf6ba5186a3663606aa843a2f247709173f073/nova/compute/api.py#L3978

I'm not sure if this is a security vulnerability per se, but it's definitely a bug. Sylvain Bauza is probably someone that should look at this.

[Matt Riedemann (mriedem)]

I may be naive here, but the fix seems rather trivial. In rebuild_instance in conductor manager, in the rebuild case (where host is not None), we could call _schedule_instances and assert that the provided host must be in the returned list, else we can't rebuild the instance on that host with that image.

[Sylvain Bauza (sylvain-bauza)]

Yeah, the bug is latent since https://github.com/openstack/nova/commit/d6ca1cc02ab58615a6bb1b337db34dd06525112e

Basically, we accepted to not verify the scheduler in case of a rebuild because it was staying with the compute node. What we missed is that in case people pass a different image, then it could be something not acceptable for the scheduler.

Is that a security issue ? Probably. Should it be embargoed ? Not sure, but I leave other commenting that.

About the possible solution, I think we could easily mimic the evacuate behaviour and use the RequestSpec.requested_destination field for calling the scheduler but only for that specific host.

See https://github.com/openstack/nova/blob/93bf6ba5186a3663606aa843a2f247709173f073/nova/compute/api.py#L3977-L3994 where we set that field if the operator doesn't want to force the target.

[Sylvain Bauza (sylvain-bauza)]

Here is the possible solution for that bug.

[George Shuklin (george-shuklin)]

About severity of the issue: I've just imaged scenario where this would lead to compromise.

Let's says we got some PCI-passthrough installation which relies on image-level (operating system level) permissions to allow or disallow some operations with hardware. Tenants are allowed to use images with properly configured permissions system and they are forbidden to run random (untrusted/unverified) code on those nodes.

By issuing 'rebuild' command with specially crafted image malicious tenant may compromise hardware by means of bypassing image-level security and Openstack's restriction on which images are allowed to run on the node with sensitive hardware.

I don't think that this is a common use case, but who knows whom Stuxnet would use this bug on whom nuclear factory.

[George Shuklin (george-shuklin)]

One more thing I thought about: What should nova do during 'nova rebuild' command if this host's nova-compute is marked as 'disabled' by system administrator?

Should it allows to rebuild or should it rejects?

[Sylvain Bauza (sylvain-bauza)]

George, that's something optionable by operators : if they disabled ComputeFilter in their nova.conf, then the scheduler stops only accepting active computes, and can return disabled nodes too.

Nothing on a case-by-case tho, that would require an admin-only API modification hence a blueprint to address that.

[Tristan Cacqueray (tristan-cacqueray)]

This looks like a class A type of bug according to VMT's taxonomy ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ), though I'm in favor for fixing this in the open since the impact sounds limited.

Impact description draft:

Title: Nova Filter Scheduler bypass through rebuild action
Reporter: George Shuklin (servers.com)
Products: Nova
Affects: >=13.0.0 <=13.1.3, >=14.0.0 <=14.0.4, ==15.0.0

Description:
George Shuklin from servers.com reported a vulnerability in Nova.
By rebuilding a malicious instance, an authenticated user may be able to bypass
Filter Scheduler resulting in restrictions violation such as the
ImagePropertiesFilter and the IsolatedHostsFilter. All setups using Nova
Filter Scheduler are affected.

[Jeremy Stanley (fungi)]

I agree that if these controls are relied on by some deployments as a security measure, then bypass by users does constitute a vulnerability.

In light of the Nova stable/ocata point release which was tagged a few hours after your impact description draft, the affects line should now be...

    Affects: >=13.0.0 <=13.1.3, >=14.0.0 <=14.0.4, >=15.0.0 <=15.0.1

The middle sentence of the description is also a little hard for me to follow. How about...

    By rebuilding an instance, an authenticated user may be able to
    circumvent the Filter Scheduler bypassing imposed filters (for
    example, the ImagePropertiesFilter or the IsolatedHostsFilter).

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the review, here is the updated impact description draft:

Title: Nova Filter Scheduler bypass through rebuild action
Reporter: George Shuklin (servers.com)
Products: Nova
Affects: >=13.0.0 <=13.1.3, >=14.0.0 <=14.0.4, >=15.0.0 <=15.0.1

George Shuklin from servers.com reported a vulnerability in Nova. By rebuilding an instance, an authenticated user may be able to circumvent the Filter Scheduler bypassing imposed filters (for example, the ImagePropertiesFilter or the IsolatedHostsFilter). All setups using Nova Filter Scheduler are affected.

I'm still on the fence to keep this under embargo if the filter scheduler aren't a common use-case, should we subscribe ossg-coresec to weigh on this (as explained here: https://security.openstack.org/vmt-process.html#embargo-exceptions ) ?

Otherwise we need coresec review on the proposed patch in comment #7 and backport to mitaka, newton and ocata, in particular since the patch doesn't apply cleanly on mitaka.

[Matt Riedemann (mriedem)]

Note that there is another public bug for the same issue, but it doesn't mention a possible security impact:

https://bugs.launchpad.net/nova/+bug/1667667

[Matt Riedemann (mriedem)]

> I'm still on the fence to keep this under embargo if the filter scheduler aren't a common use-case

Where are we saying that the filter scheduler is not a common use case? The filter scheduler is the scheduler driver we use in nova by default:

https://github.com/openstack/nova/blob/be6146657fee7e48d77451108345834abd8d2fbf/nova/conf/scheduler.py#L59

[Tristan Cacqueray (tristan-cacqueray)]

Matt, I meant a common use-case to secure hardware repartition against aggressive tenant as described in comment #8.

[Tristan Cacqueray (tristan-cacqueray)]

Re comment #13, any progress nova-coresec?

[Matt Riedemann (mriedem)]

There haven't been any updates on this bug. It's a known issue, we just haven't had someone push a fix yet.

[George Shuklin (george-shuklin)]

I'd like to point that this bug is already more than half-year old. Is there any meaning to have 'private bug report' when bug wouldn't be fixed for two duration of openstack release cycles? I think if this bug had been disclosed as normal, it would have been fixed in ocata.

[Jeremy Stanley (fungi)]

George: I'm in full agreement with you there. I've been suggesting for a while that the OpenStack VMT should set a hard limit on the acceptable duration of report embargoes and switch bugs to public security type for wider assistance if not resolved in that timeframe. There has been pushback on that philosophy in the past, but it's certainly worth bringing up again.

I've also subscribed the security team's advisory group to give them a heads up that there is interest in switching this report to our public security workflow, and get their input.

[Jeremy Stanley (fungi)]

George: It's also worth mentioning that if you, as the reporter of this issue, prefer for it to be made public, that's fine. The OpenStack VMT typically follows the reporter's wishes in such matters.

[Matt Riedemann (mriedem)]

I'm working on a recreate of the bug in a functional test within the nova repo, then will apply Sylvain's change on top of that to verify it fixes the bug. Then we can make this public.

[Matt Riedemann (mriedem)]

@Sylvain, your patch won't work because of this in the conductor rebuild_instance method:

https://github.com/openstack/nova/blob/6e0e2460b3e1f422b822744ea34da87590e38f39/nova/conductor/manager.py#L838

That's going to modify the request_spec to tell the scheduler to ignore the instance.host, which is exactly where we want to go if we're doing a rebuild (not evacuate).

[Matt Riedemann (mriedem)]

I've got a local patch that I'll be posting shortly in here. It adds a functional recreate of the bug and the fix on top. It has a few TODOs in it that need to be resolved, and will require some review from Sylvain, but it's mostly there. Should have it attached by EOD.

[Matt Riedemann (mriedem)]

Here is the patch with the recreate test and fix.

There are some TODOs in here so this is for initial review by nova core-sec.

Also note that the test changes could be a bit simpler if I changed some of the fixture usage and ran it as an admin client - that might be desirable for making the fix more self-contained for backports.

[Dan Smith (danms)]

+2 on the patch from comment #25

[Matt Riedemann (mriedem)]

Here is the final version of the patch for master (queens). Will start working on stable branch patches.

[Matt Riedemann (mriedem)]

This is the stable/pike backport patch. Minor changes to the functional test setup to work in Pike with the FilterScheduler.

[Matt Riedemann (mriedem)]

Here is the stable/ocata patch, it required a few minor changes to resolve merge conflicts and add some missing stuff that wasn't in ocata.

[Matt Riedemann (mriedem)]

Here is the stable/newton patch, some more minor changes needed for the test to work.

[Matt Riedemann (mriedem)]

Oh wait the newton patch in comment 30 needs some work.

[Matt Riedemann (mriedem)]

Alright, some of the USE_NEUTRON stuff that I changed in the original master branch patch had a side effect on some other tests, so I'm just going to have to remove that stuff, and thus redo the patches on stable once that's done. So hold off on sending these patches to people. I'll work on it over the weekend and post them all in a single comment when I'm done and sure they are all passing all unit and functional tests.

[Matt Riedemann (mriedem)]

Updated master branch patch.

[Matt Riedemann (mriedem)]

Updated pike patch.

[Matt Riedemann (mriedem)]

Updated ocata patch.

[Matt Riedemann (mriedem)]

Updated newton patch.

[Matt Riedemann (mriedem)]

OK should be good for disclosure.

[Tony Breeds (o-tony)]

Backports look good to me. I'm happy to hold the nova EOL until after this has been merged. I do worry that a late change like this could be a little disruptive so I wonder if it's worth releasing nova (newton) as is and then again after this has merged that way if we've made a mistake there is a pretty easy work around.

Of course having said that newton *is* EOL so I'd hope that operators are just fast-forwarding through it rather than spending any real time running it. :D

[Jeremy Stanley (fungi)]

Here's an updated impact description to cover latest releases and EOLs in the time since comment #13 (the lowest affected release may need altering if we follow Tony's recommendation to have a 14.0.9 without the fix and then tag 14.0.10 with it):

---

Title: Nova Filter Scheduler bypass through rebuild action
Reporter: George Shuklin (servers.com)
Products: Nova
Affects: <=14.0.8, >=15.0.0 <=15.0.7, >=16.0.0 <=16.0.2

George Shuklin from servers.com reported a vulnerability in Nova. By rebuilding an instance, an authenticated user may be able to circumvent the Filter Scheduler bypassing imposed filters (for example, the ImagePropertiesFilter or the IsolatedHostsFilter). All setups using Nova Filter Scheduler are affected.

---

If I send a pre-OSSA to our downstream stakeholders today we *could* schedule the public advisory for this Thursday (November 2), but I'm concerned that we have limited VMT coverage to handle publication that day due to some of us traveling to Sydney for the OpenStack Summit and have further concerns that the Nova team may have trouble finding someone to shepherd the patches through Gerrit in a timely fashion as well as stable release management to approve subsequent point release requests. And then there's the problem that many of the staff from organizations impacted by this could fail to notice the advisory on Thursday for the same reasons.

At this point I'm leaning toward notifying downstreams via pre-OSSA this week but scheduling publication for Tuesday November 14 (first non-Monday business day after the Summit week) when more people will be around to help and to notice the advisory. I need to know, however, how much of a pain it is to delay Newton EOL for two weeks without being able to publicly say why.

An alternative is to declare that the criticality/impact of this bug is minimal enough to switch to our public disclosure workflow today so we can send an advisory earlier this week... opinions?

[Matt Riedemann (mriedem)]

> I need to know, however, how much of a pain it is to delay Newton EOL for two weeks without being able to publicly say why.

Keeping stable/newton around a couple of weeks for nova isn't a problem for the nova team, but I can't speak for infra and/or Tony for stable maintenance. I think we have people around in nova that can shepherd fixes through the stable branches this week, but again can't speak for getting the stable branch point releases done this week while people are traveling.

[Morgan Fainberg (mdrnstm)]

I like the delay (pre-ossa this week) but the delayed push until post Sydney. I feel like it's best to let folks focus on the summit instead of pushing through an embargoed set of patches.

[Jeremy Stanley (fungi)]

MITRE has assigned CVE-2017-16239 for this issue.

[Jeremy Stanley (fungi)]

Given the feedback so far on the available options, in an effort to avoid surprising our community while it's in the midst of its busiest conference week of the season I'm inclined to send the pre-OSSA (impact description and patches) to the downstream stakeholders on our embargo notice list today indicating Tuesday November 14 as the planned public disclosure for this advisory. That's 10 business days instead of our usual 3-5, but if someone happens to leak it early this is at least a limited enough vulnerability that it won't be too terrible of an emergency.

[Tristan Cacqueray (tristan-cacqueray)]

That sounds fair enough, thanks fungi!

[Jeremy Stanley (fungi)]

Pre-OSSA for this has been sent to the embargo notice mailing list.

[Tony Breeds (o-tony)]

We just releases nova 14.0.9 so as discussed the actual OSSA (and perhaps the pre-OSSA) will need to alter the affected versions to show the 14.0.9 is affected

[Jeremy Stanley (fungi)]

Tristan just sent errata for the affected versions to the embargo notice list reflecting the fact that the new 14.0.9 is also impacted.

[Nolwenn Cauchois (nolwenn)]

I tested the patch in newton and it works well.
But when nova rejects the rebuild operation, image of the instance is still changed.
For example, if we rebuild an ubuntu instance with a new windows image and it is rejected by the nova scheduler, when we do a "nova show" we see that the instance has a windows image.

If an exception is raised here:
https://github.com/openstack/nova/blob/stable/newton/nova/conductor/manager.py#L699
it may be necessary to update the instance with the old image:
instance.task_state = None
instance.image_ref = orig_image_ref
instance.save()

[Matt Riedemann (mriedem)]

@Nolwenn, thanks for testing the patch. The issue you described about the image_ref is a long-standing issue in any case where rebuild fails once we RPC cast from the API to the conductor (or compute) service, and is related to bug 1482040. I think that is a separate, latent issue, worth fixing on it's own, not as part of this security fix.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/519662

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/519672

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/519681

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/519684

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/519686

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0001-Rebuild-should-verify-the-host.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519686
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=cb43ec59595e9875a179008da346852aee42fd41
Submitter: Zuul
Branch: master

commit cb43ec59595e9875a179008da346852aee42fd41
Author: Tristan Cacqueray <email address hidden>
Date: Tue Nov 14 10:37:44 2017 +0000

    Adds OSSA-2017-005 (CVE-2017-16239)

    Change-Id: Ib03875ae5b6ad95ceecf00714704ac9676ef32a7
    Related-Bug: #1664931

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/519738

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519662
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=984dd8ad6add4523d93c7ce5a666a32233e02e34
Submitter: Zuul
Branch: master

commit 984dd8ad6add4523d93c7ce5a666a32233e02e34
Author: Matt Riedemann <email address hidden>
Date: Fri Oct 27 16:03:15 2017 -0400

    Validate new image via scheduler during rebuild

    During a rebuild we bypass the scheduler because we are
    always rebuilding the instance on the same host it's already
    on. However, we allow passing a new image during rebuild
    and that new image needs to be validated to work with the
    instance host by running it through the scheduler filters,
    like the ImagePropertiesFilter. Otherwise the new image
    could violate constraints placed on the host by the admin.

    This change checks to see if there is a new image provided
    and if so, modifies the request spec passed to the scheduler
    so that the new image is validated all while restricting
    the scheduler to still pick the same host that the instance
    is running on. If the image is not valid for the host, the
    scheduler will raise NoValidHost and the rebuild stops.

    A functional test is added to show the recreate of the bug
    and that we probably stop the rebuild now in conductor by
    calling the scheduler to validate the image.

    Co-Authored-By: Sylvain Bauza <email address hidden>

    Closes-Bug: #1664931

    Change-Id: I11746d1ea996a0f18b7c54b4c9c21df58cc4714b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519672
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9e2d63da94db63d97bd02e373bfc53d95808b833
Submitter: Zuul
Branch: stable/pike

commit 9e2d63da94db63d97bd02e373bfc53d95808b833
Author: Matt Riedemann <email address hidden>
Date: Fri Oct 27 16:03:15 2017 -0400

    Validate new image via scheduler during rebuild

    During a rebuild we bypass the scheduler because we are
    always rebuilding the instance on the same host it's already
    on. However, we allow passing a new image during rebuild
    and that new image needs to be validated to work with the
    instance host by running it through the scheduler filters,
    like the ImagePropertiesFilter. Otherwise the new image
    could violate constraints placed on the host by the admin.

    This change checks to see if there is a new image provided
    and if so, modifies the request spec passed to the scheduler
    so that the new image is validated all while restricting
    the scheduler to still pick the same host that the instance
    is running on. If the image is not valid for the host, the
    scheduler will raise NoValidHost and the rebuild stops.

    A functional test is added to show the recreate of the bug
    and that we probably stop the rebuild now in conductor by
    calling the scheduler to validate the image.

    Co-Authored-By: Sylvain Bauza <email address hidden>

    Closes-Bug: #1664931

    NOTE(mriedem): The backport has to explicitly use the
    filter scheduler in the test setup and move the start of
    the compute service to after the placement fixture usage.
    This is because cc833359870d3962326c35094adea2f525ec8141
    is not in stable/pike.

    Change-Id: I11746d1ea996a0f18b7c54b4c9c21df58cc4714b
    (cherry picked from commit 984dd8ad6add4523d93c7ce5a666a32233e02e34)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519681
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b72105c1c49fcddc94992af63fc2f8078023491a
Submitter: Zuul
Branch: stable/ocata

commit b72105c1c49fcddc94992af63fc2f8078023491a
Author: Matt Riedemann <email address hidden>
Date: Fri Oct 27 16:03:15 2017 -0400

    Validate new image via scheduler during rebuild

    During a rebuild we bypass the scheduler because we are
    always rebuilding the instance on the same host it's already
    on. However, we allow passing a new image during rebuild
    and that new image needs to be validated to work with the
    instance host by running it through the scheduler filters,
    like the ImagePropertiesFilter. Otherwise the new image
    could violate constraints placed on the host by the admin.

    This change checks to see if there is a new image provided
    and if so, modifies the request spec passed to the scheduler
    so that the new image is validated all while restricting
    the scheduler to still pick the same host that the instance
    is running on. If the image is not valid for the host, the
    scheduler will raise NoValidHost and the rebuild stops.

    A functional test is added to show the recreate of the bug
    and that we probably stop the rebuild now in conductor by
    calling the scheduler to validate the image.

    Co-Authored-By: Sylvain Bauza <email address hidden>

    Closes-Bug: #1664931

    Conflicts:
          nova/conductor/manager.py
          nova/tests/functional/integrated_helpers.py
          nova/tests/functional/test_servers.py

    NOTE(mriedem): There are a few changes needed for Ocata:

    1. I6590f0eda4ec4996543ad40d8c2640b83fc3dd9d changed some
       of the conditional logic in the conductor rebuild_instance
       method. That refactor is not replayed here, just the
       necessary change for the fix.
    2. The _wait_for_action_fail_completion method didn't exist
       in Ocata.
    3. The PlacementFixture wasn't used in _IntegratedTestBase
       so it's done as part of the test setup.
    4. The default scheduler filters were different in Ocata so
       the test just restricts to using ImagePropertiesFilter.
    5. A few imports were needed in the test module.

    Change-Id: I11746d1ea996a0f18b7c54b4c9c21df58cc4714b
    (cherry picked from commit 984dd8ad6add4523d93c7ce5a666a32233e02e34)
    (cherry picked from commit 9e2d63da94db63d97bd02e373bfc53d95808b833)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/519752

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/519753

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/519755

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519752
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3f63d057a64b688b66ff1903c1afc4d97ba6df6d
Submitter: Zuul
Branch: stable/pike

commit 3f63d057a64b688b66ff1903c1afc4d97ba6df6d
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 14 15:01:52 2017 -0500

    Add security release note for OSSA-2017-005

    Change-Id: I053f1bbc56481bddce8792aa4b5460a55cc0db2d
    Related-Bug: #1664931
    (cherry picked from commit 31d28eef95ab82bdfce2221cd5633bcf4bc13653)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519753
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ffd4f72d16dacd6ca1e703f9bab37b8917d253e7
Submitter: Zuul
Branch: stable/ocata

commit ffd4f72d16dacd6ca1e703f9bab37b8917d253e7
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 14 15:01:52 2017 -0500

    Add security release note for OSSA-2017-005

    Change-Id: I053f1bbc56481bddce8792aa4b5460a55cc0db2d
    Related-Bug: #1664931
    (cherry picked from commit 31d28eef95ab82bdfce2221cd5633bcf4bc13653)
    (cherry picked from commit 3f63d057a64b688b66ff1903c1afc4d97ba6df6d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519738
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=31d28eef95ab82bdfce2221cd5633bcf4bc13653
Submitter: Zuul
Branch: master

commit 31d28eef95ab82bdfce2221cd5633bcf4bc13653
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 14 15:01:52 2017 -0500

    Add security release note for OSSA-2017-005

    Change-Id: I053f1bbc56481bddce8792aa4b5460a55cc0db2d
    Related-Bug: #1664931

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.8 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.3 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519684
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=97a51d981bd603b964b04b3568218ce57ac57338
Submitter: Zuul
Branch: stable/newton

commit 97a51d981bd603b964b04b3568218ce57ac57338
Author: Matt Riedemann <email address hidden>
Date: Fri Oct 27 16:03:15 2017 -0400

    Validate new image via scheduler during rebuild

    During a rebuild we bypass the scheduler because we are
    always rebuilding the instance on the same host it's already
    on. However, we allow passing a new image during rebuild
    and that new image needs to be validated to work with the
    instance host by running it through the scheduler filters,
    like the ImagePropertiesFilter. Otherwise the new image
    could violate constraints placed on the host by the admin.

    This change checks to see if there is a new image provided
    and if so, modifies the request spec passed to the scheduler
    so that the new image is validated all while restricting
    the scheduler to still pick the same host that the instance
    is running on. If the image is not valid for the host, the
    scheduler will raise NoValidHost and the rebuild stops.

    A functional test is added to show the recreate of the bug
    and that we probably stop the rebuild now in conductor by
    calling the scheduler to validate the image.

    Co-Authored-By: Sylvain Bauza <email address hidden>

    Closes-Bug: #1664931

    NOTE(mriedem): There were a few changes needed for Newton:

    1. There is no PlacementFixture but it's not needed.
    2. The API client needs to have the microversion set from
       the test.
    3. The enabled_filters config option wasn't in Newton.
    4. The scheduler has to be started before compute otherwise
       we get a MessagingTimeout due to the CastAsCall fixture
       during the compute startup.

    Depends-On: I344d8fdded9b7d5385fcb41b699f1352acb4cda7

    Change-Id: I11746d1ea996a0f18b7c54b4c9c21df58cc4714b
    (cherry picked from commit 984dd8ad6add4523d93c7ce5a666a32233e02e34)
    (cherry picked from commit 9e2d63da94db63d97bd02e373bfc53d95808b833)
    (cherry picked from commit b72105c1c49fcddc94992af63fc2f8078023491a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519755
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=698b261a5a2a6c0f31ef5059046ef7196d5cba30
Submitter: Zuul
Branch: stable/newton

commit 698b261a5a2a6c0f31ef5059046ef7196d5cba30
Author: Matt Riedemann <email address hidden>
Date: Tue Nov 14 15:01:52 2017 -0500

    Add security release note for OSSA-2017-005

    Change-Id: I053f1bbc56481bddce8792aa4b5460a55cc0db2d
    Related-Bug: #1664931
    (cherry picked from commit 31d28eef95ab82bdfce2221cd5633bcf4bc13653)
    (cherry picked from commit 3f63d057a64b688b66ff1903c1afc4d97ba6df6d)
    (cherry picked from commit ffd4f72d16dacd6ca1e703f9bab37b8917d253e7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.10 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/521153

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/523212

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/523213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521186
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f7c688b8ef88a7390f5b09719a2b3e80368438c0
Submitter: Zuul
Branch: master

commit f7c688b8ef88a7390f5b09719a2b3e80368438c0
Author: Dan Smith <email address hidden>
Date: Fri Nov 17 12:27:34 2017 -0800

    Refined fix for validating image on rebuild

    This aims to fix the issue described in bug 1664931 where a rebuild
    fails to validate the existing host with the scheduler when a new
    image is provided. The previous attempt to do this could cause rebuilds
    to fail unnecessarily because we ran _all_ of the filters during a
    rebuild, which could cause usage/resource filters to prevent an otherwise
    valid rebuild from succeeding.

    This aims to classify filters as useful for rebuild or not, and only apply
    the former during a rebuild scheduler check. We do this by using an internal
    scheduler hint, indicating our intent. This should (a) filter out
    all hosts other than the one we're running on and (b) be detectable by
    the filtering infrastructure as an internally-generated scheduling request
    in order to trigger the correct filtering behavior.

    Closes-Bug: #1664931
    Change-Id: I1a46ef1503be2febcd20f4594f44344d05525446

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/523427

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/523434

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/521153
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cacfd372acb4eb056c4391db3c988bfe91c957df
Submitter: Zuul
Branch: master

commit cacfd372acb4eb056c4391db3c988bfe91c957df
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 13:17:16 2017 -0500

    Add regression test for rebuild with new image doubling allocations

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes a rebuild
    with a new image go through the scheduler again to validate the
    image against the instance.host (we rebuild to the same host that
    the instance already lives on).

    The problem is that change introduced a regression where the
    FilterScheduler is going to think it's doing a resize to the same
    host and double the allocations for the instance (and usage for the
    compute node provider) in Placement, which is wrong since the
    flavor is the same.

    This adds a regression test to show the bug.

    Change-Id: Ie0949b4e6101f0b29ec4542146d523a07a683991
    Related-Bug: #1664931

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/523649

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523212
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b29a461a8bc05c9b171c0574abb2e7e5b62a2ed7
Submitter: Zuul
Branch: stable/pike

commit b29a461a8bc05c9b171c0574abb2e7e5b62a2ed7
Author: Dan Smith <email address hidden>
Date: Fri Nov 17 12:27:34 2017 -0800

    Refined fix for validating image on rebuild

    This aims to fix the issue described in bug 1664931 where a rebuild
    fails to validate the existing host with the scheduler when a new
    image is provided. The previous attempt to do this could cause rebuilds
    to fail unnecessarily because we ran _all_ of the filters during a
    rebuild, which could cause usage/resource filters to prevent an otherwise
    valid rebuild from succeeding.

    This aims to classify filters as useful for rebuild or not, and only apply
    the former during a rebuild scheduler check. We do this by using an internal
    scheduler hint, indicating our intent. This should (a) filter out
    all hosts other than the one we're running on and (b) be detectable by
    the filtering infrastructure as an internally-generated scheduling request
    in order to trigger the correct filtering behavior.

    Conflicts:
          nova/tests/functional/test_servers.py

    NOTE(mriedem): The conflict is due to the need to override the
    _setup_scheduler_service method in Pike.

    Closes-Bug: #1664931
    Change-Id: I1a46ef1503be2febcd20f4594f44344d05525446
    (cherry picked from commit f7c688b8ef88a7390f5b09719a2b3e80368438c0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523427
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bbfc4230efe3299fa51f9451f54062f32590ed3d
Submitter: Zuul
Branch: stable/ocata

commit bbfc4230efe3299fa51f9451f54062f32590ed3d
Author: Dan Smith <email address hidden>
Date: Fri Nov 17 12:27:34 2017 -0800

    Refined fix for validating image on rebuild

    This aims to fix the issue described in bug 1664931 where a rebuild
    fails to validate the existing host with the scheduler when a new
    image is provided. The previous attempt to do this could cause rebuilds
    to fail unnecessarily because we ran _all_ of the filters during a
    rebuild, which could cause usage/resource filters to prevent an otherwise
    valid rebuild from succeeding.

    This aims to classify filters as useful for rebuild or not, and only apply
    the former during a rebuild scheduler check. We do this by using an internal
    scheduler hint, indicating our intent. This should (a) filter out
    all hosts other than the one we're running on and (b) be detectable by
    the filtering infrastructure as an internally-generated scheduling request
    in order to trigger the correct filtering behavior.

    Conflicts:
          nova/scheduler/filters/exact_core_filter.py
          nova/scheduler/filters/exact_disk_filter.py
          nova/scheduler/filters/exact_ram_filter.py
          nova/scheduler/filters/type_filter.py

    NOTE(mriedem): The conflicts were due to deprecation warnings
    which don't exist in Ocata. The functional test also needed a missing
    import and to use api_put() instead of put_service().

    Closes-Bug: #1664931
    Change-Id: I1a46ef1503be2febcd20f4594f44344d05525446
    (cherry picked from commit f7c688b8ef88a7390f5b09719a2b3e80368438c0)
    (cherry picked from commit b29a461a8bc05c9b171c0574abb2e7e5b62a2ed7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523434
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4cbfcc590c17134fd14e3aab90ffbb7c17006a95
Submitter: Zuul
Branch: stable/newton

commit 4cbfcc590c17134fd14e3aab90ffbb7c17006a95
Author: Dan Smith <email address hidden>
Date: Fri Nov 17 12:27:34 2017 -0800

    Refined fix for validating image on rebuild

    This aims to fix the issue described in bug 1664931 where a rebuild
    fails to validate the existing host with the scheduler when a new
    image is provided. The previous attempt to do this could cause rebuilds
    to fail unnecessarily because we ran _all_ of the filters during a
    rebuild, which could cause usage/resource filters to prevent an otherwise
    valid rebuild from succeeding.

    This aims to classify filters as useful for rebuild or not, and only apply
    the former during a rebuild scheduler check. We do this by using an internal
    scheduler hint, indicating our intent. This should (a) filter out
    all hosts other than the one we're running on and (b) be detectable by
    the filtering infrastructure as an internally-generated scheduling request
    in order to trigger the correct filtering behavior.

    Conflicts:
          nova/scheduler/utils.py
          nova/tests/unit/compute/test_compute_api.py

    NOTE(mriedem): The conflicts are due to not having
    7d0381c91a6ba8a45ae6527f046f382166eb158d or
    4a7502a5c9e84a8c8cef7f355d72425b26b8c379 in Newton.

    Closes-Bug: #1664931
    Change-Id: I1a46ef1503be2febcd20f4594f44344d05525446
    (cherry picked from commit f7c688b8ef88a7390f5b09719a2b3e80368438c0)
    (cherry picked from commit b29a461a8bc05c9b171c0574abb2e7e5b62a2ed7)
    (cherry picked from commit bbfc4230efe3299fa51f9451f54062f32590ed3d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523649
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=ab54ce232d6db2b3e126ac61ee015d8b180e2ad6
Submitter: Zuul
Branch: master

commit ab54ce232d6db2b3e126ac61ee015d8b180e2ad6
Author: Tristan Cacqueray <email address hidden>
Date: Wed Nov 29 01:50:17 2017 +0000

    Updates OSSA 2017-005 with ERRATA#1

    Change-Id: Id747d793c430ac2e5b3c27b68e55c0f12cb8c3f7
    Related-Bug: #1664931

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523213
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=234ade29a39cf2d51e08157e149e0cbd0c5047be
Submitter: Zuul
Branch: stable/pike

commit 234ade29a39cf2d51e08157e149e0cbd0c5047be
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 13:17:16 2017 -0500

    Add regression test for rebuild with new image doubling allocations

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes a rebuild
    with a new image go through the scheduler again to validate the
    image against the instance.host (we rebuild to the same host that
    the instance already lives on).

    The problem is that change introduced a regression where the
    FilterScheduler is going to think it's doing a resize to the same
    host and double the allocations for the instance (and usage for the
    compute node provider) in Placement, which is wrong since the
    flavor is the same.

    This adds a regression test to show the bug.

    NOTE(mriedem): Due to cc833359870d3962326c35094adea2f525ec8141 not
    being in Pike, we have to use the MediumFakeDriver until the bug
    is fixed. Also, the ComputeManager._get_nodename method doesn't exist
    in Pike so we have to just get the hypervisor uuid from the os-hypervisors
    API.

    Change-Id: Ie0949b4e6101f0b29ec4542146d523a07a683991
    Related-Bug: #1664931
    (cherry picked from commit cacfd372acb4eb056c4391db3c988bfe91c957df)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.1.0 release.

[Matt Riedemann (mriedem)]

For anyone keeping track, bug 1750618 was yet another regression introduced by the first change for this bug.