By rebuilding twice with the same "forbidden" image one can circumvent scheduler rebuild restrictions
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | OpenStack Compute (nova) |
High
|
Matt Riedemann | ||
| | Newton |
Undecided
|
Unassigned | ||
| | Ocata |
High
|
Matt Riedemann | ||
| | Pike |
High
|
Matt Riedemann | ||
Bug Description
Description
===========
Since CVE-2017-16239, we call to the scheduler when doing a rebuild with a new image. If the scheduler refuses a rebuild because a filter forbids the new image on the instance's host (for example, IsolatedHostsFi
Steps to reproduce
==================
1. Configure IsolatedHostsFi
[filter_
enabled_filters = [...],IsolatedH
isolated_images = 41d3e5ca-
isolated_hosts = ubuntu
restrict_
2. Have two images, one isolated and one not:
$ openstack image list
8d0581a5-
41d3e5ca-
cirros is the isolated one
3. Have only one hypervisor (the isolated one):
$ openstack hypervisor list
ubuntu | QEMU | 192.168.100.194 | up
5. Boot a cirros (isolated) image:
$ openstack server create \
--image 41d3e5ca-
--flavor m1.nano \
cirros-
$ openstack server list
cirros-
6. Rebuild the cirros instance with centos (this should be refused by the scheduler):
$ nova --debug rebuild cirros-
DEBUG (session:722) POST call to compute for
http://
used request id req-c234346a-
8. Observe the instance going to ERROR,
but still showing the new centos image :
$ nova show cirros-
[...]
status | ERROR
image | centos (8d0581a5-
[...]
9. Rebuild again with the same centos image:
$ nova rebuild cirros-
10. The rebuild goes through.
Expected result
===============
At step 10, the rebuild should still be refused.
Actual result
=============
The rebuild is allowed.
Environment
===========
1. Exact version of OpenStack you are running. See the following
Was reported in Red Hat OpenStack 12, affects newton through master.
2. Which hypervisor did you use?
libvirt+kvm
| Changed in nova: | |
| assignee: | nobody → Matt Riedemann (mriedem) |
| status: | New → In Progress |
| Changed in nova: | |
| importance: | Undecided → High |
| Matt Riedemann (mriedem) wrote : | #1 |
Fix proposed to branch: stable/pike
Review: https:/
Fix proposed to branch: stable/ocata
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 4a2c9a4887a219a
Author: Matt Riedemann <email address hidden>
Date: Mon Jan 29 10:50:36 2018 -0500
Rollback instance.image_ref on failed rebuild
When rebuilding and changing the image, we run the new image
through the scheduler to see if it's valid for the instance
on its current compute host. The API saves off the new image
ref on the instance before casting to conductor to run through
the scheduler. If the scheduler fails, the instance.image_ref was
not being rolled back, which meant a user could attempt the rebuild
with the same invalid image a second time and the API, seeing the
instance.
backing image for the server), will bypass the scheduler and rebuild
the instance with that invalid image.
This fixes the issue by using the original image ref, passed from
API to conductor during rebuild, to reset the instance.image_ref
in the case of a failure.
Note that there are other things changed on the instance in the API
which this patch does not attempt to recover as that's a bigger
work item which likely involves substantial refactoring of the code.
Closes-Bug: #1746032
Change-Id: I3399a66fe9b129
| Changed in nova: | |
| status: | In Progress → Fix Released |
This issue was fixed in the openstack/nova 17.0.0.0rc1 release candidate.
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/pike
commit 834adeae9a3ff1b
Author: Matt Riedemann <email address hidden>
Date: Mon Jan 29 10:50:36 2018 -0500
Rollback instance.image_ref on failed rebuild
When rebuilding and changing the image, we run the new image
through the scheduler to see if it's valid for the instance
on its current compute host. The API saves off the new image
ref on the instance before casting to conductor to run through
the scheduler. If the scheduler fails, the instance.image_ref was
not being rolled back, which meant a user could attempt the rebuild
with the same invalid image a second time and the API, seeing the
instance.
backing image for the server), will bypass the scheduler and rebuild
the instance with that invalid image.
This fixes the issue by using the original image ref, passed from
API to conductor during rebuild, to reset the instance.image_ref
in the case of a failure.
Note that there are other things changed on the instance in the API
which this patch does not attempt to recover as that's a bigger
work item which likely involves substantial refactoring of the code.
Closes-Bug: #1746032
Conflicts:
NOTE(mriedem): The conflict is due to not having change
Ibc44e3b226
had to be imported in the functional test.
Change-Id: I3399a66fe9b129
(cherry picked from commit 4a2c9a4887a219a
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/ocata
commit 2efe3f6b8844bd3
Author: Matt Riedemann <email address hidden>
Date: Mon Jan 29 10:50:36 2018 -0500
Rollback instance.image_ref on failed rebuild
When rebuilding and changing the image, we run the new image
through the scheduler to see if it's valid for the instance
on its current compute host. The API saves off the new image
ref on the instance before casting to conductor to run through
the scheduler. If the scheduler fails, the instance.image_ref was
not being rolled back, which meant a user could attempt the rebuild
with the same invalid image a second time and the API, seeing the
instance.
backing image for the server), will bypass the scheduler and rebuild
the instance with that invalid image.
This fixes the issue by using the original image ref, passed from
API to conductor during rebuild, to reset the instance.image_ref
in the case of a failure.
Note that there are other things changed on the instance in the API
which this patch does not attempt to recover as that's a bigger
work item which likely involves substantial refactoring of the code.
Closes-Bug: #1746032
Conflicts:
NOTE(mriedem): The conflicts in manager.py are due to not having
I06d78c744f
test conflict is due to tests that existed in Pike which don't exist
in Ocata.
Change-Id: I3399a66fe9b129
(cherry picked from commit 4a2c9a4887a219a
(cherry picked from commit 834adeae9a3ff1b
This issue was fixed in the openstack/nova 16.1.1 release.
This issue was fixed in the openstack/nova 15.1.1 release.
This will also be an issue in newton but we're waiting to end of life newton so we won't fix this upstream there.