volume-backed instance rebuild with no image change is still going through scheduler

Bug #1732947 reported by Matt Riedemann on 2017-11-17
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Matt Riedemann
Newton
High
Matt Riedemann
Ocata
High
Matt Riedemann
Pike
High
Matt Riedemann

Bug Description

Due to this recent change:

https://github.com/openstack/nova/commit/984dd8ad6add4523d93c7ce5a666a32233e02e34

And the fact that volume-backed instances don't store the instance.image_ref value to indicate they are volume-backed, this condition will always evaluate to True:

https://github.com/openstack/nova/blob/984dd8ad6add4523d93c7ce5a666a32233e02e34/nova/compute/api.py#L2959

And we'll unnecessarily call through the scheduler again to validate the instance on the original host during the rebuild even if the image isn't changing.

For a volume-backed instance, we have to get the original image_id from the volume's volume_image_metadata field:

https://review.openstack.org/#/c/520686/

Matt Riedemann (mriedem) on 2017-11-17
Changed in nova:
status: New → Triaged
Changed in nova:
assignee: nobody → Chris Friesen (cbf123)
status: Triaged → In Progress
Matt Riedemann (mriedem) on 2017-11-17
Changed in nova:
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.openstack.org/521391

Changed in nova:
assignee: Chris Friesen (cbf123) → Matt Riedemann (mriedem)

Reviewed: https://review.openstack.org/521200
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a4eebd5de7ef4b63082536718ba7b993a66d47e7
Submitter: Zuul
Branch: master

commit a4eebd5de7ef4b63082536718ba7b993a66d47e7
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947

Reviewed: https://review.openstack.org/521391
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54407afef3b446e34c44ba5392ccbf121e266de2
Submitter: Zuul
Branch: master

commit 54407afef3b446e34c44ba5392ccbf121e266de2
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947

Changed in nova:
status: In Progress → Fix Released

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

Reviewed: https://review.openstack.org/525632
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c7991653a4e7cbb86656e6ba7579a86236801567
Submitter: Zuul
Branch: stable/pike

commit c7991653a4e7cbb86656e6ba7579a86236801567
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947
    (cherry picked from commit a4eebd5de7ef4b63082536718ba7b993a66d47e7)

tags: added: in-stable-pike

Reviewed: https://review.openstack.org/525634
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4e36c4bd0eab2be619991ae00454906e171720b6
Submitter: Zuul
Branch: stable/pike

commit 4e36c4bd0eab2be619991ae00454906e171720b6
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947
    (cherry picked from commit 54407afef3b446e34c44ba5392ccbf121e266de2)

Reviewed: https://review.openstack.org/525664
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fe5e3e7e61d879f7a28eb177e4bdbc7ead3ae4ac
Submitter: Zuul
Branch: stable/ocata

commit fe5e3e7e61d879f7a28eb177e4bdbc7ead3ae4ac
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Conflicts:
          nova/tests/fixtures.py

    NOTE(mriedem): The CinderFixture conflict is due to not having
    1928ec5606491de95abba8791b1d7b888690dd3c in Ocata. However, this
    change needs the part of that patch to the CinderFixture to work,
    so it's added here.

    NOTE(mriedem): The functional test needed to change slightly to
    use the PlacementFixture and disable the DiskFilter.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947
    (cherry picked from commit a4eebd5de7ef4b63082536718ba7b993a66d47e7)
    (cherry picked from commit c7991653a4e7cbb86656e6ba7579a86236801567)

tags: added: in-stable-ocata

Reviewed: https://review.openstack.org/525666
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=10bc4fdbc568670dfd8bec109e8b854ba8ad7949
Submitter: Zuul
Branch: stable/ocata

commit 10bc4fdbc568670dfd8bec109e8b854ba8ad7949
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947
    (cherry picked from commit 54407afef3b446e34c44ba5392ccbf121e266de2)
    (cherry picked from commit 4e36c4bd0eab2be619991ae00454906e171720b6)

Reviewed: https://review.openstack.org/526542
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ec5886fe74514c65908ff3adc2ca46610b595ad
Submitter: Zuul
Branch: stable/newton

commit 5ec5886fe74514c65908ff3adc2ca46610b595ad
Author: Matt Riedemann <email address hidden>
Date: Fri Nov 17 16:53:39 2017 -0500

    Add regression test for rebuilding a volume-backed server

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 makes rebuild
    check to see if the user is rebuilding an instance with a new
    image and if so, to run the scheduler filters again since the
    new image might not work with the current host for the instance,
    and we rebuild to the same host that the instance is already
    running on.

    The problem is the instance.image_ref attribute is not set for
    a volume-backed (boot-from-volume) instance, so the conditional
    in the rebuild() method is always True, which means we always run
    through the scheduler for volume-backed instances during rebuild,
    even if the image in the root disk isn't changing.

    This adds a functional regression test to recreate the bug.

    Conflicts:
          nova/tests/fixtures.py

    NOTE(mriedem): The fixture conflict is due to not having the
    CinderFixture in Newton. That was added in Ocata via commit
    47fb8b757994688a995ec3bdfa5dccc72f870f34. It's added here in
    it's entirety since that is simpler than trying to basically
    re-write it from scratch in a backport.

    NOTE(mriedem): The functional test is slightly modified to
    not rely on Placement and because of old config option names.

    Change-Id: If79c554b46c44a7f70f8367426e7da362d7234c8
    Related-Bug: #1732947
    (cherry picked from commit a4eebd5de7ef4b63082536718ba7b993a66d47e7)
    (cherry picked from commit c7991653a4e7cbb86656e6ba7579a86236801567)
    (cherry picked from commit fe5e3e7e61d879f7a28eb177e4bdbc7ead3ae4ac)

tags: added: in-stable-newton

Reviewed: https://review.openstack.org/526544
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=24953a95e5989f44acaf488c253230e476c2fa59
Submitter: Zuul
Branch: stable/newton

commit 24953a95e5989f44acaf488c253230e476c2fa59
Author: Matt Riedemann <email address hidden>
Date: Sun Nov 19 17:25:56 2017 -0500

    Get original image_id from volume for volume-backed instance rebuild

    A volume-backed instance will not have the instance.image_ref
    attribute set, to indicate to the API user that it is a volume-backed
    instance.

    Commit 984dd8ad6add4523d93c7ce5a666a32233e02e34 missed this subtle
    difference with how instance.image_ref is used, which means a rebuild
    of any volume-backed instance will now run through the scheduler, even
    if the image_href passed to rebuild is the same image ID as for the
    root volume.

    This fixes that case in rebuild by getting the image metadata off the
    root volume for a volume-backed instance and compares that to the
    image_href passed to rebuild.

    Change-Id: I48cda813b9effa37f6c3e0cd2e8a22bb78c79d72
    Closes-Bug: #1732947
    (cherry picked from commit 54407afef3b446e34c44ba5392ccbf121e266de2)
    (cherry picked from commit 4e36c4bd0eab2be619991ae00454906e171720b6)
    (cherry picked from commit 10bc4fdbc568670dfd8bec109e8b854ba8ad7949)

This issue was fixed in the openstack/nova 16.0.4 release.

This issue was fixed in the openstack/nova 15.1.0 release.

This issue was fixed in the openstack/nova 14.1.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers