Bug: resize instance after edit flavor with horizon

Bug #1570748 reported by Hyun Ha on 2016-04-15
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Dan Smith
Kilo
Medium
Matt Riedemann
Liberty
Medium
Matt Riedemann
Mitaka
Medium
Matt Riedemann
Ubuntu Cloud Archive
Medium
Unassigned
Liberty
Medium
Unassigned
Mitaka
Medium
Unassigned
nova-powervm
Medium
Drew Thorstensen
tempest
Wishlist
Matt Riedemann
nova (Ubuntu)
Undecided
Unassigned
Wily
Medium
Unassigned
Xenial
Medium
Unassigned
Yakkety
Undecided
Unassigned

Bug Description

Error occured when resize instance after edit flavor with horizon (and also delete flavor used by instance)

Reproduce step :

1. create flavor A
2. boot instance using flavor A
3. edit flavor with horizon (or delete flavor A)
    -> the result is same to edit or to delelet flavor because edit flavor means delete/recreate flavor)
4. resize or migrate instance
5. Error occured

Log :
nova-compute.log
   File "/opt/openstack/src/nova/nova/conductor/manager.py", line 422, in _object_dispatch
     return getattr(target, method)(*args, **kwargs)

   File "/opt/openstack/src/nova/nova/objects/base.py", line 163, in wrapper
     result = fn(cls, context, *args, **kwargs)

   File "/opt/openstack/src/nova/nova/objects/flavor.py", line 132, in get_by_id
     db_flavor = db.flavor_get(context, id)

   File "/opt/openstack/src/nova/nova/db/api.py", line 1479, in flavor_get
     return IMPL.flavor_get(context, id)

   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
     return f(*args, **kwargs)

   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 4732, in flavor_get
     raise exception.FlavorNotFound(flavor_id=id)

 FlavorNotFound: Flavor 7 could not be found.

This Error is occured because of below code:
/opt/openstack/src/nova/nova/compute/manager.py

    def resize_instance(self, context, instance, image,
                        reservations, migration, instance_type,
                        clean_shutdown=True):
....
            if (not instance_type or
                not isinstance(instance_type, objects.Flavor)):
                instance_type = objects.Flavor.get_by_id(
                    context, migration['new_instance_type_id'])
....

I think that deleted flavor should be taken when resize instance.
I tested this in stable/kilo, but I think stable/liberty and stable/mitaka has same bug because source code is not changed.

thanks.

Hyun Ha (raymon-ha) on 2016-04-15
summary: - Bug: Error resize instance after edit flavor with horizon
+ Bug: resize instance after edit flavor with horizon
tags: added: nova
tags: added: migrate resize

Its working fine in stable/mitaka . I am able to resize the instance after deleting the flavor with which the instance is launched.

Qijing Li (qijing-lee) wrote :

I can reproduce this bug in master.

Sean Dague (sdague) on 2016-04-18
Changed in nova:
status: New → Confirmed
importance: Undecided → High
Dan Smith (danms) wrote :

If this works in mitaka but not master, I'd want to know: Have you run online migrations to get the flavors moved over to the api database yet?

Also, I need to see more of the logs around the failure please.

Matt Riedemann (mriedem) wrote :

I was able to confirm this with a newish (created last week) newton devstack. The failure wasn't on the resize, it was on the revert. Steps:

1. create a duplicate of flavor 42, call it flavor 43 m1.nano2
2. boot instance using flavor 42
3. delete flavor 42
4. resize instance to flavor 43
5. revert the resize, at this point it fails:

ERROR (BadRequest): Flavor used by the instance could not be found. (HTTP 400) (Request-ID: req-98916fd8-846e-4564-ac45-a09933883bb0)

This is my version of nova:

stack@swapvol:/opt/stack/nova$ git log -1
commit 6ae2b4534a8a0d5394317533a41ad1ee3fa13964
Merge: 177a2c4 9e305c2
Author: Jenkins <email address hidden>
Date: Thu Apr 14 14:17:51 2016 +0000

    Merge "cleanup some issues in parameters.yaml"
stack@swapvol:/opt/stack/nova$

This is all I see in the n-api logs:

2016-04-18 18:24:59.406 DEBUG nova.compute.api [req-98916fd8-846e-4564-ac45-a09933883bb0 admin admin] [instance: 630f2bbe-b104-4497-b1a8-936b75fe9356] Fetching instance by UUID from (pid=27352) get /opt/stack/nova/nova/compute/api.py:2041
2016-04-18 18:24:59.497 INFO nova.osapi_compute.wsgi.server [req-98916fd8-846e-4564-ac45-a09933883bb0 admin admin] 9.5.125.128 "POST /v2.1/servers/630f2bbe-b104-4497-b1a8-936b75fe9356/action HTTP/1.1" status: 400 len: 383 time: 0.2037680

Matt Riedemann (mriedem) wrote :

It looks like the problem is here:

https://github.com/openstack/nova/blob/9989a75384aeb6c6b398e7caf8dee140c6bf0423/nova/compute/utils.py#L462

When doing a revert_resize from the compute API, it calls nova.compute.utils.reverse_upsize_quota_delta which tries to lookup the old flavor using the flavor ID from the migration record rather than the instance, and since the flavor was deleted we get a 404.

Matt Riedemann (mriedem) wrote :

The fix is that reverse_upsize_quota_delta should be using instance.old_flavor rather than relying on the old_instance_type in the migration object.

Matt Riedemann (mriedem) wrote :

We should also have a tempest test for this scenario since we missed it. We have tempest testing for reverting a resize, but not one where the flavor was deleted in between the resize and the revert.

Changed in nova:
status: Confirmed → Triaged
tags: added: mitaka-backport-potential
Matt Riedemann (mriedem) on 2016-04-18
Changed in tempest:
status: New → Confirmed
importance: Undecided → Wishlist

Fix proposed to branch: master
Review: https://review.openstack.org/307438

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/307518

Changed in tempest:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/307438
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0
Submitter: Jenkins
Branch: master

commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/309168
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7e3d3abf50732a8dd9350a1ab65376780e7ee0bd
Submitter: Jenkins
Branch: stable/kilo

commit 7e3d3abf50732a8dd9350a1ab65376780e7ee0bd
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Conflicts:
            nova/compute/api.py
            nova/compute/utils.py
            nova/tests/unit/compute/test_compute_api.py
            nova/tests/unit/compute/test_compute_utils.py

    NOTE(mriedem): The conflicts are due to _reverse_upsize_quota_delta
    being moved from nova.compute.api to nova.compute.utils in liberty
    in commit 60411ddf1bada3629f0588de8726c63b5ab00357. So the
    nova.compute.utils changes are omitted here. Also, there was no direct
    testing of _reverse_upsize_quota_delta until that change in liberty
    so as part of this cherry pick we pull back the test_reverse_quota_delta
    test. It's modified slightly since it's in a NoDBTestCase.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)
    (cherry picked from commit 30aad1e796778a94da7719b1d4586bd3cdecb821)
    (cherry picked from commit 5374568fadebf639886febe6073baf352c2a9d83)

Reviewed: https://review.openstack.org/309040
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=30aad1e796778a94da7719b1d4586bd3cdecb821
Submitter: Jenkins
Branch: stable/mitaka

commit 30aad1e796778a94da7719b1d4586bd3cdecb821
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)

Reviewed: https://review.openstack.org/309110
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5374568fadebf639886febe6073baf352c2a9d83
Submitter: Jenkins
Branch: stable/liberty

commit 5374568fadebf639886febe6073baf352c2a9d83
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Conflicts:
            nova/tests/unit/compute/test_compute_utils.py

    NOTE(mriedem): The conflict is due to uuidsentinel not being imported
    in liberty (and is not needed here) and test_migration still being
    used in liberty (but not mitaka).

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)
    (cherry picked from commit 30aad1e796778a94da7719b1d4586bd3cdecb821)

Reviewed: https://review.openstack.org/307518
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=98c32b7860f03bedbe2d4d8e35ec53cfa0c0c5cb
Submitter: Jenkins
Branch: master

commit 98c32b7860f03bedbe2d4d8e35ec53cfa0c0c5cb
Author: Matt Riedemann <email address hidden>
Date: Mon Apr 18 19:34:47 2016 -0400

    Add a test for reverting a resize with a deleted flavor

    A user should be able to boot a server, resize it and
    then reject the resize even though the original flavor
    used to boot the instance is deleted. This is because
    the old flavor information is stored with the instance
    in the nova database, so the original flavor doesn't
    actually need to exist anymore.

    Depends-On: I5f95021410a309ac07fe9f474cbcd0214d1af208

    Change-Id: I356411f96a601f1443d75ac90e42567bef1f8228
    Closes-Bug: #1570748

Changed in tempest:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/311145
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=e72a505860b2f7ce2dd4a492f65d426f8b091e1f
Submitter: Jenkins
Branch: master

commit e72a505860b2f7ce2dd4a492f65d426f8b091e1f
Author: Drew Thorstensen <email address hidden>
Date: Fri Apr 29 10:12:33 2016 -0500

    Fix revert migration if flavor deleted

    After an instance is created, the flavor could be deleted. But
    operations still need to be able to occur against the VM when that
    occurs. A new test in Tempest showed that we had been using the
    incorrect (and potentially deleted) flavor when a resize/migration was
    reverting.

    This change updates the code to use the flavor that is stored on the
    instance itself.

    Change-Id: I16254d8eb74a724e3e0a538f47259e9627c4233a
    Closes-Bug: 1570748

Changed in nova-powervm:
status: New → Fix Released
Eric Fried (efried) on 2016-05-02
Changed in nova-powervm:
status: Fix Released → Fix Committed
assignee: nobody → Drew Thorstensen (thorst)
importance: Undecided → Medium

Reviewed: https://review.openstack.org/311521
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=f98749a2d3154452d8edb8c0e95b54f5cde579d1
Submitter: Jenkins
Branch: stable/mitaka

commit f98749a2d3154452d8edb8c0e95b54f5cde579d1
Author: Drew Thorstensen <email address hidden>
Date: Fri Apr 29 10:12:33 2016 -0500

    Fix revert migration if flavor deleted

    After an instance is created, the flavor could be deleted. But
    operations still need to be able to occur against the VM when that
    occurs. A new test in Tempest showed that we had been using the
    incorrect (and potentially deleted) flavor when a resize/migration was
    reverting.

    This change updates the code to use the flavor that is stored on the
    instance itself.

    Change-Id: I16254d8eb74a724e3e0a538f47259e9627c4233a
    Closes-Bug: 1570748
    (cherry picked from commit e72a505860b2f7ce2dd4a492f65d426f8b091e1f)
    Conflicts:
     nova_powervm/tests/virt/powervm/test_driver.py

This issue was fixed in the openstack/nova 2015.1.4 release.

Louis Bouchard (louis) on 2016-05-23
Changed in nova (Ubuntu Yakkety):
status: New → Fix Released
tags: added: sts-sru
no longer affects: cloud-archive/kilo
Changed in nova-powervm:
status: Fix Committed → Fix Released

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 12.0.4 release.

Changed in nova (Ubuntu Wily):
importance: Undecided → Medium
status: New → Triaged
Changed in nova (Ubuntu Xenial):
importance: Undecided → Medium
status: New → Triaged
Changed in cloud-archive:
importance: Undecided → Medium
status: New → Fix Released

This issue was fixed in the openstack/nova 13.1.0 release.

Louis Bouchard (louis) on 2016-11-09
tags: removed: sts-sru

This issue was fixed in the openstack/nova 2015.1.4 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers