Bug: resize instance after edit flavor with horizon

[Hyun Ha]

Error occured when resize instance after edit flavor with horizon (and also delete flavor used by instance)

Reproduce step :

1. create flavor A
2. boot instance using flavor A
3. edit flavor with horizon (or delete flavor A)
    -> the result is same to edit or to delelet flavor because edit flavor means delete/recreate flavor)
4. resize or migrate instance
5. Error occured

Log :
nova-compute.log
   File "/opt/openstack/src/nova/nova/conductor/manager.py", line 422, in _object_dispatch
     return getattr(target, method)(*args, **kwargs)

   File "/opt/openstack/src/nova/nova/objects/base.py", line 163, in wrapper
     result = fn(cls, context, *args, **kwargs)

   File "/opt/openstack/src/nova/nova/objects/flavor.py", line 132, in get_by_id
     db_flavor = db.flavor_get(context, id)

   File "/opt/openstack/src/nova/nova/db/api.py", line 1479, in flavor_get
     return IMPL.flavor_get(context, id)

   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
     return f(*args, **kwargs)

   File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 4732, in flavor_get
     raise exception.FlavorNotFound(flavor_id=id)

 FlavorNotFound: Flavor 7 could not be found.

This Error is occured because of below code:
/opt/openstack/src/nova/nova/compute/manager.py

    def resize_instance(self, context, instance, image,
                        reservations, migration, instance_type,
                        clean_shutdown=True):
....
            if (not instance_type or
                not isinstance(instance_type, objects.Flavor)):
                instance_type = objects.Flavor.get_by_id(
                    context, migration['new_instance_type_id'])
....

I think that deleted flavor should be taken when resize instance.
I tested this in stable/kilo, but I think stable/liberty and stable/mitaka has same bug because source code is not changed.

thanks.

[gundarapu kalyan reddy (gundarapu-reddy)]

Its working fine in stable/mitaka . I am able to resize the instance after deleting the flavor with which the instance is launched.

[Qijing Li (qijing-lee)]

I can reproduce this bug in master.

[Dan Smith (danms)]

If this works in mitaka but not master, I'd want to know: Have you run online migrations to get the flavors moved over to the api database yet?

Also, I need to see more of the logs around the failure please.

[Matt Riedemann (mriedem)]

I was able to confirm this with a newish (created last week) newton devstack. The failure wasn't on the resize, it was on the revert. Steps:

1. create a duplicate of flavor 42, call it flavor 43 m1.nano2
2. boot instance using flavor 42
3. delete flavor 42
4. resize instance to flavor 43
5. revert the resize, at this point it fails:

ERROR (BadRequest): Flavor used by the instance could not be found. (HTTP 400) (Request-ID: req-98916fd8-846e-4564-ac45-a09933883bb0)

This is my version of nova:

stack@swapvol:/opt/stack/nova$ git log -1
commit 6ae2b4534a8a0d5394317533a41ad1ee3fa13964
Merge: 177a2c4 9e305c2
Author: Jenkins <email address hidden>
Date: Thu Apr 14 14:17:51 2016 +0000

    Merge "cleanup some issues in parameters.yaml"
stack@swapvol:/opt/stack/nova$

This is all I see in the n-api logs:

2016-04-18 18:24:59.406 DEBUG nova.compute.api [req-98916fd8-846e-4564-ac45-a09933883bb0 admin admin] [instance: 630f2bbe-b104-4497-b1a8-936b75fe9356] Fetching instance by UUID from (pid=27352) get /opt/stack/nova/nova/compute/api.py:2041
2016-04-18 18:24:59.497 INFO nova.osapi_compute.wsgi.server [req-98916fd8-846e-4564-ac45-a09933883bb0 admin admin] 9.5.125.128 "POST /v2.1/servers/630f2bbe-b104-4497-b1a8-936b75fe9356/action HTTP/1.1" status: 400 len: 383 time: 0.2037680

[Matt Riedemann (mriedem)]

It looks like the problem is here:

https://github.com/openstack/nova/blob/9989a75384aeb6c6b398e7caf8dee140c6bf0423/nova/compute/utils.py#L462

When doing a revert_resize from the compute API, it calls nova.compute.utils.reverse_upsize_quota_delta which tries to lookup the old flavor using the flavor ID from the migration record rather than the instance, and since the flavor was deleted we get a 404.

[Matt Riedemann (mriedem)]

The fix is that reverse_upsize_quota_delta should be using instance.old_flavor rather than relying on the old_instance_type in the migration object.

[Matt Riedemann (mriedem)]

We should also have a tempest test for this scenario since we missed it. We have tempest testing for reverting a resize, but not one where the flavor was deleted in between the resize and the revert.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307438

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/307518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307438
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0
Submitter: Jenkins
Branch: master

commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309040

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/309110

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/309168

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309168
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7e3d3abf50732a8dd9350a1ab65376780e7ee0bd
Submitter: Jenkins
Branch: stable/kilo

commit 7e3d3abf50732a8dd9350a1ab65376780e7ee0bd
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Conflicts:
            nova/compute/api.py
            nova/compute/utils.py
            nova/tests/unit/compute/test_compute_api.py
            nova/tests/unit/compute/test_compute_utils.py

    NOTE(mriedem): The conflicts are due to _reverse_upsize_quota_delta
    being moved from nova.compute.api to nova.compute.utils in liberty
    in commit 60411ddf1bada3629f0588de8726c63b5ab00357. So the
    nova.compute.utils changes are omitted here. Also, there was no direct
    testing of _reverse_upsize_quota_delta until that change in liberty
    so as part of this cherry pick we pull back the test_reverse_quota_delta
    test. It's modified slightly since it's in a NoDBTestCase.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)
    (cherry picked from commit 30aad1e796778a94da7719b1d4586bd3cdecb821)
    (cherry picked from commit 5374568fadebf639886febe6073baf352c2a9d83)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309040
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=30aad1e796778a94da7719b1d4586bd3cdecb821
Submitter: Jenkins
Branch: stable/mitaka

commit 30aad1e796778a94da7719b1d4586bd3cdecb821
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309110
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5374568fadebf639886febe6073baf352c2a9d83
Submitter: Jenkins
Branch: stable/liberty

commit 5374568fadebf639886febe6073baf352c2a9d83
Author: Dan Smith <email address hidden>
Date: Mon Apr 18 12:40:54 2016 -0700

    Fix reverse_upsize_quota_delta attempt to look up deleted flavors

    When we did the "great flavor migration of 2015" we missed a quota method
    which still looks up flavors by id from the migration. Now that flavors
    are moved to the api database and actually removed when deleted, this no
    longer works. The problem manifests itself as a failure when trying to
    revert a migration or resize operation when the original flavor has been
    deleted.

    Conflicts:
            nova/tests/unit/compute/test_compute_utils.py

    NOTE(mriedem): The conflict is due to uuidsentinel not being imported
    in liberty (and is not needed here) and test_migration still being
    used in liberty (but not mitaka).

    Change-Id: I5f95021410a309ac07fe9f474cbcd0214d1af208
    Closes-Bug: #1570748
    (cherry picked from commit a46e847aad4ee7edbb63eb08f97f6635e6c9ccb0)
    (cherry picked from commit 30aad1e796778a94da7719b1d4586bd3cdecb821)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307518
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=98c32b7860f03bedbe2d4d8e35ec53cfa0c0c5cb
Submitter: Jenkins
Branch: master

commit 98c32b7860f03bedbe2d4d8e35ec53cfa0c0c5cb
Author: Matt Riedemann <email address hidden>
Date: Mon Apr 18 19:34:47 2016 -0400

    Add a test for reverting a resize with a deleted flavor

    A user should be able to boot a server, resize it and
    then reject the resize even though the original flavor
    used to boot the instance is deleted. This is because
    the old flavor information is stored with the instance
    in the nova database, so the original flavor doesn't
    actually need to exist anymore.

    Depends-On: I5f95021410a309ac07fe9f474cbcd0214d1af208

    Change-Id: I356411f96a601f1443d75ac90e42567bef1f8228
    Closes-Bug: #1570748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/311145
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=e72a505860b2f7ce2dd4a492f65d426f8b091e1f
Submitter: Jenkins
Branch: master

commit e72a505860b2f7ce2dd4a492f65d426f8b091e1f
Author: Drew Thorstensen <email address hidden>
Date: Fri Apr 29 10:12:33 2016 -0500

    Fix revert migration if flavor deleted

    After an instance is created, the flavor could be deleted. But
    operations still need to be able to occur against the VM when that
    occurs. A new test in Tempest showed that we had been using the
    incorrect (and potentially deleted) flavor when a resize/migration was
    reverting.

    This change updates the code to use the flavor that is stored on the
    instance itself.

    Change-Id: I16254d8eb74a724e3e0a538f47259e9627c4233a
    Closes-Bug: 1570748

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/311521

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/311521
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=f98749a2d3154452d8edb8c0e95b54f5cde579d1
Submitter: Jenkins
Branch: stable/mitaka

commit f98749a2d3154452d8edb8c0e95b54f5cde579d1
Author: Drew Thorstensen <email address hidden>
Date: Fri Apr 29 10:12:33 2016 -0500

    Fix revert migration if flavor deleted

    After an instance is created, the flavor could be deleted. But
    operations still need to be able to occur against the VM when that
    occurs. A new test in Tempest showed that we had been using the
    incorrect (and potentially deleted) flavor when a resize/migration was
    reverting.

    This change updates the code to use the flavor that is stored on the
    instance itself.

    Change-Id: I16254d8eb74a724e3e0a538f47259e9627c4233a
    Closes-Bug: 1570748
    (cherry picked from commit e72a505860b2f7ce2dd4a492f65d426f8b091e1f)
    Conflicts:
     nova_powervm/tests/virt/powervm/test_driver.py

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 2015.1.4 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.4 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 2015.1.4 release.