unable to destroy bare-metal instance when flavor is deleted

Bug #1400269 reported by Tan Lin
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Paul Murray
Juno
Fix Released
Undecided
Unassigned

Bug Description

Get error if delete flavor before delete instance.
This is caused by ironic driver in nova.
We have below code in _cleanup_deploy function in nova/virt/ironic/driver.py
if flavor is None:
            # TODO(mrda): It would be better to use instance.get_flavor() here
            # but right now that doesn't include extra_specs which are required
            flavor = objects.Flavor.get_by_id(context,
                                              instance['instance_type_id'])
So if the flavor is deleted before we destroy bare metal node, we get FlavorNotFoud exception without handling.

At same time, I found the flavor is used to clean deploy ramdisk/kernel in driver_info, which was plan to remove in Kilo, are we ready for that?

Tan Lin (tan-lin-good)
Changed in nova:
assignee: nobody → Tan Lin (tan-lin-good)
Changed in ironic:
assignee: nobody → Tan Lin (tan-lin-good)
description: updated
Revision history for this message
Dmitry Tantsur (divius) wrote :

Thank you for reporting. This bug should be fixed within Nova tree, as to my best knowledge Ironic is not even aware of flavor concept.

Changed in nova:
status: New → Confirmed
Changed in ironic:
status: New → Invalid
Revision history for this message
Clint Byrum (clint-fewbar) wrote :
Download full text (4.3 KiB)

I can confirm this error on a very recent nova+ironic. While some nodes were failing deploy, I deleted and re-created the flavor to have less disk requirement, and now I cannot delete the instances:

+--------------------------------------+---------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+---------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | seed |
| OS-EXT-SRV-ATTR:hypervisor_hostname | 772af6ac-d053-47cf-88b5-3cce94191bb4 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | deleting |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-12-11T10:27:04Z |
| fault | {"message": "Flavor 7 could not be found.", "code": 404, "created": "2014-12-11T10:37:51Z"} |
| flavor | baremetal (82f959f5-b20e-44b2-9604-f68abc19e214) |
| hostId | e7fd3fa395a116811e038357c0c1fc0f37af09d8c600294e1631a1d6 |
| id | 44a7bb32-9563-492a-b20e-1354193651d2 |
| image | overcloud-compute (6250ce83-56f6-4b5f-8bcd-d3f170b9b7ae) |
| key_name ...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/141104

Changed in nova:
assignee: Tan Lin (tan-lin-good) → Jim Rollenhagen (jim-rollenhagen)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/142123

Changed in nova:
assignee: Jim Rollenhagen (jim-rollenhagen) → Paul Murray (pmurray)
Tan Lin (tan-lin-good)
no longer affects: ironic
Revision history for this message
Paul Murray (pmurray) wrote :

Devananda's comments in https://review.openstack.org/141104 note that there are no log messages warning about the changes related to ramdisk/kernal, so it has to be left for now. So this fix should be done anyway.

The fix https://review.openstack.org/142123 does that single change to address this bug.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/142123
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c4eab7062301b8f3b2de2358c589aee4c53074ef
Submitter: Jenkins
Branch: master

commit c4eab7062301b8f3b2de2358c589aee4c53074ef
Author: Paul Murray <email address hidden>
Date: Tue Dec 16 14:20:32 2014 +0000

    Fix ironic delete fails when flavor deleted

    The ironic virt driver looks up the flavor of an instance
    when it is going to delete it. This is to obtain extra
    specs details that are not available in the instance details.
    If the flavor has been deleted this lookup fails and causes
    the delete to fail.

    The fix makes the lookup include deleted flavors. Note that
    extra specs handling is changing in nova, so this code is
    likely to become obsolete when they are available by
    other means.

    Change-Id: I47ba78abfe60e82226acc6a17752db503d9f21d8
    Co-Authored-By: Nicholas Randon <email address hidden>
    Co-Authored-By: Phil Day <email address hidden>
    Closes-Bug: #1400269

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/143132

Paul Murray (pmurray)
tags: added: juno-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/143132
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee66c0436ba969adae519fa5033b8da0fd04a30f
Submitter: Jenkins
Branch: stable/juno

commit ee66c0436ba969adae519fa5033b8da0fd04a30f
Author: Paul Murray <email address hidden>
Date: Tue Dec 16 14:20:32 2014 +0000

    Fix ironic delete fails when flavor deleted

    The ironic virt driver looks up the flavor of an instance
    when it is going to delete it. This is to obtain extra
    specs details that are not available in the instance details.
    If the flavor has been deleted this lookup fails and causes
    the delete to fail.

    The fix makes the lookup include deleted flavors. Note that
    extra specs handling is changing in nova, so this code is
    likely to become obsolete when they are available by
    other means.

    Change-Id: I47ba78abfe60e82226acc6a17752db503d9f21d8
    Co-Authored-By: Nicholas Randon <email address hidden>
    Co-Authored-By: Phil Day <email address hidden>
    Closes-Bug: #1400269
    (cherry picked from commit c4eab7062301b8f3b2de2358c589aee4c53074ef)
    Conflicts:
            nova/tests/unit/virt/ironic/test_driver.py
            nova/virt/ironic/driver.py

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-2 → 2015.1.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/141104
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.