vif unplug is invoked after vm delete

[Sridhar Venkat]

During vm delete operation, vif unplug is invoked after delete is performed. Due to this problem, since vm is deleted, unplug reports an error.
This is related to liberty.
Reproduce steps :
1. Deploy a VM
2. Delete VM
Does not happen very often, but in continuous deploy/delete scenarios, this problem is reproduced.

VM got deleted successfully, but vif unplug failed to locate vm to remove vlan associated with it.

[Drew Thorstensen (thorst)]

So, there does need to be a question why the manager allowed this. Sridhar, I believe you have a stack trace though that shows that we are not raising a proper NovaException in this case? Can you add that here?

[Sridhar Venkat (svenkat)]

Stack trace :

2015-11-06 18:27:18.778 34392 INFO nova_powervm.virt.powervm.tasks.vm [req-3eaee1d9-a23b-4301-87bb-39099dec253b 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 b7ed54d96abd4b14b0444827eebc3eb3 - - -] Deleting instance HAR_RHEL_7DT_-468bf5c8-000000cd from system.
2015-11-06 18:27:18.779 34392 INFO nova_powervm.virt.powervm.vm [req-3eaee1d9-a23b-4301-87bb-39099dec253b 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 b7ed54d96abd4b14b0444827eebc3eb3 - - -] Deleting virtual machine. LPARID: 468BF5C8-D533-484E-B35A-61313EA4E65B
2015-11-06 18:27:19.190 34392 INFO nova_powervm.virt.powervm.vm [req-3eaee1d9-a23b-4301-87bb-39099dec253b 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 b7ed54d96abd4b14b0444827eebc3eb3 - - -] Virtual machine delete status: 204

...

2015-11-06 18:27:22.601 34392 INFO nova_powervm.virt.powervm.driver [req-20cba4d3-8cd7-4c44-8215-3f80ee8866bc ccef2cf310f3c87a1f7694fd1efec05df65170786399ba92e1aa07fbe72f746f f86572c7774b4d30a3eea74ab72c5011 - - -] Operation: unplug_vifs. Virtual machine display name: HAR_RHEL_7DT_001_199, name: HAR_RHEL_7DT_-468bf5c8-000000cd, UUID: 468bf5c8-d533-484e-b35a-61313ea4e65b
...

2015-11-06 18:27:22.299 34392 INFO nova_powervm.virt.powervm.driver [req-3536e9f3-6fc6-4a28-ad19-c6895ec27d22 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 b7ed54d96abd4b14b0444827eebc3eb3 - - -] Operation: destroy. Virtual machine display name: HAR_RHEL_7DT_002_009, name: HAR_RHEL_7DT_-9dfb56ab-000000d7, UUID: 9dfb56ab-0ed2-478d-9ed6-809f7eddcd1f
2015-11-06 18:27:22.524 34392 INFO nova.compute.manager [req-20cba4d3-8cd7-4c44-8215-3f80ee8866bc ccef2cf310f3c87a1f7694fd1efec05df65170786399ba92e1aa07fbe72f746f f86572c7774b4d30a3eea74ab72c5011 - - -] [instance: 468bf5c8-d533-484e-b35a-61313ea4e65b] Neutron deleted interface 64c735a4-f18d-4e9b-9846-2abe834d8b0d; detaching it from the instance and deleting it from the info cache
2015-11-06 18:27:22.601 34392 INFO nova_powervm.virt.powervm.driver [req-20cba4d3-8cd7-4c44-8215-3f80ee8866bc ccef2cf310f3c87a1f7694fd1efec05df65170786399ba92e1aa07fbe72f746f f86572c7774b4d30a3eea74ab72c5011 - - -] Operation: unplug_vifs. Virtual machine display name: HAR_RHEL_7DT_001_199, name: HAR_RHEL_7DT_-468bf5c8-000000cd, UUID: 468bf5c8-d533-484e-b35a-61313ea4e65b
2015-11-06 18:27:23.561 34392 INFO pypowervm.helpers.log_helper [req-20cba4d3-8cd7-4c44-8215-3f80ee8866bc ccef2cf310f3c87a1f7694fd1efec05df65170786399ba92e1aa07fbe72f746f f86572c7774b4d30a3eea74ab72c5011 - - -] REQUEST: {'auditmemento': 'None', 'sensitive': 'False', 'headers': "{'Accept': 'application/atom+xml', 'X-Audit-Memento': 'nova'}", 'timeout': '-1', 'path': '/rest/api/uom/ManagedSystem/7bf5091c-0a4e-3c79-968c-22eb636be47d/LogicalPartition/468BF5C8-D533-484E-B35A-61313EA4E65B?group=None', 'method': 'GET'}
2015-11-06 18:27:23.562 34392 INFO pypowervm.helpers.log_helper [req-20cba4d3-8cd7-4c44-8215-3f80ee8866bc ccef2cf310f3c87a1f7694fd1efec05df65170786399ba92e1aa07fbe72f746f f86572c7774b4d30a3eea74ab72c5011 - - -] RESPONSE: {'status': '404', 'adapter': 'None', 'reqbody': '', 'reqpath': '/rest/api/uom/ManagedSystem/7bf5091c-0a4e-3c79-968c-22eb636be47d...

[Markus Zoeller (markus_z) (mzoeller)]

That's an issue which happened with the out-of-tree PowerVM driver, right? IIRC PowerVM does *not* use the in-tree manager, which is responsible for the order of calls. Is that true? I'm asking because I'm unsure if such a bug report would fit here. Any thoughts?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Drew Thorstensen (thorst)]

I got a hold of Sridhar on this. The short answer is that I think this defect can be canceled.

What happened:
 - User started the destroy operation for a VM.
 - User immediately kicked off a detach_interface operation while the destroy was still ongoing

Had the destroy finished before the detach_interface was called, a 404 (instance not found) error would have been exposed. However, the instance did still exist when the command came in.

That led to the issue. I think that this is more a usability issue. It is perfectly OK for the compute log to provide messaging around this.