delete an shelved_offloaded server cause failure in cinder

Bug #1579667 reported by Peter Wang on 2016-05-09
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned

Bug Description

When deleting an shelved_offloaded STATE VM instance with volume attached, nova passes a connector dictionary:

  connector = {'ip': '127.0.0.1', 'initiator': 'iqn.fake'}

to cinder for terminate connnection, this causes KeyError in cinder driver
code :
https://github.com/openstack/nova/blame/master/nova/compute/api.py#L1803

    def _local_cleanup_bdm_volumes(self, bdms, instance, context):
1804 """The method deletes the bdm records and, if a bdm is a volume, call
1805 the terminate connection and the detach volume via the Volume API.
1806 Note that at this point we do not have the information about the
1807 correct connector so we pass a fake one.
1808 """
1809 elevated = context.elevated()
1810 for bdm in bdms:
1811 if bdm.is_volume:
1812 # NOTE(vish): We don't have access to correct volume
1813 # connector info, so just pass a fake
1814 # connector. This can be improved when we
1815 # expose get_volume_connector to rpc.
1816 connector = {'ip': '127.0.0.1', 'initiator': 'iqn.fake'}
1817 try:
1818 self.volume_api.terminate_connection(context,
1819 bdm.volume_id,
1820 connector)
1821 self.volume_api.detach(elevated, bdm.volume_id,
1822 instance.uuid)
1823 if bdm.delete_on_termination:
1824 self.volume_api.delete(context, bdm.volume_id)
1825 except Exception as exc:
1826 err_str = _LW("Ignoring volume cleanup failure due to %s")
1827 LOG.warn(err_str % exc, instance=instance)
1828 bdm.destroy()
1829
https://github.com/openstack/nova/blame/master/nova/compute/api.py#L1828

according to my debug, the connector info for terminate_connection is already there( in bdm object):

so Nova should build correct connection info for terminate_connection.

======Steps to reproduce

1. create a server: nova boot ----
2. shelve the server: nova shelve <server_id>
3. delete the server: nova delete <server_id>

Thanks
Peter

Peter Wang (peter.wang) wrote :

This should affect all cinder drivers

description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/314083

Changed in nova:
assignee: nobody → Peter Wang (peter.wang)
status: New → In Progress
Peter Wang (peter.wang) wrote :

We found this issue via VNX cinder CI, see log here:
http://publiclogs.emc.com/vnx_ostack/EMC_VNX_iSCSI/276/logs/screen-c-vol.txt.gz

grep by req-id "req-177fbb90-ebdd-411f-8775-d11f4abe005d"

Peter Wang (peter.wang) wrote :

Change abandoned by Peter Wang (<email address hidden>) on branch: master
Review: https://review.openstack.org/314083
Reason: Already fixed in nova now

Peter Wang (peter.wang) on 2016-09-14
Changed in nova:
assignee: Peter Wang (peter.wang) → nobody
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → New
Sean Dague (sdague) wrote :

Is this still an actively worked issue? If so please put into the Confirmed state.

Changed in nova:
status: New → Incomplete
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

Changed in nova:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers