Ophaned Volume Not Removed on Instance Delete

Bug #1344113 reported by Mat Lowery
22
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Ali Asgar Adil

Bug Description

When the trove.taskmanager.models.FreshInstanceTasks#_create_server_volume_individually path is used, cinderclient is used to create the volume and then that volume is supplied in the block_device_mapping (BDM) in the novaclient call. delete_on_terminate is set to True in that BDM. However, in a scenario in which cinderclient succeeds and the subsequent novaclient fails (e.g. quota), the volume is not cleaned up since there is no nova server.

relevant code-path: https://github.com/openstack/trove/blob/1654031cc6b0fa28efb322717efabf2bebffabf8/trove/taskmanager/models.py#L583-L597

this is not good, because the quotas are rolled back, meaning there's a provisioned volume that's not accounted for.

Changed in trove:
status: New → Confirmed
Changed in trove:
status: Confirmed → Triaged
importance: Undecided → Medium
milestone: none → juno-3
milestone: juno-3 → ongoing
description: updated
Changed in trove:
milestone: ongoing → kilo-1
Changed in trove:
assignee: nobody → Auston McReynolds (amcrn)
summary: - volume is not deleted if nova call fails after cinder call
+ Ophaned Volume Not Removed on Instance Delete
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/125550

Changed in trove:
status: Triaged → In Progress
Changed in trove:
milestone: kilo-1 → kilo-2
Changed in trove:
milestone: kilo-2 → kilo-3
Changed in trove:
milestone: kilo-3 → kilo-rc1
Changed in trove:
milestone: kilo-rc1 → liberty-1
Changed in trove:
milestone: liberty-1 → liberty-2
Changed in trove:
assignee: Auston McReynolds (amcrn) → Nikhil Manchanda (slicknik)
milestone: liberty-2 → liberty-3
Changed in trove:
milestone: liberty-3 → ongoing
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on trove (master)

Change abandoned by Craig Vyvial (<email address hidden>) on branch: master
Review: https://review.openstack.org/125550
Reason: Abandoning due to lack activity updating this patch. Feel free to open this patch back up. Thanks.

Revision history for this message
Matt Fischer (mfisch) wrote :

I would argue that this should be high. When this happens you can run out of quota which is not apparent in the trove logs and is confusing to customers.

Amrith Kumar (amrith)
Changed in trove:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/334722

Changed in trove:
assignee: Nikhil Manchanda (slicknik) → Ali Asgar Adil (aadil-b)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/334722
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=c3d670c2186e681d89cfa511984eff7255ef0826
Submitter: Jenkins
Branch: master

commit c3d670c2186e681d89cfa511984eff7255ef0826
Author: Ali Adil <email address hidden>
Date: Mon Jun 27 20:47:23 2016 -0400

    Ophaned Volume Not Removed on Instance Delete

    When an instance is created with quota issues, the cinder volume is
    created butthe nova instance fails to be created. Deleting the
    instance would fail todelete the volume. Check the status of the
    volume when deleting the instance, if the status of the volume is
    "available" delete the volume as it is not connected to any instance.

    Change-Id: Ie921a8ff2851e2d9d76a3c3836945c750f090c4e
    Closes-Bug: #1344113

Changed in trove:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/trove 6.0.0.0b3

This issue was fixed in the openstack/trove 6.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/trove 6.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.