Detached and deleted RBD volumes remain associated with instance
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Cinder |
Undecided
|
Adam Gandelman | ||
| Folsom |
Critical
|
Adam Gandelman | ||
| OpenStack Compute (nova) |
Undecided
|
Unassigned | ||
| Folsom |
Critical
|
Adam Gandelman | ||
| cinder (Ubuntu) |
Undecided
|
Unassigned | ||
| Quantal |
Undecided
|
Unassigned | ||
| nova (Ubuntu) |
Undecided
|
Unassigned | ||
| Quantal |
Undecided
|
Unassigned |
Bug Description
Using the RBD driver with nova-volume, volumes can be created and attached to instances okay. However, once they have been detached and deleted, an association between the instance the volume remains, causing issues for the EC2 API.
$ euca-run-instances -t m1.tiny ami-01
$ euca-create-volume -s 1 -z nova
$ euca-describe-
$ euca-attach-volume -i i-014 vol-0f -d /dev/vdc
$ euca-describe-
$ euca-describe-
VolumeNotFound: Volume vol-0000000f could not be found.
The equivalent OSAPI commands can be used to trigger the same end-result (a broken euca-describe-
Shortly after the volume has been detached from the instance, the nova-volume.log shows the following trace:
2012-11-27 17:42:45 8727 ERROR nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 TRACE nova.openstack.
2012-11-27 17:42:45 8727 ERROR nova.openstack.
The nova-api-ec2.log for describe_instances request shows:
2012-11-27 17:55:33 DEBUG nova.api.ec2 [req-ec9747d4-
2012-11-27 17:55:33 DEBUG nova.compute.api [req-ec9747d4-
2012-11-27 17:55:33 INFO nova.api.ec2 [req-ec9747d4-
2012-11-27 17:55:33 ERROR nova.api.ec2 [req-ec9747d4-
2012-11-27 17:55:33 INFO nova.api.ec2 [req-ec9747d4-
2012-11-27 17:55:33 INFO nova.ec2.
At this point, there is an entry in the block_device_
Found when testing the current stable/folsom in preparation for 2012.2.1. Have not tested using Grizzly / master.
Related branches
- Openstack Ubuntu Testers: Pending requested 2012-11-30
-
Diff: 38 lines (+11/-9)1 file modifieddebian/changelog (+11/-9)
- Openstack Ubuntu Testers: Pending requested 2012-11-30
-
Diff: 121 lines (+100/-4)1 file modifieddebian/changelog (+100/-4)
CVE References
Adam Gandelman (gandelman-a) wrote : | #1 |
Fix proposed to branch: master
Review: https:/
Changed in cinder: | |
assignee: | nobody → Adam Gandelman (gandelman-a) |
status: | New → In Progress |
Fix proposed to branch: stable/folsom
Review: https:/
Reviewed: https:/
Committed: http://
Submitter: Jenkins
Branch: master
commit d030c5b10e9e8d7
Author: Adam Gandelman <email address hidden>
Date: Tue Nov 27 17:51:26 2012 -0800
Improve provider_location cleanup code for RBD.
The RBD driver does not make use of the 'provider_location' field
but the current cleanup code assumes it does. Ensure the field
is in use before testing whether or not it needs fixing.
Fixes bug 1083818.
Change-Id: Id6ff85101f85e7
Changed in cinder: | |
status: | In Progress → Fix Committed |
Changed in nova: | |
status: | New → Invalid |
Reviewed: https:/
Committed: http://
Submitter: Jenkins
Branch: stable/folsom
commit e7877868b13ff61
Author: Adam Gandelman <email address hidden>
Date: Tue Nov 27 17:44:27 2012 -0800
Improve provider_location cleanup code for RBD.
The RBD driver does not make use of the 'provider_location' field
but the current cleanup code assumes it does. Ensure the field
is in use before testing whether or not it needs fixing.
Fixes bug 1083818.
Note: Applying to stable/folsom nova-volume code. Same fix
proposed to cinder @ https:/
Change-Id: I814bd253d2e850
Fix proposed to branch: stable/folsom
Review: https:/
Reviewed: https:/
Committed: http://
Submitter: Jenkins
Branch: stable/folsom
commit 940f363dafd8a51
Author: Adam Gandelman <email address hidden>
Date: Tue Nov 27 17:51:26 2012 -0800
Improve provider_location cleanup code for RBD.
The RBD driver does not make use of the 'provider_location' field
but the current cleanup code assumes it does. Ensure the field
is in use before testing whether or not it needs fixing.
Fixes bug 1083818.
Change-Id: Id6ff85101f85e7
(cherry picked from commit d030c5b10e9e8d7
Changed in cinder (Ubuntu): | |
status: | New → Fix Released |
Changed in cinder (Ubuntu Quantal): | |
status: | New → Confirmed |
Changed in nova (Ubuntu): | |
status: | New → Fix Released |
Changed in nova (Ubuntu Quantal): | |
status: | New → Confirmed |
summary: |
- Detached and deleted RBD volumes remain associated with insance + Detached and deleted RBD volumes remain associated with instance |
Hello Adam, or anyone else affected,
Accepted cinder into quantal-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in cinder (Ubuntu Quantal): | |
status: | Confirmed → Fix Committed |
tags: | added: verification-needed |
Clint Byrum (clint-fewbar) wrote : | #9 |
Hello Adam, or anyone else affected,
Accepted nova into quantal-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in nova (Ubuntu Quantal): | |
status: | Confirmed → Fix Committed |
Changed in cinder: | |
milestone: | none → grizzly-2 |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #10 |
This bug was fixed in the package cinder - 2012.2.1-0ubuntu1
---------------
cinder (2012.2.1-0ubuntu1) quantal-proposed; urgency=low
* Ubuntu updates:
- Cinder should suggest ceph-common, not python-ceph (LP: #1065901):
- debian/control: cinder-volume Suggests: python-ceph -> ceph-common
* Resynchronize with stable/folsom (87d839a5) (LP: #1085255):
- [f990ff0] Remove unused python-daemon dependency
- [940f363] Detached and deleted RBD volumes remain associated with insance
(LP: #1083818)
- [7f34ba3] After folsom upgrade, instances can no longer access existing
volumes. (LP: #1065702)
- [1c99b24] Jenkins jobs fail because of incompatibility between sqlalchemy-
migrate and the newest sqlalchemy-0.8.0b1 (LP: #1073569)
- [d12d4b6] Add SIGPIPE handler to subprocess execution in rootwrap and
utils.execute (LP: #1053364)
- [ce5e002] Set defaultbranch in .gitreview to stable/folsom
-- Adam Gandelman <email address hidden> Tue, 04 Dec 2012 09:19:29 -0800
Changed in cinder (Ubuntu Quantal): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #11 |
This bug was fixed in the package nova - 2012.2.
---------------
nova (2012.2.
* Ubuntu updates:
- debian/control: Ensure novaclient is upgraded with nova,
require python-
- d/p/avoid_
* Dropped patches, applied upstream:
- debian/
* Resynchronize with stable/folsom (b55014ca) (LP: #1085255):
- [a99a802] create_lvm_image allocates dirty blocks (LP: #1070539)
- [670b388] RPC exchange name defaults to 'openstack' (LP: #1083944)
- [3ede373] disassociate_
(LP: #1074437)
- [22d7c3b] libvirt imagecache should handle shared image storage
(LP: #1075018)
- [e787786] Detached and deleted RBD volumes remain associated with insance
(LP: #1083818)
- [9265eb0] live_migration missing migrate_data parameter in Hyper-V driver
(LP: #1066513)
- [3d99848] use_single_
(LP: #1075859)
- [65a2d0a] resize does not migrate DHCP host information (LP: #1065440)
- [102c76b] Nova backup image fails (LP: #1065053)
- [48a3521] Fix config-file overrides for nova-dhcpbridge
- [69663ee] Cloudpipe in Folsom: no such option: cnt_vpn_clients
(LP: #1069573)
- [6e47cc8] DisassociateAddress can cause Internal Server Error
(LP: #1080406)
- [22c3d7b] API calls to dis-associate an auto-assigned floating IP should
return proper warning (LP: #1061499)
- [bd11d15] libvirt: if exception raised during volume_detach, volume state
is inconsistent (LP: #1057756)
- [dcb59c3] admin can't describe all images in ec2 api (LP: #1070138)
- [78de622] Incorrect Exception raised during Create server when metadata
over 255 characters (LP: #1004007)
- [c313de4] Fixed IP isn't released before updating DHCP host file
(LP: #1078718)
- [f4ab42d] Enabling Return Reservation ID with XML create server request
returns no body (LP: #1061124)
- [3db2a38] 'BackupCreate' should accept rotation parameter greater than or
equal to zero (LP: #1071168)
- [f7e5dde] libvirt reboot sometimes fails to reattach volumes
(LP: #1073720)
- [ff776d4] libvirt: detaching volume may fail while terminating other
instances on the same host concurrently (LP: #1060836)
- [85a8bc2] Used instance uuid rather than id in remove-fixed-ip
- [42a85c0] Fix error on invalid delete_
- [6a17579] xenapi migrations fail w/ swap (LP: #1064083)
- [97649b8] attach-time field for volumes is not updated for detach volume
(LP: #1056122)
- [8f6a718] libvirt: rebuild is not using kernel and ramdisk associated with
the new image (LP: #1060925)
- [fbe835f] live-migration and volume host assignement (LP: #1066887)
- [c2a9150] typo prevents volume_tmp_dir flag from working (LP: #1071536)
- [93efa21] Instances deleted during spawn leak network allocations
(LP: #1068716)
- [ebabd02] After restartin...
Changed in nova (Ubuntu Quantal): | |
status: | Fix Committed → Fix Released |
Changed in cinder: | |
milestone: | grizzly-2 → 2013.1 |
Bisected back to e3d7f8c7de9bad7 3bf1f9b5ee9b2cf 46eb452351 / https:/ /review. openstack. org/#/c/ 15005/