cinder volume deletion fails with ceph backend

Bug #1292433 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
Edward Hope-Morley
cinder (Ubuntu)
Fix Released
Critical
Unassigned
Trusty
Fix Released
Critical
Unassigned

Bug Description

Detected while running the tempest.api.compute.servers.test_server_rescue tempest smoke test:

2014-03-14 09:26:14.567 17626 INFO cinder.volume.flows.manager.create_volume [req-5525c9a2-e57e-4b7e-95df-4a83eb9d3118 None] Volume a3efff36-7803-4a71-9764-00faf481132d: being created using CreateVolumeFromSpecTask._create_raw_volume with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-a3efff36-7803-4a71-9764-00faf481132d'}
2014-03-14 09:26:14.643 17626 INFO cinder.volume.flows.manager.create_volume [req-5525c9a2-e57e-4b7e-95df-4a83eb9d3118 None] Volume volume-a3efff36-7803-4a71-9764-00faf481132d (a3efff36-7803-4a71-9764-00faf481132d): created successfully
2014-03-14 09:26:39.615 17626 INFO cinder.volume.manager [-] Updating volume status
2014-03-14 09:26:46.594 17626 INFO cinder.volume.manager [req-c19a3ec4-df8f-48ab-9360-8f695d83502d None] volume a3efff36-7803-4a71-9764-00faf481132d: deleting
2014-03-14 09:26:46.653 17626 ERROR cinder.openstack.common.rpc.amqp [req-c19a3ec4-df8f-48ab-9360-8f695d83502d None] Exception during message handling
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp **args)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 143, in lvo_inner1
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp return lvo_inner2(inst, context, volume_id, **kwargs)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/lockutils.py", line 233, in inner
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 142, in lvo_inner2
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp return f(*_args, **_kwargs)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 413, in delete_volume
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'})
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 402, in delete_volume
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp self.driver.delete_volume(volume_ref)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 591, in delete_volume
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp rbd_image = self.rbd.Image(client.ioctx, volume_name)
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/rbd.py", line 337, in __init__
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp raise TypeError('name must be a string')
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp TypeError: name must be a string
2014-03-14 09:26:46.653 17626 TRACE cinder.openstack.common.rpc.amqp

volume goes into state error_deleting:

+--------------------------------------+----------------+----------------------------------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+----------------------------------------+------+-------------+----------+-------------+
| 8ce566a4-cde5-4b84-9272-7edb47bc2e21 | error_deleting | Volume-json--1711121506 | 1 | None | false | |
| a3efff36-7803-4a71-9764-00faf481132d | error_deleting | ServerRescueTestXML_volume-648856801 | 1 | None | false | |
| dd43b650-36ae-4200-a1a0-54522bcf313a | error_deleting | ServerRescueTestJSON_volume-1077148935 | 1 | None | false | |
+--------------------------------------+----------------+----------------------------------------+------+-------------+----------+-------------+

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: cinder-volume 1:2014.1~b3-0ubuntu2
ProcVersionSignature: User Name 3.13.0-17.37-generic 3.13.6
Uname: Linux 3.13.0-17-generic x86_64
ApportVersion: 2.13.3-0ubuntu1
Architecture: amd64
Date: Fri Mar 14 09:30:19 2014
Ec2AMI: ami-0000003c
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: serverstack-az-1
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cinder
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.cinder.rootwrap.d.volume.filters: [deleted]
upstart.cinder-volume.override: env CEPH_ARGS="--id cinder"

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Note that is can be reproduced by trying to delete any cinder volume:

cinder create 5
cinder delete <uuid>

+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 72d9c32a-1474-41b9-b6b2-26b3ff641944 | error_deleting | None | 1 | None | false | |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

Revision history for this message
Edward Hope-Morley (hopem) wrote :

I have seen this sort of error previously when the name supplied to rbd.Image is unicode formatted (or non-utf8). We should call strutils.safe_encode() on the name to be safe like in cinder.backup.drivers.ceph.

Changed in cinder:
assignee: nobody → Edward Hope-Morley (hopem)
status: New → In Progress
Revision history for this message
James Page (james-page) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/80530

James Page (james-page)
Changed in cinder (Ubuntu Trusty):
importance: Undecided → Critical
status: New → Triaged
James Page (james-page)
tags: added: ubuntu-ci
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cinder - 1:2014.1~b3-0ubuntu3

---------------
cinder (1:2014.1~b3-0ubuntu3) trusty; urgency=medium

  * d/p/fixup-rbd-str-handling.patch: Cherry pick fix from upstream
    Gerrit to resolve issue deleting Ceph volumes and snapshots
    (LP: #1292433).
  * d/control,rules: Use upstream run_tests.sh wrapper to execute unit tests,
    add subunit to BD's to ensure output is correctly formatted.
 -- James Page <email address hidden> Fri, 14 Mar 2014 11:19:16 +0000

Changed in cinder (Ubuntu Trusty):
status: Triaged → Fix Released
Changed in cinder:
milestone: none → icehouse-rc1
importance: Undecided → Critical
Revision history for this message
John Griffith (john-griffith) wrote :

Merged: https://review.openstack.org/#/c/80530/

Patch had an incorrect bug-id associated with it.

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.