oslo-incubator

delete volume failed due to unicode problems

Bug #1368527 reported by Eric Lee on 2014-09-12

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
Cinder	Fix Released	Medium	Jay Bryant	Cinder 2014.2 "juno"
Icehouse	Fix Released	Medium	Eric Harney
oslo-incubator	Fix Released	High	James Carey	oslo-incubator 2014.2
oslo.concurrency	Invalid	High	James Carey
oslo.utils	Fix Released	High	James Carey

Bug Description

A problem I found when I delete a volume:

2014-09-12 12:39:28.922 25582 ERROR oslo.messaging.rpc.dispatcher [req-df3daaef-5725-48a3-9963-846dbaff1d73 10c46b22ba294231abdfaeb9b1b66791 c8f88a72e4e141f29339a9d30fe93d04 - - -] Exception during message handling: 'ascii' codec can't decode byte 0xe2 in position 19: ordinal not in range(128)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 127, in lvo_inner1
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher return lvo_inner2(inst, context, volume_id, **kwargs)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/lockutils.py", line 233, in inner
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher retval = f(*args, **kwargs)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 126, in lvo_inner2
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher return f(*_args, **_kwargs)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 454, in delete_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher {'status': 'error_deleting'})
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/excutils.py", line 82, in __exit__
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 443, in delete_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher self.driver.delete_volume(volume_ref)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 234, in delete_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher self._delete_volume(volume)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 129, in _delete_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher self._clear_volume(volume, is_snapshot)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 172, in _clear_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher volume_clear_size=self.configuration.volume_clear_size)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/utils.py", line 361, in clear_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher ionice=volume_clear_ionice)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/volume/utils.py", line 296, in copy_volume
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher flag, run_as_root=True)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/utils.py", line 134, in execute
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/processutils.py", line 196, in execute
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher sanitized_stderr = strutils.mask_password(stderr)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/strutils.py", line 295, in mask_password
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher message = six.text_type(message)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19: ordinal not in range(128)
2014-09-12 12:39:28.922 25582 TRACE oslo.messaging.rpc.dispatcher
2014-09-12 12:39:28.925 25582 ERROR oslo.messaging._drivers.common [req-df3daaef-5725-48a3-9963-846dbaff1d73 10c46b22ba294231abdfaeb9b1b66791 c8f88a72e4e141f29339a9d30fe93d04 - - -] Returning exception 'ascii' codec can't decode byte 0xe2 in position 19: ordinal not in range(128) to caller
2014-09-12 12:39:28.926 25582 ERROR oslo.messaging._drivers.common [req-df3daaef-5725-48a3-9963-846dbaff1d73 10c46b22ba294231abdfaeb9b1b66791 c8f88a72e4e141f29339a9d30fe93d04 - - -] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 127, in lvo_inner1\n return lvo_inner2(inst, context, volume_id, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/lockutils.py", line 233, in inner\n retval = f(*args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 126, in lvo_inner2\n return f(*_args, **_kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 454, in delete_volume\n {\'status\': \'error_deleting\'})\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/manager.py", line 443, in delete_volume\n self.driver.delete_volume(volume_ref)\n', ' File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 234, in delete_volume\n self._delete_volume(volume)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 129, in _delete_volume\n self._clear_volume(volume, is_snapshot)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 172, in _clear_volume\n volume_clear_size=self.configuration.volume_clear_size)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/utils.py", line 361, in clear_volume\n ionice=volume_clear_ionice)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/volume/utils.py", line 296, in copy_volume\n flag, run_as_root=True)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/utils.py", line 134, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/processutils.py", line 196, in execute\n sanitized_stderr = strutils.mask_password(stderr)\n', ' File "/usr/local/lib/python2.7/dist-packages/cinder/openstack/common/strutils.py", line 295, in mask_password\n message = six.text_type(message)\n', "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 19: ordinal not in range(128)\n"]

In order to solve this problem, I simply add the following line in line 295 in mask_password method in cinder/openstack/common/strutils.py:
message = message.decode('utf-8')

Tags:

Revision history for this message

Jon Bernard (jbernard) wrote on 2014-09-12:

This is true, nice catch. Running tests on the patch now, should be posted shortly.

Changed in cinder:
assignee:	nobody → Jon Bernard (jbernard)

Jon Bernard (jbernard) on 2014-09-12

Changed in cinder:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-12: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/121143

Revision history for this message

Jon Bernard (jbernard) wrote on 2014-09-12:

A related bug: https://bugs.launchpad.net/oslo.utils/+bug/1366189

Jon Bernard (jbernard) on 2014-09-13

Changed in cinder:
status:	In Progress → Confirmed
assignee:	Jon Bernard (jbernard) → nobody

Revision history for this message

Jay Bryant (jsbryant) wrote on 2014-09-13:

Jim Carey is working the related bug. Assigning to him for now to make sure that we don't forget to pull the fix back once he reaches a solution in oslo.

Changed in cinder:
importance:	Undecided → Medium
milestone:	none → juno-rc1
assignee:	nobody → James Carey (jecarey)
tags:	added: oslo

Doug Hellmann (doug-hellmann) on 2014-09-15

Changed in oslo.concurrency:
milestone:	none → next-kilo
status:	New → Triaged
importance:	Undecided → High
Changed in oslo.utils:
milestone:	none → next-kilo
status:	New → Triaged
importance:	Undecided → High

James Carey (jecarey) on 2014-09-15

Changed in oslo-incubator:
assignee:	nobody → James Carey (jecarey)

Doug Hellmann (doug-hellmann) on 2014-09-15

Changed in oslo-incubator:
importance:	Undecided → High
milestone:	none → next-juno
status:	New → In Progress

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2014-09-16:

Incubator fix: https://review.openstack.org/121632

Changed in oslo-incubator:
status:	In Progress → Fix Committed

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2014-09-16:

oslo.utils temporary fix for juno: https://review.openstack.org/121657

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-16:

Fix proposed to branch: master
Review: https://review.openstack.org/121885

Changed in cinder:
status:	Confirmed → In Progress

OpenStack Infra (hudson-openstack) on 2014-09-16

Changed in cinder:
assignee:	James Carey (jecarey) → Jay Bryant (jsbryant)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-17: Fix merged to cinder (master)

Reviewed: https://review.openstack.org/121885
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=cb5bcca025f9a34ecb9bc72b8fe570e6a63e442d
Submitter: Jenkins
Branch: master

commit cb5bcca025f9a34ecb9bc72b8fe570e6a63e442d
Author: James Carey <email address hidden>
Date: Tue Sep 16 14:13:51 2014 +0000

Sync latest strutils from oslo-incubator for mask_password fix

This sync pulls in:
1131b56 Enable mask_password to handle byte code strings

This is the only update since the last sync:
beca4db Sync latest strutils from oslo-incubator

This is needed to fix a delete volume failure when processutils
calls mask_password with a byte code string.

Change-Id: Ie8a8c8e26abc0e387830bc9246e9a44913ae9b24
Closes-bug: #1368527

Changed in cinder:
status:	In Progress → Fix Committed

Doug Hellmann (doug-hellmann) on 2014-09-19

Changed in oslo.concurrency:
status:	Triaged → Confirmed
Changed in oslo.utils:
status:	Triaged → Confirmed

Doug Hellmann (doug-hellmann) on 2014-09-23

Changed in oslo-incubator:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-01

Changed in cinder:
status:	Fix Committed → Fix Released

Doug Hellmann (doug-hellmann) on 2014-10-08

Changed in oslo.concurrency:
assignee:	nobody → James Carey (jecarey)
Changed in oslo.utils:
assignee:	nobody → James Carey (jecarey)

Thierry Carrez (ttx) on 2014-10-16

Changed in cinder:
milestone:	juno-rc1 → 2014.2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-20: Fix proposed to cinder (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/129613

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-22: Fix merged to cinder (stable/icehouse)

#10

Reviewed: https://review.openstack.org/129613
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=3821206ff1455ffc682739911f7f59bdde6e18d2
Submitter: Jenkins
Branch: stable/icehouse

commit 3821206ff1455ffc682739911f7f59bdde6e18d2
Author: James Carey <email address hidden>
Date: Tue Sep 16 14:13:51 2014 +0000

Sync latest strutils from oslo-incubator for mask_password fix

This sync pulls in:
1131b56 Enable mask_password to handle byte code strings

This is the only update since the last sync:
beca4db Sync latest strutils from oslo-incubator

This is needed to fix a delete volume failure when processutils
calls mask_password with a byte code string.

    Change-Id: Ie8a8c8e26abc0e387830bc9246e9a44913ae9b24
    Closes-bug: #1368527
    (cherry picked from commit cb5bcca025f9a34ecb9bc72b8fe570e6a63e442d)

tags:

added: in-stable-icehouse

Doug Hellmann (doug-hellmann) on 2014-10-24

Changed in oslo.concurrency:
milestone:	0.1.0 → next-kilo

Doug Hellmann (doug-hellmann) on 2014-10-27

Changed in oslo.concurrency:
milestone:	0.2.0 → kilo-next

Revision history for this message

Ben Nemec (bnemec) wrote on 2014-10-27:

#11

oslo.concurrency uses strutils from oslo.utils now, so it will pick up the fix from there.

Changed in oslo.concurrency:
status:	Confirmed → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-17: Change abandoned on cinder (master)

#12

Change abandoned by Mike Perez (<email address hidden>) on branch: master
Review: https://review.openstack.org/121143

Davanum Srinivas (DIMS) (dims-v) on 2014-12-02

Changed in oslo.utils:
milestone:	1.1.0 → next-kilo

Ben Nemec (bnemec) on 2014-12-02

Changed in oslo.concurrency:
milestone:	next-kilo → none

Doug Hellmann (doug-hellmann) on 2015-01-08

Changed in oslo.utils:
milestone:	1.1.1 → none

ChangBo Guo(gcb) (glongwave) on 2015-11-16

Changed in oslo.utils:
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.