cinderclient.volumes.set_metadata and volume_snapshots.set_metadata() does not support unicode

[Paul Karikh]

Looks like cinderclient.volumes.set_metadata() and cinderclient.volume_snapshots.set_metadata() does not support unicode values for keys in metadata.
If cal these methods with unicode chars (for example, metadata= {u'тестовый': u'тестовое_значение'}) it fails with followng error:

File "/horizon/.venv/local/lib/python2.7/site-packages/cinderclient/v3/volume_snapshots.py", line 175, in set_metadata
    body, "metadata")
  File "/horizon/.venv/local/lib/python2.7/site-packages/cinderclient/base.py", line 323, in _create
    return self.resource_class(self, body[response_key], resp=resp)
  File "/horizon/.venv/local/lib/python2.7/site-packages/cinderclient/openstack/common/apiclient/base.py", line 468, in __init__
    self._add_details(info)
  File "/horizon/.venv/local/lib/python2.7/site-packages/cinderclient/openstack/common/apiclient/base.py", line 494, in _add_details
    setattr(self, k, v)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

Looks like actually request to cinder service is finished correctly with 200 status and the issue happens when cinderclient tries to return Volumes/VolumeSnapshot Resource type.

[Ivan Kolodyazhny (e0ne)]

Upstream bug: https://bugs.launchpad.net/python-cinderclient/+bug/1622631

[Rodion Tikunov (rtikunov)]

Proposed fix: https://review.fuel-infra.org/#/c/27146/

[Rodion Tikunov (rtikunov)]

Patch https://review.fuel-infra.org/#/c/27146/ for 7.0 allows set Unicode character for volumes and volume snapshots but after that snapshots cannot be deleted. They freeze in the deleting state [0].
In /var/log/cinder/cinder-volume.log appears error [1]

[0] http://paste.openstack.org/show/584676/
[1] http://paste.openstack.org/show/584673/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/27236
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 4630bd31777abdecd68730f29ea32964103d0a72
Author: Ivan Kolodyazhny <email address hidden>
Date: Thu Oct 6 14:22:26 2016

Fix error during set unicode metadata key

Change-Id: I2940670b4174dae800e76667ceecc80c568e613f
Closes-Bug: #1622629
(cherry picked from commit 0fd11f30f4a06f7b8cb701fa476aae4f193ac963)

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Rodion Tikunov <email address hidden>
Review: https://review.fuel-infra.org/28007

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/28007
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 02d484c93a6a280062ddd6399ccbf9cba5b33eb3
Author: Rodion Tikunov <email address hidden>
Date: Mon Oct 31 08:45:22 2016

Fix typo in set unicode metadata key

Fixes typo in commit 4630bd31777abdecd68730f29ea32964103d0a72

Change-Id: I50c59b92f6f65f7b6aa23fd530f6c4b105ad6b1d
Closes-Bug: #1622629
(cherry picked from commit eb4bdd8d944feded57b9eccdfffb1764ca47544a)

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #511.

Steps to verify:
1) Set metadata with unicode key/value to volume's snapshot:
cinder create --metadata t=test --name test1 1
cinder snapshot-create test1
cinder snapshot-metadata <snapshot-id> set t='тест'
cinder snapshot-metadata <snapshot-id> set тест1='тест'
cinder snapshot-metadata-show <snapshot-id>
2) Set metadata with unicode key/value to volume:
cinder metadata test1 set тест='тест2'

Actual results:
All commands are finished successfully.

[Michael Dovgal (mdovgal)]

Link to the merged patch
https://review.fuel-infra.org/#/c/30962/

[TatyanaGladysheva (tgladysheva)]

Verified on 10.0 build #1569.

Steps to verify:
1) Set metadata with unicode key/value to volume's snapshot:
cinder create --metadata t=test --name test1 1
cinder snapshot-create test1
cinder snapshot-metadata <snapshot-id> set t='тест'
cinder snapshot-metadata <snapshot-id> set тест1='тест'
cinder snapshot-metadata-show <snapshot-id>
2) Set metadata with unicode key/value to volume:
cinder metadata test1 set тест='тест2'

Actual results:
All commands are finished successfully.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/27146
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 8f6f7446389cde834ebd38b37f27b35a3aac75af
Author: Ivan Kolodyazhny <email address hidden>
Date: Fri Oct 28 15:20:13 2016

Fix error during set unicode metadata key

Change-Id: I2940670b4174dae800e76667ceecc80c568e613f
Closes-Bug: #1622629
(cherry picked from commit 0fd11f30f4a06f7b8cb701fa476aae4f193ac963)

[TatyanaGladysheva (tgladysheva)]

Verified on 8.0 + MU4 updates.

Steps to verify:
1) Set metadata with unicode key/value to volume's snapshot:
cinder create --metadata t=test --name test1 1
cinder snapshot-create test1
cinder snapshot-metadata <snapshot-id> set t='тест'
cinder snapshot-metadata <snapshot-id> set тест1='тест'
cinder snapshot-metadata-show <snapshot-id>
2) Set metadata with unicode key/value to volume:
cinder metadata test1 set тест='тест2'

Actual results:
All commands are finished successfully.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/33636

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Rodion Tikunov <email address hidden>
Review: https://review.fuel-infra.org/33638

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33636

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/33638
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: bdbabfc071319b4ff2c21aefc2db560077510ae4
Author: Rodion Tikunov <email address hidden>
Date: Wed Apr 26 09:30:12 2017

Fix typo in set unicode metadata key

Fixes typo in commit 4630bd31777abdecd68730f29ea32964103d0a72

Change-Id: I50c59b92f6f65f7b6aa23fd530f6c4b105ad6b1d
Closes-Bug: #1622629
(cherry picked from commit eb4bdd8d944feded57b9eccdfffb1764ca47544a)

[Alexey Stupnikov (astupnikov)]

It turns out that bug's description was too detailed and it led to incomplete bug fix: we are still unable to execute the following commands if unicode is used (example commands):

COMMAND:
cinder create --metadata 'test'='тест' 1
RESULT:
error: argument --metadata: invalid str value: u'test=\u0442\u0435\u0441\u0442'
Try 'cinder help create' for more information.

COMMAND:
# cinder show ba132381-fd76-45f7-a099-8bcf27d65546

RESULT (MOS7):
| metadata | {u'\u0442\u0435\u0441\u0442\u043e\u0432\u044b\u0439': u'\u0442\u0435\u0441\u0442\u043e\u0432\u043e\u0435_\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435'} |

RESULT(MOS9):
ERROR: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)

[Alexey Stupnikov (astupnikov)]

Opened upstream bug #1693151

[Alexey Stupnikov (astupnikov)]

bug #1693151 is solved, we need to backport a fix to our branches.

[Alexey Stupnikov (astupnikov)]

Opened related bugs:

- bug #1693710 to backport a patch from bug #1693151
- bug #1693713 to solve another encoding issue MOS9

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0/mitaka
Change author: Yuriy Nesenenko <email address hidden>
Review: https://review.fuel-infra.org/35455

[Alexey Stupnikov (astupnikov)]

MOS7 patch: https://review.fuel-infra.org/#/c/35457/

[Alexey Stupnikov (astupnikov)]

A patch https://review.fuel-infra.org/#/c/35457/ was tested by Vladimir and he confirmed that fix works. Moving to Fix Released for 7.0-mu8

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/35455
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 9e78c026f42fec2340f52a7dcde694fce8404c0d
Author: Yuriy Nesenenko <email address hidden>
Date: Thu Jun 1 10:25:22 2017

Fix Unicode error printing extra-specs

If type-key is set to unicode the command cinder extra-specs-list
fails with the ERROR: 'ascii' codec can't encode characters in position.
This patch fixes it.

Change-Id: Id82bfbe8870351605f53c7ca029b9aa7e6089f6a
Closes-Bug: #1693713
Related-Bug: #1622629