Auth token header should not be unicode

[Vincent Untz]

I'm trying to create an image from an instance with non-ascii characters in the name, by using this command:

 nova image-create e6bd72dd-ab8f-4802-8e29-faf90eeceb1c ééé

It turns out that nova-compute fails with this:

015-04-22 16:23:20.537 8401 ERROR oslo.messaging.rpc.dispatcher [req-e35df82c-d6f6-4dd0-9c16-0b399327b528 ] Exception during message handling: 'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher payload)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 307, in decorated_function
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 284, in decorated_function
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 335, in decorated_function
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 323, in decorated_function
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 383, in decorated_function
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher instance=instance)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 373, in decorated_function
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher *args, **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 3039, in snapshot_instance
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 3070, in _snapshot_instance
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher update_task_state)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1795, in snapshot
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher image_file)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/image/api.py", line 126, in update
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher purge_props=purge_props)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/image/glance.py", line 405, in update
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher _reraise_translated_image_exception(image_id)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/image/glance.py", line 403, in update
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher image_id, **image_meta)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/image/glance.py", line 232, in call
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher return getattr(client.images, method)(*args, **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/glanceclient/v1/images.py", line 329, in update
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher resp, body = self.client.put(url, headers=hdrs, data=image_data)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/glanceclient/common/http.py", line 265, in put
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher return self._request('PUT', url, **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/glanceclient/common/http.py", line 206, in _request
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/requests/sessions.py", line 456, in request
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher resp = self.send(prep, **send_kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/requests/sessions.py", line 559, in send
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher r = adapter.send(request, **kwargs)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/requests/adapters.py", line 342, in send
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher low_conn.endheaders()
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/httplib.py", line 914, in endheaders
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher self._send_output()
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/httplib.py", line 784, in _send_output
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher msg = "\r\n".join(self._buffer)
2015-04-22 16:23:20.537 8401 TRACE oslo.messaging.rpc.dispatcher UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)

After digging a bit, it seems self._buffer in httplib is (simplified version):

['PUT /v1/images/e24da13e-71f6-44aa-b26d-972a3b8cf123 HTTP/1.1', 'Host: d52-54-01-77-77-01.c19:9292', 'X-Service-Catalog: $catalog', 'Accept: */*', 'X-Tenant-Id: 8f39cfc227764567b339674018df250f', 'x-image-meta-property-image_state: available', 'User-Agent: python-glanceclient', 'x-image-meta-name: \xc3\xa9\xc3\xa9\xc3\xa9', 'x-image-meta-container_format: ami', u'X-Auth-Token: $long-token', 'x-image-meta-property-image_location: snapshot', 'Accept-Encoding: gzip, deflate', 'Transfer-Encoding: chunked', 'x-glance-registry-purge-props: false', 'x-image-meta-property-ramdisk_id: a55b7c7c-9c74-4aef-9451-8b963eabf5a5', 'X-User-Id: 88a1fea444c24989bdf67980a9270bb5', 'x-image-meta-property-kernel_id: fcaafb77-3144-4b4b-9ebe-b62e5de354bc', 'X-Identity-Status: Confirmed', 'Content-Type: application/octet-stream', 'X-Roles: admin,heat_stack_owner', 'x-image-meta-size: 11010048', 'x-image-meta-is_public: False', 'x-image-meta-property-owner_id: 8f39cfc227764567b339674018df250f', 'x-image-meta-disk_format: ami', '', '']

As you can see, the X-Auth-Token line is an unicode string. It turns out that this indeed causes the failure:

>>> "\r\n".join([u'', 'x-image-meta-name: \xc3\xa9\xc3\xa9\xc3\xa9'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)

So clearly we should make sure to encode the token and not assume anything about it.

[Vincent Untz (vuntz)]

A simpler way to reproduce is:

$ glance image-create --disk-format qcow2 --container-format bare --name ééé --file cirros-0.3.3-x86_64-disk.img --progress
'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)

I'm not totally sure why this doesn't happen for everyone, though :/

[Kamil Rykowski (kamil-rykowski)]

I wasn't able to reproduce the issue, so have two questions:

1. What version of glanceclient are you using?
glance --version
0.17.0

2. What version of glance are you using?
I'm working on the trunk version - cbffaf4db1f24533ae1e51ab264ae60b87d07764

[Vincent Untz (vuntz)]

I was using 0.15.0; I'll try with 0.17.0 as soon as I can, but I don't think there was any change that really impacts this. And to be clear, I think it's likely that even if you go back to 0.15.0, you won't be able to reproduce it. I'm thinking this might be my python version or something like that -- but regardless of that, I'm thinking the token header needs to go through the same process as the other headers.

(the glance version doesn't matter as this happens before talking to glance, afaict -- but that's with stable/juno as of a couple of days ago)

[Vincent Untz (vuntz)]

Checking further, I think it's really an incompatibility with python 2.6, in the putheader method of httplib. It got improved in 2.7 to be more robust when dealing with non-ascii strings.

[Vincent Untz (vuntz)]

That's the python commit that makes it work in 2.7: https://hg.python.org/cpython/rev/e1f6ce836fc5

[Vincent Untz (vuntz)]

Do we still care about 2.6 support for the client code?

[Ian Cordasco (icordasc)]

Vincent, we do. I'll take a look at this today or next week.

[Vincent Untz (vuntz)]

Ian: I submitted https://review.openstack.org/#/c/177183/ for this.

[Stuart McLaren (stuart-mclaren)]

Confirmed:

'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/python-glanceclient 2.0.0 release.