[nova-client] nova-client returns 500 Error in case of custom field in the glance image

[Michael Kraynov]

If we add some custom field to the glance image through the Horizon (by mistake or specially) nova image-show returns 500 BadGateway. It can broke ceilometer when it tries to get the list of the instances.

Steps to reproduce:
1. Deploy environment
2. Login to Horizon
3. Open Images tab
4. Click update metadata
5. Add custom field -> "os_distro="
6. Add value -> rhel
7. Execute nova image-show <image_id>

Expected results:
The command should return the output

Actual result:
root@node-6:~# nova image-show 23ee6b59-4f35-474d-b07e-c4c46fe99853
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.HTTPBadGateway'> (HTTP 500) (Request-ID: req-46c41d2d-ffec-4af7-b9dc-01d8bebaabb8)
root@node-6:~# glance image-show 23ee6b59-4f35-474d-b07e-c4c46fe99853
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 88d6c77b58fd40a7cb7f44b62bd5ad98 |
| container_format | bare |
| created_at | 2016-10-13T06:32:37Z |
| disk_format | qcow2 |
| id | 23ee6b59-4f35-474d-b07e-c4c46fe99853 |
| min_disk | 0 |
| min_ram | 64 |
| name | TestVM |
| os_distro | rhel1 |
| os_distro= | rhel |

2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions [req-46c41d2d-ffec-4af7-b9dc-01d8bebaabb8 1d133857189940db9c09870bd23f9b02 a775a3c542f24cad8f9cdbbfc1a59c55 - - -] Unexpected exception in API method
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/images.py", line 87, in show
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions image = self._image_api.get(context, id)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 93, in get
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions show_deleted=show_deleted)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 333, in show
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions _reraise_translated_image_exception(image_id)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 682, in _reraise_translated_image_exception
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions six.reraise(new_exc, None, exc_trace)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 331, in show
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions image = self._client.call(context, version, 'get', image_id)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 250, in call
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions result = getattr(client.images, method)(*args, **kwargs)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 132, in get
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions % urlparse.quote(str(image_id)))
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in head
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions return self._request('HEAD', url, **kwargs)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 267, in _request
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions resp, body_iter = self._handle_response(resp)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 97, in _handle_response
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions raise exc.from_response(resp, resp.content)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions HTTPBadGateway: HTTPBadGateway (HTTP 502)
2016-10-20 12:35:33.604 743 ERROR nova.api.openstack.extensions

Impact:
It brakes ceilometer on the compute host

2016-10-20 12:37:05.331 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] REQ: curl -g -i -X GET http://172.16.33.3:8774/v2.1/flavors/1 -H "User-Agent: python-novaclient" -H
"Accept: application/json" -H "X-Auth-Token: {SHA1}c4fa1fdfaafd980dc6dc9d0ba17f9ae756848a2b" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-10-20 12:37:05.443 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] RESP: [200] Content-Length: 356 X-Compute-Request-Id: req-4001eaf0-a756-4932-836c-ba63591a722c Vary:
 X-OpenStack-Nova-API-Version Connection: close X-Openstack-Nova-Api-Version: 2.1 Date: Thu, 20 Oct 2016 12:37:05 GMT Content-Type: application/json
RESP BODY: {"flavor": {"name": "m1.tiny", "links": [{"href": "http://172.16.33.3:8774/v2.1/flavors/1", "rel": "self"}, {"href": "http://172.16.33.3:8774/flavors/1", "rel": "bookmark"}], "ram": 512, "OS-FLV-DISA
BLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 1, "id": "1"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-10-20 12:37:05.444 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] REQ: curl -g -i -X GET http://172.16.33.3:8774/v2.1/images/23ee6b59-4f35-474d-b07e-c4c46fe99853 -H "
User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}c4fa1fdfaafd980dc6dc9d0ba17f9ae756848a2b" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-10-20 12:37:05.896 32020 DEBUG novaclient-debug [req-80c09252-4204-4a7b-a0c3-41c7472cfad1 admin - - - -] RESP: [500] Content-Length: 205 X-Compute-Request-Id: req-a524a7df-c3ad-4982-8c41-bc97fb2ea0cc Vary:
 X-OpenStack-Nova-API-Version Connection: close X-Openstack-Nova-Api-Version: 2.1 Date: Thu, 20 Oct 2016 12:37:05 GMT Content-Type: application/json; charset=UTF-8
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'glanceclient.exc.HTTPBadGateway'>", "code":
 500}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277

[Vitaly Sedelnik (vsedelnik)]

Michael - please provide MOS version and attach diagnostic snapshot.

[Michael Kraynov (mkraynov)]

MOS: 9.0

Snapshot: https://drive.google.com/a/mirantis.com/file/d/0B7jmmA7m2Wc3WU9lU1lfTHUwSlU/view?usp=sharing

[Roman Podoliaka (rpodolyaka)]

I can confirm this is reproduced on MOS 9.0. This is what I see in haproxy logs:

http://paste.openstack.org/show/589873/

i.e. nova-api simply shows the error it gets from haproxy - 502 bad gateway. Note, that this request is served successfully from glance-api standpoint, but haproxy rejects it due to malformed header (= in header name? - http://agiletesting.blogspot.com/2014/07/troubleshooting-haproxy-502-errors.html)

[Roman Podoliaka (rpodolyaka)]

MOS Glance, could you please take a look at this one? I suggest we either add validation to prevent PATCH with such header or somehow escape the header name properly if it contains "=" character.

[Kairat Kushaev (kkushaev)]

Unfortunately, there is no simple solution for this fix.
The proper one is not use glance v1 api because it interprets all metadata as headers. In glance v2 this gap was fixed and header body was sent as response body. So long term solution is move ceilometer to v2 (nova already moved to v2 in Newton).
I am trying to figure out if we can somehow escape there characters because we can't prohibit creation of this metadata(this breaks backward compatibility).

[Kairat Kushaev (kkushaev)]

Ok, I almost sure it will not be easy to fix this in 9.x because Nova didn't move to glance v2 in Mitaka.
Let me describe the situation in details.
By default Ceilometer uses glance v1 to request image info. This api was deprecated in Newton because it had several design flows and one of them is passing image metadata as http headers.
This issue was fixed in glance v2 and now all image metadata send as request body.
In the flow above image was created through glance v2 and requested through glance v1. HAProxy doesn't allow any special characters as header names in response and we have a bug because of this. Currently I couldn't find any possibility to exclude these characters from request without issues.
Unfortunately image metadata property name can contain different characters so we cannot just prohibit creation of properties with "=", "," and other symbols because it breaks backward compatibility (remember glance v2 doesn't use http headers for metadata at all).
It seems we can solve this in several ways:
1. We can relax validation for header in haproxy (see accept-invalid-http-response in http://www.haproxy.org/download/1.4/doc/configuration.txt), we need to update glance-api/registry haproxy config because they use v1 by default.
2. We can try to custom properties from instance-list request on nova or ceilometer side, so only specific set of properties will be requested. Need nova team feedback on this.
3. Move Nova to glance v2 (impossible due to high risk).
So I would liek to ask Fuel team, is it possible to relax response header restriction for glance-api?

[Vladimir Kuklin (vkuklin)]

This bug is obviously not related to Fuel, but is clearly a bug of glance v1 API.

RFC2616 States the following

https://tools.ietf.org/html/rfc2616#section-4.2

CHAR = <any US-ASCII character (octets 0 - 127)>
...
token = 1*<any CHAR except CTLs or separators>
separators = "(" | ")" | "<" | ">" | "@"
                      | "," | ";" | ":" | "\" | <">
                      | "/" | "[" | "]" | "?" | "="
                      | "{" | "}" | SP | HT
....
       message-header = field-name ":" [ field-value ]
       field-name = token

Thus, it means, that the response headers for custom fields MUST be URL encoded and decoded on the client side. Or this particular feature should be mentioned in the documentation. In any case this is an invalid bug for Fuel

[Vitaly Sedelnik (vsedelnik)]

Should be documented as known issue for 9.2 - adding custom field to glance image leads to 500 error for nova image-show command

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/29769

[Vitaly Sedelnik (vsedelnik)]

Retargeted to 9.3 to consider actual fix.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29769
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: 1f2f3fbf7f5ca40bc2fb1e122507bf7d3e2d27ef
Author: Mariia Zlatkova <email address hidden>
Date: Tue Jan 17 14:01:21 2017

[RN-9.2] Glance known issue

Change-Id: Ie8246783d13d589452f939c2efd1c6c6fe598091
Related-Bug: #1635241

[Denis Meltsaykin (dmeltsaykin)]

It seems the bug is tricky to solve and it is not ready yet. Since we already have a known issue in the docs and there is a workaround (set the metadata of an image through the nova cli rather than horizon), I'm moving the bug from the scope of 9.2-MU-1 to 9.x-updates to have more time to fix it properly.