get_highest_client_server_version is failing with the new uWSGI changes in Devstack

[Ildiko Vancsa]

Since https://review.openstack.org/#/c/441266/ got merged the get_highest_client_server_version() call is failing in the client due to hitting the Horizon endpoint as opposed to Cinder here:
https://github.com/openstack/python-cinderclient/blob/master/cinderclient/client.py#L87

Aug 01 13:38:49.416030 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions version = _check_microversion(url, microversion)
Aug 01 13:38:49.420189 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/volume/cinder.py", line 71, in _check_microversion
Aug 01 13:38:49.420332 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions max_api_version = cinder_client.get_highest_client_server_version(url)
Aug 01 13:38:49.420422 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 104, in get_highest_client_server_version
Aug 01 13:38:49.420508 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions min_server, max_server = get_server_version(url)
Aug 01 13:38:49.420606 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 88, in get_server_version
Aug 01 13:38:49.420695 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions data = json.loads(response.text)
Aug 01 13:38:49.420778 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
Aug 01 13:38:49.420862 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions return _default_decoder.decode(s)
Aug 01 13:38:49.420945 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
Aug 01 13:38:49.421027 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Aug 01 13:38:49.421115 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
Aug 01 13:38:49.421199 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions raise ValueError("No JSON object could be decoded")
Aug 01 13:38:49.421307 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions ValueError: No JSON object could be decoded
Aug 01 13:38:49.421391 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions
Aug 01 13:40:58.581980 ubuntu-xenial-osic-cloud1-disk-10193603 <email address hidden>[14655]: ERROR nova.api.openstack.extensions [None req-8c102323-b716-488b-857e-aecaaadd4e4f tempest-TestVolumeSwap-387319544 tempest-TestVolumeSwap-387319544] Unexpected exception in API method: ValueError: No JSON object could be decoded

Full logs from a recent Tempest run are here: http://logs.openstack.org/85/330285/107/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/a68338b/logs/screen-n-api.txt.gz?level=ERROR

The issue seems to be related to the Devstack changes as well by the public_url not being constructed correctly. However the client might need to be prepared as well to handle different URL's given to it.

Recent attempts to fix the issue:
https://review.openstack.org/#/c/489715/
https://review.openstack.org/#/c/490040/

[Scott DAngelo (scott-dangelo)]

I'm not clear what the issue is here. The get_highest_client_server_version takes a URL, which should be he naked endpoint of cinder, without any path like "volume" or "volumev3".
It should not be handed the horizon endpoint, that won't work.
The point of the call is that it does not need authentication, so Horizon isn't needed. If the caller needs to call Horizon (and thus have authentication) to get the Cinder endpoint, then everything should be stripped so only the IP:port for cinder is used.

[Ildiko Vancsa (ildiko-vancsa)]

With the uWSGI changes in DevStack for Cinder it surely does not get the right URL so it hits the Horizon endpoint as opposed to Cinder. It would be great if it's purely a DevStack fix, I'm less experienced with how the client is supposed to be structured to handle the URL's and endpoints.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490040
Committed: https://git.openstack.org/cgit/openstack/python-cinderclient/commit/?id=19bb7cdf4fcba4a8db540d584d1e03c9276885bb
Submitter: Jenkins
Branch: master

commit 19bb7cdf4fcba4a8db540d584d1e03c9276885bb
Author: Ivan Kolodyazhny <email address hidden>
Date: Wed Aug 2 17:22:43 2017 +0300

    Fix get_highest_client_server_version with Cinder API + uWSGI

    get_highest_client_server_version should work with any endpoint type:
    cinder-api with eventlet, uWSGI, etc.

    This patch is based on python-novaclient patch [1]. In a future, we can
    deprecate get_highest_client_server_version in flavor of keystoneauth
    descovery feature.

    [1] Icba858b496855e2ffd71b35168e8057b28236119

    Closes-Bug: #1708188
    Change-Id: I9acf6dc84c32b25bfe3254eb0f97248736498d32

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/491519

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/491519
Committed: https://git.openstack.org/cgit/openstack/python-cinderclient/commit/?id=b321dd798575a4e39c01f3b6db29d875cd142fe2
Submitter: Jenkins
Branch: stable/pike

commit b321dd798575a4e39c01f3b6db29d875cd142fe2
Author: Ivan Kolodyazhny <email address hidden>
Date: Wed Aug 2 17:22:43 2017 +0300

    Fix get_highest_client_server_version with Cinder API + uWSGI

    get_highest_client_server_version should work with any endpoint type:
    cinder-api with eventlet, uWSGI, etc.

    This patch is based on python-novaclient patch [1]. In a future, we can
    deprecate get_highest_client_server_version in flavor of keystoneauth
    descovery feature.

    [1] Icba858b496855e2ffd71b35168e8057b28236119

    Closes-Bug: #1708188
    Change-Id: I9acf6dc84c32b25bfe3254eb0f97248736498d32
    (cherry picked from commit 19bb7cdf4fcba4a8db540d584d1e03c9276885bb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-cinderclient 3.2.0 release.

[Dr. Jens Harbott (j-harbott)]

Seems that this was resolved in the cinder client, please update if there is still an issue with devstack.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-cinderclient 3.1.1 release.