openstacksdk 0.9.2 and higher does have one issue "SDKException: Connection failure that may be retried." when using python-openstackclient performing neutron part action

[Zhao Jian]

Today when using python-openstackclient doing some test on neutron function, I got this error

##########
REQ: curl -g -i --cacert "/opt/stack/ssl/openstack.crt" -X GET https://openstack.example.org:9696 -H "User-Agent: openstacksdk/0.9.2 keystoneauth1/2.11.1 python-requests/2.11.1 CPython/2.7.6" -H "X-Auth-Token: {SHA1}c7d960e926620fa81d23b2ddaba291be2e744507"
Starting new HTTPS connection (1): openstack.example.org
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for openstack.example.org has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
"GET / HTTP/1.1" 200 129
RESP: [200] Content-Type: application/json; charset=UTF-8 Content-Length: 129 Date: Wed, 24 Aug 2016 14:23:15 GMT Connection: close
RESP BODY: {"versions": [{"status": "CURRENT", "id": "v2.0", "links": [{"href": "http://openstack.example.org:9696/v2.0", "rel": "self"}]}]}

REQ: curl -g -i --cacert "/opt/stack/ssl/openstack.crt" -X PUT http://openstack.example.org:9696/v2.0/quotas/023fa1b33ac14a6ab41476b62a50329b -H "User-Agent: openstacksdk/0.9.2 keystoneauth1/2.11.1 python-requests/2.11.1 CPython/2.7.6" -H "Content-Type: application/json" -H "X-Auth-Token: {SHA1}c7d960e926620fa81d23b2ddaba291be2e744507" -d '{"quota": {"floatingip": 30}}'
Starting new HTTP connection (1): openstack.example.org
Connection failure that may be retried.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/command.py", line 55, in run
    self.take_action(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/quota.py", line 177, in take_action
    **network_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/network/v2/_proxy.py", line 1217, in update_quota
    return self._update(_quota.Quota, quota, **attrs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/proxy.py", line 36, in check
    return method(self, expected, actual, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/proxy.py", line 160, in _update
    return res.update(self.session)
  File "/usr/local/lib/python2.7/dist-packages/openstack/resource.py", line 795, in update
    resp = self.update_by_id(session, self.id, dirty_attrs, path_args=self)
  File "/usr/local/lib/python2.7/dist-packages/openstack/resource.py", line 770, in update_by_id
    resp = session.put(url, endpoint_filter=cls.service, **args)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 683, in put
    return self.request(url, 'PUT', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 54, in map_exceptions_wrapper
    raise exceptions.SDKException(message=e.message, cause=e)
SDKException: Connection failure that may be retried.
clean_up SetQuota: Connection failure that may be retried.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/shell.py", line 133, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 226, in run
    result = self.run_subcommand(remainder)
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/shell.py", line 178, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/command.py", line 55, in run
    self.take_action(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/quota.py", line 177, in take_action
    **network_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/network/v2/_proxy.py", line 1217, in update_quota
    return self._update(_quota.Quota, quota, **attrs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/proxy.py", line 36, in check
    return method(self, expected, actual, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/proxy.py", line 160, in _update
    return res.update(self.session)
  File "/usr/local/lib/python2.7/dist-packages/openstack/resource.py", line 795, in update
    resp = self.update_by_id(session, self.id, dirty_attrs, path_args=self)
  File "/usr/local/lib/python2.7/dist-packages/openstack/resource.py", line 770, in update_by_id
    resp = session.put(url, endpoint_filter=cls.service, **args)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 683, in put
    return self.request(url, 'PUT', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 54, in map_exceptions_wrapper
    raise exceptions.SDKException(message=e.message, cause=e)
SDKException: Connection failure that may be retried.
##########

My cloud service endpoints are using https scheme, my python-openstackclient is 3.0.1 and my openstacksdk is in 0.9.2.

After doing some investigation, I do think this issue does exist in openstacksdk 0.9.2 and higher, you could reproduce this issue when you configure your could services endpoints in https, especially for neutron service, and when using openstack command to preform neutron service action, you will meet this issue.

This issue is related to this change https://github.com/openstack/python-openstacksdk/commit/d5149c9df057758aed9cfb66984856274305a666

I think mostly this change is great, but just one concern for one place about finding the suitable endpoint for service.

Since Keystone service is used to store other services endpoints, I think we could use base endpoint url and combine with version that we got using rest api provided by service.

This issue happen just because when you are getting version from Neutron service, maybe links are not expected what you want, just like this:

##########
root@allinone:/usr/local/lib/python2.7/dist-packages/openstack# curl -g -i --cacert "/opt/stack/ssl/openstack.crt" -X GET https://openstack.example.org:9696
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 129
Date: Wed, 24 Aug 2016 15:53:53 GMT
Connection: close

{"versions": [{"status": "CURRENT", "id": "v2.0", "links": [{"href": "http://openstack.example.org:9696/v2.0", "rel": "self"}]}]}
##########

This issue seems making a blocker for users who is using openstack client to operate their cloud and whose openstacksdk is 0.9.2 or 0.9.3, and they could not do much thing for neutron part.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/359992

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/360343

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zhao Jian (<email address hidden>) on branch: master
Review: https://review.openstack.org/359992
Reason: Sorry Brain, I did some mistake and my local repo, and my codes are not good. So I have to abandon this to create a new one.

[Zhao Jian (jianzj)]

Hi Brain,

Thanks for your suggestion, and I did some code change, hopes could fix this issue.

By the way, I did some investigation, and found, for each component, the implement of "/versions" are different.

One weird component is Neutron, I saw in openstackclient, I post a GET request, with https url, but in versions data, you could see href is http scheme.

For example :

##########
root@allinone:/usr/local/lib/python2.7/dist-packages/openstack# curl -g -i --cacert "/opt/stack/ssl/openstack.crt" -X GET https://openstack.example.org:9696
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 129
Date: Thu, 25 Aug 2016 09:04:13 GMT
Connection: close

{"versions": [{"status": "CURRENT", "id": "v2.0", "links": [{"href": "http://openstack.example.org:9696/v2.0", "rel": "self"}]}]}root@allinone:/usr/local/lib/
##########

For Neutron versions code, could be found here : https://github.com/openstack/neutron/blob/master/neutron/api/views/versions.py

So this makes me think that using href link to be the endpoint is not suitable for the moment, and since Keystone is using managing other component endpoints, and seems endpoints registering into Keystone is good to use.

But version problem is just as you mentioned, seems a problem, so I just combine the root_endpoint with version data together.

I am sure this is not the best way to solve this problem, but hope this will could help .

Any suggestions , feel free to contact me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/360343
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=b003c3d1cf02e132083ac26dab42520716f39610
Submitter: Jenkins
Branch: master

commit b003c3d1cf02e132083ac26dab42520716f39610
Author: Jian Zhao <email address hidden>
Date: Thu Aug 25 04:45:08 2016 -0400

    Fix issue "SDKException: Connection failure that may be retried."

    This issue will happen when your cloud service endpoints are in
      https and you are using openstack client to perform some
    actions related to neutron service.

    This issue happened in openstacksdk-0.9.2 and higher.

    This issue is because something not expected when finding
      a suitable version of Neutron service endpoint. You will
    find you will get an endpoint which is http instead of https.

    Change-Id: Iad5187a03b939fcd237d79e9bd868421d360cc63
    Closes-Bug: #1616522