httplib.BadStatusLine errors in calls to Manila Quobyte Backend

[Silvan Kaiser]

RPC calls to the Quobyte backend made from the Quobyte Manila driver sometimes return with httplib.BadStatusLine (error value ''), for example:

2015-07-01 12:33:46.1 DEBUG manila.share.drivers.quobyte.jsonrpc [req-ac57bd40-1f92-4ce8-9149-d0a687cc14b9 33dba3a037354c89a5158f2656c87dd0 6445aaa68ae54d6faf4608780dee8bfb] Posting to Quobyte backend: {"params"
: {"retry": "INFINITELY", "tenant_domain": "6445aaa68ae54d6faf4608780dee8bfb", "volume_name": "share-abf977c2-f470-4168-8dcd-dbf22a679486"}, "jsonrpc": "2.0", "method": "resolveVolumeName", "id": "2"} from (pid=
25689) call /opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py:127
2015-07-01 12:33:46.2 ERROR manila.share.manager [req-ac57bd40-1f92-4ce8-9149-d0a687cc14b9 33dba3a037354c89a5158f2656c87dd0 6445aaa68ae54d6faf4608780dee8bfb] Share abf977c2-f470-4168-8dcd-dbf22a679486 failed on
creation.
2015-07-01 12:33:46.3 WARNING manila.share.manager [req-ac57bd40-1f92-4ce8-9149-d0a687cc14b9 33dba3a037354c89a5158f2656c87dd0 6445aaa68ae54d6faf4608780dee8bfb] Share information in exception can not be written t
o db because it contains {} and it is not a dictionary.
2015-07-01 12:33:46.100 ERROR oslo_messaging.rpc.dispatcher [req-ac57bd40-1f92-4ce8-9149-d0a687cc14b9 33dba3a037354c89a5158f2656c87dd0 6445aaa68ae54d6faf4608780dee8bfb] Exception during message handling: If SSL
is enabled for the API service, the URL must start with 'https://' for the URL. Failed to parse status code from server response. Error was ''
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 356, in create_share
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher context, share_id, {'status': constants.STATUS_ERROR})
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 329, in create_share
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher context, share_ref, share_server=share_server)
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/quobyte.py", line 134, in create_share
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher self._get_project_name(context, share['project_id']))
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/quobyte.py", line 121, in _resolve_volume_name
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher tenant_domain=tenant_domain))
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py", line 159, in call
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher % e)
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher QBException: If SSL is enabled for the API service, the URL must start with 'https://' for the URL. Failed to parse status code from server response. E
rror was ''
2015-07-01 12:33:46.100 TRACE oslo_messaging.rpc.dispatcher

The occurrence of this seems to be environment related (if it occurs at all i can reproduce it all the time in my setup) and seems at the base to be a httplib timing issue. Some related discussion can be found at http://stackoverflow.com/questions/1767934/why-am-i-getting-this-error-in-python-httplib . The comments on that page also pointed me to a possible solution that i successfully tested in my setup. I'll create a change for this in the driver.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/202991

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/202991
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=dabe8c7a24a8cf81796d2b0f38886dae99b75775
Submitter: Jenkins
Branch: master

commit dabe8c7a24a8cf81796d2b0f38886dae99b75775
Author: Silvan Kaiser <email address hidden>
Date: Fri Jul 17 13:49:48 2015 +0200

    Explicit backend connect call in Quobyte RPCs

    Explicitely connect to backend prior to sending an rpc.
    This prevents httplib.BadStatusLine Exceptions that may
    occur due to keep alive timeouts for the connection
    in httplib.
    The test additions include one generally improved unit
    test.

    Closes-Bug: 1470516

    Change-Id: Ia021bfb1cb3795fcded45ebae7e940fb006c58e7