OpenStack Shared File Systems Service (Manila)

CannotSendRequest in Quobyte driver

Bug #1560345 reported by Silvan Kaiser on 2016-03-22

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	Low	Silvan Kaiser

Bug Description

Random occurrences of CannotSendRequest happen during some Quobyte CI runs.

Example log (from http://78.46.57.153:8081/refs-changes-04-288504-14/logs/screen-m-shr.log.txt):
2016-03-21 20:53:52.029 22801 DEBUG manila.share.drivers.quobyte.jsonrpc [req-dc135702-d762-4d44-8781-df3ce60e9acb bdabaa03fa354add96d9ba5aea5752a2 f418251524be4edc812d49bcd0ce7a5f - - -] Posting to Quobyte backend: {"params": {"retry": "INFINITELY", "remove_allow_ip": "1.2.3.4", "volume_uuid": "3bda6bfd-2a40-480d-be56-ca9e61d3d248"}, "jsonrpc": "2.0", "method": "exportVolume", "id": "132"} call /opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py:127
2016-03-21 20:53:52.669 22801 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:201
2016-03-21 20:53:52.670 22801 WARNING manila.context [-] Arguments dropped when creating context: {u'read_only': False, u'domain': None, u'show_deleted': False, u'user_identity': u'bdabaa03fa354add96d9ba5aea5752a2 f418251524be4edc812d49bcd0ce7a5f - - -', u'project_domain': None, u'resource_uuid': None, u'user_domain': None}.
2016-03-21 20:53:52.848 22801 DEBUG manila.share.drivers.quobyte.jsonrpc [req-8ecf0cd8-313a-49e6-a343-8ed298002dc8 bdabaa03fa354add96d9ba5aea5752a2 f418251524be4edc812d49bcd0ce7a5f - - -] Posting to Quobyte backend: {"params": {"retry": "INFINITELY", "tenant_domain": "f418251524be4edc812d49bcd0ce7a5f", "volume_name": "share-805cdfeb-eae4-4f9e-8c00-7ce9feba7c16"}, "jsonrpc": "2.0", "method": "resolveVolumeName", "id": "133"} call /opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py:127
2016-03-21 20:53:52.849 22801 ERROR manila.share.manager [req-8ecf0cd8-313a-49e6-a343-8ed298002dc8 bdabaa03fa354add96d9ba5aea5752a2 f418251524be4edc812d49bcd0ce7a5f - - -] Share instance 805cdfeb-eae4-4f9e-8c00-7ce9feba7c16 failed on creation.
2016-03-21 20:53:52.849 22801 WARNING manila.share.manager [req-8ecf0cd8-313a-49e6-a343-8ed298002dc8 bdabaa03fa354add96d9ba5aea5752a2 f418251524be4edc812d49bcd0ce7a5f - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher [req-8ecf0cd8-313a-49e6-a343-8ed298002dc8 bdabaa03fa354add96d9ba5aea5752a2 f418251524be4edc812d49bcd0ce7a5f - - -] Exception during message handling:
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 146, in wrapped
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher return f(self, *args, **kwargs)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/utils.py", line 616, in wrapper
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher return func(self, *args, **kwargs)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 1014, in create_share_instance
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 985, in create_share_instance
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher context, share_instance, share_server=share_server)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/quobyte.py", line 210, in create_share
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher self._get_project_name(context, share['project_id']))
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/quobyte.py", line 180, in _resolve_volume_name
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher tenant_domain=tenant_domain))
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py", line 178, in call
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher ctxt.reraise = False
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py", line 130, in call
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher dict(Authorization=(self._credentials.
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/httplib.py", line 979, in request
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher self._send_request(method, url, body, headers)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher self.putrequest(method, url, **skips)
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/httplib.py", line 877, in putrequest
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher raise CannotSendRequest()
2016-03-21 20:53:52.885 22801 ERROR oslo_messaging.rpc.dispatcher CannotSendRequest

Silvan Kaiser (2-silvan) on 2016-03-22

Changed in manila:
assignee:	nobody → Silvan Kaiser (2-silvan)

Valeriy Ponomaryov (vponomaryov) on 2016-06-08

Changed in manila:
importance:	Undecided → Low

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-07-21: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/345219

Changed in manila:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-08-01: Fix merged to manila (master)

Reviewed: https://review.openstack.org/345219
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=b75ab85582abcb22152dc53db631dc40f1dc5ac5
Submitter: Jenkins
Branch: master

commit b75ab85582abcb22152dc53db631dc40f1dc5ac5
Author: Silvan Kaiser <email address hidden>
Date: Thu Jul 21 09:49:11 2016 +0200

Replaces httplib with requests lib in Quobyte RPC layer

    Simplifies the Manila Quobyte drivers rpc layer by using
    requests library instead of httplib. This reduces overall
    complexity and fixes randomly occurring rpc send issues.

Closes-Bug: #1560345

Change-Id: I4cc602722ece3761331bb3fa1c6bce2bbd35c78f

Changed in manila:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-01: Fix included in openstack/manila 3.0.0.0b3

This issue was fixed in the openstack/manila 3.0.0.0b3 development milestone.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.