Insufficient error reporting in RPC exception in Quobyte driver

[Silvan Kaiser]

Backend API calls in the Quobyte Manila driver for some specific errors fail without usable error output, example from http://176.9.158.139:8081/refs-changes-73-245573-3/logs/screen-m-shr.log.txt :

2015-11-26 15:34:02.791 24735 DEBUG manila.share.drivers.quobyte.jsonrpc [req-db52a359-61fe-4ecd-805d-1f40c51454d8 - - - - -] Posting to Quobyte backend: {"params": {"retry": "INFINITELY", "tenant_domain": "b61f4b98551f481bad494d479bbbaa65", "volume_name": "share-a890f616-cfb9-4c03-8176-9240e8c3ad41"}, "jsonrpc": "2.0", "method": "resolveVolumeName", "id": "106"} call /opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py:128
2015-11-26 15:34:02.848 24735 DEBUG manila.share.drivers.quobyte.jsonrpc [req-9be9d23c-d404-45a8-a614-d475b50c865d - - - - -] Retrieved data from Quobyte backend: {u'jsonrpc': u'2.0', u'result': {u'nfs_export_path': u'/adb2e5f0-25fe-4a77-b66b-2c0b02721cd5', u'nfs_server_ip': u'127.0.0.1'}, u'id': u'105'} call /opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py:137
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher [req-db52a359-61fe-4ecd-805d-1f40c51454d8 - - - - -] Exception during message handling: Quobyte exception occurred:
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 110, in wrapped
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher return f(self, *args, **kwargs)
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/utils.py", line 613, in wrapper
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher return func(self, *args, **kwargs)
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 952, in delete_share_instance
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR_DELETING})
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 946, in delete_share_instance
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher share_server=share_server)
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/quobyte.py", line 178, in delete_share
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher self._get_project_name(context, share['project_id']))
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/quobyte.py", line 143, in _resolve_volume_name
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher tenant_domain=tenant_domain))
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/quobyte/jsonrpc.py", line 163, in call
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher raise exception.QBException(msg=six.text_type(e))
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher QBException: Quobyte exception occurred:
2015-11-26 15:34:02.976 24735 ERROR oslo_messaging.rpc.dispatcher
2015-11-26 15:34:03.035 24735 INFO manila.share.manager [req-9be9d23c-d404-45a8-a614-d475b50c865d - - - - -]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/273135

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/273135
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=c965f297d4d5b1603dfc9ba0a9150bd295c1c4dd
Submitter: Jenkins
Branch: master

commit c965f297d4d5b1603dfc9ba0a9150bd295c1c4dd
Author: Silvan Kaiser <email address hidden>
Date: Wed Jan 27 17:16:45 2016 +0100

    Ext. exception handling for httplib and socket errors in Quobyte driver

    This extends exception handling of socket.error and
    http_client.HTTPException issues to be more verbose. Current handling
    drops some exception information. This should help improve the
    currently bad reliability of the Quobyte Manila CI.

    Closes-Bug: 1520496

    Change-Id: Ifccf993c95817e2edffee98777e58b843db45b88

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/manila 2.0.0.0b3 development milestone.