Timeout triggers failures running tempest for ZFSSA iSCSI Cinder driver.

[Juan Zuluaga]

Running tempest in a long distance link between the cinder controller and ZFS Storage Appliance, ZFSSA iSCSI Cinder driver timeouts, resulting in tests failures.
e.g
...
{1} tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_from_image [355.316953s] ... FAILED
{1} tempest.api.volume.test_volumes_get.VolumesV2GetTestXML.test_volume_create_get_update_delete_as_clone [389.165116s] ... FAILED

...
Or
{1} tempest.api.volume.test_volumes_get.VolumesV1GetTest.test_volume_create_get_update_delete [507.216176s] ... FAILED

Some of these errors are the results of a volume not been reported as created, or an error uploading an image to a volume.
e.g (from cinder volume service)
2014-09-12 11:15:43.940 ERROR oslo.messaging.rpc.dispatcher [req-1879fa63-44c7-4f22-92fd-b139cce4bae7 85f8e9058af848aa9e10d2615e2e47dc 43
ab44ae2c7d4606ae30bc3b551cfe25] Exception during message handling: timed out
2014-09-12 11:15:44.643 ERROR cinder.volume.flows.manager.create_volume [req-da06124c-366b-48b7-95ca-96720967438c 85f8e9058af848aa9e10d26
15e2e47dc 43ab44ae2c7d4606ae30bc3b551cfe25] Volume c74b1a80-0528-41dd-b3d5-275410fa557a: create failed
2014-09-12 11:15:44.648 ERROR oslo.messaging.rpc.dispatcher [req-da06124c-366b-48b7-95ca-96720967438c 85f8e9058af848aa9e10d2615e2e47dc 43
ab44ae2c7d4606ae30bc3b551cfe25] Exception during message handling: timed out

[Juan Zuluaga (juan-c-zuluaga)]

Patch related: https://review.openstack.org/#/c/106075/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121900

[Juan Zuluaga (juan-c-zuluaga)]

Typical stack from the driver looks like this:
2014-09-18 11:28:12.561 ERROR oslo.messaging.rpc.dispatcher [req-f23bdb6c-48ab-450a-890e-0869ac13a3ec e1a88ac2b1714ad3b048b6b8327c5110 f9cb176a788b4110991ad79d9e7e4e80] Exception during message handling: timed out
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 127, in lvo_inner1
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher return lvo_inner2(inst, context, volume_id, **kwargs)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher retval = f(*args, **kwargs)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 126, in lvo_inner2
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher return f(*_args, **_kwargs)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 499, in delete_volume
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher self.publish_service_capabilities(context)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1232, in publish_service_capabilities
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher self._report_driver_status(context)
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1207, in _report_driver_status
2014-09-18 11:28:12.561 TRACE oslo.messaging.rpc.dispatcher volume_stats = self.driver.get_volume_stats(refresh=True)
2014-09-18 11:28:12.561 TRACE oslo.me...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121900
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b04a1dd063d9678747194e416ed157109a17fdc7
Submitter: Jenkins
Branch: master

commit b04a1dd063d9678747194e416ed157109a17fdc7
Author: Juan Zuluaga <email address hidden>
Date: Tue Sep 16 11:09:02 2014 -0400

    Timeout triggers failures running tempest for ZFSSA driver.

    Adding a property to setup the RESTAPI connection timeout with
    ZFS Storage Appliance, allow tempest to pass with slower network links.

    DocImpact: New config parameter is added to allow configuring timeout value.
    Closes-Bug: #1369136

    Change-Id: I8de19f56a18106324ed414cf41ee2323de639359