We seem to be hitting this in nova again with a notifications functional test:

http://logs.openstack.org/06/641806/16/check/nova-tox-functional-py36/6ffca82/job-output.txt.gz#_2019-05-02_03_23_47_450578

2019-05-02 03:23:47.450578 | ubuntu-bionic | {0} nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSample.test_instance_action [] ... inprogress

That test runs a bunch of actions on a server which generates a lot of logs, plus tracebacks during volume attach:

2019-05-02 03:23:47.482528 | ubuntu-bionic |     b'2019-05-02 03:23:44,276 INFO [nova.api.openstack.requestlog] 127.0.0.1 "POST /v2.1/6f70656e737461636b20342065766572/servers/e61aff9b-8ce8-4b02-a726-e90c9c5eaa9c/os-volume_attachments" status: 200 len: 207 microversion: 2.72 time: 0.078540'
2019-05-02 03:23:47.482635 | ubuntu-bionic |     b'2019-05-02 03:23:44,314 INFO [nova.compute.manager] Attaching volume 227cc671-f30b-4488-96fd-7d0bf13648d8 to /dev/sdc'
2019-05-02 03:23:47.482747 | ubuntu-bionic |     b'2019-05-02 03:23:44,355 ERROR [nova.compute.manager] Failed to attach 227cc671-f30b-4488-96fd-7d0bf13648d8 at /dev/sdc'
2019-05-02 03:23:47.482796 | ubuntu-bionic |     b'Traceback (most recent call last):'
2019-05-02 03:23:47.482889 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 5628, in _attach_volume'
2019-05-02 03:23:47.482932 | ubuntu-bionic |     b'    do_driver_attach=True)'
2019-05-02 03:23:47.483021 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 46, in wrapped'
2019-05-02 03:23:47.483081 | ubuntu-bionic |     b'    ret_val = method(obj, context, *args, **kwargs)'
2019-05-02 03:23:47.483169 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 651, in attach'
2019-05-02 03:23:47.483217 | ubuntu-bionic |     b'    virt_driver, do_driver_attach)'
2019-05-02 03:23:47.483308 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 629, in _do_attach'
2019-05-02 03:23:47.483347 | ubuntu-bionic |     b'    do_driver_attach)'
2019-05-02 03:23:47.483451 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 542, in _volume_attach'
2019-05-02 03:23:47.483508 | ubuntu-bionic |     b"    self['mount_device'])['connection_info']"
2019-05-02 03:23:47.483625 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/mock/mock.py", line 1062, in __call__'
2019-05-02 03:23:47.483684 | ubuntu-bionic |     b'    return _mock_self._mock_call(*args, **kwargs)'
2019-05-02 03:23:47.483800 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/mock/mock.py", line 1128, in _mock_call'
2019-05-02 03:23:47.483850 | ubuntu-bionic |     b'    ret_val = effect(*args, **kwargs)'
2019-05-02 03:23:47.483970 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_instance.py", line 1773, in attach_volume'
2019-05-02 03:23:47.484031 | ubuntu-bionic |     b'    reason="Connection timed out")'
2019-05-02 03:23:47.484120 | ubuntu-bionic |     b'nova.exception.CinderConnectionFailed: Connection to cinder host failed: Connection timed out'
2019-05-02 03:23:47.484271 | ubuntu-bionic |     b'2019-05-02 03:23:44,356 INFO [nova.tests.fixtures] Deleted attachment 65b5ae04-66d6-4fce-98a9-9d68d3dc94db for volume 227cc671-f30b-4488-96fd-7d0bf13648d8. Total attachments for volume: 0'
2019-05-02 03:23:47.484356 | ubuntu-bionic |     b'2019-05-02 03:23:44,434 ERROR [oslo_messaging.rpc.server] Exception during message handling'
2019-05-02 03:23:47.484426 | ubuntu-bionic |     b'Traceback (most recent call last):'
2019-05-02 03:23:47.484564 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming'
2019-05-02 03:23:47.484619 | ubuntu-bionic |     b'    res = self.dispatcher.dispatch(message)'
2019-05-02 03:23:47.484748 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch'
2019-05-02 03:23:47.484835 | ubuntu-bionic |     b'    return self._do_dispatch(endpoint, method, ctxt, args)'
2019-05-02 03:23:47.484969 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch'
2019-05-02 03:23:47.485025 | ubuntu-bionic |     b'    result = func(ctxt, **new_args)'
2019-05-02 03:23:47.485116 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/exception_wrapper.py", line 79, in wrapped'
2019-05-02 03:23:47.485175 | ubuntu-bionic |     b'    function_name, call_dict, binary, tb)'
2019-05-02 03:23:47.485296 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__'
2019-05-02 03:23:47.485338 | ubuntu-bionic |     b'    self.force_reraise()'
2019-05-02 03:23:47.485474 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise'
2019-05-02 03:23:47.485536 | ubuntu-bionic |     b'    six.reraise(self.type_, self.value, self.tb)'
2019-05-02 03:23:47.485653 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/six.py", line 693, in reraise'
2019-05-02 03:23:47.485690 | ubuntu-bionic |     b'    raise value'
2019-05-02 03:23:47.485779 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/exception_wrapper.py", line 69, in wrapped'
2019-05-02 03:23:47.485830 | ubuntu-bionic |     b'    return f(self, context, *args, **kw)'
2019-05-02 03:23:47.485925 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/utils.py", line 1323, in decorated_function'
2019-05-02 03:23:47.485983 | ubuntu-bionic |     b'    return function(self, context, *args, **kwargs)'
2019-05-02 03:23:47.486079 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 215, in decorated_function'
2019-05-02 03:23:47.486132 | ubuntu-bionic |     b"    kwargs['instance'], e, sys.exc_info())"
2019-05-02 03:23:47.486252 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__'
2019-05-02 03:23:47.486293 | ubuntu-bionic |     b'    self.force_reraise()'
2019-05-02 03:23:47.486427 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise'
2019-05-02 03:23:47.486488 | ubuntu-bionic |     b'    six.reraise(self.type_, self.value, self.tb)'
2019-05-02 03:23:47.486599 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/six.py", line 693, in reraise'
2019-05-02 03:23:47.486635 | ubuntu-bionic |     b'    raise value'
2019-05-02 03:23:47.486743 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 203, in decorated_function'
2019-05-02 03:23:47.486805 | ubuntu-bionic |     b'    return function(self, context, *args, **kwargs)'
2019-05-02 03:23:47.486898 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 5613, in attach_volume'
2019-05-02 03:23:47.486957 | ubuntu-bionic |     b'    do_attach_volume(context, instance, driver_bdm)'
2019-05-02 03:23:47.487082 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner'
2019-05-02 03:23:47.487127 | ubuntu-bionic |     b'    return f(*args, **kwargs)'
2019-05-02 03:23:47.487223 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 5611, in do_attach_volume'
2019-05-02 03:23:47.487260 | ubuntu-bionic |     b'    bdm.destroy()'
2019-05-02 03:23:47.487391 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__'
2019-05-02 03:23:47.487436 | ubuntu-bionic |     b'    self.force_reraise()'
2019-05-02 03:23:47.487563 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise'
2019-05-02 03:23:47.487621 | ubuntu-bionic |     b'    six.reraise(self.type_, self.value, self.tb)'
2019-05-02 03:23:47.487731 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/six.py", line 693, in reraise'
2019-05-02 03:23:47.487767 | ubuntu-bionic |     b'    raise value'
2019-05-02 03:23:47.487873 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 5608, in do_attach_volume'
2019-05-02 03:23:47.487941 | ubuntu-bionic |     b'    return self._attach_volume(context, instance, driver_bdm)'
2019-05-02 03:23:47.488036 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 5655, in _attach_volume'
2019-05-02 03:23:47.488084 | ubuntu-bionic |     b'    volume_id=bdm.volume_id, tb=tb)'
2019-05-02 03:23:47.488206 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__'
2019-05-02 03:23:47.488247 | ubuntu-bionic |     b'    self.force_reraise()'
2019-05-02 03:23:47.488373 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise'
2019-05-02 03:23:47.488444 | ubuntu-bionic |     b'    six.reraise(self.type_, self.value, self.tb)'
2019-05-02 03:23:47.488555 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/six.py", line 693, in reraise'
2019-05-02 03:23:47.488595 | ubuntu-bionic |     b'    raise value'
2019-05-02 03:23:47.488689 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 5628, in _attach_volume'
2019-05-02 03:23:47.488732 | ubuntu-bionic |     b'    do_driver_attach=True)'
2019-05-02 03:23:47.488838 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 46, in wrapped'
2019-05-02 03:23:47.488909 | ubuntu-bionic |     b'    ret_val = method(obj, context, *args, **kwargs)'
2019-05-02 03:23:47.489000 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 651, in attach'
2019-05-02 03:23:47.489048 | ubuntu-bionic |     b'    virt_driver, do_driver_attach)'
2019-05-02 03:23:47.489140 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 629, in _do_attach'
2019-05-02 03:23:47.489179 | ubuntu-bionic |     b'    do_driver_attach)'
2019-05-02 03:23:47.489274 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/block_device.py", line 542, in _volume_attach'
2019-05-02 03:23:47.489328 | ubuntu-bionic |     b"    self['mount_device'])['connection_info']"
2019-05-02 03:23:47.489452 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/mock/mock.py", line 1062, in __call__'
2019-05-02 03:23:47.489514 | ubuntu-bionic |     b'    return _mock_self._mock_call(*args, **kwargs)'
2019-05-02 03:23:47.489633 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/mock/mock.py", line 1128, in _mock_call'
2019-05-02 03:23:47.489684 | ubuntu-bionic |     b'    ret_val = effect(*args, **kwargs)'
2019-05-02 03:23:47.489804 | ubuntu-bionic |     b'  File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_instance.py", line 1773, in attach_volume'
2019-05-02 03:23:47.489853 | ubuntu-bionic |     b'    reason="Connection timed out")'
2019-05-02 03:23:47.489940 | ubuntu-bionic |     b'nova.exception.CinderConnectionFailed: Connection to cinder host failed: Connection timed out'

It looks like the cinder fixture is failing to stub something out during volume attach.