test_volume_swap_server_with_error produce a suspicious stack trace

[Balazs Gibizer]

Sometimes test_volume_swap_server_with_error functional test fails [1]. It feels like a timing issue in the test itself. But regardless if it fails or passes there is a suspicious stack trace in the log showing that something is wrong with the swap volume codepath:

2019-02-18 22:40:07,103 ERROR [nova.compute.manager] Failed to connect to volume 9c6d9c2d-7a8f-4c80-938d-3bf062b8d489 with volume at /dev/sdb
Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5701, in _swap_volume
    bdm, new_attachment_id, mountpoint)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5688, in _init_volume_connection
    new_cinfo['serial'] = new_volume_id
TypeError: 'tuple' object does not support item assignment
2019-02-18 22:40:07,521 ERROR [oslo_messaging.rpc.server] Exception during message handling
Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/exception_wrapper.py", line 79, in wrapped
    function_name, call_dict, binary, tb)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/exception_wrapper.py", line 69, in wrapped
    return f(self, context, *args, **kw)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/utils.py", line 1301, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 213, in decorated_function
    kwargs['instance'], e, sys.exc_info())
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 201, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5859, in swap_volume
    is_cinder_migration)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5757, in _swap_volume
    context, new_attachment_id)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5701, in _swap_volume
    bdm, new_attachment_id, mountpoint)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5688, in _init_volume_connection
    new_cinfo['serial'] = new_volume_id
TypeError: 'tuple' object does not support item assignment

[1] http://logs.openstack.org/78/637578/1/check/nova-tox-functional-py35/14da3dc/testr_results.html.gz

[Lee Yarwood (lyarwood)]

AFAICT this isn't an issue with swap_volume itself. The fixture is causing the TypeError on purpose here to exercise the notification code path when dealing with errors during a swap volume call:

https://github.com/openstack/nova/blob/d2a87a44468f91d5a468e1262e7e2fd8c7add35c/nova/tests/functional/notification_sample_tests/test_instance.py#L1564-L1565

https://github.com/openstack/nova/blob/d2a87a44468f91d5a468e1262e7e2fd8c7add35c/nova/tests/fixtures.py#L1597-L1601

You missed the trace showing the issue with the notification path itself from your initial comment:

Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_instance.py", line 1609, in test_volume_swap_server_with_error
    tb2 = fake_notifier.VERSIONED_NOTIFICATIONS[7]['payload'][
IndexError: list index out of range

[Balazs Gibizer (balazs-gibizer)]

Thanks Lee to figure out that is an intentional error. It would be nicer to raise something like a TestException instead.

I did notice the missing notification, hence the sentence from the bug report: "It feels like a timing issue in the test itself."

Anyhow that timing issue is unrelated to this bug report and this bug is not a bug. So I'm marking it invalid.