TestInstanceNotificationSample.test_volume_swap_server fails with "testtools.matchers._impl.MismatchError: 7 != 6"

[Matt Riedemann]

http://logs.openstack.org/17/483917/1/check/gate-nova-tox-functional-ubuntu-xenial/f92a89d/console.html#_2017-07-14_13_14_22_687228

2017-07-14 13:14:22.687228 | nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSample.test_volume_swap_server
2017-07-14 13:14:22.687273 | --------------------------------------------------------------------------------------------------------------------
2017-07-14 13:14:22.687286 |
2017-07-14 13:14:22.687304 | Captured traceback:
2017-07-14 13:14:22.687322 | ~~~~~~~~~~~~~~~~~~~
2017-07-14 13:14:22.687345 | Traceback (most recent call last):
2017-07-14 13:14:22.687388 | File "nova/tests/functional/notification_sample_tests/test_instance.py", line 837, in test_volume_swap_server
2017-07-14 13:14:22.687419 | self.assertEqual(7, len(fake_notifier.VERSIONED_NOTIFICATIONS))
2017-07-14 13:14:22.687479 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
2017-07-14 13:14:22.687507 | self.assertThat(observed, matcher, message)
2017-07-14 13:14:22.687565 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
2017-07-14 13:14:22.687586 | raise mismatch_error
2017-07-14 13:14:22.687611 | testtools.matchers._impl.MismatchError: 7 != 6

This could be due to the recent change in the CinderFixture here:

https://review.openstack.org/#/c/448779/

We need the dump the notifications in the error message to get context when this fails to debug it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/483939

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483939
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1ed77e95012432bca99e2e9d11606c7bad84b824
Submitter: Jenkins
Branch: master

commit 1ed77e95012432bca99e2e9d11606c7bad84b824
Author: Matt Riedemann <email address hidden>
Date: Fri Jul 14 09:42:26 2017 -0400

    Dump tracked version notifications when swap volume tests fail

    We're seeing random failures in the swap volume versioned
    notification functional tests, but need more information about
    what notifications have been tracked when we fail in order to
    debug this, so this change dumps the tracked notifications when
    the tests fail.

    Change-Id: I07592cc4e58dc7664aede16bf32ac1e0282d23eb
    Related-Bug: #1704392

[Matt Riedemann (mriedem)]

Here is an example failure with the payload:

http://logs.openstack.org/82/482382/4/check/gate-nova-tox-functional-ubuntu-xenial/38a4cb4/console.html#_2017-07-16_01_14_36_313757

http://paste.openstack.org/show/615507/

[Balazs Gibizer (balazs-gibizer)]

Based on the logs the volume_swap.end notification is missing. Currently the test waits for the new volume id to show up on volume attachment of the server but based on the compute manager code that happens _before_ the end notification is sent. I will propose a patch that wait for the notification instead of waiting for just the volume_id to appear.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/484288

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484288
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cf7f61c180c294f533e228697885a0ffb7e429bc
Submitter: Jenkins
Branch: master

commit cf7f61c180c294f533e228697885a0ffb7e429bc
Author: Balazs Gibizer <email address hidden>
Date: Mon Jul 17 10:32:00 2017 +0200

    fix test_volume_swap_server instability

    The test_volume_swap_server notification sample test was unstable
    becase sometimes volume_swap.end notification was missing. The test was
    waiting for the new volume id to appear on the REST API before
    asserting the received notifications but the compute manager updates
    the BDM earlier and then emits the volume_swap.end.

    This patch modifies the test to explicitly wait for the
    volume_swap.end notification. This expected to remove the test
    instability.

    Change-Id: Id6eefa7c85c4f63562344b552f027f1d513a90e1
    Closes-Bug: #1704392

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b3 development milestone.