test_pre_live_migration_volume_backed* intermittently fails serialized json compare

Bug #1779711 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Critical
Matt Riedemann

Bug Description

Introduced in these new tests:

https://review.openstack.org/#/c/540679/

They are doing a primitive dict compare with nested dicts and the keys can be in random orders. We need to use something like self.assertJsonEqual.

Seen here:

http://logs.openstack.org/04/523604/21/check/openstack-tox-lower-constraints/8a273cd/testr_results.html.gz

ft1.600: nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_pre_live_migration_volume_backed_StringException: pythonlogging:'': {{{2018-06-29 18:43:57,584 WARNING [os_brick.initiator.connectors.remotefs] Connection details not present. RemoteFsClient may not initialize properly.}}}
stderr: {{{
/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py:323: DeprecationWarning: Using class 'MoxStubout' (either directly or via inheritance) is deprecated in version '3.5.0'
  mox_fixture = self.useFixture(moxstubout.MoxStubout())
}}}

Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/unit/virt/libvirt/test_driver.py", line 11972, in test_pre_live_migration_volume_backed
    self._test_pre_live_migration_volume_backed()
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/unit/virt/libvirt/test_driver.py", line 11969, in _test_pre_live_migration_volume_backed
    returned_migrate_data.obj_to_primitive())
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/lower-constraints/lib/python3.5/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/lower-constraints/lib/python3.5/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = {'nova_object.data': {'bdms': [{'nova_object.data': {'boot_index': None,
                                                     'bus': 'scsi',
                                                     'connection_info_json': '{"data": '
                                                                             '{"device_path": '
                                                                             '"/dev/disk/path/lun-X"}, '
                                                                             '"serial": '
                                                                             '"53641be9-7553-4b55-b2fd-cb3cf6373505"}',
                                                     'dev': 'sda',
                                                     'format': None,
                                                     'serial': '53641be9-7553-4b55-b2fd-cb3cf6373505',
                                                     'type': 'disk'},
                                'nova_object.name': 'LibvirtLiveMigrateBDMInfo',
                                'nova_object.namespace': 'nova',
                                'nova_object.version': '1.1'},
                               {'nova_object.data': {'boot_index': None,
                                                     'bus': 'scsi',
                                                     'connection_info_json': '{"data": '
                                                                             '{"device_path": '
                                                                             '"/dev/disk/path/lun-Z"}, '
                                                                             '"serial": '
                                                                             '"1f5cf6f3-6afb-45d2-a4a9-7a206f3a3cf8"}',
                                                     'dev': 'sdb',
                                                     'format': None,
                                                     'serial': '1f5cf6f3-6afb-45d2-a4a9-7a206f3a3cf8',
                                                     'type': 'disk'},
                                'nova_object.name': 'LibvirtLiveMigrateBDMInfo',
                                'nova_object.namespace': 'nova',
                                'nova_object.version': '1.1'}],
                      'block_migration': False,
                      'disk_available_mb': 123,
                      'disk_over_commit': False,
                      'filename': 'foo',
                      'image_type': 'qcow2',
                      'instance_relative_path': 'instance-00000001',
                      'is_shared_block_storage': False,
                      'is_shared_instance_path': True,
                      'is_volume_backed': True,
                      'serial_listen_ports': [],
                      'src_supports_native_luks': True,
                      'supported_perf_events': [],
                      'target_connect_addr': None},
 'nova_object.name': 'LibvirtLiveMigrateData',
 'nova_object.namespace': 'nova',
 'nova_object.version': '1.8'}
actual = {'nova_object.data': {'bdms': [{'nova_object.data': {'boot_index': None,
                                                     'bus': 'scsi',
                                                     'connection_info_json': '{"serial": '
                                                                             '"53641be9-7553-4b55-b2fd-cb3cf6373505", '
                                                                             '"data": '
                                                                             '{"device_path": '
                                                                             '"/dev/disk/path/lun-X"}}',
                                                     'dev': 'sda',
                                                     'format': None,
                                                     'serial': '53641be9-7553-4b55-b2fd-cb3cf6373505',
                                                     'type': 'disk'},
                                'nova_object.name': 'LibvirtLiveMigrateBDMInfo',
                                'nova_object.namespace': 'nova',
                                'nova_object.version': '1.1'},
                               {'nova_object.data': {'boot_index': None,
                                                     'bus': 'scsi',
                                                     'connection_info_json': '{"serial": '
                                                                             '"1f5cf6f3-6afb-45d2-a4a9-7a206f3a3cf8", '
                                                                             '"data": '
                                                                             '{"device_path": '
                                                                             '"/dev/disk/path/lun-Z"}}',
                                                     'dev': 'sdb',
                                                     'format': None,
                                                     'serial': '1f5cf6f3-6afb-45d2-a4a9-7a206f3a3cf8',
                                                     'type': 'disk'},
                                'nova_object.name': 'LibvirtLiveMigrateBDMInfo',
                                'nova_object.namespace': 'nova',
                                'nova_object.version': '1.1'}],
                      'block_migration': False,
                      'disk_available_mb': 123,
                      'disk_over_commit': False,
                      'filename': 'foo',
                      'image_type': 'qcow2',
                      'instance_relative_path': 'instance-00000001',
                      'is_shared_block_storage': False,
                      'is_shared_instance_path': True,
                      'is_volume_backed': True,
                      'serial_listen_ports': [],
                      'src_supports_native_luks': True,
                      'supported_perf_events': [],
                      'target_connect_addr': None},
 'nova_object.name': 'LibvirtLiveMigrateData',
 'nova_object.namespace': 'nova',
 'nova_object.version': '1.8'}

Tags: testing
Revision history for this message
Matt Riedemann (mriedem) wrote :
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/579615

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/579615
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ac9acce283a28326c6f9af4a158e6f3c1e09db45
Submitter: Zuul
Branch: master

commit ac9acce283a28326c6f9af4a158e6f3c1e09db45
Author: Matt Riedemann <email address hidden>
Date: Mon Jul 2 11:50:18 2018 -0400

    Handle nested serialized json entries in assertJsonEqual

    test_pre_live_migration_volume_backed* tests are comparing
    primitive versions of LibvirtLiveMigrateData objects which
    nest primitive LibvirtLiveMigrateBDMInfo which have
    serialized connection_info_json fields in them, which can
    have a random key order when comparing which makes the tests
    fail. This changes assertJsonEqual to try and deserialize
    nested json strings like it does for the top level expected
    and observed variables, and updates the test to use
    assertJsonEqual.

    Change-Id: Ief1634f7e31c473b226e9f19240ecd21840ebdb3
    Closes-Bug: #1779711

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/580702

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/580702
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2baa223c029c6efcd138768e739dec451693f0f3
Submitter: Zuul
Branch: master

commit 2baa223c029c6efcd138768e739dec451693f0f3
Author: Matt Riedemann <email address hidden>
Date: Fri Jul 6 11:24:23 2018 -0400

    Handle compare in test_pre_live_migration_volume_backed* directly

    We can't use assertJsonEqual here even if that function deserializes
    the nested connection_info_json. When deserializing the "bdms" entry
    in the migrate_data object, it creates a random order on the list
    which can still cause mismatch errors.

    So rather than rely on assertJsonEqual handling this, that change
    is reverted and we simply deserialize the connection_info_json
    in the test in question before comparing to our expected data.

    This reverts commit ac9acce283a28326c6f9af4a158e6f3c1e09db45.

    Change-Id: I7bba91e490a9ac5d0bbffa90e9cbefe60bb31f27
    Closes-Bug: #1779711

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b3

This issue was fixed in the openstack/nova 18.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.