TestServiceUpdateNotificationSample fails intermittently: u'host2' != u'host1': path: root.payload.nova_object.data.host

[Balazs Gibizer]

Tests in both nova.tests.functional.notification_sample_tests.test_service.TestServiceUpdateNotificationSampleLatest and nova.tests.functional.notification_sample_tests.test_service.TestServiceUpdateNotificationSamplev2_52 fails with a similar error:

Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_service.py", line 121, in test_service_disabled
    'uuid': self.service_uuid})
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_service.py", line 37, in _verify_notification
    base._verify_notification(sample_file_name, replacements, actual)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/tests/functional/notification_sample_tests/notification_sample_base.py", line 150, in _verify_notification
    self.assertJsonEqual(sample_obj, notification)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/test.py", line 533, in assertJsonEqual
    raise error
AssertionError:
expected:
{'event_type': 'service.update',
 'payload': {'nova_object.data': {'availability_zone': None,
                                  'binary': 'nova-compute',
                                  'disabled': True,
                                  'disabled_reason': None,
                                  'forced_down': False,
                                  'host': 'host2',
                                  'last_seen_up': '2012-10-29T13:42:05Z',
                                  'report_count': 1,
                                  'topic': 'compute',
                                  'uuid': 'e81d66a4-ddd3-4aba-8a84-171d1cb4d339',
                                  'version': 26},
             'nova_object.name': 'ServiceStatusPayload',
             'nova_object.namespace': 'nova',
             'nova_object.version': '1.1'},
 'priority': 'INFO',
 'publisher_id': 'nova-compute:host1'}
actual:
{'event_type': 'service.update',
 'payload': {'nova_object.data': {'availability_zone': None,
                                  'binary': 'nova-compute',
                                  'disabled': True,
                                  'disabled_reason': None,
                                  'forced_down': False,
                                  'host': 'host1',
                                  'last_seen_up': '2012-10-29T13:42:05Z',
                                  'report_count': 1,
                                  'topic': 'compute',
                                  'uuid': 'e81d66a4-ddd3-4aba-8a84-171d1cb4d339',
                                  'version': 26},
             'nova_object.name': 'ServiceStatusPayload',
             'nova_object.namespace': 'nova',
             'nova_object.version': '1.1'},
 'priority': 'INFO',
 'publisher_id': 'nova-compute:host1'}
difference:
'host2' != 'host1': path: root.payload.nova_object.data.host

It happens quite frequently on the gate, 64 occurrences in half a day:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22u'host2'%20!%3D%20u'host1'%3A%20path%3A%20root.payload.nova_object.data.host%5C%22

[Balazs Gibizer (balazs-gibizer)]

It is consistently broken on master since today. The offending commit is https://review.openstack.org/#/c/531381/ . The functional test does not run on that patch as it only contains modification under nova/docs. But the notification samples are used in the functional test so a change in the samples can cause a functional test to fail.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/533190

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533190
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3881f9e1efe9183232c6273732a344ff2f9e2021
Submitter: Zuul
Branch: master

commit 3881f9e1efe9183232c6273732a344ff2f9e2021
Author: Balazs Gibizer <email address hidden>
Date: Fri Jan 12 15:11:35 2018 +0100

    Revert "Deduplicate service status notification samples"

    This reverts commit 39f9ef719eacb2fbcf0c37680b845152fcff01cc.

    The reverted change made the service.update notification sample tests
    failing consistently on master. The CI did not catch this problem as
    the functional test does not run for changes that only touches
    nova/docs. This is a quick fix to unblock the gate and fix the
    functional job defintion in a separate patch.

    Change-Id: Idabbf70ae46f36be8343c0a00d812cbebf7f7670
    Closes-Bug: #1742935

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.