The unshelve notification functional sample test fails intermittently

Bug #1835070 reported by Balazs Gibizer on 2019-07-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Balazs Gibizer
Pike
Low
Balazs Gibizer
Queens
Low
Balazs Gibizer
Rocky
Low
Balazs Gibizer
Stein
Low
Balazs Gibizer

Bug Description

The notification sample test for unshelve waits for the instance to reach ACTIVE state and then asserts if unshelve.end notification is emitted properly. However the instance.vm_state is set to ACTIVE earlier[1] than emitting the unshelve.end notification[2]. This can cause two different test case failures.

1) _test_unshelve_server() fails with no ushelve.end notification received.

2) _test_shelve_and_shelve_offload_server() also has a unshelve action at the end and that test step also only waits for the the ACTIVE state. So the unshelve.end notification from the end of _test_shelve_and_shelve_offload_server() can bleed into the _test_unshelve_server() step causing that it receive one more notifications.

[1] https://github.com/openstack/nova/blob/5c6c1f8fce7cd976dedc0a1ad28836ed87af2780/nova/compute/manager.py#L5322-L5326
[2] https://github.com/openstack/nova/blob/5c6c1f8fce7cd976dedc0a1ad28836ed87af2780/nova/compute/manager.py#L5329
[3] https://github.com/openstack/nova/blob/5c6c1f8fce7cd976dedc0a1ad28836ed87af2780/nova/tests/functional/notification_sample_tests/test_instance.py#L836

Balazs Gibizer (balazs-gibizer) wrote :

I got reports that both cases happening in a downstream env and I can reproduce it locally. But I did not found occurences of this fault in logstash

Changed in nova:
assignee: nobody → Balazs Gibizer (balazs-gibizer)
status: New → In Progress
Matt Riedemann (mriedem) on 2019-07-02
tags: added: testing
Changed in nova:
importance: Undecided → Low

Reviewed: https://review.opendev.org/668675
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=40f1e7c4c22dc6023614b34f28bb7fc416b668a8
Submitter: Zuul
Branch: master

commit 40f1e7c4c22dc6023614b34f28bb7fc416b668a8
Author: Balazs Gibizer <email address hidden>
Date: Tue Jul 2 14:56:40 2019 +0200

    Stabilize unshelve notification sample tests

    The notification sample test for unshelve waits for the instance to
    reach ACTIVE state and then asserts if unshelve.end notification is
    emitted properly. However the instance.vm_state is set to ACTIVE earlier[1]
    than emitting the unshelve.end notification[2]. This can cause two
    different test case failure.

    1) _test_unshelve_server() fails with no ushelve.end notification
    received.

    2) _test_shelve_and_shelve_offload_server() also has a unshelve action
    at the end and that test step also only waits for the the ACTIVE state.
    So the unshelve.end notification from the end of
    _test_shelve_and_shelve_offload_server() can bleed into the
    _test_unshelve_server() step causing that it receive one more
    notifications.

    So this patch adds an extra
    self._wait_for_notification('instance.unshelve.end') call to each test
    step to prevent the instability.

    [1] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5322-L5326
    [2] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5329
    [3] https://github.com/openstack/nova/blob/5c6c1f8f/nova/tests/functional/notification_sample_tests/test_instance.py#L836

    Closes-Bug: #1835070

    Change-Id: Ie217523a8969326b27930d7f74e50e9b352ab7a1

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/668806
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3bc6ff029ff24083844db363010988d8d08cff00
Submitter: Zuul
Branch: stable/stein

commit 3bc6ff029ff24083844db363010988d8d08cff00
Author: Balazs Gibizer <email address hidden>
Date: Tue Jul 2 14:56:40 2019 +0200

    Stabilize unshelve notification sample tests

    The notification sample test for unshelve waits for the instance to
    reach ACTIVE state and then asserts if unshelve.end notification is
    emitted properly. However the instance.vm_state is set to ACTIVE earlier[1]
    than emitting the unshelve.end notification[2]. This can cause two
    different test case failure.

    1) _test_unshelve_server() fails with no ushelve.end notification
    received.

    2) _test_shelve_and_shelve_offload_server() also has a unshelve action
    at the end and that test step also only waits for the the ACTIVE state.
    So the unshelve.end notification from the end of
    _test_shelve_and_shelve_offload_server() can bleed into the
    _test_unshelve_server() step causing that it receive one more
    notifications.

    So this patch adds an extra
    self._wait_for_notification('instance.unshelve.end') call to each test
    step to prevent the instability.

    [1] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5322-L5326
    [2] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5329
    [3] https://github.com/openstack/nova/blob/5c6c1f8f/nova/tests/functional/notification_sample_tests/test_instance.py#L836

    Closes-Bug: #1835070

    Change-Id: Ie217523a8969326b27930d7f74e50e9b352ab7a1
    (cherry picked from commit 40f1e7c4c22dc6023614b34f28bb7fc416b668a8)

Reviewed: https://review.opendev.org/669118
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a85ce04fa39e60e672e4fa2d7912f6880079c6ef
Submitter: Zuul
Branch: stable/rocky

commit a85ce04fa39e60e672e4fa2d7912f6880079c6ef
Author: Balazs Gibizer <email address hidden>
Date: Tue Jul 2 14:56:40 2019 +0200

    Stabilize unshelve notification sample tests

    The notification sample test for unshelve waits for the instance to
    reach ACTIVE state and then asserts if unshelve.end notification is
    emitted properly. However the instance.vm_state is set to ACTIVE earlier[1]
    than emitting the unshelve.end notification[2]. This can cause two
    different test case failure.

    1) _test_unshelve_server() fails with no ushelve.end notification
    received.

    2) _test_shelve_and_shelve_offload_server() also has a unshelve action
    at the end and that test step also only waits for the the ACTIVE state.
    So the unshelve.end notification from the end of
    _test_shelve_and_shelve_offload_server() can bleed into the
    _test_unshelve_server() step causing that it receive one more
    notifications.

    So this patch adds an extra
    self._wait_for_notification('instance.unshelve.end') call to each test
    step to prevent the instability.

    [1] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5322-L5326
    [2] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5329
    [3] https://github.com/openstack/nova/blob/5c6c1f8f/nova/tests/functional/notification_sample_tests/test_instance.py#L836

    Conflicts:
          nova/tests/functional/notification_sample_tests/test_instance.py
    Conflicts due to:
    * I019e88fabd1d386c0d6395a7b1969315873485fd

    Closes-Bug: #1835070

    Change-Id: Ie217523a8969326b27930d7f74e50e9b352ab7a1
    (cherry picked from commit 40f1e7c4c22dc6023614b34f28bb7fc416b668a8)
    (cherry picked from commit 3bc6ff029ff24083844db363010988d8d08cff00)

Reviewed: https://review.opendev.org/674636
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8afc39a2c748a8fde070493dd5633788f782c69a
Submitter: Zuul
Branch: stable/queens

commit 8afc39a2c748a8fde070493dd5633788f782c69a
Author: Balazs Gibizer <email address hidden>
Date: Tue Jul 2 14:56:40 2019 +0200

    Stabilize unshelve notification sample tests

    The notification sample test for unshelve waits for the instance to
    reach ACTIVE state and then asserts if unshelve.end notification is
    emitted properly. However the instance.vm_state is set to ACTIVE earlier[1]
    than emitting the unshelve.end notification[2]. This can cause two
    different test case failure.

    1) _test_unshelve_server() fails with no ushelve.end notification
    received.

    2) _test_shelve_and_shelve_offload_server() also has a unshelve action
    at the end and that test step also only waits for the the ACTIVE state.
    So the unshelve.end notification from the end of
    _test_shelve_and_shelve_offload_server() can bleed into the
    _test_unshelve_server() step causing that it receive one more
    notifications.

    So this patch adds an extra
    self._wait_for_notification('instance.unshelve.end') call to each test
    step to prevent the instability.

    [1] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5322-L5326
    [2] https://github.com/openstack/nova/blob/5c6c1f8f/nova/compute/manager.py#L5329
    [3] https://github.com/openstack/nova/blob/5c6c1f8f/nova/tests/functional/notification_sample_tests/test_instance.py#L836

    Conflicts:
          nova/tests/functional/notification_sample_tests/test_instance.py
    Conflicts due to:
    * I1a0afa0e8740c229db77c18b932e316196880de5

    Closes-Bug: #1835070

    Change-Id: Ie217523a8969326b27930d7f74e50e9b352ab7a1
    (cherry picked from commit 40f1e7c4c22dc6023614b34f28bb7fc416b668a8)
    (cherry picked from commit 3bc6ff029ff24083844db363010988d8d08cff00)
    (cherry picked from commit a85ce04fa39e60e672e4fa2d7912f6880079c6ef)

This issue was fixed in the openstack/nova 19.0.2 release.

This issue was fixed in the openstack/nova 18.2.2 release.

This issue was fixed in the openstack/nova 17.0.12 release.

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers