Functional test test_keepalived_state_change_notification timeouts waiting for enque_state_change monitor calls to be made

[Slawek Kaplonski]

Test failure:

ft1.3: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_keepalived_state_change_notificationtesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 711, in wait_until_true
    eventlet.sleep(sleep)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_ha_router.py", line 49, in test_keepalived_state_change_notification
    common_utils.wait_until_true(lambda:
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 716, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds

Example: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d32/797537/33/check/neutron-functional-with-uwsgi/d325301/testr_results.html

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2049%2C%20in%20test_keepalived_state_change_notification%5C%22

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/822051

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822051
Committed: https://opendev.org/openstack/neutron/commit/c6a6c5ae127a97856c880da3f82257977a06a571
Submitter: "Zuul (22348)"
Branch: master

commit c6a6c5ae127a97856c880da3f82257977a06a571
Author: Slawek Kaplonski <email address hidden>
Date: Thu Dec 16 22:25:27 2021 +0100

    [Functional] Fix expected number of the enqueue_state_change calls

    In the HA router's keepalived state change monitor tests, it was
    expected that enqueue_state_change method will be called 3 or 4 times.
    But after some changes in the keepalived_state_change monitor which were
    done some time ago, it may be now that it will be called just 2 or 3
    times:
    - 2 when initial status will be "primary" and it will be just
      transition to "backup",
    - 3 when initial status will be "backup", then it will transition to
      "primary" and finally to "backup" again.

    To reflect those 2 possibilities, test was changed that it will expect
    2 or 3 calls and will check only that last 2 will be always transition
    to "primary" and then to "backup".

    Additionally this patch adds some extra logging in that test so it will
    be easier to check what was going on in that test.

    Closes-Bug: #1954751
    Change-Id: Ib5de7e65839f52c35c43801969e3f0c16dead5bb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/822673

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/822674

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/822675

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/822709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/822710

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822673
Committed: https://opendev.org/openstack/neutron/commit/6265ab319e11006cc707890b2d5fb5ab2cfabb94
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 6265ab319e11006cc707890b2d5fb5ab2cfabb94
Author: Slawek Kaplonski <email address hidden>
Date: Thu Dec 16 22:25:27 2021 +0100

    [Functional] Fix expected number of the enqueue_state_change calls

    In the HA router's keepalived state change monitor tests, it was
    expected that enqueue_state_change method will be called 3 or 4 times.
    But after some changes in the keepalived_state_change monitor which were
    done some time ago, it may be now that it will be called just 2 or 3
    times:
    - 2 when initial status will be "primary" and it will be just
      transition to "backup",
    - 3 when initial status will be "backup", then it will transition to
      "primary" and finally to "backup" again.

    To reflect those 2 possibilities, test was changed that it will expect
    2 or 3 calls and will check only that last 2 will be always transition
    to "primary" and then to "backup".

    Additionally this patch adds some extra logging in that test so it will
    be easier to check what was going on in that test.

    Closes-Bug: #1954751
    Change-Id: Ib5de7e65839f52c35c43801969e3f0c16dead5bb
    (cherry picked from commit c6a6c5ae127a97856c880da3f82257977a06a571)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822674
Committed: https://opendev.org/openstack/neutron/commit/39b138ca26bc9d975804835c285fe5d4970439ed
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 39b138ca26bc9d975804835c285fe5d4970439ed
Author: Slawek Kaplonski <email address hidden>
Date: Thu Dec 16 22:25:27 2021 +0100

    [Functional] Fix expected number of the enqueue_state_change calls

    In the HA router's keepalived state change monitor tests, it was
    expected that enqueue_state_change method will be called 3 or 4 times.
    But after some changes in the keepalived_state_change monitor which were
    done some time ago, it may be now that it will be called just 2 or 3
    times:
    - 2 when initial status will be "primary" and it will be just
      transition to "backup",
    - 3 when initial status will be "backup", then it will transition to
      "primary" and finally to "backup" again.

    To reflect those 2 possibilities, test was changed that it will expect
    2 or 3 calls and will check only that last 2 will be always transition
    to "primary" and then to "backup".

    Additionally this patch adds some extra logging in that test so it will
    be easier to check what was going on in that test.

    Closes-Bug: #1954751
    Change-Id: Ib5de7e65839f52c35c43801969e3f0c16dead5bb
    (cherry picked from commit c6a6c5ae127a97856c880da3f82257977a06a571)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822675
Committed: https://opendev.org/openstack/neutron/commit/533ca2c99c454c171f2b8d6577a39bb7be6376e6
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 533ca2c99c454c171f2b8d6577a39bb7be6376e6
Author: Slawek Kaplonski <email address hidden>
Date: Thu Dec 16 22:25:27 2021 +0100

    [Functional] Fix expected number of the enqueue_state_change calls

    In the HA router's keepalived state change monitor tests, it was
    expected that enqueue_state_change method will be called 3 or 4 times.
    But after some changes in the keepalived_state_change monitor which were
    done some time ago, it may be now that it will be called just 2 or 3
    times:
    - 2 when initial status will be "primary" and it will be just
      transition to "backup",
    - 3 when initial status will be "backup", then it will transition to
      "primary" and finally to "backup" again.

    To reflect those 2 possibilities, test was changed that it will expect
    2 or 3 calls and will check only that last 2 will be always transition
    to "primary" and then to "backup".

    Additionally this patch adds some extra logging in that test so it will
    be easier to check what was going on in that test.

    Closes-Bug: #1954751
    Change-Id: Ib5de7e65839f52c35c43801969e3f0c16dead5bb
    (cherry picked from commit c6a6c5ae127a97856c880da3f82257977a06a571)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822709
Committed: https://opendev.org/openstack/neutron/commit/bed8c6d58028b7d5a426dbd474dccbe34048fd63
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit bed8c6d58028b7d5a426dbd474dccbe34048fd63
Author: Slawek Kaplonski <email address hidden>
Date: Thu Dec 16 22:25:27 2021 +0100

    [Functional] Fix expected number of the enqueue_state_change calls

    In the HA router's keepalived state change monitor tests, it was
    expected that enqueue_state_change method will be called 3 or 4 times.
    But after some changes in the keepalived_state_change monitor which were
    done some time ago, it may be now that it will be called just 2 or 3
    times:
    - 2 when initial status will be "primary" and it will be just
      transition to "backup",
    - 3 when initial status will be "backup", then it will transition to
      "primary" and finally to "backup" again.

    To reflect those 2 possibilities, test was changed that it will expect
    2 or 3 calls and will check only that last 2 will be always transition
    to "primary" and then to "backup".

    Additionally this patch adds some extra logging in that test so it will
    be easier to check what was going on in that test.

    Conflicts:
        neutron/tests/functional/agent/l3/test_ha_router.py

    Closes-Bug: #1954751
    Change-Id: Ib5de7e65839f52c35c43801969e3f0c16dead5bb
    (cherry picked from commit c6a6c5ae127a97856c880da3f82257977a06a571)
    (cherry picked from commit 533ca2c99c454c171f2b8d6577a39bb7be6376e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822710
Committed: https://opendev.org/openstack/neutron/commit/0d037895c75a9ca401a2b3cdd519be6739983157
Submitter: "Zuul (22348)"
Branch: stable/train

commit 0d037895c75a9ca401a2b3cdd519be6739983157
Author: Slawek Kaplonski <email address hidden>
Date: Thu Dec 16 22:25:27 2021 +0100

    [Functional] Fix expected number of the enqueue_state_change calls

    In the HA router's keepalived state change monitor tests, it was
    expected that enqueue_state_change method will be called 3 or 4 times.
    But after some changes in the keepalived_state_change monitor which were
    done some time ago, it may be now that it will be called just 2 or 3
    times:
    - 2 when initial status will be "primary" and it will be just
      transition to "backup",
    - 3 when initial status will be "backup", then it will transition to
      "primary" and finally to "backup" again.

    To reflect those 2 possibilities, test was changed that it will expect
    2 or 3 calls and will check only that last 2 will be always transition
    to "primary" and then to "backup".

    Additionally this patch adds some extra logging in that test so it will
    be easier to check what was going on in that test.

    Conflicts:
        neutron/tests/functional/agent/l3/test_ha_router.py

    Closes-Bug: #1954751
    Change-Id: Ib5de7e65839f52c35c43801969e3f0c16dead5bb
    (cherry picked from commit c6a6c5ae127a97856c880da3f82257977a06a571)
    (cherry picked from commit 533ca2c99c454c171f2b8d6577a39bb7be6376e6)
    (cherry picked from commit bed8c6d58028b7d5a426dbd474dccbe34048fd63)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.