test_init_application_called_twice unit test fails intermittently

[Balazs Gibizer]

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_350/782634/3/check/openstack-tox-py38/35000f9/testr_results.html

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py38/lib/python3.8/site-packages/mock/mock.py", line 1346, in patched
    return func(*newargs, **newkeywargs)
  File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/unit/api/test_wsgi.py", line 65, in test_init_application_passes_sys_argv_to_config
    mock_parse_args.assert_called_once_with(
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py38/lib/python3.8/site-packages/mock/mock.py", line 925, in assert_called_once_with
    raise AssertionError(msg)
AssertionError: Expected 'parse_args' to be called once. Called 0 times.

It seems that after [1] merged the two init tests test_init_application_passes_sys_argv_to_config and test_init_application_called_twice cannot be run in the same executor without the second failing.

It is probably due the global state introduced in [1] not handled properly in the test.

[1] https://review.opendev.org/c/openstack/nova/+/733627

[Balazs Gibizer (balazs-gibizer)]

It can be reproduce with

tox -e py38 "'test_init_application_passes_sys_argv_to_config|test_init_application_called_twice'" -- --serial

[Balazs Gibizer (balazs-gibizer)]

fix proposed to master: https://review.opendev.org/c/openstack/nova/+/782691

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

[melanie witt (melwitt)]

https://review.opendev.org/c/openstack/nova/+/785060 proposed to stable/victoria

[melanie witt (melwitt)]

https://review.opendev.org/c/openstack/nova/+/785062 proposed to stable/ussuri

[melanie witt (melwitt)]

https://review.opendev.org/c/openstack/nova/+/785065 proposed to stable/train

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/785060
Reason: squashed into https://review.opendev.org/c/openstack/nova/+/785059

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/785059
Committed: https://opendev.org/openstack/nova/commit/e3085fa6310ddeaafa493c3f718aab0ce64f0994
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit e3085fa6310ddeaafa493c3f718aab0ce64f0994
Author: Hervé Beraud <email address hidden>
Date: Thu Jun 4 09:49:59 2020 +0200

    Initialize global data separately and run_once in WSGI app init

    NOTE(melwitt): This is a combination of two changes to avoid
    intermittent test failure that was introduced by the original bug fix,
    and was fixed by change I2bd360dcc6501feea7baf02d4510b282205fc061.

    We have discovered that if an exception is raised at any point during
    the running of the init_application WSGI script in an apache/mod_wsgi
    Daemon Mode environment, it will prompt apache/mod_wsgi to re-run the
    script without starting a fresh python process. Because we initialize
    global data structures during app init, subsequent runs of the script
    blow up as some global data do *not* support re-initialization. It is
    anyway not safe to assume that init of global data is safe to run
    multiple times.

    This mod_wsgi behavior appears to be a special situation that does not
    behave the same as a normal reload in Daemon Mode as the script file is
    being reloaded upon failure instead of the daemon process being
    shutdown and restarted as described in the documentation [1].

    In order to handle this situation, we can move the initialization of
    global data structures to a helper method that is decorated to run only
    once per python interpreter instance. This way, we will not attempt to
    re-initialize global data that are not safe to init more than once.

    Co-Authored-By: Michele Baldessari <email address hidden>
    Co-Authored-By: melanie witt <email address hidden>

    Conflicts:
        nova/api/openstack/wsgi_app.py

    NOTE(melwitt): The conflict is because change
    If4783adda92da33d512d7c2834f0bb2e2a9b9654 (Support sys.argv in wsgi
    app) is not in Victoria.

    Closes-Bug: #1882094

    [1] https://modwsgi.readthedocs.io/en/develop/user-guides/reloading-source-code.html#reloading-in-daemon-mode

    Reset global wsgi app state in unit test

    Since I2bd360dcc6501feea7baf02d4510b282205fc061 there is a global state
    set during the wsgi_app init making our unit test cases
    non-deterministic based on the order of them. This patch makes sure
    that the global state is reset for each test case.

    Closes-Bug: #1921098
    (cherry picked from commit bc2c19bb2db901af0c48d34fb15a335f4e343361)

    Change-Id: I2bd360dcc6501feea7baf02d4510b282205fc061
    (cherry picked from commit 7c9edc02eda45aafbbb539b759e6b92f7aeb5ea8)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/785061
Reason: Using https://review.opendev.org/c/openstack/nova/+/785061 instead.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/785062
Reason: Using https://review.opendev.org/c/openstack/nova/+/785061 instead.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "melanie witt <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/785065
Reason: Squashed into https://review.opendev.org/c/openstack/nova/+/785064

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 22.2.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/785061
Committed: https://opendev.org/openstack/nova/commit/59249697bf09ca1a560defdd550be7e3c439b5b7
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 59249697bf09ca1a560defdd550be7e3c439b5b7
Author: Hervé Beraud <email address hidden>
Date: Thu Jun 4 09:49:59 2020 +0200

    Initialize global data separately and run_once in WSGI app init

    NOTE(melwitt): This is a combination of two changes to avoid
    intermittent test failure that was introduced by the original bug fix,
    and was fixed by change I2bd360dcc6501feea7baf02d4510b282205fc061.

    We have discovered that if an exception is raised at any point during
    the running of the init_application WSGI script in an apache/mod_wsgi
    Daemon Mode environment, it will prompt apache/mod_wsgi to re-run the
    script without starting a fresh python process. Because we initialize
    global data structures during app init, subsequent runs of the script
    blow up as some global data do *not* support re-initialization. It is
    anyway not safe to assume that init of global data is safe to run
    multiple times.

    This mod_wsgi behavior appears to be a special situation that does not
    behave the same as a normal reload in Daemon Mode as the script file is
    being reloaded upon failure instead of the daemon process being
    shutdown and restarted as described in the documentation [1].

    In order to handle this situation, we can move the initialization of
    global data structures to a helper method that is decorated to run only
    once per python interpreter instance. This way, we will not attempt to
    re-initialize global data that are not safe to init more than once.

    Co-Authored-By: Michele Baldessari <email address hidden>
    Co-Authored-By: melanie witt <email address hidden>

    Conflicts:
      nova/test.py

    NOTE(melwitt): The conflict is because change
    I1fea14d5be10bb4e884f52e0ae8be722519ddd3f (Poison
    netifaces.interfaces() in tests) is not in Ussuri.

    Closes-Bug: #1882094

    [1] https://modwsgi.readthedocs.io/en/develop/user-guides/reloading-source-code.html#reloading-in-daemon-mode

    Reset global wsgi app state in unit test

    Since I2bd360dcc6501feea7baf02d4510b282205fc061 there is a global state
    set during the wsgi_app init making our unit test cases
    non-deterministic based on the order of them. This patch makes sure
    that the global state is reset for each test case.

    Closes-Bug: #1921098
    (cherry picked from commit bc2c19bb2db901af0c48d34fb15a335f4e343361)

    Change-Id: I2bd360dcc6501feea7baf02d4510b282205fc061
    (cherry picked from commit 7c9edc02eda45aafbbb539b759e6b92f7aeb5ea8)
    (cherry picked from commit e3085fa6310ddeaafa493c3f718aab0ce64f0994)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 21.2.3 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/785064
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.