openstack-tox-py36 failing on tripleoclient.tests.v1.overcloud_deploy.test_overcloud_deploy.TestDeployOvercloud

[Pooja Jadhav]

openstack-tox-py36 failing on tripleoclient.tests.v1.overcloud_deploy.test_overcloud_deploy.TestDeployOvercloud

Below tests are failing :

[1] tripleoclient.tests.v1.overcloud_deploy.test_overcloud_deploy.TestDeployOvercloud.test_answers_file

2022-04-19 04:58:48.045938 | ubuntu-bionic | ---------------------------------------------------------------------------------------------------
2022-04-19 04:58:48.045954 | ubuntu-bionic |
2022-04-19 04:58:48.045970 | ubuntu-bionic | Captured traceback:
2022-04-19 04:58:48.045986 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~
2022-04-19 04:58:48.046002 | ubuntu-bionic | Traceback (most recent call last):
2022-04-19 04:58:48.046019 | ubuntu-bionic |
2022-04-19 04:58:48.046034 | ubuntu-bionic | File "/usr/lib/python3.6/unittest/mock.py", line 1183, in patched
2022-04-19 04:58:48.046050 | ubuntu-bionic | return func(*args, **keywargs)
2022-04-19 04:58:48.046066 | ubuntu-bionic |
2022-04-19 04:58:48.046082 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/python-tripleoclient/tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py", line 902, in test_answers_file
2022-04-19 04:58:48.046099 | ubuntu-bionic | self.cmd.take_action(parsed_args)
2022-04-19 04:58:48.046115 | ubuntu-bionic |
2022-04-19 04:58:48.046131 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/python-tripleoclient/tripleoclient/v1/overcloud_deploy.py", line 1065, in take_action
2022-04-19 04:58:48.046147 | ubuntu-bionic | utils.check_deploy_backups(self.working_dir)
2022-04-19 04:58:48.046164 | ubuntu-bionic |
2022-04-19 04:58:48.046180 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/python-tripleoclient/tripleoclient/utils.py", line 3471, in check_deploy_backups
2022-04-19 04:58:48.046196 | ubuntu-bionic | statvfs = os.statvfs(backup_file)
2022-04-19 04:58:48.046212 | ubuntu-bionic |
2022-04-19 04:58:48.046228 | ubuntu-bionic | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpuaysui6_/../tmps51s47n8/overcloud-install-20220419045841.tar.bzip2'
tripleoclient.tests.v1.overcloud_deploy.test_overcloud_deploy.TestDeployOvercloud.test_config_download_timeout

[2] tripleoclient.tests.v1.overcloud_deploy.test_overcloud_deploy.TestDeployOvercloud.test_config_download_timeout

2022-04-19 04:45:43.205688 | ubuntu-bionic | --------------------------------------------------------------------------------------------------------------
2022-04-19 04:45:43.205696 | ubuntu-bionic |
2022-04-19 04:45:43.205713 | ubuntu-bionic | Captured traceback:
2022-04-19 04:45:43.205721 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~
2022-04-19 04:45:43.205728 | ubuntu-bionic | Traceback (most recent call last):
2022-04-19 04:45:43.205736 | ubuntu-bionic |
2022-04-19 04:45:43.205744 | ubuntu-bionic | File "/usr/lib/python3.6/unittest/mock.py", line 1183, in patched
2022-04-19 04:45:43.205752 | ubuntu-bionic | return func(*args, **keywargs)
2022-04-19 04:45:43.205760 | ubuntu-bionic |
2022-04-19 04:45:43.205768 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/python-tripleoclient/tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py", line 1436, in test_config_download_timeout
2022-04-19 04:45:43.205776 | ubuntu-bionic | fixture.mock_config_download.mock_calls)
2022-04-19 04:45:43.205784 | ubuntu-bionic |
2022-04-19 04:45:43.205791 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/python-tripleoclient/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 399, in assertIn
2022-04-19 04:45:43.205799 | ubuntu-bionic | self.assertThat(haystack, Contains(needle), message)
2022-04-19 04:45:43.205807 | ubuntu-bionic |
2022-04-19 04:45:43.205815 | ubuntu-bionic | File "/home/zuul/src/opendev.org/openstack/python-tripleoclient/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 480, in assertThat
2022-04-19 04:45:43.205823 | ubuntu-bionic | raise mismatch_error
2022-04-19 04:45:43.205831 | ubuntu-bionic |
2022-04-19 04:45:43.205841 | ubuntu-bionic | testtools.matchers._impl.MismatchError: [call(<ANY>, <ANY>, <ANY>, 'ctlplane', '/tmp/tmpng8ctg9d/config-download', None, denyed_hostnames=None, deployment_options={}, deployment_timeout=448, forks=None, in_flight_validations=False, limit_hosts=None, skip_tags=None, tags=None, timeout=42, verbosity=3)] not in [call(<KeywordArgumentAdapter tripleoclient.v1.overcloud_deploy.DeployOvercloud (INFO)>, <osc_lib.tests.fakes.FakeClientManager object at 0x7fcc96cce5c0>, 'overcloud', 'ctlplane', '/tmp/tmpng8ctg9d/config-download', None, denyed_hostnames='', deployment_options={}, deployment_timeout=448, forks=None, in_flight_validations=False, limit_hosts=None, skip_tags=None, tags=None, timeout=42, verbosity=3)]

Logs :

https://5cc827387c6796d678c4-848f29e6bb7f92a80e59238721ad95f7.ssl.cf1.rackcdn.com/830974/6/check/openstack-tox-py36/7581a4e/job-output.txt

https://7c49628f39888fb28033-84d03368ee2dce2090b5b216e6b5a046.ssl.cf5.rackcdn.com/820078/21/check/openstack-tox-py36/ff9d48e/job-output.txt

https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py36&project=openstack/python-tripleoclient

[chandan kumar (chkumar246)]

Based on DNM patch https://review.opendev.org/c/openstack/python-tripleoclient/+/838447

openstack-tox-py36success is passing and https://zuul.opendev.org/t/openstack/build/886c9c86738d4ec6aa2c027279157030

[Marios Andreou (marios-b)]

seems to be transient - started yesterday (see attachment here) then we had a few today but latest on the top are now green

besides the one chkumar pointed to also green there https://review.opendev.org/c/openstack/python-tripleoclient/+/830974/ (didn't report yet https://zuul.opendev.org/t/openstack/build/4a1cb348352041c298276992d8e1c5a9 )

[Jiri Podivin (jpodivin)]

I can confirm that this bug is still present and now occurs during local tox test runs as well.

The solution would be to properly mock the functions concerned. I've already started some preliminary work in that regard.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/839712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/839712
Committed: https://opendev.org/openstack/python-tripleoclient/commit/1a59023772d23c3c870621f02657536a4da57e42
Submitter: "Zuul (22348)"
Branch: master

commit 1a59023772d23c3c870621f02657536a4da57e42
Author: Jiri Podivin <email address hidden>
Date: Thu Apr 28 10:51:16 2022 +0200

    Mocking utils.check_deploy_backups in OvercloudDeploy tests

    The `check_deploy_backups` makes calls to `stat` which can potentially
    lead to I/O error in testing scenarios when the relevant file is
    removed after path retrieval but before `stat` is called.

    This can lead to tox test failure, in both CI and local environments.
    Mocking the utils.check_deploy_backups in tests where it is indirectly
    called should alleviate the problem.

    Long term solution would be mocking all built in functions performing I/O calls by default.

    Closes-Bug: #1969425

    Signed-off-by: Jiri Podivin <email address hidden>
    Change-Id: I96d4bfce84ffe36d476e3383ee264cea6fd93c24

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/840838

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/840838
Committed: https://opendev.org/openstack/python-tripleoclient/commit/9d3fa0a2efa17e4c9953e846d206cc0ddc0ef4c2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9d3fa0a2efa17e4c9953e846d206cc0ddc0ef4c2
Author: Jiri Podivin <email address hidden>
Date: Thu Apr 28 10:51:16 2022 +0200

    Mocking utils.check_deploy_backups in OvercloudDeploy tests

    The `check_deploy_backups` makes calls to `stat` which can potentially
    lead to I/O error in testing scenarios when the relevant file is
    removed after path retrieval but before `stat` is called.

    This can lead to tox test failure, in both CI and local environments.
    Mocking the utils.check_deploy_backups in tests where it is indirectly
    called should alleviate the problem.

    Long term solution would be mocking all built in functions performing I/O calls by default.

    Closes-Bug: #1969425

    Signed-off-by: Jiri Podivin <email address hidden>
    Change-Id: I96d4bfce84ffe36d476e3383ee264cea6fd93c24
    (cherry picked from commit 1a59023772d23c3c870621f02657536a4da57e42)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 19.0.0 release.