object storage container and quota tempest tests are failing with connnection refused on swift healthcheck api on scenario4 stable/stein branch

[chandan kumar]

http://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario004-standalone&pipeline=check&branch=stable%2Fstein

tripleo-ci-centos-7-scenario004-standalone on stable/stein is broken from very long time on tempest with following issues:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bc6/706148/2/check/tripleo-ci-centos-7-scenario004-standalone/bc6e24b/logs/undercloud/home/zuul/tempest.log

Below is the failing tests:
* tempest.api.object_storage.test_account_quotas.AccountQuotasTest -> Details: No "ResellerAdmin" role found

* tempest.api.object_storage.test_account_services.AccountTest
* tempest.api.object_storage.test_object_services.ObjectTest
* tempest.api.object_storage.test_container_quotas.ContainerQuotasTest
* tempest.api.object_storage.test_container_services.ContainerTest

2020-02-06 01:15:07 | File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2020-02-06 01:15:07 | _stacktrace=sys.exc_info()[2])
2020-02-06 01:15:07 | File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 398, in increment
2020-02-06 01:15:07 | raise MaxRetryError(_pool, url, error or ResponseError(cause))
2020-02-06 01:15:07 | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.1', port=8080): Max retries exceeded with url: /info (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4812fd34d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2020-02-06 01:15:07 |

Tempestconf is also failing with the same issue.
2020-02-06 00:58:41 | 2020-02-06 00:58:41.274 169082 WARNING config_tempest.constants [-] Healthcheck API not discovered giving HTTPConnectionPool(host='192.168.24.1', port=8080): Max retries exceeded with url: /swift/healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcb38d252d0>: Failed to establish a new connection: [Errno 111] Connection refused',)): MaxRetryError: HTTPConnectionPool(host='192.168.24.1', port=8080): Max retries exceeded with url: /swift/healthcheck (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcb38d252d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

It might be related to ceph. Currently we are skipping it till we fix the test.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/706215

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/706218

[yatin (yatinkarel)]

Pasting from IRC, before it's get lost post weekend.

<karel|away> raukadah, so scenario004 stein also have similar issue
<ykarel|away> tempest_test_whitelist is specific to os-tempest, post moving away from os-tempest, for tempest run need to set equivalent
<raukadah> ykarel|away, but tests are different so settings vary
<ykarel|away> raukadah, u mean based on how tempest runs, different tests will run?
<raukadah> ykarel|away, we have different scenarios and services configured there
<ykarel|away> raukadah, so scenario004 whether runs with os-tempest or tempest run, i think it will have same services
<ykarel|away> and same tests should run
<ykarel|away> so i would say need to set tempest_whitelist
<ykarel|away> same as tempest_test_whitelist

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706215
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=699355b71b5272832a1a3c4632dbc3ecaeee66ad
Submitter: Zuul
Branch: master

commit 699355b71b5272832a1a3c4632dbc3ecaeee66ad
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 6 15:48:21 2020 +0530

    Updated tempest skip list for stein

    Currently scenario 1 and 4 standalone stable/stein check and gate
    job is broken due to volume encryption and swift related tempest
    tests failure.

    In order to unblock the gates, we are currently moving tests to
    skip list.

    Related-Bug: #1862143
    Related-Bug: #1862142

    Depends-On: https://review.opendev.org/#/c/706196/

    Change-Id: Ieabc0d56e01ced15860da5642c9d62c620e9c0db
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706218
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=1f5cf689725cec8df7bc41bc3d5adcfcb8f77191
Submitter: Zuul
Branch: master

commit 1f5cf689725cec8df7bc41bc3d5adcfcb8f77191
Author: Soniya Vyas <email address hidden>
Date: Thu Feb 6 15:52:39 2020 +0530

    Added missing role

    There was an absence of ResellerAdmin role in
    the configure-tempest.sh.j2
    This patch fixes it.

    Related-Bug: #1862143
    Signed-off by: Soniya Vyas<email address hidden>

    Change-Id: I34b76c70986da82fde8c74cbbcfd7a5d55ea9405