[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'

[wes hayutin]

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/ad35888/logs/undercloud/home/zuul/tempest.log.txt.gz

2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 CRITICAL tempest [-] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest Traceback (most recent call last):
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest File "/usr/bin/discover-tempest-config", line 10, in <module>
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest sys.exit(main())
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/main.py", line 597, in main
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest verbose=args.verbose
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/main.py", line 523, in config_tempest
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest users.create_tempest_users(services.is_service('orchestration'))
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/users.py", line 58, in create_tempest_users
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest role_required=False)
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest File "/usr/lib/python2.7/site-packages/config_tempest/users.py", line 76, in give_role_to_user
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest if self._conf.get('auth', 'tempest_roles') not in roles_names:
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest File "/usr/lib64/python2.7/ConfigParser.py", line 618, in get
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest raise NoOptionError(option, section)
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest NoOptionError: No option 'tempest_roles' in section: 'auth'
2019-05-08 06:37:41 | 2019-05-08 06:37:41.394 16213 ERROR tempest

Same failure May-08 - May 13th

[Marios Andreou (marios-b)]

adding promotion blocker since this job is in criteria periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens @ https://github.com/rdo-infra/ci-config/blob/4fa8eda72d664733acbe9f2650405592ef68affb/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini#L24

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660036

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660036
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=6c1cafd985fe9b168997347e7f5638792c465f3e
Submitter: Zuul
Branch: master

commit 6c1cafd985fe9b168997347e7f5638792c465f3e
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 20 12:43:59 2019 +0530

    Remove auth.tempest_role from extra tempest overrides

    As https://review.opendev.org/#/c/638617/ takes cares for looking
    for Member role so we do not need to specifically specify the same
    in tempest config extra overrides.

    Depends-On: https://review.opendev.org/#/c/660060/

    Closes-Bug: #1828859

    Change-Id: I478e5f036e3c57c09dae6623bf00e4b4f40e55cb
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[Marios Andreou (marios-b)]

no periodic run yet but this should be done hopefull will get a green one today lets see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens

[Marios Andreou (marios-b)]

hmmm @chandan still seeing the same in the periodic run today can you check please

2019-05-21 06:27:01 | 2019-05-21 06:27:01.054 14357 ERROR tempest NoOptionError: No option 'tempest_roles' in section: 'auth'

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/d657319/logs/undercloud/home/zuul/tempest.log.txt.gz

[chandan kumar (chkumar246)]

I think the fix is not yet picked by in the periodic promotion run, we need to wait for next run.

[Marios Andreou (marios-b)]

so lest also remove from member role from featureset 017? hmm unless that would break non queens i.e. other things using the fs

13:30 < ykarel> chandankumar, marios i see tempest_roles in fs017, shouldn't that be cleaned instead of the patch that merged?

13:31 < ykarel> chandankumar, marios_|ruck i mean
                https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset017.yml#L171

13:33 < marios_|ruck> ykarel: not sure question for chandankumar but i don't see any changes in the fs from reviews we can post one
                      though unless chandankumar disagrees

[Martin Schuppert (mschuppert)]

I see the same on queens in https://review.opendev.org/#/c/660585/ - tripleo-ci-centos-7-undercloud-oooq [1].

[1] http://logs.openstack.org/85/660585/1/check/tripleo-ci-centos-7-undercloud-oooq/ccfcf68/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-22_08_22_42

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660649

[Martin Kopec (mkopec)]

This review [1] is wrong, it builds on a wrong presumption that [2] does something it doesn't. Based on that [3] is just a workaround, but it doesn't fix the original issue.

The original issue is, that python-tempestconf doesn't take into account that tempest_roles might be a list, therefore this condition [4] is wrong.

Give me some time, I'll send a review to fix that within today.

[1] https://review.opendev.org/#/c/660036/
[2] https://review.opendev.org/#/c/638617/
[3] https://review.opendev.org/#/c/660649/
[4] https://opendev.org/openstack/python-tempestconf/src/commit/e0d5dd6725f9a21090cae69a12bf087004a17e13/config_tempest/users.py#L76

[Martin Kopec (mkopec)]

https://storyboard.openstack.org/#!/story/2005758

And https://review.opendev.org/#/c/660036/ will probably need to be reverted.

[Marios Andreou (marios-b)]

review from mkopec https://review.opendev.org/#/c/660748 Fix checking of roles

10:14 < chandankumar> marios_|ruck: testing here https://review.rdoproject.org/r/#/c/20899/
10:15 < kopecmartin> marios_|ruck, revert of that patch might not be needed at the end , we'll see, and the fix for the issue should be here https://review.opendev.org/#/c/660748/2

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.opendev.org/660649

[Marios Andreou (marios-b)]

another related patch here https://review.rdoproject.org/r/#/c/20909/ Added patch for fixing Member role

[Marios Andreou (marios-b)]

09:50 < marios_|ruck> chandankumar: so short term https://review.rdoproject.org/r/#/c/20909/ but longer term fix https://review.opendev.org/#/c/660748
09:50 < chandankumar> marios_|ruck: yes

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.opendev.org/660351

[Marios Andreou (marios-b)]

so https://review.rdoproject.org/r/#/c/20909/ should unblock the queens gate for this bug

looking now however not seeing green runs yet at

http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-oooq&branch=stable%2Fqueens

or for the periodic yet

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens