periodic rhel-8-ovb-3ctlr_1comp-featureset001-master times out during overcloud-prep-images

[Marios Andreou]

at [1][2][3] periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master times out during image prepare with trace like

        2019-11-25 09:02:34.742005 | primary | TASK [overcloud-prep-images : Prepare the overcloud images for deploy] *********
        2019-11-25 09:02:34.762246 | primary | Monday 25 November 2019 09:02:34 -0500 (0:00:01.927) 0:00:17.076 *******
        2019-11-25 17:11:48.176341 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
        2019-11-25 17:11:48.177376 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]

This is rhel8 promotion blocker as job is in promotion criteria

[1] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/12609c1/job-output.txt
[2] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/6093de8/job-output.txt
[3] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0142e8e/job-output.txt

[Marios Andreou (marios-b)]

thanks to rlandy for digging - this seems to be related to selinux ... indeed the denials seems very busy @ http://logs.rdoproject.org/22/23822/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/2052e10/logs/undercloud/var/log/extra/denials.txt.gz

17:02 < rlandy> periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-masterIopenstack/tripleo-ciImasterIopenstack-periodic-masterImasterI2208I2019-11-18T13:12:45IFAILURE
17:02 < rlandy> periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-masterIopenstack/tripleo-ciImasterIopenstack-periodic-masterImasterI8442I2019-11-18T01:02:36ISUCCESS
17:02 < rlandy> marios|ruck: ^^ that's the date/tie of the permissive/enforcing change
17:17 < rlandy> https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/RedHat-8/promotion-testing-hash-master.yml#L89
17:17 < rlandy> marios|ruck: ^^ should be permissive

[wes hayutin (weshayutin)]

https://github.com/openstack/tripleo-quickstart/commit/ae400a009f8bb16adf8c4714c6dd9543e32d5d39

[wes hayutin (weshayutin)]

ugh.. selinux is enforcing on the undercloud

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/12609c1/logs/undercloud/etc/selinux/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/696119

[wes hayutin (weshayutin)]

Set the undercloud to permissive:
https://review.opendev.org/696119

[Marios Andreou (marios-b)]

thanks weshay and rlandy

trying to get the green run we need for promotion with testproject there (depends-on wes patch from coment #5) https://review.rdoproject.org/r/#/c/23822/

[Alex Schultz (alex-schultz)]

ironic cleaning is failing

2019-11-25 21:31:36.869 7 ERROR ironic.conductor.task_manager [req-3901637e-b5c6-495f-b5a9-b4012564985b - - - - -] Node c7a55df4-2b93-431c-90ba-1e38bb708bc7 moved to provision state "clean failed" from state "clean wait"; target provision state is "available"

[Marios Andreou (marios-b)]

Alex is right... the job still times out at the test review https://review.rdoproject.org/r/#/c/23822/

it is permissive there http://logs.rdoproject.org/22/23822/4/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0277d63/logs/undercloud/var/log/extra/selinux.txt.gz

but i see the ironic errors from comment #7 @ http://logs.rdoproject.org/22/23822/4/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0277d63/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz

        * 2019-11-26 12:55:10.982 7 ERROR ironic.conductor.task_manager [req-a0acfd3a-f21c-4c53-8fb0-0084146cefc1 - - - - -] Node 5f447471-6375-40a3-939c-811f681da7bc moved to provision state "clean failed" from state "clean wait"; target provision state is "available"

        * good log/green job doesn't have that ^^^ e.g. there http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0f19b3c/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz

[Marios Andreou (marios-b)]

this is blocking the rhel8 promotion, again in the latest run today @ https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master rhel8 fs1 is the only job missing for promotion.

We haven't promoted in a while for rhel, and it is also a blocker for https://bugs.launchpad.net/tripleo/+bug/1854062 as it looks like the previous promotion was not clean.

[Marios Andreou (marios-b)]

last green run for that was openstack/tripleo-ci master openstack-periodic-master master 8442 2019-11-18T01:02:36 SUCCESS

i just went looking at tripleo-heat-templates/tripleo-common/tripleo-quickstart & extras but did not quickly see anything related ~18th but i may well have missed it trying to get relevant folks to sanity check here for more info

[Marios Andreou (marios-b)]

i think duplicate https://bugs.launchpad.net/tripleo/+bug/1853652

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696119
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=b3203c68b55a0a997e86e26372dde2095b47d8da
Submitter: Zuul
Branch: master

commit b3203c68b55a0a997e86e26372dde2095b47d8da
Author: Wes Hayutin <email address hidden>
Date: Tue Nov 26 09:03:26 2019 -0700

    set undercloud to permissive for rhel upstream

    selinux on the undercloud/overcloud in upstream should
    always be permissive

    Related-Bug: #1854073
    Change-Id: I79c22f0ed530b94aefca52ee0339e75dcd512d9c