train beaker job is failing because httpd fails to start

[Takashi Kajinami]

Currently beaker jobs are continuously failing in stable/train.

An example can be found here.
 https://review.opendev.org/c/openstack/puppet-nova/+/789760
 https://zuul.opendev.org/t/openstack/build/b0f4bf20d6af44a09717d699de8df9b1

Looking at the log, it is observed that httpd fails to start because of Permission error.

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b0f/789760/1/check/puppet-openstack-beaker-centos-7/b0f4bf2/job-output.txt
~~~
2021-05-05 09:38:32.724469 | centos-7 | Error: /Stage[main]/Apache::Service/Service[httpd]/ensure: change from 'stopped' to 'running' failed: Systemd start for httpd failed!
2021-05-05 09:38:32.724561 | centos-7 | journalctl log for httpd:
2021-05-05 09:38:32.724753 | centos-7 | -- Logs begin at Wed 2021-05-05 09:20:13 UTC, end at Wed 2021-05-05 09:38:32 UTC. --
2021-05-05 09:38:32.724979 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 systemd[1]: Starting The Apache HTTP Server...
2021-05-05 09:38:32.725317 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 httpd[9904]: (13)Permission denied: AH00072: make_sock: could not bind to address 127.0.0.1:5000
2021-05-05 09:38:32.725608 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 httpd[9904]: no listening sockets available, shutting down
2021-05-05 09:38:32.725810 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 httpd[9904]: AH00015: Unable to open logs
2021-05-05 09:38:32.726101 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
2021-05-05 09:38:32.726318 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 systemd[1]: Failed to start The Apache HTTP Server.
2021-05-05 09:38:32.726536 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 systemd[1]: Unit httpd.service entered failed state.
2021-05-05 09:38:32.726743 | centos-7 | May 05 09:38:32 centos-7-rax-dfw-0024556763 systemd[1]: httpd.service failed.
~~~

It seems that the error is caused by selinux denial.

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b0f/789760/1/check/puppet-openstack-beaker-centos-7/b0f4bf2/logs/audit.log.txt
~~~
type=AVC msg=audit(1620207512.662:8885): avc: denied { name_bind } for pid=9904 comm="httpd" src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket permissive=0
~~~

The latest openstack-selinux is installed

~~~
2021-05-05 09:32:55.586547 | centos-7 | Notice: /Stage[main]/Openstack_integration/Package[openstack-selinux]/ensure: created
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b0f/789760/1/check/puppet-openstack-beaker-centos-7/b0f4bf2/logs/rpm-qa.txt
~~~
openstack-selinux-0.8.26-0.20210326113922.f9bf0f2.el7.noarch
...
selinux-policy-3.13.1-268.el7_9.2.noarch
selinux-policy-targeted-3.13.1-268.el7_9.2.noarch
~~~

[Takashi Kajinami (kajinamit)]

Oh, wait, where do we get that openstack-selinux package ?
As far as I see cloud7-openstack-queens-release provides openstack-selinux-0.8.13-1.el7 .

[Takashi Kajinami (kajinamit)]

OK. The 0.8.26 package is what rdo train provides.

https://trunk.rdoproject.org/centos7-train/current-passed-ci/
 openstack-selinux-0.8.26-0.20210326113922.f9bf0f2.el7.noarch.rpm

Maybe something is outdated in rdopkg command. Ignore my previous comment...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789795

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789816

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789795

[Tobias Urdin (tobias-urdin)]

See https://bugs.launchpad.net/packstack/+bug/1923005

[Tobias Urdin (tobias-urdin)]

14:32 < tobias-urdin> ykarel: is there a newer container-selinux available for centos7 we are just not updating it or it's just not there at all?
14:33 < ykarel> tobias-urdin, container-selinux good version which is compatible with latest openstack-selinux not available in centos 7 base repos

[Tobias Urdin (tobias-urdin)]

14:26 < ykarel> tkajinam, mainly this is the culprit https://github.com/redhat-openstack/openstack-selinux/commit/1f3ab78f0d9b5e1d76ca420873889e9c6f54faf0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789816
Committed: https://opendev.org/openstack/puppet-openstack-integration/commit/952ae0e3f152ed55eeebe8f099a6e8c5c6da4553
Submitter: "Zuul (22348)"
Branch: stable/train

commit 952ae0e3f152ed55eeebe8f099a6e8c5c6da4553
Author: Takashi Kajinami <email address hidden>
Date: Wed May 5 20:36:37 2021 +0900

    Disable selinux in CentOS7 beaker job

    This change ensures selinux is disable on CentOS7 beaker job to
    workaround an selinux denial currently blocking the gate.

    Related-Bug: #1927210
    Change-Id: Ie1bf3ebdb2642e24beb10d80878b8eecd67ce9e0

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789923

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789923
Committed: https://opendev.org/openstack/puppet-openstack-integration/commit/e7d7ba5c8fbfa388d6dcd8c92a07114a5824e575
Submitter: "Zuul (22348)"
Branch: stable/train

commit e7d7ba5c8fbfa388d6dcd8c92a07114a5824e575
Author: Takashi Kajinami <email address hidden>
Date: Thu May 6 01:12:29 2021 +0900

    Fix ansible error when making selinux permissive

    This fixes the following error caused by the parent commit[1].

    {
        "msg": "Policy is required if state is not 'disabled'"
    }

    [1] 952ae0e3f152ed55eeebe8f099a6e8c5c6da4553

    Related-Bug: #1927210
    Change-Id: I272f0cc5c2e35752b44111425ed2f676376f26f4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789960

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/789960
Committed: https://opendev.org/openstack/puppet-openstack-integration/commit/ba73591f2208d03cb499c6f588e10b86e8327259
Submitter: "Zuul (22348)"
Branch: stable/train

commit ba73591f2208d03cb499c6f588e10b86e8327259
Author: Takashi Kajinami <email address hidden>
Date: Thu May 6 08:30:52 2021 +0900

    Update selinux mode by root

    This change fixes the remaining error while setting selinux permissive.

    TASK [Disable selinux (CentOS/RHEL<=7) (bug/1927210)]
    ...
    centos-7 | OSError: [Errno 13] Permission denied

    Related-Bug: #1927210
    Change-Id: Id04e62980501f2f8e94ef6ab9b53c221ba8e231a

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/792982

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/792983

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/792984

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/792984
Committed: https://opendev.org/openstack/puppet-openstack-integration/commit/1c6881dc0469797416d0b21583bccddf1dca3eb8
Submitter: "Zuul (22348)"
Branch: stable/queens

commit 1c6881dc0469797416d0b21583bccddf1dca3eb8
Author: Takashi Kajinami <email address hidden>
Date: Wed May 5 20:36:37 2021 +0900

    Disable selinux in CentOS7 beaker job

    This change ensures selinux is disable on CentOS7 beaker job to
    workaround an selinux denial currently blocking the gate.

    Backport note:
    This commit includes the following two commits which fixed some issues
    with the task to disable selinux.

    e7d7ba5c8fbfa388d6dcd8c92a07114a5824e575
    Fix ansible error when making selinux permissive

    ba73591f2208d03cb499c6f588e10b86e8327259
    Update selinux mode by root

    Related-Bug: #1927210
    Change-Id: Ie1bf3ebdb2642e24beb10d80878b8eecd67ce9e0
    (cherry picked from commit 952ae0e3f152ed55eeebe8f099a6e8c5c6da4553)
    (cherry picked from commit 1d06ff2e11d7b894d5a2e2627224a9578f56551e)
    (cherry picked from commit 71a07238b1c3e06d1eddf5187a427e8742bc168c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/792983
Committed: https://opendev.org/openstack/puppet-openstack-integration/commit/71a07238b1c3e06d1eddf5187a427e8742bc168c
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 71a07238b1c3e06d1eddf5187a427e8742bc168c
Author: Takashi Kajinami <email address hidden>
Date: Wed May 5 20:36:37 2021 +0900

    Disable selinux in CentOS7 beaker job

    This change ensures selinux is disable on CentOS7 beaker job to
    workaround an selinux denial currently blocking the gate.

    Backport note:
    This commit includes the following two commits which fixed some issues
    with the task to disable selinux.

    e7d7ba5c8fbfa388d6dcd8c92a07114a5824e575
    Fix ansible error when making selinux permissive

    ba73591f2208d03cb499c6f588e10b86e8327259
    Update selinux mode by root

    Related-Bug: #1927210
    Change-Id: Ie1bf3ebdb2642e24beb10d80878b8eecd67ce9e0
    (cherry picked from commit 952ae0e3f152ed55eeebe8f099a6e8c5c6da4553)
    (cherry picked from commit 1d06ff2e11d7b894d5a2e2627224a9578f56551e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-openstack-integration/+/792982
Committed: https://opendev.org/openstack/puppet-openstack-integration/commit/1d06ff2e11d7b894d5a2e2627224a9578f56551e
Submitter: "Zuul (22348)"
Branch: stable/stein

commit 1d06ff2e11d7b894d5a2e2627224a9578f56551e
Author: Takashi Kajinami <email address hidden>
Date: Wed May 5 20:36:37 2021 +0900

    Disable selinux in CentOS7 beaker job

    This change ensures selinux is disable on CentOS7 beaker job to
    workaround an selinux denial currently blocking the gate.

    Backport note:
    This commit includes the following two commits which fixed some issues
    with the task to disable selinux.

    e7d7ba5c8fbfa388d6dcd8c92a07114a5824e575
    Fix ansible error when making selinux permissive

    ba73591f2208d03cb499c6f588e10b86e8327259
    Update selinux mode by root

    Related-Bug: #1927210
    Change-Id: Ie1bf3ebdb2642e24beb10d80878b8eecd67ce9e0
    (cherry picked from commit 952ae0e3f152ed55eeebe8f099a6e8c5c6da4553)