rdo on rhel 7.4, undercloud httpd service fails to start due to could not bind to address 192.168.24.1:6385

[wes hayutin]

https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-62/undercloud/home/stack/undercloud_install.log.txt.gz#_2017-09-19_16_14_56

https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtha-3ctlr_1comp_192gb-38/undercloud/home/stack/undercloud_install.log.txt.gz#_2017-09-19_06_52_35

https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-61/undercloud/home/stack/undercloud_install.log.txt.gz#_2017-09-19_10_53_28

https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-62/undercloud/home/stack/undercloud_install.log.txt.gz

https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtha-3ctlr_1comp_192gb-39/undercloud/home/stack/undercloud_install.log.txt.gz

2017-09-19 16:14:56 | 2017-09-19 16:14:56,131 INFO: Error: Systemd start for httpd failed!
2017-09-19 16:14:56 | 2017-09-19 16:14:56,132 INFO: journalctl log for httpd:
2017-09-19 16:14:56 | 2017-09-19 16:14:56,132 INFO: -- Logs begin at Tue 2017-09-19 16:00:39 EDT, end at Tue 2017-09-19 16:14:56 EDT. --
2017-09-19 16:14:56 | 2017-09-19 16:14:56,132 INFO: Sep 19 16:14:12 undercloud.localdomain systemd[1]: Starting The Apache HTTP Server...
2017-09-19 16:14:56 | 2017-09-19 16:14:56,132 INFO: Sep 19 16:14:55 undercloud.localdomain python[10939]: Compressing... done
2017-09-19 16:14:56 | 2017-09-19 16:14:56,133 INFO: Sep 19 16:14:55 undercloud.localdomain python[10939]: Compressed 6 block(s) from 3 template(s) for 2 context(s).
2017-09-19 16:14:56 | 2017-09-19 16:14:56,133 INFO: Sep 19 16:14:56 undercloud.localdomain httpd[11012]: (13)Permission denied: AH00072: make_sock: could not bind to address 192.168.24.1:6385
2017-09-19 16:14:56 | 2017-09-19 16:14:56,133 INFO: Sep 19 16:14:56 undercloud.localdomain httpd[11012]: no listening sockets available, shutting down
2017-09-19 16:14:56 | 2017-09-19 16:14:56,133 INFO: Sep 19 16:14:56 undercloud.localdomain httpd[11012]: AH00015: Unable to open logs
2017-09-19 16:14:56 | 2017-09-19 16:14:56,133 INFO: Sep 19 16:14:56 undercloud.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
2017-09-19 16:14:56 | 2017-09-19 16:14:56,134 INFO: Sep 19 16:14:56 undercloud.localdomain kill[11015]: kill: cannot find process ""
2017-09-19 16:14:56 | 2017-09-19 16:14:56,134 INFO: Sep 19 16:14:56 undercloud.localdomain systemd[1]: httpd.service: control process exited, code=exited status=1
2017-09-19 16:14:56 | 2017-09-19 16:14:56,134 INFO: Sep 19 16:14:56 undercloud.localdomain systemd[1]: Failed to start The Apache HTTP Server.
2017-09-19 16:14:56 | 2017-09-19 16:14:56,134 INFO: Sep 19 16:14:56 undercloud.localdomain systemd[1]: Unit httpd.service entered failed state.
2017-09-19 16:14:56 | 2017-09-19 16:14:56,134 INFO: Sep 19 16:14:56 undercloud.localdomain systemd[1]: httpd.service failed.

/var/log/messages
Sep 19 16:14:56 undercloud httpd: no listening sockets available, shutting down
Sep 19 16:14:56 undercloud httpd: AH00015: Unable to open logs
Sep 19 16:14:56 undercloud systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
Sep 19 16:14:56 undercloud kill: kill: cannot find process ""
Sep 19 16:14:56 undercloud systemd: httpd.service: control process exited, code=exited status=1
Sep 19 16:14:56 undercloud systemd: Failed to start The Apache HTTP Server.
Sep 19 16:14:56 undercloud systemd: Unit httpd.service entered failed state.
Sep 19 16:14:56 undercloud systemd: httpd.service failed.

[wes hayutin (weshayutin)]

a net-stat is here

https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-62/undercloud/var/log/extra/netstat.txt.gz

[Alan Pevec (apevec)]

192.168.24.1:6385 is not occupied in the netstat output, but there is AVC

type=AVC msg=audit(1505852096.031:1377): avc: denied { name_bind } for pid=11012 comm="httpd" src=6385 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

in https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-62/undercloud/var/log/audit/audit.log.txt.gz

[Alan Pevec (apevec)]

centos job for the same pike hash has the same AVC denial recorded, but it was obviously running permissive so it continued:

https://thirdparty.logs.rdoproject.org/jenkins-periodic-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-59/undercloud/var/log/extra/denials.txt.gz

[Alan Pevec (apevec)]

comparing https://thirdparty.logs.rdoproject.org/jenkins-promote-rhel-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-62/undercloud/var/log/journal.txt.gz
vs
https://thirdparty.logs.rdoproject.org/jenkins-periodic-pike-rdo_trunk-virtbasic-1ctlr_1comp_64gb-59/undercloud/var/log/journal.txt.gz

* both start with SELinux: Starting in permissive mode
* in rhel there is more going on re. selinux 23 vs 15 matches

[wes hayutin (weshayutin)]

Additional fix was required due to how images are built. This has been built downstream.

https://bugzilla.redhat.com/show_bug.cgi?id=1489863#c19
redhat-openstack/openstack-selinux: 52b3fe8

Downstream one-off for OSP12:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=598760

openstack-selinux-0.8.10-0.20170914195211.e16a8f8.2.el7ost