CI: apache fails to start on overcloud controller

[Sagi (Sergey) Shnaidman]

Apache fails to start on overcloud.

[1;31mError: /Stage[main]/Apache::Service/Service[httpd]: Failed to call refresh: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service failed because a timeout was exceeded. See "systemctl status httpd.service" and "journalctl -xe" for details.
2016-11-10 16:37:04.000 | Error: /Stage[main]/Apache::Service/Service[httpd]: Could not restart Service[httpd]: Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service failed because a timeout was exceeded. See "systemctl status httpd.service" and "journalctl -xe" for details.

http://logs.openstack.org/48/375348/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/f6a3a55/logs/postci.txt.gz#_2016-11-09_12_46_48_000

http://logs.openstack.org/21/395521/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/715a49d/logs/postci.txt.gz#_2016-11-09_13_02_36_000

http://logs.openstack.org/02/395702/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/70abfa6/logs/postci.txt.gz#_2016-11-10_16_37_04_000

See also may be related: https://bugs.launchpad.net/tripleo/+bug/1572976

[Sagi (Sergey) Shnaidman (sshnaidm)]

It's pretty rare, but happens.

http://logs.openstack.org/13/386713/9/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/e60f6cb/logs/postci.txt.gz#_2016-11-15_02_48_18_000

[Sagi (Sergey) Shnaidman (sshnaidm)]

When tried to run it with quickstart it appears that httpd has "*:8774" in its virtualhost configuration, but nova-api listens on non-vip 8774 port and haproxy on vip 8774 port. So when httpd tries to start it fails with "can not bind socket" error.
But it may be a root cause only for this job.

[Emilien Macchi (emilienm)]

We promoted CI and have new robotop and horizon. It works now: https://review.openstack.org/#/c/400944/