periodic nonha job fails to install undercloud

[Gabriele Cerami]

In the past two days' job logs this error shows up

[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/usr/bin/openstack token issue --format value' returned 1: Unable to establish connection to https://192.0.2.2:13000/v3/auth/tokens (tried 30, for a total of 170 seconds)

No other errors prior to this.

You can check the logs here

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/9697c38/logs/undercloud/var/log/undercloud_install.txt.gz#_2016-10-18_06_35_22_000

[Gabriele Cerami (gcerami)]

Debugging showed one problem with two major causes
the problem

Keystone is unable to contact ceilometer from haproxy port 13000 on undercloud, haproxy is not running.

the causes
1 - ceilometer binds on the same port on 0.0.0.0 as haproxy should, haproxy fails on binding.
2 - puppet haproxy module is unable to detect that haproxy is not started correclty, and signals a success.

[Gabriele Cerami (gcerami)]

cause #2 (haproxy) it's actually a bug in haproxy code itself. Haproxy logs show

Oct 18 09:25:28 tripleo-centos-7-tripleo-test-cloud-rh1-4970270 haproxy-systemd-wrapper[29397]: [ALERT] 291/062528 (29398) : Starting proxy ceilometer: cannot bind socket [192.0.2.3:8777]
Oct 18 09:25:28 tripleo-centos-7-tripleo-test-cloud-rh1-4970270 haproxy-systemd-wrapper[29397]: haproxy-systemd-wrapper: exit, haproxy RC=256

A RC of 256 is actually cut down to 0 in exit code standard (only least significant 8 bits of the exit code are actually used) So the wrapper returns as if it succeded.

[Gabriele Cerami (gcerami)]

https://review.openstack.org/#/c/388035/ was proposed to fix the ceilometer/haproxy bind conflic problem

[Gabriele Cerami (gcerami)]

posted a bug for haproxy here
https://<email address hidden>/msg23896.html