init-keystone fails with ConnectionRefused

[Giulio Fidente]

init-keystone against overcloud fails CI with the following in console.html

keystoneclient.openstack.common.apiclient.exceptions.ConnectionRefused: Unable to establish connection to http://192.0.2.6:35357/v2.0/OS-KSADM/roles

a typical failure with stacktrace:

http://logs.openstack.org/50/108650/10/check-tripleo/check-tripleo-novabm-overcloud-f20-nonha/1e698b1/console.html#_2014-08-04_11_21_09_661

keystone seems to be up; seems to me a networking issue, maybe a race when bringing up the additional VIP on the controller

[Jan Provaznik (jan-provaznik)]

The issue is that haproxy reload on machine with VIP fails, then because haproxy is not running keystone is not accessible through VIP. I agree that it looks like a race issue when bringing up keepalive and haproxy services This is relevant part from my log (looks like haproxy reload is done when keepalived setup is not finished yet):

Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 Keepalived_vrrp[7065]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 Keepalived_vrrp[7065]: Configuration is using : 62226 Bytes
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 Keepalived_vrrp[7065]: Using LinkWatch kernel netlink reflector...
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 Keepalived_vrrp[7065]: VRRP sockpool: [ifindex(5), proto(112), unicast(0), fd(10,11)]
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 os-collect-config[780]: Redirecting to /bin/systemctl reload haproxy.service
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 systemd[1]: Reloading HAProxy For TCP And HTTP Based Applications.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'keystone_admin' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'keystone_public' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'horizon' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'neutron' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'cinder' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'glance_api' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'glance_registry' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'heat_api' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'heat_cloudwatch' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'heat_cfn' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'nova_ec2' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvdgd5 bash[7075]: [WARNING] 215/184222 (7075) : config : log format ignored for proxy 'nova_osapi' since it has no log address.
Aug 04 18:42:22 overcloud-controller1-3qnbafdvd...

[Ben Nemec (bnemec)]

This is a huge problem in the gate right now. Bumping to critical.

[Ben Nemec (bnemec)]

This appears to be a bug in the haproxy reload on Fedora. I can reproduce the same failure on a clean VM by simple installing haproxy, starting it, and trying to reload. I get the same error in the haproxy log that I'm seeing on our CI runs.

See https://bugzilla.redhat.com/show_bug.cgi?id=1126955

[Ben Nemec (bnemec)]

Bleh, forgot to reference this bug in the revert commit. This is fixed by https://review.openstack.org/#/c/112091/

[James Polley (tchaypo)]

Ben, I've assigned to you as you had flagged this as fixed. Please feel free to reassign to someone more appropriate, or me if there's no-one else.