Comment 14 for bug 1338608

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

There's no keystone or nova logs in the snapshot from comment #12, even though ps output indicates that both were running at the time when snapshot was generated. Based on haproxy.log, I can derive that at the time the failure occurred, keystone wasn't running or at least wasn't responding to haproxy:

2014-10-28T16:10:56.011090+00:00 err: (/Stage[main]/Osnailyfacter::Cluster_ha/Nova_floating_range[10.108.36.128-10.108.36.254]) Could not evaluate: Authentication failed with response code 504

2014-10-28T16:09:57.895115+00:00 alert: Server keystone-1/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2014-10-28T16:09:57.961115+00:00 alert: Server keystone-1/node-4 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2014-10-28T16:09:58.093883+00:00 alert: Server keystone-2/node-3 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2014-10-28T16:09:58.160867+00:00 alert: Server keystone-2/node-4 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2014-10-28T16:10:55.999073+00:00 info: 10.108.37.3:33993 [28/Oct/2014:16:09:55.991] keystone-1 keystone-1/node-1 0/0/1/-1/60004 504 194 - - sH-- 1/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"