If master node fails over, horizon session broken in browser

[Matthew Mosesohn]

This may be a Horizon bug or it may be how our implementation works.

How to reproduce:
1 - Deploy HA ubuntu (any network and storage type)
2 - Log into horizon
3 - crm_resource -r vip__management_old --move --node (any other node than current master)
4 - Refresh horizon or try to log in again

Result:
504 Gateway Time-out
The server didn't respond in time.

[Matthew Mosesohn (raytrac3r)]

https://bugs.launchpad.net/fuel/+bug/1285449 may be related

[Ryan Moe (rmoe)]

I believe it's 100% related to that bug. Dmitry and I ran into this repeatedly when working on 1285449. It appears that other services don't handle the VIP changing gracefully. This isn't related to rabbit either as we only saw this once we changed all services to talk to rabbit directly. When this issue occurs you can see that sometimes CLI commands fail to return results.

For example "nova list" will return a 504 (only sometimes though) and looking in the logs you can see that the Nova client makes an API call to the Neutron client which never succeeds. Nova is configured to talk to Neutron over the VIP. All services talk to Keystone this way as well. We saw issues where restarting Keystone on a node fixed all CLI commands and subsequently Horizon too.

[Sergey Vasilenko (xenolog)]

may be it's happens because Horizon store his session in memcache.
When controller was dead part of memcache servers go out.

[Ryan Moe (rmoe)]

It's not just Horizon that breaks though. All CLI commands stop working too.